More and more deep learning techniques have been used to classify pests in recent years, but most of the research objects involved in the existing work are adult pests. The identification of adult pests will delay the best time for pest control. Therefore, this paper aims to improve the identification of pest larvae. The classification of these similar caterpillars can be regarded as fine-grained image classification. However, taking existing fine-grained classification methods to classify the pest larvae still has the following problems. First, the object categories covered by the existing standard fine-grained image datasets (such as Birds, dogs, cars, and airplanes) have the same characteristics, and the various parts of the object are distinguished, with prominent outlines. However, the body parts of some larvae are not visually prominent. Second, compared with the samples in the standard data set, the background environment of the larvae samples is mostly more complicated, and some larvae use mimicry to disguise themselves, which brings difficulties to identification and classification. This paper proposes a fine-grained image classification algorithm that can extract more subtle features to solve the above problems. Experiments show that the model proposed in this paper surpasses the existing general image classification model and fine-grained image classification model on the agricultural pest fine-grained dataset AgrFIP20 and the large-scale agricultural pest dataset AgrIP138.