Pests and diseases are the major concerns in plantain tree cultivation. Thrips are one of the major pests found in banana fields. It is also known as Thysanoptera. These are the tiny insets about 1-3 mm in size and about 5500 species, are the widely seen pests in the plantain trees of India. Their attack is infecting the health and growth of the host plants. The detection of thrips at the early stages is very difficult because of their minute size. Traditional methods such as printed dichotomous taxonomic and genetic markers require laboratory facilities and human expertise. Automated identification of pests is essential for controlling various plant diseases effectively. Deep learning techniques, especially, Convolutional Neural Networks (CNNs) is already established their proficiency in the automatic detection of diseases and pests in humans. This research work proposes a CNN model that combines the efficiency of the Visual Geometric Group (VGG) model with the Xgboost technique for identifying pests. The experiments are carried out on banana plants for detecting the thrips. Banana image dataset is a real-time dataset collected from the state named Tamil Nadu situated in the Southern part of India. The proposed model is compared with the basic VGG-19 model and the result shows the proposed Xgboost-VGG model outperforms in terms of the evaluation metrics: recall, accuracy, f1-score, precision, and specificity.