High-accuracy recognition of litchi fruits is the key to yield estimation. To solve the problem of low detection rates of dense, small targets in orchard scenes, this paper proposes an improved YOLOv3 method for litchi fruit recognition. The litchi fruit target anchor frames in the dataset are re-clustered to obtain nine predefined anchor frames. The prediction scale of the network is adjusted, a 160 × 160 feature prediction scale is added to improve the detection of small targets, and the feature prediction scale for large-target detection in YOLOv3 is removed to simplify the network model. A dense connection module is added to the feature-extraction network to enhance the feature propagation capability and improve network performance. To train and test the litchi fruit dataset, immature (young fruit and expansion stage) and mature stages were constructed in an orchard scene, and 227018 labels were generated using LabelImg software. The F1 score and mean accuracy precision (mAP) were used to evaluate the network model, and the proposed method was experimentally compared with YOLOv3 for litchi images. The experiments showed that the recognition effect was significantly improved using the proposed model, with F1 and mAP of 0.851 and 88.9%, respectively, on the full dataset, which are better than YOLOv3 by 0.238 and 31.1 percentage points, respectively. Therefore, the method has a good effect for dense litchi fruit recognition in orchard scenarios and provides technical support for litchi yield estimation.