Apples, a popular fruit, require thorough quality inspections to enhance their commercial value. To assess apple quality in terms of the presence of defects, this study proposes a real-time apple sorting system that combines a single RGB camera with the YOLOv7 object detection model and the SORT tracking algorithm. Using images from the apple sorting machine, the system trains the YOLOv7 model to detect five defect types, stems, and calyxes while differentiating between defective and non-defective apples. The SORT algorithm integrates multi-frame data, tracks, and keeps apple labels. With YOLOv7 achieving 71.7% mAP and 69.4 fps in defect detection and SORT achieving 91.2% MOTA and 85.2% MOTP, the proposed system demonstrates high efficiency. Combining detection and tracking yields an overall quality rating of 95.8% F1 and 60.9 fps. However, defect-specific detection, such as insect damage (46.9% mAP) and physical damage (38.3% mAP), requires improvement. This system—effective for apple quality inspection, shows potential for broader agricultural product sorting applications.