Black tomatoes are typically classified according to their ripeness immediately after harvesting to maintain optimal quality and minimize the loss caused by uneven ripening. The ripeness of black tomatoes is traditionally evaluated either visually or using a colorimeter. The visual observation technique is time- and labor-intensive and may yield unreliable results, and the colorimeter-based technique can be implemented for only a small number of tomatoes. To address these problems, this paper proposes a method to effectively classify the ripeness of black tomatoes based on machine vision and YOLOv4. A total of 4,080 digital images of Black Change, a variety of black tomatoes, were collected by capturing the top, bottom, left side, and right side of each sample in illuminance conditions of 570 lx, 1,240 lx, and 2,780 lx. The results showed that the model trained with images gathered under a single illuminance condition could effectively classify the ripeness for images with the same condition. When images with mixed lighting conditions were used, the model achieved a classification performance of nearly 100%. However, its performance deteriorated when the model trained with an independent illuminance condition applied to the images for other conditions. The model trained with over 1632 images in mixed illuminance conditions for over 3000 iterations achieved a classification accuracy of at least 96.00%, and time required for image collection, labeling, and training was minimized.