A sampling device that can collect high-quality images of wheat grain was designed, and the image was processed by Mask-RCNN algorithm. Then the calculation model which can calculate the impurity rate and breakage rate based on image information was established. Wheat grain images collected in the field were labeled and trained. The trained algorithm can identify the number of wheat kernels, the pixel area of damaged wheat kernels, and the pixel area of impurities in the image. Then the three data were input into the calculation model to obtain the impurity rate and damage rate. This protocol enabled the fast recognition and segmentation of wheat grain images with 90.46% recognition accuracy for intact wheat seeds, 88.85% and 90.32% segmentation accuracy for broken wheat seeds and impurities. The relative error of the method for monitoring the impurity rate and breakage rate of wheat is less than or equal to 9.86%. The research in this paper can monitor the wheat impurity rate and breakage rate when the combine harvester is harvesting wheat, which helps the driver adjust the combine harvester in time to improve the harvesting efficiency, and it can provide a reference basis for intelligent regulation of combine harvesters.