The accuracy and speed of tomato detection were increased to facilitate fully automated harvesting by improving the Real-Time Detection Transformer (RTDETR) to develop a new lightweight tomato detection model called LiteTom-RTDETR. This model employed RepViT as a lightweight backbone network instead of the original RTDETR backbone, considerably reducing both the number of parameters in and computational complexity of the model. Furthermore, a context guide fusion module was designed to enhance multiscale feature extraction efficiency, and an adaptive sliding weight mechanism was integrated into the loss function to mitigate class-imbalance issues. The proposed LiteTom-RTDETR model was shown to balance high tomato identification accuracy (88.2%) with an excellent real-time inference speed (52.2 frames per second) and computational efficiency (36.3 GFLOPs). Notably, the average detection accuracy of LiteTom-RTDETR was 0.6% higher, its detection speed was 15.5% faster, and its computational load and model size were 36.2% and 31.6% smaller, respectively, than those of the original RTDETR model. Therefore, the proposed model provides a practical approach for realizing visual recognition tasks in resource-constrained mobile automated tomato harvesting equipment.