This paper presents a methodology for detecting unhealthy lemons in a lemon dataset based on a deep Convo-lutional Neural Network (CNN) and transfer learning. Initially, a CNN model was developed and trained, after which a new model was framed on the existing CNN model using transfer learning. The base model was trained to a satisfying level of accuracy (of 98.55%). Afterwards, we incorporated this pre-trained model into a customized transfer learning framework. The final model was tested using a different augmented lemon dataset. In this case, no additional training was performed on the transfer learning model and the resultant model achieved an accuracy of 95.91 %.