Background Hepatocellular carcinoma (HCC) and cholangiocarcinoma (CCA) represent the primary liver cancer types. Traditional diagnostic techniques, reliant on radiologist interpretation, are both time-intensive and often inadequate for detecting the less prevalent CCA. There is an emergent need to explore automated diagnostic methods using deep learning to address these challenges. Methods This study introduces HTRecNet, a novel deep learning framework for enhanced diagnostic precision and efficiency. The model incorporates sophisticated data augmentation strategies to optimize feature extraction, ensuring robust performance even with constrained sample sizes. A comprehensive dataset of 5,432 histopathological images was divided into 5,096 for training and validation, and 336 for external testing. Evaluation was conducted using five-fold cross-validation and external validation, applying metrics such as accuracy, area under the receiver operating characteristic curve (AUC), and Matthews correlation coefficient (MCC) against established clinical benchmarks. Results The training and validation cohorts comprised 1,536 images of normal liver tissue, 3,380 of HCC, and 180 of CCA. HTRecNet showed exceptional efficacy, consistently achieving AUC values over 0.99 across all categories. In external testing, the model reached an accuracy of 0.97 and an MCC of 0.95, affirming its reliability in distinguishing between normal, HCC, and CCA tissues. Conclusion HTRecNet markedly enhances the capability for early and accurate differentiation of HCC and CCA from normal liver tissues. Its high diagnostic accuracy and efficiency position it as an invaluable tool in clinical settings, potentially transforming liver cancer diagnostic protocols. This system offers substantial support for refining diagnostic workflows in healthcare environments focused on liver malignancies.