Tumor vascular microenvironment (TVME) critically governs biological properties in primary liver cancers (PLC), yet noninvasive tools to decode its heterogeneity remain clinically unavailable. In this study, data from six clinical cohorts encompassing hepatocellular carcinoma (HCC), intrahepatic cholangiocarcinoma (ICC), and combined hepatocellular-cholangiocarcinoma, were comprehensively analyzed. Based on quantitative vascular features extracted from computed tomography (CT) images, a novel multi-task learning computational framework (MTV-Net) was constructed to generate two imaging biomarkers: TAVSPHE for classifying PLC based on vascular phenotype similarity to HCC or ICC, and TAVSRE for predicting post-resection recurrence risk. Patients classified as "ICC-like" by TAVSPHE exhibited significantly worse survival outcomes than "HCC-like" counterparts. Meanwhile, TAVSRE effectively stratified recurrence risk across all three PLC subtypes: high-risk groups showed substantially higher recurrence rates compared to low-risk groups (all P < 0.001) and enhanced risk discrimination when integrated with established clinical staging systems. The resulting MTV-Net-Clinic model demonstrated superior prognostic accuracy, with concordance index ranging from 0.731 to 0.823 across validation cohorts. Radiogenomics analysis revealed that enrichment of the extracellular matrix remodeling signaling pathway underlies the shared biological foundation of the two biomarkers. Collectively, MTV-Net serves as a TVME-targeted computational framework, enabling PLC reclassification from routine CT scans and thereby improving prognostic stratification.