Abstract Purpose The aim of this study was to evaluate the novel rhein derivative Y01 for its anti-HCC activity and potential molecular mechanisms, thereby positioning it as a promising candidate for HCC therapy. Methods CCK-8, EdU incorporation and clone formation assay were employed to assess the impact of Y01 on the cell viability and proliferation of HCC cells. While apoptosis was assessed through flow cytometry and western blotting techniques. Additionally, the impact of Y01 on cell mobility was evaluated via wound healing and transwell migration assays, with western blotting analyses providing further insights. Mechanistically, transcriptomics and western blotting assays were used to explore the potential signaling pathways. Results Y01 markedly suppressed the growth, colony formation, and migratory capacity of HCC cells, induced apoptosis and affected the expression of apoptosis-related proteins. Transcriptomics initially pointed toward the PI3K/AKT/mTOR pathway as a potential target, which was corroborated by western blotting results showing decreased levels of phosphorylated PI3K, AKT, and mTOR following Y01 treatment, highlighting its role in mediating the compound’s anticancer effects. Conclusions Y01 inhibited the proliferation, migration, and induced apoptosis of HCC cells possibly by blocking PI3K/AKT/mTOR signaling pathway.