Seed size is an important agronomic trait determining crop yield. Identifying key genes involved in seed size regulation and elucidating their molecular mechanisms are of great significance for crop breeding. Recent studies in crops have uncovered numerous genes that control seed size and weight, many of which function by modulating phytohormone biosynthesis, metabolism, or signaling pathways. This review provides a comprehensive overview of the genetic and molecular mechanisms by which phytohormones regulate seed size and weight and their cross-talks in modulating seed size. We highlight the functional conservation and divergence of homologous genes that control seed size across species. A particular focus is placed on those genes that have promising potential for yield improvement. Finally, we discuss current challenges in phytohormone regulation of seed size and molecular design breeding strategies for translating this knowledge into crop improvement.