Mathematics skills relate to lifelong career, health and financial outcomes. Individuals’ cognitive abilities predict mathematics performance and there is growing recognition that environmental influences, including differences in culture and variability in mathematics engagement, also affect mathematics performance. In this Review, we summarize evidence indicating that differences between languages, exposure to maths-focused language, socioeconomic status, attitudes and beliefs about mathematics, and engagement with mathematics activities influence young children’s mathematics performance. These influences play out at the community and individual levels. However, research on the role of these environmental influences for foundational number skills, including understanding of number words, is limited. Future research is needed to understand individual differences in the development of early emerging mathematics skills such as number word skills, examining to what extent different types of environmental input are necessary and how children’s cognitive abilities shape the impact of environmental input. Children’s individual abilities and environment influence their mathematics skills. In this Review, Silver and Libertus examine how language, socioeconomic status and other environmental factors influence mathematics skills across childhood, with a focus on number word acquisition.