Lightweight casting aluminum structural components, in particular shaped castings, are usually designed on the criteria of either yield strength or stiffness (Young’s modulus). Currently, there are limited options for the aluminum alloys with outstanding Young’s modulus. Following our review on the stiffness of aluminum-based composites and bimetallics, this review focuses on the effect of different chemical elements on the Young’s modulus of cast aluminum alloys and their microstructures. The influence of alloying elements on the Young’s modulus depends on the state. If the alloying elements are in a solid solution phase, the magnitude of the Young’s modulus is determined by the nature of the atomic interactions. If the alloying elements form second phases, the magnitude of the Young’s modulus is determined by the volume fraction and the intrinsic modulus of the second phase. Among the available elements, Be, Li, Si, Mn, and Ni are favorite candidates to enhance the modulus of cast aluminum alloys. The increase of Young’s modulus in commercial cast aluminum alloys can be significantly improved through adding appropriate alloying elements. Therefore, further research into the improvement of Young’s modulus of aluminum alloys is necessary.