(Ba,Ca)TiO$_3$ and Ba(Ti,Zr)O$_3$ solid solutions are the building blocks of\nlead-free piezoelectric materials that attract a renewed interest. We\ninvestigate the properties of these systems by means of first-principles\ncalculations, with a focus on the lattice dynamics and the competition between\ndifferent ferroelectric phases. We first analyze the four parent compounds in\norder to compare their properties and their different tendency towards\nferroelectricity. The core of our study is systematic characterization of the\nbinary systems (Ba,Ca)TiO$_3$ and Ba(Ti,Zr)O$_3$ within both the Virtual\nCrystal Approximation and direct supercell calculations. In the case of Ca\ndoping, we find a gradual transformation from $A$-site to $B$-site\nferroelectricity due to steric effects that largely determines the behavior of\nthe system. In the case of Zr doping, in contrast, the behavior is eventually\ndominated by cooperative Zr-Ti motions and the local electrostatics. In\naddition, our comparative study reveals that the specific microscopic physics\nof these solid sets severe limits to the applicability of the Virtual Crystal\nApproximation for these systems.\n