Pseudopotentials simulate the interaction of the valence-electron system with frozen atomic cores using radially nodeless pseudo-orbitals. This leads to computational simplifications absent in frozen-core all-electron calculations. It is argued here that applying density fitting allows for using essentially the same (reduced) auxiliary basis sets for the valence interaction in both pseudopotential and all-electron calculations. We furthermore show that reduced auxiliary basis sets may also be made use of for fitting core Coulomb and exchange operators beyond on-site matrix elements. This leads to efficient substitutes for pseudopotentials in frozen-core all-electron calculations. At some pilot examples (Aun, HfO, RnF6), we demonstrate the possibility to systematically improve the accuracy of standard pseudopotential calculations with limited additional computational effort.