Phosphorus (P) is an essential element for plant growth, so proper application of P fertilizers to farmland is necessary. High levels of P fertilization often cause P accumulation in soil and thereby increase P loss to the environment. The effect of long-term P fertilization on soil inorganic P (Pi) fractions and available P (Pa) stocks were investigated in order to provide a reference for rational management of P fertilization. A 27-year experiment was initiated in September 1984 in Changwu County on the southern part of the Loess Plateau, northern China. The experiment included five treatments of P fertilization: 0, 20, 40, 60, and 80 kg P ha−1. With zero P application, soil Pi fractions decreased and were even depleted over time. In fertilized plots, soil Pi content in available and unavailable fractions increased over time, regardless of the application rate. P fertilization increased the content and change rate of soil Pi fractions between three sampling times (1991, 2001, and 2011). Soil Pa stocks and plant P uptake increased with increasing application rate of 20–60 kg P ha−1, indicating increased input and output of P in the soil–plant system. Higher application rates (≥60 kg P ha−1) did not change soil Pa stocks or plant P uptake but increased the annual change rate of Pi fractions, maintaining a balance between the supply and demand of P. This study has implications for reducing P fertilization level and decreasing associated environmental risks in agricultural soil on the Loess Plateau.