Inorganic crystals doped with luminescent ions have attracted considerable interest due to their abundant luminescence spectra. Particularly, these luminescent materials can show highly variable luminescence in response to external stimuli, rendering widespread applications ranging from super-resolution techniques to high-security information encryption. In this Perspective, we highlight the recent developments of controlling optical response in doped inorganic crystals under various stimuli, including light, temperature, water/humidity, and mechanical force. Moreover, we delve into the underlying mechanism and discuss relevant applications, simultaneously pointing out challenges and future directions for designing superior stimuli-responsive materials.