We propose a mechanism to understand the phenomenon of self-organization of the stylized facts in financial markets. This question is addressed within a workable agent-based model which is particularly simple and mathematically well posed. A key element is the non-stationarity of the number of agents, that can enter or exit the market depending on the signal they perceive. This leads to a market dynamics which evolves spontaneously towards a regime with stylized facts. All situations without stylized facts are shown to be unstable. The stylized facts correspond to finite-size effects with respect to time and number of agents. These results have conceptual and practical implications that can be tested with suitable data. Copyright (C) EPLA, 2009