From millimeter observations of classical T Tauri stars, it is suggested that dust grains in circumstellar disks have grown to millimeter size or larger. However, gas drag on such large grains induces rapid accretion of the dust. We examined the evolution of dust disks composed of millimeter sized grains, and showed that rapid accretion of the dust disk causes attenuation of millimeter continuum emission. If a dust disk is composed mainly of grains of 1 cm - 1 m, its millimeter emission goes off within 10 Myr. Hence, grains in this size range cannot be a main population of the dust. Considering our results together with grain growth suggested by the millimeter continuum observations, we expect that the millimeter continuum emission of disks comes mainly from grains in a narrow size range of 1 mm -1 cm. This suggests either that growth of millimeter sized grains to centimeter size takes more than 10 Myr, or that millimeter sized grains are continuously replenished. In the former case, planet formation is probably difficult, especially in the outer disks. In the latter case, reservoirs of millimeter grains are possibly large (> 10 m) bodies, which can reside in the disk more than 10 Myr. Constraints on the grain growth time-scale are discussed for the above two cases.