Time series are widely used in many classification and regression tasks. However, numerous time series contain unavoidable missing data, making it challenging to model the temporal dynamics of sequential data. Various data imputation methods have been proposed to infer missing values in time series. Although sequences recorded at fixed time intervals are presented in discrete form, they possess an inherent temporal continuity, which is ignored in most existing approaches. In this paper, we propose an end-to-end Attentive Continuous-Time Generative Adversarial Network (ACGANet) to estimate unobserved values in irregular sequences. ACGANet captures the temporal dynamics by transforming the discrete sequence into the continuous-time flow, thereby modeling the underlying distribution of the real data. Furthermore, ACGANet employs an adversarial learning strategy to alleviate the error introduced by imputed values, with the discriminator distinguishing between real and generated samples. Additionally, ACGANet introduces the log-density of hidden temporal states as an auxiliary loss to further optimize the generator. This allows the model to simultaneously focus on the overall temporal dynamics of the time series and the underlying distribution of the missing data. Extensive experiments on three publicly available real-world datasets demonstrate that ACGANet achieves state-of-the-art performance in imputing incomplete time series. Moreover, both qualitative and quantitative analyses validate the effectiveness of the proposed model.