Change detection (CD) identifies pixel-level land cover changes in multi-temporal remote sensing imagery, playing a critical role in environmental monitoring and disaster management. Despite the advantages of high-resolution imagery in providing detailed spatial and contextual information, it significantly intensifies the complexity of fine-grained CD by exacerbating sensitivity to spatial misalignment. While accurate CD conventionally hinges on precise image registration, inevitable misalignments in practical scenarios often lead to spurious change responses and compromise the stability of feature representation learning. To address these challenges, this paper proposes the Intra- and Cross-temporal Graph Interaction Network (ICGINet), a novel architecture that enhances CD robustness under registration uncertainty by modeling intra- and cross-temporal dependencies through adaptive graph-based message passing and relational reasoning. Specifically, ICGINet employs a dual-branch design that combines convolutional neural networks (CNNs) for local semantic extraction with graph neural networks (GNNs) for global relational modeling, enabling the integration of fine-grained spatial details and long-range contextual dependencies. To reconcile intra-temporal features from these heterogeneous branches, a Dual-Focus Feature Fusion (DFFF) module is proposed to facilitate attention-guided information exchange and semantic alignment. Furthermore, an Adaptive Neighborhood Graph Awareness (ANGA) module constructs a displacement-aware dynamic graph to capture cross-temporal neighborhood correspondences, thereby mitigating spatial misalignment during temporal interaction. Comprehensive experiments on two challenging datasets demonstrate that ICGINet achieves superior performance compared to state-of-the-art methods, both in terms of accuracy and robustness to registration error.