Change detection (CD) represents a critical task in the analysis of remote sensing (RS) images. In particular, the CD of heterogeneous images that integrate optical and synthetic aperture radar (SAR) modalities has attracted growing attention owing to their complementary characteristics. However, the optical and SAR images (OSIs) CD tasks face specific issues including structural inconsistency between modalities, speckle noise in SAR images, and insufficient utilization of complementary cross-modal features, resulting in limitations in feature representation, temporal causal modeling, and feature fusion for existing methods. Therefore, proposes an end-to-end causal attention and frequency-domain gating fusion enhanced dual-branch OSIs CD network (CAFGFNet). This network adopts a dual-branch encoding architecture with modality-specific designs, where the optical branch emphasizes preserving spectral and texture features, and the SAR branch integrates denoising with deformable convolution to mitigate speckle noise and geometric deformation. On this basis, the Causal Attention (CA) mechanism is introduced to model temporal dependencies to suppress false changes, and the Frequency-domain Gating Fusion Module (FGFM) is designed to adaptively fuse complementary frequency components from optical and SAR features, thus effectively boosting the accuracy and robustness of CD. Compared with six state-of-the-art (SOTA) methods, CAFGFNet has increased the relative suboptimal indicators of Precision by 2.2% and 8.93% on two public datasets. The experimental results show that CAFGFNet can achieve high-precision detection of changes in OSIs while ensuring the full expression and effective fusion of cross-modal features.