Three-dimensional (3D) gravity inversion remains a fundamentally ill-posed and non-unique problem, where reconstructing subsurface density distributions from two-dimensional (2D) surface gravity observations is severely constrained by depth-dependent sensitivity decay and ambiguity in mass placement. Classical Tikhonov-based inversion algorithms partially alleviate these issues through explicit regularization but often suffer from the skin effect, oversmoothed deep anomalies, and substantial computational cost. Recent deep learning (DL) approaches provide a data-driven alternative; however, conventional convolutional architectures still struggle with semantic inconsistency during multi-scale feature fusion, limited depth-awareness, and extreme class imbalance in geophysical models.This paper presents GravityVNet, a physics-aware deep neural framework tailored for high-fidelity 3D gravity inversion. The architecture is built upon a Residual V-Net backbone enhanced with a 3D Coordinate Attention (CoordAtt) mechanism to encode long-range spatial dependencies and explicitly preserve depth-direction information, which helps compensate for depth-dependent attenuation effects commonly associated with the skin effect. To further improve structural recovery, we introduce Multi-Scale Attentional Feature Fusion (AFF) modules, which adaptively recalibrate skip-connected features and bridge semantic gaps between encoder and decoder pathways. A Deep Supervision training strategy, combined with a Hybrid Loss function integrating Focal Tversky Loss and Total Variation (TV) regularization, enforces boundary sharpness, suppresses artifacts, and strengthens convergence stability. Extensive experiments on large-scale, physics-consistent synthetic datasets generated by SimPEG indicate that GravityVNet achieves competitive performance compared to standard baseline methods. The results indicate that the model has improved in terms of geometric fidelity, quantitative accuracy, and physical consistency with the observed gravity response. In addition, the validation of the measured data in the Vinton Salt Dome also proves the effectiveness of the model in solving composite anomalies and deep structures.