A holographic display enables three-dimensional scene reconstruction, but conventional methods suffer from high computational cost and artifacts in multi-depth scenes. We propose an end-to-end light-field hologram generation framework that integrates physics-based modeling with deep neural networks for efficient, high-fidelity hologram prediction. The framework directly maps light-field inputs to refocused representations and subsequently to phase-only holograms using real-valued and complex-valued networks under optical constraints, thereby avoiding redundant three-dimensional object computations while preserving physical consistency. Simulations and optical experiments demonstrate that the method yields high-quality reconstructions with smooth depth transitions and reduced edge artifacts while enabling fast and accurate holographic display.