DEEP: Differentiable Explicit Projection for Deep Networks with Unitary Constraints

Maintaining stability in deep and recurrent neural networks is essential, yet gradient vanishing and explosion remain major challenges. In this paper, we introduce DEEP (Differentiable Explicit Projection), a framework that enforces unitary constraints on network weights through a fully differentiable explicit projection mechanism. DEEP leverages the Björck iteration to efficiently project unconstrained weights onto the unitary manifold, preserving norms and inner products while remaining fully compatible with backpropagation. Unlike previous implicit or computationally expensive methods, DEEP achieves strict orthogonality with minimal computational overhead. We evaluate DEEP across multiple benchmarks, including random unitary matrix learning, long-sequence Copying Tasks, pixel-by-pixel MNIST, and convolutional networks on CIFAR-10, demonstrating that a small number of Björck iterations is sufficient to maintain numerical stability, improve training convergence, and enhance overall performance. These results highlight DEEP as a practical and general approach for enforcing unitary constraints, offering an effective trade-off between stability, efficiency, and predictive accuracy in deep learning.