In recent years, Physics-Informed Neural Networks (PINNs) have emerged as a powerful alternative to traditional numerical methods for solving partial differential equations (PDEs).Despite their remarkable success, growing evidence indicates that standard PINNs can sometimes fail to deal with complicated PDEs.Existing research has primarily improved PINNs through optimising the training process.Comparatively less attention has been paid to limitations arising from the neural architecture itself.Conventional multilayer perceptrons used in PINNs suffer from insufficient nonlinear expressiveness, exhibit performance degradation as network depth increases, and show internal covariate shift (ICS) during training.These unresolved architectural limitations restrict the accuracy, scalability, and robustness of PINNs.This thesis addresses these gaps through three architectural developments.The practical value of these developments is demonstrated in fracture mechanics by combining Deeper-PINNs with a residual-guided adaptive refinement strategy in a Deep Ritz formulation.The resulting framework accelerates convergence and improves the prediction of displacement, stress, and fracture parameters, particularly near crack tips and material interfaces.Overall, this thesis advances PINN architectures in terms of expressiveness, scalability, and stability and demonstrates their applicability to complex engineering analysis.