Strong Coupling of Resonant Metasurfaces with Epsilon-Near-Zero Guided Modes

We study, both theoretically and experimentally, strong interaction between a quasi-bound state in the continuum (QBIC) supported by a resonant metasurface with an epsilon-near-zero (ENZ) guided mode excited in an ultrathin ITO layer. We observe and quantify the strong coupling regime of the QBIC-ENZ interaction in the hybrid metasurface manifested through the mode splitting over 200 meV. We also measure experimentally the resonant nonlinear response enhanced near the ENZ frequency and observe the effective nonlinear refractive index up to ∼4 × 10–13 m2/W in the ITO-integrated dielectric nanoresonators, which provides a promising platform for low-power nonlinear photonic devices.