We investigate the nonlinearity of ultra-low loss Si3N4-core and SiO2-cladding rectangular waveguides. The nonlinearity is modeled using Maxwell's wave equation with a small amount of refractive index perturbation. Effective n2 is used to describe the third-order nonlinearity, which is linearly proportional to the optical intensity. The effective n2 measured using continuous-wave self-phase modulation shows agreement with the theoretical calculation. The waveguide with 2.8-μm wide and 80-nm thick Si3N4 core has low loss and high power handling capability, with an effective n2 of about 9×10(-16) cm2/W.