A Note on Penalized Spline Smoothing With Correlated Errors

AbstractWe investigate the behavior of data-driven smoothing parameters for penalized spline regression in the presence of correlated data. It has been shown for other smoothing methods that mean squared error minimizers, such as (generalized) cross-validation or the Akaike information criterion, are extremely sensitive to misspecifications of the correlation structure resulting in over- or (under-)fitting the data. In contrast to this, we show that a maximum likelihood–based choice of the smoothing parameter is more robust and that for a moderately misspecified correlation structure over- or (under-)fitting does not occur. This is demonstrated in simulations and data examples and is supported by theoretical investigations.KEY WORDS: Correlation structure misspecificationLinear mixed modelSmoothing parameter selection