Propensity score estimators for the average treatment effect and the average treatment effect on the treated may yield very different estimates

OBJECTIVE: Propensity score matching is typically used to estimate the average treatment effect for the treated while inverse probability of treatment weighting aims at estimating the population average treatment effect. We illustrate how different estimands can result in very different conclusions. STUDY DESIGN: We applied the two propensity score methods to assess the effect of continuous positive airway pressure on mortality in patients hospitalized for acute heart failure. We used Monte Carlo simulations to investigate the important differences in the two estimates. RESULTS: Continuous positive airway pressure application increased hospital mortality overall, but no continuous positive airway pressure effect was found on the treated. Potential reasons were (1) violation of the positivity assumption; (2) treatment effect was not uniform across the distribution of the propensity score. From simulations, we concluded that positivity bias was of limited magnitude and did not explain the large differences in the point estimates. However, when treatment effect varies according to the propensity score (E[Y(1)-Y(0)|g(X)] is not constant, Y being the outcome and g(X) the propensity score), propensity score matching ATT estimate could strongly differ from the inverse probability of treatment weighting-average treatment effect estimate. We show that this empirical result is supported by theory. CONCLUSION: Although both approaches are recommended as valid methods for causal inference, propensity score-matching for ATT and inverse probability of treatment weighting for average treatment effect yield substantially different estimates of treatment effect. The choice of the estimand should drive the choice of the method.