Quantiles of the distribution of treatment effects

​

 

My research builds on a newly developed method for identifying features, such as percentiles (or more generally: ‘quantiles’), of the distribution of treatment effects. The method allows researchers to understand, in unprecedented detail, the distribution of treatment effects and parameters that depend on it, including the fraction of beneficiaries (winners) and those harmed (losers), and the gains and losses resulting from policy interventions. 

 

Estimating quantiles of the distribution of treatment effects (QDTE) under plausible assumptions is ground-breaking as it reveals if an intervention affects people equally or if some benefit while others are being harmed. This is crucial, for example, if a welfare reform increases earnings of workers for certain segments of the distribution while simultaneously reducing earnings for other segments. Relying solely on average treatment effects (ATE) might obscure this heterogeneity. 

 

The study of heterogeneous treatment effects has traditionally relied on average treatment effects within subgroups that share the same observed characteristics. Recently, more adaptable approaches for estimating subgroup-specific average treatment effects have emerged, often leveraging machine learning techniques that rely on large-scale datasets. While subgroup analysis can yield valuable insights, it may overlook critical aspects of heterogeneity when essential information is unobserved. Most notably, it cannot be used to determine the distribution of treatment effects and the parameters derived from it.

​

Researchers have also made use of quantile treatment effects (QTE) to study heterogenous treatment responses. QTE are defined as differences between quantiles of separate marginal distributions of treatment and control outcomes. While experiments are informative about ATE without further assumptions, QTE rely on a rank invariance assumption. Rank invariance is a strong assumption because it implies that observation units maintain their relative position in the potential outcomes distribution regardless of whether they are being assigned to the treatment group or the control group. QTE are different from QDTE if the rank invariance assumption is violated. (More precisely, QDTE are only identical to QTE if the rank invariance assumption holds and if the QTE are monotonically increasing along the distribution.)

​

My newly developed method can be used to identify QDTE under plausible assumptions that can either be tested or subjected to sensitivity checks to assess their realism. Specifically, the method uses rank correlation coefficients between actual and predicted control state outcomes under the testable assumption that highly predictive covariates are available. In the case of random assignment, these rank correlation coefficients are identical, regardless of whether they are calculated using in-sample predictions or out-of-sample predictions. Assuming that all permutations of observation units satisfying this property are equally likely, it becomes possible to identify the rank correlation coefficient between potential treatment and control outcomes, along with the corresponding QDTE. 

 

Reweighting techniques can be applied to extend the method to situations in which treatment assignment is as good as random after controlling for a set of covariates. Monte Carlo simulations demonstrate that the estimators are unbiased, consistent, and asymptotically normal. Rearranging QDTE yields generalised quantile treatment effects (GQTE), which do not require a rank invariance assumption but align with conventional QTE under rank invariance.