Using covariates to regress out unwanted effects ...

My take on Simpson's Paradox, Lord's Paradox, and Suppression Effect

Reflections inspired by Tu, Y-K, Gunnell, D. & Gilthorpe, M.S. (2008). Simpson's Paradox, Lord's Paradox, and Suppression Effects are the same phenomenon - the reversal paradox. Emerging Themes in Epidemiology, 5.2

What Paradox?

The Simpson's Paradox [1] [2] is observed when the relationship between two variables differs within subgroups compared to that observed for the aggregated data. Typically, whilst regression lines within groups go in one direction, the regression line across groups go the other way, as it can be seen e.g. in a group x repeated measures design. This is explained by the dispersion of data between groups and their means showing opposite effects.

Tu et al. (2008) point out that 'Simpson's paradox has broad implications [...] since it indicates that making causal inference from any non-randomised study (e.g. cohort studies, case-control studies) can be difficult, because [...] there will always be the possibility that an unobserved and therefore unadjusted confounder might attenuate the association (or even reverse its direction) between the independent variables and dependent variables, due to the difference in the mean values or the distribution of confounders between the case or control group.'


Lord's paradox [3] [4] relates to a similar effects appearing in analyses of covariance (ANCOVA) and is observed when the relationship between a continuous dependent variable and a categorical (independent) variable is reversed when the covariate is introduced (i.e. ANOVA and ANCOVA give opposite results). A typical example is the use of a measurement made at baseline within a longitudinal study; the ANOVA shows some effect and this effect disappear or is reversed when baseline measurement is introduced.



The suppression effect [5] is observed when a potential covariate (x2) that is unrelated to the dependent variable (y) increases the overall model fit within regression when it is added to the model. This occurs when the additional covariate (x2) is positively correlated with a covariate already present in the design (x1), and this one does correlate with the data. The new covariate x2 'suppresses' the part of x1 that is uncorrelated with y, thereby increasing not only the overall predictability but also the coefficient of x1, while the coefficient for x2 becomes negative.

Illustration in neuroimaging

One major difference between epidemiology and neuro-imaging is that, 2 types of confonding variables can be distinguished: 1 - covariates that impact of the measurements and relates to the effects being studied, 2 - covariates that directly relate to the measurements.

Confounding variables in VBM studies

A classic example in VBM studies is the use of Total Intracranial Volume (TIV) or total gray matter volume, as confound. The reason is simple, we want to look for local changes in gray matter but voxel values in standard space are related to the total amount of warping performed, and thus on the total brain size. This is an example of confounding that relates to the measurement itself.

Another typical covariate in VBM studies is age. Because we know that age influences the local concentration / volume of gray matter, it is usually regressed out. This is an example of confounding that relates to the effect being studied and we state in the statistical model that gray matter is the sum an age effect and whatever other variable. Importantly, many biological and psychological variables are affected by age, and thus many VBM studies might be afflicted by the Lord's paradox or the suppression effect - but nobody (including me) ever checks.

Confounding variables in fMRI studies

A typical example of confounding variable that relates to the measurement is motion. Because a voxel time course necessarily reflect the effect observed and the motion history / correction applied, motion parameters are often used as covariates. Interestingly, motion by itself (assuming reasonable amount) doesn't explain that much of the data - it's the relationship motion / design that matters the most. For instance in Gorgolewski et al. (2012) [6] we showed that total displacement contributed a little to between session variance while the interaction motion / design matrix explained ~17% of the variance.

What can we do?

One of the point made by Tu et al. (2008) is that all 3 paradoxes reflect the same phenomenon: one can observe an effect or the absence of an effect between variables, and the introduction of a covariate changes the results - that is conditioning the data can dramatically modify the results in one direction or the other. An interesting point made by the author is that the introduction of such covariate should depends upon biological and clinical knowledge, not ad hoc statistical analyses and changes in the estimated effects.

The 1st thing to do is therefore to think about the statistical model: does the model represent accurately the data and how do the covariate relates to each other.

The 2nd thing to do is to run the model twice, with and without confounding variables, and also a model with those confounding variables only - this will allow inquire their impact on the data (where in the brain and how do they explain the data), and their impact on the model and variables of interest (which effects are changing, and how).

A 3rd and last thing to do, is to be explicit about the observed effects: say if an effect is there with/without the covariates and if some differences are observed try your best to understand and explain what is going on.

References

[1] Simpson EH: The interpretation of interaction in contingency tables. J R Stat Soc Ser B 1951, 13:238-41.
[2] Yule GU: Notes on the theory of association of attributes in statistics. Biometrika 1903, 2:121-34.
[3] Lord FM: A paradox in the interpretation of group comparisons. Psychol Bull 1967, 68:304-5
[4] Lord FM: Statistical adjustments when comparing preexisting groups. Psychol Bull 1969, 72:337-8
[5] Horst P: The role of prediction variables which are independent of the criterion. In The Prediction of Personal Adjustment. Edited by Horst P. New York: Social Science Research Council; 1941::431-6.
[6] Gorgolewski, KJ, Storkey AJ, Bastin ME, Whittle I, Pernet C. (2013). Single subject fMRI test-retest reliability metrics and confounding factors. NeuroImage, 69, 231-243.