Challenges of interpreting patient reported outcomes from clinical trials

As Bell et al. (1) indicated in their paper entitled “Statistical Controversies in Cancer Research: Using standardized effect size graphs to enhance interpretability of cancer-related clinical trials with patient reported outcomes,” Patient Reported Outcomes (PROs) are frequently being gathered in clinical trials. The primary challenge is that these outcomes are multidimensional; rarely is there a single outcome but rather a number of symptoms measures or components of quality of life or health status. While there are formal statistical procedures to insure that the conclusions from a trial about any single outcome is valid (not a false positive), it does not address the clinically relevant questions as to whether the intervention is generally beneficial across the multiple outcomes.

Forest plots graphically display results from either multiple studies (e.g., meta-analyses) or multiple outcomes in a single trial. They date back to at least the 1970s and the name ‘forest plots’ appears to be first used in 1996 (2). The premise is that we can process information better visually than when presented in a tabular form. I know this is true for me even given my quantitative training as a statistician and in my experience this is also true for clinicians.

For continuous variables, as is typical of PRO measures, there are two versions of these plots. The first reports the measures relative to the precision that they are measured for the trial (e.g., the standard error) and thus the scale is related to the statistic that is used to test hypotheses. The second reports the measures relative to the variability of the outcome (e.g., the standard deviation) in the individuals being studied. In meta-analyses (trials of varying size) these two types of plots can look quite difference, but with multiple outcomes from the same trial (fixed size), they will appear quite similar, only different in the scale of the horizontal axis. But that scale is important to the issue of interpretation. When the scale is related to the standard error, it is influenced by the size of the trial and it is hard to compare to other trials. In contrast, when the scale is related to the standard deviation (e.g., standardized effect sizes) these effect size estimates can be easily compared across trials or with alternative interventions. Cohen (3) provided guidelines for interpreting effect size estimates with small, medium, and large effect sizes corresponding to D=0.2, 0.5 and 0.8.

So what is the challenge? There are two in my mind that are related. The first is familiarity with typical effect sizes for the type of intervention we are considering. For some clinical researcher and those working in certain fields (e.g., behavioral interventions), this concept is familiar. But for most practicing clinicians, standardized effect sizes are meaningless. The second is that continuous outcomes do not fit easily into the clinical decision making process (to treat with the interventions or not) (4). Following the recommendations of Bell et al. (1), I agree that the use of plots of standardized effect sizes when reporting trials with multiple PROs will improve the interpretability of these trials. But I also believe that we need to continue to think about how we can present results in a way that encourages implementation when warranted.

Acknowledgements

None.

Footnote

Conflicts of Interest: The author has no conflicts of interest to declare.

References

1. Bell ML, Fiero MH, Dhillon HM, et al. Statistical Controversies in Cancer Research: Using standardized effect size graphs to enhance interpretability of cancer-related clinical trials with patient reported outcomes. Ann Oncol 2017;28:1730-3. [Crossref] [PubMed]
2. Lewis S, Clarke M. Forest plots: trying to see the wood and the trees. BMJ 2001;322:1479-80. [Crossref] [PubMed]
3. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: Lawrence Earlbaum Associates, 1988.
4. Schünemann HJ, Akl EA, Guyatt GH. Interpreting the results of patient reported outcome measures in clinical trials: the clinician's perspective. Health Qual Life Outcomes 2006;4:62. [Crossref] [PubMed]