The questions are being asked across the pond. It is now a global concern to better understand the accuracy of predictions.
The rehabilitation industry has move ahead to assume more responsibility than just measuring a person's functional ability. Systems are now in place to also predict how much change should occur for a person receiving rehabilitation services. This is the first paper that I am aware of that is asking the next question: How accurate is the prediction? In my opinion, a systematic review proves to be a challenging method to answer questions. When you think about it, systematic reviews typically have quite a bit of heterogeneity which makes it difficult for clinicians to actually apply what is learned into practice. This particular publication that I reviewed did it's best to focus on what we in the United States would term risk adjustment. I was introduced to a new term: case-mix adjustment (which as I read the paper seems to be the same as risk adjustment).
Predictive ability across US study models ranged from 18-42% and in UK models from 23-30%, demonstrating moderate to strong predictive ability across models.
If we use common sense, a model that only focuses on patient factors will be 100% accurate. A final treatment outcome includes far more than just patient factors. If you think of what rehabilitation looks like, there are other factors that play a role in treatment outcomes: clinician factors, patient-clinician factors and clinic factors. I'd like to think that a clinician's belief system affects an outcome. I'd like to believe that the strength of the therapeutic alliance has a role. I'd also like to think that the way an organization structures patient experience also has a role. In my mind, if a model that only focuses on patient factors is able to accurately predict the outcome 40-50% of the time, the model is very strong.
Because there are now more products on the market that risk adjust, I take the stand that the industry needs to demand the percentage of variation in the product's ability to predict the clinical outcome. The next step then includes defining the power of the product's predictive ability. As an example 0-15% variation explained = poor; 16-25% variation explained = limited; 26-30% variation explained = average; 31-42% variation explained = good; >42% variation explained = excellent. Since I am just thinking out loud with this concept, I would also propose that 26-42% would be defined as the typical industry standard to meet when providing predictive analytics for clinicians. The reason I believe this is important is because outcomes are compared in aggregated data. If the risk adjustment process is not able to adequately capture the important factors to increase the predictive accuracy, then clinicians will be unfairly compared.
The abstract is included below for you to review.
Case-mix adjustment is an established method to take account of variations across cohorts in baseline patient factors, when comparing health outcomes. Although commonplace, there is a lack of evidence as to the most appropriate case-mix adjustment model to use to enable fair comparisons of PROM data in musculoskeletal services.
To conduct a systematic review summarising evidence of the development, validation, and performance of musculoskeletal case-mix adjustment models, and to make recommendations for future methods.
Searches included; AMED, CINAHL, EMBASE, HMIC, MEDLINE, and grey literature.
Studies; from January 1992-May 2017, English language, musculoskeletal adult population, developing or validating a case-mix adjustment model, using a relevant PROM, and using patient factors feasible for clinical collection.
Two reviewers evaluated selected papers. The CASP Cohort Tool was used to assess quality.
Fourteen studies were included; eight US studies on the Focus on Therapeutic Outcomes model (pooled n=546,726 patients (with pre/post treatment data)) and six UK studies related to the UK National PROMs Programme model (pooled n=282,424 patients (with pre/post treatment data)). The majority used retrospective data, restricted to complete datasets. Both US and UK models showed good predictive ability (R2 18-42%). Common model variables were; baseline PROM score, age, sex, comorbidities, symptom duration, and surgical history. Reduced quality scores were mainly due to acceptability of patient recruitment, and completeness and length of patient follow up.
Significant methodological crossover was found. Further studies are however needed to externally validate and develop models across musculoskeletal settings.
Physiotherapy. 2018 Nov 9. pii: S0031-9406(18)30292-X. doi: 10.1016/j.physio.2018.10.002. [Epub ahead of print]