Statistical Fragility of Randomized Clinical Trials in Shoulder Arthroplasty
The Fragility Index (FI) is a tool for assessing the robustness of a "statistically significant" result from clinical research. The FI is defined as the number of patients in a study that would flip an outcome from being statistically significant to not statistically significant or vice versa.
Let's consider an example of a randomized controlled trial of 50 patients in which no patients were lost to two year followup. All 25 patients receiving "my prosthesis" improved by the MCID while only 19 patients receiving "your prosthesis" improved by the MCID. Fischer's exact test declares me the winner with p = 0.022!
Hang on. Without asking me, my fellow reanalyzed the data and found that actually one more of your patients achieved the MCID. No big deal? Except that with your 20 successes, p became 0.050, no statistically significant difference.
Now let's add another dose of reality. Actually we enrolled 54 patients in this study, but 4 were lost to followup. That's a 93% followup rate - Spectacular! However, this same fellow had the nerve to point out that with the outcome of 4 patients being unknown and a fragility index of only 1, the outcome for those 4 patients could easily have flipped the significance of the result in either direction.
Reference: see this link.
With that introduction, the primary purpose of this study was to evaluate the statistical robustness of clinical trials with regards to shoulder arthroplasty using FI. The secondary goal of this study is to identify trial characteristics associated greater statistical fragility.
The authors found a total of 13 randomized controlled trials (RCTs) on shoulder arthroplasty were identified and evaluated, and were found to have a median sample size of 47 patients and a median number of 7 patients lost to follow-up
The median FI was 6.
A majority of outcomes (74.4%) had a Fragility Index that was less than the number of patients lost to follow-up and the majority of outcomes (89.7%) were statistically not significant.
The take home point is that we readers need to be aware of the fragility of the statistical result, even if p<.05.
The thinkers among you will be interested in:
Sir Austin Bradford Hill (1897-1991) and Richard Doll were the first scientists to establish the causation of cancer by cigarette smoking. Even though Hill pioneered the randomized clinical trial, he recognized that there were many circumstances in clinical research were the RCT could not be used.
(1) the strength of the association
(2) the consistency of the observed association
(3) the specificity of the association
(5) the biological gradient (dose/response relationship)
(6) the biological plausibility – the mechanism by which local anesthetics cause chondrolysis is consistent with the current understanding of mechanisms of cytotoxicity;
(7) coherence - does the cause-and-effect interpretation of the data conflict with the generally known facts of the natural history and biology?
(8) experimental evidence
(9) analogy
You may be interested in his writing "The Environment and Disease: Associationor Causation?"
He concludes this article with the following 'poke' at those of us who excessively emphasize "p<":
“I wonder whether the pendulum has not swung too far – not only with the attentive pupils but even with the statisticians themselves. To decline to draw conclusions without standard errors can surely be just as silly? Fortunately I believe we have not yet gone so far as our friends in the USA where, I am told, some editors of journals will return an article because tests of significance have not been applied. Yet there are innumerable situations in which they are totally unnecessary – because the difference is grotesquely obvious, because it is negligible, or because, whether it be formally significant or not, it is too small to be of any practical importance. What is worse the glitter of the t table diverts attention from the inadequacies of the fare. Only a tithe, and an unknown tithe, of the factory personnel volunteer for some procedure or interview, 20% of patients treated in some particular way are lost to sight, 30% of a randomly-drawn sample are never contracted. The sample may, indeed, be akin to that of the man who, according to Swift, ‘had a mind to sell his house and carried a piece of brick in his pocket, which he showed as a pattern to encourage purchasers.’ The writer, the editor and the reader are unmoved. The magic formulae are there.
Of course I exaggerate. Yet too often I suspect we waste a deal of time, we grasp the shadow and lose the substance, we weaken our capacity to interpret the data and to take reasonable decisions whatever the value of P. And far too often we deduce ‘no difference’ from ‘no significant difference.’ Like fire, the chi-squared test is an excellent servant and a bad master.”
Our approach to the ream and run can be viewed by clicking here.
===
How you can support research in shoulder surgery Click on this link.
We have a new set of shoulder youtubes about the shoulder, check them out at this link.
Be sure to visit "Ream and Run - the state of the art" regarding this radically conservative approach to shoulder arthritis at this link and this link
Use the "Search" box to the right to find other topics of interest to you.
