How fragile is our knowledge about reverse total shoulder arthroplasty?

We tend to have highest confidence in Level I randomized clinical trials because they control for variables that may be overlooked in comparative case series - even with devices such as propensity matching. 

Level I randomized clinical trials been reported that explore important aspects of reverse total shoulder arthroplasty. Some recent examples are listed below:

Inferior tilt of the glenoid component does not decrease scapular notching in reverse shoulder arthroplasty: results of a prospective randomized study.

Results after primary reverse shoulder arthroplasty with and without subscapularis repair: a prospective randomized trial.

Reverse Shoulder Arthroplasty Is Superior to Plate Fixation at 2 Years for Displaced Proximal Humeral Fractures in the Elderly: A Multicenter Randomized Controlled Trial.

Can a functional difference be detected in reverse arthroplasty with 135° versus 155° prosthesis for the treatment of rotator cuff arthropathy: a prospective randomized study.

Clinical performance of lateralized versus non-lateralized reverse shoulder arthroplasty: a prospective randomized study

A randomized single-blinded trial of early rehabilitation versus immobilization after reverse total shoulder arthroplasty

Reverse total shoulder arthroplasty provides better shoulder function than hemiarthroplasty for displaced 3- and 4-part proximal humeral fractures in patients aged 70 years or older: a multicenter randomized controlled trial.

Reverse or Hemi Shoulder Arthroplasty in Proximal Humerus Fractures: A Single-Blinded Prospective Multicenter Randomized Clinical Trial.

Reverse shoulder arthroplasty versus nonoperative treatment for 3- or 4-part proximal humeral fractures in elderly patients: a prospective randomized controlled trial.

A comparison of concentric and eccentric glenospheres in reverse shoulder arthroplasty: a randomized controlled trial

Reverse shoulder arthroplasty versus hemiarthroplasty for acute proximal humeral fractures.A blinded, randomized, controlled, prospective study.

Positioning of the metaglene in reverse shoulder arthroplasty: deltopectoral versus anterosuperior
approach: a prospective randomized trial

Influence of glenosphere size on the development of scapular notching: a prospective randomized study.

Can small glenospheres with eccentricity reduce scapular notching as effectively as large glenospheres without eccentricity?

Bony increased-offset reverse shoulder arthroplasty vs. metal augments in reverse shoulder arthroplasty: a prospective, randomized clinical trial with 2-year follow-up.

Intravenous tranexamic acid reduces total blood loss in reverse total shoulder arthroplasty: a prospective, double-blinded, randomized, controlled trial.

Reverse shoulder arthroplasty with and without latissimus and teres major transfer for patients with
combined loss of elevation and external rotation: a prospective, randomized investigation.

While these Level I studies assess the statistical significance (i.e. the p values) for their findings, the p value does not indicate how robust or fragile the findings are. In some cases a change in outcome for just a few patients could flip the p value from significant (<.05) to insignificant (>.05) or, conversely, from insignificant (p>.05) to significant (p<.05).

A useful indicator of the robustness of the result of an RCT is the fragility index (FI), which is the minimum number of results which must be reversed to change the statistical significance of a given outcome. The Fisher exact test provides convenient way of measuring the FI (readers are encouraged to experiment with it using this link). 

Here is a hypothetical example comparing the incidence of of acromial fractures in a randomized study of hemiarthroplasty versus reverse total shoulder. In the first scenario, none of the 30 patients having hemiathroplasty experienced acromial fractures, whereas 6 of the 30 patients having reverse total shoulders experienced acromial fractures. The result was statistically significant at p < .05.

However, if one of the 30 patients having hemiarthroplasty was discovered to have an acromial fracture, the result was no longer statistically significant at p < .05.

Thus in this example, the fragility index was 1, indicating the fragility of the result - a flip in significance with a change in the result for only one patient.

The fragility quotient (FQ) is the fragility index divided by the number of patients in the study. In this example it would be 1/60 or 0.01667.

Studies with lower fragility indices and lower fragility quotients are less robust, even if p<.05.

The fragility index can also be compared to the number of patients lost to followup (LTFU), i.e. for whom the outcome is unknown. If the number of patients lost to followup is greater than the fragility index, the observed difference is not robust, even if p<.05

The authors of The Statistical Fragility of The Management Options for Reverse Shoulder Arthroplasty: A Systematic Review of Randomized Control Trial with Fragility Analysis examined the statistical fragility of Randomized Control Trials (RCTs) reporting outcomes from reverse total shoulder arthroplasty (RSA). They included 19 RCTS representing 1,146 patients, of which 41.2% were male, with a mean age of 74.2  and mean follow-up of 22.1 months. 

The median number of patients in the studies was 59, with a median of 9 patients Lost To Follow-up (LTFU). The median FI was 4.5, indicating more patients were lost from the trial than the number of outcomes which would have to change to reverse the finding of significance. The median FQ was 0.083.

Comment: Among these Level I RSA studies, the clinical evidence was vulnerable to statistical fragility, with a median FI of 4.5 indicating that the reversal of just a handful of outcomes was sufficient to change the finding of statistical significance. This should be viewed in the context of the median number of patients LTFU being equal to 9. Thus the median trial lost more patients to follow-up than the number of outcomes which would have to be changed to reverse a finding of significance.  Approximately 2/3s of included events may have had reversed significance findings had there been a more complete follow-up. 

When interpreting the results of randomized control clinical trials, it is important to consider both the fragility index in comparison to the number o patients lost to followup.

This post received and interesting reaction from Mark Cote, Director of Outcomes Research for Sports Medicine & Orthopaedic Surgery. Massachusetts General Hospital. He gave permission to quote it here:

"I think it’s great to go beyond p-values; however, this type of approach does little to advance interpretation. We’ve discussed this from time to time in the past. The fragility index is trying to move away from “statistical significance” yet it’s rooted in statistical significance, i.e. how many patients would have to change their outcome in order to flip the p-value to above 0.05. In a randomized clinical trial (RCT) of bio-augmented RCR versus standard RCR, if the bio-aug group had less failures than standard and the result was statistically significant (p<0.05), then the fragility index would be the number of patients in the bio-aug group that would have needed to fail before the difference is no longer p<0.05.

The problem is the fragility index can be misleading. For one, RCTs are designed with the MINIMUM number of patients needed to reach statistical significance, yet we act surprised when these studies are “fragile” because the fragility index equals 2 or 3 patients. Of course it does, it was deliberately designed to barely reach statistical significance! When's the last time you, or a medical student, resident, fellow, or faculty came into my office and asked for a sample size that went well beyond p<0.05? The findings in the blog should not be surprising, nor should the growing number of systematic reviews of RCTs involving the elbow, knee, etc., all of which conclude the same thing, i.e. the available evidence is fragile. If we did this in RCR or instability, it’s likely to be fragile as well.

Finally, a large fragility index doesn’t equal robust findings. In fact, it can be quite the opposite. If you recall, we were looking at the fragility index for RCTs on proximal humerus fractures a few months back. The one study that had a large index seemed to be celebrated yet when you looked closer at all of the studies in the review, it had the biggest loss to follow-up as well as other biases . Here, the large fragility index was a product of a bias study. This is also not surprising. You would expect well planned RCTs that by tradition enroll the MINIMUM number of patients needed to reach statistical significance to have small fragility indexes. Conversely, a RCT that despite planning (or alleged planning) that produced results that are wildly inconsistent with their power analysis can indicate a number of things, almost all of which indicate pronounced bias. Dr. Matsen does a good job noting the importance of losses to follow up but the general premise of the fragility index is flawed. Specifically, there IS or IS NOT a difference in revision rates between arthroplasty groups. It would be nice if nature was so straightforward, but unfortunately it isn’t. All null hypotheses, i.e. no difference between groups, are wrong. The question is how well a study estimated a difference and whether it’s of a size that we care about.

Again, I think it’s great to see folks trying to move away from statistical significance. It’s like acknowledging that you need to stop smoking 3 packs a day. But to turn to the fragility index as some sort of panacea or surrogate for robustness is about the same as thinking that switching to light cigarettes will mitigate the health risk of your 3-pack a day habit.
That said, a paper on these problems may be worth pursuing…

Mark"

You can support cutting edge shoulder research that is leading to better care for patients with shoulder problems, click on this link.

Follow on twitter: https://twitter.com/shoulderarth
Follow on facebook: click on this link
Follow on facebook: https://www.facebook.com/frederick.matsen
Follow on LinkedIn: https://www.linkedin.com/in/rick-matsen-88b1a8133/

Here are some videos that are of shoulder interest
Shoulder arthritis - what you need to know (see this link).
How to x-ray the shoulder (see this link).
The ream and run procedure (see this link).
The total shoulder arthroplasty (see this link).
The cuff tear arthropathy arthroplasty (see this link).
The reverse total shoulder arthroplasty (see this link).
The smooth and move procedure for irreparable rotator cuff tears (see this link).
Shoulder rehabilitation exercises (see this link).

Graft augmentation for rotator cuff repair - how fragile is the supporting evidence?

Let's say we wanted to compare re-tear rates between rotator cuff repairs with and without graft augmentation. Our research fellow found these results showing a statistically significant benefit of the graft.



However, she rechecked her data and found that one patient in the no-graft group originally thought to have had a re-tear actually had an intact repair. When the data were re-analyzed, the significance of the benefit of grafting disappeared.




As can be determined from the Fragility index calculator, our study had a fragility index of one: only one result needed to change for the result to loose statistical significance.

The authors of The Statistical Fragility of Studies on Rotator Cuff Repair with Graft Augmentation sought to determine the fragility index (FI) and fragility quotient (FQ) for studies investigating the value of augmenting rotator cuff repair (RCR) with a graft. A lost to follow-up (LTF) value greater than the FI indicates statistical instability for the reported outcomes and conclusions.

They performed a systematic review to identify studies of RCR with graft augmentation. They included 17 studies (1,098 patients) that compared least one statistically analyzed dichotomous outcome variable for graft and no graft repairs (eg cuff integrity at followup). The fragility index (FI) was determined by changing each reported outcome event within 2 x 2 contingency tables until statistical significance or non-significance was reversed. The associated fragility quotient (FQ) was determined by dividing the FI by the sample size. Lost to follow-up (LTF) values were also extracted from each included study.

The median FI was 4 (IQR 2-5), indicating that the reversal of 4 patients’ outcomes would have reversed the finding of significant difference from significant to non-significant or vice-versa. The median FQ was 0.08 (IQR 0.04-0.15), indicating that in a sample of 100 patients, reversal of 8 patients’ outcomes would reverse statistical significance. 56% of reported outcomes were statistically fragile, having LTF greater than their respective FI.

They concluded that studies of RCR with graft augmentation lack statistical stability: a few altered outcome events would reverse statistical significance. They recommend that future studies of graft augmentation include FI and FQ along with traditional statistical significance analyses to provide better context on the strength of conclusions.

Comment: Assessment of fragility should be a part of all comparative studies. See:

PRP for Cuff Disease: Data Fragility of a Level I Study

You can support cutting edge shoulder research that is leading to better care for patients with shoulder problems, click on this link.

Follow on twitter: https://twitter.com/shoulderarth

Follow on facebook: click on this link

Follow on facebook: https://www.facebook.com/frederick.matsen

Follow on LinkedIn: https://www.linkedin.com/in/rick-matsen-88b1a8133/

Here are some videos that are of shoulder interest
Shoulder arthritis - what you need to know (see this link).
How to x-ray the shoulder (see this link).
The ream and run procedure (see this link).
The total shoulder arthroplasty (see this link).
The cuff tear arthropathy arthroplasty (see this link).
The reverse total shoulder arthroplasty (see this link).
The smooth and move procedure for irreparable rotator cuff tears (see this link).
Shoulder rehabilitation exercises (see this link).

PRP for Cuff Disease: Data Fragility of a Level I Study

Allogeneic Platelet-Rich Plasma Versus Corticosteroid Injection for the Treatment of Rotator Cuff Disease A Randomized Controlled Trial

These authors investigated the safety and efficacy of a fully characterized allogeneic pure PRP injection into the subacromial space of patients with "rotator cuff disease" in comparison with corticosteroid injection. Inclusion criteria: ≥18 years ofd age, unilateral shoulder pain for at least 3 months, a Neer or Hawkins impingement sign,  either a painful arc or a positive result on the Jobe test. Exclusion criteria included previous subacromial injections within the past 3 months, a history of shoulder trauma, a full-thickness rotator cuff tear as demonstrated with magnetic resonance imaging (MRI) or ultrasonography, and limitation of both active and passive movement of the glenohumeral joint of 25% in at least 2 directions as compared with the contralateral shoulder or with normal values.

60 patients with rotator cuff disease were randomly assigned to receive a subacromial injection of either 4 mL of allogeneic pure PRP or a 4-mL mixture of 1 mL of 40-mg/mL triamcinolone acetonide and 3 mL of 2% lidocaine under ultrasonographic guidance. 

The authors concluded that " Constant score at 1 month did not significantly differ between the PRP and corticosteroid groups. At 6 months, the DASH (Disabilities of the Arm, Shoulder and Hand) score, overall function, and external rotation were significantly better in the PRP group than in the corticosteroid group, and the other clinical outcomes did not show significant differences. All pain measurements, the strength of the supraspinatus and infraspinatus, and 5 functional scores also improved slowly and steadily after injection, becoming significantly better at 6 months compared with those before the injection, whereas those in the corticosteroid group responded promptly but did not further improve. Allogeneic PRP injections for the treatment of rotator cuff disease are safe but are not definitely superior to corticosteroid injections with respect to pain relief and functional improvement during 6 months. The DASH score, overall function, and external rotation were significantly better in the PRP group than in the steroid group at 6 months. Generally, PRP slowly but steadily reduced pain and improved function of the shoulder until 6 months, whereas corticosteroid did not."

Comment: Let's take a look at the DASH data. Recalling that a lower DASH score is good, we can see that at later time points the average DASH scores for the available patients are lower (i.e. better).

A few blog posts ago we discussed data fragility (see this link). Fragility is an indication of how sensitive the result is to the loss of patients to followup. 

In this study, we see that the average DASH score for the patients available at each time point after treatment is related to the percentage of patients lost to followup. The best (lowest) DASH score is at 6 months when 24% of the original patients were not included. The conclusion that the DASH improved with time is fragile because it is possible that the missing 24% could have had higher (worse) DASH scores at 6 months - we just don't know. 

It is also of interest that the percent of patients lost to followup (especially at 6 months) is quite different for the PRR and the steroid groups as shown below.

This limitation is not discussed in the limitations discussion of the paper.

===

How you can support research in shoulder surgery Click on this link.

We have a new set of shoulder youtubes about the shoulder, check them out at this link.

Be sure to visit "Ream and Run - the state of the art" regarding this radically conservative approach to shoulder arthritis at this link and this link

Use the "Search" box to the right to find other topics of interest to you.

You may be interested in some of our most visited web pages  arthritis, total shoulder, ream and run, reverse total shoulder, CTA arthroplasty, and rotator cuff surgery as well as the 'ream and run essentials'

 

How fragile is our knowledge about shoulder arthroplasty?

Statistical Fragility of Randomized Clinical Trials in Shoulder Arthroplasty

The Fragility Index (FI) is a tool for assessing the robustness of a "statistically significant" result from clinical research. The FI is defined as the number of patients in a study that would flip an outcome from being statistically significant to not  statistically significant or vice versa. 

Let's consider an example of a randomized controlled trial of 50 patients in which no patients were lost to two year followup. All 25 patients receiving "my prosthesis" improved by the MCID while only 19 patients receiving "your prosthesis" improved by the MCID. Fischer's exact test declares me the winner with p = 0.022! 

Hang on. Without asking me, my fellow reanalyzed the data and found that actually one more of your patients achieved the MCID. No big deal? Except that with your 20 successes, p became 0.050, no statistically significant difference. 

This is an example of a study with a fragility index of 1: only one patient needed to flip in order for the statistically significant result to become statistically insignificant.

Now let's add another dose of reality. Actually we enrolled 54 patients in this study, but 4 were lost to followup. That's a 93% followup rate - Spectacular! However, this same fellow had the nerve to point out that with the outcome of 4 patients being unknown and a fragility index of only 1, the outcome for those 4 patients could easily have flipped the significance of the result in either direction. 

Reference: see this link.

With that introduction, the primary purpose of this study was to evaluate the statistical robustness of clinical trials with regards to shoulder arthroplasty using FI. The secondary goal of this study is to identify trial characteristics associated greater statistical fragility.

The authors found a total of 13 randomized controlled trials (RCTs) on shoulder arthroplasty were identified and evaluated, and were found to have a median sample size of 47 patients and a median number of 7 patients lost to follow-up

The median FI was 6.

A majority of outcomes (74.4%) had a Fragility Index that was less than the number of patients lost to follow-up and the majority of outcomes (89.7%) were statistically not significant.

The take home point is that we readers need to be aware of the fragility of the statistical result, even if p<.05.  

The thinkers among you will be interested in:

Sir Austin Bradford Hill (1897-1991) and Richard Doll were the first scientists to establish the causation of cancer by cigarette smoking. Even though Hill pioneered the randomized clinical trial, he recognized that there were many circumstances in clinical research were the RCT could not be used.

He was particularly interested in proving causation; he developed nine criteria and used them to convince the scientific community that smoking caused cancer. Here are the Hill Criteria:

(1) the strength of the association
(2) the consistency of the observed association
(3) the specificity of the association

(4) the temporal relationship of the association
(5) the biological gradient (dose/response relationship)
(6) the biological plausibility – the mechanism by which local anesthetics cause chondrolysis is consistent with the current understanding of mechanisms of cytotoxicity; 
(7) coherence - does the cause-and-effect interpretation of the data conflict with the generally known facts of the natural history and biology?
(8) experimental evidence
(9) analogy 

You may be interested in his writing "The Environment and Disease: Associationor Causation?"

He concludes this article with the following 'poke' at those of us who excessively emphasize "p<":

“I wonder whether the pendulum has not swung too far – not only with the attentive pupils but even with the statisticians themselves. To decline to draw conclusions without standard errors can surely be just as silly? Fortunately I believe we have not yet gone so far as our friends in the USA where, I am told, some editors of journals will return an article because tests of significance have not been applied. Yet there are innumerable situations in which they are totally unnecessary – because the difference is grotesquely obvious, because it is negligible, or because, whether it be formally significant or not, it is too small to be of any practical importance. What is worse the glitter of the t table diverts attention from the inadequacies of the fare. Only a tithe, and an unknown tithe, of the factory personnel volunteer for some procedure or interview, 20% of patients treated in some particular way are lost to sight, 30% of a randomly-drawn sample are never contracted. The sample may, indeed, be akin to that of the man who, according to Swift, ‘had a mind to sell his house and carried a piece of brick in his pocket, which he showed as a pattern to encourage purchasers.’ The writer, the editor and the reader are unmoved. The magic formulae are there.
Of course I exaggerate. Yet too often I suspect we waste a deal of time, we grasp the shadow and lose the substance, we weaken our capacity to interpret the data and to take reasonable decisions whatever the value of P. And far too often we deduce ‘no difference’ from ‘no significant difference.’ Like fire, the chi-squared test is an excellent servant and a bad master.”

Our approach to the ream and run can be viewed by clicking here.

Our approach to total shoulder arthroplasty can be viewed by clicking here.

To support our research to improve outcomes for patients with shoulder problems, click here.

To subscribe to this blog, enter your email in the box to your right that looks like the below

===
How you can support research in shoulder surgery Click on this link.

We have a new set of shoulder youtubes about the shoulder, check them out at this link.

Be sure to visit "Ream and Run - the state of the art" regarding this radically conservative approach to shoulder arthritis at this link and this link

Use the "Search" box to the right to find other topics of interest to you.

You may be interested in some of our most visited web pages  arthritis, total shoulder, ream and run, reverse total shoulder, CTA arthroplasty, and rotator cuff surgery as well as the 'ream and run essentials'