What's the difference?

Editorial: No-difference Studies Make a Big Difference

This article points out some of the reasons that it is important for journals to publish articles that show no difference between treatments:

(1) No difference studies can retard the adoption of unproven ideas. 

(2) The readers and meta-analyses will come to the wrong conclusions if research demonstrating 'no difference' is 

-prematurely terminated, 

-or not submitted for publication because the investigators are disappointed in the outcome or because the industry supporting the research doesn't want the news to get out, 

-or because journals are more attracted to 'positive' results.

(3) If negative results are not published, investigators may unknowingly repeat the study. 

Comment: We agree with the author. Articles should be published if they address an important question with sound methodology. The results should not be the deciding factor. 

Another reason for publishing 'no difference' studies is to prompt investigators to do better research. A recent Cochrane review found 'no difference in functional outcome scores between surgical and active non-operative treatment' of rotator cuff disease. How is it, then, that rotator cuff surgery is by far the most common shoulder procedure performed. 

A recent review of the literature found that there was no difference in the results of rotator cuff surgery over time, in spite of the introduction of new surgical methods. How is it, then, that new cuff surgery techniques are introduced each year without demonstration of their superiority over extant methods.

On the other hand, we read studies that show 'differences' but that fail to show that the differences are of clinical relevance or of value to the patient (examples here and here ).

What surgeons need from the journals is not more numbers, but rather what we call 'actionable intelligence' - information that improves our ability to understand, evaluate and mange the problems faced by our patients. We need news we can use. To the point of the editorial referenced above, it is equally important to know that the clinical outcomes are the same with treatments A and B as to know that B is clinically a substantial improvement over A.

Finally a quote by Sir Austin Bradford Hill (1897-1991) the first scientist to establish the causation of cancer by cigarette smoking from a prior post:

“I wonder whether the pendulum has not swung too far – not only with the attentive pupils but even with the statisticians themselves. To decline to draw conclusions without standard errors can surely be just as silly? Fortunately I believe we have not yet gone so far as our friends in the USA where, I am told, some editors of journals will return an article because tests of significance have not been applied. Yet there are innumerable situations in which they are totally unnecessary – because the difference is grotesquely obvious, because it is negligible, or because, whether it be formally significant or not, it is too small to be of any practical importance. What is worse the glitter of the t table diverts attention from the inadequacies of the fare. Only a tithe, and an unknown tithe, of the factory personnel volunteer for some procedure or interview, 20% of patients treated in some particular way are lost to sight, 30% of a randomly-drawn sample are never contracted. The sample may, indeed, be akin to that of the man who, according to Swift, ‘had a mind to sell his house and carried a piece of brick in his pocket, which he showed as a pattern to encourage purchasers.’ The writer, the editor and the reader are unmoved. The magic formulae are there.
Of course I exaggerate. Yet too often I suspect we waste a deal of time, we grasp the shadow and lose the substance, we weaken our capacity to interpret the data and to take reasonable decisions whatever the value of P. And far too often we deduce ‘no difference’ from ‘no significant difference.’ Like fire, the chi-squared test is an excellent servant and a bad master.”

===

Consultation for those who live a distance away from Seattle.

Click here to see the new Shoulder Arthritis Book

Click here to see the new Rotator Cuff Book

Use the "Search" box to the right to find other topics of interest to you.

You may be interested in some of our most visited web pages including:shoulder arthritis, total shoulder, ream and run, reverse total shoulder, CTA arthroplasty, and rotator cuff surgery as well as the 'ream and run essentials'

See from which cities our patients come.

See the countries from which our readers come on this post.