These authors concluded that "Outcome measures may consist of simple questions or they may be more complex instruments that evaluate multiple interrelated domains that influence patient function. Outcome measures should be relevant to patients, easy to use, reliable, valid, and responsive to clinical changes."
We are now in an era where we must use simple tools to assess the degree of disability experienced by our patients and to measure the change in this disability after our treatment. The value of our treatment is then measured by the change in disability divided by the cost of the treatment (including the cost of resulting complications).
The topic of outcome questionnaires recalls an editorial in the JBJS and our response to it, which is reproduced here:
To The Editor:
The editorial "Are Validated Questionnaires Valid?" (2005;87:1671-2), by my respected colleague Bert Zarins, was a provocative piece. While what he says rings true, there are other aspects to the "validity" issue that should be added to our thoughtful consideration. I will enlist the help of another Massachusetts General surgeon who is often, but incompletely, quoted:
"Already in 1900 I had become interested in what I have called the End Result Idea, which was merely the common-sense notion that every hospital should follow every patient it treats, long enough to determine whether or not the treatment has been successful, and then to inquire `if not, why not?'... We had found that this routine tracing of every case, interesting or uninteresting, had brought to our notice many things in which our knowledge, our technique, our organization, our own skill or wisdom, and perhaps even our care and our consciences, needed attention."
1 When he presented this idea in 1913 in the great hall of the Philadelphia Academy of Medicine, Codman pointed out that answering these questions is of primary interest to the patient, the public, and those in the medical field. He then asked, "Who represents or acts for these interests?" and answered, "Strangely enough the answer is: No one."
1 In his infamous cartoon of the "Back Bay Golden Goose Ostrich," he showed the bird producing golden eggs of profit while hiding her head in the sand so she could not see how much (or how little) the care was benefiting the patient
1. For his insolence, he was fired from the hospital.
A century later we are struggling with the same issues.
If we are to have a valid assessment of the effectiveness of an operation, we first need a method for determining the result in a reasonable statistical sample of patients having this procedure. As an example of the difficulty in achieving this objective, Hasan et al. showed that the great majority of shoulder arthroplasties are done by surgeons who perform fewer than four per year
2. Because these common cases are not included in studies of outcomes, the surgical results data published in the peer-reviewed literature are not particularly relevant, because they lack common external validity—that is, they cannot be applied to the common surgical experience. We are surely a long way from complying with Codman's notion of following "every" patient. The issue, as he pointed out, is that no one is representing or acting in the interest of the average patient having the surgery.
The two key steps that will help us to obtain valid data on the common experience are to apply the simplest effective tool for measuring the result of treatment (such as a series of "yes or no" questions about comfort and function that the patient can complete before and sequentially after the procedure) and to exert leadership at all levels for the broad collection and analysis of these data. The argument for the most practical metrics for this purpose must be made forcefully to balance the view expressed by Dr. Zarins that more costly "objective measures, such as physical examination findings, radiographs, and arthrometer measurements" are required for validity. The impracticality of rigorously applying these "objective measures" in common practice systematically and selectively excludes the largest and most important groups of patients and surgeons from the sample—those in general orthopaedic practices.
As the expense of medicine becomes progressively unaffordable, the need for externally valid, generalizable analyses of surgical effectiveness will be pressed on us by the employers, who are responsible for the medical benefits of their employees, and by the government, which is responsible for the medical benefits of those without medical benefits from their employers. It is in their interest.
The benefit from a surgical procedure can only be determined if we assess the patient in the same way before and after the treatment—a point emphasized by Dr. Zarins. While there is inconsistent use of the word "outcome" in our literature, common sense indicates that it means what comes out of the treatment in terms of the chosen measurement tool applied at a reasonable time after surgery. In order to determine the effectiveness of the treatment, we need also to know what goes into the treatment or the "ingo." Thus, the effectiveness or result of the arthroplasty is the difference between the outcome and the ingo as indicated by the same metric.
We recognize that the result of surgery is determined by three major factors: the patient, the surgeon, and the implant or prosthesis, if one is used. Let us consider these determinants in reverse order.
The first factor is the prosthesis. There are essentially no data indicating that "improvements" in prosthetic design have a significant effect on the benefit of an arthroplasty to the patient. For example, two recent articles by Mileti et al.
3 and Churchill et al.
4 found no difference between "first" and "second" generation humeral components. In this light, it is of concern that many tens of millions of dollars have been spent on trying to "improve" the humeral prostheses used in shoulder arthroplasty. For sure, the charges for implants are rising faster than either the established benefit to the patient from their use or the funds available in the health-care system to cover the increased charges. This is in contrast to the application of vascular stents, which both increased the quality of the result and reduced the cost of treating arterial occlusive disease. Were he alive today, Codman might ask, "In whose interest is the design of additional and more expensive varieties of orthopaedic prostheses?"
The second factor is the surgeon. It has been determined that patients cared for by surgeons who performed fewer than two shoulder arthroplasties per year were more likely to die postoperatively and had more surgical complications and longer lengths of stay than those managed by surgeons with a volume of five procedures or more per year. In one study, Jain et al.
5 concluded that "patients who have a total shoulder arthroplasty or hemiarthroplasty performed by a high-volume surgeon or in a high-volume hospital are more likely to have a better outcome." Hammond et al.
6 found that "the patients of surgeons with higher average annual case-loads of total shoulder arthroplasties and hemiarthroplasties have decreased complication rates and hospital lengths of stay compared with the patients of surgeons who perform fewer of these procedures." "Who holds the knife" appears to have a strong effect on the surgical result of a procedure.
The effects of intersurgeon differences extend beyond the surgical technique in the operating room. Harryman et al.
7 demonstrated that groups of patients with the same diagnosis in the practices of different surgeons were quite different from each other. Thus, the generalizability of the results obtained by a given surgeon treating his or her own patients may not be any more generalizable to all patients than the results obtained by a particular math professor in an elite university would be generalizable to all college students.
The surgeon is the method.
The third factor is the patient—the complex of the specific surgical problem and the human being who is affected by it. William Osler is credited with the statement: "It is more important to know what sort of patient has the disease than to know what sort of disease has the patient." Although it would seem intuitive that a patient's physical, emotional, and social welfare would all have a powerful effect on his or her ability to benefit from a surgical reconstruction, there has been little research on this important determinant of surgical effectiveness. Yet, as shown by Rozencwaig et al.
8, the tools for this critical research, such as the Short Form-36 (SF-36) patient self-assessment, are at hand providing a compelling and practical approach for exploring the relationship of the patient's perceived physical and mental health to the result of orthopaedic procedures.
As Dr. Zarins and we consider the validity of approaches to clinical research in orthopaedic surgery, we must ask, "What question are we trying to answer?" and "Will the answer we get from our study apply generally across our specialty, or only to a relatively few patients, practices, or surgeons?" If we are looking for a way to determine the answer to the question "Is prosthesis A or prosthesis B better in the hands of a surgeon who does nothing but this type of surgery?" we would want a sophisticated set of tools for controlling the variability in patients and their pathology as well as for documenting the function, mechanics, and radiographic anatomy before and sequentially after surgery—much as suggested by Dr. Zarins. If we are asking, "Do surgeons with high volumes, special training, or memberships in specialty societies get better results?" we need a method for including as many patients and as many surgeons as possible so that we have a statistically and externally valid sample. This will require the simplest and least taxing methodologies if it is to have a chance of succeeding. Finally, if we want to know which patients have the best results and which patients fail to benefit from a procedure, we need to capture data from the broadest possible sample of what is really happening. It is not a question of subjective versus objective measures; validity comes from matching the tool to the task.
