As pointed out by E.A.Codman almost a century ago, the goal of outcome research is to follow every patient long enough to determine whether the treatment was successful and to ask, "if not, why not". The goal of clinical research using patient self-assessment is to capture the largest possible percentage of those potentially eligible, without the risk of non-response or selection bias that may systematically exclude certain categories of patients. Is computer adaptive testing the ideal tool in this regard?

Let's consider a recent article, Performance and responsiveness to change of PROMIS UE in patients undergoing total shoulder arthroplasty These authors studied the Patient-Reported Outcomes Measurement Information System Upper Extremity Computer Adaptive Test (PROMIS UE CAT) with respect to its responsiveness in patients undergoing shoulder arthroplasty. They found that the PROMIS UE demonstrated excellent correlation (range: 0.68-0.84) with the standard legacy instruments, the Simple Shoulder Test, the American Shoulder and Elbow Surgeons Score and the Oxford Shoulder Score at all postoperative time-points.

Comment: The authors pointed out that "the mean number of questions administered by PROMIS UE CAT was 4, compared to 11 in ASES and 12 in both SST and OSS, which translates to a lower patient response burden when filling out PROMIS surveys"; however, it is not at all clear that this "patient response burden" of an extra 34 seconds (see below) compromises the patient participation in postoperative followup.

Instead, consider "Patients were required to have access to a working email and a computer/phone to participate in this study so socioeconomic factors and advanced age could have limited participation and confound our results." While "all patients undergoing TSA were offered enrollment in this study", only 97 of all the TSAs performed by this busy arthroplasty service between March 2019 and April 2021 could be included.

Computer adaptive testing requires that the patient have access to and is trained in the use of a computer interface that they will use before surgery and after surgery at the designed followup intervals. While this may be straightforward for some patients, other patients may not have the necessary training and access.

It seems important to use a followup system that does not carry the risk of systematically excluding patients who are older, less educated, less healthy or socioeconomically disadvantaged.

For example in a study discussed in this link, one third of the eligible patients did not provide a 12-month PROMIS response. It appears that the characteristics of the PROMIS system has the potential for excluding a substantial number of patients - possibly those with inferior results or those from less advantaged socio-economic situations (see this link).

For contrast compare the response rate from the above study with that in What is a Successful Outcome Following Reverse Total Shoulder Arthroplasty?, a study in which the more easily accessible Simple Shoulder Test enabled 87% of the patients in the original sample to provide two year followup. In contrast to the PROMIS, the SST can be completed anywhere and requires only a pencil or a pen.

Another article is relevant: Correlation of PROMIS Physical Function Upper Extremity Computer Adaptive Test with American Shoulder and Elbow Surgeons shoulder assessment form and Simple Shoulder Test in patients with shoulder arthritis

The purpose of this study was to evaluate the Patient-Reported Outcomes Measurement Informative System Physical Function Upper Extremity Computer Adaptive Test (PROMIS PFUE CAT) measurement tool against the already validated American Shoulder and Elbow Surgeons (ASES) shoulder assessment form and the Simple Shoulder Test (SST) in patients with shoulder arthritis.

The average times to complete the SST and PROMIS PFUE CAT were determined to be 96.9 ± 25.1 seconds and 62.6 ± 22.8 seconds, respectively. The question is whether the saving of 34 seconds is worth the limitations of the PROMIS?

The scatter plot from this article also brings up another issue with the PROMIS: four patients who indicated that they could perform none of the 12 functions of the SST, still had PROMIS scores in the same range as three patients what could perform eight of these functions. Thus, the PROMIS was unable to discriminate between a non-functioning shoulder and a reasonably functional one.

Another recent article is relevant:PROMIS Upper Extremity Underperforms Psychometrically Relative to American Shoulder and Elbow Surgeons Score in Patients Undergoing Primary Rotator Cuff Repair

PROMIS UE-CAT correlated to a degree with the ASES (r=0.684) and had a 4% floor effect and no ceiling effect. While PROMIS UE-CAT initially required fewer test items for overall equivalent coverage of shoulder function assessment, final models after recursive item elimination revealed the ASES instrument to have more well-fitting items over a broader range of shoulder function.

The authors concluded that: "Until further refinements in the PROMIS UE-CAT instrument are made, it should not replace the ASES instrument in patients undergoing primary RCR."

Performance and responsiveness to change of PROMIS UE in patients undergoing total shoulder arthroplasty described the so called "ceiling effect" they observed with the OSS (17%) and the Simple Shoulder Test (18%). How big an issue is this really? In the graph below from The prognosis for improvement in comfort and function after the ream-and-run arthroplasty for glenohumeral arthritis: an analysis of 176 consecutive cases ...

..it can be seen a large number of patients having the ream and run for osteoarthritis "hit the ceiling" of 12 out of 12 on the SST. This means that

-the shoulder was comfortable at the side

-the shoulder allowed the patient to sleep comfortably

-the shoulder allowed reach to the small of the back to tuck in a shirt

-the shoulder allowed placement of the hand behind the head with the elbow straight out to the side

-the shoulder could lift a coin, a one pound weight, and an eight pound weight to the level of the top of the head without bending the elbow

-the shoulder allowed carrying 20 pounds at the side

-the shoulder allowed tossing a softball 20 yards underhand

-the shoulder allowed throwing a soft ball 20 yards overhand

-the shoulder allowed washing the back of the opposite shoulder

-the shoulder allowed work full time at the patient's usual job

In our view that's a pretty high ceiling; it is remarkable that so many patients can hit it after the ream and run. Obviously one could avoid the "ceiling effect" by adding a question such as, "would your shoulder allow you to throw 100 yards?", but it seems that "yes" responses to each of the 12 existing questions indicates a comfortable and highly functional shoulder.

Performance and responsiveness to change of PROMIS UE in patients undergoing total shoulder arthroplasty points out that while the minimal clinically important difference (MCID) has been well established for the Simple Shoulder Test (see Is the Simple Shoulder Test a valid outcome instrument for shoulder arthroplasty?) and many of the other legacy scales, but that "further quantification of meaningful responsiveness to change will require estimation of the minimal clinically important difference and substantial clinical benefit for PROMIS UE CAT".

Finally, Performance and responsiveness to change of PROMIS UE in patients undergoing total shoulder arthroplasty erroneously states that "all except SST have been translated and adapted to several other languages." See below:

Simple shoulder test and Oxford Shoulder Score: Persian translation and cross-cultural validation

Validation of the Simple Shoulder Test in a Portuguese-Brazilian Population. Is the Latent Variable Structure and Validation of the Simple Shoulder Test Stable across Cultures?

Validation and reliability of a Japanese version of the Simple Shoulder Test: a cross-sectional study