UW Shoulder and Elbow Academy: response bias

Tuesday, September 21, 2021

What is the value of the PROMIS innovation in assessing the results of reverse total shoulder - the risk of non-response bias.

Establishing clinically significant outcomes of the Patient-Reported Outcomes Measurement Information System Upper Extremity questionnaire after primary reverse total shoulder arthroplasty

These authors reviewed consecutive patients undergoing primary RTSA between 2018 and 2019 who received preoperative baseline and follow-up PROMIS UE assessments at 12 months after surgery.

Domain-specific anchor questions pertaining to pain and function assessed at 12 months after surgery were used to determine minimal clinically important difference (MCID), substantial clinical benefit (SCB), and patient acceptable symptomatic state (PASS) values for the PROMIS UE.

Notably, all patients were distributed PROMIS UE assessments by trained research personnel at baseline before surgery. At 1 year after surgery, patients were contacted by e-mail every 5 days to complete follow-up assessments. At the end of the 1-month period, those who did not respond were recorded as lost to follow-up and excluded from this investigation.

They identified a total of 141 patients undergoing primary RTSA with completed preoperative PROMIS UE assessments in the study period, of whom only 95 (67.3%) completed 12-month assessments. There was a significantly greater proportion of male patients in the included cohort when compared with those lost to follow-up.

Among patients who completed both assessments, the average PROMIS UE score improved from 28.6 at baseline to 40.78 at 12 months. The visual analog scale score and American Shoulder and Elbow Surgeons (ASES) shoulder score also improved from 6.56 to 2.99 and from 38.58 to 77.89, respectively.

By use of an anchor-based method, the PASS value was 36.68 and the SCB value was 11.62. By use of a distribution-based method, the MCID value was calculated to be 4.27.

Higher preoperative PROMIS UE scores were a positive predictor in achievement of the PASS. This is an intuitive result - higher scores before surgery are likely to correlate with higher scores after surgery. 69.5% of the responding patients achieved the PASS.

Lower preoperative PROMIS UE scores were associated with obtaining SCB. This is an intuitive result - higher scores before surgery are likely to correlate with less improvement (benefit). 47.4% of the responding patients achieved the SCB.

Greater baseline forward flexion was negatively associated with achievement of the PASS and MCID. This result is not explained.

Comment: As noted above, all patients were distributed PROMIS UE assessments by trained research personnel at baseline before surgery. At 1 year after surgery, patients were contacted by e-mail every 5 days to complete follow-up assessments. Thus the methods for data collection after surgery (email) were not the same as the staff intensive data collection before surgery. At the end of the 1-month period, those who did not respond (one out of three) were recorded as lost to follow-up and excluded from the investigation.

This study demonstrates that non-response bias (see this link) may be associated with the use of the PROMIS: one third of the eligible patients did not provide a 12-month response. Even though the preoperative characteristics of the responders and non-responders were similar, there is no way to know if the outcomes of the missing 1/3 were similar to those of the responders. It appears that the characteristics of the PROMIS system has the potential for excluding a substantial number of them - possibly those with inferior results or those from less advantaged socio-economic situations (see this link).

For contrast compare the response rate from the above study with that in What is a Successful Outcome Following Reverse Total Shoulder Arthroplasty?, a study in which the more easily accessible Simple Shoulder Test enabled 87% of the patients in the original sample to provide two year followup.

While there have been many studies evaluating the characteristics of different outcome measures (see this link), there clearly needs to be further study of the response rates and the risk of non-response bias for each metric.

Follow on twitter: https://twitter.com/shoulderarth

Follow on facebook: https://www.facebook.com/frederick.matsen

Follow on LinkedIn: https://www.linkedin.com/in/rick-matsen-88b1a8133/

How you can support research in shoulder surgery Click on this link.

Here are some videos that are of shoulder interest

Shoulder arthritis - what you need to know (see this link)

The smooth and move for irreparable cuff tears (see this link)

The total shoulder arthroplasty (see this link).

The ream and run technique is shown in this link.

The cuff tear arthropathy arthroplasty (see this link).

The reverse total shoulder arthroplasty (see this link).

Shoulder rehabilitation exercises (see this link).

Friday, September 3, 2021

Do we know what our patients think about the surgical care we give?

Press Ganey Surveys in Patients Undergoing Upper-Extremity Surgical Procedures Response Rate and Evidence of Nonresponse Bias

These authors point out that patient satisfaction surveys are important measures of the patient experience and these surveys can provide data for quality improvement.

They sought to determine the response rate and the factors associated with the completion of the Press Ganey (PG) Ambulatory Surgery Survey (PGAS) in patients who underwent ambulatory upper extremity surgical procedures.

The PGAS poses questions such as the below regarding the provider of their care

They conducted a review of the orthopaedic registry at a single academic ambulatory surgical center for

patients who underwent an upper-extremity surgical procedure from 2015 to 2019. The institutional Press Ganey database was queried to determine the patients who completed the PGAS postoperatively.

They calculated the response rate, and compared the baseline characteristics and patient-reported outcome measures between responders and nonresponders.

Of the 1,489 patients included, 201 (13.5%) were responders and 1,288 (86.5%) were nonresponders.

Differences existed in baseline characteristics between groups, with responders being significantly older (p = 0.004) and having significantly higher proportions of White race (p < 0.001), college education (p = 0.011), employment (p = 0.005),marriage (p = 0.006), and higher income earners (p < 0.001).

Responders had significantly better baseline Patient-Reported Outcomes Measurement

Information System scores across multiple domains (p < 0.05), but these differences did not exceed the MCID.

The authors concluded that the PGAS response rates were low (13.5%) and that the responders may not be representative of all patients.

Comment: This study points out the perils of non-response bias in the Press-Ganey scores. It provides clear evidence that the 13.5% of patients who responded did not represent the diversity of the population of interest.

We suggest that a similar non-response bias may be present in the application of computer-based outcome measures, such as the computer-based PROMIS. We suspect that PROMIS responders may differ from PROMIS non-responders and that the differences in demographics may be similar to those seen for the PG scores. This is important in that both the Press-Ganey and the PROMIS scores may yield results that are not representative of the total group of patients under consideration; furthermore because of the non-response bias effect the results obtained from the responders may be better than they would be for the entire group of interest.

There seem to be two lessons here: (1) in presenting results with measures such as PG and PROMIS, we must be alert to the response rate as well as to the risk of non-response bias and (2) we must seek and use measures that are as inclusive as possible, so that access, questionnaire fatigue, and technology do not form barriers to patient participation.