Showing posts with label propensity matching. Show all posts
Showing posts with label propensity matching. Show all posts

Sunday, June 8, 2025

The Perils of Propensity Matching (post revised with the assistance of ChatGPT)

In a prior post, I discussed an article titled "Medicaid status is independently predictive of increased complications, readmission, and mortality following primary total shoulder arthroplasty." The authors claimed to show that Medicaid status was an independent predictor of adverse outcomes using a national administrative claims database. To interrogate that conclusion, I submitted the article and my blog comments to ChatGPT and asked for a refined critique. Here’s what we arrived at:


Is Medicaid Really the Risk? Are we asking the right question?

Recent work using national databases has reported that patients insured by Medicaid are at increased risk for complications, readmission, and mortality following elective total shoulder arthroplasty (TSA). These findings have led to the conclusion that "Medicaid status is independently predictive" of adverse outcomes. But is Medicaid status truly the cause of poorer outcomes, or is it a proxy for other, more fundamental patient characteristics and circumstances?

Let’s start with a thought experiment: Imagine a 63-year-old widow living alone with diabetes, osteoporosis, and shoulder arthritis. She qualifies for Medicaid due to low income and limited assets. Her shoulder arthroplasty is scheduled in two months. Now imagine she marries a childhood friend who provides her with different health insurance. Her comorbidities, living situation, and health literacy remain unchanged. Should we expect her surgical risk to suddenly drop?

Of course not. The administrative label of her insurer has changed, but her biological and social risk factors have not. Yet the implication of the aforementioned study is that the change in payer alone would significantly reduce her odds of a complication. This framing fails what I would call the "sniff test."


Propensity Matching Pitfalls

The study used propensity score matching (PSM) to compare outcomes between patients on Medicaid and those not on Medicaid. However, matching included only age, sex, and discharge weight. Critically, key confounders such as comorbidity burden (Charlson Comorbidity Index) were excluded from the matching algorithm. The groups remained imbalanced across multiple preoperative risk factors: congestive heart failure, chronic lung disease, anemia, renal failure, substance use, and more. These are not minor oversights—they are core determinants of perioperative risk.

More broadly, Medicaid enrollment is intertwined with social determinants of health: poverty, underemployment, poor nutrition, limited access to preventive care, and systemic racism. These powerful drivers of outcome often go unmeasured in administrative data. To ignore them is to risk mistaking correlation for causation.

This misattribution reflects a well-known cognitive bias: the fundamental attribution error. As Daniel Kahneman explains in Thinking, Fast and Slow, we often over-attribute outcomes to individual traits while neglecting situational forces. Medicaid status is being treated as a standalone risk factor, rather than a marker of cumulative disadvantage.

The authors did apply multivariate regression after matching, but residual differences in baseline health status suggest significant causes of confounding were unaddressed. The assertion that "Medicaid status was independently predictive" risks directing attention toward an insurance label—a variable that is neither biologically meaningful nor directly actionable for most patients—instead of toward modifiable drivers of inequity.

Instead, we should ask: Do patients insured by Medicaid fare worse after adjusting for what we can measure? If so, it tells us that we have incompletely identified the barriers these patients face in accessing and benefitting from coordinated, high-quality care.

Clinically speaking, we do not treat Medicaid status. We treat people. Let us not confuse an administrative variable with the medical and social factors that determine the outcome of treatment.


Revised Abstract (from ChatGPT)

Background: Prior studies report worse surgical outcomes in Medicaid-insured patients, but the extent to which these reflect insurance status versus underlying comorbidity and social disadvantage remains unclear. This study assessed the association between Medicaid insurance and short-term adverse outcomes after elective total shoulder arthroplasty (TSA) using a national administrative database.

Methods: We conducted a retrospective cohort study using the Nationwide Readmissions Database (NRD) from 2016 to 2020. Patients undergoing elective primary TSA were identified by ICD-10-PCS codes. Medicaid patients were matched 1:1 to non-Medicaid patients on age, sex, and discharge weight using nearest-neighbor propensity score matching. Covariate balance was assessed descriptively but not using standardized mean differences. Logistic regression adjusted for residual covariates including the Charlson-Deyo Comorbidity Index (CCI), select diagnoses, and geographic factors. Outcomes included 180-day complications, readmissions, mortality, and other surgical and medical complications.

Results: 15,448 patients on Medicaid and 15,374 patients not on Medicaid were analyzed. Patients on Medicaid had higher CCI scores, greater comorbidity burden, and lower income. Adjusted analyses showed increased odds of any complication (OR 1.2), readmission (OR 1.2), and mortality (OR 1.4) in the Medicaid group. Significant associations also emerged for dislocation, pneumonia, sepsis, and other events.

Conclusion: Medicaid insurance was associated with increased adverse outcomes after TSA, even after adjustment for the variables that were identified. These differences must be due to social and clinical risks that were not identified in this study. Medicaid should be viewed not as an intrinsic risk factor, but as a proxy for structural disadvantage. Policy and care improvements must focus on optimizing comorbidity management, care access, and coordination for all patients, independent of insurance status, i.e. patients on other insurance programs are subject to the same risk factors as those covered by Medicaid. We should focus on identifying and managing these social, economic and medical risk factors rather than the type of insurance.

I'd like to illuminate this last point with a chart from the paper


When looking at this chart our eyes are drawn to the two right hand columns. However, if we look at the second and third columns we see that very similar complications and rates of complications occur both in patients not covered by Medicaid (Control) and those that are (Medicaid). Non-Medicaid insurance is not protective against complications any more than Medicaid insurance causes complications.
 



Simple Guidelines for Propensity Score Matching (PSM)

  1. Define the Exposure Clearly

    • Must be binary (e.g., Medicaid vs. non-Medicaid insurance).

    • Clearly identify the treated and comparison groups.

  2. Choose Confounders Thoughtfully

    • Include only variables measured before treatment.

    • Choose variables that affect both treatment and outcomes.

    • Avoid variables on the causal pathway (e.g., complications).

  3. Estimate the Propensity Score

    • Use logistic regression.

    • Include relevant confounders.

  4. Match Patients

    • Use 1:1 nearest neighbor with a caliper (e.g., 0.2 × SD of logit).

    • Prefer matching without replacement.

    • Exclude unmatched cases.

  5. Check Balance

    • Use standardized mean differences (SMD < 0.1 ideal).

    • Visualize using Love plots.



  6. Analyze Outcomes Appropriately

    • Use matched methods (e.g., conditional logistic regression).

    • Don’t assume matched pairs are independent.

  7. Report Transparently

    • Describe covariates used.

    • State matching method and caliper.

    • Report group sizes pre/post-matching.

    • Show covariate balance.

    • Detail outcome analysis.


A Fun Illustration: Osprey vs. Bald Eagle



If we were to use PSM to compare the hunting success of an osprey and a bald eagle:

  • Outcome: Hunting success rate = successful captures / attempts

  • Exposure: Species (osprey vs. eagle)

  • Confounders to Match:

    • Habitat (river, lake, ocean)

    • Wind/weather

    • Time of day/season

    • Prey density

    • Flight altitude

    • Visibility conditions

Not Confounders:

  • Bird color

  • Vocalization

  • Rarity or popularity

  • Endangered status

As with surgery, outcomes depend not just on species — or insurance — but on context, environment, and opportunity.


Let us not confuse administrative categories for biological truths. Statistical adjustment is no substitute for structural insight.



You can support cutting edge shoulder research that is leading to better care for patients with shoulder problems, click on this link

Follow on twitter/X: https://x.com/RickMatsen
Follow on facebook: https://www.facebook.com/shoulder.arthritis
Follow on LinkedIn: https://www.linkedin.com/in/rick-matsen-88b1a8133/

Here are some videos that are of shoulder interest
Shoulder arthritis - what you need to know (see this link).
How to x-ray the shoulder (see this link).
The ream and run procedure (see this link).
The total shoulder arthroplasty (see this link).
The cuff tear arthropathy arthroplasty (see this link).
The reverse total shoulder arthroplasty (see this link).
The smooth and move procedure for irreparable rotator cuff tears (see this link)
Shoulder rehabilitation exercises (see this link).


Friday, April 5, 2024

Osteoarthritis: Hemiarthroplasty vs Total Shoulder. A pitfall of propensity score matched analysis

Randomization enables unbiased estimation of treatment effects; randomization attempts to assure that treatment-groups are balanced with respect to the important covariates. Unfortunately for us shoulder surgeons, surgical treatments are rarely assigned randomly.

Propensity matching is an attempt to use observational data to compare two treatment groups by accounting for the covariates that are associated with the outcome. 

The possibility of bias arises because a difference in the outcome between treatment groups may be caused by factors that predict which treatment the patient receives rather than the effectiveness of each treatment. For example if an observational study matching patients for age and sex alone retrospectively compared the recurrence rates after Bankart repair and after the Latarjet procedure, it would be at risk for an incorrect conclusion because it did not match for the size of the glenoid defect which may have affected the choice of treatment.




However, the title itself gives pause: why should a smaller operation (hemiarthroplasty) have a higher short term postoperative complication rate than a more involved procedure (total shoulder arthroplasty)? Sounds like a fundamental attribution error.

Let's take a deeper dive. The authors searched the American College of Surgeons National Surgical Quality Improvement Program database for records of patients who underwent either TSA or HA for glenohumeral osteoarthritis of the glenohumeral joint. 

Patients in each group underwent a 1:1 propensity match for age, sex, BMI, ASA classification, diabetes mellitus, hypertension requiring medication, congestive heart failure, chronic obstructive pulmonary disease, inpatient/outpatient status, smoking status, and bleeding disorders.  2188 received TSA and 2188 received HA. The question is, "among these supposedly similar patients, what determined whether they wound up getting HA or TSA? We'll get back to that question shortly.

The HA patients had a higher rate of any adverse event (7.18% vs 4.8%), death (0.69% vs 0.1%), sepsis (0.46% vs 0.1%), postoperative transfusion (4.62% vs 2.2%), postoperative intubation (0.5% vs 0.1%), and extended length of stay (23.77% vs 13.1%). 

Comment: While these differences are striking, it is apparent that putting in a plastic glenoid does not reduce the risk of death, sepsis, transfusion, intubation or extended length of stay.

As stated above, the possibility of bias arises because a difference in the outcome between treatment groups may be caused by factors that predict which treatment the patient receives rather than the effectiveness of each treatment. HA patients had a statistically significantly higher mortality probability (0.004±0.010 vs 0.002±0.003 and morbidity probability (0.027±0.015 vs 0.021±0.011) at baseline compared with the TSA cohort, even after propensity score matching.  Surgeons may prefer to perform HA for high-risk patients and those with more complex pathology. Less experienced surgeons may elect to perform HA because of its simplicity. Surgeons may be more likely to perform HA on patients that have worse social determinants of health (Economic Stability, Education Access and Quality, Health Care Access and Quality, Neighborhood and Built Environment, and Social and Community Context), which are known to be associated with inferior outcomes.

Thus, while the authors state 
"HA was found to increase the odds of developing these complications when baseline demographics were controlled",
 perhaps a more accurate statement would be 
"Patients for whom the surgeons chose HA were found have increased odds of developing these complications when the selected baseline demographics were controlled."

You can support cutting edge shoulder research that is leading to better care for patients with shoulder problems, click on this link.

Follow on twitter: https://twitter.com/RickMatsen or https://twitter.com/shoulderarth
Follow on facebook: click on this link
Follow on facebook: https://www.facebook.com/frederick.matsen
Follow on LinkedIn: https://www.linkedin.com/in/rick-matsen-88b1a8133/

Here are some videos that are of shoulder interest
Shoulder arthritis - what you need to know (see this link).
How to x-ray the shoulder (see this link).
The ream and run procedure (see this link).
The total shoulder arthroplasty (see this link).
The cuff tear arthropathy arthroplasty (see this link).
The reverse total shoulder arthroplasty (see this link).
The smooth and move procedure for irreparable rotator cuff tears (see this link).
Shoulder rehabilitation exercises (see this link).