Scientific deep-dive
How to Trust an Observational Study
We have twice warned that a very large hazard ratio from an observational study is telling you about the study. This one, reporting 0.42 and 0.58, deserves trust, and the article names the specific techniques that earn it.
We have twice warned that a very large hazard ratio from an observational study is usually telling you about the study rather than the drug. This one reports hazard ratios of 0.42 and 0.58 and deserves considerably more trust, because of a set of specific techniques you can learn to recognize.[1]
The technique that earns the trust
Before asking their real question, the researchers used their claims database to emulate two randomized trials whose results were already published, STEP-HFpEF DM and SUMMIT. They rebuilt each trial’s eligibility criteria, exposures and outcomes inside the data, ran the analysis, and compared what came out against what the trials had found.
The emulations agreed with the trials on every prespecified metric.
Only then did they widen the eligibility criteria to the kind of patients seen in ordinary practice (the people trials exclude) and ask the question nobody had randomized evidence for.
The second technique is negative control outcomes: outcomes the drug has no plausible reason to affect. If the results were being produced by healthier people getting the newer drug, those outcomes would improve too. They did not, which is evidence the effect is not simply a healthier group in disguise.
What they found
The question was heart failure with preserved ejection fraction alongside obesity or type 2 diabetes, a condition where early trials were promising but rested on very few actual events. The outcome was a composite of hospitalization for heart failure or death from any cause, over up to 52 weeks. Sitagliptin, a drug with no expected effect here, served as the comparator.
| Comparison | Patients | Hazard ratio (95% CI) |
|---|---|---|
| Semaglutide vs sitagliptin | 58,333 | 0.58 (0.51–0.65) |
| Tirzepatide vs sitagliptin | 11,257 | 0.42 (0.31–0.57) |
| Tirzepatide vs semaglutide | 28,100 | 0.86 (0.70–1.06) |
Secondary endpoints, subgroups and sensitivity analyses were consistent, and no substantial safety signal appeared.
The third row is a null, and it matters
Set head to head, tirzepatide showed no meaningfully lower risk than semaglutide. The hazard ratio was 0.86, with an interval running from 0.70 to 1.06 that includes no difference.
This matters because tirzepatide produces substantially more weight loss. If the benefit here ran through weight, the drug that removes more of it should win. It did not, which fits the larger pattern we have covered: the cardiovascular benefit does not track weight loss, and a synthesis of 262 trials similarly found the drug with the mortality evidence sitting fifth on the weight table.
What still limits it
- Sitagliptin is a placebo proxy, not a placebo. It is chosen because it should not affect these outcomes, and it is still a drug given to people for reasons.
- Claims data cannot see an echocardiogram. Preserved ejection fraction is inferred from diagnostic codes, so some patients are certainly misclassified.
- Fifty-two weeks, in a chronic condition.
- Benchmarking validates the method against two specific trials, in the populations those trials enrolled. It does not guarantee the method behaves as well in the expanded population, which is precisely where no randomized answer exists.
That last caveat is the real limit of the whole approach, but it does not undo the value of the exercise. A method shown to work where it can be checked is a better instrument than one never checked at all.
The portable version
When you meet a large effect from an observational study, these are the questions that separate the credible from the merely striking.
- Did they benchmark against a known answer? Emulating an existing trial and reproducing it is the strongest signal available.
- Are there negative control outcomes? If everything improved, the patients differed, not the treatment.
- Is the comparator an active drug or nothing? Comparing against another treatment partly controls for who gets prescribed what, which is why a study of GLP-1s and epilepsy measured them against a different diabetes drug.
- Was the outcome recorded for everyone? One surgery-versus-drug comparison balanced 30,458 people, then took its weight figures from the 1.6% of the drug arm whose weight was on record.
- Is the effect size physiologically plausible? This remains the first filter (see when a hazard ratio is too good).
Frequently Asked Questions
References
- 1.Krüger N, Schneeweiss S, Fuse K, et al. Semaglutide and Tirzepatide in Patients With Heart Failure With Preserved Ejection Fraction JAMA. 2025. PMID: 40886075.
Compare the sellers
Related research
The Biggest Number Is the Weakest
A prespecified SELECT analysis reports cardiovascular reductions of 26%, 21% and 34% across three definitions of liver fibrosis risk. The 34% is the only one that was not statistically significant.
7 min read
When Pooling Studies Hides the Answer
A meta-analysis of 323,439 patients found no significant cardiovascular advantage for tirzepatide. The more important number sits beside it: the studies being pooled disagree with each other almost completely.
7 min read
A Biomarker Is Not an Outcome
Sixteen cardiovascular risk markers were measured in SURMOUNT-1 at 72 weeks, and most improved substantially. Cardiology has learned twice, expensively, why that is not the same as preventing events.
8 min read
A Benefit Nobody Observed
A cardiology journal reports GLP-1 therapy could cut ten-year cardiovascular risk by 22% in people who have never had heart disease. Nobody in the study took the drug.
6 min read
Seizures When You Already Have Epilepsy
A careful study of epilepsy and type 2 diabetes found fewer seizures with GLP-1s, and 60% lower mortality. No glucose-lowering drug does that, and the implausible number is the informative one.
7 min read
Real, and Below the Threshold
Semaglutide improves patient-reported physical functioning by 1.71 points, with an interval excluding zero. The threshold at which a patient notices a change is 3.7 points. Both are true at once.
6 min read
Where to get GLP-1 online, safely: sellers our editors have checked
These are telehealth sellers our editors have checked. For each one we hold a price, the form the drug comes in, and the states it reaches.
No insurance needed · vetted by our editors
Some of the links on this page earn us money. If you sign up with a provider after following one, that provider may pay GLP Watchdog a commission. Learn more
Embody
A flat compounded tirzepatide or semaglutide price at every dose, from a seller that names its pharmacies
From $79/mo
Get started →Direct Meds
Compounded semaglutide at $249/month
From $249/mo
Get started →Collective
Flat any-dose pricing, if you can absorb a $199 annual membership on top
From $59/mo + membership
Get started →