Comparing Trials That Never Met

We recently refused to compare two obesity drugs across separate trials, and explained why that is not a comparison.^[1] Here is the methodologically proper version of the thing we refused — an anchored indirect treatment comparison, using individual patient data. It is genuinely better than lining two headlines up. It is still not a head-to-head, and one of its numbers has a confidence interval spanning a factor of nearly fifty.

What an anchored comparison does

The problem with reading across two trials is that the participants differ. If one enrolled heavier people, or more women, or people with worse blood sugar, then the difference between the headline figures partly measures the difference between the populations.

An anchored comparison exploits the fact that both trials contained a placebo arm. Instead of comparing drug A’s result to drug B’s, it compares each drug to its own placebo group, and then compares those two differences. The placebo arms act as a shared reference point — the anchor — which cancels out a great deal of what makes the trials incomparable.

This analysis went further. It had the individual patient records from one trial and only summary figures from the other, and used that to statistically reweight the first population to resemble the second on sex, body weight and blood sugar status. That is a real methodological step up from arithmetic on two press releases.

It is the difference between comparing two results and comparing two effects.

What it found

The weight difference is on firmer footing: about three percentage points, with intervals that stay comfortably on one side of zero, and consistent across both ways of analyzing it.

It seems to contradict a head-to-head trial. It does not.

A reader who followed our coverage of ACHIEVE-3 will notice a problem. That was an actual randomized head-to-head between these same two drugs, and orforglipron won. Here oral semaglutide wins. Both cannot be right — except they are measuring different things.

Different outcome, a semaglutide dose nearly twice as high, and a completely different population. These are two answers to two questions, and neither refutes the other. What would settle the weight question is the trial nobody has run: these two drugs, at these doses, randomized against each other, in this population.

On one point the two agree, which is worth more than either alone: orforglipron produced more discontinuation for side effects in both. ACHIEVE-3 recorded roughly double, and this comparison points the same way. Agreement across two independent designs is stronger evidence than either provides on its own.

What still limits it

• It is not a head-to-head. Anchoring removes much of the population difference and cannot remove differences in how the trials were run, who dropped out, or what care participants received alongside.
• Adjustment only covers what was measured. Sex, weight and glycemic status were balanced. Whatever else differed was not.
• Several authors work for Novo Nordisk, which makes oral semaglutide, and the comparator is a competitor’s drug. That is disclosed, ordinary, and worth knowing.
• The tolerability estimate is barely constrained, as the interval above shows.

Our position from the earlier article stands, slightly refined. A cross-trial comparison assembled by a reader from two headlines is not evidence. A properly anchored, population-adjusted comparison published with its confidence intervals is evidence — weaker than a randomized head-to-head, stronger than nothing, and only as good as the intervals it reports.