Different Views of the Truth: Drug Efficacy

Based upon data from a drug study, the results may be viewed from many perspectives, ranging from sounding optimistic to trivializing its value.

Revised: 11-10-2023

Calculating the Real World Costs and Benefits of a Drug

The ancient fable about the blind men who examined an elephant and reached laughably divergent and erroneous conclusions about what they had experienced was, obviously, meant to teach the lesson that our senses may deceive us and, perhaps, that we shouldn’t jump to conclusions. It’s not all that different with how the data from a drug study gets used or abused. Have you ever wondered just how effective a drug (or equivalently, some other treatment) really was? It’s not easily answered and even when one looks, there may still be much uncertainty. I’ve chosen a single drug study to explore, but I’m sure the general principles will apply to any drug or other medical product.

This essay looks at the JUPITER trial which tested rosuvastatin (Crestor). I had a personal interest here, as I took the drug for about three years. Even if you’re not one the hundreds of milllions on a statin, please note that all the comments here can be generalized to similar studies of any drug, indeed, of any trial that compares treatment vs. control.

Assuming data in the NEJM are true, I offer some very different observations all directly derived from that data. Unless I’ve made an error, all the figures I will claim are “true”, that is they follow from the study’s reported data or can be derived from those. I’m not so confident about my estimates of cost of treatment as those are based on rough estimates. I’m even less sure of the accuracy of my projections of added life expectancy; those doubts are better explained in their specific sections.

Recall the old saying that figures don't lie but liars can figure? Beyond the most basic truth claims, very few things are patently true or false. Our messy, complicated real world instead offers us many shades of gray. Too, one must discount the understandable tendency to spin the figures to suit the message. If you're in marketing or otherwise advocating a treatment, you will promote those relative improvements. A government committee should be concerned with the billions of dollars a public health intervention will cost. Will the benefits be worth the costs? And then there's cynics like me and perhaps you, who wanted to "run the numbers, " to look under the hood, and to see how (in)effective this drug is. There always exists the problem of outright fraud and other deception and it’s apparently pretty common in drug trials. Millions, even billions of dollars of potential income are at state, not to mention the future career prospects of all involved. These and related factors are beyond the scope of this short essay, but they are mentioned in passing because they will always bias results. More relevant to this essay, my thesis is that there are almost always different perspectives on the truth. Different players will give different reports, shaped not only by what they considered valid observations, but also by conscious or unconscious motivations to spin the narrative for their own ends.

I've tried to arrange the following statements from more to less favorable.

Patients taking Crestor (Rosuvastatin) could realistically expect...

Non-fatal MI (heart attack) reduced by 64% RRR (relative risk reduction)

Non-fatal stroke reduced by 48% (RRR).

Death from any cause reduced by 20% RRR.)

Incidence of fatal stroke reduced 50% (RRR.)

Now all those sound great, which is why they use relative ratios. But when one looks at absolute numbers, they appear much more modest:

(Improvements)

Non-fatal MI: 62-22 = 40/8,901 => 0.0045/1.9 years = 0.0024 = 0.24%/year ARR (absolute risk reduction; note Note that a time unit should be specified).

Non-fatal stroke: 58-30 = 28/8,901 => 0.0031/1.9 years = 0.0017 = 0.17%/year ARR.

Death from any cause: 247-198 => 49/8,901 = 0.0055/1.9 years = 0.0029 = 0.29%/year ARR.

Fatal MI: an INCREASE from 6 to 9, absolute increase = 0.018%/year, relative jump 50% (Yes, I could claim that fatal heart attacks jump by 50%!!!)

Fatal Stroke: 6-3 in favor of treatment; absolute reduction = 0.018%/year ARR; 50% RRR in fatal strokes. (I'll copy that into the "more favorable" zone).

The authors went to some effort to disguise the unfavorable data. In fact Kendrick (Doctoring Data, pp. 72-73) credits another researcher for showing him how to arrive at the “hidden” death statistics. Treatment and control had 12 deaths each from MI and Stroke. That’s a rather embarrassing result for a drug that’s intended to reduce such, one might think. Is it any wonder that the authors disguised the data as well as they could?

What is NNT (number needed to treat?) Those are derived from the absolute rates. Cost to treat: This will vary according to many assumptions, but here are my estimates: Annual cost of treatment: 1 lab test + 4 office visits + drugs = $100 + $400 + $100 = $600 (for now, don't worry about insurance paid vs. out-of-pocket.) Let's call this a ballpark estimate of the cost to treat one patient for one year. Imagine how much pricier when a drug is still on patent.

To “prevent” (delay) a single instance of each event requires this number of years of treatment and (approx.) cost:

Non-fatal MI: 1/0.0024 = 417 patient-years ($250,200).

Non-fatal Stroke: 1/0.0017 = 588 patient-years ($352,800).

Fatal MI: Er, let's change the subject!

Fatal Stroke: 1/0.0018 = 556 patient-years ($333,600).

Death from any cause: 1/0.0029 = 345 patient-years ($207,000).

Treatment for me (61 years old) adds 2.1 years of LE (by my best calcuations, which I admit are of low confidence!) The cost to treat me for the rest of my life (rough estimate of course!) is about $600/year. Using the endpoint to prevent one death, if I do the math right it's 345 patient-years; thus the cost to delay one death should be $600 X 345 = $207,000. That budget could treat about 17 men for their remaining LE. From the standpoint of public and private health costs, is it worthwhile to spend $207,000 to extend the life expectancy of a group of 17 61-year-old men by two years? Stated another way, those two "extra" years of life actually cost 207,000/17 => $12,000 per man. Mind you, this with their shiny 20% risk reduction. More typical statin RRR would more than double that cost. (Reiterating the disclaimer: I'm still very much at the kindergarten stage when it comes to this actuarial stuff, so it's entirely possible I've made a mistake in these figures.)

Statins are one of the world's most widely prescribed drugs. As a public health measure, they would seem to have a fairly low value. For that one man whose death will be "prevented" (delayed, really) 344 men will receive no benefit, as well as lose time spent seeing doctors and pharmacy, as well as bear the risks of adverse drug effects. Unless all his medical costs are paid by others, he'll likely pay several grand out of pocket, in constant dollars, over the decades of treatment. Cynics might suspect that such large-scale, low-value interventions mayhap are promoted more to line the pockets of drug makers and the rest of the medical guild, rather than actually help patients.

Change in Life Expectancy (LE): This is one of the more difficult to find or calculate figures. Given the promise of a 20% reduction in all-cause death rates, how does that translate into LE? The aswer is dependent upon the risk reduction, the time period, and the underlying actuarial death rates, usually diferent for male and female. For this example, I'‘ve tailored it to my own age (61). For reference, the SSA Life Table gives my “control” LE = 82 years.

Most statin studies disclose more modest RRR, in the 8% range. But to be fair, the study I’m using claimed 20%, so I’ve used that in my calculations. I used a custom spreadsheet to do the analysis. Please note that I have very low confidence in my life expectancy calculations. I’ve tried to estimate the LE (50% survivial) rates. With treatment my LE is about 83.1, or a net gain of 2.1 years, call it 25 months. Problem: my estimate is far higher than what others claim is typical.

Will the real LE please stand up: 25 days, 25 months, or 7.25 days?

As I’ve stated multiple times, I have low confidence in my models. It seems my estimates are on the high side. Apparently reputable sources consistently estimate far lower benefit from statin treatment. Now I found a paper that looked at statin studies, including JUPITER which I played with in this essay, and it finds an even shorter LE. What’s going on???

While re-reading his The Clot Thickens, he makes notes the following in Chapter 8 (Increase in Life Expectancy):

“… you could chew gloomily on statins for 40 years. An activity that will require regular appointments with your GP. Added to this, you will have to pay for a prescription every month. Then you must remember to pick it up from the pharmacy. You will also need blood tests from time to time to check your cholesterol level – and suchlike. As an added bonus you may well get to suffer adverse effects for the next 40 years. You can then obsess and worry about your cholesterol level. For all this you may gain just under a month. Three days extra for every five years taking the statin.” (bolding mine)

This alerted me to the possibility that my “corrected” computation method may over-estimate the benefit. Or perhaps not. I’ll need to check Kenrick’s cite and see what if anything I can learn.

Here’s the study he cites:

Kristensen et al. The effect of statins on average survival in randomised trials, an analysis of end point postponement. BMJ Open. May 2015.

https://bmjopen.bmj.com/content/5/9/e007118

A quick look at the Kristenson et al Study shows several trials.I’m most familiar with JUPITER and see the RR (“Risk Reduction”) given as 0.80, which is consistent with the 20% reduction I found earlier in the present essay. They claim “Postponement, quick method, days” = 7.25 days. The methods the researchers used were to examine graphs of surival vs. time for the studies available. In some cases they literally counted pixels; the “quick method” was to draw a triangle and obtain its area. Some of my other statin essays discuss this study, or the reader may refer to it directly.

Conclusion

There you have it, a more or less real world calculation, using Pharma's own study, of a drug I took for nearly three years. Other than the cost assumptions (and assuming no mistakes in my math), all the above you see are different perspectives on The Truth what the actual costs and benefits of that drug would be for me.

I have a certain admiration for the trickery that the authors of these studies resort to.

Yes, indeed, it’s true that this drug will reduce your risk of a heart attack or stroke from half to two-thirds, and even better, it’ll cut your chance of dying from any cause by 20%. (Shamlessly borrowing from Kendrick, I call this the “Buy two tickets to the lottery and you improve your chances of winning by 100%” theory.)

All true enough, but when one digs into the numbers, it looks a lot less alluring. “It all looks fine, to the naked eye, but it don’t really happen that way at all,” sang The Who. Or maybe Tom Waits is more apropos: “The big print giveth and the small print taketh away.”

[Credit where due: I relied heavily on Kendrick and others I cannot recall. What I’ve done here is learn how the different values are computed, then rework the numbers for this specific example.]

Works Used:

Kendrick, Malcolm. Doctoring Data. How to Sort Out Medical Advice from Medical Nonsense.

Kristensen et al. The effect of statins on average survival in randomised trials, an analysis of end point postponement. BMJ Open. May 2015.

Ridler et al, Rosuvastatin to Prevent Vascular Events in Men and Women with Elevated C-Reactive Protein, New England Journal of Medicine 11-20-08 (2195-2207).