The Little Secret Hiding Behind Treatment Success Rates

I was going through an old hard drive the other day when I found the audio recording I made of my wife's office visit with her vascular surgeon from 2010. My wife had had an MRI several years earlier for tinnitus and radiologists spotted something unrelated and unexpected—an unburst aneurysm along her right cerebral artery situated about an inch inside her brain. It had remained stable until some months before the office visit in question when we learned that in less than a year it had nearly doubled in size.

This doctor is a world-renowned surgeon and a world-class jack-ass. Nearing retirement, he had founded the neurosurgery department at this hospital and was the only surgeon within the entire Southwestern United States who had any experience with this exact type of aneurysm. We were fortunate to be living just three hours from him. Other patients were coming from two and three states away for his services.

Ultimately, he performed an open craniotomy (hole in the skull) on my wife, excised the aneurysm, and since the aneurysm was kind of folded over on itself, he was able to simply sew the two ends together, eliminating the need to use clips or coils or anything unnecessary. It took nearly seven hours but the result was masterful, and she made a full recovery.

Both my wife and I learned a lot about a lot of different things related to medicine during that entire process, and listening to that audio recording reminded me of one very valuable lesson.

After he finished explaining what he might do during the surgery (he was always vague about his exact intentions, except to say he wouldn't know for sure what the best course was until he "got in there"), my wife asked him what the success rate of this procedure was.

He said the success rate was 91 percent.

Naturally, my wife heard him say that the failure rate was nine percent, but I was impressed: Ninety-one percent success rate on such tricky brain surgery? That's amazing! We talked about a lot of things on the drive home, but I couldn't shake how astonished I was at that success rate.

I also couldn't shake a feeling of déjà vu. I'd seen that number before, but I wasn't sure where, or when. Not until we got home did I figure it out.

That consult had been the second consult with him. After the first one, I had searched his name on PubMed and found a paper he co-authored in the journal Neurosurgery. The paper was a retrospective study of outcomes of patients with brain aneurysms of any kind who had undergone surgery at that hospital.

Retrospective studies are inherently weakened by the fact that there are no controls in place, no assured consistency in treatment from one patient to the next. A retrospective study like this is merely looking at a bunch of similar cases from the past and drawing a conclusion about them. The contrary to a retrospective study is a prospective study, wherein patients with the same condition are recruited and receive the same treatment (whether that is the treatment being tested or placebo). Prospective studies are far more valuable than retrospective ones.

At any rate, re-reading the study I found the number: 91 percent.

Then I was reminded of how small the retrospective study had been: just 38 patients, with maybe a handful having an aneurysm even remotely similar to my wife's. Still, if you did the math, you saw that 34 patients experienced a successful surgery.

If you read the paper, you saw a definition of 'successful' that wasn't how you would define it. In short, 'success' meant you survived your time on the operating table and for at least six months after that. You didn't have a stroke or otherwise die in surgery. You may not have emerged from the procedure anywhere near 100 percent-- In fact, 66 percent of patients developed new, postoperative problems including severe headaches, cognitive issues and diminished neurological function-- but that didn't stop the authors—including my wife's future surgeon—from lumping them into the 'success' group.

During the consult, he delivered this 91 percent figure with all the authority a physician carries by virtue of his being a physician. We bought it hook, line, and sinker. Clearly he hadn't been dishonest or especially misleading when citing that figure.

But let's be straight here: when he did cite that figure, he had stopped talking to us as normal, non-medical people and for just a moment he had started addressing us as medical colleagues—he just omitted mentioning as much. When citing that figure, he was momentarily treating us as people who would understand that 91 percent didn't really mean 91 percent the way the average person believes it to mean, but 91 percent in medicine, where nothing is nearly as concrete or cut-and-dried as many of us think it is.

If this weren't the case, then he would have laid out what 91 percent meant; how it sounded like a nice clean figure but was in reality loaded with modifiers and qualifiers that, if actually taken into account, would have resulted in a number a bit lower than 91.

Cancer is overloaded with these kinds of figures: success rates, survival rates, five-year survival percentages, and more. Oncologists routinely cite these figures when pressed by their patients. They have even been known to hedge these figures by telling their patients that while the published success rate of this or that treatment is X, they see much better results in their practice.

Hearing the recording reminded me of how shocked I was to learn that this surgeon was telling us that the success rate for this procedure was 91 percent, and he either made it sound like this was some nationally published and widely accepted figure or he merely allowed us to think that it was because he knew that we would, when the figure was actually based on a miniscule little retrospective study.

I concede that this figure is all that he had to give us. I concede that he didn't really mislead us or do anything unethical. And I admit that it was a very comforting thing to hear (at least for me).

But I also learned that in medicine nothing is quite as certain as it seems. I learned that contrary to perceptions, uncertainty is the defining principle in modern medicine. And I learned that your doctor is acutely aware of this.

You should be too.

Photo by Francisco Osorio