FYI.

Donald

DJP Update 6-29-2006 Medical Error Statistics - WSJ "Numbers Guy" 
discussion

The Wall Street Journal Online

June 29, 2006

THE NUMBERS GUY
By CARL BIALIK


In December 2004, Harvard professor Donald Berwick launched an  ambitious effort aimed at improving U.S. hospitals. His goal was to  save 100,000 lives in the next 18 months by convincing hospitals to take steps to cut down on errors and improve care.

Two weeks ago, a group led by Dr. Berwick announced the results to great fanfare: Hospitals participating in the effort had saved 122,300 lives, exceeding expectations. Much of the favorable press coverage mentioned that this was comparable with the estimated 44,000 to 98,000 lives lost each year to medical errors in hospitals.

But such estimates have an inherent drawback: It is difficult, in many cases, to connect whether a patient dies to a single medical error or procedure. Dr. Berwick's group compared death rates during the study with those before the study, but there's no way to know that the improvements came from the group's campaign and not other factors. Also, the group relied on self-reported numbers from hospitals. It's possible that only hospitals with positive outcomes shared information.

Questioning the Numbers

Several of Dr. Berwick's colleagues told me they admire his goals but question his number. "I have no doubt the campaign was a good thing and saved a lot of lives," said Robert Wachter, a professor in the department of medicine at the University of California, San Francisco, and the author of a book about medical errors. "I don't think it saved 122,300." He added that, like in a political campaign, the health-care campaign used "statistics selectively to try to mobilize your base to do good. It's understandable. It's not good science."

H. Gilbert Welch, senior research associate with the Department of Veteran Affairs in White River Junction, Vt., and a critic1 of medical-error estimates, said, "I think there's been a tendency in the errors business to first overstate the size of the problem, and now, I'm afraid, to overstate the effect of interventions on the other side."

The study received broad press coverage, with many accounts headlining the 122,300 number. U.S. News & World Report's headline read2, "122,000 Who Lived." An Associated Press report on the study was published in several newspapers, including The Wall Street Journal3.

The Institute for Healthcare Improvement4, the group behind the study, promised hospitals it wouldn't release the data they provided. However, it offered a customizable press release5 for those that wished to trumpet their results, and several newspaper reports (such as those in the Toledo Blade6 and the Fresno Bee7) included anecdotes and quotes supplied by nearby participating hospitals.

The IHI press release8 was careful to say only that hospitals in the campaign saved 122,300 lives, and Dr. Berwick, who serves as IHI's president and chief executive, told me "it would be stupid" to say the group's campaign was solely responsible. "We think we added to it," he said. IHI is a Boston-based nonprofit organization aimed at improving health care. Its funding comes from charitable foundations and hospitals.

Other Factors

Many hospitals had already undertaken their own efforts to boost the quality of care, prodded by groups such as the Centers for Disease Control and Prevention. But many of the news articles simply stated that the campaign saved all those lives, without qualification. (Numbers Guy readers Dean Anderson, Kirk Jeffrey and Curtis L. Russell spotted some of these articles and suggested I look at this number.)

The IHI initiative called on hospitals to institute six steps9. While one of the steps was aimed at reducing mistakes in administering drugs, the rest appear to focus less on errors and more broadly on improving care: Hospitals were encouraged to take additional steps to prevent infections, for instance. (Not all participating hospitals followed all of the recommended steps.)

By the 18-month mark of the campaign, more than 3,000 hospitals were participating, representing about three-quarters of all the nation's hospital beds. Most of these hospitals reported to IHI the number of admissions and deaths during the campaign period and in 2004 -- the group says information it received was about 86% complete.

IHI filled in the numbers for the rest by extrapolating, a step criticized by Dr. Wachter, who said that those hospitals with the best results may have been more likely to report. "It would be like going to your high-school reunion and extrapolating from the divorce rate and waist line [of those who show up] that everyone is married and stays thin," he said.

A simple comparison of the death rates before and during the campaign showed that roughly 33,000 fewer patients died in participating hospitals than would have been expected, based on the year-earlier results. But that calculation was flawed, Dr. Berwick argued, because the mix of patients changes -- you wouldn't compare mortality among 10 heart-attack victims and 10 sufferers of the flu.

So the group tried to come up with a number that would capture how much sicker patients were during the study than the year earlier. It relied on estimates from three companies that have access to a wide range of data from hospitals, including information on patients' diagnoses and their ages. Partly because the population is aging, those companies all reported that patients are, on average, arriving at hospitals in worse condition than they had in previous years. Based on the data, the group adjusted its estimate upward to 122,300. (IHI explains its methods in a document10 on its Web site.)

Andy Hackbarth, an IHI senior engineer who helped crunch the numbers, said the campaign's effect won't be understood fully until the group can compare its hospitals with others that didn't participate, and compare the results with hospitals' death-rate trends before the campaign began. The group plans to gather those numbers in the next six months and include them in a paper to be submitted to a peer-reviewed journal, Mr. Hackbarth told me.

A Personal Connection

Dr. Berwick found himself at the center of the quality-of-care issue following the hospitalization of his wife, Ann, in 1999. She waited 60 hours for treatment while gravely ill, and three times was left alone on a gurney, though she did eventually recover and leave the hospital. He was the subject of a front-page article11 in the Journal in 2002.

When he launched the campaign in 2004, Dr. Berwick said he was losing patience with the health-care system's halting efforts to respond to a 1999 estimate of deaths due to medical error. That study, from the Institute of Medicine, a National Academy of Sciences group that advises Congress on health, found that 44,000 to 98,000 people die each year because of medical errors. Those numbers remain frequently quoted to this day, even though the numbers are based in part on the experiences of hospital patients 22 years ago.

Much of the public's outraged reaction to the Institute of Medicine report focused on rare, terrifying cases such as botched surgeries or gross misdiagnoses. In an episode of her talk show, Oprah Winfrey asked about 260 members of her studio audience to stand up, and told them that, by a "conservative estimate ... this is the number of people who die every day in hospitals from medical mistakes." Then, her show featured one guest who was misdiagnosed with cancer and underwent a hysterectomy and six months of chemotherapy. Another guest's breasts were removed after she was falsely diagnosed with cancer because of a paperwork mix-up.

But critics of the study pointed out that many of the errors were less blatant. The estimates were based on reviews of hospital discharges in three states -- New York in 1984 and Colorado and Utah in 1992 -- looking for any "adverse events" caused by treatment, and not the underlying condition being treated. But not all adverse events are preventable. For example, if a drug deemed necessary causes dangerous side effects in 1% of patients, it's not known which 1% will suffer. For those who do, that's unquestionably an adverse event, but it may not have been a medical error -- it's certainly less clear-cut than having the wrong leg removed in surgery. (The study tried to estimate which adverse events were preventable, but 
had difficulty establishing clear cause-and-effect relationships.)

Dr. Berwick told me he considers the Institute of Medicine study "as good a quantitative estimate as we've got," adding, "the point is that [errors] are very large causes of morbidity." (Some estimates have put the figure higher than 98,000. Health Grades Inc., which maintains a Web site12 that measures hospitals based on mortality and complication rates, counted13 195,000 deaths annually between 2000 and 2002.)

Both the recent study and the 1999 medical-errors analysis raise the question: Are all deaths equal? Both counted each life lost or saved the same way. "A patient who has a terminal illness, who will die next week, but would have died this week because of medical error, counts as a life saved," Dr. Berwick told me. So does the life of a child saved from medical error who is discharged from the hospital in good health.

Harold C. Sox, Jr., editor of the Annals of Internal Medicine in Philadelphia, told me that averting deaths is, of course, a good goal, but there are other improvements to care worth pursuing. "Another target could be trying to improve the quality of death, for people for whom death is inevitable, and, as much as possible, to match where death actually occurs with what the patient desires," he said.

Although Dr. Wachter remains critical of the lived-saved estimate trumpeted by Dr. Berwick's group, he told me that he believes even questionable numbers can galvanize the public for the improvement of health care. "There will always be tension between good science and the laudable goal of [Dr. Berwick] and others to move the calcified health-care system forward," he said.

Write to Carl Bialik at numbersguy@wsj.com

* * *