Research Reports From Consumer Reports WebWatch Analyze Problems Facing Consumers And Health Web Site Publishers

September 30, 2003, YONKERS, N.Y. -- Consumers need better help to navigate the rapidly growing amount of health care information available on the World Wide Web, even though at least two dozen sets of guidelines and criteria for improving health Web site quality have been published. At the same time, newspapers, trade journals and other media should do more to call attention to those attributes that make a health Web site relevant and useful, according to two studies published today by Consumer Reports WebWatch.

The first study, which evaluated existing criteria sets for health sites put forth by medical institutions, professional groups, government organizations and others, is a joint project of the Health Information Institute, Bethesda, Maryland, and Consumer Reports WebWatch, and was authored by Peter Goldschmidt, M.D., PhD., and the institute's founder and president. The second study, analyzing discussions of health Web sites in the medical and popular media, was commissioned by Consumer Reports WebWatch from the Annenberg School for Communication at the University of Pennsylvania, Philadelphia. It was directed and edited by Joseph Turow, Ph.D., Robert Lewis Shayon Professor of Communication, and a member of Consumer Reports WebWatch's advisory board.

Together, the studies illustrate problems facing consumers and health Web site publishers, as the number of health sites grows and as more people use them to seek information. At least 22 sets of criteria exist for evaluating Web site information. Many are duplicative, only a quarter are intended by their authors for consumer use, and no single set has emerged as useful to consumers.

In addition, there is disagreement among the health community and the medical press about what constitutes a "quality" health Web site, what Web sites actually exhibit, what Web sites actually achieve "quality," and how important it is that a Web site be "clinically accurate" — in other words, that it presents information matching accepted medical wisdom on a particular treatment, syndrome, disease or drug. Further, in the popular press, health site "seals-of-approval," accreditations and other means of judging quality have had little to no visibility.

"This is a difficult situation for consumers, and an almost impossible one for health sites," said Beau Brendler, director of Consumer Reports WebWatch. "Who are consumers supposed to listen to, and how can health Web sites be expected to measure up to so many different sets of rules?"

The good news, Brendler said, is that more work has been done on credibility and quality issues among health Web sites than any other type of site. "But there's been a lot of talking and not much leading," Brendler said. "A trusted organization, group or individual needs to come forward with a set of guidelines consumers can use, that make sense to health Web site publishers, and that guide people toward better sites and away from bad ones.

"Highlights of the Health Improvement Institute/Consumer Reports WebWatch study:

• None of the existing 22 criteria sets for health site quality met the Institute's evaluation criteria. The most telling failures were: lack of testing, lack of consumer-friendliness, and failure to define criteria in operational terms — in other words, criteria are not useful to create evaluations or ratings.
  
  
• There are no criteria sets readily usable by consumers, and none readily usable by professionals to reliably assess health Web sites.
  
  
• One previously published study, found by the Institute, evaluated the quality of Web sites related to asthma. That study's authors used the Health On the Net Foundation (HON, http://www.hon.ch/) code as the basis for their evaluation. They found that 16 percent of the health sites they examined met HON Code principles. Of sites they found that displayed the HON logo, only 38 percent actually complied with the HON principles.

"There are no user-friendly tools for consumers to use, and they cannot rely on existing seals of approval, to assess the credibility or reliability of health websites," said Goldschmidt, who authored the study. "Our findings to date validate the Institute's work and the need for programs such as Consumer Reports WebWatch to draw attention to a significant problem and the lack of adequate solutions, and to work toward enabling consumers to evaluate Web sites and health information."

Highlights of the University of Pennsylvania Annenberg School study:

• In popular media, the Web is overwhelmingly treated positively as a way to search for health information.
  
  
• Systematic critiques of Web sites are rare.
  
  
• Health Web sites are mentioned largely in passing to quote information or to send people to information.
  
  
• Seal-of-approval programs, such as HON, are almost never mentioned.
  
  
• Mentions of best practices — indicators of quality that Web sites have and which audiences ought to consider — are rare.
  
  
• The "digital divide" — that Web information is most accessible to those who can afford it — is not presented as an issue. Neither is Web literacy.
  
  
• Sites of government medical organizations, often among the most respected for objectivity, alter medical information for political, as opposed to medical, reasons.

"Our findings suggest that the popular press, which is filled with health stories, is doing little to help the public understand one of the biggest health stories in decades — the rise of an enormous and often confusing online health information universe," said Professor Turow, who directed the study. "Advocates of better health information ought to work with producers of magazines, newspapers and TV programs to address how their audiences can become critical users of health information online."

In the future, Consumer Reports WebWatch and the Health Improvement Institute plan to develop independent ratings of health Web sites. In the meantime, concerned consumers may want to judge health sites considering these quality characteristics, based on the Institute's research and Consumer Reports WebWatch's guidelines for improving Web site credibility:

• Sponsorship. Who owns and operates the site? Is it a medica; institution, a group of doctors in private practice, a company selling nutritional supplements, a pharmaceutical manufacturer?
• Purpose. What is the Web site's intent? What is the scope of information presented? Does it publish positive and negative information about a particular treatment or drug?
• Audience. Who is the Web site intended for? What is the intended use for the information on the site?
• Currency. How often is the site updated? Does the site say how frequently information is updated, and the date of last update?
• Sources and credentials. What is the origin of information cited, and what is the expertise of the person who wrote it? Does the site contain this information?

METHODOLOGIES
The Health Improvement Institute conducted Web and literature searches in January and February 2003 to identify criteria sets used to evaluate the credibility of Web sites that offer health information. The institute defined the purpose and scope of the search to identify criteria of relevance; searched Web sites using various search engines including Yahoo!, Google, MSN and Webcrawler; and used the search terms "health website evaluation criteria," "website evaluation criteria," and "website evaluation." For each search, the first 100 results were evaluated. Each Web site was assigned an initial grade based on its description, and a final grade based on a review of Web site contents. Further details can be found here.

Professor Turow directed a research group at the University of Pennsylvania to review and analyze quantitative scholarly research on the quality of health Web sites, and to examine the general discussion of health Web sites over six months in 47 media outlets, from medical research journals to television network news operations. The range of media were: five medical research journals; 16 health industry trade periodicals; five major daily newspapers; 14 consumer magazines; two Internet online trade periodicals; and five television network news operations. A copy of the report can be found here.

Copies of the Health Improvement Institute's report are also available at http://www.hii.org.

A transcript of a public discussion of these reports, authors' presentations and comments, audience questions and feedback at Consumer Reports WebWatch's First National Summit on Web Credibility, "Building Trust on the Web," held April 24, 2003 in New York City, is available here.

Consumer Reports WebWatch is a project of Consumers Union, the non-profit publisher of Consumer Reports Magazine, and is supported by grants from The Pew Charitable Trusts, the John S. and James L. Knight Foundation, and the Open Society Institute.

The Health Improvement Institute is a non-profit, tax-exempt, 501(c)3 charitable organization dedicated to improving the quality and productivity of America's health care. The Institute is a voluntary organization. The Institute's primary goal is to provide information to patients, providers, payors, purchasers, policymakers, and the public about available alternatives to ensure that all Americans have the opportunity to make informed health care choices.

Joseph Turow, Ph.D, conceived, directed and edited the University of Pennsylvania Annenberg School of Communication's report. The individual authors are graduate students at the University of Pennsylvania: Kara Coluccio, Alyssa Hersh, Lee Humphreys, Lela Jacobsohn and Nadia Sawicki.