Experts vs. Online Consumers:

A Comparative Credibility Study of Health and Finance Web Sites (Abstract)

Leslie Marable
Consumer Reports WebWatch

Consumer Reports WebWatch
101 Truman Ave.
Yonkers, N.Y., USA 10703-1057

View Report   |   Download Report (PDF)   |   Download PDF Reader

Abstract

Many researchers have conducted studies on consumers to understand various aspects of Web site credibility. However, some consumers may not be well equipped to make informed decisions about the accuracy of information in technical fields such as health or finance. In fact, previous research in other mediums showed that consumers have had mixed results when judging credibility of information. Given the varying results of previous studies of consumer credibility assessments, we are left wondering whether consumers' credibility evaluations of Web sites are correct. These evaluations are increasingly important as people use the Web today to carry out a variety of vital tasks and research. Consumers are faced with important decisions about the information sources that they choose to believe for making important health or financial decisions. Do these everyday people know which Web sites are really credible, especially in vital areas such as finance and health? What do industry experts say about the credibility of sites in their fields? And, finally, how do the experts' assessments compare to how the average person decides which sites to trust?

To answer these credibility-related questions, Sliced Bread Design and Consumer Reports WebWatch produced this expert study, titled Experts vs. Online Consumers: A Comparative Credibility Study of Health and Finance Web Sites, in collaboration with Stanford University's Persuasive Technology Lab (Stanford PTL). As a usability and interaction design agency, Sliced Bread Design, LLC is interested in understanding how to create ethical online experiences that communicate credibility to their users. Consumer Reports WebWatch, which commissioned this study, has the goal to investigate, inform, and improve the credibility of information published on the World Wide Web. Due to these shared goals, the two organizations decided to embark on a study of how health and finance experts judge the credibility of Web sites within their domains of expertise.

This expert study was launched jointly with a parallel, consumer-focused study, How Do People Evaluate a Web Site's Credibility? Results from a Large Study, conducted by the Stanford PTL and Consumer Reports WebWatch, which asked over 2,600 average people to rate the credibility of Web sites in 10 content areas. Our study then compared how experts and consumers evaluated the same health and finance sites to understand if and how consumers failed in their assessments. By comparing the expert and consumer evaluations, we hoped to identify any gaps in consumer education and begin to design guidelines for improving consumer understanding of online credibility. Furthermore, by studying experts in two diverse fields — health and finance — we aimed to learn about field-specific credibility in order to inform Web design guidelines and consumer education needs.

About These Joint Studies

Sliced Bread Design: In this study, 15 experts from the health and finance fields were asked to assess the credibility of sites in their respective areas. A total of 8 health and 7 finance experts visited the same 10 health sites or 10 finance sites as the consumers in the Stanford PTL study. They were asked to rank from 1-to-10 the credibility of the sites specific to their area of expertise, as well as provide detailed written assessments of each site under review. (See Appendix B for a list of the expert participants and brief bios.)

Stanford Persuasive Technology Lab: In the parallel consumer study, 2,684 people evaluated the credibility of two Web sites randomly assigned from one of 10content categories (e-commerce, entertainment, finance, health, news, nonprofit, opinion or review, search engines, sports, and travel.) A total of 100 sites were assessed. The 2,440 comments that people wrote about each site's credibility were gathered and analyzed to track which features of a Web site were noticed — or went unnoticed — when consumers evaluated credibility online. (See Appendix A in the Stanford PTL study report for a complete list of Web sites selected for the consumer study.)

Key Findings

The online consumers in the Stanford PTL study and our expert panel of evaluators diverged greatly in their credibility assessment criteria. Overall, our experts were far less concerned about visual appeal as a marker of credibility than the Stanford PTL consumers, and they were more concerned about the quality of a site's information. Among other conclusions, this study found that health experts assigned more credibility to health sites that provided information from reputable sources and cited the names and credentials of authors for each article published. Finance experts assigned more credibility to finance sites that provided investors with a great deal of unbiased educational information and research, rather than nudging consumers toward their own products or services.

The Stanford PTL study found that consumers tended to rely heavily on overall visual design when assessing Web sites, including layout, typography and color schemes. About half (54.6%) of the comments by the consumers regarding finance sites referred to design look, which relates to the visual appeal of a site's design, compared to only 16.4 percent of finance expert comments on this topic. Likewise, 41.8 percent of consumer comments regarding health sites made note of a site's design look, compared to 7.6 percent of surveyed health experts' comments which mentioned this topic.

Our health experts most often relied on the name reputation of a site, its operator, or that of its affiliates, when assessing the credibility of health Web sites (43.9% of health expert comments related to this credibility criterion). The next most common issues mentioned when evaluating health site credibility were information source, which relates to the citation of a site's information sources (25.8%), and company motive, which relates to a user's perception of the motive of the organization behind the site, whether good or bad (22.7%).

Our finance experts most often relied on a site's scope or information focus when assessing the credibility of finance Web sites, which includes consideration of the quantity of information provided (40.3% of finance expert comments related to this credibility criterion). The next most common issues mentioned when evaluating finance site credibility were company motive (35.8%), and information bias (29.9%), which relates to a user's perception of bias in the site's content.

This study also reveals which specific elements lend credibility to a site's perception, according to each health and finance expert group. In addition, the report provides recommendations to Web publishers, particularly those in the health and finance fields, which aim to increase the credibility of sites among each type of Internet audience.

* * *

About Consumer Reports WebWatch
Consumer Reports WebWatch is a project of Consumers Union, the non-profit publisher of Consumer Reports magazine and ConsumerReports.org. The project is supported by The Pew Charitable Trusts, which invests in ideas that fuel timely action and results; the John S. and James L. Knight Foundation, which promotes excellence in journalism worldwide and invests in the vitality of 26 U.S. communities; and the Open Society Institute, which encourages debate in areas in which one view of an issue dominates all others. Consumer Reports WebWatch's Web site launched April 16, 2002.

About Sliced Bread Design, LLC
Sliced Bread Design is a usability and interaction design agency established by Julie Stanford and Ellen Tauber to help people effectively use and enjoy interactive products. Sliced Bread Design provides user study and interface design services to help organizations create compelling online, desktop, voice, and wireless software. Its work appears in a variety of products, ranging from online enterprise resource planning software for Fortune 500 companies to consumer software for mobile phones. The company is located in California's Silicon Valley.

About Stanford Persuasive Technology Lab
The Stanford Persuasive Technology Lab creates insight into how computers can be designed to change what people think and do, an area called "captology." Directed by experimental psychologist B.J. Fogg, the Stanford University team includes social scientists, designers, and technologists who research and design interactive products (from Web sites to mobile phones) that motivate and influence their users. Although this area can be controversial, the lab's focus is on using the power of persuasive technology to promote positive changes in domains such as health, education, business, and safety. As part of the lab's mission, since 1998 the team has investigated what causes people to believe — or not believe — what they find online.