Note: This is an edited transcript of the proceedings.

Chuck Bell:
--Consumers Union in our Yonkers, New York office. I helped obtain the funding for Consumer Reports WebWatch several years ago, from the John S. and James L. Knight Foundation, The Pew Charitable Trusts, which really believed in this project. It was at one point sort of the brainchild of a very enterprising New York Times reporter named Denise Caruso, who pointed out that we don’t have on the Web some of the conventions we have in the print world that bring about the separation of editorial content and advertising. And they allowed us to take this project into major watchdog organizations for many different types of Web sites.

I do want to thank our funders, because a lot of what we do in this project is sort of a public good. If foundations or government or other angels don’t help finance it, it doesn’t necessarily happen. So we’ve been very privileged to have this project.

And we’ve been very privileged to have Beau Brendler, who I don’t know if this was made clear, but he used to be the editorial director of ABCNews.com. So he’s been very – he comes from the dot-com world, he’s a journalist, he really brings a lot of passion to this work.

So all of you in the audience today are very fortunate that we have a very excellent, expert, eclectic panel, drawn from throughout the country, to talk here today about how we go about improving health care Web sites and health search.

I’m just going to give a short introduction for who’s here, they’ll make some opening remarks, and then we’ll get right into it. Then we’ll open it up for questions and discussion.

And my background is, I’m a health advocate. I work for expansion of health care access. I’ve been a watchdog on nutritional supplements and have worked in managing grant-funded projects on healthcare information for consumers, such as issues of nursing homes and soon.

To my left, we have Dr. Tom Eng. He’s the founder of the eHealth Institute, which is a non-profit organization that works to explore exciting new applications of health technology and how that can be used to improve public health. He’s also the CEO of Helia, which is developing a next generation health search engine. And he works with another firm called Evalumetrics, which is a strategic consulting firm that helps companies, organizations trying to deal with health technology issues.

To his left, we have Nicole Spelhaug, from Rochester, Minnesota, who is chief of product development for the Mayo Foundation and MayoClinic- dot-com or dot-org?

Nicole Spelhaug
Dot-com.

Chuck Bell
Dot-com. Okay. And she’s been on the staff there since 1979. She’s a former managing editor of the Mayo Clinic Health Letter, among other functions, and has been working on the Web for quite some time.

To her left, we have Nan Forte, who is Executive Vice President for Consumer Services at WebMD. And Nan joins us, you’re based now in Atlanta, is that right, Nan?

Nan Forte
Atlanta and New York and California.

Chuck Bell
Okay, so I caught her shuttling back and forth to both places. We’re very pleased that you could join us here today. And she has a lot of experience in working in health information issues, was involved with launching iVillage’s health Web site before coming to WebMD, and has been involved in healthcare and patient education for more than 20 years.

To her left, we have Dr. Peter Goldschmidt, who you heard from this morning, who’s the founder and president of the Health Improvement Institute in Bethesda, Maryland. Peter’s partnered with Consumer Reports WebWatch on our health care ratings and works in many other areas involving health and public interest.

To his left, we have Dr. Stephen Barrett, who is the operator of 19 different Web sites, the best known of which is Quackwatch, which is dedicated to debunking dubious health information on the Web and elsewhere. Dr. Barrett is a retired psychiatrist, who lives in Allentown, Pennsylvania. He’s also Vice President of the National Council Against Health Fraud and has been involved for many years and served as a watchdog over the quality and integrity of health information in many types of media and now, especially, on the Internet.

To his left, we have Pat Thomas, who is the first holder of the Knight Chair in Health and Medical Journalism at the Grady College of Journalism and Mass Communication at the University of Georgia. This is a new position, and she’s just going there now, but she has a long background of writing about medicine and public health, having served from 1991 to 1997 as the editor of the Harvard Health Letter. And she’s also been a fellow for the Knight Science Fellow at MIT in 2000 and 2003.

And, finally, we have at the very end, last but not least, Pam Dixon, who’s Executive Director of the World Privacy Forum from San Diego. The World Privacy Forum is a non-partisan organization, focused on conducting in-depth research and consumer education on technology and privacy issues. She’s the author of seven books and many investigative reports and formerly was a researcher and investigator at Denver University School of Law, the Stern School of Law’s Privacy Foundation.

I think this is an excellent group. I really want to thank all of you for taking the time out of your busy schedules to be with us here today. And we are looking forward to lots of comments and interaction with the audience and among the panelists themselves.

So I was just going a couple of opening remarks about some of the questions that I posed to the panelists of things I thought we might want to talk about here today. But the floor is open.

Jørgen [Wouters] mentioned in his talk this important study that was done by the California Healthcare Foundation and RAND about health search, and I think it gives an excellent overview of some of the challenges that are involved, even though it was carried out in 2001. It points out that many Americans are using search engines to find information online. However, the search engine users only had a one in five chance of finding relevant information in the first page of results. Many sites were found to provide incomplete information, and conflicting information was not uncommon. And, as mentioned this morning, we have a lot of issues regarding literacy, people with disabilities, language barriers, that influence access to healthcare information.

So there were three pieces of general consumer advice which I think are worth mentioning, which is they recommended that consumers should allow ample time to search for answers. They should be aware that a single site may not answer all questions. And they should certainly discuss information that you find on the Internet with your healthcare provider, provided one can find time on the provider’s schedule to do that.

Some of the general challenges – a lot were mentioned this morning, I just wanted to mention three here that we have with healthcare information on the Internet. One is the phenomenon of unmediated search. It’s like being in a vast medical library, but there’s no librarian who’s ordering the books or is available to steer you to one stack of books or magazines or another.

Second, there’s a very low barrier to entry. Anybody can be a publisher, a health advisor, and/or a merchant, or maybe all three, if they want to do that.

And there’s an information glut. It’s like sucking from a fire hose, and you don’t even know where the source of the water is, or what the source of the water is.

So the question I posed for us to discuss are: What strategies, policies, systems or tools are needed to broadly improve the credibility and quality of healthcare information on the Web? What can we do now? A lot has been done in various forms; I’m sure everyone on this panel has been involved in a lot of different things. I think the question is, how do we go forward? And are there some specific initiatives that we could collaborate on or propose as coming from – we have many different stakeholder groups represented here today. We have consumers and publishers, content providers, people involved in the technology area and journalists. Also watchdog organizations. So I think we have kind of a unique chance to frame an agenda of things to do.

One thing, just speaking for Consumer Reports WebWatch, obviously, as we’ve said, we feel very strongly about the importance of basic disclosures on the Web. And it sounds a little boring and mundane – a lot of sites are still not doing it. And so we’re committed to going forward and beating the drum for disclosure.

The Department of Health & Human Services, they have a plan that’s called Healthy People 2010 that lays out many different types of goals for the nation for improving health status of Americans and people living in this country. And one of them is that health Web sites, they hope, will adopt basic disclosures across the board by the year 2010. Their disclosures that they call for are very similar to the ones we have advocated, and Hi-Ethics and many other organizations.

The one that they have that we have not been championing to date is they suggest that a treatment site – like a site that offers diets or other types of behavior modification – actually post whatever evidence it has or scientific information indicating whether that intervention is effective for consumers. So that’s – we think that could be a very useful thing, if sites were to do that.

So that’s disclosure. But beyond disclosure, what else remains to be done to improve the quality and integrity, accuracy and reliability of health information online? That’s one broad question.

Okay, the second question is: How can we improve the ability of consumers to find reliable information through search engines or other means, such as portals or bookmarks or brochures? And to that end, I would recommend people go to a report that we have on our Web site, which is called Setting the Public Agenda for Online Health Search. This is a report we did with URAC (Utilization Review Advisory Committee), which is an accreditation organization. And it was done through several stakeholder dialogues and a literature search, just looking at all the issues that are involved with searching for Web sites and getting good information.

And one of the points that was noted in that report is the quality of what you get from a health search depends on a lot of different factors, but principally the consumer behavior, what types of search terms people input into the engine. People may not be very skilled at using electronic search technologies. How they interpret the information they get. How long they stay with it before abandoning the search, issues like that.

There’s also the issue of search engine behavior, and what do search engines do in terms of returning results – do they use credibility or reliability as a factor in the search technology that they are using? And what comes back in terms of listings? Are they paid placements? So there are a lot of things that search engines do that affect where consumers get steered or where they end up.

And then, of course, there’s Web site behavior. What do Web sites do to provide information and ensure the quality of their content?

So obviously it’s a complicated process. We noted this sort of five-step process that takes place in any search and at each of these five points, the consumer decides they have a health need and goes online, enters a search term. There’s tremendous room for variation, and many research questions about how, what consumers do when they review the information, how they select sites.

So, out of that report, there are a number of recommendations that were made. We haven’t followed up on this so much so far, but we’re quite interested in what people think about this report. We hope people will read it and wrestle with some of the questions. But one of the recommendations is clearly we’ve got to do more to teach consumers about using a search engine and using the Internet to search for health information. So we said we want to promote health search literacy and build the capacity of consumers to use these new tools. And there’s a whole public education agenda that could go along with that.

Also, the report raises the question whether there could be consumer-directed tools, such as new types of search technologies, new types of bookmarks or filtered portals that give people access only to vetted or filtered information. And, finally, there were many research questions that were identified, and we see this as something where we’re going to need many years of work from researchers and academics and people, practitioners who are in the field.

But I just wanted to get that out there, as I think it’s a useful resource. I hope we can touch on some of those issues now.

So, to lead us off on the panel, I’d like to now call on Dr. Tom Eng from the eHealth Institute. He’s going to give us some overview of things that have been tried to improve the quality of information on the Web, and also talk to us from his experience at a company that’s working to improve health search technologies so consumers can find better information.

Dr. Tom Eng
Thanks a lot, Chuck. When Chuck mentioned that we have an expert and eclectic panel, I’m the eclectic one.

Sorry for that technical delay. Anyone have any questions so far? Let me just take some time while Chuck is pulling out that PowerPoint.

Basically, I’ve been in the – first of all, full disclosure. I’ve been looking at the whole issue of health information quality for a long time. I spent most of my life in public health and government employment, looking at the issues of health and technology. And now I am probably mostly in, basically, the technology business. So I put my two feet in two worlds, basically. So that’s sort of my biases and perspectives when I talk to you today.

I spend probably about 95% of my time on a new company called Helia, and it’s a health technology startup. We developed a new search engine for specifically health content. The eHealth Institute is a 501(c)(3) organization which I also founded. And the purpose of that organization is to see how we can use emerging technologies to improve public health.

[PowerPoint fixed.] Thanks, Chuck. And please feel free to interrupt me anytime, if you have any questions.

So what Chuck asked me to do is provide a general overview of what we’ve done in the past few years. I’m just going to mention a little bit then about what we’ve been doing as well on the company side.

So what we do know here is that consumers have a lot of problems identifying high quality and relevant health information. We know that about 70% of clinical questions that come up in a daily practice situation go unanswered during the course of our day and that’s primarily because docs don’t have the time to answer those questions. And also because they don’t have the necessary tools to find that information efficiently.

Second point. This is a study from JAMA that showed that 70% of the studies that have been published so far on health information quality online have shown that quality is a problem. We don’t need a full-blown study to show you this, right? It’s just common sense.

Third point here is that only about a quarter of health information seekers actually follow expert advice about checking content source and the currency of that content before they actually use that content. And that’s from the Pew Internet [& American Life] folks. [1]

And even people who are interested in looking and evaluating information online before they actually use it don’t do it right. So this is a study from B.J. [Fogg’s] group at Stanford University. [2] What he found here was, this is not exclusive to health information sites, but just generic Web sites. What he found is that the number one criteria that people use to evaluate quality information is the aesthetics of a site, the design, look and feel of the site. Obviously, that’s probably not the best correlative information [unintelligible] content quality.

The problem these days is not that there’s too little information, but there’s just way too much. And so the challenge to us is to try to help consumers find that needle in the haystack, that piece of information that is high quality, but also personally relevant for the personal situation. What we do know is that 86% of the people use multiple health Web sites. And about 81% of them use a search engine to find that information. They don’t all go to, unfortunately, WebMD or the Mayo Clinic, but they actually go to a search engine first to find that information.

And when you Google some of these common health terms, you know, you’ve done these yourselves, you’ll see millions and millions of hits. And even for a rare genetic syndrome, like Danny Walker Syndrome, you get more than 90,000 hits. And there’s no way in the world that anyone can actually go through these hits to find that document or that piece of content that’s totally relevant to your situation.

Here’s another issue. We know that – this is more like an hypothesis than something that’s documented by evidential studies, but I hypothesize that there are actually many types of health decisions that people make on a daily basis. It’s not one kind or type of decision. This is why it’s important. I think that a lot of people make decisions, for example, about health service providers. Like, for example, which doctor you go to, what specialist you go to, what hospitals are best for your surgical procedure or what clinic you go to.

Second, they also make decisions on a daily basis on interventions. That is, what kind of screening programs should I be taking in my age group? What kind of procedures are best for me if I have X condition, as well as treatments?

A third major kind of health decision that is made is decisions about personal health products. And this is done on a routine basis. For example, you walk in a drugstore, you think about what dandruff shampoo to buy. That’s a health decision. You think about what decongestant to buy for a son or a daughter who has congestion.

Fourth kind of health decision that’s made is lifestyle choices. Again, this is done sometimes unconsciously by people. That is, what do we eat? What did you actually choose on the menu for lunch in the buffet, for example, today? Whether we choose to exercise or smoke. Those are all decisions that are made on a very frequent basis.

And my hypothesis is that the user's needs, and your quality criteria that they use to assess whether or not they choose to use a piece of content not only varies from group to group – that is from, say, demographic group to demographic group – but also within those groups, it varies from person to person. Then within the person themselves, which I think is very important, it varies from query to query.

So, for example, if my question were about what’s the best kind of diaper to prevent diaper rash for my daughter, my needs and quality criteria I use to assess that information -- that piece of content I use is totally different than if my question were what kind of procedure is best for my diagnosis of prostate cancer. Very different questions.

So what I’m saying is that the needs of the individual and the quality criteria that could be used to assess information for that individual will vary from question to question, not just from person to person. And why that’s important is I think we need technology that will accommodate that need.

Let me provide just a very general overview and generic overview of what we’ve done in the past in looking at the histories of online health quality. Rating instruments. These are instruments or checklists for users to use to rate the quality of Web sites. Now, the advantage of this kind of approach is a very old approach, it’s been with us for decades, basically, well, since the Internet, I should say, of course, or online content. Of course, offline content’s been with us for a lot longer than that.

Well, the pros of this approach are the following. It’s familiar to use, people know how to use these kinds of checklists, by and large. And there are many of them available. The cons of this kind of approach are that it’s very hard to interpret the results for a consumer. For example, if you have a list of 10 questions to look for in a Web site, what if six out of 10 are checked off? What does that mean? How does the consumer then use those results to actually evaluate the information? [unintelligible] leave them without an approach that they could then follow through.

It’s also totally dependent upon the users. That is, the best quality health, online health instrument or checklist here is only good if the person uses it correctly and actually then takes that information and puts that in the context of their choice of the content.

Self-certified seals of approval. This is basically something I think most of you know about. This is the use of recognized seals or logos to indicate quality – the most famous one of these is the HON [Health on the Net Foundation] from the EU. This is a group based in Geneva, Switzerland. They were the first folks to come out with this concept about a seal of approval. And the corollary, of course, is like the Good Housekeeping Seal of Approval that we used to see in the 60s, 70s and 80s, I guess.

The pros of this approach is: It’s familiar to people. And another thing I like about this approach is that people can actually – what I call a glance-indicator. That is, people don’t actually think too much about the quality, they just look and see the appearance of a seal, if it’s there, then it triggers a green light for them, if you will.

The cons of this are the following: That it’s really just an honor system, especially a HON code. There’s no way to validate that a site that puts this logo on actually follows those guidelines, there’s totally no oversight of that.

Next approach is quality standards. And basically what I mean by this is the following: It’s voluntary compliance with a published set of quality standards, usually by independent third-party organizations. So examples of this are the AMA guidelines – there’s a British healthcare group, as well, several groups in the EU. The Hi-Ethics group, for example. The pros of this are the following: It’s easy to implement. Basically, you have organizations that actually sign up and say they’ll adhere to these quality standards and say so on their Web site.

The disadvantage of this kind of approach is that there’s a lack of incentive for actually adhering to these kinds of standards. And it’s also hard to validate, again. A lot of these are voluntary standards so, again, they’re very hard to validate.

And then the last one I put here is there’s no “standards standard.” You know, there’s an old saying that people love standards and the reason why is they can pick and choose which ones to abide by. So that’s the trouble with standards – that you actually have a menu of standards to follow and some of these are not consistent with [unintelligible].

Chuck had mentioned in the Healthy People 2010 guidelines that there’s a Healthy People objective about disclosure. I was one of the folks who actually helped put that guideline in Healthy People 2010. And about six years ago, we came out with a book called Wired for Health and Well-Being, [unintelligible] interactive health communication. Where as one of the recommendations of this government-sponsored group, we proposed that Web sites actually put a disclosure statement on the Web site. And not unusual with most government-run projects or government-sponsored, it didn’t really get a lot of dissemination, or adoption.

But we basically had a lot of the same things that Peter [Goldschmidt] had mentioned today were on that disclosure statement. It’s easy to implement. The cons, again, are lack of incentives for adherence, and it’s also difficult to validate whether or not what people say in the disclosure statement really is the truth.

This is also, this book is still available online, if you’re interested. It’s health.gov/scipich.

Another approach is accreditation. This is the approach where we have an independent body that comes in and certifies that the producer of content has appropriate processes and practices for ensuring quality content development. And the most famous one of these, of course, is URAC and, on the smaller scale, Trust-E is another example of this, where they actually audit privacy policies of Web sites.

It’s very easy to verify this, because somebody actually comes on board and actually looks at these documents and looks at the way these things are – the processes by which these things are developed.

The disadvantages: It’s something that’s not familiar to a lot of consumers. People -- like for example the person this morning who asked about what is URAC? What is that? They’d never heard about URAC, probably. And only health professionals or very [unintelligible] health professionals, for that matter, have ever heard of what URAC means or the significance of a URAC seal, anyway.

And it’s also not scalable. We think there are about 70,000, maybe 80,000 health Web sites in the world. That’s probably a gross underestimate. So that is the number of health Web sites are logged into open directory. Now, there are probably a lot of other sites, especially in foreign languages, that are actually not using the open directory.

It’s also a very expensive approach. I forgot exactly how much URAC charges, but it’s in the thousands of dollars. And so what you have is, once they get through all the clients they get in the next few years, what they’re going to have is a biased set of URAC-accredited sites, a bias towards well-funded companies, basically – a large group of companies. Is that fair? I don’t know. At least it’s not scalable. There are thousands and thousands of quality health domains out there that are not going to be able to afford to pay that kind of money for that kind of accreditation model.

Another approach that’s been taken is the [unintelligible] concept. Has anyone heard of the Medcertain approach? Have you guys heard about that? A few people have. So, basically, in this approach, what they do is they use meta-tags to describe and disclose site characteristics of the Web site. An example is this Medcircle/Medcertain – Medcertain actually went out of business I think a couple of years ago. So now the newest one is Medcircle. And I’m not sure – you know more about this than I do, Peter, about Medcircle or not?

Peter Goldschmidt
I don’t know if I know more than you. I think it’s still going.

Tom Eng
Yeah, it’s an EU initiative. And it’s basically focused on European Web sites. But what they do is to propose that the Web site developers tag their content with meta-data so that different kinds of technology tools can come in and filter out certain kinds of sites based on those attributes.

Now, the pros of this is that it can then be used by very different tools, like search engines or other kinds of technology. The problem is that it’s technically harder to implement than any of the things we talked about right now, because you actually need someone with IT experience to actually implement this.

It’s also a totally dependent solution on developers to comply with this thing. And then there are also no incentives to implement it. And I’m not sure exactly how many Web site developers actually comply with this. My guess is that it’s something like .0000-fraction of the total number of Web sites.

And then, lastly, the approach that my company’s taking, I’m putting my company hat on, is basically use a machine assessment of quality. So what we do is use algorithms to automatically assess the nature and the quality of the content. So the advantage of this approach is it’s independent – the approach and the solution is actually independent of what developers and users actually do. And it’s very scalable, because we’re relying on computers to do this, instead of people.

The problems with this are, one, it’s very new. By nature, it’s a new thing that probably is not proven in some ways. And also, the algorithms can never be perfect. Now, I would say this, as well, though, is that if you got, say, 10 health librarians or 10 world-recognized physicians in a room and you put 10 Web pages in front of them about a certain disease, they probably would not have a unanimous decision about the nature or the quality of that content, anyway. So there’s really no gold standard in this and that’s one of my points. It’s that there is no gold standard in quality. And so, because of that, we cannot expect software to give us a perfect result, either.

This company is called Helia. We’re based on Belleview, Washington, pretty close to Microsoft. It’s an ASP health search engine. We were funded for about four years, actually, to do this by the National Cancer Institute; it’s an SBRI award. And we use patent pending technology, we use algorithms and methods to assess quality and other page attributes.

The difference between what we’re doing and what other major search engines are doing are the following: It only provides high quality results. The second thing is that it enables someone to personalize a search in a way you cannot do anywhere. So, for example, if you’re a Hispanic woman looking for breast cancer information, below reading level, and you want pages that are more likely to be focused, say, for a senior citizen, we can provide a matched set for you.

It also allows customers to push contextual and target messages to the user and this is about – what we try to do is take advantage of the moment in time when a user is most likely to be receptive to health messages and health communication. So in health communication theory, the more tailored and more appropriate, just-in-time message you can push to a person, the more likely it is that they’ll take up on it and use it.

This is an independent, blind trial that we did of the search engine against search engine A, major search engine A, which I have blinded to protect the guilty, actually. And the second one is us and the third one is MedHunt, which is a search engine that is sponsored by the HON folks. And we did this [unintelligible] Health Sciences University, blinded to the results of the search engine. So we had three physicians there evaluate the quality of the search results, blinded to where the results came from. And so we came up statistically ahead in all three categories of information quality.

Speaker
Now, have you come across Healthdash yet?

Tom Eng
We know about them, yes.

Speaker
Is that the same thing or different?

Tom Eng
What we’re doing is very different.

Speaker
Very – it’s not algorithm-based, but they’re still filtering health searches by some sort of quality rating.

Tom Eng
I actually don’t know exactly what technology they use.

Speaker
Okay, I just didn’t know, because you said yours is the only one. So, because of the algorithm?

Tom Eng
Oh, no, we’re not saying we’re the only one to do this. There are lots, there are dozens of health search engines out there.

Speaker
There are?

Tom Eng
Yes. There are dozens of health search engines. We’re the only ASP health search engine. And, to our knowledge, we’re the only one that takes and uses algorithms to look at quality.

Speaker
Got it.

Tom Eng
Okay, now I’m putting my non-profit hat back on.

One of the things that I think we should think about in this conference, that I’d like everyone to think about at this point is that information is only the tip of the iceberg in consumer e-health applications. For example, I think a just as important, if not more important, thing we should look at is the quality of interactive tools. For example, lots of people use online peer support – you know, chat rooms and disease bulletin boards, etc. Health risk assessment software products. You know, how likely am I going to get cancer within X number of years? What’s my likelihood of dying from cardiovascular disease given my diet? Those kinds of things. Clinical decisions support. There’s lots of products out there that support shared decision-making between the patients and doctors. How good are those things in terms of providing someone what they need to know in terms of making the best clinical decision?

Online consultation messaging. There are lots of companies out there that dial into this space on looking at how do we message between patients and doctors, an environment that’s more cost-effective than actually doing the clinical visit. Personal health records, a big push, as you know, by the current administration to have inter-operable EMRs [Electronic Medical Records] and such. And it’s also a big push to do PHRs, [Personal Health Records] as well.

And then soon I would say, within maybe not five years, but maybe 10, we could have something called virtual health cultures. And I talk about this as well to other groups. Where we actually have agents or software products that actually help us make health decisions. And these are things that can actually sense the environment in which we’re making a health decision, know about our medical history and know and predict what’s likely to be the best decision for us and help us to make those best decisions.

What are the knowledge gaps? This is sort of a research agenda, if I could. One is, what should we be striving for? What’s good enough? For example, should we expect that, in Peter’s framework and the Consumer Reports framework [for HealthRatings.org] that all Web sites should have an excellent rating? Or is basically 50% -- what is the standard we’re reaching for? What’s good enough? What’s the cost-benefit ratio? And how do groups [unintelligible] vary in their perceptions of quality in search needs? What are sustainable models for quality assurance and quality improvement and who’s going to pay for it? And, then, what effective methods and metrics for measuring [unintelligible].

And then how can we improve the quality of e-health? Just two simple things, I think. One is to explore more scalable and less user-dependent approaches to quality assessment. And then the second one, I think, is just try to incentivize the best way we can different stakeholders. Meaning the content developers, the search engine providers and companies, the advertisers to do the right thing. And I think with those two things, that we’ll be able to move forward a little bit in this space.

Any questions, feel free to e-mail me. If you’re interested in seeing the actual product we’ve developed, just give me a call and we can do a demo for you. Thank you.

Chuck Bell
That was a really helpful overview. And you really can feel a lot of excitement about the potential of consumer health technology to improve the well-being of entire populations, too. I think that's a very exciting opportunity for us.

Okay, now I'm going to turn us over to Nicole Spelhaug, who is our chief of product development for the Mayo Foundation and MayoClinic.com.

Nicole Spelhaug
When I was first asked to consider these two questions, I thought: Well, shoot, how do I make this generally applicable? When you have a brand like MayoClinic.com, it's pretty easy to hide behind that, or stand behind that brand.

But what I hope I've covered are the kinds of things that any help Web site not only can do but should do, if you're in the information business of helping people manage their health.

I'm going to quickly move through these first slides, because we've talked a lot about what the issues are, about the pervasiveness of health Web site use. And just to comment about how important it is, how important the role of health information is to consumers, in terms of how they use it.

The empowering aspect is what we try to reflect on the site. That people are using this information to make, as Joel [Gurin] mentioned this morning, life and death decisions about their health.

They're using it to communicate better with their doctor. They're changing their lifestyle based on the information that they're using. So this is no small business. This is important stuff.

And when you look at the fact that the access to all of this information is counterbalanced by inaccurate and sometimes misleading information, it only raises the bar for those of us who provide this type of information.

So what's a health site to do? I challenge all providers of health information to take the higher ground that goes beyond the basic disclosure and transparency issues that we've talked about, that I'm not trying to minimize. But it goes beyond that to a higher level of credibility, that I essentially boil down to these three or four areas.

If you're going to be in the business of providing health information on the Web, make sure that your business is making people healthier. The content that you're using to do this should not be a commodity, but a tool to get you to that end.

Content is not a commodity. We did some market research a few years ago, where we took content, both branded and unbranded content, and consumers can tell a difference. We evaluated the quality of the content in that research based on: Did the content give people the answers they were looking for? Was it useful to them?

And if it did, we considered that high-quality content. We didn't get into the medical science aspects of it, but was it useful? And people could tell the difference when it did answer their questions, and when it didn't.

Second area is, if you're going to be in this business, be sure you're the health expert. And if you're not, then collaborate with the health experts. Because it's not the kind of information that you can just win.

The gold standard for all of this is really what MayoClinic.com prides itself on, is the editorial process. That the product that we have is the result of a collaborative team of multidisciplinary experts.

Sure, because we're Mayo Clinic, we've got access to physicians and researchers and health educators, and we can hire professional journalists. It's easy. It can be replicated in any scenario, whether they're onsite or they're contract or they're freelance, the point is that you need a variety of experts.

The clinical experience that was represented in the content is also, I think, important to its credibility. That not only is it subject matter experts, but also those who are practicing and are in the business now, and know what kinds of questions people ask, and what kinds of answers people need.

And finally, the redundant medical review. One of the weaknesses of our site that was pointed out this morning is that we don't list a single author for content.

We don't do that, because we don't believe any of our content is the result of a single author. Because there is this collaborative process from the beginning, and there is a redundant medical review. No one specialist knows all of the answers. In fact, we use specialists and generalists in all the review of our content, to benefit from that redundancy.

The business model is important, I think, for a health site. That dedication to excellence comes in addition to the need to make money. And to use the product or the content, the money that will result from that stems from the value of the content and not vice versa.

And finally, if you're going to provide credible content or strive to provide credible content, you have to know what drives content to the top of the way search engines currently operate, anyway.

That takes us to the second issue – counterbalancing those less credible sites. I have very simple points on this. I think the variety of seals of approval are counterproductive. I would like to see one accreditation.

We talked a little bit about standards this morning. Web designers are moving toward standardized design that is seamless across platforms and browsers. Why can't there be standardization in accreditation bodies, and through search engine practices also? I know it's a simple point for a very complicated issue, but I'd like to at least challenge us to talk more about that, if anything.

I would hope that accreditation becomes a household term. Tom mentioned, again, the fact that nobody even knows who URAC is. We should be able to do something about this one standard seal of approval that does come to mean something to people.

And I think the media efforts that the Consumer Reports WebWatch group has done with the pledge is a good example of the positive effects of this, but I think it has to be ongoing. And there might be an opportunity for those of us who are accredited, or have a vested interest in this accreditation issue, to support financially an ongoing media campaign.

And then the collaboration with the search engines, I think, is key, because as we mentioned this morning also, they hold the cards.

We try in everything that we do to put out credible content. But if you search on many, many of the topics that we publish on, you will not see our site come up to the top. One of the reasons is because we're not totally optimized right now. And I think that's a challenge anybody who provides health information needs to address. If you've got health information, you've got to optimize it.

But we also need the search engines' help to do that. And I think what I'd like to challenge them to do is to help us combine this quality listing with relevance. So that if you've got quality content, maybe it's the way you're accredited, and there's this symbol, and the site actually lists the results based on those accredited sites first.

The second, real easy thing, I think, is that search engines could require the source of the content to be listed in the line or the abstract.

Obviously we list our brand there, because that does us a lot of good, and it helps the consumer also, we believe. But as all of the examples that Joel [Gurin] showed us this morning, there are a lot of cases where you don't know where that's coming from till you click there. And even then you don't know. At a minimum, you should know that that information's provided by Pfizer before you get there. Or by Joe Schmoe, or whoever it is that's got the arthritis cure.

And then the results – I don't know how it's done. I just know that it would be nice if the results could be optimized for accredited sites.

And so I challenge anybody who's in the health information business to look at some of these things and say: Well, jeez, that's not very easy. That's a little idealistic.

Well, that's the Mayo Clinic charge from the beginning, to turn idealism into action. And I challenge us all today to try to do that with the credibility and relevance issue. Thank you.

Any questions?

Chuck Bell
Yes. We'll take maybe one quick question, but I want to make sure we get through our panelists.

Speaker
I want to know what an "expert" is. Are you talking about consultants or state-of-the-art researchers or your doctor, your chemist, your [unintelligible]? We have to get to what an expert is and how you qualify to be called an expert before we get much further. Everything seems to depend upon expertise.

Chuck Bell
Yeah, I think we can take that as comments, and maybe we can come back to that in the discussion. We really need to try to get through our panelists. I promised people 8-10 minutes each, and we got started a little bit late here, but thank you, Nicole. That was fabulous

I'd like to next go to Nan Forte, who is executive vice president for consumer services at WebMD.

Nan Forte
Okay, here's what I'm going to do. I'm not going to do my presentation now. Just because I just think it's not going to work. It's not the right environment. So I'm just going to throw out some random thoughts, and then let people ask me questions, and then get – is that okay?

Because if I give my presentation, no matter how good I think it is, or my kids think it is, it's going to lower the glucose level.

Chuck Bell
Or we can do that. We can put your full presentation on the Web, if that's okay.

Nan Forte
Well, I'll have to pass that through Legal and [TALKOVER], and I'll have to get some experts to comment, but I will, yeah. We'll make something available. And I do good PowerPoint. That is my other title at the company. I am the Chief PowerPoint Officer.

Just a few comments on, first of all, we're really happy to be here. Second of all, myself personally, I sold my company to WebMD. I actually started at the Center for Biomedical Communications at Columbia, dedicated in the early 80s during the AIDS crisis to the first time a patient showed up at medical meetings.

And I myself was a medical journalist, and so my career was launched there, in a hospital, with research. I was there when the patients chained themselves to the lab stuff down at Burroughs Wellcome with AZT. It all kind of started there. This was before the Internet.

But first it was the big concept of health journalism before we even got to how we were going to disseminate it. Just how are we going to write.

And I think one of the things that's interesting, not just about search, but about credibility in general, is the constant balance of credibility and, at the same time, because we're all dedicated to health improvement and we're all dedicated to advancing knowledge, so that people can make more empowered decisions.

We're very dedicated towards the democratization of medicine, which is a very complex topic. To your point of “What is an expert?” For us at WebMD, we have as much glee from our users.

I was going to show some examples of experts who come onto our site, Andrew Weil, others. And they throw out these things of what they do. Here's Dr. Andrew Weil saying that what he does to prevent the flu is carry alcohol wipes. Here on the news side of our site we have a whole thing saying these alcohol wipes don't do anything for you. And then he says he uses some Chinese herb to prevent immunity, agro something. And then we have to flash the sign that says “This is not FDA approved.” And yet he has a following.

He's just one of many. We have Dean Ornish on our site, who of course everything that he says and does makes perfect sense and works for people, but it's very, very difficult for – it attracts just a certain amount of people.

And so, Gerri Laybourne is a friend of mine – she’s at Oxygen, and she started Nickelodeon – and she used to always say, “It tastes good, and it's good for you.” Obviously there's the HBO model of television, which is no commercials. And we did contemplate that at WebMD. Just so you know, when WebMD first launched, we were actually a non-commercial site.

So for two years, we actually operated without advertising. So that's probably why we're even around today, because there's some inkling of the true essence of what we started to do, which was in a non-commercial nature at the beginning.

So all along we – myself and John and our staff of people – have been struggling with the balance. Obviously we've adopted choice and disclosure, choice and disclosure. We're so happy that we came in No. 1 [in WebWatch’s health evaluations, HealthRatings.org]. But we ourselves are so very humble, because we know we're only scratching the surface.

Choice and disclosure is just one of those things – I think Dr. Barrett mentioned it earlier – it's about the content. Let me just give you an example of what we go through at WebMD, because of our size. Last month along we had 26 million people in a given month that will come to the site. And just so that you understand that, one, we don't buy our traffic, and two, the smallest percentage of that comes from search engines. We are not search engine optimized.

I'll tell you why we're not. Because when we tell our doctors that they have to put the word “treatment” in the title so they’ll come up higher on Google, they say: “No, no, no, that changes the context of what we're saying, because if you say the word ‘treatment’ and it's not a treatment, and you...”

So we have all these fights within WebMD of the doctors saying, “We have to write what we write, and we have to say what we say,” and our producer saying, “But you won't go to the top of Google,” etc.

But actually, we know that our marketplace is probably twice the size, as we start looking at the meta-tagging and doing other things that would bring us higher in the search engines. And we're working very closely with Google.

Just so you know, we're very intrigued by a search engine for health. Right now our search engine only searches our site, which is not really a great public service. So we’ve been doing a lot of research and usability testing in health search.

And what's so interesting – back to this “tastes good and good for you” point – is that Google, you have to remember, Google set the standard. So every time we show them filtered searches: “Okay, how about this?” We’ll use these icons – .gov, .org – and here's a filtered search.” And we go into our usability lab, because we're very focused on this. We're going to be very interested in your company and others who are looking at this as a service.

You know, they don’t want the filtering. And we know we have to give them some sort of filtering, some sort of credibility rating, some sort of something. And then we say to them, “Well, gosh, when you're looking at Google, everything looks the same.” How are we going to guide you here, etc.?

And we're trying right now to really understand just from the consumer perspective how to best address a filtering rating concept.

The other thing that we struggle with at WebMD, again, because of our size, is the issue of subjectivity is very difficult. We own MedScape – MedScape is our partner. We publish The American College of Physicians textbook, and The American College of Surgeons textbook.

We own MedicineNet. They're doctors for doctors on the Internet. They're at a higher level of content. All of those doctors comment on a daily basis on the consumer Web site, because that's the flagship and they all own it.

On top of that, we work with CMS. We are part of Healthy People. We work with HHS.

You mentioned FDA. I live in Washington, and let me tell you, it is very hard to get that type of content in a way that we try to disseminate it. We really do, but it takes a long time to the issue of review and search and everything to get this out.

We're not crying the blues. We're just saying that we want to raise the level, and we need everyone in this room, and we need more affiliations and more partners doing this.

But how we view it at WebMD is, that's almost an academic exercise amongst us, until we find the icons and we find the things that the users do. That's okay. Things follow into sort of just getting organized and then getting good added.

It was interesting, I was down at the FDA about a month ago, and I was asked, “What did you think of that new ad that Johnson & Johnson did on TV?” I don't know if anybody saw this ad they did on TV; it was about birth control. They showed a woman – actually, it's the first ad of its kind – talking to her doctor about the risks and the benefits.

So it's the first time ever in the history of direct-to-consumer advertising that, instead of it running on the bottom like we saw in the Dancing Pill, because it gave us such a great idea to try to explain health in that type of format.

So it was a woman, and she was saying, "Hey, Doc, I really want this oral contraceptive.” And the doctor is saying “Are you a smoker? Are you this, are you that, are you sure?” So it began the dialogue.

We weren't a part of it, we have nothing to do with it, I know no one at – full disclosure, I know no one at J&J – but it was something that those of us who'd been in the industry forever took note of. Because we thought: Hmm, the dialogue is beginning. And, yet again, [unintelligible] starting it.

And these are all random thoughts. These are things – I'm not following my thing on all the sort of factors that go into even a paragraph on hypertension or even a conversation about an oral contraceptive. At the same time, obviously, we have to stringently label when this was originally created, when it was medically reviewed, which is critical, who wrote it, what the background of that person is.

At WebMD, we try to adhere to standard clinical guidelines. Of course, then we also, because of MedScape, work with the societies and associations. So ACC [American College of Cardiology] is telling us one thing, American Heart Association is telling us the other.

You have to understand, the La Leche League will come and strike outside of our offices. When I was very green in this field, because Enfamil was advertising on our site, and we spent a lot of time and money – we agreed with them that potentially, because we didn't have enough content, just the bulk of content on breast-feeding, that that was not fair balance. So, therefore, if we would accept an advertiser, therefore then we need to have fair balance. We agree with that.

A lot of times it's helpful for us with management, because if a big sponsor comes in and we don't have something in that area, then we need to step up to the plate.

It's why it's interesting now, with all these new health claims, and a lot of alternative medicine finally – not alternative medicine, but a lot of non-FDA approved type nutriceuticals on the market – advertising with advertising budgets, that's now forcing us to – that's our greatest way to go to management and say,” We have to have fair balance here. We have to do this.”

Another thing to throw out is that, interestingly enough, we had a grant from the Markle Foundation three years ago. And what's really interesting is, again, it took us a year to get this breast cancer module together. It was together with the television show with Ally Sheedy. And then we met the real woman afterwards, who unfortunately didn't make it with her struggle through breast cancer.

And it was so funny. We put every ad, every promotion we could behind that to get people there, and yet at the same time, we can get [thousands of] people in a certain amount of hours to take a breast cancer IQ quiz.

So it goes back to that balance of, the one thing that we look at at WebMD – and we hope that we also can engage in the dialogue and be seen for sort of the perspective of, we're out there with a little teeny-tiny quiz just trying to ask someone to engage. At the same time, we offer personalized health portals for employees to all the largest corporations in America. So we make personalized health portals that have health records, etc. on our same division, for PepsiCo, for Starbucks, for Microsoft, for Kraft, Delta, Raytheon.

And we're looking. We can watch all that behavior, right? We can see in an aggregated form what they're clicking on, what they do. And we do the same thing with MedScape.

And it's mind-blowing as we try to look at: Where do we go with content for our future? Obviously what they tell us and what they do are two different things across the board.

We know they want interactivity. We know all these other things. And, interestingly enough, the search is the one thing that – even though we're looking at it very closely because of the issues that are out there – they don't look to us for, because they view us as – they view search as navigation to a destination. And they're more concerned with that destination that finally gets them there.

One of the things I had in my PowerPoint was a visual that so powerfully shows that. When you pop up – it was sort of the thing that was done earlier – when you pop up what you actually find and put them all next to each other behind it, that's what people are really, really looking for. It's why they walk the mall as opposed to scan the sign, because it's a very different experience.

So we're looking at that. Anyway, I've probably gone on too long, but those were those random thoughts and comments that aren't part of PowerPoint, and I would just encourage anyone to ask questions or come back or make some comments based on those things.

Chuck Bell
I'd like to hold the questions to the end. I just want to make sure we get through all of our panelists, and then we'll open it up and give you some time.

And also, Peter, if it's okay with you, I'd like to, since we haven't heard from the people to your left, let's hear from Dr. Stephen Barrett, board chairman of Quackwatch, at this point. And we'll get through everyone's presentation, and we'll open it up for questions and discussion.

Thank you very much, Nan. You get the picture that there's a lot involved in this. There are many different publics and stakeholders that the producers of the content must relate to, while bearing in mind that they still want to get consumers to read it, and you're going to keep the end-user, the end-consumer in mind. Thank you.

Stephen Barrett, MD
My site is called Quackwatch. There's a reason for that, and that is that I define quackery and I attack it. Most people on the Internet are concerned about either getting out good information or selling you something.

I'm not interested in selling you anything. And I guess I'm willing to give out good information. But my information is primarily negative. That is to say, I want to warn people not to do things. It's not a very popular cause.

If you think about the other people who are doing – there aren't very many, and I wish there were more. One of the reasons I came here today is because I hoped that I would make some connections with people who have much more power than I do, who might be interested in helping to spread the kind of information that I put out.

It's really a shame that quackery is really, if you think about it, if you really look back and think of the publications and Web sites and so on that have talked much about how to avoid things that are based on pseudoscience, I don't think you'll come up with a long list.

There are a lot of reasons for this. To some extent, because it requires a lot of work. To another extent, because people are afraid of getting other people upset. Maybe they're afraid of being sued.

But the Q-word has almost disappeared from the American vocabulary in the last 15 years. You won't find articles on quackery on many sites. And it's now called “complementary and alternative medicine,” and it's a movement.

I'm not going to get into the details of what I think about it, except to mention that you won't find much on these topics. And what you find on most of the big sites is not reliable.

You wa