A Report on the Evaluation of Criteria Sets for Assessing Health Web Sites
(Full Report)
Author: A Joint Project of The Health Improvement Institute
and Consumer Reports WebWatch
TABLE OF CONTENTS
Introduction
Methods
Results
Discussion
References/footnotes
Table 1: Websites and articles evaluated
Table 2: Density of website assessement criteria
Table 3: Top 10 specific health website assessment criteria
Table 4: Number of criteria sets containing a meta-criterion
Table 5: Website criteria sets meeting main evaluation criteria
Table 6: Health website criteria sets meeting evaluation criteria
Appendix 1: Meta-Criteria
Appendix 2: Health website quality assessment criteria set evaluation form
Introduction
In 2003 Health Improvement Institute (Institute) began a research project conducted collaboratively with Consumer Reports WebWatch. Its purpose is to enable consumers to become informed users of Internet health information sites (referred to as health websites). The project consists of a number of phases, to culminate in independent ratings of health websites. The objective of Phase-I is to develop a consumer-friendly rating system, based on basic principles and past efforts to develop evaluation criteria. Phase-I includes:
- Search for previous evaluations of health websites
- Identify existing criteria sets to
- Describe criteria used or suggested for use in evaluating health websites
- Evaluate the consumer-friendliness of identified criteria sets.
This report describes the results of our search for and analysis and evaluation of criteria sets intended to assess the "reliability" or "credibility" of health websites.
back to top 
Methods
We describe methods as follows:
- Search to identify criteria sets
- Analysis of criteria in sets within the project's scope
- Evaluation of criteria sets' utility
Search to identify criteria sets:
In January and February 2003, we conducted World Wide Web (WWW) and literature searches to identify criteria sets that are used to evaluate the credibility of websites that offer health information. We:
- Defined the purpose and scope of the search to identify criteria of relevance
- Searched websites
- Used various search engines (Google, MSN, Webcrawler, and, Yahoo) and search terms (health website evaluation criteria, website evaluation criteria, website evaluation)
- For each website search, evaluated the first 100 results. Each website was assigned an initial grade based on its description; a final grade, based on a review of website contents.
- Followed links on websites judged to be relevant
- Searched the medical literature
- Searched Medline (the on-line reference search of the National Library of Medicine) and other sites using the same or similar terms used for website searches
- Evaluated the relevance of retrieved articles, first based on the title of the article; then, for articles retrieved as relevant, the content of the article
- Checked and retrieved the references of relevant articles
- Found through searches
- Found serendipitously
- On hand at the start of the project
Through these searches we identified:
- Articles that were reports of health website evaluation criteria or relevant evaluations of health websites
- 151 criteria sets (125 websites and 26 articles) among the approximately 411 websites URLs and 60 citations identified.
We included a criteria set in our final data set if it was used specifically for evaluating health information on websites or if the criteria set contained elements that may be used to evaluate health information on websites. We judged 22 (20 websites and 2 articles) of the 151 criteria sets to be directly relevant to the comprehensive evaluation of health websites; 7 were a derivative of another criteria set [1]. See Table 1.
Analysis of criteria
In order to describe criteria in sets within the projects' scope, we developed a set of meta-criteria. To construct this set, we:
- Developed a conceptual framework
- Reviewed the criteria in the 22 sets to be analyzed
- Developed a logical structure to array them
- Arrayed criteria in sets according to this structure
- Elaborated the structure within this context to include criteria that might reasonably be contained within criteria sets to be analyzed.
To describe criteria used or suggested for use in evaluating health websites, we:
- Parsed the criteria expressed in the 22 criteria sets. This step was necessary because authors often provided narratives or made conjoint statements ( for example, "this….and….that")
- Coded these criteria elements according to our meta-criteria, which encompassed every criterion in all 22 criteria sets.
- Reprinted criteria sets from our database to check
- The accuracy of captured statements by comparing database records to the original criteria sets.
- We had correctly classified headings, definitions, and elaboration's of criteria that appeared in criteria sets to distinguish them from criteria.
- Reported criteria according to meta-criteria set codes in order to check the accuracy, specificity, and consistency of coding (updated the meta-criteria set and recoded records accordingly).
- Counted the number of times a specific meta-criterion appeared in criteria sets.
The meta-criteria had 9 domains, and a total of 115 criteria elements (appendix 1 provides a detailed description of the meta-criteria):
- Content relevance
- Content accessibility
- Content selection
- Content validity
- Content interchange
- Site transparency
- Links
- Quality assurance
- Safeguards.
Evaluation of criteria sets' utility
To evaluate the utility of identified criteria sets to consumers and their consumer-friendliness, we:
• Developed a basic set of evaluation criteria, consisting of 22 criteria elements organized into 6 domains. See Appendix 2 for the set of evaluation criteria.
• Asked 2 project staff members to rate independently the identified criteria sets using these evaluation criteria: each criterion was scored "met," "not met" or "partially met."
• Compared ratings.
• Resolved any differences in ratings by asking a third project staff to determine the final rating.
• Checked the logical consistency of ratings, and resolved any inconsistencies by examining individual ratings.
• Tabulated evaluation results.
back to top
 |
Results
Findings of analysis of criteria
Our search yielded 22 criteria sets within the project's scope. These 22 criteria sets contained 466 criteria elements for assessing health websites. Table 2 shows the "density" of criteria, i.e., the number of criteria elements in each domain as a percentage of the total of all criteria elements contained within all 22 criteria sets. Almost two-thirds of criteria addressed content validity (27%), content accessibility (19%), and site transparency or credibility (17%). Table 3 lists the top 10 most frequent specific criteria (which appeared in almost half or more of the 22 criteria sets). Table 4 shows the number of criteria sets containing each of the 115 meta-criteria elements.
Findings of evaluations of criteria sets utility
None of the 22 criteria sets examined met all of our evaluation criteria; 4 sets met 5 of 6 evaluation criteria. See Table 5. Three-quarters of the criteria sets defined their purpose clearly, but only 18% in operational terms; none reported reliability testing. See Table 6. Only one-quarter of the criteria sets evaluated stated they were intended for use by consumers. Of these sets, we rated 40% (2 sets: [2,3]) as being consumer-friendly. One of these sets [3] met all of our top-level evaluation criteria except reported reliability testing.
back to top
|
Discussion
We identified one study that had evaluated the quality of health websites related to asthma. The authors used the HON code as the basis for their evaluation [4]. They found that:
- 16% of the health websites examined met HON code criteria (principles)
- Of the websites that displayed the HON code logo (signifying that they subscribed to the principles), 38% actually complied with HON code principles [5].
We identified 22 criteria sets for assessing the "reliability" or "credibility" of health websites. They contained from 8 to 37 criteria. Most criteria pertained to content validity or accessibility or website transparency or credibility, including the currency of information. None of the 22 criteria sets met all 6 of our evaluation criteria. The most telling failures were lack of testing, lack of consumer-friendliness, and failure to define criteria in operational terms (virtually precluding their use to produce reliable ratings of health websites). Only one of the criteria sets was developed for use, and none was usable, by consumers.
There are no criteria sets (assessment tools) that are readily usable by consumers to assess health websites. Moreover, there are none that are readily usable by professionals to assess reliably health websites. In the next phase of the project, the Institute intends to develop, test, and apply such a tool to produce independent ratings of health websites. The Institute welcomes participation in this process. In the meantime, consumers may want to judge the "quality" of health websites based on a few consumer-friendly criteria. The website should be easy-to-use, and it should disclose at least the following information:
- Sponsorship -- who owns/operates (pays) for it (to rule-out obvious potential conflicts of interest)
- Purpose/scope -- the website's purpose and the scope of information presented (to judge the extent to which it matches the consumer's interests)
- Audience -- for whom the information is intended, and its intended use (in this instance, it should be for consumers and the specified use of information should match the site visitor's reason for searching for information)
- Currency -- when the website and/or webpage was last updated (and that should be recently)
- Source/credentials -- for "facts" (articles), the source of the information, a description of authors' credentials, and, preferably, a link to the source.
back to top
References/Footnotes
- Silberg W, Lundberg G, Musacchio R. Assessing, Controlling, and Assuring the Quality of Medical Information on the Internet: Caveat Lector et Viewor -- Let the Reader and Viewer Beware. JAMA 1997; 277:1244-1245.
- Criteria for Assessing the Quality of Health Information on the Internet -- Policy paper. hitiweb.mitretek.org/docs/policy.html.
- NetScoring: Criteria to assess the quality of Health Internet information.
www.chu-rouen.fr/netscoring/netscoringeng.html.
- HON Code of Conduct (HON code) for medical and health websites. www.hon.ch/HONcode/Conduct.html.
- Croft D, Peterson M. An evaluation of the Quality and Contents of Asthma Education on the World Wide Web. Chest 2002; 121:1301-1307.
- Goldschmidt PG, Liao J (eds). The quality of health information on the Internet: Report of a workshop. Bethesda, MD: Health Improvement Institute, 1997.
back to top
Table 1: Websites and articles evaluated
| URL (20) |
Organization |
www.webfeetguides.com/
criteria_WFforHealth.cfm |
Web Feet |
www.asc.upenn.edu/courses/comm240/
medsitecriteria.cfm |
University of Pennsylvania |
www.meddean.luc.edu/library/
WEBEVAL.htm |
Loyola University - Health Sciences Library |
| www.hon.ch/HONcode/Conduct.cfm |
Health On the Net Foundation |
| www.nnlm.gov/gmr/publish/eval.cfm |
National Network of Libraries of Medicine |
www.ihealthcaregroup.com/
rheumatologyguide/criteria.htm |
Internet Health Care Group |
www.sph.emory.edu/WELLNESS/
instrument.cfm |
Rollins School of Public Health |
| hitiweb.mitretek.org/docs/policy.cfm |
Mitretek Systems |
www.nursingworld.org/ojin/
infocol/info_1.htm |
Kent State University |
atlas.ici.ro/ehto/medinf99/papers
criteria_for_evaluating_the_qual.htm |
Medical Informatics Team Projects |
europa.eu.int/information_society/eeurope/
ehealth/quality/draft_guidelines/index_en.htm |
Information Society- European Commission |
www.chu-rouen.fr/netscoring/
netscoringeng.cfm |
Centrale Paris |
www.umanitoba.ca/libraries/units/health/
help/ihp/evaluate.shtml |
University of Manitoba- Health Sciences Libraries |
www-personal.umich.edu/%7Epfa/pro/
courses/WebEvalNew.pdf |
University of Michigan |
www.uic.edu/depts/lib/lhsu/resources/
guides/web-evaluation.shtml |
LHSU-Library of the Health Sciences-Urbana |
www.library.dal.ca/kellogg/internet/
evaluate.htm |
Dal Libraries Kellogg library |
www.nyu.edu/education/hepr/resources/
online/adq.pdf |
Department of Health and Nutrition Sciences |
| www2.auckland.ac.nz/lbr/instruct/evaluate.htm |
University of Auckland Library |
| www.ourbodiesourselves.org/eval.htm |
Boston Women's Health Collective |
www.canadian-health-network.ca/
html/help/checklist1.cfm |
CHN (Canadian Health Network) |
| Articles (2) |
| Silberg W, Lundberg G, Musacchio R. Assessing, Controlling, and Assuring the Quality of Medical Information on the Internet: Caveat Lector et Viewor -- Let the Reader and Viewer Beware. JAMA 1997; 277:1244-1245. |
| Nicoll, Leslie H. Quick and Effective Website Evaluation. Lippincott's Case Management 2001; 6(5):220-221 |
back to top
Table 2: Density of website assessment criteria
| Code |
Meta-Criteria Domain |
Number |
Percent |
| 1.0 |
Content relevance |
56 |
12.0% |
| 2.0 |
Content accessibility |
90 |
19.3% |
| 3.0 |
Content selection |
34 |
7.3% |
| 4.0 |
Content validity |
128 |
27.5% |
| 5.0 |
Content interchange |
10 |
2.2% |
| 6.0 |
Site transparency/credibility |
78 |
16.7% |
| 7.0 |
Links |
35 |
7.5% |
| 8.0 |
Quality assurance |
9 |
1.9% |
| 9.0 |
Safeguards |
26 |
5.6% |
| |
Total |
466 |
100.0% |
Note: The percentages shown above are the number of criteria elements in all 22 criteria sets in the specified domain divided by the total number of criteria elements (i.e., 466), expressed as a percentage, that is, "criteria density." Our meta-criteria set lists 9 domains and 115 criteria elements.
back to top
Table 3: Top-10 specific health website assessment criteria
| Code |
Criteria Description |
Number |
Percent |
|
6.1 |
Does the site say who owns or pays for or created it? |
17 |
77% |
|
3.2 |
Conflict of interest, bias, disclosure of sponsorship, funding sources |
14 |
64% |
|
2.2 |
Navigability, logical organization |
13 |
59% |
|
6.3 |
Does the site indicate when it was last updated? |
13 |
59% |
|
1.1 |
Purpose/Mission/Scope: Does site state its purpose? |
12 |
55% |
|
1.2 |
Audience: Does the site state its intended audience? |
12 |
55% |
|
2.2 |
Author's credentials |
12 |
55% |
|
4.4 |
For reports of facts does site give source of specific information? |
12 |
55% |
|
3.1 |
Editorial policy, site evaluation process, peer review process |
10 |
46% |
|
7.1 |
Does the site link to other sites as sources of health information? |
10 |
46% |
|
9.3 |
Disclaimers, privacy policy, sponsorship, advertising policy |
10 |
46% |
Total number of criteria sets = 22
back to top
Table 4: Number of criteria sets containing a meta-criterion
|
Code |
Criteria Description |
Number |
Percent |
|
1 |
Content relevance |
1 |
5 |
|
1.1 |
Purpose/Mission/Scope: Does site state its purpose? |
12 |
55 |
|
1.1.1 |
Does the mission statement convey purpose & scope of website? |
3 |
14 |
|
1.1.2 |
Scope: Does the site state its scope? |
1 |
5 |
|
1.1.3 |
Objective/Goals: Does the site state its objective? |
4 |
18 |
|
1.1.4 |
Does the site indicate the title/subject of site? |
1 |
5 |
|
1.2 |
Audience: Does the site state its intended audience? |
12 |
55 |
|
1.3 |
Relevance: Can one tell easily if what's on site is relevant to search purpose |
7 |
32 |
|
1.4 |
Breadth & depth, Coverage, Uniqueness |
6 |
27 |
|
1.4.1 |
Is the information unique or can it be found elsewhere? |
7 |
32 |
|
1.5 |
Site consists only of links (I.e. no substantive information)? |
2 |
9 |
|
1.9 |
Content, NEC |
0 |
0 |
|
2 |
Content accessibility |
7 |
32 |
|
2.1 |
Comprehensibility: Is site prepared at a level useful to intended audience? |
9 |
41 |
|
2.1.1 |
Is it comprehensible by average consumer? |
0 |
0 |
|
2.2 |
Navigability, logical organization |
13 |
59 |
|
2.2.1 |
Is the information needed to use the eval checklist easily accessible on site? |
0 |
0 |
|
2.2.2 |
Is site content easily accessible? |
8 |
36 |
|
2.2.3 |
Is the site easily navigable? |
6 |
27 |
|
2.2.4 |
Does it contain a search engine? |
5 |
23 |
|
2.2.5 |
Is website designed in a user-friendly manner? |
4 |
18 |
|
2.2.6 |
Is there a site index, table of contents? |
3 |
14 |
|
2.3 |
Does site organize information in a consumer-friendly way? |
6 |
27 |
|
2.4 |
Readability: Is text, graphics, video etc consumer-friendly? |
7 |
32 |
|
2.4.1 |
Does site contain (useful) images? |
1 |
5 |
|
2.4.2 |
Is site available in text only mode? |
3 |
14 |
|
2.4.3 |
If user can't access images, etc, is information still "complete" (useful)? |
1 |
5 |
|
2.5 |
Downloadability |
7 |
32 |
|
2.6 |
Are there conditions to accessing content? |
3 |
14 |
|
2.6.1 |
Is there a fee to access content? |
2 |
9 |
|
2.6.2 |
Does the user have to register? |
0 |
0 |
|
2.6.3 |
Does accessing content require acceptance of cookies? |
0 |
0 |
|
2.7 |
Presentation of website: Is it appealing? |
5 |
23 |
|
2.9 |
Content accessibility, NEC |
0 |
0 |
|
3 |
Content selection/editorial policy |
2 |
9 |
|
3.1 |
Editorial policy, site evaluation process, peer review process |
10 |
46 |
|
3.1.1 |
Was search strategy adequate to the site's purpose? |
0 |
0 |
|
3.1.2 |
Was the selection method adequate to the site's purpose? |
0 |
0 |
|
3.2 |
Conflict of interest, bias, disclosure of sponsorship, funding sources |
14 |
64 |
|
3.2.1 |
Does the site provide a conflict of interest policy? |
2 |
9 |
|
3.2.2 |
Does site report any relationship between authors of content & owner, etc. |
4 |
18 |
|
3.9 |
Content selection, NEC |
2 |
9 |
|
4 |
Content validity |
12 |
55 |
|
4.1 |
Does site differentiate between essays/commentaries/
opinions/facts? |
4 |
18 |
|
4.1.1 |
Is the information/content thorough and detailed? |
1 |
5 |
|
4.1.2 |
Is the information accurate? |
7 |
32 |
|
4.2 |
Does site give author's information for opinions/reports of facts? |
9 |
41 |
|
4.2.1 |
Author's name |
6 |
27 |
|
4.2.2 |
Author's credentials |
12 |
55 |
|
4.3 |
Does site indicate the date the opinion was written/last updated? |
1 |
5 |
|
4.4 |
For reports of facts does site give source of specific information? |
12 |
55 |
|
4.4.1 |
Link to the source? |
1 |
5 |
|
4.4.2 |
Authors of the source |
9 |
41 |
|
4.4.3 |
Date source information was generated or date the report was published |
8 |
36 |
|
4.4.4 |
Date information was posted on the site |
4 |
18 |
|
4.4.5 |
Bibliography/resource list |
3 |
14 |
|
4.4.6 |
Type of source |
2 |
9 |
|
4.5 |
Does site indicate whether or not it has evaluated specific information? |
7 |
32 |
|
4.5.1 |
If so, does the site describe how the specific information was evaluated? |
2 |
9 |
|
4.5.2 |
Are the methods adequate to their purpose? |
0 |
0 |
|
4.6 |
Does the site state the evidence grade of specific information |
9 |
41 |
|
4.6.1 |
If so, does the site describe how the evidence grade was assessed? |
0 |
0 |
|
4.6.2 |
Are the methods adequate to their purpose? |
0 |
0 |
| 4.7 |
Does site indicate limitations on applicability of specific information? |
8 |
36 |
| 4.8 |
Does site state whether or not it carries advertising? |
3 |
14 |
| 4.8.1 |
Does site distinguish between advertising and content? |
3 |
14 |
| 4.8.2 |
Does it provide its advertising policy? |
3 |
14 |
| 4.8.3 |
Is its advertising policy appropriate for consumers? |
1 |
5 |
| 4.9 |
Content validity, NEC |
1 |
5 |
| 5 |
Content interchange |
2 |
9 |
| 5.1 |
Is there a way for consumers to request additional information? |
6 |
27 |
| 5.2 |
Does site offer decision support technology? |
1 |
5 |
| 5.2.1 |
If so, does site describe the expert system used for this purpose? |
0 |
0 |
| 5.2.2 |
Are specific caveats given regarding the system's use? |
0 |
0 |
| 5.2.3 |
Does site describe how the DST was developed and validated? |
0 |
0 |
| 5.2.4 |
When was the DST last updated and/or validated? |
0 |
0 |
| 5.3 |
If site offers exchange of information between consumers and health professionals, does the site describe their credentials? |
1 |
5 |
| 5.3.1 |
Is it obvious to the consumers when they are interacting with a professional? |
0 |
0 |
| 5.9 |
Content interchange, NEC |
0 |
0 |
| 6 |
Site transparency/credibility |
6 |
27 |
| 6.1 |
Does the site say who owns or pays for or created it? |
17 |
77 |
| 6.1.1 |
Does it list a physical location, tel. number, contact person etc? |
1 |
5 |
| 6.2 |
Does the site say who runs it? |
2 |
9 |
| 6.2.1 |
Does it list a physical location, tel. number, contact person etc? |
0 |
0 |
| 6.3 |
Does the site provide email address for the webmaster? |
5 |
23 |
| 6.4 |
Does the site describe the sponsor's purpose or business? |
1 |
5 |
| 6.5 |
Does the site disclose sources of financing of the sponsoring organization? |
7 |
32 |
| 6.6 |
Currency |
10 |
46 |
| 6.6.1 |
When was the site first created? |
3 |
14 |
| 6.6.2 |
How frequently or often is the site/site content updated? |
5 |
23 |
| 6.6.3 |
Does the site indicate when it was last updated? |
13 |
59 |
| 6.6.4 |
Does the site indicate date until which the content/information is valid? |
3 |
14 |
| 6.7 |
Is the site essential "complete"? |
1 |
5 |
| 6.8 |
If site is moderated, is moderator's identity disclosed? |
0 |
0 |
| 6.8.1 |
Are credentials, affiliations, etc of the moderator disclosed? |
1 |
5 |
| 6.8.2 |
Is method used to select moderator disclosed? |
1 |
5 |
| 6.9 |
Site transparency, NEC |
2 |
9 |
| 7 |
Links |
3 |
14 |
| 7.1 |
Does the site link to other sites as sources of health information? |
10 |
46 |
| 7.2 |
Does the site indicate whether or not it has evaluated adequacy of sites? |
6 |
27 |
| 7.2.1 |
If so, does the site describe the basis for such evaluation? |
1 |
5 |
| 7.3 |
Does the site provide appropriate descriptions, disclosures etc about links? |
2 |
9 |
| 7.4 |
Are consumers alerted when they leave the site? |
3 |
14 |
| 7.5 |
Does the site report its policy for periodically checking integrity of links? |
8 |
36 |
| 7.6 |
Does the website contain email links to the authors of the content? |
2 |
9 |
| 7.9 |
Links, NEC |
0 |
0 |
| 8 |
Quality assurance |
0 |
0 |
| 8.1 |
Does the site describe any quality management or QA/QI process? |
0 |
0 |
| 8.1.1 |
Does QA/QI include accessibility? |
0 |
0 |
| 8.1.2 |
Relevance? |
0 |
0 |
| 8.1.3 |
Content selection, validity, and/or interchange? |
0 |
0 |
| 8.1.4 |
Transparency/credibility? |
2 |
9 |
| 8.1.5 |
Links? |
0 |
0 |
| 7.1.6 |
Safeguards? |
0 |
0 |
| 8.2 |
If so, are QA/QI processes adequate to their purpose? |
0 |
0 |
| 8.3 |
Does the site permit easy feedback from consumers? |
7 |
32 |
| 8.9 |
QA/QI, NEC |
0 |
0 |
| 9 |
Safeguards |
0 |
0 |
| 9.1 |
Does site display terms & conditions of use of the website? |
2 |
9 |
| 9.2 |
Does site display its privacy or confidentiality policy? |
5 |
36 |
| 9.2.1 |
Is it adequate to the task of informing consumers of what info is captured etc? |
5 |
23 |
| 9.3 |
Disclaimers, privacy policy, sponsorship, advertising policy |
10 |
46 |
| 9.3.1 |
Are they adequate to the task of informing consumers on limitations? |
1 |
5 |
| 9.9 |
Safeguards, NEC |
0 |
0 |
back to top
Table 5: Website criteria sets meeting main evaluation criteria
| Criteria Met |
Number |
Percent |
| All (6) |
0 |
0% |
| 5 |
4 |
18% |
| 4 |
1 |
5% |
| 3 |
7 |
32% |
| 2 |
6 |
27% |
| 1 |
4 |
18% |
| 0 |
0 |
0% |
| TOTAL |
22 |
100% |
back to top
Table 6: Health website criteria sets meeting evaluation criteria
| Code |
Evaluation Criterion |
Number
Meeting |
Percent
Meeting |
| 1.0 |
Is purpose of criteria set defined clearly? |
17 |
77% |
| 1.1 |
Is purpose defined operationally? |
4 |
18% |
| 1.2 |
Is criteria set's scope (operational range/limit) described? |
5 |
23% |
| 2.0 |
Is intended user of criteria set (tool) specified? |
9 |
41% |
| 2.1 |
Does intended user include consumers? |
5 |
23% |
| 3.0 |
Is criteria set sufficiently comprehensive for purpose? |
15 |
68% |
| 3.1 |
Is context/basis of criteria (that constitute set) described? |
10 |
46% |
| 3.2 |
Do criteria encompass all essential dimensions/
concepts? |
2 |
9% |
| 4.0 |
Are criteria (in set) defined in operational terms? |
6 |
27% |
| 4.1 |
Are applicable terms defined operationally? |
6 |
27% |
| 4.2 |
Are criteria clear, unambiguous, etc? |
5 |
23% |
| 4.3 |
Is scoring of individual criteria described clearly? |
3 |
14% |
| 4.4 |
Is use (e.g., aggregation) of criteria scores explained clearly? |
3 |
14% |
| 5.0 |
Has reliability/validity of criteria set (tool) been tested? |
0 |
0% |
| 5.1 |
Is test design adequate/appropriate for criteria purpose? |
0 |
0% |
| 5.1.1 |
With info given, could another person replicate test? |
0 |
0% |
| 5.2 |
Did test include intended users of criteria set (tool)? |
0 |
0% |
| 5.2.1 |
Did test include consumers? |
0 |
0% |
| 5.3 |
Were sources of variation in test results appraised? |
0 |
0% |
| 5.4 |
Were test results analyzed appropriately? |
0 |
0% |
| 6.0 |
Is criteria set (tool) user-friendly (forms, instructions, etc)? |
14 |
64% |
| 6.1 |
Is it consumer-friendly? |
13 |
59% |
back to top
Appendix 1: Meta -Criteria
1. Content Relevance
1.1 Purpose/Mission/Scope: Does the site state its purpose? If yes: Is it to provide information on health topics to consumers?
1.1.1 Is the mission stated? Does the mission statement convey purpose & scope of website?
1.1.1 Scope: Does the site state its scope?
1.1.2 Objective/Goals: Does the site state its objective?
1.1.3 Does the site indicate the title/subject of site?
1.2 Audience: Does the site state its intended audience?
1.3 Relevance: Can one tell easily if what's on the site is relevant to one's search purpose?
1.4 Breadth and depth, Coverage, Completeness, Uniqueness
1.4.1 Is the information unique or can it be found elsewhere?
1.5 Site consists only of links (i.e. no substantive information)?
1.6 Content relevance, NEC
2. Content accessibility, presentation,design:
2.1 Comprehensibility: Is the site prepared at a level that is useful to the intended audience?
2.1.1 Or, failing any statement about intended audience, is it comprehensible by the average consumer? Note: Consumers will be likely able to assess whether or not they can comprehend content (the materials on the site).
2.2 Navigability, logical organization:
2.2.1 Is the information needed to use the evaluation checklist easily accessible on the site?
2.2.1 Is site content easily accessible? E.g., doesn't require plug-ins
2.2.2 Is the site easily navigable?
2.2.3 Does it contain a search engine (when such is necessary to access easily the site's content)?
2.2.4 Is website designed in a user-friendly manner?
2.2.5 Is there a site index, table of contents?
2.3 Does the site organize information in a consumer-friendly way, e.g., by health topic (e.g., cures for breast cancer), rather than simply providing research reports that the consumer has to search?
2.4 Readability: Is text, graphics, video etc consumer-friendly (readable)?
2.4.1 Does site contain (useful) images?
2.4.2 Is site available in text only mode?
2.4.3 If user can't access images, etc, is information still "complete" (useful)?
2.5 Downloadability
2.6 Are there conditions to accessing content?
2.6.1 Is there a fee to access content?
2.6.2 Does the user have to register?
2.6.3 Does accessing content require acceptance of cookies
2.7 Presentation of website: Is it appealing?
2.9 Content accessibility, NEC
3. Content selection/editorial policy (an important aspect of evaluating potential bias).
3.1 Editorial policy, site evaluation process, peer review process: Does the site describe how it searched for and selected the information displayed from the universe of information that exists? Note: This concept might be captured by "editorial policy."
3.1.1 Was the search strategy adequate to the site's purpose? If no, purpose stated, was it reasonable? Does the site display or communicate search results?
3.1.2 Was the selection method adequate to the site's purpose? If no, purpose stated, was it reasonable? Does the site display or communicate selection results?
Note: A consumer-oriented rating system would need to assess these aspects of the website. A critical issue is sponsors' or advertisers' participation in these processes.
3.2 Conflict of interest, bias, disclosure of sponsorship, funding sources
3.2.1 Does the site provide a conflict of interest policy?
3.2.2 Specifically, does site report any relationship between authors of content and the owner, sponsor, or operator of the website?
3.9 Content selection, NEC
4. Content validity/accuracy
4.1 Does the site differentiate clearly between (or it's obvious what are) essays/ commentaries/ "opinions" and reports/ information/ "facts?"
4.1.1 Is the information/content thorough and detailed?
4.1.2 Is the information accurate?
4.2 Does the site give the authors' information for "opinions" and/or reports of "facts" ("articles" presented on the website)
4.2.1 Author's name
4.2.2 Author's credentials (qualifications and experience)
4.3 Does the site indicate the date the "opinion" or report of "facts" was written or last updated?
4.4 For reports of "facts" (articles), does the site give the source of specific information it is communicating?
For each source referenced, does the citation to the source give all of the following information (regardless of whether or not there is a link, to promote user-friendliness):
4.4.1 Link to the source (best practice, whether on- or off-site)? Enough information to obtain it?
4.4.2 Authors of the source
4.4.3 Date the source information was generated or the date the report containing the referenced information was published.
4.4.4 Date information was posted on the site
4.4.5 Bibliography/resource list
4.4.6 Type of source e.g. government, educational institution etc.
Note: Specific information refers to a statement that is the equivalent of, eg, "this treatment has been shown to cure breast cancer."
4.5 Does the site indicate whether or not it has evaluated specific information (source) before displaying it on the website?
4.5.1 Is so, does the site describe how the specific information was evaluated?
4.5.2 Are the methods adequate to their purpose?
4.6 Does the site state the evidence grade of specific information?
4.6.1 If so, does the site describe how the evidence grade was assessed?
4.6.2 Are the methods adequate to their purpose?
Note: Evidence grade refers to the likely validity of the information for its purpose based on an assessment of the type of research study that produced it, and, better yet, an assessment of the study, e.g., based on assessing the research report (the adequacy of methods in relation to reported findings)
4.7 Does the site indicate the generalizability or limitations on applicability of specific information, or provide similar assistance to consumers in its use for some spectrum of purposes?
4.8 Does the site state whether or not it carries advertising (if not obvious that it does)?
4.8.1 If the site carries advertising, does the site clearly distinguish between advertising and content?
4.8.2 Does it provide its advertising policy?
4.8.3 Is its advertising policy appropriate for consumers (eg, doesn't accept advertisement for products with dubious claims of effectiveness)?
Note: The term "advertising" includes any payment for display of (or link to) content.
4.9 Content validity, NEC
5 Content interchange, interactivity
5.1 Is there a way for consumers to request additional information? Does the site state how long it usually takes for the inquiry to be answered? Note: This is useful mostly for rating websites.
5.2 Does the site offer decision support technology (a way that consumers can access information specific to their needs or concerns), such as, eg, a consumer self-assessment tool?
5.2.1 If so, does the site describe the expert system used for this purpose? Is the system adequate for its purpose?
5.2.2 Are specific caveats given regarding the system's use (eg, consumer must give honest information if results are to be meaningful) and limitation of relevant information?
5.2.3 Does the site describe how the DST was developed and validated? If so, was the method sufficient to validate the DST?
5.2.4 When was the DST last updated and/or validated?
5.3 If the site offers exchange of information between consumers and health professionals, does the site describe their credentials (qualifications, experience, etc). Does the site indicate whether or not it has verified their credentials? If so, were the methods appropriate?
5.3.1 Is it obvious to the consumers when they are (and are not) interacting with a specific health professional (whose credentials have been described)?
5.9 Content interchange, NEC
6 Site transparency/credibility/currency
6.1 Does the site say who owns or pays for (sponsors) or created it?
6.1.1 Does it list a physical location, telephone number, contact person, etc?
6.2 Does the site say who runs it (if not the sponsor)?
6.1.2 Does it list a physical location, telephone number, contact person, etc?
6.3 Does the site provide the email address (or other contact information) for the webmaster (or other contact at the sponsoring organization)?
6.4 Does the site describe the sponsor's purpose or business?
6.5 Does the site disclose sources of financing of the sponsoring organization? Of the operating organization?
6.6 Currency
6.6.1 When was the site first created?
6.6.2 How frequently or often is the site/site content updated?
6.6.3 Does the site indicate when it was last updated? Better yet, when specific sections were last updated?
6.6.4 Does the site indicate date until which the content/information is valid?
6.7 Is the site essential "complete" (no or few "under-construction" banners)?
6.8 If site is moderated, is moderator's identity disclosed?
6.8.1 Are credentials, affiliations, etc of the moderator disclosed?
6.8.2 Is method used to select moderator disclosed?
6.9 Site transparency, NEC
7 Links
7.1 Does the site link to other sites as sources of health information? If so, does it state the basis for displaying links (eg, its linkage policy)?
7.2 Does the site indicate whether or not it has evaluated the adequacy of sites to which it displays links for additional health information?
7.2.1 If so, does the site describe the basis for such evaluation? Subsidiary questions: Are the methods adequate to their purpose?
7.3 Does the site provide appropriate descriptions, disclosures, and disclaimers about links?
7.4 Are consumers alerted when they leave the
|
