False Oracles: Consumer Reaction to Learning the Truth About How Search Engines Work

Results of an Ethnographic Study (Full Report)

Based on field research conducted by Context-Based Research Group

Acknowledgements

TABLE OF CONTENTS

Executive Summary

A year after the U.S. Federal Trade Commission (FTC) determined there was a need for "clear and conspicuous disclosures of paid placement" on search engines (1), Consumer Reports WebWatch released data from a new study that shows consumers can't always discern paid search from pure search results (2). Key findings were released April 24, 2003, in New York City at Consumer Reports WebWatch's First National Summit on Web Credibility.

This study, which tested 15 major search and navigation sites, used an ethnographic approach that allowed researchers to observe experienced Web searchers in their natural surroundings. Search companies have done their own usability-related studies, but the results are usually proprietary. Context-Based Research Group, a cultural anthropology research firm based in Baltimore, conducted the field research.

Four anthropologists recruited 17 Web consumers from four U.S. metropolitan areas (Kansas City, Mo., Phoenix, Ariz., Providence, R.I., and Raleigh-Durham, N.C.). None of the participants knew about pay-for-placement on search engines prior to joining the study. Each candidate answered a pre-screener open-ended question that asked for his or her understanding of how a typical search engine ranked its results. (See FIGURE 1: Participant demographic profiles.)

Once admitted to the study, each consumer performed two online searches (one e-commerce, the other informational) on five pre-assigned search sites while being observed and interviewed by the anthropologist. A total of 163 completed searches were performed in which a participant selected at least one link from the delivered results pages. (The 15 sites assessed in this study were About.com, AlltheWeb.com, AltaVista.com, AOL.com, Ask.com, Go.com, Google.com, InfoSpace.com, iWon.com, Kanoodle.com, LookSmart.com, Lycos.com, MSN.com, Overture.com and Yahoo.com.) See APPENDIX A: Background and study methodology for information on how we selected these sites.

After the planned searches, participants were told how pay-for-placement works on each search site under assessment. To lead into this discussion, the ethnographer pointed out each of the "disclosure" pages and "About Search" links posted on each site under evaluation and asked the participant to read through them. (Many of these disclosure pages were located in areas geared toward advertisers, not consumers.) One-on-one field interviews with participants, which each averaged two hours in duration, were conducted pre- and post-enlightenment. After being enlightened, participants were asked to do "homework" assignments, 10 additional searches on sites and topics of their choosing, before a second field interview with the ethnographer. (Participants were asked to record in a search journal their thoughts and behaviors recalled during these "homework" sessions.) The participants' pre- and post-enlightenment online searching was observed to monitor changes in attitudes and behaviors over time.

back to top

Major Findings

• Most participants had little understanding of how search engines retrieve Web pages or how they rank or prioritize links on a results page.
  
  
• The majority of participants never clicked beyond the first page of search results. They trusted search engines to present only the best or most accurate, unbiased results on the first page. As a result, two-in-five links (or 41%) selected by our participants during the assigned search sessions were paid search results.
  Once enlightened about pay-for-placement, each participant expressed surprise about this search engine marketing practice. Some had negative, emotional reactions.
  
  
• All participants said paid search links on search and navigation sites were often too difficult to recognize or find on many sites, and the disclosure information available was clearly written for the advertiser, not the consumer. Search engine sites that were perceived to be less transparent about these related disclosures lost credibility amongst the group.

1. Federal Trade Commission letter to Commercial Alert, "Complaint Requesting Investigation of Various Internet Search Engine Companies for Paid Placement and Paid Inclusion Programs." Available online at http://www.ftc.gov/os/closings/staff/commercialalertletter.htm
   
   
2. Paid search (a.k.a., pay-for-placement, pay-for-performance, pay-per-click, paid listings): Guaranteed prominent placement of a Web page link or ad in the search results of a search engine, triggered by keyword terms an advertiser has purchased.

back to top

back to top

Note: ConsumerWebWatch.org does not participate in any pay-for-placement or paid inclusion/submission program on any search engine or navigation Web site.

back to top

About Context-Based Research Group:

Context-Based Research Group is an ethnographic research and consulting firm located in Baltimore, M.D. with a global network of cultural anthropologists located around the world. Professional anthropologists Robbie Blinkoff, Ph.D. and Belinda Blinkoff, M.A., A.B.D. created Context in partnership with Chuck Donofrio, the President and C.E.O. of Carton Donofrio Partners, Inc., an international brand experience design firm also located in Baltimore.
http://www.contextresearch.com

back to top

Introduction

Consumer Reports WebWatch learned in its first commissioned study -- a national telephone survey of 1,500 Web-savvy U.S. adults conducted by Princeton Survey Research Associates -- that 60 percent did not know some search engines accept fees in exchange for giving prominent placement to advertiser Web pages within the results pages. (i) In addition, four-in-five respondents (or 80%) said it was important search engines disclose their paid search policies in the search results or an easy-to-find page on the site.

A more recent Consumer Reports WebWatch credibility-related joint study, lead by Stanford University's Persuasive Technology Lab, found half the consumers (or 46.1%) judged the credibility of a Web site based, in part, on superficial aspects such as Web design or overall visual appeal. (ii) These Stanford participants were more likely to assess a search engine site's credibility based on visual cues (52.6% vs. 46.1% overall). (3) In addition, less than four percent of consumers who assessed the 10 search sites in this Web-based study commented about information bias. In comparison, 11.6 percent of information bias-related comments were made about the study sites overall, and 30.2 percent of comments were derived while assessing news-specific sites, where the issue of information bias and sponsorship influence is presumably more understood (and expected) amongst the public.

In this most recent study on search engines and navigation sites, Consumer Reports WebWatch decided to use an ethnographic or anthropological approach in order to dive deeper into how consumers use these popular Web navigational points-of-entry. We commissioned Context-Based Research Group, an ethnographic research and consulting firm based in Baltimore, to conduct this project. Robbie Blinkoff, Ph.D., a managing partner and principal anthropologist, directed the project. (See also APPENDIX B: The merits of ethnographic research.)

The 15 sites assessed in this study:

About.com
AlltheWeb.com
AltaVista.com
AOL Search (AOL.com)
Ask.com
Go.com
Google.com
InfoSpace.com
iWon.com
Kanoodle.com
LookSmart.com
Lycos.com
MSN Search (MSN.com)
Overture.com
Yahoo.com

Why focus on search engines? (4)

Switching stations: Searchers use these navigational portals to move from one site to another, particularly when the user is unsure of where to surf next. Findings from various research reports show at least 85 percent of Web users query search engines with great frequency. (iii) Other research indicates the percentage of search engine referrals worldwide has increased significantly over the past year: up to 13.5 percent of global referrals from 7.2 percent the previous year, according to online marketing analytics firm WebSideStory.com. (iv) (Search sites in the U.S. account for 15 percent of all referrals, up from 8 percent a year ago.) Clearly, online consumers rely on search engine and portal sites to find and to navigate Web pages more quickly and efficiently.

Paid search factor: These advertiser-based results can nudge online consumers to click on Web pages, links or ads listed prominently in the results, yet not necessarily the most relevant to their search query. Even well informed or Web-savvy users may not know whether a listing is a paid ad. In addition, in the wake of the dot-com advertising bust in 2001, and the decreasing effectiveness of banner ads, online advertisers and marketers are anxious to find tactics that will boost visibility with consumers. Keyword and other forms of paid search can make a significant difference to a company's bottom line. A recent Jupiter Research poll found that 76 percent of marketing executives who have used search engine marketing rate it more effective than banner-style online advertising. (v) About two-thirds (or 64%) said they intended to increase their search engine marketing budgets in the next 12 months. (Indeed, a June 2003 report released by the Interactive Advertising Bureau said keyword search made up 15 percent of last year's total Internet advertising revenues, up from 4 percent in year 2001.) According to industry analysts U.S. Bancorp Piper Jaffray, total paid search revenues in the U.S. will approach $2 billion by the end of 2003, then jump to nearly $5 billion by 2007. (vi)

Consumer detriment: There's a higher risk to the Web consumer of making flawed buying and other decisions as a result of reliance on information from a prominently ranked site that doesn't necessarily have the most accurate or most authoritative information. This risk factor is highest when consumers search for information on health or financial issues in which actions based on faulty, misleading or incomplete information could adversely affect their health or money matters. (vii) For example, a November 2002 Consumers International study of 460 Web sites found half the sites giving advice on medical and financial matters failed to provide full information about the authority and credentials of the people behind that advice. (5)

3. The Stanford study asked participants to assess the credibility of 10 search sites, randomly assigned in pairs, that were pre-selected by researchers for a variety of reasons, including Web design, user interface, brand name, and the presence or absence of disclosure information. The search sites assessed between June 15 - August 15, 2002 were About.com, AlltheWeb.com, Ask.com, Google.com, Insider.com, iWon.com, LookSmart.com, Overture.com, Vivisimo.com and Yahoo.com.

4. Consumer Reports WebWatch's definition of a search engine: Any major site with the ability to point or navigate hundreds of thousands of Internet users to different points of entry online, including search services, directories, portals and community sites.

5. Consumers International (2002), Credibility on the Web: An international study of the credibility of consumer information on the Internet.


back to top

What Consumer Reports WebWatch hoped to accomplish:

• First, to elaborate on the Princeton survey statistics about search engines. Consumer Reports WebWatch wanted more confirmation Web consumers are still largely unaware of paid search, and that such paid results might influence the ranking of their search results.
  
  
• Second, to assess whether consumers could tell the difference between paid search results and pure or algorithmic search results, or not.
  
  
• Third, to assess whether consumers noticed and understood the "About Search" links or related disclosure information posted on such sites a year after the FTC determined there was "the need for clear and conspicuous disclosures of paid placement…so that businesses may avoid possible future Commission action." (viii)
  
  
• Fourth, to understand whether consumers' online behavior would change once they learned more about search engine pay-for-placement marketing and how related results rankings might be biased toward advertisers.
  
  
• Fifth, to alert the search engine community that current marketing and advertising practices could lead to long-term credibility problems for the industry.
  
  
• Sixth, to call attention to potential concerns or inadequacies in the FTC recommendations to search engines to make their practices more transparent.

What the study does not cover:

The study did not focus on the concept of paid submission (a.k.a., paid inclusion) (6) because such techniques do not typically taint search engine result relevancy or rankings. In addition, Consumer Reports WebWatch thought that adding the concept of paid inclusion to the concept of pay-for-placement would only confuse and possibly frustrate participants.

Consumer Reports WebWatch decided to focus on consumer understanding of paid placement, as its influence can adversely affect search results positioning, thus potentially denying consumers the ability to make the best buying and related household decisions while online. Further, basing a health- or finance-related decision on inaccurate information can be literally dangerous to a consumer's well-being.

Secondly, the study did not attempt to determine whether the search results the participants received or clicked on were in fact accurate or relevant to their query. However, we took notice of occurrences in which the participant remarked to the ethnographer about the perceived accuracy or inaccuracy of search results received during search sessions.

6. Paid submission: Submitting a Web site or pages for guaranteed inclusion into a search engine's database or listings. This does not guarantee prominent listings or a rankings boost over those pages retrieved from the engine's indexing (crawlers), but it does guarantee a site is "crawled" more deeply or with greater frequently.

back to top

The names of participants have been changed. Participants were assigned pseudonyms to protect their privacy and to encourage candidness with the ethnographer.

Excerpted ethnographer field notes and long quotations from participants appear in block quote format. Direct quotes from participants are set off in double quotation marks.

Some participant quotes or ethnographer field notes were edited for grammar or clarity.

PRE-ENLIGHTENMENT ATTITUDES AND BEHAVIORS

1. All participants used one search engine as their sole or primary point of Web search entry.

Thirteen participants reported using either Yahoo or Google for all or nearly all of their online searching. (Google garnered eight loyalty votes, while Yahoo had five. AltaVista, Dogpile, MSN Search and Search.com each received one vote apiece.)

Users were loyal to Google because they perceived it to deliver "relevant" and "satisfactory search results." Participants also mentioned its ease-of-use due to an "uncluttered" Web page with "no distractions" like pop-up ads, images and a lot of text. Yahoo's popularity amongst participants was credited to its perceived delivery of "the best results," "appearance and accessibility" and "brand name and overall usability." Three participants admitted heavy usage of one particular search engine simply because it was the default Web home page setting on their PC's Internet browser.

For example, the Kansas City ethnographer reported Geena, 48, a legal assistant who does a lot of work-related searches, has used Yahoo ever since she first went online 10 years ago:

When she first began using the Internet [Yahoo] was set as her default start page. Since then she has not felt a need to change the way in which she searches on the Web. It is also set as her default page at her workplace, however, she did not set it as her start page. She explained that, "it was just like that so I use it."

MSN.com is the default choice for fellow Kansas City participant Kareem, 37, a restaurant owner and home/office Web user for five years.

MSN serves as the homepage on his computer and he therefore uses it the most frequently…When he first used the Internet, and first got a computer, the homepage was set to MSN, which is the reason he has used it as his primary search engine all along.

back to top

2. Most participants did not understand how search engines retrieve information from the Web or how they rank or prioritize links on a results page.

About half the participants (8 of 17) had a vague understanding that search engine results are influenced by keyword or a closest match relevancy. In other words, they had a fuzzy idea search engines relied on the key words or phrases typed into the search boxes to find Web pages, but didn't understand much else.

For example, Penny, 34, a stay-at-home mom in Phoenix said "the search engine goes out and looks at the keywords in the codes of Web pages and if the words match it pulls it down for the results page." Similarly, Vivian, 19, a full-time college student in Providence, suggested search sites "pull up things that have the closest matches to what you typed in and then maybe it ranks them in order of what page or where on the page the words show up."

Two people characterized search and navigation sites as "something you can give a word to and it goes out and finds matches that most closely match your word" (Sara, 30, elementary school teacher, Raleigh-Durham) or "software that goes out on the Internet using search words to find Web sites." (Yvonne, 32, physician's assistant, Phoenix)

Bob, a 46-year old Phoenix-based geographic consultant with six years online, gave this response:

"A search engine is an interface with a Web page you go to in order to query where to find information. You type in keywords and you get 100,000's of returns [that are] prioritized by how well your keywords match the Web page's keywords."

Likewise, Dennis of Providence, 40, a jewelry-accessories company vice president with a background in trademark law, explained:

"I would say [search engines] allow you to find information about a topic that is located elsewhere on the Internet by clicking in key phrases related to what you are interested in."

Not surprisingly, some participants were more advanced in their understanding than others. Two in particular, Gary of Phoenix (a 40-year-old environmental compliance coordinator) and Larry of Providence (a 62-year-old marina and boating consultant) had the most sophisticated understanding of how search engines work:

Here's how Gary, who manages a personal home page, explains how search engines work to the Phoenix ethnographer:

"[E]ach search engine is set up differently to do Web searches. Each engine has an algorithm that determines how the engine's 'spider' or Web 'crawler' (the information retrieval thing) will chose the Web sites for the results page. Some spiders are sent out to seek 'meta-tags,' but some reject 'meta-tags.' Some accept hidden text, while others reject hidden text. It depends on the search engine's goals, which are determined by whoever runs/owns the search engine."

Larry of Providence, who has experience compiling a database of Web pages related to environmental issues, conveyed a higher level of understanding to the researcher:

"[It] operates on keyword searches, which might be words embedded in text. It scans through and reads names of individual places and products and with some systems, they will even list whatever keyword they think you want to have…That is what gets registered with the search engine."

Three participants (Jack and Kareem of Kansas City and Mike of Raleigh-Durham) had naïve perceptions of how search engine result links are ranked, believing priority listing was based, in part, on some kind of popularity- or seniority-related measurement.

According to the ethnographer's field notes, Jack, a 28-year-old graphic artist with eight years of online experience, assumed the results were selected based on the "most frequently visited Web sites." He also stated that Web sites ended up in the results based on their relevancy to keywords in a query.

Fellow Missouri statesman Kareem, who also maintains a Web page for his small business, "assumes" certain links are ranked higher than others because they are logged on more frequently.

Mike of Raleigh-Durham, 25, an electric meter reader and part-time student with five years' Web experience, suggested results were ranked based on seniority:

He believes the sites that are ranked higher are there "because they are the most established or oldest. Perhaps because they have the most hits."

Four participants (Geena of Kansas City, and Anna, Claude, and Diane of Raleigh-Durham) admitted they had little understanding of how search engines prioritize results.

Anna, a 55-year-old retired banker, guessed search engines rank results based on the "more words I entered are on that site." Claude, 36, a construction worker with six years online, believed search results are ranked, "listed by the best site." And Diane, 20, an undergraduate university student with five years of Web experience, thought search engines "scanned [the Web]…and the first ones are just like you want it."

While no participant knew about pay-for-placement on search sites, two participants suspected results rankings weren't based entirely on a pure search or algorithmic method.
The Kansas City ethnographer noted this about Lucy, 24, a child daycare coordinator with seven years of Web experience:

She alluded to pay-for-placement a tiny bit when asked about the reason certain items are ranked higher than others did in a search result, but then backed off. "I don't know why, I guess because of pay or something, or actually it's because of keyword I think."

Likewise, Providence resident Maria, 27, an office manager for a non-profit art organization with nine years online, suggested there were factors at hand that drive search results other than unbiased or algorithmic rankings.

"I always [thought]…it was depend[ent] on what the search engine wanted to present to people and also what the places [Web sites] kind of give to the search engine. I don't know if it is a money deal or whatever, I don't know if they exchange [money], but it has something to do with helping each other out."

back to top

3. The majority of participants never clicked beyond the first page of search results as they had trust in the search engine to present only the best or most accurate results on the first page, making it unnecessary to review later results pages.

Each of the 15 search engines under evaluation was assessed by a minimum of five participants. A total of 170 unique search sessions were performed during the planned browse phases of Field Interview #1, an equal number of commercial searches and informational searches for health, finance or travel information (i.e., 85 searches of each type.) Seven searches were aborted before a result link was clicked because the participant reported he or she didn't see a link of interest to them, or because the results were redundant to those seen on other search engines tested. (There were two aborted searches on MSN.com and one apiece on About.com, Go.com, InfoSpace.com, iWon.com, and Kanoodle.com.)

(See APPENDIX C: Search engine and navigational sites evaluated, by participant.)

Not one participant explored search results beyond the fifth delivered page. Of the 163 searches in which a participant selected a link to explore, the majority or 88 percent of the result links selected were located on the first page. The number of pages explored by participants decreased considerably after page one. There were only 16 cases in which a participant viewed the second page of results, two cases where the third page was reviewed, and one case in which the fifth page was scanned.

There was a slight difference between commercial and informational searches. Of the 144 cases in which a link was selected from the first results page, 75 were picked during commercial searches, while 69 were chosen during informational searches.

Prior to the planned browse searches, in which the ethnographer probed a participant on his or her searching behavior, pre-enlightenment, most participants had a fairly accurate depiction of their online behavior. The majority (or 12 of 17 participants) stated they only clicked on links listed on the first page of search results, which later proved to be accurate once these statements were compared to their actual behavior displayed during respective planned searches.

Two participants, Anna and Diane of Raleigh-Durham, were overly conservative in what they reported to the ethnographer. While each stated she typically stopped at the first page of results, in three of 10 cases during the planned search each reviewed two or three pages of results, certainly far more than the group on average. (Anna assessed About.com, AltaVista.com, Google.com, MSN.com, and Overture.com. Diane assessed Go.com, InfoSpace, iWon.com, Kanoodle.com and LookSmart.com.)

Another participant not only reported he was diligent about reviewing a number of pages before making a link selection, but his actual behavior during the planned search bore this out. Kareem of Kansas City's exchange with the anthropologist indicates how thorough he is when reviewing the results he receives:

He typically scrolls through several pages, "at least three," before choosing a result, however, he said it ranges from one to six [pages]. He explained, "because when I've found my appropriate search phrase, everything will be in those results pages. I click on lots of pages because there's lots of good information." He scrolls through anywhere from "one to twenty to fifty" actual results before selecting a link.

True to form, an analysis of his 10 planned searches revealed Kareem had clicked on a first-page result only three times. He reviewed through the second page of results half the time (5 of 10 times), reviewed through the third page of results one time, and through the fifth page one time. (Kareem performed his 10 planned searches on AOL.com, AlltheWeb.com, Ask.com, Lycos.com, and Yahoo.com.)

There were, however, two participants who said one thing, but actually did another. (i.e., there was a disconnect in reported behavior versus actual behavior.) The first is Lucy of Kansas City. According to the ethnographer's field notes, Lucy reported the following:

She always scrolls through all the results, glancing at the descriptions quickly, before making any decisions. Then, she zeroes in on the one she wants and reads the details and sometimes the Web address, if it is offered. She explained that, "the top one isn't always the one I want so I look through them all." I noticed that although she looks through "them all," she only looked on the first page. When I asked her about that, she concurred, stating that she "almost never goes beyond the first page because it would take too much time."

It seems the answer extracted from Lucy, triggered by probing from the anthropologist, proved to be most accurate. A review of Lucy's planned searches, in which she assessed AOL.com, AlltheWeb.com, Ask.com, Lycos.com and Yahoo.com, showed she only selected links from the first page of results.

Similarly, the Raleigh-Durham researcher recorded that Mike said:

He typically will "either pick a result on the first two pages, or by the third I may skip to the farthest one I can find, just to see what is there." He will search through three pages of 25 results for a more directed search.

His actual planned searching behavior, however, indicated he never went beyond the first page of results served by assigned sites Go.com, InfoSpace.com, iWon.com, Kanoodle.com and LookSmart.com.

Perhaps more importantly, four participants justified their behavior of stopping at the first page of results because they assumed a search engine would present only the most trustworthy Web sites first, therefore there was no need for them to read beyond the first page.

According to the Providence anthropologist, Larry generally doesn't scroll through any search pages.

Larry chooses one of the first five or so links he sees on his first search results page…If Larry doesn't see anything "in the ballpark" of what he is looking for, he will restructure his search because he believes results are ranked by relevancy and the frequency of the occurrence of his search term in the results that are generated. If he doesn't find what he wants in the first five, he believes that he won't find them further down the screen or on subsequent screens.

Yvonne justified her behavior to the Phoenix ethnographer in this way:

Often she doesn't even scroll down to view all of the results on that page and makes her link choice from the links at the top of the page. It's okay to do this because the search engine has prioritized the links to have the best matches at the top.

Two Kansas City participants expressed similar blind trust of search engines:

Jack claimed, based on his assumption that the first results are the Web sites most frequently visited, that "the most trustworthy sites are the first to come up." When using Google.com, he chose a sponsored link at the top of the results page because he said it looked "trustworthy."

The same ethnographer wrote these notes in reference to Geena:

She does not think there is any difference between the results provided by different search engines.

back to top

4. Nearly half the links selected by participants were paid search results.

Of the 163 searches in which a link was selected, 67 results (or 41%) could be classified as paid search. These included links euphemistically labeled "sponsored" link, listing, match, or result, and "featured site" or "featured sponsor." Most results generated by "pay-for-performance" site Kanoodle were coded as paid search results (7), as well as all results generated by Overture, whether these Overture results came directly from its home page or picked during a search of partner sites Go.com or InfoSpace.com. Similarly, links that could be attributed to the Google AdWords or Sponsored Links programs were coded as paid search, including some results that appeared on partner sites such as iWon and InfoSpace.com.

Slightly more paid links were selected during the commercial searches compared to the informational searches. A total of 36 links were selected during pre-enlightenment e-commerce searches. Fewer paid links (31 to be exact) were selected during informational searches. On the other hand, there were slightly more pure search links selected during informational searches. (50 pure links selected during informational searches vs. 46 pure links selected during commercial searches.)

(See the POST-ENLIGHTENMENT ATTITUDES AND BEHAVORS section for information on consumer perceptions of pay-for-placement results received during commercial vs. informational searches.)

There were some cases in which it was difficult for the Consumer Reports WebWatch researcher to determine whether a selected link should be coded as paid versus non-paid. This is because the disclosure and "About Search" page language for a particular engine was so vaguely worded it was impossible to determine what the site classified as a paid search result. That said, the researcher coded 96 links (or, 59%) as derived from pure or algorithmic generated results, including those Web pages in which there was reason to believe they were present due to paid inclusion or paid submission programs.

7. Other results "powered by Google" were coded as "pure" search results.

back to top

5. Very few participants noticed the paid search labeled "sponsored" links or listings, but only when they were easy to see (i.e., at the top of the search page or clearly marked in a box).

These same participants reported skipping over links labeled "sponsored" because they assumed these results didn't have what they wanted, even though they couldn't always articulate why.

For example, the Phoenix ethnographer reported this about Gary:

When he did the search on About.com (a site he had never used before), he quickly skipped the top section of results. On Google, he skipped over the highlighted results and behaved similarly on the other sites. He said he skipped those results because they seemed less (or not at all) relevant. That is when he started noticing the sponsored link notes and other clues to pay-for-placement and recognized those links had paid to be included in searches.

The same researcher noted:

Penny skips over sponsored links on any results page, not because they are pay-for-placement, but because they don't look right. Because they are highlighted or separated or made to stand out somehow, they don't fit into the format that she expects on the results page, so she ignores them.

In a different city, the Providence ethnographer recorded similar online behavior for Vivian:

She says she never looks at sponsored links, though she didn't know they were paid for. When we talk about premium sponsorship on Google, she says she never looks at the information at the top. "I never actually look at them very closely, but I know they are trying to sell me something. I think that these are advertising and that these (points to sponsored links on side) are the same, so I just never look at them."

The Providence researcher noted Larry also ignored sponsored links, such as those labeled "sponsor matches":

"I don't look at those as a general rule if it is in a separate box as I know it is a sponsored thing. Well, you know, I'll give them a quick glance, but I know that those aren't what I want."

back to top

6. Paid search disclosures were overlooked in every case.

The four ethnographers noted none of the participants in his or her city noticed the "About Search" or other disclosure information pages on the 15 search engines under assessment during the 10 planned searches conducted pre-enlightenment.

back to top

POST-ENLIGHTENMENT ATTITUDES AND BEHAVIORS

1. Each participant expressed surprise after learning about this search engine marketing practice. Some had negative, emotional reactions.

Sara of Raleigh-Durham reacted emotionally when told about pay-for-placement. The ethnographer recorded this exchange:

"Well, that figures!" I believe she may get a little depressed about this information. She remained quite melancholy throughout the rest of the visit. I could tell her mind was rolling with the repercussions of pay-for-placement. She seems concerned her students [will have to grow up and have] to learn how to see bias while using a computer.

Similarly, fellow North Carolina resident Anna said she was concerned about those more vulnerable to information bias, like children:

"I expect information bias and, therefore, take it into consideration when making decisions because I'm old and jaded." She later added, "I feel like there are people, like family, that are more gullible and [will] accept anything they read on the Internet as gospel." When I asked, whom, she responded, "My mother, my daughter, and other people's families."

Four participants reacted angrily or expressed a sense of betrayal, including Kareem of Kansas City:

Upon enlightenment, his face contorted a bit. He expressed immediate dismay, stating, "Well, that explains it. No wonder I run into so much [expletive]." Yet, he tempered his displeasure quickly, stating, "I expect it though, I guess I kind of suspected something like this. I just never thought too much about it. It just annoyed me."

Mike of Raleigh-Durham, also had an angry reaction.

I found he is not happy he has to "be careful" when searching "so I don't get sites I don't want," the ethnographer recorded in the field notes.

Geena of Kansas City was taken aback at first, and then she blamed herself for not noticing disclosure information sooner.

When we visited the disclosure links she was somewhat stunned. She seems to be a very, very trusting person that thinks the best of everyone and everything, so she was shocked. She felt extremely betrayed. She expressed a slight bit of disappointment with herself because she said she should have visited the disclosure link sooner, but then brought the site to task for not disclosing it in a more explicit manner.

Similarly, Claude of Raleigh-Durham expressed a sense of betrayal and then became disturbed when he suspected this practice might have influenced his more serious informational searching.

The participant insisted informational searches "are no place for advertising." The participant made an excellent point when reflecting on pay-for-placement. "People like me may only get to learn stuff online, and if we can't then that's just as biased as having to pay lots of money to go to some college to learn."

The ethnographers recorded three exchanges with participants that indicated they were confused or disappointed:

Vivian of Providence expressed disappointment, but like Geena, blamed herself.

"Oh, I'm such a sucker!" (She laughs.) I tell her she isn't [a sucker], but that she might want to know this when she is shopping or looking for information. I also point out the different headings for search results as they are listed on sites like iWon, LookSmart and Kanoodle. She hadn't noticed these before. "My little fantasy world is wrecked!"

Diane of Raleigh-Durham felt somewhat defenseless, as if unsure what to do next.

"I feel helpless and must tolerate it. It's annoying everywhere it happens, but what can you do about it."

Lucy of Kansas City was confused at first, until she recalled a previous experience:

When Lucy and I visited the disclosure pages on the search engines, she was a bit puzzled, at least the look on her face seemed to be a mix of bewilderment, confusion, and displeasure. She expressed a sort of 'ah-ha' and continued, "So that is why I kept getting sent to eBay and other annoying product sites I didn't want." She then expressed her appreciation for being informed of this fact, but then went on to chastise herself for not ever looking at the disclosure links on the search engine homepages.

Two participants questioned the trustworthiness of advertisers that choose to participate in search engine marketing programs.

Gary of Phoenix called into question the ethics of advertisers that pay-for-placement:

"They are being placed there because of money, not content. I am after content." He summed it up by saying, "the bottom line is, if you've got a good Web site don't pay. Let the search engines do their thing."

Providence participant Dennis questioned the ethics of some Web companies. He expressed confusion about what is paid and what is not paid, in this case on a Google results page.

Dennis doesn't click on sponsored links on Google because "they are biased. They are paying Google to be there." He appears to still be confused. "The advertisements are over there (motions to right side of an imaginary screen), and the results are over here. Whether they mix them in there, I'm not sure. To get on the first page maybe they do? Because sometimes I have seen, like, why is this on the first page when what I really need is on the third page?"

The last six participants, while initially surprised about pay-for-placement, eventually came to a more philosophical or realistic outlook. They characterized search engine marketing as a sort of price to pay, like other forms of Internet advertising that keep online information free.

The Phoenix anthropologist recorded this response from Bob:

Being as skeptical as he is, he said he "figured" the results were influenced that way, but he had never bothered to investigate. He was very interested in the 'about the search' pages. "[I]nformation bias is tolerable because any venture has to be profitable and if the Internet wasn't profitable it wouldn't exist." Anyway sometimes he's looking for commercial sites so it's good that they pay to get to the top. He said, "Even if the sites didn't pay-for-placement, the listings would still be biased somehow."

Maria of Providence responded in a similar fashion:

Maria seemed surprised by the information, but not outraged. As she said, "I understand how it works." She added, "I kinda had an idea a little bit that that was how it works, but I didn't understand how money was exchanged or how exactly everything is going together. It is interesting, I didn't realize how much of the beginning pages [were] money pages, like people paying. To find the meat of what it is that I want, what I really want, I have to look farther [down the search results.]"

Jack of Kansas City, said he expects information bias to occur in the commercial realm.

Jack seemed unfazed. He said, "Well, I guess I should have known." He went on to express his displeasure with pay-for-placement, stating he "needs to look more into it because my art [stored on an online gallery site] will never get seen because big sites will always get visited first." Jack seemed a bit unsure, but threw out the idea that some corporations are [more] forceful with their money, which leads to some bias [on search engines.]

Yvonne of Phoenix had this reaction:

Yvonne's searches were a great lead in to the pay-for-placement enlightenment. Even though she had noticed the different [pages] of Amazon were coming up in different order, she still hadn't noticed some were listed as "sponsored" or "recommended" sites. She said learning about pay-for-placement is "kind of a big bummer." She was not really surprised, but a little disappointed. She elaborated, "It's a little bit disturbing, especially when I'm searching for information, I don't know. I guess I expect to be duped when I'm searching for things [to buy], but then again when I put in a specific string I don't want that string to get dummied down or dulled by the fact that somebody paid money to get to the top of the pile." She compares [her post-enlightenment knowledge of] pay-for-placement on search engines to commercials on television. "It's like commercials on television. I don't particularly care [that they are there], but it's good to know the information you are getting is being filtered through that lens. Then you can adjust your perception of that information. You are buyer beware because you are aware."

The same Phoenix researcher noted a similar kind of exchange with Penny:

She was very realistic about pay-for-placement saying, "It's just like going to the mailbox, you just have to filter through the junk -- there might be a check for $1,000 in there, but there will also be at least four pieces of junk mail." Penny defines information bias as the presentation of material in a manner to achieve a desired outcome. "It's what people want. They have a particular outcome in mind so they present information in a particular way." Though Penny isn't fond of all the advertising on the Internet, especially the flickering ads, she believes that it is worth tolerating because it allows access to the Web sites to remain free of cost.

Providence resident Larry was the sole participant to express an interest in the disclosure information on the search site so he could use it for personal business purposes.

Larry's reaction to enlightenment about pay-for-placement was that of a businessman…Once made aware, he wanted to use it himself. He hadn't understood how Google made money. We explored this (how they sell their search services to AOL, to Cox Cable, etc.) and… pay-for-placement. As we read the descriptions of different pay-for-placement programs on [Google], I could see the gears turning and hear his comments, mostly about how he could use this method to get people to look at his not-for-profit Web site and Web-based database for information about marinas he writes for the federal government.

back to top

2. No participant stopped using a favored search engine after learning about pay-for-placement practices, although they reviewed search results with a more critical eye.

For example, Sara of Raleigh-Durham thinks, post-enlightenment, the results across search engines are generally "all the same, but I'm just more in tune to it, or mindful."

Some participants reported they scrutinized the results pages more carefully. Jack of Kansas City tends to scroll through more options and go to more pages to look for a site to choose, according to a verbal report to the anthropologist.

Some participants used more than one search engine before making a final decision to click a link, as was the case of Lucy of Kansas City:

Since her enlightenment [Lucy] has begun to use more search engines than before because she was previously unaware of all the different engines, but also to compare their results. Prior to enlightenment, Lucy never used a search engine other than Yahoo, and never "went past the first page of results."

Some participants used a combination of methods to compare results across engines before making a decision to click on a link. For example, Geena of Kansas City reported viewing more results pages across multiple search sites.

[Geena] has become a much more astute and observant searcher and Web user. She went on to explain, "I have begun to go through more pages of results. When I went to look for cruises the other day, I noticed the same results came up multiple times at different spots in the results section."

Similarly, Yvonne of Phoenix reviews more results pages and checks non-paid links before making a decision to click.

After learning about pay-for-