Booking and Bidding in the Blind:

An In-Depth Examination of Opaque Travel Web Sites (Full Report)

TABLE OF CONTENTS

Executive Summary

Consumer Reports WebWatch Mission Statement

Consumer Reports WebWatch Travel Projects in 2003-04

Overview of Opaque Travel Web Sites

Testing Methodology and Parameters
Selection of Web Sites
Methodology
Testing Parameters and Specific Testing Criteria

• Airlines
• Hotels
• Car Rentals

Booking Procedures
Selection of Results
Currency Issues
Rounding of Fares and Rates
Availability of Fares
Valid Tests
Invalid Data
Technical or System Failures

Testing Results, Rankings and Conclusions
Lowest Rates for Airline Tests
Lowest Rates for Hotel Tests
Lowest Rates for Car Rental Tests
Lowest Rates for All Tests
Online Customer Service Policies

Booking and Bidding Procedures
Priceline
Hotwire

Findings and Concerns
Pricing Display Issues
Bias Issues

Consumer Reports WebWatch Tips for Using Opaque Travel Web Sites

Appendix I: Booking Fees

Appendix II: Invalid Rates Data

Figure 1: Lowest Airline Rates

Figure 2: Lowest Hotel Rates

Figure 3: Lowest Car Rental Rates

Figure 4: Lowest Rates (All Tests)

Figure 5: Booking Fees

Figure 6: Invalid Hotel Test Results

Executive Summary

In fewer than 10 years, the burden of booking travel has largely shifted from professionals — reservations agents, travel agents, corporate travel managers — to travelers themselves. This has occurred for both leisure and business trips, as more and more consumers have begun tackling the intricacies of booking airline seats, hotel rooms, rental cars, vacation packages, and even more complicated purchases such as tours and cruises.

For the travel industry, it has meant huge reductions in the cost of distributing travel products. It's also meant the emergence of the new and burgeoning online travel industry.

For consumers, it's meant both good and bad news. There's no doubt the Internet has brought lower fares and rates in many cases. But the risks and responsibilities of booking have been assumed by consumers as well.

Booking travel can be complicated, but the complexities increase tremendously when a consumer books through an "opaque" travel Web site such as Hotwire or Priceline that does not reveal key aspects of the itinerary — including the name of the airline, hotel property, or car rental firm — until after the non-refundable reservation has been made. Further, that process can be even more complex if the opaque site requires the consumer to bid for the travel product, as Priceline does.

Consumer Reports WebWatch has been testing travel Web sites for more than a year, and has expanded upon testing methodologies first developed more than three years ago by Consumer Reports Travel Letter (which ceased publication in 2002). Over time, Consumer Reports WebWatch has examined bookings for airlines, hotels, and car rentals. After much development, Consumer Reports WebWatch finally created testing systems to examine the opaque sites.

This research report focuses on that testing, which encompassed both Hotwire and Priceline, as well as non-opaque travel Web sites Expedia, Orbitz, Quikbook, and Travelocity, and Sabre, a global distribution system (GDS) used by travel agencies. For the first time, Consumer Reports WebWatch not only examined the fare and rate information but actually booked travel products. Approximately $38,000 of airline seats, hotel rooms, and rental cars were processed by Consumer Reports WebWatch testers for this project. These reservations were then donated to the Make-A-Wish Foundation of America.

Testing "integrated" travel Web sites — such as Expedia, Orbitz, and Travelocity — presented certain challenges, but ultimately there emerged a strong "apples-to-apples" methodology. The same applied to testing "branded" travel Web sites maintained by airlines, hotel chains, and car rental firms. But testing opaque travel Web sites presented new hurdles for Consumer Reports WebWatch. The primary challenge was to develop methodologies that would allow all of the Web sites to be compared in a fair and equitable manner, in real time.

Based on the inherent differences between opaque and transparent travel Web sites, this project was conceived to answer two fundamental questions:

1. Are the fares and rates provided by either or both of the leading opaque travel Web sites — Hotwire and Priceline — truly lower than fares and rates provided by leading transparent travel Web sites when tested under real-time, apples-to-apples conditions?
   
   
2. If the fares and rates provided by either or both Hotwire and Priceline truly are lower than fares and rates provided by leading transparent travel Web sites, are the trade-offs required by the opaque sites worth the savings?

Consumer Reports WebWatch believes it has fully addressed both these questions. These and many other issues are covered in this research report.

After months of examining, testing, and analyzing these Web sites, some of the most significant findings emerged. Among them were the following:

• In all three sectors of this testing — airlines, hotels, and car rentals — either Priceline or Hotwire provided the highest number and percentage of lowest rates.
  
  
• Priceline emerged as the clear winner at providing the highest number and percentage of lowest rates for airlines, hotels, and car rentals, and did so 47% of the time.
  
  
• Hotwire fared well, ranking third by providing lowest fares and rates 19% of the time, but lagged behind both Priceline and Orbitz in the overall rankings for lowest fares and rates.
  
  
• The trade-offs required by Hotwire and especially by Priceline make using these opaque Web sites a very subjective decision for many consumers. Yes, they're usually better than non-opaque Web sites at delivering lower fares and rates. But both conceal key details of the trip, both offer non-refundable bookings, both always charge booking fees, and both usually do not provide loyalty program miles or points.
  
  
• In addition, Priceline is more cumbersome and time-consuming to use and may require changing key trip aspects, including departure dates and times, airports, hotel locations, and rental vehicle types.
  
  
• Two non-opaque Web sites posted impressive numbers without requiring these trade-offs. Orbitz ranked second overall by providing lowest fares and rates 20% of the time. And Quikbook, which only sells hotel rooms, led the non-opaque Web sites in providing the lowest hotel rates.
  
  
• Unfortunately, concerns over deceptive pricing displays, discussed in a previous research report, were raised once again when an incorrect (lower) rate was posted by Orbitz and the user was not notified that this rate was no longer available. This issue was first addressed by Consumer Reports WebWatch in an examination of car-rental booking sites.
  
  
• In addition, concerns over potential bias emerged again as well. Expedia's initial display of car rental companies continued to omit key rental firms that often offered lower rates. Previously Expedia stated that the rental firms not listed on the initial displays are not "part of our preferred partner program at this time."

The project was directed by William J. McGee, a travel journalist and consultant to Consumer Reports WebWatch and the Editor of Consumer Reports Travel Letter from 2000 to 2002. Harrell Associates Inc., based in New York City, provided paid assistance by helping to develop testing methodologies and providing fares and rates from Sabre, a global distribution system (GDS) used by travel agencies, during the testing period; however, Harrell Associates was not involved in the preparation of this research report. Additional research assistance was not compensated for but provided by Forrester Research, a Cambridge, Mass.-based technology research firm, and PhoCusWright, a Sherman, Conn.-based tourism and hospitality industry intelligence consulting firm.

The project was completely funded by Consumer Reports WebWatch. Employees of Consumer Reports WebWatch assisted in drafting the methodology, participated in the testing, and contributed to this research report. The research report was edited by Beau Brendler, the director of Consumer Reports WebWatch.

back to top

Consumer Reports WebWatch Travel Projects,
2003-2004

Consumer Reports WebWatch remains committed to testing, researching, and reporting on the Internet. But since travel is the single largest source of online commerce, Consumer Reports WebWatch is particularly dedicated to providing unbiased, detailed, accurate, and repetitive testing of travel Web sites.

Here is a summary of the travel Web site projects undertaken to date:

• Consumer Reports WebWatch teamed with Consumer Reports Travel Letter (which ceased publication in December 2002) to extensively test travel Web sites providing domestic airfares and released the results in June 2002. This report is available here.
• "Booking Hotels Online: An In-Depth Examination of Leading Hotel Web Sites," was a research report that was released in April 2003. This report is available here.
• "An Analysis of the Potential Benefits and Dangers of Booking Through a Car Rental Web Site," was a research report that was released in October 2003. This report is available here.
• "Booking and Bidding in the Blind: An In-Depth Examination of Opaque Travel Web Sites," a research report that examined alternative "opaque" travel booking Web sites, was released on December 8, 2003.

Consumer Reports WebWatch will continue to test and evaluate travel Web sites throughout 2004. A minimum of three projects will be undertaken. This will include a forthcoming examination of international airfare booking sites, slated for publication in Spring 2004.

back to top

A Brief Overview of Opaque Travel Web Sites

As Consumer Reports WebWatch has noted in the past, the Internet has remained the least expensive method for travel suppliers such as airlines, hotel chains, and car rental firms to distribute their products. Undoubtedly this has contributed to travel's ranking as the largest source of online commerce.

Many travel companies are using all the marketing tools at their disposal — including discounting prices and offering bonus loyalty-program mileage and points - to direct more consumers to the Web in an ongoing effort to continually reduce distribution costs. The latest research indicates that these efforts are paying off: Travel booked through the Internet continues to grow briskly. An estimated 30.7 million households in the U.S. will buy travel online in 2004, an 8% increase from the 28.5 mark reached in 2003, according to technology research firm Forrester Research, based in Cambridge, Mass.

It's important to note that there are critical differences in the sales and booking tools employed by online travel companies. In general, travel Web sites can be broken down into three separate and distinct categories:

• BRANDED SITES: these sites are owned by one or more travel suppliers and are basically dealerships selling a single line of products (e.g., American Airlines) or consortia of partner products (e.g., Northwest Airlines-KLM Royal Dutch Airlines).
  
  
• INTEGRATED TRANSPARENT SITES: these sites may or may not be owned by travel suppliers but they act as online travel agencies offering multiple products from competing companies at varying fares and rates (e.g., Expedia, Orbitz, Travelocity); the identities of the travel suppliers as well as the fares and rates are provided to the consumer prior to booking.
  
  
• INTEGRATED OPAQUE SITES: these sites may or may not be owned by travel suppliers but they offer multiple products from competing companies at varying fares and rates and the identities of the travel suppliers are not provided to the consumer prior to booking (e.g., Hotwire, Priceline). In addition, the consumer may be required to bid for a fare or rate prior to booking (e.g., Priceline).

Opaque travel Web sites sell excess inventory (airline seats, hotel rooms, car rental vehicles, vacation packages, and cruises) directly to consumers. They claim to offer steeper discounts on fares and rates than those offered by non-opaque travel Web sites or traditional offline methods of distribution, such as "brick-and-mortar" travel agencies, telephone reservations centers, or ticket offices.

The two largest opaque sites, Hotwire and Priceline, are not shy about their sales claims. Hotwire, for example, says it negotiates "special prices" that "can't be found anywhere else." These discounts, dubbed "Hot-Fares" and "Hot-Rates," were the basis of a national television advertising campaign launched in June 2003, which celebrated "cheap" behavior with the slogan, "You're Our Kind of Customer." In fact, Hotwire's slogan is "Fly. Sleep. Drive. Cheap."

Priceline, meanwhile, claims on its Web site that "a substantial percentage of consumers" who purchased airline tickets through Priceline have saved up to 40% off "lowest published fares" concurrently available for sale, and up to 60% for last-minute travel booked within 7 days. A key issue is whether "lowest published fares" refers only to airfares provided to reservations clearing houses and global distribution systems (GDSs) such as Sabre, or also refers to "Web-only" fares offered by transparent rivals such as Expedia, Orbitz, and Travelocity. In this study, Consumer Reports WebWatch directly addressed this question by querying fares and rates side-by-side from Hotwire and Priceline, as well as Sabre, Expedia, Orbitz, Travelocity, and the hotel-booking site Quikbook.

Hotwire and Priceline have the benefit of a sizeable built-in audience: One of every five consumers who book leisure or pleasure trips online is a "mercenary traveler" most concerned about getting the lowest fare or rate, according to data provided by Forrester. "Most people are not using an opaque site to plan the vacation of a lifetime," says Henry Harteveldt, a principal analyst for Forrester. "These sites are, however, perfectly good ways to find a quality hotel room and to stretch your travel dollars."

The Forrester report revealed that users of leading integrated travel Web sites differ on what factors are most important to them. The lowest price was more important to certain users than the travel company brand or the existence of a traveler loyalty program offering mileage or points. The breakdown of those who cited the importance of obtaining the lowest price was as follows:

• 26% of Hotwire users
• 19% of Priceline users
• 18% of Orbitz users
• 15% of Travelocity users
• 13% of Expedia users

Obviously, these differences divided the users of opaque and transparent travel Web sites. And in a direct comparison between Hotwire and Expedia, for example, there were twice as many Hotwire users who cited the lowest price factor. In some ways, these findings may indicate a rapid maturation and brand differentiation within the online travel industry, which is itself less than a decade old. Some travel Web sites may have successfully found market niches for certain types of consumers. Hotwire and Priceline would seem to be two examples.

Hotwire, based in San Francisco, was launched online in October 2000. Although Priceline is not supported directly by travel suppliers, Hotwire was founded with the financial backing of six of the nation's largest airlines (American, America West, Continental, Northwest, United, and US Airways). In July 2003, Hotwire reported a record-breaking second quarter, with $195 million in gross travel bookings (a 64% increase over the previous quarter) and net income of $2.5 million.

Priceline, based in Norwalk, Conn., came on the scene in April 1998. In November 2003, the company reported strong financial results for the third quarter, including a gross profit of $40.6 million and gross travel bookings of $275.3 million. This marked a strong turnaround from late 2000, when the company was forced to reorganize and lay off employees due to a combination of financial setbacks and customer-service problems that reached its nadir when the Better Business Bureau in its home state of Connecticut temporarily revoked its membership. At that time Priceline also scrapped ambitious plans for expansion into additional non-travel arenas, including term life insurance and cellular telephones (Priceline still maintains a personal finance division).

It's worth noting that both of the leading opaque travel Web sites have ties to non-opaque travel sites. In 2002, Priceline acquired Lowestfare, an integrated transparent travel Web site, and also operates a non-booking travel site called RentalCars.com. Hotwire is now a sister company of Expedia and Hotels.com, having been acquired in November 2003 by IAC/InterActiveCorp (USA Interactive), a rapidly expanding travel distribution network. Interestingly, at press time Hotwire still maintained a marketing partnership with Orbitz, an integrated transparent travel Web site and a direct competitor of Expedia.

Together these two Web sites dominate the opaque travel market, collectively accounting for $1.547 billion in travel bookings in the U.S. in 2002, according to PhoCusWright, a tourism and hospitality industry intelligence consulting firm based in Sherman, Conn. And the opaque market is larger than many may suspect. PhoCusWright says that Hotwire and Priceline combined for a healthy 10.4% of the total $14.85 billion spent in online leisure and unmanaged business travel agency bookings in the U.S. in 2002.

Although both Hotwire and Priceline are categorized together under the broad heading of opaque travel sites, there are crucial differences in their booking methodologies and consumer interfaces. Examining these differences was a key component of the testing that is described in this research paper. In short, Consumer Reports WebWatch sought to compare Hotwire and Priceline to each other, and to transparent integrated Web sites as well.

Generally speaking, a consumer can use either Hotwire's or Priceline's search functions to indicate the broad parameters of an airline, hotel, or car rental itinerary. Each site then sifts through available inventory for appropriate products. There the similarities end, for these sites differ drastically in defining how the term "opaque" is presented to consumers.

Hotwire discloses the price of available inventory upfront. However, prior to a non-refundable booking, Hotwire does not reveal the name of the travel supplier or certain particulars of the itinerary which would reveal the identity of the supplier (such as exact flight times or exact location of a hotel property).

Priceline, on the other hand, offers a much more complicated and time-consuming process of bidding and even counter-bidding. These procedures are summed up in the company's "Name Your Own Price" slogan. The consumer not only does not know the name of the travel supplier, but he or she also does not know many of the particulars of the itinerary, and in certain cases must be flexible about changes such as flight times, airports, length of stay, and rental vehicle types. Priceline claims to notify customers whether their bid is accepted within 3 minutes, on average, but Consumer Reports WebWatch testers found that this often took much longer. The Priceline process is examined at length on page 29.

Both Hotwire and Priceline reveal the complete travel itinerary details only after the booking is confirmed for the airline seat, hotel room, or car rental vehicle (and the consumer's charge card has been billed for a non-refundable sale). Both Web sites charge booking fees, which are detailed at length on pages 37 and 40. And consumers who book through Hotwire and Priceline forgo accruing mileage or points in traveler loyalty programs in nearly all cases.

Consumers are advised that such trade-offs are worth the savings offered by Hotwire and Priceline in comparison to other booking channels. By requiring consumers to be flexible about their travel plans, in exchange for the claim of deeply-discounted fares and rates, Hotwire and Priceline have quickly remade the face of online travel. This is one reason why Consumer Reports WebWatch had been eager to develop proper and reliable testing methodologies for opaque travel Web sites, so that such claims about discounting could be accurately and fairly examined.

For travel suppliers, opaque Web sites provide yet another sales strategy as they seek to continually lower their distribution costs. Opaque Web sites undoubtedly help travel companies unload inventory that would have remained unsold, without cannibalizing existing distribution and retail sales channels. This point is made even more compelling by another piece of Forrester data: More than 10 million U.S. households reported taking at least one unplanned, unbudgeted trip in the last year. It seems certain that many of them have turned to opaque travel Web sites to help plan those trips.

back to top

Consumer Reports WebWatch announced early in 2003 that it would launch several in-depth examinations of travel Web sites. This research paper focuses solely upon testing of opaque travel Web sites.

What follows is an in-depth presentation of the testing methodology and parameters.

back to top

SELECTION OF WEB SITES

Consumer Reports WebWatch analyzed market share data and other factors before deciding which Web sites to include in this testing. Consumer Reports WebWatch was assisted in this analysis by Harrell Associates Inc., based in New York City.

For these tests, the two largest opaque travel Web sites were selected: Priceline and Hotwire. Sabre, the largest global distribution system (GDS) in North America, also was included so that testers would receive ballpark bidding figures in real-time. In addition, the three largest non-opaque, integrated travel Web sites — Expedia, Orbitz, and Travelocity — were included as benchmarks for the opaque fares and rates. Finally, a discount hotel-booking site, Quikbook, was included as another benchmark for hotel tests only.

Here is further information about the six travel Web sites and the one GDS:

• Expedia (www.expedia.com) is based in Bellevue, Wash. and is owned by IAC/InterActiveCorp (USA Interactive). Expedia was formerly owned by Microsoft. Expedia is a sister company of Hotels.com and Hotwire.
  
  
• Hotwire (www.hotwire.com) is based in San Francisco. In November 2003, after this testing was completed, Hotwire was acquired by IAC/InterActiveCorp (USA Interactive). Hotwire was formerly backed by Texas Pacific Group and six major U.S. airlines: America West Airlines, American Airlines, Continental Airlines, Northwest Airlines, United Airlines, and US Airways. Hotwire is a sister company of Expedia and Hotels.com. In addition, Hotwire is a marketing partner of Orbitz.
  
  
• Orbitz (www.orbitz.com) is based in Chicago and is owned by the nation's five largest airlines: American Airlines, Continental Airlines, Delta Air Lines, Northwest Airlines, and United Airlines. Orbitz has no travel-related sister companies. Orbitz is a marketing partner of Hotwire.
  
  
• Priceline (www.priceline.com) is based in Norwalk, Conn., and its stock is publicly traded on Nasdaq (PCLN). Priceline is a sister company of Lowestfare.com and RentalCars.com. In addition, Priceline.com is part-owner of Travelweb LLC.
  
  
• Quikbook (www.quikbook.com) is based in New York City and is privately held. Quikbook has no travel-related sister companies.
  
  
• Sabre Travel Network is based in Southlake, Texas and is owned by Sabre Holdings. Sabre is the largest global distribution system (GDS) used by travel agencies in North America, and claims its market share is nearly 50%. Sabre was formerly owned by AMR, the parent company of American Airlines. Sabre is a sister company of Travelocity.
  
  
• Travelocity (www.travelocity.com) is based in Fort Worth, Texas and is owned by Sabre Holdings. Travelocity was formerly owned by AMR, the parent company of American Airlines. Travelocity is a sister company of Sabre Travel Network.

An important note: Consumer Reports WebWatch and Consumer Reports Travel Letter have found that the results provided by Sabre and the results provided by its sister company, Travelocity, are rarely consistent, even though Travelocity is "powered" by Sabre. In recent tests, Travelocity has repeatedly provided lower rates than Sabre. In addition, we have found that the results provided by other travel Web sites "powered" by Sabre (or the other GDS used in our testing) have rarely been consistent as well.

back to top

METHODOLOGY

All testing conducted by Consumer Reports WebWatch was performed by trained individuals upon completion of dry-run testing. Statistical analysis provided by Consumers Union led to the creation of eight separate tests, grouped into five separate testing days. Testing times were varied throughout the course of several weeks in late August and early September 2003.

All testing was scheduled in advance and completed simultaneously in real-time. However, the nature of this testing required minor adjustments to the standard Consumer Reports WebWatch testing procedures. First, fares and rates were queried in Sabre and immediately forwarded to the testers, and then bids were placed in Priceline, while simultaneously fares and rates were queried in the remaining Web sites.

Two factors differentiated this project from previous travel Web site testing projects undertaken by Consumer Reports WebWatch and Consumer Reports Travel Letter.

First, these tests were not confined to a single sector of the travel industry, but instead all three major categories were included: airlines, hotels, and car rentals.

Second, for the first time, actual travel products were purchased. This was done in Hotwire and Priceline only, so that the testers could determine the specifics of the itineraries being booked. (In TEST #5 only, the tester queried fares and rates from Hotwire but did not book, since it was determined which travel companies would be among those provided for the specific itineraries requested.)

During the course of this project, Consumer Reports WebWatch spent
approximately $38,000 for airline seats, hotel rooms, and rental cars. Most of these purchases — totaling $36,671 — were donated to the Make-A-Wish Foundation of America, a Phoenix-based non-profit organization that grants special trips and other gifts to children with life-threatening medical conditions. A small percentage of the products purchased in the tests were bought by Consumers Union employees and consultants.

An important note: For the first time in history of its testing of online travel sites, Consumer Reports WebWatch invalidated an entire test because it was determined that the results were not statistically valid, when most of the tested Web sites were unable to provide rates for the specific itinerary requested. This occurred with TEST #1, a set of 15 trials for hotel rates, totaling 105 queries. These results were not tabulated with the other test results. However, these results are included and discussed in Appendix II, for informational purposes.

Therefore, this project consisted of 120 separate trials, for a total of 840 queries across all six Web sites and Sabre. The breakdown was as follows:

• AIRLINES (270 queries)
• HOTELS (210 queries)
• CAR RENTALS (360 queries)

back to top

TESTING PARAMETERS AND SPECIFIC TESTING CRITERIA

Each test consisted of searching for specific fares and rates in U.S. dollars throughout the mainland U.S., Alaska, Hawaii, and Puerto Rico. What follows are detailed breakdowns for each of the three travel products: airlines, hotels, and car rentals.

AIRLINES
Each test consisted of searching for airfares in U.S. dollars on 15 high-volume domestic routes between major airports in cities throughout the mainland U.S. and Hawaii. Consumer Reports WebWatch examined passenger traffic and airfare data provided by the U.S. Department of Transportation in selecting these routes. Most of these routes offered nonstop service, multiple carriers, and low fare service. Furthermore, these routes represented a mix of both leisure and business trips.

For TEST #1, the routes were:

1) Boston - Tampa
2) New York - Orlando
3) Baltimore - Phoenix
4) Pittsburgh - New Orleans
5) Atlanta - San Diego
6) Miami - St. Louis
7) Cleveland - Orlando
8) Cincinnati - Miami
9) Detroit - Seattle
10) Chicago - Los Angeles
11) Minneapolis-St. Paul - Las Vegas
12) Denver - Honolulu
13) San Francisco - Baltimore
14) Los Angeles - Atlanta
15) San Diego - Houston

For TEST #4 and TEST #5 (identical routes but different booking windows), the routes were:

1) New York - Orlando
2) New York - Fort Lauderdale
3) New York - Tampa
4) New York - West Palm Beach
5) Atlanta - Orlando
6) Atlanta - Tampa
7) Atlanta - Miami
8) Atlanta - Las Vegas
9) Dallas - Houston
10) Dallas - Las Vegas
11) Dallas - New Orleans
12) Dallas - San Diego
13) Denver - Minneapolis-St. Paul
14) Denver - Las Vegas
15) Denver - Seattle

• Each test consisted of searching for airfares. Booking criteria were established in advance.

For TEST #1, these criteria included:

• One adult fare
• Economy or coach class
• Round-trip
• Any local airport
• Non-stop flight
• Single-airline itineraries
• Flight times at any time of day are acceptable for both departure and arrival
• No special or corporate rate programs (government, military, AARP, AAA, etc.)
• No frequent flyer program membership

Note that not all Web sites allowed such specificity for each test, but these parameters were established in advance to ensure consistency.

For TEST #4 and TEST #5, these criteria included:

• One adult fare
• Economy or coach class
• Round-trip
• Any of the listed airports
• Connecting flights are acceptable
• Multiple-airline itineraries are acceptable
• Flight times at any time of day are acceptable for both departure and arrival
• No special or corporate rate programs (government, military, AARP, AAA, etc.)
• No frequent flyer program membership

NOTE: Again, not all Web sites allowed such specificity for each test, but these parameters were established in advance to ensure consistency.

• In order to simulate a variety of trips, the advance booking windows varied. The booking times were:

TEST #1 (SHORT BOOKING): 5 days in advance
TEST #4 (LONG ADVANCE BOOKING): 28 days in advance
TEST #5 (VERY SHORT BOOKING): 1 day in advance

• In a further effort to simulate a variety of trips, the length of the stays varied as well. The stays were:

TEST #3: 2 days
TEST #4: 4 days
TEST #5: 2 days

back to top

TESTING PARAMETERS AND SPECIFIC TESTING CRITERIA

HOTELS
This test consisted of searching for hotel rates in U.S. dollars in 30 major cities throughout the mainland U.S., Alaska, Hawaii, and Puerto Rico. These cities represented a mix of both leisure and business destinations.

For TEST #2, the cities were:

1) Boston/Logan International Airport (BOS)
2) New York/LaGuardia Airport (LGA)
3) Philadelphia/International Airport (PHL)
4) Pittsburgh/International Airport (PIT)
5) Baltimore/Baltimore-Washington International Airport (BWI)
6) Washington, D.C./Dulles International Airport (IAD)
7) Charlotte/Douglas International Airport (CLT)
8) Atlanta/Hartsfield International Airport (ATL)
9) Orlando/International Airport (MCO)
10) Tampa/International Airport (TPA)
11) Miami/International Airport (MIA)
12) San Juan/Luis Munoz Marin International Airport (SJU)
13) Cleveland/Hopkins International Airport (CLE)
14) Cincinnati/International Airport (CVG)
15) Minneapolis/Minneapolis-St. Paul International Airport (MSP)
16) Chicago/O'Hare International Airport (ORD)
17) St. Louis/Lambert International Airport (STL)
18) New Orleans/International Airport (MSY)
19) Dallas/Dallas-Fort Worth International Airport (DFW)
20) Houston/George Bush Intercontinental Airport (IAH)
21) Denver/International Airport (DEN)
22) Salt Lake City/International Airport (SLC)
23) Phoenix/Sky Harbor Airport (PHX)
24) Las Vegas/McCarron International Airport (LAS)
25) Anchorage/International Airport (ANC)
26) Seattle/Seattle-Tacoma International Airport (SEA)
27) San Francisco/International Airport (SFO)
28) Los Angeles/International Airport (LAX)
29) San Diego/Lindbergh International Airport (SAN)
30) Honolulu/International Airport (HNL)

• Each test consisted of searching for the same room type: a three-star property (or equivalent) near a major airport. Note that not all Web sites allowed such specificity for each test, but these parameters were established in advance to ensure consistency.

• Booking criteria were established in advance. These criteria included:

• Two adults; no children
• One queen-sized room
• Non-smoking
• No special or corporate rate programs (government, military, AARP, AAA, etc.)
• No frequent guest program membership

NOTE: Again, not all Web sites allowed such specificity for each test, but these parameters were established in advance to ensure consistency.

• The booking times were:

 

TEST #2 (ADVANCE BOOKING): 14 days in advance

• The length of the hotel stay was:

TEST #2: 2 nights

back to top

TESTING PARAMETERS AND SPECIFIC TESTING CRITERIA

CAR RENTALS
Each test consisted of searching for car rental rates in U.S. dollars at either 30 major airports or 15 major airports in cities throughout the mainland U.S., Alaska, Hawaii, and Puerto Rico. These airports represented a mix of both leisure and business destinations.

For TEST #3, the airports were:

1) Boston/Logan International Airport (BOS)
2) New York/LaGuardia Airport (LGA)
3) Newark/Liberty International Airport (EWR)
4) Philadelphia/International Airport (PHL)
5) Washington, D.C./Dulles International Airport (IAD)
6) Washington/Reagan Washington National Airport (DCA)
7) Atlanta/Hartsfield International Airport (ATL)
8) Orlando/International Airport (MCO)
9) Tampa/International Airport (TPA)
10) West Palm Beach/Palm Beach International Airport (PBI)
11) Fort Lauderdale/Fort Lauderdale-Hollywood Airport (FLL)
12) Miami/International Airport (MIA)
13) San Juan/Luis Munoz Marin International Airport (SJU)
14) Cleveland/Hopkins International Airport (CLE)
15) Chicago/O'Hare International Airport (ORD)
16) St. Louis/Lambert International Airport (STL)
17) New Orleans/International Airport (MSY)
18) Dallas/Dallas-Fort Worth International Airport (DFW)
19) Houston/George Bush Intercontinental Airport (IAH)
20) Denver/International Airport (DEN)
21) Salt Lake City/International Airport (SLC)
22) Phoenix/Sky Harbor Airport (PHX)
23) Las Vegas/McCarron International Airport (LAS)
24) Anchorage/International Airport (ANC)
25) Seattle/Seattle-Tacoma International Airport (SEA)
26) San Francisco/International Airport (SFO)
27) Los Angeles/International Airport (LAX)
28) San Diego/Lindbergh International Airport (SAN)
29) Honolulu/International Airport (HNL)
30) Maui/Kahului Airport (OGG)

For TEST #4 and TEST #5 (identical airports but different booking windows), the airports were:

1) Newark/Liberty International Airport (EWR)
2) Atlanta/Hartsfield International Airport (ATL)
3) Orlando/International Airport (MCO)
4) Tampa/International Airport (TPA)
5) West Palm Beach/Palm Beach International Airport (PBI)
6) Fort Lauderdale/Fort Lauderdale-Hollywood Airport (FLL)
7) Miami/International Airport (MIA)
8) New Orleans/International Airport (MSY)
9) Denver/International Airport (DEN)
10) Phoenix/Sky Harbor Airport (PHX)
11) Las Vegas/McCarron International Airport (LAS)
12) San Francisco/International Airport (SFO)
13) Los Angeles/International Airport (LAX)
14) San Diego/Lindbergh International Airport (SAN)
15) Honolulu/International Airport (HNL)

• Rates were accepted for car rental companies located at these airports. In most cases, the companies provided "on-airport" service but in some cases they provided "off-airport" service. However, rates from other locations were not accepted.

• Each test consisted of searching for the same vehicle type: a midsize/intermediate car. The vehicle specifics (number of doors, air conditioning, automatic transmission, etc.) were established in advance. No optional extras (child seats, ski racks, cell phones, etc.) were requested.

Note that not all Web sites allowed such specificity for each test, but these parameters were established in advance to ensure consistency.

• Booking criteria were established in advance. These criteria included:

• Picking up from and returning to the same location
• No second driver
• No underage driver
• No special or corporate rate programs (government, military, AARP, AAA, etc.)
• No frequent renter program membership

NOTE: Again, not all Web sites allowed such specificity for each test, but these parameters were established in advance to ensure consistency.

• In order to simulate a variety of trips, the advance booking windows varied. The booking times were:

TEST #3 (SHORT BOOKING): 2 days in advance
TEST #4 (LONG ADVANCE BOOKING): 29 days in advance
TEST #5 (VERY SHORT BOOKING): 1 day in advance

• In a further effort to simulate a variety of trips, the length of the rental periods varied as well. The rental periods were:

TEST #3: 3 days
TEST #4: 2 days
TEST #5: 1 day

back to top

BOOKING PROCEDURES

As noted, Consumer Reports WebWatch purchased nearly $38,000 of airline seats, hotel rooms, and rental cars through Hotwire and Priceline. Since these and other travel Web sites employ security safeguards that prevent suspicious and/or multiple travel bookings, this meant using a variety of booker identities, traveler identities, charge card accounts, and computer terminals.

Consumers Union employees obtained charge cards and E-mail accounts specifically for this testing. Since anonymity is a crucial component of all Consumer Reports WebWatch testing, Hotwire and Priceline were not notified about these booking procedures. However, despite our best efforts, on several occasions throughout the testing period the repetitive bookings triggered security alerts from both the charge card vendors and the Web sites themselves. When this happened, booking continued using other identities.

During the testing, Consumers Union pro-actively notified the charge card vendors and asked that any relevant authorities be notified as well. However, it was not until the testing was completed that Consumers Union notified Hotwire and Priceline, in an effort to avoid a security-based investigation.

back to top

SELECTION OF RESULTS

As noted in previous reports, the technological tools and viewer interfaces employed by travel Web sites have greatly evolved in recent years, so Consumer Reports WebWatch broadened its testing methodology. Specifically, Consumer Reports WebWatch no longer analyzes lowest rates and fares based upon the first rate or fare returned, a method that was previously employed by Consumer Reports Travel Letter and Consumer Reports WebWatch. This had been done because at one time all integrated travel Web site screens resembled the vertical presentations pioneered by global distribution systems (GDSs). Because these screens have evolved, the testing methodology evolved as well.

This project proved to be the most challenging for Consumer Reports WebWatch in many ways. One of the primary challenges was developing "apples-to-apples" methodologies for comparing results. The Web sites tested employed a variety of methods to display their fares and rates and we strove to be fair and consistent in comparing these prices.

For Expedia, the lowest fares and rates were culled from either the first five returns or the first full page of returns, whichever was greater. But it's important to note that when querying rates for car rentals, the lowest price was not always listed first. Expedia employed a two-tier system of displaying car rental rates, and the lowest prices were sometimes "buried" in the second set of integrated rates. This issue was discussed at length in the Consumer Reports WebWatch research report on car rental rates published in October 2003. Note that once again Consumer Reports WebWatch tabulated rates from the first displays only, since competing Web sites did not require a two-step process.

Both Orbitz and Travelocity employed "matrix" displays, which offered a combination of vertical and horizontal interfaces with multiple fares and rates, and multiple airlines, hotel properties, and car rental companies presented on a single screen. For these results, the lowest fare or rate was selected within the appropriate category from among all the vendors presented within the matrix.

QuikBook employed a display method that was unlike the methods used by any of the other Web sites to display hotel rates. Quikbook did not allow searching by hotel rating or specific location, and instead required searching by price range. The tester queried rates within a very broad range and then later, in a very time-consuming process, culled the lowest rates displayed by cross-checking the parameters of the specific itinerary.

For Hotwire, the lowest fare or rate was listed in a traditional horizontal format and no selection was necessary.

Illustration A: Example of Hotwire Hotel Search
(click on image below to view full screen image)

For Priceline, if the lowest fare or rate was provided through the bidding process, then no selection was necessary.
However, in some cases, Priceline stated it could not provide a fare or rate for a given itinerary, and displayed fares or rates provided by its sister company, Lowestfare. These fares and rates were listed in a traditional horizontal format. In these cases, the lowest fares and rates were culled from either the first five returns or the first full page of returns, whichever was greater.

Illustration B: Example of Priceline Airfare Search
(click on image below to view full screen image)

Note that all rankings included ties. Therefore it was theoretically possible that every Web site and Sabre could have provided the lowest rate for every available query.

back to top

CURRENCY ISSUES

All rates were provided in U.S. dollars. None of the rates included federal, state, local, airport, lodging, or corporate taxes, fees, or surcharges. The Web site's own booking fees were not included as well. These fees are discussed here.

back to top

ROUNDING OF FARES AND RATES

Consumer Reports WebWatch's rounding of amounts for this project was consistent with its methodology for past projects. That is, amounts were rounded off to the nearest dollar for all airfares and hotel rates. Because car rental rates are so intensely competitive, actual rates were used in all cases in these tests, and no amounts were rounded to the nearest dollar.

back to top

AVAILABILITY OF RATES

As stated, Consumer Reports WebWatch booked and purchased airline seats, hotel rooms, and rental cars in Hotwire and Priceline only. Among the remaining Web sites, Consumer Reports WebWatch did not book any of the fares and rates provided. In all cases, the Web sites stated that the airline seats, hotel rooms, and vehicles requested were available. When the fare or rate provided was not available, the next lowest fare or rate that was available was used.

back to top

VALID TESTS

All Web sites were evaluated solely on valid tests. If there was an error on the part of Consumer Reports WebWatch, this was deemed an invalid test. Invalid tests were eliminated and did not affect final rankings. These errors included incorrect data entries, insufficient returns of data, and printing errors. Overall, there were only two errors committed by Consumer Reports WebWatch testers for all 840 queries. This translated into a completion percentage of 99.8%.

back to top

Through no fault of the Consumer Reports WebWatch testers, all six Web sites failed to provide valid data on some tests. Sabre provided valid data for all 150 queries.

In some cases, these failures affected the Web site's final rankings. These could have been due to a variety of factors, including:

• Results outside the specific parameters requested;
• Airline flight, hotel room, or vehicle type not available for the specific itinerary requested;
• Airline flight, hotel room, or vehicle type sold out on those dates;
• Specific airport or location not served by that Web site
• Technical or system failures.

back to top

TECHNICAL OR SYSTEM FAILURES

Although our testers experienced several technical or system failures in which a Web site was unable to process a request, for the most part these incidents were temporary and did not prevent the tester from completing the query. However, in two cases, a Web site was unable to process the request in time due to an apparent technical failure.

Both of these incidents occurred with Hotwire, in back-to-back queries for airline fares in TEST #4, for Boston's Logan International Airport and New York's LaGuardia Airport. Hotwire resumed processing requests shortly thereafter and there were no other incidents throughout the remainder of the testing.

back to top

Testing Results, Rankings, and Conclusions

The findings of Consumer Reports WebWatch's opaque travel Web site testing are presented in the attached figures and are described below. There are four major categories of rankings:

1. Lowest rates for airline tests
2. Lowest rates for hotel tests
3. Lowest rates for car rental tests
4. Lowest rates for all tests

The results for each of the three travel sectors — airlines, hotels, and car rentals — varied, in some cases significantly. It's instructive to closely analyze those results, both individually and collectively.

It's important to note that all rankings included ties. Therefore it was theoretically possible for all six Web sites and Sabre to have provided the lowest rate in response to every single query.

back to top

LOWEST FARES FOR AIRLINES

Airline bookings comprise the largest slice of the online travel pie, so it's fair to conclude these tests were the most important conducted by Consumer Reports WebWatch.

Priceline led the five other travel Web sites and Sabre in providing lowest airline rates. In 45 trials, Priceline provided the lowest fare 44% of the time. Although this was the best percentage posted by any of the travel Web sites during the airline testing, it still means Priceline failed to provide the lowest airlines rates more than half the time.

Hotwire was not adept at providing lowest airline rates, ranking fourth overall, behind not only Priceline but Expedia and Travelocity as well. Hotwire provided lowest rates 18% of the time when searching for airfares, fewer than 1 of every 5 tries. This means that consumers whose primary criterion is lowest fares should not book airfares on Hotwire until they have searched on other travel Web sites.

Hotwire and Priceline did offer similar savings over non-opaque Web sites in most cases. For each query in which one or both of the opaque Web sites provided the lowest fare, Consumer WebWach calculated the percentage of savings Hotwire and Priceline offered compared to the lowest rate provided by a non-opaque site for the same query. For airline fares, Priceline averaged savings of 21% and Hotwire averaged savings of 20%. But Hotwire's average soared to 70% with the inclusion of a single itinerary that offered savings of $1,012 on a San Francisco — Baltimore flight. Without that flight, the savings offered by both Hotwire and Priceline were very close.

Consumer Reports WebWatch also calculated the percentage of Priceline's lowest rates that required a change of itinerary of some kind. This included changing flight days or times or airports. For airlines, Priceline required such a change 40% of the time, the second-highest percentage after car rentals.

Expedia and Travelocity performed very well when it came to searching for airline rates, probably the most critical component of a travel Web site. Expedia provided lowest airfares in 36% of the trials, behind Priceline's 44% and ahead of Travelocity's 27%. Consumers eager to find online airline bargains but