Global Concerns:

An In-Depth Examination of Travel Web Sites Selling International Airline Tickets (Full Report)

TABLE OF CONTENTS

Executive Summary

Consumer Reports WebWatch has noted for some time now that travel remains the largest sector of Internet commerce. Even as it retains that position, the online travel industry continues to make surprising gains. Forrester Research of Cambridge, Mass., estimates nearly half (46%) of all leisure travelers now book at least part of their vacations online.

The most important component of most travel itineraries is, of course, the purchase of airline seats. Separately and together, Consumer Reports WebWatch and the now-defunct Consumer Reports Travel Letter have repeatedly analyzed a wide variety of online airline products, but in all instances this testing focused on searching for domestic airfares. This project expands that focus to the international airline sector for the first time.

According to the Bureau of Transportation Statistics of the U.S. Department of Transportation, 23.6 million airline passengers flew to a foreign destination from the U.S. in 2002 (while 25.0 million passengers flew here from other countries). Not surprisingly, these numbers were down from pre-Sept. 11, 2001 figures, but that did not reduce—and in fact may have increased—efforts by online travel sellers and the airlines themselves to spur more consumers to buy international airline tickets via the Internet.

For the first time, Consumer Reports WebWatch conducted testing of airline Web sites while searching for flight and fare data on international rather than domestic routes. This was an effort to determine how certain key factors affect the international market, including the following:

• base ticket prices are usually higher on average for international routes than for domestic routes
• the international tax and fee structure is generally more complex;
• airline “code-sharing” and marketing partnerships are more common among international carriers;
• currency conversions may affect total pricing, that is, inclusive of all taxes, fees and surcharges.

Consumer Reports WebWatch wanted to determine if competition was more or less intense in the international market. The conclusion of this test is that online competition within the international airfare arena is indeed robust. Despite relatively high ticket prices, many of the sites tested closely or exactly matched their competitors’ fares repeatedly throughout.

This project involved several months of planning, testing, and analyzing results. Consumer Reports WebWatch selected the three leading travel Web sites that offer integrated listings of competing carriers: Expedia, Orbitz, and Travelocity. These three sites were included in all 150 trials. In addition, “branded” Web sites maintained by individual airlines were also tested, though no single site was included in all 150 trials. They included sites maintained by five U.S. carriers (American, Continental, Delta, Northwest/KLM, and United) and eight foreign carriers (Air France, Air Jamaica, British Airways, Korean Air, Lufthansa, Singapore, Thai Airways, and Virgin Atlantic).

A close analysis of the results determined that Orbitz, one of the three major integrated sites, is the source of the best and the worst news for consumers. Orbitz was so adept at providing lowest fares that it posted some of the most impressive statistics ever recorded by Consumer Reports WebWatch. However, testers found the Orbitz fare display tools to be inaccurate and cumbersome. The Orbitz site chronically presented listings for flights that were unavailable and returned fares that changed during the shopping process, most often increasing.

Among the key findings:

• The Orbitz flight selection “Matrix™,” while in theory an extremely consumer-friendly comparison shopping tool, failed repeatedly. In 150 trials, the lowest posted fares increased in price 21% of the time and decreased in price 5% of the time. In addition, the lowest-priced flights were not available 19% of the time and could not be booked online an additional 5% of the time. In many cases, these problems occurred multiple times within a single query and required extensive cross-checking of posted listings.
  
  
• Expedia and Travelocity also displayed inaccurate pricing and booking information, though neither site’s displays returned inaccurate results as often as the Orbitz Matrix™. With Expedia, lowest fares increased 5% of the time and decreased 2% of the time and lowest-priced flights were not available for booking 2% of the time. With Travelocity, lowest fares increased 1% of the time and decreased 1% of the time and lowest-priced flights were not available for booking 1% of the time. There were no such problems with any of the branded airline sites.
  
  
• The “fare-jumping” problem, when a site displays a low fare that, in fact, cannot be booked, reached a nadir with Orbitz, when the posted price of a round-trip flight between San Francisco and Brussels suddenly increased by $1,098.
  
  
• To their credit, all three integrated travel Web sites now post printed warnings to users when fares suddenly increase or decrease mid-booking.
  
  
• Tax and fee information was unclear on some sites and Consumer Reports WebWatch discarded results from two additional airline sites (EVA Airways and Japan Airlines) because viable and accurate pricing comparisons were not possible. Also, the total fare mechanism repeatedly malfunctioned on the Continental Airlines site, when taxes and fees were posted incorrectly 43% of the time. (Continental later told Consumer Reports WebWatch that this problem was corrected.)
  
  
• Despite the problems discussed, Orbitz comfortably led all 16 sites by providing the highest percentage of lowest fares (30%), ahead of Northwest/KLM (27%), Expedia (23%), and Travelocity (19%).
  
  
• When the results were adjusted to include not just lowest fares but also “closest” fares (within $10 of the lowest fare), Orbitz dominated all competitors by providing the lowest or closest fare in four out of five cases. Orbitz led with 79%, followed by Expedia with 55% and Travelocity with 35%.
  
  
• Individually, all three integrated travel Web sites—Orbitz, Expedia, and Travelocity—consistently provided a higher number and percentage of lowest fares than the branded airline sites did collectively. It’s important to note that this was accomplished despite all three of these integrated sites charging booking fees in most cases.
  
  
• Three major U.S. airline sites—Northwest/KLM, United, and American—performed well at providing lowest fares. Some foreign airline sites also performed well, albeit with much smaller test samples.
  
  
• When individual results were tabulated, in one test American led all sites when searching for fares on European routes and in another test United led all sites when searching for fares on Asian routes.
  
  
• As for functionality and ease-of-use criteria, some of the branded airline sites repeatedly returned technical “glitches” when searching for fares. The three integrated sites performed without any serious problems.
  
  
• One final observation: Two of these Web sites—Orbitz and Travelocity—always presented integrated displays with the dollar amounts rounded down, regardless of the cent value.
  

The project was directed by William J. McGee, a travel journalist and consultant to Consumer Reports WebWatch and the Editor of Consumer Reports Travel Letter from 2000 to 2002. The project was completely funded by Consumer Reports WebWatch. Employees of Consumer Reports WebWatch assisted in drafting the methodology, participated in the testing, and contributed to this research report. The research report was edited by Beau Brendler, the director of Consumer Reports WebWatch.

back to top

• Consumer Reports WebWatch teamed with Consumer Reports Travel Letter (which ceased publication in December 2002) to extensively test travel Web sites providing domestic airfares and released the results in June 2002. This report is available at ConsumerReports.org.
  
  
• “Booking Hotels Online: An In-Depth Examination of Leading Hotel Web Sites,” was a research report that was released in April 2003. This report is available at http://www.consumerwebwatch.org/dynamic/
  travel-report-booking-hotels-online-abstract.cfm.
  
  
• “An Analysis of the Potential Benefits and Dangers of Booking Through a Car Rental Web Site,” was a research report that was released in October 2003. This report is available at http://www.consumerwebwatch.org/dynamic/
  travel-report-renting-cars-online-abstract.cfm.
  
  
• “Booking and Bidding Site Unseen: A Consumer’s Guide to Opaque Travel Web Sites,” was a research report that examined alternative “opaque” travel booking Web sites and was released in December, 2003. This report is available at http://www.consumerwebwatch.org/dynamic/
  travel-report-booking-bidding-abstract.cfm.

Consumer Reports WebWatch will continue to test and evaluate travel Web sites throughout 2004 and 2005. Forthcoming projects include an international study of airline ticket-booking sites in conjunction with consumer organizations in the Netherlands, the United Kingdom, France, Germany, Belgium and Denmark, set for publication in Fall 2004. In addition, an examination of booking sites for premium (first-class and business-class) airline seats is scheduled for publication early in 2005.

back to top

Testing Methodology and Parameters

As part of its ongoing commitment to examine all aspects of online travel, Consumer Reports WebWatch recognized that a clear need had arisen to expand past testing parameters. Previously, testing of airline booking sites had focused exclusively on domestic airfares. However, a combination of factors indicated that the online display of international airfares might present issues unique to this sector of the travel industry.

What follows is an in-depth presentation of the testing methodology and parameters.

back to top

SELECTION OF WEB SITES

Consumer Reports WebWatch analyzed market share data and other factors before deciding which Web sites to include in this testing. Included were the three largest integrated travel Web sites: Expedia, Orbitz, and Travelocity.

In addition, “branded” booking sites maintained by individual airlines were included as well. These included four sites owned by domestic carriers, eight sites owned by foreign carriers, and one site jointly owned by both a domestic carrier and a foreign carrier. Thus the five largest U.S. airlines were included. As for the foreign airlines, they were selected after the routes were selected (see below). This was to ensure that the testing would include dominant carriers on these international routes, which were selected due to passenger traffic volume as well as geographical balance.

What follows is a complete listing of these travel Web sites.

1. Integrated Travel Web Sites:
   
   1. Expedia (www.expedia.com) is based in Bellevue, Wash., and is owned by IAC/InterActiveCorp (USA Interactive). Expedia was formerly owned by Microsoft. Expedia is a sister company of Hotels.com and Hotwire.
      
      
   2. Orbitz (http://www.orbitz.com/) is based in Chicago and is owned by the nation’s five largest airlines: American Airlines, Continental Airlines, Delta Air Lines, Northwest Airlines, and United Airlines. Orbitz has no travel-related sister companies.
      
      
   3. Travelocity (www.travelocity.com) is based in Fort Worth, Texas and is owned by Sabre Holdings. Travelocity was formerly owned by AMR, the parent company of American Airlines. Travelocity is a sister company of Sabre Travel Network, the global distribution system (GDS) used by travel agencies.
      
      
2. Domestic Branded Airline Web Sites:
   
   1. American Airlines (www.aa.com), is based in Fort Worth, Texas.
      
      
   2. Continental Airlines (www.continental.com), is based in Houston.
      
      
   3. Delta Air Lines (www.delta.com), is based in Atlanta.
      
      
   4. United Airlines (www.united.com), is based in Chicago.
      
      
3. Foreign Branded Airline Web Sites:
   
   
   1. Air France (www.airfrance.us), is based in France.
      
      
   2. Air Jamaica (www.airjamaica.com), is based in Jamaica.
      
      
   3. British Airways (www.britishairways.com), is based in Great Britain.
      
      
   4. Korean Air (www.koreanair.com), is based in South Korea.
      
      
   5. Lufthansa German Airlines (http://cms.lufthansa.com/fly/us/en/homepage), is based in Germany.
      
      
   6. Singapore Airlines (www.singaporeair.com), is based in Singapore.
      
      
   7. Thai Airways International (www.thaiair.com), is based in Thailand.
      
      
   8. Virgin Atlantic Airways (www.virgin-atlantic.com), is based in Great Britain.
      
      
4. Joint Domestic/Foreign Branded Airline Web Sites:
   
   1. Northwest Airlines/KLM Royal Dutch Airlines (www.nwa.com), are based in St. Paul, Minn. and the Netherlands, respectively.

It should be noted that this was the first Consumer Reports WebWatch testing project that included Orbitz, as well as the Web sites of all five of its airline owners.

In addition, Consumer Reports WebWatch included two other foreign airline Web sites in this testing, but results were discarded because total prices were not labeled clearly, making head-to-head comparisons of airfares impossible:

• EVA Airways (www.evaair.com), is based in Taiwan.
  
  
• Japan Airlines (www.japanair.com), is based in Japan.

back to top

All testing conducted by Consumer Reports WebWatch was performed by trained individuals upon completion of dry-run testing. Statistical analysis provided by Consumers Union led to the creation of five separate tests, grouped into four separate testing days. Testing times were varied throughout the course of two weeks in mid-December 2003.

All testing was scheduled in advance and completed simultaneously in real-time. All testers queried airfares from previously distributed itineraries. In all cases, airfares were available for booking, but in no cases were airline seats purchased.

In total, this project consisted of 150 separate trials, for a total of 911 queries across all 16 Web sites. Of these queries, 903 were deemed valid, after eight were discarded due to errors on the part of Consumer Reports WebWatch.

Expedia, Orbitz, and Travelocity were included in all 150 trials, but none of the branded airline sites were included in all 150 trials. In addition, an airline site was not included in any trial if that airline did not serve the route in question.

back to top

TESTING PARAMETERS AND SPECIFIC TESTING CRITERIA

Each test consisted of searching for specific airfares in U.S. dollars on high-volume international routes. All flights originated from airports within the United States. The routes included destinations in the Caribbean, Latin America, Europe, and Asia.

Consumer Reports WebWatch examined passenger traffic and airfare data provided by the Bureau of Transportation Statistics of the U.S. Department of Transportation in selecting these routes. All of the foreign countries selected were among the busiest destinations for air travelers departing the United States in 2002, the most recent year that annual statistics were available at time of testing.

As for the cities of departure, care was taken to ensure that the origins included in this testing were not only high-volume markets and geographically diverse, but that they also represented a fair sampling of domestic airline gateways. For example, each departure city (New York/Newark, Chicago, Atlanta, etc.) was a major hub and/or international gateway for at least one of the U.S. carriers being tested, as well as one or more of the foreign carriers. Conversely, each U.S. carrier was represented in each of the five tests by at least one of its hubs and/or international gateways as a point of origin.

Many but not all of these routes offered nonstop service and all offered competition in the form of service provided by multiple carriers. Due to the nature of international airline travel, the “mix” of leisure and business routes was less critical than it was for domestic airfare testing.

• For TEST #1, the routes were:

1) New York/Newark—London
2) New York/Newark—Paris
3) New York/Newark—Amsterdam
4) New York/Newark—Frankfurt
5) New York/Newark—Brussels
6) Chicago—London
7) Chicago—Paris
8) Chicago—Amsterdam
9) Chicago—Frankfurt
10) Chicago—Brussels
11) Atlanta—London
12) Atlanta—Paris
13) Atlanta—Amsterdam
14) Atlanta—Frankfurt
15) Atlanta—Brussels
16) Dallas/Ft. Worth—London
17) Dallas/Ft. Worth—Paris
18) Dallas/Ft. Worth—Amsterdam
19) Dallas/Ft. Worth—Frankfurt
20) Dallas/Ft. Worth—Brussels
21) San Francisco/Oakland—London
22) San Francisco/Oakland—Paris
23) San Francisco/Oakland—Amsterdam
24) San Francisco/Oakland—Frankfurt
25) San Francisco/Oakland—Brussels
26) Los Angeles—London
27) Los Angeles—Paris
28) Los Angeles—Amsterdam
29) Los Angeles—Frankfurt
30) Los Angeles—Brussels

• For TEST #2, the routes were:

1) New York/Newark—London
2) New York/Newark—Paris
3) New York/Newark—Amsterdam
4) New York/Newark—Frankfurt
5) New York/Newark—Brussels
6) Chicago—London
7) Chicago—Paris
8) Chicago—Amsterdam
9) Chicago—Frankfurt
10) Chicago—Brussels
11) Atlanta—London
12) Atlanta—Paris
13) Atlanta—Amsterdam
14) Atlanta—Frankfurt
15) Atlanta—Brussels
16) Dallas/Ft. Worth—London
17) Dallas/Ft. Worth—Paris
18) Dallas/Ft. Worth—Amsterdam
19) Dallas/Ft. Worth—Frankfurt
20) Dallas/Ft. Worth—Brussels
21) Minneapolis/St. Paul—London
22) Minneapolis/St. Paul—Paris
23) Minneapolis/St. Paul—Amsterdam
24) Minneapolis/St. Paul—Frankfurt
25) Minneapolis/St. Paul—Brussels
26) Los Angeles—London
27) Los Angeles—Paris
28) Los Angeles—Amsterdam
29) Los Angeles—Frankfurt
30) Los Angeles—Brussels

• For TEST #3, the routes were:

1) New York/Newark—London
2) Chicago—London
3) Atlanta—London
4) Dallas/Ft. Worth—London
5) Minneapolis/St. Paul—London
6) Los Angeles—London
7) New York/Newark—Paris
8) Chicago—Paris
9) Atlanta—Paris
10) Dallas/Ft. Worth—Paris
11) Minneapolis/St. Paul—Paris
12) Los Angeles—Paris
13) New York/Newark—Amsterdam
14) Chicago—Amsterdam
15) Atlanta—Amsterdam
16) Dallas/Ft. Worth—Amsterdam
17) Minneapolis/St. Paul—Amsterdam
18) Los Angeles—Amsterdam
19) New York/Newark—Frankfurt
20) Chicago—Frankfurt
21) Atlanta—Frankfurt
22) Dallas/Ft. Worth—Frankfurt
23) Minneapolis/St. Paul—Frankfurt
24) Los Angeles—Frankfurt
25) New York/Newark—Brussels
26) Chicago—Brussels
27) Atlanta—Brussels
28) Dallas/Ft. Worth—Brussels
29) Minneapolis/St. Paul—Brussels
30) Los Angeles—Brussels

• For TEST #4, the routes were:

1) New York/Newark—San Juan
2) New York/Newark—Montego Bay
3) New York/Newark—Cancun
4) New York/Newark—Grand Cayman
5) New York/Newark—Rio de Janeiro
6) Chicago—San Juan
7) Chicago—Montego Bay
8) Chicago—Cancun
9) Chicago—Grand Cayman
10) Chicago—Rio de Janeiro
11) Atlanta—San Juan
12) Atlanta—Montego Bay
13) Atlanta—Cancun
14) Atlanta—Grand Cayman
15) Atlanta—Rio de Janeiro
16) Miami—San Juan
17) Miami—Montego Bay
18) Miami—Cancun
19) Miami—Grand Cayman
20) Miami—Rio de Janeiro
21) Minneapolis/St. Paul—San Juan
22) Minneapolis/St. Paul—Montego Bay
23) Minneapolis/St. Paul—Cancun
24) Minneapolis/St. Paul—Grand Cayman
25) Minneapolis/St. Paul—Rio de Janeiro
26) Los Angeles—San Juan
27) Los Angeles—Montego Bay
28) Los Angeles—Cancun
29) Los Angeles—Grand Cayman
30) Los Angeles—Rio de Janeiro

• For TEST #5, the routes were:

1) New York/Newark—Tokyo
2) Chicago—Tokyo
3) Atlanta—Tokyo
4) San Francisco—Tokyo
5) Los Angeles—Tokyo
6) New York/Newark—Seoul
7) Chicago—Seoul
8) Atlanta—Seoul
9) San Francisco—Seoul
10) Los Angeles—Seoul
11) New York/Newark—Bangkok
12) Chicago—Bangkok
13) Atlanta—Bangkok
14) San Francisco—Bangkok
15) Los Angeles—Bangkok
16) New York/Newark—Taipei
17) Chicago—Taipei
18) Atlanta—Taipei
19) San Francisco—Taipei
20) Los Angeles—Taipei
21) New York/Newark—Singapore
22) Chicago—Singapore
23) Atlanta—Singapore
24) San Francisco—Singapore
25) Los Angeles—Singapore
26) Minneapolis/St. Paul—Tokyo
27) Minneapolis/St. Paul—Seoul
28) Minneapolis/St. Paul—Bangkok
29) Minneapolis/St. Paul—Taipei
30) Minneapolis/St. Paul—Singapore

• Each test consisted of searching for international airfares. Booking criteria were established in advance.

For all five tests, these criteria included:

• One adult fare
• Economy or coach class
• Round-trip
• Any local airport specified
• Connecting flights acceptable
• Multiple-airline itineraries acceptable
• Flight times for any time of day acceptable for both departure and arrival
• No special or corporate rate programs (government, military, AAA, AARP, senior, student, child, etc.)
• No frequent flyer program membership

NOTE: Not all Web sites allowed such specificity for each test, but these parameters were established in advance to ensure consistency.

The nature of searching for international airfares required changes to established Consumer Reports WebWatch airline booking criteria. Specifically, this meant accepting 1) connecting flights in addition to non-stop and direct flights; 2) multiple-airline itineraries in addition to single-airline itineraries; and 3) departure and arrival times throughout the day. These changes were implemented to reflect the differences inherent in the airlines’ international route structures, since many foreign destinations are served only once a day and/or require connecting flights to international gateways, either on the same carrier or on that carrier’s marketing and code-sharing partner(s).

• In order to simulate a variety of trips, the advance booking windows varied. The booking times were:
TEST #1 (ADVANCE BOOKING): 28 days in advance
TEST #2 (ADVANCE BOOKING): 28 days in advance
TEST #3 (ADVANCE BOOKING): 28 days in advance
TEST #4 (SHORTER ADVANCE BOOKING): 14 days in advance
TEST #5 (LONG ADVANCE BOOKING): 90 days in advance

• Because of the nature of international travel, Consumer Reports WebWatch departed from earlier methodology and did not simulate a variety of trips by varying the length of the stays with 2-day trips, 3-day trips, etc. However, all trips included a Saturday-night stay-over, which often reduces the cost of the airfare. For this project, the stays were:
TEST #1: 7 days
TEST #2: 7 days
TEST #3: 7 days
TEST #4: 7 days
TEST #5: 7 days

back to top

SELECTION OF RESULTS

As noted in previous reports, Consumer Reports WebWatch no longer analyzes the lowest fares based upon the first fare returned, a method previously employed by Consumer Reports WebWatch (and earlier employed by Consumer Reports Travel Letter). This is because most integrated travel Web site screens no longer resemble the vertical presentations pioneered by global distribution systems (GDSs), and the lowest fare is not always listed first. Indeed, with some screens, it’s not always clear which position the “first” listing occupies.

In general, Consumer Reports WebWatch selected fares culled from either the first five returns or the first full page of returns, whichever was greater. Most of the branded airline sites provided traditional vertical listings, and in some cases the airline sites provided only one fare for each query. But the three integrated Web sites—Expedia, Orbitz, and Travelocity—offered additional display tools.

Orbitz employed its Matrix™ display, which provided a combination of vertical and horizontal interfaces with multiple fares and multiple airlines presented on a single screen. For these results, the lowest fare was selected within the appropriate category from among all the carriers presented within Orbitz’s grid. A detailed discussion of the Orbitz grid is available in "Inaccurate Displays of Flight Fares and Availability."

Both Expedia and Travelocity provided more traditional vertical listings. However, both Web sites offered horizontal summaries of the lowest fares at the top of these vertical listings, in a similar grid. As noted by Expedia, these proved to be useful as a “Quick Compare” tool.

Note that all rankings included ties. Therefore it was theoretically possible that for every fare query, every Web site tested could have provided the lowest airfare for that query.

A final note: Consumer Reports WebWatch testers were often provided with attractive fares that were outside the parameters of the query (e.g., $50 less expensive for a flight departing on the next day). While the average consumer may very well opt to book such a fare, in all cases our testers only selected fares that were applicable to that specific itinerary. Selecting lower fares that did not adhere to these specifics would provide that Web site with an advantage over other Web sites that were not queried outside the original itinerary.

back to top

CURRENCY ISSUES

All airfares were provided in U.S. dollars, including fares provided by foreign airlines. However, Consumer Reports WebWatch altered its previous methodology by tabulating total airfares that included all applicable taxes, fees, and booking costs. Previously, base fares and base rates were used in most cases.

There were two reasons for this change. 1) For most of these international airfare listings, only total fares were provided. Therefore, using total fares provided a better apples-to-apples comparison. 2) In many cases, total airfares included booking fees levied by the Web site itself. Therefore, using total fares provided a more accurate tabulation of a consumer’s “bottom-line” cost.

It should be noted, for example, that in many head-to-head comparisons, the branded airline sites did not charge a booking fee, while Expedia and Travelocity charged a $5 booking fee, and Orbitz charged a $10 booking fee. A detailed discussion of this issue is available in "Tax and Fee Information."

Once the decision was made to tabulate total airfares, the testing results from two branded airline Web sites—EVA Airways and Japan Airlines—were discarded as previously noted. This was because the total fares provided were not labeled clearly with taxes and fees, so apples-to-apples comparisons with other fares were not possible.

back to top

ROUNDING OF FARES

Consumer Reports WebWatch’s rounding of amounts for this project was consistent with its methodology for past projects. That is, amounts were rounded off to the nearest dollar for all airfares.

However, two of the integrated Web sites—Orbitz and Travelocity—always presented displays with dollar amounts rounded down, regardless of the cent value. When this occurred, Consumer Reports WebWatch corrected the rounding of the amount (i.e., if the cent amount was between 50¢ and 99¢, the amount was rounded up).

back to top

AVAILABILITY OF FARES

For this project, Consumer Reports WebWatch did not book any of the airline fares provided. In all cases, the Web sites stated that the airline seats requested were available. When the airfare provided was not available, the next lowest fare that was available was used. In some cases, this meant that multiple fares repeatedly were not available for the same query. Specific issues concerning availability of fares are discussed in "Inaccurate Displays of Flight Fares and Availability."

back to top

VALID TESTS

All of the Web sites were evaluated solely on valid tests. If there was an error on the part of Consumer Reports WebWatch, this was deemed an invalid test. Invalid tests were eliminated and did not affect the final rankings. These errors included incorrect data entries, insufficient returns of data, and printing errors. Overall, there were eight errors committed by Consumer Reports WebWatch testers for all 911 queries. This translated into a completion percentage of 99.1%.

back to top

INVALID DATA

Through no fault of the Consumer Reports WebWatch testers, many of these Web sites failed to provide valid data on some tests.

In some cases, these failures affected the Web site’s final rankings. These could have been due to a variety of factors, including:

• Results outside the specific parameters requested;
• Space not available for the specific itinerary requested;
• Airline seats sold out on those dates;
• The Web site could not price the flight selected;
• The Web site could not tabulate total airfare with all taxes and fees;
• Technical or system failures.

The following Web sites failed to provide valid data for at least one query:

• Air Jamaica
• American
• Continental
• Delta
• Expedia
• Korean Air
• Northwest/KLM
• Travelocity
• United

back to top

Although our testers experienced several technical or system failures in which a Web site was unable to process a request, these incidents were temporary and did not prevent the tester from completing the query. For this project, no Web sites were unable to process a request in time due to an apparent technical failure.

back to top

Testing Results, Rankings, and Conclusions

The findings of Consumer Reports WebWatch’s international airline Web site testing are presented in the attached figures and are described below. There are three major categories of rankings:

1) Lowest fares for all tests;
2) Closest fares for all tests;
3) Lowest fares for each of the five individual tests.

The results are detailed in Figures 1 through 7.

It’s important to note that all rankings included ties. Therefore it was theoretically possible that every Web site tested could have provided the lowest airfare when queried.

back to top

LOWEST FARES FOR ALL TESTS

Figure 1 shows how each of the Web sites performed at providing the lowest fares. The first section of the chart breaks down the results for the three integrated Web sites — Expedia, Orbitz, and Travelocity — as well as for all branded airline Web sites combined. Roughly half the results were posted by the three integrated sites collectively (450 queries) and half by the airline sites collectively (453 queries).

The second section of the chart presents individual rankings for every Web site tested, and clearly these findings are important as well. But context is critical. Because of the methodology of this testing, it would be inappropriate, for example, to rank Lufthansa (33%) with six valid queries performed ahead of Orbitz (30%) with 150 valid queries performed.

Therefore, when the branded airline sites are viewed collectively, it’s clear that Orbitz was best at providing lowest fares. Orbitz accomplished this 30% of the time, followed by its rivals Expedia (23%) and Travelocity (19%). Orbitz’s performance is particularly noteworthy because in many cases it charged a higher booking fee than both Expedia and Travelocity.

However, some of the branded airline sites performed well overall. Although the individual samplings for Lufthansa (6 valid queries) and Singapore (3 valid queries) were very small, they led all sites with 33% each.

More important, perhaps, was the performance of Northwest/KLM, which provided the lowest fares 27% of the time with a higher sampling of 49 valid queries. Northwest/KLM, in fact, provided lowest fares more often than both Expedia and Travelocity, when it competed head-to-head with these two sites in one-third of all queries.

It’s also important to note the rankings of both United and American. United provided lowest fares 17% of the time with 117 valid queries and American provided lowest fares 16% of the time with 119 valid queries. Put another way, these two branded airline sites provided lowest fares almost as often as Expedia and Travelocity, with nearly as great a test sampling.

The two remaining domestic branded airline sites—Continental (6%) and Delta (1%)—performed poorly at providing lowest fares, despite testing samples greater than Northwest/KLM’s (50 queries and 80 queries, respectively).

None of the six remaining foreign branded airline sites—Virgin Atlantic, Air Jamaica, Korean Air, Thai Airways, Air France, and British Airways—provided any lowest fares, with valid queries ranging from two to six apiece.

back to top

CLOSEST FARES FOR ALL TESTS

Consumer Reports WebWatch maintains that the inclusion of “closest” fares is critical to any evaluation of the competitive online travel market. In many cases, the difference in fares and rates provided by competing Web sites is just a few dollars for identical itineraries, and sometimes the difference can even be measured in cents. Even the most cost-sensitive consumers can make better decisions when they factor such small price differentials into their buying equations.

Closest fares are even more critical when evaluating more expensive products, such as international airline tickets, because the price differentials are considerably less when viewed as a percentage of the total cost. Therefore Consumer Reports WebWatch determined that $10 would be a fair and appropriate amount to delineate the closest fares, for two reasons. First, $10 was not more than 5% of the lowest fare provided for any of the 150 queries (American Airlines site, $197, Miami—San Juan, TEST #4). Second, in most cases $10 was the highest amount of the booking fee charged by an integrated site.

Figure 2 presents the findings when lowest and closest airfares are combined. Once again, the collective performance of the branded airline Web sites is compared to each of the three integrated Web sites in the first section of the chart. These results are segregated from the individual site results presented in the second section of the chart.

These results clearly show how easily Orbitz dominated its competitors by providing the lowest and closest fares 79% of the time. Discounting the fine performance posted by Singapore (67% for only 3 valid queries), Orbitz maintained a 24% lead over second-place Expedia at 55%, followed by Travelocity at 35%. Again, this was based on a total of 150 separate queries each for Orbitz, Expedia, and Travelocity. This was by far the largest spread between first-place and second-place rankings since Consumer Reports WebWatch began evaluating closest fares and rates for travel products.

Collectively, the branded airline sites did not perform well by providing lowest and closest fares only 19% of the time. Individually, however, there were impressive performances by some of the airline sites once again. And the inclusion of closest fares did not alter the lowest fare rankings of all five domestic airlines.

In addition to Singapore (67%, 3 valid queries) and Lufthansa (33%, 6 valid queries), Northwest/KLM led the domestic carriers by providing the lowest fare 31% of the time, for 49 valid queries. Once again, Northwest/KLM was followed by United (29%) and American (18%), with 117 valid queries and 119 valid queries, respectively. Including closest fares boosted Continental’s performance to 16% but did little to help Delta’s at 1%.

Among the foreign carriers, the inclusion of closest fares helped British Airways (17%), but did not assist five foreign airlines—Virgin Atlantic, Air Jamaica, Korean Air, Thai Airways, and Air France—that did not provide any closest or lowest fares.

Consumer Reports WebWatch’s long-standing advice to consumers is to shop around for the lowest fares and rates. Specifically, shoppers should always check the branded Web sites of airlines (or hotel or car rental companies) if an integrated site provides an attractive fare or rate, to see if the branded site can beat that price. This advice seems particularly relevant for three of the branded airline sites tested for this project: Northwest/KLM, United, and American. These three airline Web sites matched or beat Orbitz, Expedia, and Travelocity often enough that even the most time-pressed consumers would be well served to visit these branded sites when searching for low airfares.

back to top

LOWEST FARES FOR INDIVIDUAL TESTS

As noted, this project consisted of five separate tests, each consisting of 30 separate queries for airfares, for a project total of 150 valid queries. The results posted by each of these individual tests warrant a closer examination, since there were variables in the Web sites tested, the destinations, the routes, the dates of travel, and the testing dates themselves. However, it should be stressed again that the small test samples for many of the individual airline Web sites require that these results—both good and bad—be viewed in context.

Figures 3 through 7 detail each of the results in order for TEST #1 through TEST #5.

• Results for TEST #1, TEST #2, and TEST #3 (European routes)

The first three of the five separate tests focused on routes between the United States and Europe. Consumer Reports WebWatch felt it was appropriate to allot the division of routes in this manner, for two key reasons: 1) the sheer volume of passenger traffic on transatlantic routes, according to Bureau of Transportation Statistics data, and 2) the more robust competition on many of these routes, compared overall to routes between the U.S. and the Caribbean & Latin America and the U.S. and Asia.

What’s striking is that despite all the similarities in the itineraries for these three tests, the results are quite different. With few exceptions, the routes themselves remained fairly consistent throughout all three tests, and the specifics of the itineraries remained consistent as well. Also note that for all three tests, the airfares were queried 28 days in advance, and in all three tests, the length of stay was 7 days, including a Saturday-night stay-over in all cases. Yet the rankings of both the integrated and branded airline Web sites differed considerably among all three tests.

The key difference was the date of departure. For TEST #1, this was a Thursday; for TEST #2, this was a Friday; and for TEST #3, this was a Monday. The availability of seats, let alone the availability of low fares, was a critical component of this testing, and proved to be the deciding factor on which Web sites provided the highest number and percentage of lowest fares.

As Consumer Reports WebWatch has noted before, this confirms the vital importance of the date of travel. Consumers who are flexible on their departure and return dates—even by as small a margin as one day—are always likely to find better bargains. Comparison shopping will bear this out.

Here’s a brief synopsis of the three European route tests:

• TEST #1

As shown in Figure 3, American Airlines not only led seven Web sites in this test, but by providing 15 lowest fares among 30 valid queries, it posted the best performance of any Web site for all three European route tests.

• TEST #2

As shown in Figure 4, Orbitz led seven Web sites by providing lowest fares for 10 of 30 valid queries. Conversely, American’s performance plummeted, as the site provided just one lowest fare among 30 valid queries.

• TEST #3

As shown in Figure 5, Expedia led nine Web sites by providing lowest fares for 9 of 30 valid queries. It was followed by Travelocity, which ranked third, second, and second in these three tests.

• TESTS #1 THROUGH #3 COMBINED

At first glance of the rankings, it would appear the most interesting results were posted by both American and Orbitz, due to their fluctuations in providing lowest airfares. In two tests consisting of 30 queries each, American provided lowest fares of 15 and 1. In three tests consisting of 30 queries each, Orbitz provided lowest fares of 2, 10, and 7.

It also would appear that the most consistent results in the rankings were posted by Expedia and Travelocity. After both Web sites were queried 30 times in each of these three tests, Expedia provided lowest fares 6, 7, and 9 times, while Travelocity provided lowest fares 4, 8, and 8 times.

But if all three European route tests are combined, these are the rankings for all participating Web sites:

The three integrated sites, meanwhile, were clustered close together, with Expedia (24%) barely leading both Travelocity (22%) and Orbitz (21%). This may be a further indication of the competition that exists on routes between the U.S. and Europe. In numerous cases, Expedia, Orbitz, and Travelocity all tied by providing the lowest and/or closest airfare in these three tests.

• Results for TEST #4 (Caribbean & Latin American routes)

The most intriguing results of this test can be summed up in one word: Orbitz. The integrated site completely dominated not only its two prime competitors (Expedia and Travelocity), but five branded airline sites: four operated by domestic carriers and one operated by a Caribbean-based carrier.

As shown in Figure 6, Orbitz provided 18 lowest fares among 30 valid queries (60%), easily leading both Expedia and Travelocity. Both of these sites provided five lowest fares among 30 valid queries (17% each).

Among the five airline sites, only American provided any lowest fares: 3 among 29 valid queries (10%). Air Jamaica, Northwest/KLM, Continental, and Delta did not provide any lowest fares, among a varied number of queries each.

Even so, the poor performance posted by American is perhaps most noteworthy, since that carrier has long dominated the Caribbean & Latin American market, and operates busy gateways in both Miami and San Juan. In fact, both of these cities were included in the itineraries for TEST #4. But American’s primary distinction was that when queried for a fare between Miami and San Juan, the site provided the single lowest airfare ($373) among all 903 queries for this project.

It should be noted that in addition to the routes employed, there was another key difference with TEST #4: This was the shortest booking window of all five tests, with itineraries planned for just 14 days in advance. What remains unclear is whether or not a longer booking window would have led to a higher percentage of lowest fares for the any or all of these Web sites.

But whether it was due to the route structures or the time frame of the bookings, Orbitz clearly excelled. In fact, by providing lowest fares for 60% of these queries, it posted the single highest ranking for any of the Web sites among all five tests in this project.

• Results for TEST #5 (Asian routes)

For all 30 itineraries in TEST #5, the longest booking window was used (90 days). This seemed consistent with the nature of such long-haul travel between the U.S. and Asia, since many such trips are planned well in advance.

TEST #5 also produced a clear winner at providing lowest airfares: United Airlines. As shown in Figure 7, United’s ability to provide 11 lowest fares among 30 valid queries resulted in a mark of 37% and led eight other Web sites. United particularly shone on routes from the U.S. to Tokyo, Seoul, and Taipei.

United was followed by the three integrated sites: Expedia and Orbitz provided lowest fares 27% of the time and Travelocity provided lowest fares 13% of the time. However, all three integrated Web sites proved to be quite competitive at providing closest fares for these Asian routes. In fact, for all 11 queries in which United provided a lowest airfare, at least one and sometimes all three of the integrated sites provided a closest airfare. The small price differential was usually due to the fee charged by the integrated Web site(s).

Northwest/KLM performed very well with a small sampling, providing three lowest fares among five valid queries (60%). Similarly, Singapore performed very well on a much smaller scale, with one lowest fare among three valid queries (33%).

Conversely, Korean Air and Thai Airways both failed to provide any lowest fares among five valid queries each. Unfortunately, American also failed to provide any lowest fares, but with a much larger sampling of 30 valid queries.

As noted elsewhere in this report, two other foreign carriers were included in TEST #5. But the results posted by both EVA Airways of Taiwan and Japan Airlines of Japan were discarded because their airfares were not labeled clearly enough to be used for accurate apples-to-apples comparisons.

back to top

Findings and Concerns

Unfortunately, each online travel project undertaken by Consumer Reports WebWatch has uncovered serious consumer issues, and this test of international airline fares was no exception. The most alarming problems concerned displays of incorrect fares and unavailable flights. This section chronicles these and other areas of concern.

Happily, this project uncovered good news for consumers as well, through the posting of more consumer-friendly advisories and warnings to users of the three largest integrated travel Web sites.

Detailed breakdowns of these findings follow.

back to top

INACCURATE DISP