  |
||
|
Print Economists Ink Winter 2009 | |
Sampling Methodology in Merger Analysis: the Whole Foods Survey A survey may be a valuable tool in merger analysis, as it can shed light on consumers’ likely reactions to price increases, the core issue in market definition. A survey used in litigation, however, must be designed and executed with care, to minimize sources of inaccuracies and statistical bias. In FTC v. Whole Foods, US District Court Judge Paul Friedman decided against giving “any weight or consideration” to the survey submitted by Whole Foods’ expert. The judge relied on the FTC’s expert, who argued that the methodological flaws of the survey rendered the data and results unreliable. A better sampling plan would have eliminated many of the problems. Sampling methodology involves four steps: 1) defining the population of interest, 2) selecting a sampling frame, 3) developing a sampling method, and 4) deciding on a sample size. The first step in sampling is to define the population of interest to which survey results will later be generalized. The correct population of interest would have been all current customers of the merging parties, Whole Foods and Wild Oats, because these are all of the identifiable individuals who will be affected by the merger. (While future customers would also be affected by the merger, it is difficult to reliably ascertain whether someone who is not a current customer will be a future customer.) However, Whole Foods’ expert mistakenly defined two populations: “Frequent” and “Cusp” shoppers at Whole Foods or Wild Oats. Frequent shoppers visited Whole Foods or Wild Oats at least once per month, while Cusp respondents ranged from shopping a few times a year to having shopped at least once or twice at Whole Foods or Wild Oats. Then in generalizing her results to all current Whole Foods and Wild Oats customers, she erroneously gave the two customer groups equal weight, even though that likely would not be representative of the population of interest. The sampling frame is the list of individuals or households that corresponds to the population of interest. Examples include a public telephone directory, a list of Fortune 500 executives, or all individuals over the age of 18 who shopped at the Pentagon City Mall between the hours of 10 AM and 9:30 PM on Friday, July 25, 2008. The last type of sampling frame is known as the mall intercept or “convenience” sample. Academic and commercial marketing researchers seldom use convenience samples, for fear that the data collected from a small number of respondents at a particular location and time are biased and cannot be extrapolated to the population of interest, which is often the general population of all American consumers. However, when the population of interest is Whole Foods and Wild Oats customers, store-intercept data may be extrapolated to a universe of customers of those two particular grocery retailers, as long as steps are taken to minimize systematic bias. In particular, store locations should be selected randomly and days of the week and times of the day for data collection should be varied. Whole Foods’ expert selected a random-digit dialing (RDD) sampling frame. RDD call centers are equipped to randomly dial telephone numbers from a list generated according to certain selection specifications, such as ZIP code, telephone exchange code (the 3 digits that follow the area code in a telephone number), or telephone company. The problem with RDD in this case is that the population of interest (current Whole Foods and Wild Oats customers) is very small compared to the sampling frame (every household in the RDD database in the selected ZIP codes). Surveyors made 427,397 phone calls to find 25,011 Whole Foods or Wild Oats customers, only 1,607 of whom ultimately completed the interview. Using RDD for this type of study may be an inordinately expensive approach. The Whole Foods survey was based on eight metropolitan areas. Because these areas were non-randomly selected, there is no statistical basis to generalize from these areas to all Whole Foods and Wild Oats stores. The areas should have been selected from a random sample, or a stratified random sample based on criteria such as geographic region, consumer demographics, and presence of Whole Foods and Wild Oats. Moreover, even within the eight areas, the survey sampled potential respondents non-randomly because it had quotas of Frequent and Cusp customers. Additionally, the survey violated the sampling methodology by collecting RDD data from respondents outside of the specified ZIP codes. While store-intercept studies can have potentially serious drawbacks and should be performed with extreme caution, a carefully-executed store-intercept study likely would have yielded better results than a poorly performed RDD study in the Whole Foods case. Researchers should carefully choose a sample size, taking into account available resources and potential response rates. The final number of responses must be large enough to allow statistical inferences. If a pilot test indicates that response rate is likely to be unacceptably low, a different sampling frame or sampling method should be considered. The survey collected 1,607 responses from a sample size of 25,011 eligible customers, for a comparatively low response rate of 6%; 94% refused the survey. That low response rate makes it probable that survey respondents were not representative of the entire population of Whole Foods and Wild Oats customers. Thus, the results are likely tainted by non-response bias. Although Judge Friedman excluded the Whole Foods survey from consideration, surveys have the potential to provide valuable information in litigation, especially in antitrust and consumer protection. Unfortunately survey data are often discounted because of the numerous sources of potential bias. Before attempting to conduct a survey, researchers must understand how to design surveys that will be defensible and to identify issues such as biased sampling procedures, problems with questionnaire design and administration, omitted control variables, and improper interpretation of results. The United States Court of Appeals has since overturned the District Court’s decision to deny a preliminary injunction to block the merger. The Court of Appeals based its decision on the argument that Whole Foods and Wild Oats cater to a submarket of core customers who the Court believes would not switch to traditional supermarkets even in the event of a small but significant non-transitory increase in price. A properly performed survey could have shed light on many pertinent issues in this case, including the share of Whole Foods and Wild Oats who are core customers and the shopping and switching behaviors of those consumers.
Additional Articles in Winter 2009 Issue of Economists Ink Implications of the Recent Cipro Decision The GAO Report on the FTC’s Policy Towards Petroleum Mergers
|
