A recent Supreme Court decision considered whether information must be “statistically significant” to be relevant. (Matrixx Initiatives, Inc., et al. v. Siracusano et al.) While the Court’s decision is limited to a narrow context, it sheds light on the importance of significance tests and the weight that should be given to results that fail these tests.
Investors in Matrixx Initiatives Inc. alleged that the company failed to disclose material information when it did not reveal reports that some users of its leading product, Zicam, had lost their sense of smell and that there were pending lawsuits from those affected. When that information was eventually reported, the company’s stock price fell. Matrixx argued that the information was not material because the reported results were not statistically significant. The Court ruled unanimously that the case could proceed and that companies could not rely solely on statistical significance to determine what information must be disclosed.
In fields that examine “noisy” data, such as medicine and economics, it is common (and often required) to note whether reported results are statistically significant. “Significant” has a very specific meaning in statistical analysis. Tests of statistical significance are commonly framed as a choice between a base hypothesis (typically that there is no relationship between two variables) and an alternative hypothesis (there is a relationship). A result is statistically significant when the base hypothesis of no relationship is rejected at a given significance level, implying that the alternative hypothesis that there is a relationship is correct. (That level is often, but not always, 5%.) Statistical significance at the 5% level is often misinterpreted to mean, “I am 95% sure there is a relationship.” The correct interpretation is that if there were no relationship, there is less than a 5% chance of finding the result shown by the data. Thus the result is very unlikely to be due to chance, so the no-relationship hypothesis is rejected in favor of the hypothesis of a relationship.
The Court found that a result may be relevant, even if it is not statistically significant. The Court noted that both medical experts and the FDA rely on evidence other than statistically significant results and that “courts frequently permit expert testimony on causation based on evidence other than statistical significance.” The Court concluded that investors could reasonably rely on results that are not statistically significant.
The Court has recognized that statistical significance is not the sole measure of relevance. Nonetheless, the decision is narrowly tailored to information that might be relevant to an investment decision. Thus, the decision’s broader implications may be limited.