Repeatedly claims are being made about food safety based on searches for differences between foods based on the concept of statistical significance (see for example Academics Review Genetic Roulette 1.3).
In many cases the differences that are claimed are not even statistically significant because the wrong statistical models are used. Quite often the assumptions used when apply the test are violated , and the old saying, a garbage in garbage out applies .When using statistical tests based on certain assumptions it is always important to always ask whether or not the assumptions are justified.
But other statistical mistakes are being made when making claims about GM feed safety. A common one is to confuse statistical with practical and biological significance.
Even when the assumptions of use a statistical test are justified, a statistical indicator that comes out of them may be of no practical medical or clinical importance. The results may be statistically significant but of no practical or biological importance. Indeed the differences can occur between feeds and foods which are of no importance whatsoever for food safety or nutrition. Most meals we eat are different from one another, but this usually does not matter.
The basics of the common misinterpretation of statistical significance is explained extremely well by statistician Schuyler Huck in his great little book about the common misconceptions about statistics.
QUOTE: Statistical Significance Versus Practical Significance
From Schuyler W Huck 2008 Statistical Misconceptions. Chapter 11
The Misconception
Statistically significant results signify strong relationships between variables or big differences between comparison groups.
Evidence That This Misconception Exists
The first of the following statements comes from a peer-reviewed journal article in the behavioral sciences. The second statement comes from a book designed to help people become better able to understand research in applied linguistics. The third statement comes from a book dealing with a subfield of statistics called forecasting. (In these passages, note the words common, misconception, often, and confuse.)
- A common misuse … is the implication that statistical significance means theoretical or practical significance. This misconception involves interpreting a statistically significant difference as a difference that has practical or clinical implications.
- Two other misconceptions are common regarding statistical significance. One is to think that because something is statistically significant, there is a strong relationship between variables or a big difference between groups…. The other common misconception about statistical significance is to confuse it with practical significance.
- [Researchers often misinterpret statistical significance…. One problem is that researchers (and editors and reviewers) often confuse statistical significance with practical significance.
Why This Misconception Is Dangerous
There are two kinds of significance—statistical significance and practical significance—and they refer to entirely different concepts. To think that one implies the other is tantamount to thinking that a bridal shower and a bathroom shower are the same thing, or that the King of Hearts is equivalent to the King of England. Whereas few people would ever confuse these two kinds of showers or these two kinds of kings, it unfortunately is the case that statistical significance is often interpreted—even by some researchers—to mean significance in a practical manner.
If you think that statistical significance implies practical significance, you are likely to disappoint others or be disappointed yourself. If you are a researcher and talk about your statistically significant findings in a way that makes others think that you’ve discovered something big, important, and noteworthy, the recipients of your results may be disappointed (or even angry) when they discover, after taking action on your study’s findings, that what they expected to be large or strong in reality is small or weak. If you are the one who spends time, energy, or money on something and expect your actions to make a big difference (because you think statistical significance = practical significance), it may be you who gets disappointed.
If you or others fail to distinguish between these two kinds of significance, what appears to you or them to be a mountain may actually be only a molehill!
Undoing the Misconception
…Practical significance, in contrast to statistical significance, is focused on a study’s possible impact on the work of practitioners or other researchers. Here, the question being asked is: Will people who read or hear about a study’s findings consider altering what they do or think?
