We are a commune of inquiring, skeptical, politically centrist, capitalist, anglophile, traditionalist New England Yankee humans, humanoids, and animals with many interests beyond and above politics. Each of us has had a high-school education (or GED), but all had ADD so didn't pay attention very well, especially the dogs. Each one of us does "try my best to be just like I am," and none of us enjoys working for others, including for Maggie, from whom we receive neither a nickel nor a dime. Freedom from nags, cranks, government, do-gooders, control-freaks and idiots is all that we ask for.
If you put enough data together (say, 100 correlations), and set your p-value high enough (say, 0.05) , you WILL get results that appear statistically significant (in this case, likely 5).
So you race to publish. But you shouldn't. You should get another data set and test only those 5 correlations. Or better, get 5 independent data sets and test each individually.
Then, maybe, publish. Or better, get a colleague who knows something about possible mechanisms that would explain the correlation and test those, then publish.
The real solution is to set your p-value lower. Physical scientists think p = 0.05 is far too high. Physicists like 0.0000003, for example, ten million times lower. 0.05 is OK for pollsters and psychologists. I think medicine and epidemiology should look a little lower, maybe 0.01.
There are four kinds of statistical correlations:
1. A causes B
2. B causes A
3. A and B are both caused by C
Until you have a defined mechanism of causation, all are possibilities, with 4 being the default assumption. Science starts at the P-value of significance, it doesn't end there.
Tobacco companies spent close to 50 years arguing 2, 3, and 4 with regard to the link between cancer and smoking (yes, there were actual proposals that cancer caused smoking), and Alzheimer's researchers are actively arguing 1, 2, and 3 about the presence of amyloid plaques in the brain.
Another Guy named Dan
I've always preferred the kneeling example.
If you study how many times a team kneels with the ball in college and pro football, it's clear that kneeling is a winning strategy. Teams that kneel 3 or more times a game are winners.
Based on that information, a good coach should plan on kneeling as often as possible.