Meta-Analyses and their birth defect — Emptying File-drawers.

This post is going to be addressing a problem with arguably one of the most important exports of science and what can be done about it.

While this still might sound like just about the most boring read ever, you have to take into account that it comes from a person who got genuinely excited when he recently found an article named — “ THE FILE-DRAWER PROBLEM REVISITED: A GENERAL WEIGHTED METHOD FOR CALCULATING FAIL-SAFE NUMBERS IN META-ANALYSIS” that led to derailing a whole post about exercise and stress.

And yes, since I do understand that reading this title also got you somewhat excited, let us just take a short stroll down what this means and why it is important.

See, in Science, there are these wonderful things called called “Meta-Analyses”. A Meta-Analysis is basically a science baby that comes about when a couple, or usually a whole group of scientists decide to have some fun (fun, in scientific language, is to gather and read through a whole lot previously published studies in a field). The reason these babies are great is because they go on to help people who are not as diligent as their parents to save time by pointing to the direction that most of the results are found.

But just like with baby Jesus, there is something out to harm the Meta-Analyses as well. This problem is in the form of a birth defect affecting some of them and goes by the name of the “File-drawer problem” (Also called “Publication bias”).

The result of the defect is that the baby is birthed skewed and will forever be pointing in the wrong direction…

This birth defect comes about because some scientists are craving a thing called “sci cred”, which is just like “street cred”, but for nerds. Because just like regular people, scientists want to impress others, and the scientific way of doing this is by telling other scientists that they found something significant, which in turn earns them the sci cred. The problem comes about when scientists have conducted their experiments, analyzed the results, and end up not finding anything significant. When this happens, the scientist has two possible ways to go about it.

Option A: They can just put it in their file-drawer and pretend it never happened, nothing happens.

Option B: They admit that they found nothing, which sounds quite lame and risks decreasing their sci cred.

While Option A sounds safe, it is not in the long term. Option A actually ends up screwing the data, or skewing* as scientists like to spell it. And it is this very data that forms the future generations of Meta-Analyses, so if it is skewed, they will also be screwed.

To be fair the problem is not only on the Scientists’ side of things but also on the publishers. You will likely never read an article in Nature that talks about a null result — a result that does not support the initial hypothesis. This is because null results are super boring to read about. To understand why we have to understand the fact that scientists like good gossip. (Nature is a super sci creddy newspaper, kind of the science equivalent to Rolling Stone or OK!).

So to put it into “real-world” talk — It would not really be classified as good gossip if a group of paparazzi’s came in, after having followed Tom Cruise for a month reporting that:

“Everything was just like we thought, Tom had the occasional nice meal, drove a nice car and lived in a nice house. No lover, no weird dungeon and no tigers in cages (i.e. nothing significant)”

These paparazzi’s would not get this story into any major paper, would they?

What needs to happen in order to save these poor little babies is that the whole scientific community changes when it comes to this issue!

Some points to start out with would be the following:

1: Scientists must start testing the data before they go on to make their babies to make sure no new defected ones are born!

The paper mentioned at the start goes into a mathematical formula for doing this kind of testing, there are also other ways of doing this.

2: The scientists must realize that the real sci cred lies in choosing Option B — Emptying their file drawers and publishing their null results!

Super sci creddy scientist Adam Grant wrote a whole book about this together with Facebook COO Sheryl Sandberg called: “Option B: Facing Adversity, Building Resilience, and Finding Joy”.

3: The publishers must realize that what the world needs is not only the sensational but also the real!

There are initiatives such as this one — “Scientists offered €1,000 to publish null results”.

P.S Joking aside, this is a real problem and an important problem.

P.P.S I’ve heard that Grants and Sandbergs book is great, although it is not at all related to this topic, so that was a joke as well.