When I taught my first undergraduate social psychology course, I was excited to explain one of my favorite findings to the class: smiling makes you happier. In the original study, participants were asked to rate how funny they thought a series of cartoons were while holding a pen in their mouth. The trick was that the way they held the pen varied: in one condition, they held it with their lips, preventing them from activating smile muscles; in another, they held it with their teeth, forcing them to activate their smile muscles. When the smile muscles were active, participants rated the cartoons as funnier! You may not realize it, but smiling changes your feelings! Activating muscles associated with a specific emotion seemed to influence people’s emotional responses in a subtle, unconscious way.
Then, that finding turned out to be wrong. In 2016, a large-scale study that collected data from 17 different labs found that the original study’s results did not replicate. Activating smile muscles did not change how funny people found cartoons. This result felt definitive because this new replication included many more people, from a more representative sample (since it wasn’t just students from one university), and with a more constrained methodology and analysis plan (because they ran everything by experts in facial feedback research who provided feedback on everything). But was it?
Just two years later, new evidence was added to the debate. Tom Noah, Yaacov Schul, and Ruth Mayo keyed in on what they believed was an essential difference between the original and the replication study: the use of a video camera. Based on feedback from one of the experts they consulted, the replication group had decided to video record each session to make sure everything had worked smoothly (including the pen being held in the mouth the right way!). But having a video camera facing you might make you more self-conscious. Different literature in psychology suggested that you’re less willing to rely on “gut feelings” when making decisions when you’re aware you’re being watched. This might have messed up the results of the replication study.
So Noah and colleagues decided to run a new study, where they randomly assigned participants to one of two different versions of the experiment: the original, no camera version, versus the replication, camera-included version. When comparing people in front of a camera, they replicated the replication; there was no effect. When they didn’t include the camera, they replicated the original; there was an effect. It was the video camera that made the difference.
It is easy to interpret this as a fight between science reformers and their opponents among social psychologists. A traditional finding was rejected, and then a counter punch was thrown, rejecting the rejection. (Editorial comments by Noah and colleagues imply that replicators’ claims “decrease cumulative science” don’t help). Yet this is actually a great example of how science is meant to work. Scientists are meant to question each other’s findings, and it’s essential to find out whether results hold up. This includes questioning the results of replications and thinking through why results might be different when a study is replicated.
One potential outcome of re-examining earlier research always needs to be “we got it wrong, this effect is not something reliable about the world.” This can be because of statistical noise–the way the data came out, it looked like there was a difference between groups, even when there really wasn’t. That’s no fault of the experimenter, but it’s something we can check by doing replications. Finding out that an earlier belief was wrong should always be an acceptable outcome of the research, and that does indeed contribute to a cumulative science.