Copernican Revolution . org
Transforming Our Lives through Self Reflection and Psychology
A psychology professor's collection of lessons fostering self-discovery through online activities, hands-on classroom experiences, engaging lectures, and effective discussion prompts.
Engaging Classroom Lesson by Katie Hope Grobman
Confirmation Bias and Scientific Discovery
In-person game played with students, the mimicking scientific discovery with Wason's 2-4-6 Hypothesis Rule Discovery Task.
My very first time teaching a college class was actually in Philosophy, not Psychology. As a Philosophy graduate student I taught Philosophy of Science to undergraduates. Science is an epistemology, a way of knowing, and even if scientists have been wrong and sometimes extraordinary immorally wrong, science works. Science helps us discover knowledge. But why? And what actually makes something science versus something else that just looks like science. It's an incredibly practical philosophical question because ideologues and con artists often try to give their views credibility by making it look like science. This activity is the first time I adapted something from one field into another - Cognitive Psychology became Philosophy of Science.
“
Whenever a theory appears to you as the only possible one, take this as a sign that you have neither understood the theory nor the problem which it was intended to solve.
Is Scientific Discovery Common Sense? How Confirmation Bias Challenges Us to Be Good Scientists
Sometimes people think science is just common sense. Isn't it just obvious viruses cause disease, the world is made of atoms, the Earth circles the Sun, and groups of like-minded people polarize? Except, it's not really so obvious because used to believe differently about all these scientific findings. But maybe doing science is common sense? The famous biologist, known as Charles Darwin's "bulldog" defending evolution, Thomas Huxley quipped, "Science is simply common sense at its best, that is, rigidly accurate in observation, and merciless to fallacy in logic." I have to reply "Accurate observation? Flawless logic? Is that what common sense is?" But he didn't study Psychology, which is kinda of ironic, because one reason "common sense" struggles so much to accept evolution is our teleological bias, our tendency to see things as having purpose even when they don't. Being a scientist means setting aside biases as best we can. This in class activity is meant to help all of recognize another unscientific inclination making science hard, the confirmation bias. But I would like to explain the activity in an odd order so you get to experience doing at least a little. That's because of yet another bias in our common sense, the hindsight bias, once we know something, we feel like we knew it all along. Once you understand the activity, you'll probably think you wouldn't make the mistake, except 80% of even very capable students mess up.
Setting Up the Class Activity
Bring to class enough copies of the 2-4-6 Game Handout for each of your students. Understanding the directions requires paying close attention so I provide precise Confirmation Bias Slides accompanying your deliberate instructions:
“
A sequence of numbers has an order to it. For example, 1-2-3 is a different sequence than 3-2-1.
Some sequences of 3 numbers make me incredibly happy; other sequences of 3 numbers make me very sad.
Your goal is to figure out the rule for what sequences make me happy.
But you can’t simply ask me my rule. Instead you can conduct experiments on me. You can make up a 3 number sequence and I’ll tell you if it makes me happy. Then you can make up another sequence, I’ll tell you again, and we’ll keep going until you’re mostly confident you know the rule inside my head.
Let’s do the first sequence together. 2-4-6 (gesture to the pre-written cell on the handout). After you write a sequence I’ll write a happy or sad face for you in the “fits my rule” column (gesture to the pre-written cell on the handout). It turns out this sequence makes me very happy!
Now that you have feedback from me, you should make your best guess for the rule that makes me happy. For example, you might guess, “counting up by 2’s.” If that’s your guess write it in this blank (gesture to cell), or you can feel free to write another hypothesis (wait for participant to write hypothesis).
To finish the row, you should make a rough estimate for how certain you are the rule you guessed really is the rule for sequences making me happy (gesture to cell). If you have absolutely no confidence and your guess is basically random, write 0%. On the other hand, if you are totally sure you have it, write 100%. You can also write anything in between, such as 50% (wait for participant to write percent).
Now let’s continue. Write a sequence of 3 numbers to test. Once I give you feedback with a happy or sad face, write your hypothesis. If it’s the same as before, feel free to put ditto marks. Say how sure you are with a percent.Then make another sequence to test what rule makes me happy. Wait for my feedback. And so forth. Once you’re nearly 100% confident we’ll stop.
Some sequences of 3 numbers make me incredibly happy; other sequences of 3 numbers make me very sad.
Your goal is to figure out the rule for what sequences make me happy.
But you can’t simply ask me my rule. Instead you can conduct experiments on me. You can make up a 3 number sequence and I’ll tell you if it makes me happy. Then you can make up another sequence, I’ll tell you again, and we’ll keep going until you’re mostly confident you know the rule inside my head.
Let’s do the first sequence together. 2-4-6 (gesture to the pre-written cell on the handout). After you write a sequence I’ll write a happy or sad face for you in the “fits my rule” column (gesture to the pre-written cell on the handout). It turns out this sequence makes me very happy!
Now that you have feedback from me, you should make your best guess for the rule that makes me happy. For example, you might guess, “counting up by 2’s.” If that’s your guess write it in this blank (gesture to cell), or you can feel free to write another hypothesis (wait for participant to write hypothesis).
To finish the row, you should make a rough estimate for how certain you are the rule you guessed really is the rule for sequences making me happy (gesture to cell). If you have absolutely no confidence and your guess is basically random, write 0%. On the other hand, if you are totally sure you have it, write 100%. You can also write anything in between, such as 50% (wait for participant to write percent).
Now let’s continue. Write a sequence of 3 numbers to test. Once I give you feedback with a happy or sad face, write your hypothesis. If it’s the same as before, feel free to put ditto marks. Say how sure you are with a percent.Then make another sequence to test what rule makes me happy. Wait for my feedback. And so forth. Once you’re nearly 100% confident we’ll stop.
Figure 1. Beginning of the 2-4-6 game.
Try It Yourself!
Before we continue, try playing the game at least a little yourself.
Please make up the next sequence. What would you write in the three little blanks on the second row?
Did the sequence you suggested fit what you think the rule is?
Being concrete, if you were guessing, "count up by two's" like my example, and you wrote something like, "6, 8, 10" then you sought to confirm the rule because it's also counting up by two's. But if you put something like, "3, 2, 1," while thinking my rule is "count up by two's," you sought to falsify the rule.
If you made a sequence confirming the rule, you made the mistake - the confirmation bias. Your students will get many more chances to break the cycle. No worries if you showed the bias. It's a natural part of being human, and following complete instructions for the game, I'll elaborate.
Please make up the next sequence. What would you write in the three little blanks on the second row?
Did the sequence you suggested fit what you think the rule is?
Being concrete, if you were guessing, "count up by two's" like my example, and you wrote something like, "6, 8, 10" then you sought to confirm the rule because it's also counting up by two's. But if you put something like, "3, 2, 1," while thinking my rule is "count up by two's," you sought to falsify the rule.
If you made a sequence confirming the rule, you made the mistake - the confirmation bias. Your students will get many more chances to break the cycle. No worries if you showed the bias. It's a natural part of being human, and following complete instructions for the game, I'll elaborate.
Applying the Rule
Now it's time for you to write a happy or sad face on each of your students' papers. To speed up the process, I zip around the room systematically and just quietly say to each student "happy" or "sad." You might need to individually help some students understand the task. If you notice lots of confusion, its feedback next time to be even more deliberate explaining the game.
To speed the process, I usually say, once you're 100% sure, please turn your paper over so I know to skip you. Afterall, you can't get more confident than 100%.
The rule is incredibly simple: any increasing sequence. Usually students write simple sequences and it requires just a little thought on your part. For example, {1,2,3}, {8,10,12}, and {7,63,99} all fit the rule. But sequences like, {3,2,1}, {7,4,7}, and {42,42,42} do not fit the rule. Occasionally students get fancy and it's fine but a little taxing on you. For example,"-7, π, 30000" fits the rule. So does {⅛, ¼, ½} and I usually get giggles saying loudly enough for the class to hear, "fractions! you're making me do fractions!"
Unless you have a small class, it's probably too unwieldy to keep giving feedback until every single student is 100% sure. So I usually stop after about 5 rounds when the vast majority of student are at least 90% sure. Before polling I say something like, "Even if you're not 100% sure, let's see what rule you figured out is pretty likely."
To speed the process, I usually say, once you're 100% sure, please turn your paper over so I know to skip you. Afterall, you can't get more confident than 100%.
The rule is incredibly simple: any increasing sequence. Usually students write simple sequences and it requires just a little thought on your part. For example, {1,2,3}, {8,10,12}, and {7,63,99} all fit the rule. But sequences like, {3,2,1}, {7,4,7}, and {42,42,42} do not fit the rule. Occasionally students get fancy and it's fine but a little taxing on you. For example,"-7, π, 30000" fits the rule. So does {⅛, ¼, ½} and I usually get giggles saying loudly enough for the class to hear, "fractions! you're making me do fractions!"
Unless you have a small class, it's probably too unwieldy to keep giving feedback until every single student is 100% sure. So I usually stop after about 5 rounds when the vast majority of student are at least 90% sure. Before polling I say something like, "Even if you're not 100% sure, let's see what rule you figured out is pretty likely."
Polling the Class
I normally write the rules on the board and write a count beside each from a show of hands. When polling class, I recommend you not ask for rules at first. It would allow students to change their mind. So I ask narrower questions first. I'm usually aware of which students have the actual sequence rule and I delay calling on them as long as possible.
“
How many of you ended with the same rule we began with, "count up by 2's?"
How many of you ended with a rule like, "count up by multiples," such as {3,6,9}, {5,10,15}, and {50,100,150}?
How many of you ended with a rule that's a formula like add the first two numbers for the third number?
Anybody have another rule?
How many of you have the same rule as so-and-so?
How many of you ended with a rule like, "count up by multiples," such as {3,6,9}, {5,10,15}, and {50,100,150}?
How many of you ended with a rule that's a formula like add the first two numbers for the third number?
Anybody have another rule?
How many of you have the same rule as so-and-so?
Typically about 80% of my classes have one of the first three rules and most of the rest has the correct answer.
Revealing the Answer
The correct answer is any increasing sequence of numbers makes me happy. So {1,2,3} and {-36, √2, 999999} make me happy. But {3,2,1} and {2,2,2} make me sad.
Why did so many of you become so confident about a wrong answer?
Animating the slides (figure 2) I reveal and say the question:
Maybe more interestingly, how come I was so confident I would animate a slide before class assuming you would not figure it out?
After asking the questions, I ask students to think about it and we'll return to before we end class today. I typically play this game at the beginning of the first day of my Social Psychology classes, even though we don't cover social cognition until mid-semester. Then I can ask rhetorically at the end of class, "I wonder if you'll make the same bias on another activity later in the semester?" They do. You might use the 2-4-6 game in a cognitive psychology or research methods class with a slightly different focus.
Why did so many of you become so confident about a wrong answer?
Animating the slides (figure 2) I reveal and say the question:
Maybe more interestingly, how come I was so confident I would animate a slide before class assuming you would not figure it out?
After asking the questions, I ask students to think about it and we'll return to before we end class today. I typically play this game at the beginning of the first day of my Social Psychology classes, even though we don't cover social cognition until mid-semester. Then I can ask rhetorically at the end of class, "I wonder if you'll make the same bias on another activity later in the semester?" They do. You might use the 2-4-6 game in a cognitive psychology or research methods class with a slightly different focus.
Figure 2 Slide revealing the rule and asking class a question.
How Do We Respond to the 2-4-6 Game?
It’s normally pretty intuitive to students how they played the game and why it went wrong (see figure 3, typical play)
Draw out of them how they had an idea and then kept testing the idea with things fitting the idea. Since every time they tried, it worked, they kept feeling more and more confident. But they didn’t consider exploring other possibilities beyond their idea.
Draw out of them how they had an idea and then kept testing the idea with things fitting the idea. Since every time they tried, it worked, they kept feeling more and more confident. But they didn’t consider exploring other possibilities beyond their idea.
Figure 3 Typical play of the 2-4-6 game, following the confirmation bias.
How Should We Respond to the 2-4-6 Game?
It’s usually a bit more challenging for students to figure out how they should have played to figure out my rule. Draw out of students how they need to actively challenge their preconception by conducting experiments that could falsify their hypothesis.
The hallmark of every science - psychology, physics, and every other - is falsifiability - a clear possibility our hypothesis could be wrong. It's why, in Psychology, we always set up our hypothesis (e.g., bar P is higher than bar Q) against a null (e.g., bar P and Q are the same) and we either reject the null (falsify it) or fail to reject the null (haven't found evidence falsifying it yet).
More simply stated, the only way to show we’re right, is to try and show we’re wrong but fail. That means, with each sequence ("experiment") we need to try and find a possible sequence where we're wrong (see figure 4, ideal play). So if I think its counting up by two's I need a test that's anything but counting up by two's. Notice how counter-intuitive that sounds? That's highlighting just how intuitive the confirmation bias is.
The hallmark of every science - psychology, physics, and every other - is falsifiability - a clear possibility our hypothesis could be wrong. It's why, in Psychology, we always set up our hypothesis (e.g., bar P is higher than bar Q) against a null (e.g., bar P and Q are the same) and we either reject the null (falsify it) or fail to reject the null (haven't found evidence falsifying it yet).
More simply stated, the only way to show we’re right, is to try and show we’re wrong but fail. That means, with each sequence ("experiment") we need to try and find a possible sequence where we're wrong (see figure 4, ideal play). So if I think its counting up by two's I need a test that's anything but counting up by two's. Notice how counter-intuitive that sounds? That's highlighting just how intuitive the confirmation bias is.
Figure 4 Typical play of the 2-4-6 game, following the confirmation bias.
Understanding Confirmation Bias
In Psychology, a bias, is any inclination away from a neutral point. Hearing "bias" often bring prejudice to mind, which is absolutely a bias because we're pre-judging someone, usually based on a stereotype. When we study research methods, we're learning how to reduce bias in science: selection bias, acquiescence bias, and our replicability crisis comes from our bias to see null results as less worthy of publication. Many biases are innocuous, like most people have a right-hand bias.
Confirmation bias means we have a natural inclination to seek our information supporting our pre-existing views and to interpret ambiguous evidence as supporting our current theories (e.g., Darley & Gross, 1983).
My favorite example of confirmation bias was a participant obersation study by social psychologists who joined a doomsday cult to see how they would respond when the world did not end on the day they predicted (Festinger et al., 1956). The result? The cult members rationalized their faith had saved the world.
Confirmation bias can have serious consequences. It's part of group polarization. And studies show psychologists and psychiatrists interpret ambiguous information about a client with their pre-existing stereotypes of groups their client belongs to (e.g., Mendell et al., 2011; O'Reilley et al., 1989)
Confirmation bias means we have a natural inclination to seek our information supporting our pre-existing views and to interpret ambiguous evidence as supporting our current theories (e.g., Darley & Gross, 1983).
My favorite example of confirmation bias was a participant obersation study by social psychologists who joined a doomsday cult to see how they would respond when the world did not end on the day they predicted (Festinger et al., 1956). The result? The cult members rationalized their faith had saved the world.
Confirmation bias can have serious consequences. It's part of group polarization. And studies show psychologists and psychiatrists interpret ambiguous information about a client with their pre-existing stereotypes of groups their client belongs to (e.g., Mendell et al., 2011; O'Reilley et al., 1989)
I hope learning how easily we make the confirmation bias can help us make the world a better place. We can actively choose to burst our bubbles and reduce group polarization. We can actively challenge our stereotypes of people different from us. And we can do better science by actively seeking to contrast our hypotheses with alternatives.
References
Darley, J. M., & Gross, P. H. (1983). A hypothesis-confirming bias in labeling effects. Journal of Personality and Social Psychology, 44(1), 20-33.
Festinger, L., Riecken, H. W., & Schachter, S. (1956). When prophecy fails. University of Minnesota Press.
Mendell, N. R., & Hahn, U. (2011). Confirmation bias in the classroom: Preserving beliefs while promoting understanding. The Journal of Experiential Education, 34(2), 123-135.
O'Reilly, K., O'Shaughnessy, J., & Mannion, J. (1989). Children's perceptions of their academic progress: A confirmation bias analysis. Journal of Social Psychology, 129(4), 559-560.
Wason, P. C. (1960). On the failure to eliminate hypotheses in a conceptual task. The Quarterly Journal of Experimental Psychology, 12, 129-140.
Festinger, L., Riecken, H. W., & Schachter, S. (1956). When prophecy fails. University of Minnesota Press.
Mendell, N. R., & Hahn, U. (2011). Confirmation bias in the classroom: Preserving beliefs while promoting understanding. The Journal of Experiential Education, 34(2), 123-135.
O'Reilly, K., O'Shaughnessy, J., & Mannion, J. (1989). Children's perceptions of their academic progress: A confirmation bias analysis. Journal of Social Psychology, 129(4), 559-560.
Wason, P. C. (1960). On the failure to eliminate hypotheses in a conceptual task. The Quarterly Journal of Experimental Psychology, 12, 129-140.
Citation
Grobman, K. H. (2003). Is Scientific Discovery Common Sense? How Confirmation Bias Challenges Us to Be Good Scientists. CopernicanRevolution.org (originally published DevPsy.org as Confirmation Bias: A class activity adapted from Wason’s 2-4-6 Hypothesis Rule Discovery Task)
Book chapter version for students to create a lab exercise:
Grobman, K. H. (2018). Confirmation bias and the 2-4-6 game: How do we test our perspectives? In R. L. Miller (Ed.) Promoting Psychological Science: A Compendium of Laboratory Exercises for Teachers of High School Psychology, (pp. 286-290), Washington, DC: American Psychological Association Society for the Teaching of Psychology
Grobman, K. H. (2003). Is Scientific Discovery Common Sense? How Confirmation Bias Challenges Us to Be Good Scientists. CopernicanRevolution.org (originally published DevPsy.org as Confirmation Bias: A class activity adapted from Wason’s 2-4-6 Hypothesis Rule Discovery Task)
Book chapter version for students to create a lab exercise:
Grobman, K. H. (2018). Confirmation bias and the 2-4-6 game: How do we test our perspectives? In R. L. Miller (Ed.) Promoting Psychological Science: A Compendium of Laboratory Exercises for Teachers of High School Psychology, (pp. 286-290), Washington, DC: American Psychological Association Society for the Teaching of Psychology
