“You Can Find Research that Proves Anything”

Sometimes teachers hear about research that SUPPORTS our current beliefs and teaching practice.

Honestly, that experience feels great. “Look,” says my interval voice, “I’ve been doing it right all along.”

And sometimes, we hear about research that CONTRADICTS our beliefs and practice.

Honestly, that’s a punch to the gut. “Wait,” says that voice, “I’ve been wrong all this time?”

This discomfort often prompts us to use a handy rejoinder: “well, you can find research that proves anything…”

The not so subtle implication: “yes, this research says that — but this research doesn’t really matter because even absurd positions can find research backing…”

So, what should we do when a colleague rejects our research-based advice with this claim? Or, what should we do when we find ourselves saying it?

Step #0: Let’s Check

In the first place, I’m honestly not so sure that we can find reseach that says anything.

Let’s take a common piece of teaching advice: “teachers should shake hands with their students as they enter the classroom.”

Can we find research supporting, or contradicting, this claim? If “we can find research that says anything,” we certainly should be able to.

Well, so far I haven’t found any research examining this question.

As I’ve written before, Dr. Clay Cook found that “positive greetings” at the door produced specific benefits for specific students.

But his research doesn’t remotely suggest that all teachers should “shake-in” at all times. (For one thing: “positive greetings” don’t have to be handshakes.)

I just asked Elicit.org this handshake-at-the-door question. The closest answer I found is a study showing that female professors get higher ratings on the first day of class when they shake in, whereas male professors get lower ratings.

But again: that study neither confirms nor contradicts the larger claim about daily handshakes.

It seems that we can’t always “find research that proves anything.”

Step #1: Start Reading

But let’s agree that we can find research supporting lots of strange conclusions — or, at least, conclusions I disagree with. What should we do when that happens?

Imagine that a friend tells me: “chewing gum increases learning.”

When I ask him if he’s found research supporting that position, he grins broadly and says: “check this out.”

So, let’s check it out!

A cursory glance suggests that — yes — my friend has found research supporting his position, but it’s not terribly persuasive research.

It includes exactly 16 participants.

It’s published in a journal that focuses on engineering (not, say, memory, or learning).

Its method of measuring attention is … well … HIGHLY unscientific.

In other words, my friend found research supporting this claim; however, I didn’t need to look very hard to find reasons to doubt it.

That is: it doesn’t really matter if I can find research that “proves anything.” What matters is if I can find GOOD research supporting a particular claim.

Step #2: Get Curious

But, is GOOD research enough? If I find one well-done study, should I accept that chewing gum does promote memory?

When I got started in this field, I noticed that the scholars I admired most shared a surprising intellectual habit:

They shift to curiosity.

That is:

Person 1 says: “research shows that chewing gum improves learning!“

Person 2 says: “nope; research shows that chewing gum has no effect at all on learning.”

At this point, person 1 might say: “well, you can find research that shows anything. You’re obviously wrong. My research is correct.”

Or, person 1 might say: “wow, I’m curious that we have seen research that arrives at contradictory conclusions! Let’s explore…”

Over the years, I’ve come to rely on two sources when I feel curious and want to explore.

Scite.ai asks how often a particular study has been cited overall; how many times its findings have been confirmed; and how many times its findings have been contradicted.

Connectedpapers.com looks at the most frequently cited papers related to the topic, and creates a cool spiderweb diagram to show their connections.

Using these websites, person 1 and person 2 can plug in their studies, and see how many OTHER studies arrive at their conclusions.

That is: rather that rely on just ONE study, we can look at a WHOLE GROUP of studies to reach our conclusion.

When I use these tools to explore the chewing gum claim, as I’ve written before, I arrive at several conclusions:

First: researchers have done a surprising amount of work on this topic. (It seems like SUCH a niche-y question that I’m surprised folks have investigated it substantively.)

Second: even the quality research in this field (i.e., more than 16 participants) arrives at contradictory results.

This overview, noting that we can find clear evidence of both benefits and detriments, concludes that “the robustness of reported effects of gum chewing on cognition has to be questioned.”

So, at this point I don’t think we can claim we have a decisive, research-informed answer to this question.

In other words: the question is not “can we find research that proves anything?”; or even “can we find GOOD research that points in a clear direction?”; but “can we find SEVERAL studies all pointing in a clear direction — and more studies pointing this way than that way?”

Only if the answer to that last question is “yes” should we teachers start changing our practice because “research says so.”

TL;DR

Can we really find research that supports any claim about education?

First: no.

Second: we don’t want research, we want good research.

Third: we don’t just want good research, we want several good studies pointing roughly toward the same conclusion.

Until we have met these criteria, we can’t really say that a particular claim merits our attention and respect.

Cook, C. R., Fiat, A., Larson, M., Daikos, C., Slemrod, T., Holland, E. A., … & Renshaw, T. (2018). Positive greetings at the door: Evaluation of a low-cost, high-yield proactive classroom management strategy. Journal of Positive Behavior Interventions, 20(3), 149-159.

Wilson, J., Stadler, J., Schwartz, B., & Goff, D. (2009). Touching your students: The impact of a handshake on the first day of class. Journal of the Scholarship of Teaching and Learning, 108-117.