Tag: methodology

Here at Learning and the Brain, we believe that research can improve education.

Specifically, research into psychology (“how the mind works”) and neuroscience (“how the brain works”) can help teachers and schools. After all, we spend all day working with students’ minds and brains!

Every now and then, we should stop and look for flaws in our assumptions.

Are there ways that research might not help learning? Might it actually limit learning?

A recent article by Yong Zhao explores this important — and troubling — question.

The Medical Model

Doctors have long relied on research to test their hypotheses.

Does this treatment work better than that treatment? Let’s run a “randomized control trial” to find out.

Notably, medical research always includes this important question: what side effects* does a treatment produce?

That is:

Any treatment might produce specific benefits.

And, it might also produce specific harms.

Medical research looks for and reports on BOTH.

Sadly — and this is Zhao’s key point — education research tends to skip the second question.

Researchers look for benefits:

Does mindfulness reduce stress?

Can retrieval practice enhance learning?

Should students exercise mid-class?

When they measure the potential upsides of those “treatments,” they don’t always look equally scrupulously for downsides.

And yet: almost everything has downsides.

What to Measure, and When?

Why do we overlook the downsides?

Zhao offers two hypotheses.

First, we all agree that education is good.

If doing X helps students learn, then X is good! Its obvious goodness makes potential badness invisible.

Second, downsides take time — and alternative methods — to discover.

An example. I hypothesize a particular method will help students sing better. So, I test my method in a randomized control trial.

Sure enough, students with the new method sang better! My method worked!

However, my new teaching method just might make students hate singing.

To discover this “side effect,” I have to measure different variables. That is:

I need to check how well they sing (one set of measurements),

AND how much they like singing (a different set of measurements).

It’s also possible that the downside takes longer to arise. The improvement (right now) results in less enjoyment of singing (later on). If I don’t keep measuring, I’ll miss this “side effect.”

New Habits

As Zhao argues, our habit of overlooking potential downsides creates real problems.

For instance, Zhao takes the example of Direct Instruction.

Its proponents can show lots of research suggesting its strengths. Its detractors likewise.

How can these contradictory realities exist?

Well, any complex teaching method will have benefits and detriments. If we focus only on one — if we measure only one — we’ll necessarily miss the other.

Instead, Zhao argues, we should develop the rigorous habit of looking for both: the benefits of any teaching strategy, and also its downsides.

This realisic, complex reality will allow us to make better decisions in classroom and schools.

One More Step

Although Zhao doesn’t mention “opportunity costs,” I think they’re an important part of this conceptual re-think.

That is:

Every time I do use a particular teaching strategy, I don’t use the other one.

If I take time for this stress-reducing technique, I don’t have time for that stress-reducing technique.

Even if a strategy has good research behind it, even if it has relatively few “side effects,” I always want to know: have I given up a better strategy to make time for this merely good strategy?

For example, this point often comes up in discussion of Learning Styles Theory.

If you’ve spent any time in this field, you know: Learning Styles Theory simply doesn’t have good research support behind it.

Alas: it has LOTS of popular support, even among teachers.

When I show teachers the comprehensive research reviews contradicting the theory, they occasionally respond this way:

“Okay, but what harm is it doing? It might be true, so why not teach to my students’ learning style?”

For me, the clear answer is opportunity cost.

If we teachers ARE spending time on teaching methods that have no research support, we ARE NOT spending time on those that do.

If students ARE studying on the treadmill because they’re “kinesthetic learners,” they ARE NOT using study strategies with research support behind them.

Measuring opportunity cost requires subtle and humble calculations. We just might have to give up a long-prized approach to make time for an even better one.

If our students learn more, that sacrifice will have been worth it.

TL;DR

Like medical researchers, we should look both for benefits and for potential harms of any teaching suggestion.

This balanced perspective might take additional time, and might require consideration of  opportunity costs.

It will, however, result in a more realistic and useful understanding of teaching and learning.

---

*  Many years ago, I read that the phrase “side effects” is misleading. It makes unwanted effects seem unlikely, even though they’re just as likely as the wanted effects.

For that reason, I’m putting the words “side effects” in quotations throughout this post.

I believe it was Oliver Sacks who made this point, but I can’t find the citation so I’m not sure.

If you know the correct source of this insight, please let me know!

---

Zhao, Y. (2017). What works may hurt: Side effects in education. Journal of Educational Change, 18(1), 1-19.

A surprising research finding to start your week: parachutes don’t reduce injury or death.

How do we know?

Researchers asked participants to jump from planes (or helicopters), and then measured their injuries once they got to the ground. (To be thorough, they checked a week later as well.)

Those who wore parachutes and those who did not suffered — on average — the same level of injury.

Being thorough researchers, Robert Yeh and his team report all sorts of variables: the participants’ average acrophobia, their family history of using parachutes, and so forth.

They also kept track of other variables. The average height from which participants jumped: 0.6 meters. (That’s a smidge under 2 feet.) The average velocity of the plane (or helicopter): 0.0 kilometers/hour.

Yes: participants jumped from stationary planes. On the ground. Parked.

Researchers include a helpful photo to illustrate their study:

Why Teachers Care

As far as I know, teachers don’t jump out of planes more than other professions. (If you’re jumping from a plane that is more than 0.6 meters off the ground, please do wear a parachute.)

We do, however, rely on research more than many.

Yeh’s study highlights an essential point: before we accept researchers’ advice, we need to know exactly what they did in their research.

Too often, we just look at headlines and apply what we learn. We should — lest we jump without parachutes — keep reading.

Does EXERCISE helps students learn?

It probably depends on when they do the exercise. (If the exercise happens during the lesson, it might disrupt learning, not enhance it.)

Does METACOGNITION help students learn?

It probably depends on exactly which metacognitive activity they undertook.

Do PARACHUTES protect us when we jump from planes?

It probably depends on how high the plane is and how fast it’s going when we jump.

In brief: yes, we should listen respectfully to researchers’ classroom guidance. AND, we should ask precise questions about that research before we use it in our classrooms.

A surprising research finding to start your week: parachutes don’t reduce injury or death.

How do we know?

Researchers asked participants to jump from planes (or helicopters), and then measured their injuries once they got to the ground. (To be thorough, they checked a week later as well.)

Those who wore parachutes and those who did not suffered — on average — the same level of injury.

Being thorough researchers, Robert Yeh and his team report all sorts of variables: the participants’ average acrophobia, their family history of using parachutes, and so forth.

They also kept track of other variables. The average height from which participants jumped: 0.6 meters. (That’s a smidge under 2 feet.) The average velocity of the plane (or helicopter): 0.0 kilometers/hour.

Yes: participants jumped from stationary planes. On the ground. Parked.

Researchers include a helpful photo to illustrate their study:

Why Teachers Care

As far as I know, teachers don’t jump out of planes more than other professions. (If you’re jumping from a plane that is more than 0.6 meters off the ground, please do wear a parachute.)

We do, however, rely on research more than many.

Yeh’s study highlights an essential point: before we accept researchers’ advice, we need to know exactly what they did in their research.

Too often, we just look at headlines and apply what we learn. We should — lest we jump without parachutes — keep reading.

Does EXERCISE helps students learn?

It probably depends on when they do the exercise. (If the exercise happens during the lesson, it might disrupt learning, not enhance it.)

Does METACOGNITION help students learn?

It probably depends on exactly which metacognitive activity they undertook.

Do PARACHUTES protect us when we jump from planes?

It probably depends on how high the plane is and how fast it’s going when we jump.

In brief: yes, we should listen respectfully to researchers’ classroom guidance. AND, we should ask precise questions about that research before we use it in our classrooms.

We don’t do a lot of cross posting here at Learning and the Brain. I believe this is the first time we’ve done so while I’ve been editor.

I think the initiative below is very exciting, and you — Learning and the Brain readers — are just the right audience to take advantage of it.

In this post, Ben Keep and Ulrich Boser of the Learning Agency Lab explain how we teachers can do valuable research in our own classrooms.

If that grabs your attention: read on!

---

New technologies can help educators become high-quality researchers. ​

When it comes to teaching, there are a million questions to ask about the nature of instruction.

What examples to use? What analogies to draw on? What sequences to teach new ideas? The people in the best position to both ask and answer these questions are often teachers.

Teacher-driven research isn’t new, but — at least in the U.S. — it’s relatively rare. Teaching loads are high and work hours are long, making teachers reluctant to lead education research projects, even when they want to. And, generally speaking, the U.S. school system is not set up to support teacher-driven research.

But in spite of the challenges, teachers want to engage in research. One survey found that over 90% of teachers wanted to influence the direction of research. And 59% wanted to participate in research themselves.

One way to engage is through action research, which certainly has its place in the field. And while the approach has clear benefits, it also has some limitations — like missing comparison groups.

A new kind of tool might help solve this problem. Over the next year, different learning platforms plan on offering tools to assist teachers in running their own research projects. Take ASSISTments ETRIALS Project. There’s already currently a small community of teachers who are performing independent classroom research on ASSISTments and that’s scheduled to expand.

RCE Coach also has plans to put out a version of their software this fall that will facilitate teacher use of the platform. They plan on fostering collaborations and providing workshops and other resources to support teacher research.

There’s also Carnegie Learning’s UpGrade platform. The company has plans to release an easy-to-use UI that lets teachers perform research on the platform. They’re particularly interested in testing whether letting students move ahead at their own pace benefits student outcomes.

These tools all help teachers run randomized controlled trials in their classrooms. That is, they help teachers to randomly assign students to different instructional conditions so that we can figure out which teaching approaches work best — and why.

Action Research Isn’t Action(able) Enough. Or why RCTs?

Current teacher-driven research efforts emphasize action research, which is an approach to deliberately reflecting on one’s own teaching practice with an aim to improve it. Under this model, teachers will often experiment with new teaching approaches, conduct interviews or surveys of students, and make detailed observations along the way. Often, the entire class makes a change, and the teacher reflects on whether the change was effective at improving learning outcomes.

This has led to a lot of fascinating work. But one of the limitations of action research is that, without a meaningful group comparison, it’s hard to know whether the proposed change made a difference.

Putting teachers in charge of running RCTs offers several intriguing benefits. First, teachers are likely to ask questions that researchers might not think of. The tests would also be in the context of a real classroom environment. And the results could be put into practice immediately.

Second, a wider group of teachers becoming involved with research might help bridge the research-practice divide. Teachers do not often learn about the science of learning during teacher training programs. Simultaneously, many teachers feel like existing education research is inaccessible, hard-to-understand, or simply not relevant.

Transparent randomly controlled trials would also give teachers the ability to hone their intuitions about instructional choices. By posting the study design before posting the results, teachers, researchers, and anyone else who was interested could make predictions about what’s likely to happen. This gives people the kind of practice they need to become expert forecasters.

Of course, the approach also comes with significant challenges. With average class sizes of around 25 students, a single class yields very small sample sizes for carrying out RCTs. Teachers also have varying experience with research methods. And it’s still unclear what platform features will best serve the teachers-as-researchers community, and which questions simply can’t be tested using learning platforms.

More Actionable ResearchRCTs In Action

Do students benefit from solving math problems with pencil and paper (as opposed to on a computer)?

Suppose we had a group of students perform a homework assignment where they solved problems with pencil and paper, while a comparable group of students solved the same homework problem on a computer (with no incentive to write it out). Would the first group learn more or less than the second?

A math teacher in Maine — Bill Hinkley — actually decided to test this very question, through an RCT. One group of students was encouraged to use paper and pencil, and had to turn in a piece of paper showing their work. The other group of students went through the homework problems as usual — through a computer screen. Both groups saw and submitted their answers through the same math platform: ASSISTments.

The result? Students who used paper and pencil outperformed those who didn’t by about 13 points. The difference was just shy of statistical significance, but suggestive given the small sample size (15 students in one condition, 12 in the other). Bill Hinkley plans to replicate and expand on the experiment in the near future.

Want To Join The RCT Teacher Research Community?

What would happen if we could scale up this style of research? There are 3.7 million teachers in the U.S. If just one percent of them started engaging in education research, there would be 37,000 teacher-researchers. The largest education research association, AERA, by comparison, has about 25,000 members.

Suppose each teacher-researcher only performed one experiment a year. That’s still 37,000 small experiments, run in realistic, noisy, classroom settings using rigorous research methods. Imagine what we might learn.

Interested in using RCTs in your classroom? Get in touch with us: Email Ulrich at [email protected]

We’re looking to build a community of teacher researchers who are doing this work in schools every day.