I hope that one of my strengths as a blogger is: I know what I don’t know — and I don’t write about those topics.
While I DO know a lot about cognitive science — working memory, self-determination theory, retrieval practice — I DON’T know a lot about technology. And: I’m only a few miles into my own A.I. journey; no doubt there will be thousands of miles to go. (My first foray along the ChatGPT path, back in February of this year, did not go well…)
Recently I came across research that looks at A.I.’s potential benefits for studying. Because I know studying research quite well, I feel confident enough to describe this particular experiment and consider its implications for our work.
But before I describe that study…
Guiding Principles
Although I’m not a student of A.I., I AM a student of thinking. Few cognitive principles have proven more enduring than Dan Willingham’s immortal sentence: “memory is the residue of thought.”
In other words, if teachers want students to remember something, we must ensure that they think about it.
More specifically:
- they should think about it successfully (so we don’t want to overload working memory)
- they should think about it many times (so spacing and interleaving will be important cognitive principles
- they should think hard about it (so desirable difficulty is a thing)
And so forth.
This core principle — “memory is the residue of thought” — prompts an obvious concern about A.I. in education.
In theory, A.I. simplifies complex tasks. In other words, it reduces the amount of time I think about that complexity.
If artificial intelligence reduces the amount of time I that I’m required to think about doing the thing, it necessarily reduces the amount of learning I’ll do about the thing.
If “memory is the residue of thought,” then less thinking means less memory, and less learning…
Who Did What?
Although discussions of generative A.I. often sound impenetrable to me, this study followed a clear and sensible design.
Researchers from the University of Pennslyvania worked with almost 1000 students at a high school in Turkey. (In this kind of research, 1000 is an unusually high number.)
These students spent time REVIEWING math concepts they had already learned. This review happened in three phases:
Phase 1: the teacher re-explained math concepts.
Phase 2: the students practiced independently.
Phase 3: the students took a test on those math concepts. (No book; no notes; nada.)
For all students, phases 1 and 3 were identical. Phase 2, however, gave researchers a chance to explore their question.
Some students (let’s call them Group A) practiced in the usual way: the textbook, their notes, paper and pencil.
Group B, on the other hand, practiced with ChatGPT at hand. They could ask it questions to assist with their review.
Group C practiced with a specially designed ChatGPT tutor. This tutor was programed not to give answers to students’ questions, but to provide hints. (There were other differences between the ChatGPT and the ChatGPT tutor, but this difference strikes me as most pertinent.)
So: did ChatGPT help?
Did the students in Groups B and C have greater success on the practice problems, compared to Group A?
Did they do better on the test?
Intriguing Results
The students who used A.I. did better on the practice problems.
Those who used ChatGPT scored 48% higher than their peers in Group A.
Those who used the ChatGPT tutor scored (are you sitting down?) 127% higher than their peers in Group A.
Numbers like these really get our attention!
And yet…we’re more interested in knowing how they did on the test; that is, how well did they do when they couldn’t look at their books, or ask Chatty questions.
In brief: had they LEARNED the math concepts?
The students who used regular ChatGPT scored 17% lower than their notes-n-textbook peers.
Those who used the ChatGPT tutor scored the same as those peers.
In brief:
A.I. helped students succeed during practice.
But, because it reduced the amount of time they had to THINK about the problems, it didn’t help them learn.
Case closed.
Case Closed?
In education, we all too easily rush to extremes. In this case, we might easily summarize this study in two sentences:
“A.I. certainly didn’t help students learn; in some cases it harmed their learning. Banish A.I.!”
While I understand that summary, I don’t think it captures the full message that this study gives us.
Yes: if we let students ask ChatGPT questions, they think less and therefore learn less. (Why do they think less? Probably they simply ask for the answer to the question.)
But: if we design a tutor that offers hints not answers, we reduce that problem … and eliminate the difference in learning. (Yes: the reseachers have data showing that the students spent more time asking the tutor questions; presumably they had to think harder while doing so.)
As a non-expert in this field, I suspect that — sooner or later — wise people somewhere will be able to design A.I. tutors that are better at asking thought-provoking hints. That is: perhaps an A.I. tutor might cause students to think even MORE than other students praticing the old-fashioned way.
That two sentence summary above might hold true today. But we’ve learned this year that A.I. evolves VERY rapidly. Who knows what next month will bring.
TL;DR
Although THIS study suggests that A.I. doesn’t help (and might harm) learning, it also suggests that more beneficial A.I. tutors might exist in the future.
If — and this is the ESSENTIAL “if” — if A.I. can prompt students to THINK MORE than they currently do while practicing, then well-established cog-sci principles suggest that our students will learn more.
* A note about the publication status of this study. It has not yet been peer reviewed and published, although it is “under review” at a well-known journal. So, it’s technically a “working paper.” If you want to get your research geek on, you can check out the link above.
Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, O., & Mariman, R. (2024). Generative ai can harm learning. Available at SSRN, 4895486.
