한국어

Is ChatGPT Study Mode Good? What AI Tutor Studies Found

At a glance
  • No published experiment has tested ChatGPT study mode itself, as far as this note could find.
  • In a close test, classes with an answer-giving AI scored 17% lower on a later exam without AI.
  • Classes with a hint-only AI tutor lost nothing on that exam, but they did not gain either.
  • The AI tutors that worked had a teacher's worked solution for every problem. Study mode has none.
  • Give study mode your own notes and solutions, then recall what you learned the next day without AI.

No published experiment has tested ChatGPT study mode itself, as far as this note could find. But AI tutors built on the same idea, hints instead of answers, have been tested in two trials. They prevented the damage an answer-giving AI did, and a well-built one taught more than a strong classroom lesson. Both of those tutors carried a teacher's worked solution for every problem, and both measured learning only on the same day.

What does ChatGPT study mode do differently?

Study mode guides you with questions and hints instead of handing over the answer. OpenAI launched it on July 29, 2025.

According to OpenAI's announcement, study mode runs on custom system instructions written with teachers, scientists, and pedagogy experts. It uses Socratic questions and hints, breaks material into steps, checks your level, and quizzes you to confirm understanding. You can switch it on and off within a conversation.

OpenAI also named a limit: because it runs on instructions, its behavior can be inconsistent across conversations and it can make mistakes. The announcement reports no measurement of what study mode does to learning or memory. So the best evidence comes from trials of AI tutors that work the same way.

Why is an answer-giving AI a problem for learning?

It raised practice scores, then lowered scores on an exam taken without AI.

Bastani and colleagues (2025) ran a field experiment in a Turkish high school with about 50 math classes and nearly 1,000 students. Each class did practice problems with textbooks only, with a standard ChatGPT-style AI (GPT Base), or with an AI told to give hints rather than answers (GPT Tutor). Then, in the same session, everyone took a closed-book exam on similar problems.

The answer-giving AI raised practice scores but lowered exam scores without AI

  • Textbook only
  • Answer-giving AI
  • Hint-only AI
020406080Textbook only · Practice with AI · 28.428.4Answer-giving AI · Practice with AI · 42.142.1Hint-only AI · Practice with AI · 64.564.5Practice with AITextbook only · Exam without AI · 32.132.1Answer-giving AI · Exam without AI · 26.726.7Hint-only AI · Exam without AI · 31.731.7Exam without AIAnswer-giving AI:17% lower
Mean scores rescaled to 100. We computed each AI arm by adding its regression estimate to the textbook arm's mean. The hint-only arm's exam score was not statistically different from the textbook arm.Source: Bastani et al. (2025), Table 1
See the numbers
Textbook onlyAnswer-giving AIHint-only AI
Practice with AI28.442.164.5
Exam without AI32.126.731.7

With the answer-giving AI, practice scores rose 48%, but exam scores came out 17% lower than in the textbook classes. Students used it as a crutch, asking for and copying solutions, and they did not notice that they had learned less. With the hint-only AI, practice scores rose 127% and exam scores were no different from the textbook classes.

Two things follow. The hint-style tutor prevented the harm. It did not add learning on the exam. Yet those students believed they had done better. Feeling that study mode is working is not the same as learning more. More on this study is in Is Using ChatGPT to Study Bad?

Does AI tutoring actually work?

A carefully built AI tutor worked well. In a Harvard physics course, students learned more than twice as much from it as from an active-learning class.

Kestin and colleagues (2025) ran a crossover trial with 194 students in an introductory physics course in fall 2023. For one week, students had an active-learning class with group problem solving. For the other week, they learned the same content at home with an AI tutor. They took a quiz before and after each lesson.

Quiz scores rose more after the AI tutor than after the active-learning class

012345Median quiz score · Baseline before · 2.752.75Baseline beforeMedian quiz score · Active-learning class · 3.53.5Active-learning classMedian quiz score · AI tutor · 4.54.5AI tutorAI tutor gain wasmore than double
Median quiz scores right after the lesson. The baseline is the pre-lesson score of both groups combined. We computed the gains (1.75 vs 0.75).Source: Kestin et al. (2025), Results
See the numbers
Median quiz score
Baseline before2.75
Active-learning class3.5
AI tutor4.5

The AI group also spent less time: the class ran 60 minutes, and the AI group's median was 49 minutes. The authors estimated an effect size of 0.73 to 1.3 standard deviations. Older evidence points the same way. A review of tutoring studies found intelligent tutoring systems nearly as effective as human tutors.

Intelligent tutoring systems came out nearly as effective as human tutors

  • Commonly believed
  • Found in the review
00.511.522.5Commonly believed · Human tutoring · 22Found in the review · Human tutoring · 0.790.79Human tutoringCommonly believed · Intelligent tutoring systems · 11Found in the review · Intelligent tutoring systems · 0.760.76Intelligent tutoringsystems0.76 vs 0.79,nearly the same
Effect size (d) compared with learning the same content without tutoring. "Commonly believed" is the value the review says was widely assumed at the time.Source: VanLehn (2011), abstract
See the numbers
Commonly believedFound in the review
Human tutoring20.79
Intelligent tutoring systems10.76

What did the AI tutors that worked have?

A teacher-written solution for every problem. Study mode has general instructions only, and no problem-specific solutions unless you supply them.

Both AI tutors that worked carried a worked solution, written by people, for every problem.

Hint-only AI (Bastani et al.)AI tutor (Kestin et al.)ChatGPT study mode
Problem solutionsSolution and suggested hints in the promptStep-by-step solutions in the promptNone (you supply them)
How it guidesHints, no direct answersWalks through each part in orderQuestions, hints, quizzes
What was measuredExam in the same sessionQuiz right after the lessonNo published results

Bastani et al. (2025) noted that the standard AI often gave wrong answers, while the hint-only AI held the solution, so students could check their work against it. Kestin et al. (2025) added full solutions to avoid trusting answers the model invented. They also found that instructions alone could not walk students through multi-part problems in order, so the platform fed the parts one at a time. In our reading, giving study mode your textbook or answer key is the simplest way to close that gap.

Does what you learn in study mode last?

Nobody knows yet. Both trials measured learning on the same day.

The Bastani et al. (2025) exam came right after practice, in the same session. The Kestin et al. (2025) quiz came right after the lesson. Kestin et al. list adding retention strategies such as spacing as future work.

Tests right away and tests days later can disagree. In Roediger and Karpicke (2006), rereading beat recall practice on a test 5 minutes later, but on tests 2 days and 1 week later, students who had practiced recall remembered substantially more. The rereaders were also more confident. A smooth session in study mode can carry the same trap.

How should you use study mode?

Give it your materials, answer first, and recall the material the next day without AI. All of this works without study mode too.

  1. Hand over your notes and solutions. Paste lecture notes or the answer key and ask: "Use this solution, and give me hints only." That is what both successful tutors had.
  2. Write your own answer before asking. A wrong guess is fine. Take the hint after.
  3. Hold off once when you want the answer. Study mode can be switched off, so it is easy to grab the answer when stuck. If you do, close the screen and solve it again in your own words.
  4. Recall it the next day without AI. Pick three things from yesterday that you want to keep and explain them with nothing in front of you. Repeat a few days apart to make them last. Intervals are in Spaced repetition schedule.

In Aloom

If study mode helps with understanding on the day, Aloom takes over from the next day. You ask AI, keep only the answers worth keeping, and each one joins what you already know in your graph. Each day you recall 1, 3, or 5 of them from a question, before you see the answer. The more you recall, the darker the ink. It covers the days-later memory that the trials above did not measure. See how Aloom works

FAQ

Does ChatGPT study mode work?

No study has tested it directly. Trials of AI tutors built the same way found that hint-only guidance prevents the losses of answer-giving AI (Bastani et al., 2025), and that a tutor with expert-written solutions can beat a strong class on a same-day quiz (Kestin et al., 2025).

Is study mode different from asking ChatGPT for hints only?

In principle, no. Study mode also runs on system instructions. In the trials, what mattered was hint-first guidance and a worked solution behind it, not the name of a mode. Either way, give it your materials.

Can an AI tutor replace a human tutor?

The evidence is partial. In VanLehn (2011), intelligent tutoring systems (0.76) were nearly as effective as human tutors (0.79), but those were systems built before generative AI. Kestin et al. (2025) suggest using AI tutors to bring everyone up to speed before class, not to replace it.

References