ChatGPT Practice Tests: Do AI Quiz Questions Help You Learn?
- An AI-made practice test is still a practice test. Answering from memory is what does the work.
- With the right answer shown after each question, short-answer practice did best on a final test three days later.
- Wrong options you read on a multiple-choice test can come back later as your answers.
- In one 2023 study, 25% of 60 questions written by ChatGPT 3.5 had wrong or misleading answers.
- Ask for short-answer questions, check each answer against the source text, and retake them two or three days later.
An AI-made practice test helps you learn, as long as you answer it from memory. What decides how much it helps is how you take it, not who wrote it. Ask for short-answer questions instead of multiple choice, check each answer against your own notes, and take the same questions again a few days later. Some AI-written questions have wrong answers, so check the answer in the source, not in the chat.
Does studying with AI practice questions work?
Yes. Whoever wrote the question, pulling the answer out of memory is what does the work.
Adesope and colleagues (2017) ran a meta-analysis of practice testing experiments. Practice tests helped learning more than restudying and every other comparison condition, and how much they helped depended on features of the practice test. The rest of this note is about those features.
One study put ChatGPT-made questions into a course. Yusof (2025) had second-year education students at a Malaysian university answer questions without notes for 10 to 15 minutes in class, after the lecture, followed by the answers and a discussion. Most questions were written by the lecturer. In one class session, students wrote questions with ChatGPT, which the lecturer reviewed and revised. In the revision week before the final, students answered the same questions again and used ChatGPT to check their answers.
| Group | Students sampled | Final exam mean |
|---|---|---|
| Earlier classes (no ChatGPT practice) | 64 of 197 | 25.44 |
| Later classes (ChatGPT practice) | 64 of 186 | 28.81 |
The difference was statistically significant. But this compares earlier and later classes, and their final exams had the same format but were not identical, so it cannot separate the AI from the weekly practice and the second round. Students wrote ChatGPT questions in only one session, and the lecturer corrected them first.
Multiple choice or short answer: which should you ask for?
Short answer. When students saw the right answer during practice, short-answer practice led to the best performance three days later.
Kang, McDermott and Roediger (2007) had students read four short journal articles. After each article they took a multiple-choice test, took a short-answer test, read statements with the same facts, or did an unrelated task. In the second experiment, students got corrective feedback during these first tests. Three days later everyone took a final test with both multiple-choice and short-answer questions.
With feedback, short-answer practice led to the best performance on the final test. The authors took this to mean that the more demanding the retrieval, the more it helps later.
The first experiment, with no feedback, came out differently. Multiple-choice practice led to better final scores than short-answer practice, and it did not differ significantly from simply reading the statements. The authors added feedback in the second experiment to even out how much of the tested information each group saw.
Short-answer questions and checking the answer go together. Without the check, short answers can do worse.
What happens if you skip checking the answers?
Some of the wrong options you read can come back later as your answers.
Roediger and Marsh (2005) found that taking a multiple-choice test helped people on a later test, and they also found the cost. The more wrong options people had read, the smaller the benefit, and the more often they gave those wrong options as answers later. They wrote that multiple-choice testing may create false knowledge without anyone meaning to.
Taking an AI quiz, glancing at the score and moving on leaves those wrong options unchecked. For what to do with the questions you miss, see Mistake Notebook: Why You Get the Same Questions Wrong.
Can you trust the questions AI writes?
Many are fine, but some are wrong. Check the answers in your source.
Ngo and colleagues (2024) pasted immunology study guides into ChatGPT 3.5 in January and February 2023 and asked it to write four multiple-choice questions with explanations, 60 questions in all. Four reviewers graded them, and the harshest grade counted.
A quarter of 60 ChatGPT 3.5 questions had wrong or misleading answers
See the numbers
| Share of questions | |
|---|---|
| Usable with minor fixes | 32% |
| Needed major edits | 43% |
| Wrong or misleading | 25% |
Another study came out more hopeful. Cheung and colleagues (2023) mixed 50 medical multiple-choice questions written by ChatGPT from textbooks with 50 written by two professors, and five assessors from the UK, Ireland, Singapore and Hong Kong graded them without knowing who wrote which. The AI questions scored a little lower only on relevance (7.56 vs 7.88 out of 10), and the total scores did not differ significantly. ChatGPT took 20 minutes 25 seconds; the professors took 211 minutes 33 seconds. The AI questions' scores were also spread more widely than the professors' questions.
ChatGPT wrote 50 exam questions in about a tenth of the professors' time
See the numbers
| Minutes | |
|---|---|
| ChatGPT | 20.4 |
| Two professors | 211.6 |
Both studies used 2023 models, and today's models may do better. This note did not find a study that graded current models the same way. Either way the advice holds. The person taking the quiz cannot tell which question is wrong, so check each answer against a sentence in the source, not against what the AI says.
How can you try this today?
Take one chunk of what you studied today. Any AI will do.
- Ask for short answers. Paste the material and say: "Using only this material, write 8 short-answer questions. For each, note the answer and the exact sentence from the material it comes from, but don't show them until I've answered. Ask one question at a time."
- Close the material and write your answer. Write at least a line, not just in your head. If you don't know, write that.
- Check the answer against the source sentence. If the quoted sentence isn't in your material, or says something else, drop the question.
- If you use multiple choice, check the wrong options too. After you pick, read why each option you didn't pick is wrong, and look hardest at the wrong one you chose.
- Retake the same questions two or three days later. Start with the ones you missed. The final test in the study above came three days later.
If you'd rather have the AI guide you with questions than hand you answers, see Is ChatGPT Study Mode Good?.
In Aloom
In Aloom, your recall questions are made from the answers you chose to keep after asking AI. Each day you answer 1, 3 or 5 of them, recalling before you see anything. They are multiple choice and true or false, so after you answer, Aloom shows why the right one is right and why the closest wrong one is not, so that a wrong option doesn't settle in as a memory. Each question comes from one answer you kept, so you always know where to check. The more you recall, the darker the ink. See how Aloom works
FAQ
Can ChatGPT make practice tests?
Yes. Ask for questions drawn only from your material, in short-answer form, with the answer and its source sentence held back until you have answered. The source sentence lets you check whether the AI's answer is right. In a 2023 study, a quarter of ChatGPT 3.5's questions had wrong or misleading answers (Ngo et al., 2024).
Is multiple choice or short answer better for studying?
Short answer, if you check the answers. In Kang and colleagues (2007), short-answer practice with feedback did best on a final test three days later. Without feedback, multiple-choice practice did better than short answer. But wrong options you read on a multiple-choice test can come back later as answers (Roediger & Marsh, 2005).
Are AI generated practice questions as good as a teacher's?
Sometimes. In one study, ChatGPT's medical questions scored about the same as professors' overall, but varied more (Cheung et al., 2023). In another, only 32% were usable with just minor formatting fixes (Ngo et al., 2024). Treat each question as a draft and check its answer in your source.
References
- Kang, S. H. K., McDermott, K. B., & Roediger, H. L. (2007). Test format and corrective feedback modify the effect of testing on long-term retention. European Journal of Cognitive Psychology, 19(4–5), 528–558.
- Roediger, H. L., & Marsh, E. J. (2005). The positive and negative consequences of multiple-choice testing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(5), 1155–1159.
- Adesope, O. O., Trevisan, D. A., & Sundararajan, N. (2017). Rethinking the use of tests: A meta-analysis of practice testing. Review of Educational Research, 87(3), 659–701.
- Yusof, I. J. (2025). ChatGPT-assisted retrieval practice and exam scores: Does it work? Journal of Information Technology Education: Research, 24, Article 8.
- Ngo, A., Gupta, S., Perrine, O., Reddy, R., Ershadi, S., & Remick, D. (2024). ChatGPT 3.5 fails to write appropriate multiple choice practice exam questions. Academic Pathology, 11(1), 100099.
- Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., et al. (2023). ChatGPT versus human in generating medical graduate exam multiple choice questions—A multinational prospective study (Hong Kong S.A.R., Singapore, Ireland, and the United Kingdom). PLOS ONE, 18(8), e0290691.