AI boosts performance but not always learning: the performance–learning paradox

Performance and learning are two different scores

When you ask a model to solve an equation, write a function, or retell a chapter, you are measuring performance: how well the task is done now, with the tool available. Learning is a different score. It answers: Can I repeat the solution myself? Will the principle hold a week later? Will it transfer to a slightly different task? Can I explain the path to another person?

These scores diverged long before chatbots. A calculator raises arithmetic performance but does not replace place-value understanding. Google search raises access to facts but still requires formulating a query, comparing sources, and assembling an answer. Generative AI takes the next step: it delivers a finished solution, not just links. That is why the gap between “it worked” and “I learned” has become especially sharp.

In HiddenLogic terms this is close to desirable difficulties and active recall. Effort, error, searching for a pattern, and trying again are not annoying noise. They are part of how skill forms. If AI removes all useful friction, performance rises while learning can stagnate or fall.

What the GPT-4 high-school experiment showed

In 2025, Proceedings of the National Academy of Sciences published work by Hamsa Bastani, Osbert Bastani, and colleagues: Generative AI without guardrails can harm learning: Evidence from high school mathematics (DOI 10.1073/pnas.2422633122). This is a field experiment with nearly a thousand high-school students (mathematics; popular accounts often note a Turkish school and Wharton/Penn involvement).

The authors compared three conditions. Control learned without generative AI. The GPT Base group got a ChatGPT-like interface on GPT-4. The GPT Tutor group — the same model class, but with pedagogical guardrails: hints and guidance, not a finished answer on the first request. AI covered a notable share of practice on the topic (on the order of 15% of the instructional material in the study description).

On practice with access to the model, the picture looked like an “AI win”:

So unconstrained GPT-4 helped perform tasks and harmed learning to perform them. The authors explicitly tie this to a “crutch” mode: students get the right answer, build less of their own skill, and on a closed-book assessment without the model end up weaker than those who worked without AI from the start.

The critical detail: for GPT Tutor, the negative exam effect largely disappeared. On practice, Tutor produced an even larger gain, and on the exam without AI the group was comparable to control — so the guardrails preserved learning without canceling help. The study’s conclusion is not “ban AI,” but “be careful with design”: without guardrails, a generative model can worsen educational outcomes.

It is worth saying separately about the “NVIDIA study.” A high-profile NVIDIA paper with a claim along the lines of “people’s academic performance falls because of AI” is not found in open sources. NVIDIA does have educational cases and tools for pedagogical assistants (hints instead of ready answers) — that lines up more with GPT Tutor logic than with the myth of a corporate study that “AI makes us dumber.” Memory often blends PNAS/Wharton, other academic work, and the NVIDIA brand.

Why a finished answer gets in the way of learning

When the model immediately delivers a solution, the brain skips the stretch of work on which skill usually forms: holding the goal, choosing a strategy, spotting an error, returning to the principle. This is a relative of cognitive outsourcing — handing part of thinking to an external tool. Outsourcing is useful when the goal is speed and product quality. It is harmful when the goal is precisely to form an internal procedure.

Useful friction in learning includes error, the effort of recall, comparing options, and self-explanation. Metacognition is exactly about noticing: “is this easy because I understand — or because someone understood for me?” A smooth AI explanation almost always feels like understanding. That is the classic illusion of competence: recognizing someone else’s solution is confused with the ability to generate your own.

Google search was less “dangerous” precisely because it left more independent operations: formulate a query, open several pages, filter noise, assemble an answer. Generative AI compresses that chain into one message. That is why, next to this guide, a breakdown of AI summaries is useful: a finished text is a hypothesis until you check the facts and rebuild the logic in your own words.

But AI does not “worsen education” by default

The picture does not reduce to one school experiment. A meta-analysis by Chen and Cheung (Educational Research Review, 2025) pooled 57 studies and 97 estimates of GenAI’s effect on university students’ outcomes and found a large positive overall effect (g+ ≈ 0.804), including for academic performance and higher-order thinking; for metacognition the effect in that paper was not statistically convincing.

A fresh three-level meta-analysis from 2026 (Frontiers in Psychology / PMC) included 36 empirical papers, 132 effect sizes, and 7229 participants. The overall GenAI effect on learning outcomes in higher education was medium positive (g = 0.499). Understanding / cognitive-creative outcomes and higher-order learning looked stronger; a significant moderator was teaching method: collaborative and blended scenarios produced larger effects.

These results do not refute PNAS. They refine the context. Meta-analyses more often measure outcomes when AI is present in instructional design — feedback, explanations, joint work — not uncontrolled answer-copying. The Bastani et al. experiment deliberately separates performance while access is open from learning after it is removed. Until you measure both layers, “AI helped” and “the person learned” are easy to confuse.

What PISA 2025 data hint at

Volume I of the PISA 2025 results (OECD) includes questions about students’ AI use. The picture is nonlinear and cautious in interpretation.

Students who do not use AI chatbots for specific school tasks (reading summarization, drafting text, preliminary search) on average show higher science scores than those who use AI for those tasks. Among those who use AI with the goal “help me learn,” the picture is different: roughly weekly use is linked to results close to non-users (after accounting for socioeconomic profile), while very rare or almost daily use is more often linked to lower ones. When schools have tasks that assess the quality of AI-generated information, frequent “learning” users look a bit better — but such opportunities are more common for socioeconomically advantaged students, and the OECD explicitly warns about the risk of a new gap in AI literacy.

Critically: these are correlations, not proof that AI “caused the scores.” Students who choose different modes of use already differ in skills, motivation, and support. Still, the pattern matches the experimental logic: purpose and intensity matter more than the bare fact of “having ChatGPT.”

Three AI modes: calculator, teacher, examiner

It helps to choose the model’s role explicitly, instead of “just opening the chat.”

AI as calculator. “Solve / write / do it for me.” Maximizes performance. Risk to learning is highest if that exact task was supposed to train the skill.

AI as teacher. “Don’t give the answer. Give one hint. Ask what I’ve already tried. Point to the error in my reasoning.” Closer to GPT Tutor and to what meta-analyses call productive embedding.

AI as examiner. “Hide the solution. Ask me five questions. Check my answer without hints. Give a similar problem.” This moves active recall into a dialogue with the model — provided you actually answer before peeking.

HiddenLogic’s practical guide to sessions, fact-checking, and cards is in “How to study with AI”. Here the strategic choice matters more: which role you assigned the model on this task.

Programming: where the paradox is especially visible

AI writes a function, a component, SQL, and tests in seconds. Performance soars. At the same time, independent design, reading errors, working with documentation, and holding APIs in memory shrink — exactly the loops from which engineering skill grows.

The developer’s new competence shifts: not “write code without AI at all costs,” but understand architecture, formulate the task, check the result, find the model’s mistakes, read generated code, and make engineering decisions. If you never close the chat and never reproduce the solution yourself, you train prompt formulation, not mastery of the stack — and that is fine only if the prompt is your actual goal.

The same five-question test after a task works here too: what did I do, what did the model do, what new did I understand, can I repeat without AI, can I transfer to a neighboring case.

Protocol: AI without losing the skill

Before AI

Spend 5–15 minutes working on your own. State the problem in your own words. Write down a hypothesis or rough plan. Even a wrong attempt creates “hooks” for the model’s explanation to land on later.

During AI

Ask for hints, not a finished solution from the first reply. Check every important claim. Ask clarifying questions (“why this step, not another?”). If the model delivered code or a proof in full — don’t copy at once: retell the next step, then check.

After AI

Close the chat. Reproduce the solution yourself. Solve a modified version of the problem. Explain the principle in your own words — to yourself, a colleague, or back to the model in examiner mode. Without this step you bought performance and did not pay for learning.

Five check questions after a serious task:

1. What did I do?
2. What did AI do?
3. What new did I understand?
4. Can I repeat it without AI?
5. Can I apply it in a new situation?

If the answer to 4–5 is “no,” AI helped complete the task but did not help form the skill.

AI as a brain exoskeleton — and as a crutch

A calculator is an exoskeleton for computation. Search is an exoskeleton for memory. Generative AI is an exoskeleton for reasoning: it can carry part of planning, wording, and draft logic. The difference in use mode is the same as between a training aid and a crutch. If you walk on crutches all the time, muscles weaken. If you use support to train movement under the right load, strength grows.

Hence the educational shift. When a machine can reproduce answers, the value of mere reproduction falls. The price of understanding, checking, framing the problem, and explaining rises — themes HiddenLogic already covers in critical thinking and in the article on what to learn in the AI era.

Common mistakes

Treating a rise in practice scores as proof of learning

Practice with an open chat measures the joint system “human + model.” An exam without the model measures the human. Confusing those metrics is the main source of false optimism.

Asking for a finished answer “to understand faster”

Often you recognize the solution faster and learn to build it more slowly. A hint and a critique of your attempt are almost always better than a clean solution if the goal is skill.

Trusting the feeling “now it’s all clear”

Clarity after someone else’s explanation is a weak signal. A strong signal is closed reproduction and transfer to a new task.

Using AI only as a calculator, then being surprised by failure without it

That is a predictable outcome under the PNAS experiment’s logic, not a personal “inability at math.”

Ignoring fact-checking

Even in tutor mode the model can be confidently wrong. Learning on a false explanation is worse than slow learning from a correct source.

FAQ

Further reading

• How to study with AI: a complete guide — sessions, fact-checking, cards
• How AI changes what is worth learning — the shift in knowledge scarcity
• Active recall — retrieval without a hint
• Desirable difficulties — why useful effort feels worse
• Metacognition in learning — calibrating “understood / seems understood”
• How to read AI summaries — not losing layers of evidence
• Critical thinking: a complete guide — checking claims and sources

Conclusion

The main problem is not that AI answers too well. The problem starts when a person stops asking: “could I do this myself?”

AI can be a crutch, a calculator, a personal teacher, or a thinking amplifier. The difference is decided not by model power, but by which cognitive work you keep for yourself. The GPT-4 high-school experiment showed the performance–learning split in clean form. Meta-analyses and PISA add nuance: with the right design and a moderate learning purpose the effect can be positive — but that is not a free pass to outsource thinking.

AI can make a person more effective without making them more competent. The question of the era is not “can the machine do this for me?” but “what will be left in my head after the machine does it?”