NON-SATIRICAL REPORT · Field Notes

When AI Help Ends, the Next Problem Can Get Harder

A study of short online tasks found that an assistant could raise performance while it was available, yet leave participants less successful on the next unaided questions. The result is an immediate design question, not a verdict on AI or learning itself.

By Field Notes Editorial Desk · October 27, 2026

Three short randomized online experiments measured performance immediately after a prepared GPT-5 assistant was removed; they do not establish lasting effects, effects across all AI systems, or effects in classrooms or jobs.

The practical question is not whether help is bad. It is whether the help leaves a person ready to continue when it is gone.

A new study gives a concrete form to a familiar unease about AI assistance. A tool can make an answer arrive faster. It can also change what happens when the next question must be answered without it. The study does not show that using AI causes permanent deskilling, and it does not judge every use of AI. It does show a short-term, task-bound pattern that schools, trainers, and product teams should take seriously: the kind of help offered now may matter for the next unaided attempt. [1] [2]

What the researchers tested

The conference paper reports three randomized online experiments involving 1,222 U.S.-based participants. Two experiments used fraction arithmetic and the third used SAT-style reading comprehension. During the initial problems, participants assigned to the AI condition had access to a sidebar presented as ChatGPT and powered by a prepared GPT-5 assistant. The assistant was then removed, and participants in both conditions completed final problems without assistance. This creates a useful comparison: after the same immediate test, did people who had been able to consult the assistant perform differently from people who had not? [1]

The help was not treated as pointless. The study found that AI availability improved performance while the tool could be used. Its central result concerns the turn after that help disappeared. In the first fraction experiment, 354 participants completed 12 initial problems and then three final unaided problems. The AI-access group solved 0.57 of the final problems on average, compared with 0.73 for the control group. The AI-access group also skipped 0.20 of those final problems, compared with 0.11 in the control group. The paper reports both differences as statistically significant. [1]

The second fraction experiment was larger and added a pretest plus a visually matched sidebar for the control condition. Its final solve rate was 0.71 for the AI condition and 0.77 for the control condition, a reported statistically significant difference. The aggregate skip rates were 0.10 and 0.07, respectively, but that difference was not statistically significant. The reading-comprehension experiment showed the same broad immediate pattern: final solve rates of 0.76 with prior AI access and 0.89 for control, with skip rates of 0.08 and 0.01. [1]

Help has a design, not just a presence

The sensible response is not to make assistance disappear. A direct answer can be exactly the right help when a person needs access, reassurance, translation, error checking, or timely completion. In many real settings, speed and a correct result are legitimate goals. The study instead raises a more precise question for any setting that also values later independent performance: is the assistant taking over the step a learner needs to practice, or is it helping that learner take the step? [1]

One result points toward that distinction, with an important warning. In the second experiment, 61% of analyzed participants in the AI condition said they had mainly used the assistant for direct answers, while 27% reported using it for hints or clarification and 12% reported no use. The direct-answer group had the weakest final results. The hint group did not show a statistically significant impairment relative to the control group. But those are observational subgroups within the AI condition, based on participants' use and self-report. They were not randomly assigned to answer-first or hint-first assistance, so the finding cannot establish that the prompting style caused the difference. [1]

That limitation does not make the distinction uninteresting. It identifies a testable design choice. A tutor, course, or workplace assistant can offer a worked solution, a partial hint, a request to try a step first, or an explanation after a response. Those choices should be judged against the purpose of the task. Completion may be the right outcome for one task; retained unaided competence may be the right outcome for another. The study offers evidence that those outcomes should not be assumed to coincide. [1]

What this study cannot settle

The experiments lasted about 13 to 15 minutes and measured performance immediately after the tool was removed. They did not measure months of use, course grades, job performance, creativity, mental health, or a lasting loss of ability. The participants were U.S.-based people recruited online, the tasks were selected arithmetic and reading questions, and the assistant was a particular prepared GPT-5 system. Different populations, subjects, assistance designs, stakes, and models could produce different results. [1]

Nor should the paper's use of persistence be turned into a judgment about character. In this study, persistence is reflected in behavior on final questions, including attempting, skipping, and solving them. The authors discuss rapid answers and expectations of immediacy as a possible explanation for the pattern, but the experiments did not directly measure that mechanism. The randomized design supports an immediate comparison between access conditions. It does not identify why every participant responded as they did. [1]

The next useful tests

The clearest next experiment would randomly assign people to answer-first, hint-first, worked-example, and no-assistant conditions, then test unaided performance days or weeks later. It should also ask when a required period of struggle is inappropriate, especially for learners using accommodations or people facing a consequential deadline. A better evidence base would distinguish the value of fast assistance from the value of practice, rather than treating either as universal. [1] [2]

For institutions already deploying assistants, practical questions come first. What does a user see when they are stuck? Can a learner choose a hint before a full answer? Does a training program measure anything after the assistant is removed? And when an interface is designed for completion, does anyone check whether it still prepares a person for the next problem? This study does not answer those questions. It makes them harder to avoid. [1]

Source: Berkeley News