Summary LLMs loop endlessly on simple tasks instead of admitting failure — a result of training incentives that reward guessing over honesty.

LLMs trained on their own generated data suffer “model collapse,” quietly eroding performance on rare, real-world tasks.

The Pain Point

I asked Gemini Pro, connected to Android Studio via the Gemini API, to add a logo to my app. It should have been simple: the image files were already in the resource folder. Instead, the model kept iterating on the idea without ever executing it.

If you’re an Android developer, you know changing a logo is a trivial task: (1) add the file to Image Assets, (2) configure the foreground layer by pointing it to the new image, (3) configure the background layer’s color. Even with the assets already in place, Gemini couldn’t complete the change. Worse, it never fell back to simply telling me the manual steps — steps easily found with a quick Google search. It failed twice over: once at the task, once at admitting it couldn’t do the task.

Claim 1: LLMs Are Trained to Finish, Not to Flag

This points to a deeper, now well-documented issue. OpenAI’s 2025 research on why language models hallucinate found that standard training and evaluation procedures reward confident guessing over acknowledging uncertainty. Just as a multiple-choice test rewards a guess over a blank answer, models are trained on benchmarks that give zero credit for saying “I don’t know” — so, over time, they learn to always produce something, even when that something is a repeated, failed loop (Kalai et al., 2025). A model that says “I can’t reliably do this — here’s how to do it yourself” would arguably be more useful than one that loops through failed attempts, but current training incentives don’t reward that kind of honesty.

Claim 2: The Copy-Machine Problem

There’s a second, related issue: benchmark performance may be masking a real decline in handling novel, simple tasks. Researchers at Oxford and Cambridge documented this formally in 2024, coining the term “model collapse” (Shumailov et al., 2024) — when models are trained recursively on data generated by earlier AI models rather than fresh human data, they progressively forget the tails of the true data distribution, converging toward repetitive, lower-variance outputs. Crucially, this degradation happens even with fine-tuning and “ideal” training methods, and it compounds across generations.

Benchmarks are often built from patterns already well-represented in web data, so models can ace them through repeated exposure — without meaningfully improving at small, contextual, real-world tasks that live in the “tail” of the distribution, like configuring a specific IDE workflow that isn’t heavily documented online.

Why This Matters

Individually, each claim explains a piece of the puzzle. Together, they explain why my logo problem wasn’t a fluke. Claims 1 and 2 aren’t separate problems — they compound each other. Training objectives reward LLMs for producing confident-sounding output rather than admitting uncertainty, while model collapse research shows that recursive training on AI-generated data quietly erodes a model’s grip on rare, real-world edge cases. Put together, this means the errors are invisible exactly where they matter most: a model that fails silently at a simple task like changing a logo, instead of saying “I’m not sure, here’s how to check manually,” is failing in the very cases current benchmarks don’t measure. The uncomfortable conclusion is that “getting smarter” on paper (higher benchmark scores) and “getting better” at helping real users with real, messy tasks may be quietly diverging — and until training incentives change to reward honest uncertainty, that gap will keep showing up in ordinary workflows like yours.

“Have you run into a similar simple-task failure? I’d like to hear about it.”

References: Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. and Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature, 631(8022), pp.755–759. doi:10.1038/s41586-024-07566-y.

Kalai, A.T., Ofir Nachum, Vempala, S.S. and Zhang, E. (2025). Why Language Models Hallucinate. [online] doi:10.48550/arXiv.2509.04664.