AI gets things wrong. Everybody knows this.

The running joke is not that AI fails, but that it fails with extraordinary confidence. It can be obviously, gloriously incorrect and still deliver the answer with the smooth assurance of someone who has never once doubted themselves in a meeting.

Funny in a demo. Less funny when the answer ends up in a report, a codebase, or a decision.

The useful thing to understand is that this is not random madness. Most of the time, AI gets things wrong for very understandable reasons. Once you see those reasons clearly, the behaviour becomes a lot less mysterious, and a lot easier to design around.

So… why does AI get it wrong?

At a high level, large language models do not retrieve truth from some neat internal filing cabinet. They generate responses from patterns learned during training and from the context available at the moment they answer. That is why OpenAI’s conversation state guide makes the point that text generation requests are independent and stateless unless prior state is explicitly preserved, while Anthropic’s context window guide describes the context window as the model’s “working memory”.

That distinction matters.

People often interact with AI as though it remembers everything, understands the whole problem, and has a stable internal model of the situation. Usually it does not. Usually it has whatever is in the current context, plus whatever patterns it learned before the conversation ever began.

That is powerful. It is also a very strange foundation for something people keep expecting to behave like a senior expert with perfect recall.

AI is not forgetful. It never knew.

A lot of AI error gets described as forgetting, but that is often the wrong mental model.

If the right facts are not in the prompt, not available through a tool, and not reliably recoverable from training, the model still has to produce something. That is where the trouble starts.

Sometimes the answer is vaguely right. Sometimes it is spectacularly wrong. Almost always it is delivered in polished, fluent prose that makes it sound more trustworthy than it is.

This is one of the reasons AI errors feel so odd. A normal software bug tends to fail loudly. A language model often fails smoothly. It gives you an answer that looks complete, sounds plausible, and only later reveals that it quietly invented a detail, merged two ideas, or filled a gap with a statistical best guess. Anthropic’s hallucination guidance is refreshingly blunt on this point: even advanced models can still generate outputs that are factually incorrect or inconsistent with the context they were given.

In other words, the model is not always remembering wrongly. Quite often, it never had the right information in front of it to begin with.

Long conversations are not long-term memory

This is where a lot of the confusion comes from.

A long chat can feel like memory because the model keeps referring to earlier parts of the conversation. But that is not the same thing as durable understanding. It is context management.

OpenAI’s documentation is explicit that text generation does not magically become stateful unless the application preserves or passes prior turns forward. Google’s long context guide for Gemini uses a short-term memory analogy for the context window, which is a useful way to think about it. The model can work with what is currently in scope. That does not mean it has formed a reliable, lasting memory in the human sense.

This matters because long threads drift.

Requirements get restated slightly differently. Assumptions pile up. Small mistakes go unchallenged and become part of the working context. A summary replaces a precise detail. Ten turns later, the model is not necessarily being irrational. It is operating on a version of the conversation that has already been compressed, blurred, or partially dropped along the way.

That is why serious AI systems spend so much effort managing context, not just writing clever prompts. The transcript is doing more heavy lifting than most people realise.

Bigger context windows help, but they do not perform miracles

One obvious response to all of this has been: fine, just give the model more context.

That absolutely helps. A larger context window means the model can consider more material at once. Google’s long context documentation shows how bigger windows expand what can be analysed in a single request, and Anthropic’s context window guide explains the same basic idea.

But bigger is not the same thing as perfect.

Anthropic makes the important point that more context is not automatically better context, and notes that accuracy and recall can degrade as token counts grow, a phenomenon it calls “context rot”. Google makes a similar point in its long context guidance: performance can vary when there are multiple details buried in large inputs rather than one obvious needle in the haystack.

So yes, long context moves the limit. It does not remove the limit.

If you hand a model a giant pile of notes, contradictory requirements, old decisions, and half-finished thoughts, you have not given it wisdom. You have given it a larger room in which to get confused.

Fluency is not verification

This is the part humans consistently trip over, because we are wired to read confidence as competence.

LLMs are very good at producing language that sounds like an answer. That is their job. They can present uncertainty in a neat paragraph, wrap a guess in the tone of authority, and fill missing logic with transitions smooth enough to make your brain stop asking questions.

That is why “confidently wrong” has become such a reliable joke. It keeps happening because the interface is persuasive. Good prose creates the impression of good reasoning. Sometimes that impression is earned. Sometimes it absolutely is not.

This becomes risky when people assume the model has checked something it has not actually checked. A citation not verified against a source. A requirement inferred from tone rather than read from documentation. A code change that looks reasonable but has not been tested against the real system. The answer looks finished, so people treat it as finished.

That is how plausible output turns into operational risk.

So what is really going on?

Most AI failure is not mysterious. It usually comes from a handful of predictable conditions:

• the model does not have the right facts in context
• the conversation has become too long or too messy
• important details have been compressed, dropped, or blurred
• the prompt is ambiguous enough to invite guesswork
• the model is asked to be helpful when it should really say, “I don’t know”

That does not make AI useless. It just makes it a system with limits, which is far less magical but much more useful.

The better way to think about it is this: AI is not a magical brain in a box. It is a probabilistic system operating on whatever information, instructions, and tools you give it at the time. When that setup is thin, noisy, or overloaded, the output suffers. Predictably.

Why this matters for software teams

In software delivery, these failure modes show up everywhere.

They show up when a coding model loses track of the architecture and patches the symptom instead of the cause. They show up when a long requirements conversation quietly drifts away from the original intent. They show up when a polished answer gets accepted because nobody wants to be the person who asks the robot to explain itself again.

That is the real issue. Not that AI sometimes gets things wrong, but that people keep treating fluent output as a substitute for grounded understanding.

And that is also why the conversation is shifting. The serious work is no longer just about better prompts. It is about better context, better retrieval, better validation, and better systems around the model.

Once you see AI failure in those terms, it stops looking like random magic gone wrong and starts looking like an engineering problem. The model is only ever as reliable as the facts, structure, and checks surrounding it. Thin context produces thin answers. Better systems produce better odds.

References

Anthropic

• Context windows: Anthropic describes the context window as a model’s working memory and notes that accuracy and recall can degrade as token counts grow, a phenomenon it calls context rot.
• Compaction: Anthropic recommends server-side compaction for long-running conversations and agentic workflows, replacing stale history with concise summaries.
• Reduce hallucinations: Anthropic provides guidance on reducing hallucinations through grounded quoting, clearer uncertainty handling, and auditable outputs with citations.

DashScope / Alibaba

• Long context (Qwen-Long): Alibaba Cloud shows file-based long-document handling through file IDs so large documents do not need to be resent with every request.
• Function Calling: Alibaba Cloud explains function calling as a way to handle tasks such as real-time information retrieval and calculations.
• Optimize RAG performance: Alibaba Cloud walks through common RAG pipeline issues across parsing, chunking, retrieval, recall, and answer generation.

Gemini / Google

• Long context: Google explains context as a form of short-term memory and discusses older workarounds such as dropping messages, summarising, and using RAG.
• Context caching: Google documents context caching as a way to reuse repeated context across requests.
• Grounding with Google Search: Google presents search grounding as a way to reduce hallucinations, access real-time information, and return cited responses.

Mistral

• Document Library: Mistral documents its built-in document library as a retrieval tool for uploaded files, with reference chunks available for citations.
• Function Calling: Mistral explains how models select tools, generate arguments, and use tool outputs in follow-up responses.
• Agents API: Mistral introduces its Agents API with web search and MCP-based tooling to provide up-to-date, evidence-supported context.

Ollama

• Context length: Ollama provides a plain-English explanation of context limits, including why agents, web search, and coding tools often need larger context settings.
• Embeddings: Ollama documents embeddings for semantic search, retrieval, and RAG workflows.
• Tool calling: Ollama explains how local models can invoke functions and run multi-turn agent loops.

OpenAI

• Conversation state: OpenAI explains that text generation requests are independent and stateless unless prior messages or conversation features are explicitly included.
• Compaction: OpenAI documents server-side compaction as a way to summarise and prune long conversations once they reach a threshold.
• Skills: OpenAI describes skills as versioned bundles of instructions, files, and scripts that package reusable procedures outside the main prompt.