Memory Persistence in LLMs

Why statelessness is a feature, not a bug, and what you lose for it

This document explains memory persistence in large language model (LLM) systems from the basics up to architectural design tradeoffs. It distinguishes between documented academic and industry work and architectural assumptions or interpretations.

The central thesis is:

LLM inference is often stateless by design because stateless workers are easier to scale, batch, schedule, isolate, and recover. Useful AI assistants, however, are usually stateful at the surrounding system layer.

A chatbot that "forgets" is often not failing because the neural network lost a human-like memory. More commonly, the system failed to preserve, retrieve, summarize, or reinsert the relevant state into the model's current context.

1. First principle: an LLM is not a database

A standard LLM at inference time can be understood as a function over tokens and fixed model weights:

next_token ~ M_theta(current_prompt_tokens)

The weights theta contain what the model learned during training. This is often called parametric memory. But during ordinary inference, the weights are not updated after a user says something.

For example, if a user says:

My daughter's name is Mira.

The model can use that fact while it is present in the current prompt or context window. But the base model has not learned that fact in the same durable way a database would store a new row.

This distinction is documented in the retrieval-augmented generation literature. Lewis et al. describe pretrained model weights as parametric memory and an external dense index, such as Wikipedia, as non-parametric memory. RAG exists precisely because model weights alone are limited for updateable, provenance-bearing knowledge. See Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.¹

So there are several different things people casually call "memory":

These are not interchangeable.

The original Transformer architecture replaced recurrence with self-attention over the input sequence. See Vaswani et al., Attention Is All You Need.² That design is powerful and parallelizable, but it means that anything the model should consider must generally be either:

1. in the model weights, or
2. in the input context assembled for that inference run, or
3. available through tools or external state that the application retrieves and inserts.

2. What stateless inference means

At the architectural level, stateless inference means:

Each model invocation is computed from the request's provided input, the fixed model weights, and temporary compute caches derived from that input. The model server does not need to remember semantic user state from previous requests in order to answer the next one.

That does not mean the entire AI product is stateless. A chat product may store messages, user preferences, uploaded files, tool results, summaries, or task state. But that state usually lives outside the model and is explicitly selected and inserted into the next model call.

A useful system-level picture is:

User message

-> Application state layer

- conversation log

- user profile

- retrieved documents

- task state

- permissions

-> Prompt/context compiler

-> Stateless LLM inference worker

-> Response

-> Memory/state update policy

The LLM worker can be stateless even when the overall assistant is stateful.

This separation is the key design tradeoff.

3. Why statelessness is a feature, not a bug

3.1 Stateless workers make horizontal scaling easier

If inference workers are stateless, a request can be routed to any available GPU replica. The system does not need the same user to always land on the same machine. This helps with:

• load balancing,
• autoscaling,
• retrying failed requests,
• routing across regions,
• batching unrelated requests,
• rolling deployments,
• failure recovery.

This is especially important because LLM inference is expensive and hardware-constrained.

Systems such as Orca and vLLM focus heavily on batching, scheduling, and KV-cache management because throughput depends on keeping accelerators busy. Orca introduced iteration-level scheduling and selective batching for Transformer generation, reporting large throughput improvements over then-existing serving approaches.³ vLLM's PagedAttention work states that high-throughput LLM serving requires batching many requests, while the KV-cache memory for each request is large and dynamically changes during generation.⁴

If every user session carried mutable hidden state inside the model server, the serving architecture would become closer to stateful session management. That creates complications such as sticky routing, state migration, consistency management, failure recovery, and isolation boundaries.

The cost of statefulness is visible in general inference serving too. NVIDIA Triton's documentation notes that stateful models require sequence batching so that all requests in a sequence are routed to the same model instance, allowing the model to maintain state correctly.⁵

Documented part: LLM serving systems optimize batching, scheduling, and KV-cache memory. Stateful sequence models require special routing/sequence handling in serving systems.

Architectural interpretation: Stateless LLM workers are easier to scale and operate because they avoid sticky session state and hidden mutable per-user server state.

3.2 Statelessness improves isolation and control

In a multi-tenant AI system, hidden persistent model state would create risks:

• one user's facts influencing another user's answer,
• private information being retained in opaque state,
• unpredictable behavior drift,
• hard-to-audit memory writes,
• difficult deletion and retention compliance.

Keeping memory outside the model allows the system to apply normal software controls:

• access permissions,
• encryption,
• retention windows,
• deletion workflows,
• audit logs,
• tenant isolation,
• versioned memory schemas.

Documented part: Standard Transformer inference operates over supplied sequences, and many memory-augmented systems add memory through external stores, memory streams, or retrieval mechanisms rather than arbitrary online weight updates.

Architectural interpretation: Explicit external state is easier to govern than implicit mutable neural state.

3.3 Statelessness keeps model behavior stable

A stateless model does not automatically "learn" from arbitrary user interactions during ordinary inference. That is often desirable. If every conversation updated the model directly, then malicious, mistaken, private, or low-quality inputs could alter future behavior.

Production systems usually prefer learning and personalization to pass through controlled mechanisms:

• logs,
• labeling,
• evaluation,
• fine-tuning,
• preference optimization,
• retrieval indexes,
• user memory stores,
• explicit memory write policies.

Several memory-augmented systems keep the base model frozen or rely on external memory. RAG retrieves from an external dense index.¹ Reflexion stores verbal reflections in an episodic memory buffer instead of updating weights.⁶ LongMem proposes a decoupled design using a frozen backbone LLM and a separate memory retriever/reader.⁷

Documented part: These systems use retrieval, memory buffers, or external memory modules.

Architectural interpretation: Controlled external memory is often preferred over uncontrolled online model updates for stability and governance.

4. What you lose because of statelessness

Statelessness buys scalability and control, but it removes several things people intuitively expect from an assistant.

4.1 No automatic long-term personalization

If a user says:

I prefer Python examples. Avoid JavaScript unless I ask for it.

that preference is not automatically present tomorrow unless the application stores it and re-injects it.

The base model's weights are unchanged. The next inference call has access only to the newly assembled prompt/context and any retrieved external memory.

4.2 No durable correction by default

Suppose an assistant repeatedly calls a project "Phoenix," and the user corrects it:

The project was renamed Mesa.

If that correction is not persisted somewhere outside the model, future calls may revert to "Phoenix," especially if old documents in the retrieval corpus still contain the old name.

This is why RAG systems need freshness, source ranking, conflict resolution, and authority handling. Retrieval alone does not guarantee that the newest or most authoritative fact wins.

4.3 Context windows are finite

Even if the system stuffs the entire conversation into the prompt, there is a context limit. When the conversation exceeds that limit, the system must drop, summarize, retrieve, or compress old information.

This is where many "forgetting" failures appear.

Long-context models reduce the problem but do not eliminate it. The "Lost in the Middle" study found that models can perform worse when relevant information appears in the middle of long contexts, even when the model technically supports long inputs.⁸

4.4 Longer context increases cost and latency

Repeatedly sending long histories, documents, tool traces, and summaries consumes tokens and compute. In standard dense Transformer attention, self-attention cost grows with sequence length. The original Transformer paper gives per-layer self-attention complexity as a function of sequence length and representation dimension.²

During serving, the KV cache also grows with the number of tokens being processed, and vLLM's PagedAttention paper identifies dynamic, large KV-cache memory as a bottleneck for high-throughput serving.⁴

4.5 No true lifelong learning by default

An assistant can appear to learn if the system saves memories, summaries, embeddings, preferences, or task state. But that is not the same as the model updating its internal parameters after each experience.

Agent papers such as Reflexion explicitly frame improvement as storing verbal feedback in memory buffers rather than changing weights.⁶

5. Concrete failure mode: the chatbot that forgets mid-session

Imagine a support chatbot for a travel insurance company. Early in the session, the user says:

My claim number is CLM-4178. I am traveling with my mother, who uses a wheelchair.

Please email me, not SMS.

Later, the conversation grows. The assistant retrieves policy documents, asks clarifying questions, invokes tools, and accumulates many turns. Eventually the prompt budget is tight. The system keeps only:

Summary: User is filing a travel insurance claim and asked about accessibility.

Recent turns: ...

Retrieved documents: policy exclusions, claim filing instructions, reimbursement limits.

Now the user asks:

Can you send me the status update the way I asked?

The chatbot replies:

Sure, I can text you the update. What is your claim number?

This feels like the model "forgot."

Architecturally, the failure could have happened at several places:

Original fact existed in conversation

-> Context grew too large

-> Compaction summary dropped "email, not SMS" and "CLM-4178"

-> Retriever did not retrieve the old turn

-> Prompt compiler did not include relevant state

-> Stateless LLM answered from incomplete context

The model did not forget in the human sense. The system failed to preserve or re-surface the relevant state.

This is the core lesson:

In stateless LLM systems, memory is not something the model simply has. Memory is an architectural pipeline.

6. Workaround 1: context stuffing

Context stuffing means putting the conversation history, retrieved documents, tool outputs, user profile, and instructions directly into the prompt.

System instructions

Developer instructions

User profile

Conversation history

Relevant documents

Tool outputs

Current user message

What is documented

Transformers attend over supplied tokens rather than maintaining recurrence across hidden sessions in the original architecture.²

Longer contexts can help when the relevant information is included, but models do not always use long context robustly. "Lost in the Middle" found that relevant information placed in the middle of long inputs can be underused.⁸

Why teams use it

Context stuffing is simple. It requires no retrieval pipeline, no memory schema, no embedding index, and no separate state manager. It is often the first architecture teams build.

What it loses

Context stuffing is:

• expensive,
• finite,
• noisy,
• sensitive to prompt ordering,
• sensitive to truncation,
• vulnerable to irrelevant context dilution.

The model may have the relevant fact somewhere in the prompt and still fail to use it.

Where it fits

Context stuffing is reasonable for:

• short sessions,
• one-off document question answering,
• small internal tools,
• prototypes,
• workflows where all relevant context is compact.

It becomes brittle for long-lived assistants, agent workflows, and systems requiring durable user state.

7. Workaround 2: RAG plus vector stores

Retrieval-Augmented Generation, or RAG, separates memory into a retrieval system and a generation system.

Knowledge corpus / past interactions

-> Chunking

-> Embedding model

-> Vector index

-> Query-time retrieval

-> Top-k chunks inserted into LLM context

-> LLM answer

RAG's core idea is documented in Lewis et al.: combine a pretrained generator's parametric memory with non-parametric memory, such as a dense vector index accessed by a retriever.¹

Dense Passage Retrieval showed that learned dense representations can retrieve passages for open-domain question answering and outperform a strong BM25 baseline on top-20 retrieval accuracy across evaluated datasets.⁹

FAISS and related approximate nearest-neighbor systems provide scalable vector similarity search. Johnson et al. describe GPU-based billion-scale similarity search.¹⁰ HNSW is a graph-based approximate nearest-neighbor method widely used for high-dimensional vector search.¹¹

Why RAG is attractive

RAG gives systems updateable memory without retraining the LLM. A system can:

• add a document,
• delete a document,
• update a policy,
• apply permissions,
• retrieve relevant snippets,
• attach citations or provenance.

RAG also avoids stuffing everything into the prompt. Instead, the system retrieves a smaller subset of potentially relevant information.

What RAG does not solve

RAG is not magic memory. It can fail if:

• the relevant fact was never stored,
• the text was chunked badly,
• the embedding model does not represent the query well,
• the relevant memory is filtered out by permissions,
• stale documents outrank fresh ones,
• semantically similar but wrong chunks are retrieved,
• the model ignores the retrieved text,
• retrieved sources conflict and the system lacks authority rules.

Self-RAG argues that indiscriminately retrieving a fixed number of passages can hurt usefulness and proposes adaptive retrieval plus reflection tokens to decide when retrieval is needed and whether retrieved passages support the answer.¹²

How RAG causes apparent forgetting

A chatbot may have the relevant memory stored somewhere, but the retriever does not fetch it. To the user, that looks identical to forgetting.

Example:

Memory store contains:

- User prefers email, not SMS.

  

User asks:

- Can you send the status update the way I asked?

  

Retriever focuses on:

- status update

- claim processing

- travel insurance policy

  

Retriever misses:

- contact method preference

The fact exists. The model never sees it.

8. Workaround 3: summarization compaction

Summarization compaction means compressing old context into shorter summaries. Common patterns include:

Rolling summary:

After every N turns, update a conversation summary.

  

Hierarchical summary:

Summarize chunks, then summarize summaries.

  

Memory extraction:

Convert turns into atomic facts.

  

Reflection:

Synthesize higher-level lessons from events.

Example memory extraction:

Raw conversation:

"My claim number is CLM-4178. Please email me, not SMS. My mother uses a wheelchair."

  

Extracted state:

claim_number = "CLM-4178"

contact_method = "email"

sms_allowed = false

accessibility_need = "wheelchair assistance for mother"

Documented related work

Generative Agents store a record of agent experiences, synthesize higher-level reflections, and retrieve memories dynamically for planning behavior.¹³

MemGPT proposes "virtual context management," inspired by hierarchical memory systems, to move information between memory tiers so LLMs can operate beyond a limited context window.¹⁴

MemoryBank proposes a long-term memory mechanism that updates and retrieves memories across interactions, including selective forgetting and reinforcement inspired by the Ebbinghaus forgetting curve.¹⁵

Why summarization is useful

Summarization can:

• reduce token cost,
• keep old context available in compact form,
• turn messy dialogue into cleaner state,
• preserve high-level intent,
• make long sessions manageable.

What summarization loses

Summaries are lossy. They can omit details that later become critical. They can also introduce unsupported interpretations.

For example:

Original:

Email me, not SMS. My phone is shared with my employer.

  

Bad summary:

User prefers digital communication.

The summary loses:

• the exact channel,
• the explicit rejection of SMS,
• the privacy reason.

Another example:

Original:

I used to live in Berlin, but now I live in Lisbon.

  

Bad memory write:

user_location = "Berlin"

Architectural lesson

Summarization should not be treated as perfect compression. It is an opinionated lossy transform.

For high-stakes or exact details, systems should store structured facts or source-linked memories, not rely only on prose summaries.

9. Workaround 4: external key-value state

Here, key-value state means application-level state, not the Transformer's internal key/value attention cache.

An external state store might hold:

user:123:preferred_language = "English"

user:123:preferred_code_language = "Python"

user:123:contact_method = "email"

claim:CLM-4178:status = "awaiting documents"

session:abc:current_task = "file travel insurance claim"

agent:run42:plan = ["verify identity", "fetch claim", "draft email"]

This is often the most reliable way to manage durable facts.

Why external key-value state differs from RAG

RAG is usually fuzzy retrieval over text chunks. Key-value state is explicit, structured, and addressable.

RAG-style lookup:

Find chunks semantically related to "how should I contact the user?"

Structured state lookup:

SELECT contact_method

FROM user_preferences

WHERE user_id = 123;

For exact state, structured lookup is usually safer.

What external state buys you

External state can provide:

• durability,
• auditability,
• deletion,
• permissions,
• conflict resolution,
• deterministic lookup,
• schema validation,
• versioning,
• separation between canonical business state and generated text.

For example, a medical, financial, legal, or insurance assistant should not rely on the LLM's prose memory for canonical state. The canonical state should live in a controlled system of record, with the LLM acting as an interface and reasoning layer.

What external state costs

External memory requires memory-writing policies. The system must decide:

• what to save,
• when to update it,
• when to forget it,
• what is private,
• what is inferred versus explicitly stated,
• what requires confirmation,
• how to handle contradictions,
• which source is authoritative.

Bad memory extraction can be dangerous.

Example:

User says:

I hate when apps assume I want dark mode.

  

Bad memory write:

prefers_dark_mode = true

External memory is powerful, but memory correctness becomes an application responsibility.

10. Internal KV cache is not long-term memory

The term KV cache often causes confusion.

In Transformer inference, the KV cache stores attention key/value tensors for previous tokens so the model does not recompute them repeatedly during autoregressive generation. vLLM's PagedAttention paper is about managing this KV-cache memory efficiently because it is large, dynamic, and affects batching throughput.⁴

vLLM also documents prefix caching, where cached KV blocks can be reused when requests share the same token prefix.¹⁶

That is compute reuse, not semantic memory.

A KV cache can help the server avoid recomputing a shared prefix such as:

System: You are a helpful assistant...

But it does not mean the model has learned:

User likes concise answers.

unless that fact is still in the prompt prefix or retrieved from external state and reinserted.

11. More advanced memory architectures

There is documented academic work on models and systems that go beyond plain stateless context windows.

Transformer-XL

Transformer-XL introduced segment-level recurrence to model dependencies beyond a fixed-length context and address context fragmentation.¹⁷

Compressive Transformer

Compressive Transformer added compressed memories for long-range sequence modeling.¹⁸

RETRO

RETRO conditions an autoregressive model on chunks retrieved from a very large corpus, using retrieval as explicit memory at scale.¹⁹

Memorizing Transformers

Memorizing Transformers add approximate k-nearest-neighbor lookup over non-differentiable memory of recent key-value pairs and report improvements as memory size increases.²⁰

LongMem

LongMem proposes a decoupled long-term memory architecture with a frozen LLM backbone and a retriever/reader side network.⁷

Agent memory systems

Generative Agents, Reflexion, MemoryBank, and MemGPT demonstrate different forms of memory streams, episodic buffers, reflections, virtual context management, and long-term user memory.^{13 6 15 14}

These works show that persistent or extended memory is technically possible. But they also show why it is not the default simple serving model: memory introduces mechanisms, retrieval policies, storage costs, update semantics, evaluation challenges, and serving complexity.

12. The architectural tradeoff in one diagram

Stateless LLM serving

Request A --\

Request B ----> Load balancer -> Any GPU worker -> Response

Request C --/

  

State needed by model = included in prompt/context

This is easier to scale and operate.

Stateful memory-centric assistant

User

-> Identity / permissions

-> Conversation log --\

-> User profile ----> Memory selection / retrieval / summarization

-> Task database --/

-> Vector store

-> Prompt compiler

-> LLM inference

-> Response

-> Memory write policy

-> External state update

This is more useful for long-lived assistants, but it has more moving parts.

The mature architecture is usually a hybrid:

Stateless model workers plus explicitly stateful application layers.

13. Documented work versus architectural assumptions

Documented academic and industry work

The following are documented in academic papers or industry documentation:

1. Transformers use attention over supplied sequences rather than recurrence in the original architecture.²
2. RAG combines model weights as parametric memory with external non-parametric memory such as a dense vector index.¹
3. Dense retrieval and vector similarity search are established mechanisms for retrieval-augmented question answering and large-scale similarity search.^{9 10 11}
4. Long-context models can still fail to use information robustly depending on where it appears in the context.⁸
5. LLM serving systems are strongly shaped by batching, autoregressive generation, and KV-cache memory management.^{3 4}
6. Stateful inference serving requires special handling such as sequence batching in systems like NVIDIA Triton.⁵
7. Memory-augmented agents and long-memory systems have been proposed using memory streams, episodic buffers, virtual context management, retrieval, and decoupled memory modules.^{13 6 14 15 7}

Architectural assumptions or interpretations

The following are reasoned architectural conclusions, not universal facts every vendor documents:

1. Stateless inference is preferred in many production settings because it simplifies horizontal scaling, batching, failover, and tenant isolation.
2. Externalized memory is safer and more governable than hidden mutable model state because it can be audited, permissioned, deleted, and tested.
3. For exact user or business state, structured key-value, SQL, or event-log memory is usually more reliable than pure vector retrieval.
4. Summarization compaction should be treated as lossy unless the system explicitly preserves source links and critical fields.
5. A chatbot "forgetting" mid-session is often not a model failure alone; it is usually a failure of context assembly, retrieval, summarization, state management, or prompt budgeting.

14. Practical design guidance

For a real AI system, do not ask only:

Does the LLM have memory?

Ask more precise questions.

What state must be exact?

Put it in a structured database, key-value store, event log, or system of record.

Examples:

claim_number

billing_status

contact_method

account_permissions

medical allergy record

legal case identifier

What state is fuzzy or semantic?

Use vector retrieval, hybrid search, or memory streams.

Examples:

past project discussions

user's writing style

related support tickets

similar design documents

previous brainstorming ideas

What state is short-lived?

Keep it in the session prompt, recent conversation window, or temporary task state.

Examples:

current question

current tool result

recent clarification

active coding task

What state is expensive but useful?

Summarize it, but preserve source links.

Examples:

long meeting transcripts

multi-hour agent traces

customer support conversations

research sessions

What state is sensitive?

Apply explicit consent, retention limits, deletion paths, permissions, and audit trails.

Examples:

health information

financial information

children's information

precise location

identity documents

private workplace data

What state should never be inferred silently?

Require user confirmation before writing it to memory.

Examples:

political views

religion

health status

sexual orientation

financial hardship

legal status

workplace conflicts

A robust architecture separates memory by reliability requirements:

Canonical facts:

SQL / key-value store / system of record

  

Semantic recall:

Vector store / hybrid search

  

Recent interaction:

Conversation tail

  

Compressed history:

Summaries with source pointers

  

Reasoning workspace:

Temporary scratchpad / tool state

  

Model:

Stateless inference over assembled context

15. Design checklist for an LLM memory system

A serious memory system should answer these questions explicitly.

Memory capture

• What events are eligible to become memory?
• Does the user need to opt in?
• Which facts require explicit confirmation?
• Are inferred facts allowed?
• Are sensitive facts excluded by default?

Memory representation

• Is memory stored as raw transcript, summary, extracted structured fact, embedding, event, or all of these?
• Does each memory have a timestamp?
• Does each memory have a source pointer?
• Does each memory record confidence?
• Does each memory distinguish user-stated facts from system inferences?

Memory retrieval

• Is retrieval semantic, lexical, structured, recency-based, permission-based, or hybrid?
• How are conflicts resolved?
• How is source authority ranked?
• How does the system avoid stale memories?
• How does the system detect that retrieval failed?

Memory injection

• What gets inserted into the prompt?
• In what order?
• With what priority?
• How is the token budget allocated?
• Are critical facts protected from summarization loss?

Memory update

• When is a memory updated versus appended?
• Can the user inspect memory?
• Can the user delete memory?
• Are deleted memories removed from embeddings, summaries, and backups?
• Are memory writes logged?

Evaluation

• Does the assistant remember exact facts across sessions?
• Does it avoid using memories from the wrong user?
• Does it update stale facts correctly?
• Does it avoid over-personalization?
• Does it refuse to store sensitive inferred facts?
• Does it behave correctly when memories conflict?

16. Bottom line

Statelessness is not an accident. It is a scalability and control choice.

A stateless LLM worker is easy to replicate, batch, schedule, cache, fail over, and isolate. That matters enormously at production scale.

The cost is that the model does not automatically accumulate durable personal, conversational, or task memory. Everything it should remember must be supplied through context, retrieved from external stores, summarized into compact form, or represented in structured state.

The right framing is:

LLMs are usually stateless at the inference layer, while useful AI assistants are stateful at the system layer.

When a chatbot "forgets," the root cause is rarely that the neural network suddenly lost a memory. More often, the architecture failed to carry the right information across the boundary between external state and the stateless model call.

References