Fixed context windows impose a hard ceiling on LLM agents: once a conversation, task history, or document set exceeds the limit, the agent either errors out or silently loses information. Long-running agents that must operate for hours, across sessions, or over unbounded task horizons need an explicit mechanism for moving data between the model's window and persistent storage. MemGPT, introduced by Packer et al. in 2023, frames this as a virtual memory problem and borrows the memory hierarchy of operating systems, treating the context window as RAM and out-of-context storage as disk. The system partitions memory into main context, recall storage, and archival storage, and equips the model itself with tool calls that page information between tiers under memory pressure. This article walks through how each tier is structured, how function-call-based paging works, how the event-driven write-back cycle reacts to memory pressure, and what the original benchmarks demonstrated. It is aimed at engineers designing agents that must exceed context limits and stay coherent over long horizons.
Operating systems give each process the illusion of a huge, contiguous address space by paging data between physical RAM and disk storage. The process sees a single logical memory, while the OS swaps pages in and out under memory pressure.
MemGPT adopts the same pattern for large‑language‑model (LLM) agents. The model’s context window—the fixed‑size prompt that the LLM can attend to at inference time—is treated as physical RAM. Data that does not fit in the window is stored in an external context (a vector database and a document archive) that acts like a disk. The combination of the in‑window “main context” and the out‑of‑window “external context” is presented to the agent as virtual context—the effective working memory of the agent, regardless of the underlying token limit. The mapping is explicit in the MemGPT documentation:
| Operating‑system concept | MemGPT analogue |
|---|---|
| Main memory / physical RAM | Main context (system instructions, working scratchpad, FIFO message buffer) |
| Disk storage | External context (recall storage + archival storage) |
| Virtual memory | Virtual context (the logical view the agent reasons over) |
The LLM itself is a fixed‑context processor: only tokens that are physically present in the prompt can influence the next token prediction. Anything kept in external storage is invisible to the model until a function‑call‑driven paging operation explicitly retrieves it and inserts it into the main context. This design lets the agent decide, at runtime, which facts, documents, or intermediate results deserve the limited RAM budget.
Consequently, treating the context window as a scarce capacity budget remains sound engineering practice, even as window sizes grow. The virtual‑memory abstraction lets agents scale their effective memory without relying on ever‑larger windows, preserving predictable cost, latency, and reasoning fidelity.
Sources: MemGPT architecture paper describes the three‑tier hierarchy and the OS‑inspired analogy【0†url】. The mapping table is reproduced in the MemGPT blog post on virtual context management【2†url】.
The main context is the only data that the LLM can see directly during inference. MemGPT partitions this fixed‑size prompt into three competing regions, each consuming a share of the total token budget.
A static block that appears at the start of every request. It contains the base directives that define the agent’s control flow, the schemas of all memory‑related function calls, and the rules for paging data between tiers. Because it never changes, this block represents a fixed overhead that must be reserved on every call 【1†L1-L4】.
A writable scratchpad that the model can modify only through the defined memory‑tool function calls. In the original MemGPT implementation the block is split into two labeled sub‑sections—<persona> for the assistant’s identity and <human> for user‑specific facts 【1†L5-L9】. The Letta platform generalizes this idea: developers declare arbitrary memory blocks (e.g., user profile, task state, session variables) and the runtime provides corresponding tools to read, write, or delete the contents 【4†L1-L4】. The size of core memory is deliberately chosen to hold the facts that must remain visible at all times; when the window fills, the agent’s paging logic decides which block entries to evict to external storage.
A first‑in‑first‑out buffer that holds the latest conversation turns—including system messages, user inputs, assistant replies, and the inputs/outputs of function calls. The head of the queue always contains a recursive summary of any earlier messages that have been evicted, giving the model a compressed view of the full dialogue history 【1†L10-L13】. This design lets the agent retain a sense of long‑term continuity while staying within the token limit.
All three regions share the same token quota. Architects must decide how many tokens to allocate to the immutable system prompt, how large the core‑memory block should be, and how much room to give the FIFO queue. The allocation is a deliberate engineering decision: a larger system prompt reduces flexibility for later regions, while a smaller core memory may force more frequent paging, increasing latency.
The concrete block names (core_memory, persona, human) and the exact function‑call signatures differ between the original research code, the open‑source MemGPT repository, and the current Letta platform. Code written against one version may require adaptation before it works with another, so developers should consult the specific SDK documentation for their target environment.
Recall storage is the verbatim log of every event the LLM processes – user‑assistant turns, system messages, intermediate reasoning steps, tool calls, and their returned values. In the reference implementation the conversation_search tool scans this store with case‑insensitive string matching and, optionally, a timestamp‑based lookup exposed through Letta【2†content】. Because the full FIFO queue is retained, the model can retrieve any past token sequence even after it has been evicted from the active window.
Archival storage is a general‑purpose read‑write datastore for documents, extracted facts, or any arbitrary text that does not belong in the immediate message history. It is backed by vector embeddings and queried with archival_memory_search, which returns the most semantically similar chunks【1†content】. Insertion is performed through archival_memory_insert, allowing the agent to tag content with descriptive metadata for future relevance【1†content】.
topic:privacy policy, expires:2027‑01). The embedding index then makes the fact reachable via semantic search even many turns later.conversation_search for literal or temporal queries (e.g., “What did the user say at 14:32?”). Use archival_memory_search for similarity‑based lookups (e.g., “Find documents about GDPR compliance”). Attempting to retrieve raw messages with the embedding API, or vice‑versa, yields imprecise results because the underlying indexing assumptions differ.Both tiers reside in external databases that are inaccessible during inference; the model must issue the appropriate tool call to bring data into the main context. This explicit boundary preserves the OS‑inspired “RAM‑disk” separation while giving agents fine‑grained control over long‑term coherence.
MemGPT equips each agent with a minimal set of six functions that act as the paging interface between the three memory tiers — main context (RAM), recall storage (disk), and archival storage — and the user 【1†L1-L8】.
| Function | Purpose | Tier |
|---|---|---|
conversation_search | Case‑insensitive string search over the full conversation log stored in recall storage | Recall |
core_memory_append | Add new text to the end of the core‑memory block | Core |
core_memory_replace | Replace the entire core‑memory block (empty string deletes) | Core |
archival_memory_insert | Insert a new record into the long‑term archival vector store | Archival |
archival_memory_search | Semantic (embedding‑based) retrieval from archival store | Archival |
send_message | Emit a response to the human user | — |
Two design decisions are worth highlighting. First, no tool is provided for searching core memory because that tier is always present in the model’s context window; the agent can read it directly without an indirection 【1†L10-L12】. Second, every call includes a request_heartbeat boolean. When set to true the harness does not immediately return control to the user; instead it allows the model to chain further calls in the same turn, enabling multi‑step paging operations such as “search → insert → summarize → replace” without breaking the interaction flow 【1†L13-L15】.
The agent runtime implements a classic “thought‑action‑observation” (TAO) cycle 【5†L1-L6】. In MemGPT the loop works as follows:
request_heartbeat flag.If a tool fails (e.g., archival storage is full), the error is fed back to the model, which can adapt by selecting an alternative strategy such as compressing the payload or moving older items to a deeper archive. This tight feedback loop lets the agent self‑edit its memory without external orchestration.
Traditional retrieval‑augmented generation (RAG) places the search‑and‑fetch logic entirely in application code: the system decides when to retrieve documents, what index to query, and which excerpts to inject into the prompt 【3†L1-L5】. MemGPT inverts this relationship. The model itself decides what to persist, when to evict, and which tags to attach, because it holds the task‑specific context that external logic lacks. The toolset therefore becomes an internal paging subsystem rather than a mere data‑access SDK.
When adapting this pattern, expose only the smallest surface that still covers the four fundamental operations per tier:
conversation_search (recall) and archival_memory_search (archival).core_memory_append, core_memory_replace, and archival_memory_insert.send_message and the request_heartbeat flag for chaining.Keeping the API minimal reduces prompt token overhead and lowers the probability that the model emits a mis‑typed or out‑of‑scope function call, resulting in more stable long‑running agents.
When the token count in the main context (the RAM analogue) climbs toward a configurable limit, MemGPT raises an internal memory‑pressure alert. In many reference implementations the trigger is set to roughly 70 % of the context capacity, but this value is not a hard rule in the original paper; it is chosen by the harness according to model size, prompt structure, and workload characteristics【source: https://serokell.io/blog/design-patterns-for-long-term-memory-in-llm-powered-architectures】.
MemGPT’s architecture is event‑driven, not continuously polling. Activities occur in response to three kinds of signals:
These heartbeats give the agent a window to perform proactive maintenance, such as archiving a fact it knows will be required later in the session. By batching several operations into one heartbeat, the system reduces the number of round‑trips to external storage and keeps latency low.
| Step | Action | Outcome |
|---|---|---|
| 1 | Detect token usage ≥ threshold (e.g., 70 % of capacity) | Interrupt raised |
| 2 | Pause reasoning | Agent enters maintenance mode |
| 3 | Identify low‑importance items | Candidate set for eviction |
| 4 | Summarise with preservation cues | Compact representation |
| 5 | Write‑back to recall/archival storage | Data persisted, RAM freed |
| 6 | Update context window | Space reclaimed for new user input |
The cycle therefore ensures no information is lost—only moved to a slower, larger store—while keeping the active context focused on the most relevant, high‑priority content. This OS‑style paging model enables long‑running agents to stay coherent over hours or across sessions without exceeding the hard limits of LLM context windows.
The 2023 MemGPT paper evaluated two representative workloads: document‑analysis over corpora that far exceed a model’s context window, and multi‑session chat where the agent must retain knowledge across many successive interactions.
Deep Memory Retrieval (DMR). On a conversation‑history task that requires pulling facts scattered throughout a long dialogue, MemGPT paired with GPT‑4 attained 92.5 % accuracy, while a vanilla GPT‑4 baseline managed only 32.1 %—a near‑three‑fold gain [source]. This gap illustrates how paging relevant snippets into the active window preserves recall where plain truncation discards the needed context.
Document‑analysis scalability. When the number of retrieved documents grew, MemGPT’s accuracy remained essentially flat, whereas baseline systems degraded because the context limit forced truncation of the document list [source]. The hierarchical memory tier (RAM → recall → archival) lets the model selectively load only the most pertinent passages, keeping the reasoning chain intact.
Nested key‑value retrieval (multi‑hop). In a synthetic benchmark that requires chained look‑ups (value → next key → …), MemGPT was the only approach that completed tasks beyond two nesting levels with 100 % accuracy; all baselines fell to 0 % [source]. This result highlights the importance of an explicit write‑back/write‑through policy: the agent can keep intermediate results in its virtual “RAM” while fetching deeper entries from the “disk”.
All numbers stem from the 2023 research implementation and the GPT‑4 generation available at that time. They should not be taken as performance guarantees for newer model families or production deployments; later benchmarks (e.g., LoCoMo, LongMemEval) now test memory systems on larger token scales and harder reasoning patterns.
Takeaway: the decisive factor across these experiments is capacity‑driven paging, not the raw score itself. By treating the context window as a limited cache and off‑loading older or less‑relevant data, MemGPT maintains stable accuracy as workloads exceed the window. Conversely, approaches that rely on naïve truncation quickly lose correctness. Future work should therefore view the paging mechanism as the transferable insight, while evaluating it on updated, more demanding benchmarks.
Implementation facts
Design decisions
| Decision | Options | Practical guidance |
|---|---|---|
| Policy ownership | • Pure model autonomy – the LLM decides what to retain or evict.<br>• Deterministic thresholds – external code forces paging.<br>• Hybrid – code detects memory pressure, the model selects the content. | A hybrid approach is usually safest: deterministic code (e.g., a simple size‑check) triggers a paging request, then the model chooses which block(s) to write back, because the model sees the full task context that raw code cannot interpret. |
| Threshold placement | • Low threshold – frequent churn, wasted RAM.<br>• High threshold – risk of a single large tool response overflowing the window. | Empirically, keep the usable window at ≈75 % of the model’s maximum tokens. This leaves headroom for the next tool call and for system messages. Adjust per model; GPT‑4‑Turbo (8 k tokens) often works well with a 6 k‑token target. |
| Guardrails on self‑editing | • Unchecked writes – potential prompt‑injection via stored content.<br>• Validate‑then‑apply – schema checks, content filters.<br>• Human‑in‑the‑loop approval for critical blocks. | Implement a lightweight validator that runs after each write_memory call. If validation fails, inject the error message back into the LLM context so it can attempt a corrected write. For high‑risk blocks (e.g., persona or policy rules), route the write through a short manual review step. |
| Persistence strategy | • Separate checkpoints – only RAM persisted.<br>• Unified checkpoint – RAM + recall + archival saved atomically. | Store a combined snapshot (RAM blocks + metadata of recall/archival indices) in a transactional database. On restart, reload the snapshot before processing the next user turn to avoid window mismatches. |
Trade‑offs
Recommendation checklist
page_out tool; let the LLM decide the victim block(s).By aligning the production stack with these concrete decisions—while keeping the OS‑inspired memory metaphor intact—engineers can build long‑running agents that stay coherent, safe, and recoverable beyond the hard limits of a single LLM context window.