Engineers evaluating token-reduction tools often face a choice between reshaping what an agent writes and reshaping what crosses the network. Caveman operates inside the prompt boundary, enforcing a terse writing style and routing decisions through a proxy layer that intercepts model calls. Headroom instead sits between the agent runtime and the provider as a transparent local layer, compressing context while preserving KV-cache alignment and reversible semantic structure. This article isolates the operational difference between an in-band discipline and an out-of-band compression proxy, drawing on the comparison passage that frames the two approaches. The intent is to give backend and AI-platform engineers a clear decision lens, not to evaluate every token-reduction tool on the market.
The clearest way to compare these two systems is to map where each one enters the request flow between an agent and a language model provider. The insertion point determines what kind of change is possible: behavior at the prompt construction stage versus behavior at the transport stage.
Caveman operates in-band. Its first intervention is stylistic: it imposes a terse writing convention on the agent itself, favoring short imperative sentences, dropped articles, and other token-compacting conventions. Because the discipline is enforced on what the agent writes, the reduction is a property of the authored prompt rather than the transmitted payload.
In addition to the style layer, Caveman routes model calls through a proxy that sits in front of the provider. This proxy can rewrite outgoing requests, enforce policy, or abort calls before they reach the model API. The pipeline therefore looks like:
agent → prompt builder → (Caveman rewrite / proxy) → provider
Anything Caveman saves is saved at authoring time, and anything it gates is gated before the network boundary.
Headroom operates out-of-band. It does not influence how the agent composes its prompt or which tool calls it selects. Instead, it is positioned as a transparent local middleware layer between the agent runtime and the provider, intercepting the serialized prompt and tool trace after the agent has already committed to sending it.
At that point, Headroom compresses the payload in a way that preserves KV-cache alignment and keeps the structure reversibly decodable on the provider side, so the model still receives a coherent input. The pipeline therefore looks like:
agent → provider-facing transport → (Headroom compress / decompress) → provider
Because Headroom sits at the transport layer, it requires no changes to the agent's code or authoring style.
The two pipelines represent fundamentally different axes of choice:
For engineers, this axis usually resolves a few practical questions up front: whether the team can tolerate agent-side style enforcement, whether a proxy in front of the provider is acceptable, and whether the savings must come from what is written or from what is sent. The remainder of the comparison follows from this single decision.
Caveman reduces token usage inside the prompt boundary by reshaping what the model is asked to produce and, optionally, by routing requests through a local proxy that can intercept calls before they hit the provider. The reduction is achieved by changing the instructions the agent reads and follows, not by transforming the wire format after the fact.
At its core, Caveman ships a compact instruction set — typically delivered as a skill, rules file, or custom output style — that the agent reads and applies to every response. The rules constrain vocabulary and sentence shape rather than the model's reasoning:
Intensity levels (lite, full, ultra, and the experimental wenyan) control how aggressively the rules apply, ranging from "professional but tight" to telegraphic prose with abbreviations like db, auth, req, and arrow causality (X → Y) (Caveman plugin guide). The system-prompt overhead is reported at roughly 120–195 tokens across the lite-to-ultra range (Caveman Code).
When invoked as caveman wrap <agent>, Caveman places a local proxy between the agent runtime and the provider. The proxy does not rewrite model output; it forwards requests as-is to the chosen provider, preserving existing credentials (including Claude Pro/Max OAuth tokens) (Caveman toolkit). Two forms of savings sit on top of the style guide:
These behaviors are configurable per agent and can be toggled in ~/.caveman-cloud/config.json (Caveman configuration).
Because the discipline lives inside the prompt, three operational consequences follow.
Combined, these properties define Caveman as an in-band discipline: the model writes less because it is told to, while optional proxy routing deduplicates and budgets the tool traffic that surrounds those replies.
Headroom operates as an out-of-band compression layer positioned between the agent runtime and the model provider. Unlike in-band approaches that reshape the agent's writing style, Headroom leaves the agent's code and prompt logic untouched. The compression and decompression happen entirely in a sidecar process or an in-process library that intercepts outbound prompts, tool traces, and inbound responses before they cross the network boundary.
The core method is Context-Compress-Retrieve (CCR), a reversible semantic compression technique. When an agent prepares a prompt, Headroom intercepts it, identifies compressible regions such as repeated tool outputs, long document excerpts, or verbose system instructions, and produces a compact representation. On the response side, any references back to those compressed regions are decompressed before the agent sees them. Because the transformation is logically reversible, the agent receives a prompt that is semantically equivalent to what it originally sent, without needing to learn new syntax, special tokens, or compression-aware prompting patterns.
This reversibility is the key contrast with lossy summarization. Summarization discards detail and rephrases content, which can subtly shift semantic intent and produce inconsistent results across runs. CCR preserves the original meaning while still reducing the token count that crosses the wire.
Headroom's most consequential design decision is preserving KV-cache alignment. Most providers implement prefix caching: if the beginning of a prompt matches a previously cached prefix, the provider reuses the stored key-value state and skips recomputation, which can substantially reduce latency and cost. The cached prefix is identified by a cache key derived from the exact token sequence at a version-specific boundary.
For out-of-band compression to remain compatible with this mechanism, the compressed token stream must:
Headroom achieves this by ensuring that the compressed representation and the original representation produce equivalent cached state at the provider's checkpoint positions. The agent sees the full semantic content; the provider sees a token sequence whose cached prefix matches what would have been cached without compression.
KV-cache behavior is not standardized across providers, and cache-key formats, prefix-match boundaries, and caching eligibility rules change between provider versions and model releases. A compression layer that aligns correctly with one provider's caching contract may silently invalidate caching on another, or may stop aligning after a provider-side update. Engineers integrating Headroom should verify alignment against the target provider's current caching documentation and test that prefix-cache hits are preserved end-to-end after compression.
KV-cache preservation is not an optimization knob for out-of-band compression — it is the technical pivot that determines whether compression pays off at all. Providers such as OpenAI and Anthropic, and self-hosted runtimes like vLLM, key their attention cache to the exact token sequence of the prompt prefix (RunPod on vLLM prefix caching). If a compressor changes a single token ID in the shared system prompt or tool definitions, the next request misses the cache, the provider must re-prefill the entire prefix, and the latency and cost savings of compression are erased (sureprompts.com on compression vs caching).
Headroom treats this as a first-class design constraint rather than a side effect. The first defense is CacheAligner, a prefix-stabilization transform that runs before any compression. It detects volatile content — timestamps, session tokens, UUIDs — and relocates it to the tail of the prompt so the static prefix remains byte-identical across requests (Headroom on GitHub). The second defense is live-zone compression, which restricts rewriting to fresh bytes (new tool output, the latest turn) while leaving the frozen prefix untouched. The verbosity-steering instruction that conditions the model to be terse is also appended to the end of the system prompt specifically so the cached prefix is not disturbed.
Cache alignment imposes a ceiling on how aggressively an out-of-band proxy can rewrite. The tighter the compression, the more likely that token IDs drift and the cache busts. Headroom's answer is CCR (Compress-Cache-Retrieve): instead of trying to embed every detail into the prompt, it stores the original locally (SQLite-backed) and injects a retrieval marker plus a headroom_retrieve tool into the agent's available tools (alphamatch.ai on Headroom CCR). The model sees compressed context, but can pull the uncompressed original in roughly 1ms when it needs the dropped detail — sometimes with an optional BM25 query to return only the relevant subset. Reversibility buys back compression ratio without paying in cache invalidation.
Caveman's in-band approach is structurally insulated from this concern. Because the proxy sits inside the request flow and is free to inspect the prompt it is about to forward, it can forward identical token sequences for any prefix that the provider would otherwise cache, and rewrite only the variable segments (new tool output, the latest turn). The stable system prompt and tool definitions cross the wire byte-identical turn after turn, so the provider's prefix cache keeps hitting. Compression work concentrates on the segments that are already going to bust the cache, not on the ones that are paying for themselves in cached prefill savings.
In practice, this is why KV-cache alignment is the cleanest lens for separating the two architectures: an out-of-band compressor must engineer around the cache; an in-band proxy can simply leave the cacheable surface alone.
The integration burden differs sharply between the two architectures, and that difference drives most of the operational trade-offs engineers encounter in practice.
Caveman's in-band discipline requires the agent, its system prompts, and any prompt templates to be rewritten in a terse, token-frugal style. This is fundamentally a code and content change: every prompt path that originates from the agent must be audited, refactored, and maintained against a style guide. The proxy that mediates model calls can itself be transparent to the network layer, but the style enforcement is not transparent to whoever authors or maintains the agent. Savings are predictable because they are a direct function of how aggressively the prompt was rewritten, but the cost is paid across the entire prompt-emitting surface area.
Headroom takes the opposite position. The agent continues to emit rich, natural prompts exactly as before. A middleware layer sits between the agent runtime and the provider, compressing context on the wire while attempting to preserve KV-cache alignment and reversible semantic structure. From the agent's perspective, nothing changed; from the provider's perspective, what arrives is a shorter, cache-aligned sequence. The deployment model is closer to drop-in: install the middleware, point the runtime at it, and the existing prompt paths remain untouched.
Engineers choosing between the two should weigh two competing profiles:
A practical decision cue:
In short, the choice reduces to whether the engineering organization prefers to invest in prompt authoring discipline or in middleware that compresses transparently downstream. Both are legitimate strategies; they impose different costs on different parts of the stack.
Both approaches introduce non-obvious failure surfaces that engineers should plan for during evaluation. The risks are largely asymmetric: Caveman concentrates failure inside the prompt boundary and the proxy layer that rewrites calls, while Headroom concentrates failure at the compression boundary that sits between the agent runtime and the provider. A small set of risks are shared by both designs and warrant common mitigations.
The comparison between Caveman and Headroom does not reduce to a token-savings leaderboard. The decisive variable is where in the pipeline the team is permitted to intervene. Once that constraint is fixed, the choice follows.
If engineers can refactor how prompts are constructed, Caveman's in-band discipline is generally the lower-friction option. The savings are deterministic — every prompt segment that crosses the proxy has been shaped by the terse-style rules — and the transformation is visible in the prompt logs, which simplifies auditing and A/B testing. The trade-off is sustained authoring cost: every new prompt template, every new tool description, and every new retry path must be written in (or migrated to) the enforced style. Teams that already maintain a strong prompt-review process absorb this overhead easily; teams that ship many small prompt changes through product managers or domain experts tend to see style drift and need a continuous linting step at the proxy.
If the agent runtime is third-party, vendored, or otherwise read-only, in-band discipline is not available. In that setting, Headroom's transparent out-of-band layer is the only viable option, subject to two preconditions:
The two approaches are not mutually exclusive. Caveman's proxy can be layered in front of Headroom's middleware: the in-band style trims the variable segments that change every turn (chain-of-thought narration, ad-hoc scratchpads), while the out-of-band layer compresses the long static context (retrieved documents, tool definitions) that would be too costly to rewrite by hand. In a stacked deployment, Caveman reduces what the agent decides to say, and Headroom reduces what has to be carried across the wire.
The mechanics, failure modes, and measured trade-offs of running the two together are covered in a sibling article on stacking in-band and out-of-band compression.