Prompt Caching

The daemon uses provider-side prompt caching to reduce costs and latency on multi-turn conversations. The system prompt (agent instructions, tool definitions, project context) is cached so that subsequent turns in the same session skip re-processing it.

How It Works

Provider Behavior

Anthropic Cache Flow

On the first turn of a session, the daemon sends the system prompt as a system message with providerOptions.anthropic.cacheControl:

Turn 1 (cache write):
  System prompt [7,464 tokens, cache_control: ephemeral] → CACHE MISS → written to cache
  User message → processed normally
  Cost: system tokens charged at 1.25x input price (cache write premium)

Turn 2+ (cache read):
  System prompt [7,464 tokens, cache_control: ephemeral] → CACHE HIT → read from cache
  Previous messages + new message → processed normally
  Cost: system tokens charged at 0.10x input price (90% discount)

The cache lives on Anthropic's servers for ~5 minutes. Each new request within that window resets the TTL.

Cost Impact

Using Bytespace pricing for Claude Opus 4.5 ($5/MTok input):

Example: 8-Turn Session with 7,464-Token System Prompt

Savings increase with conversation length. A 20-turn conversation saves ~90% on system prompt costs.

Implementation

Daemon (packages/daemon/src/agent/loop.ts)

The agent loop detects the model provider and conditionally wraps the system prompt:

const isAnthropic = model.provider?.startsWith('anthropic');

// For Anthropic: pass as system message with cache control
const systemMessages = isAnthropic && systemPrompt
  ? [{ role: 'system', content: systemPrompt,
       providerOptions: { anthropic: { cacheControl: { type: 'ephemeral' } } }
     }]
  : [];

streamText({
  model,
  system: isAnthropic ? undefined : systemPrompt,  // plain string for other providers
  messages: [...systemMessages, ...conversationMessages],
});

Proxy (apps/bytespace/src/app/api/proxy/llm/)

The proxy extracts cache metrics from provider responses without any special configuration:

• Anthropic SSE streams: SSEUsageExtractor reads cache_creation_input_tokens and cache_read_input_tokens from the message_delta event
• Non-streaming: Parsed from the JSON response body (usage.cache_creation_input_tokens)

Cache stats are surfaced back to the daemon via:

• Streaming: SSE comment appended after stream ends: :cache-stats {"cacheCreationTokens":N,"cacheReadTokens":N}
• Non-streaming: Response headers X-Cache-Creation-Tokens and X-Cache-Read-Tokens

Usage Tracking

Cache tokens are logged per-request to ctx0_llm_usage with separate columns:

The /api/proxy/usage endpoint aggregates these per-model, and the UsagePanel (/usage command) displays them as "Cache W" and "Cache R" columns with a cache hit rate.

Observability

Proxy Logs

# Turn 1 - cache write
[llm-proxy] Stream complete for anthropic: model=claude-opus-4-5-20251101,
  904in/40out, cache=7464create/0read, keySource=user

# Turn 2 - cache hit
[llm-proxy] Stream complete for anthropic: model=claude-opus-4-5-20251101,
  383in/57out, cache=0create/7464read, keySource=user

Daemon Logs

[llm] Cache: read=7464, create=0

Database Query

SELECT model, input_tokens, cache_creation_tokens, cache_read_tokens, total_cost
FROM ctx0_llm_usage
ORDER BY created_at DESC
LIMIT 10;

Key Files