API Modernization

Pipeline audit reveals phase-1-era patterns held over from early 2026: Anthropic SDK pinned at 0.39.0 (current is 0.122.0), zero prompt caching on the ~11.6KB static extraction prefix (repaid on every call to date), regex-fence JSON parsing instead of structured outputs, analyze.ts still using a stale pre-pivot prompt (batch-themes.md says "API infrastructure company"), and a latent batch.ts truncation bug (max_tokens 16000 with no stop_reason check). Separately, Mark asked whether JSON-in-git suffices or whether a database is warranted — evaluation shows it does, with clear re-evaluation triggers if conditions change. This cycle modernizes the API surface for cost and quality, records the data-architecture verdict, and absorbs c8's deferred "D2 architecture thread" (items D2.1–D2.3: storage decision, structured outputs + retry, analyze reshape).

Bring the pipeline to best-of-tech Claude API patterns (SDK upgrade, prompt caching, structured outputs, adaptive-thinking-era models), finalize the data-architecture decision with explicit re-evaluation triggers, reshape cross-call analysis from whole-corpus prompt to a map-reduce pattern (per-use-case + per-company granular views, Opus-5 reduce step for overview), fix known bugs (batch truncation, max_tokens unification), and make the site's natural load-path improvements. Outcome: extraction cost and latency drop, analysis quality rises (use-case-centric signals), development velocity increases (deterministic schema reduces prompt/code friction).

Workstreams:

• W1 — SDK upgrade + shared client (~half day): @anthropic-ai/sdk 0.39 → 0.122, extend pipeline/model.ts with shared Anthropic client, document model overrides in .env.example.
• W2 — Bug fixes (rides with W1): batch.ts truncation guard (stop_reason check), max_tokens unification (batch 16000 → EXTRACT_MAX_TOKENS 32000), SDK retry tuning (rely on built-in maxRetries).
• W3 — Structured outputs (~1 day): new schemas/output-schema.ts derives JSON Schema programmatically from vocabularies.ts arrays (no zod), observation variants via anyOf, extract.ts and batch.ts switch to .output_config, eliminate regex fence-parsing.
• W4 — Prompt caching (~half day): refactor pipeline/prompt.ts to split static/dynamic parts (split on ## Transcript marker), extract.ts adds cache_control on system prompt (≈3.5–4.5K tokens, clears Sonnet-5 minimum), batch.ts opts out (unreliable hit rates).
• W5 — analyze.ts map-reduce reshape (~1–1.5 days): replace whole-corpus-in-one prompt with Stage 0 (group observations), Map A (per-use-case via Sonnet-5), Map B (per-company via Sonnet-5), Reduce (overview via Opus-5 with adaptive thinking), write to data/analysis/{use-cases,companies}/ then consolidate to data/analysis.json.
• W6 — Site load-path small cut (~half day): call detail page fetches /data/processed/{id}.json directly, precompute search blobs once at load (not per keystroke).

Out of scope:

• Database migration (decided against; see Data-Architecture Decision below).
• Moving aggregate views to index.json (rollups need full observation objects).
• useCallData type-duplication cleanup (cross-project work, deferred).

• W1 — SDK 0.39 → 0.122; pipeline/model.ts is now the config layer: MODEL (sonnet-5), ANALYZE_MODEL (opus-5, ANTHROPIC_ANALYZE_MODEL override), EXTRACT_MAX_TOKENS=32000, shared client (maxRetries: 4). Overrides documented in .env.example. Added @types/node — npx tsc --noEmit is now a clean gate (fixed two pre-existing type errors it surfaced).
• W2 — batch.ts truncation guard (stop_reason === 'max_tokens' → failed-raw dump), max_tokens unified via EXTRACT_MAX_TOKENS, parse failures dump raw output. Retries rely on SDK built-ins. (nanoid was NOT dead — used by upload-sagetap.ts; c8 note stale.)
• W3 (amended in-flight) — structured outputs cannot cover per-call extraction: probing found per-schema API limits (16 union params, 24 optional params, "grammar too large" beyond ~50 object properties; a 42-prop subset passed, 77+ failed even enum-free). Documented in schemas/output-schema.ts. Extraction keeps prompt-driven JSON + validateCallData enforcement; the three new analyze stages DO use output_config.format (small schemas). Prompt now instructs omit/"" over null; stripEmptyStrings() cleans output.
• W4 — buildExtractionPrompt returns {staticPart, dynamicPart} (split at ## Transcript); extract.ts caches the static system block. Measured: call 1 cache_write=8,373 tokens; call 2 cache_read=8,373. batch.ts deliberately uncached. Shared _qpoint-context.md fragment () feeds extract + analyze prompts — kills the stale-pivot prompt bug class (batch-themes.md deleted).
• W5 — analyze.ts is map-reduce: Stage 0 grouping → Map A per use case + Map B per company (sonnet-5, structured output, skipped when sourceCallIds unchanged; --force regenerates) → reduce on opus-5 over map outputs + index stats only (bounded input forever). Intermediates in data/analysis/{use-cases,companies}/; overview at data/analysis.json. New prompts: analyze-use-case.md, analyze-company.md, analyze-overview.md.
• W6 (amended in-flight) — detail page fetches per-call enriched files data/calls/{id}.json written by build-index (fetching raw processed/{id}.json would have dropped sourceType/company-override/actionItem enrichment); search blobs precomputed per data load instead of per keystroke.

All six workstreams landed and verified 2026-08-31 (same day): two calls re-extracted with validation PASSED and semantic parity against phase-1 output; cache read confirmed; full map-reduce analyze ran on the 4-call corpus producing the first post-pivot analysis.json (use-case scoreboard, per-company trajectories, 7 recommendations); incremental re-run skips unchanged maps; site builds clean. The no-DB verdict stands recorded above with triggers.

Verdict: Stay on git-tracked JSON. No database migration warranted at this scale.

Evidence:

• Corpus growth: 4 calls ≈ 44KB (phase-1 peak 34 calls ≈ 388KB); phase-2 growth trajectory ~15 calls/month → ~1MB in 3 months, ~5MB in 15 months.
• Single writer (Mark), no concurrent multi-user writes, git as write log.
• No relational queries; build-index.ts full rebuild is milliseconds even at low hundreds of calls.
• Scaling problem (analyze.ts whole-corpus prompt) is an LLM-context problem a DB doesn't solve.

Re-evaluation triggers (revisit if any occur):

• Concurrent multi-user writes on the data (org growth).
• Genuine relational query needs (correlations across dimensions that index + lazy fetch can't express).
• Raw-transcript storage footprint (~1.6MB/month) becoming painful (storage or git clone time).

If ever warranted: c8 plan.md:138 prescribes SQLite before anything heavier; index + lazy per-call fetch before reaching for any DB.