c10: Pipeline Modernization — Claude API Best-of-Tech + Data Architecture Verdict

Context

Mark asked: are we using best-of-tech to analyze conversations? The pipeline was built in phase 1 (early 2026, SDK 0.39, Sonnet 4-era patterns) and only partially modernized in c8. Two questions: (a) what workflow/efficiency gains exist in the analysis pipeline, and (b) should the git-tracked-JSON data layer move to a DB (explicitly not wanting an unnecessary migration).

Cycle number is c10, not c9 — cycles/c9.project-docs already exists (completed 2026-08-20); numbers are never reused. Suggested name: cycles/c10.api-modernization/. This cycle absorbs c8's deferred "D2 architecture thread" (plan items D2.1 storage, D2.2 structured outputs + retry, D2.3 analyze reshape).

Verdict 1: No DB — stay on git-tracked JSON

Evidence: corpus is 4 calls / ~44KB (phase-1 peak: 34 calls / 388KB); growth ~15 calls/month ⇒ ~1MB in ~3 months, ~5MB in ~15 months. Single writer, git as write log, no concurrent writes, no relational queries. build-index.ts full rebuild is milliseconds and fine into the low hundreds of calls. c8 plan.md:138 already pressure-tested this and prescribed "index + lazy per-call fetch before reaching for a DB; SQLite before anything heavier if ever". The only real scaling problem (analyze.ts whole-corpus prompt) is an LLM-context problem a DB wouldn't fix.

Record in CYCLE.md as a decision with re-evaluation triggers: concurrent multi-user writes, genuine relational query needs, or repo weight from raw transcripts (~1.6MB/month) becoming painful.

Verdict 2: Pipeline is NOT best-of-tech — modernization workstreams below

Audit gaps: SDK pinned at 0.39.0 (current TS SDK is 0.122.0 — note: no 1.x exists for TS; that's Python); zero prompt caching (~11.6KB static prefix repaid on all 47 extractions to date); regex-fence JSON parsing everywhere instead of structured outputs; no retry tuning; analyze.ts sends the whole corpus in one non-streaming prompt using a stale phase-1 prompt (batch-themes.md still says "API infrastructure company"); batch.ts has a latent truncation bug (max_tokens 16000, no stop_reason check, vs extract's 32000).

Workstreams (sequenced)

W0 cycle docs
 └─ W1 SDK upgrade + shared client (foundation)
     ├─ W2 bug fixes
     ├─ W3 structured outputs ──► W4 prompt caching (cache lands on final prompt bytes)
     └─ W5 analyze map-reduce (reuses W3 schema-gen)
W6 site load-path small cut (user-approved; independent)

W0 — Cycle artifacts

cycles/c10.api-modernization/ via cycle-manager agent: initial-prompt.md (Mark's kickoff prompt verbatim), CYCLE.md (Why/Objective/Scope/Key Changes/Outcome — model on cycles/c9.project-docs/CYCLE.md), this plan as plan.md. Update cycles/CYCLES.md. Note c8 D2 items absorbed; mark c8's D2 thread resolved-by-c10.

W1 — SDK upgrade + shared client (~half day)

• @anthropic-ai/sdk ^0.39.0 → ^0.122.0 in package.json.
• Extend pipeline/model.ts (right-sized c8 D2.6 "config layer"): keep MODEL (claude-sonnet-5, ANTHROPIC_MODEL override), add ANALYZE_MODEL (claude-opus-5, ANTHROPIC_ANALYZE_MODEL override), EXTRACT_MAX_TOKENS = 32000, and a shared client = new Anthropic({ maxRetries: 4 }) replacing the three per-script constructions.
• Document both model overrides in .env.example (currently undocumented).
• Verify: npx tsc --noEmit; re-extract one call with --force, diff data/processed/<id>.json for semantic parity.

W2 — Bug fixes (rides along with W1)

1. pipeline/batch.ts downloadResults() (~line 133): check stop_reason === 'max_tokens' → dump raw to .failed-raw.txt (mirror extract.ts:174), skip write, count as failed.
2. batch.ts:56 16000 → EXTRACT_MAX_TOKENS (unify with extract.ts).
3. Retry: rely on the SDK's built-in retries (maxRetries: 4 handles 429/5xx with retry-after). Do not port avoma.ts's hand-rolled backoff to Anthropic calls.
4. nanoid stays — c8's "dead dep" note is wrong (used at pipeline/upload-sagetap.ts:8,126).

W3 — Structured outputs, schema derived from vocabularies.ts (~1 day)

• New schemas/output-schema.ts: buildCallExtractionFormat(useCaseIds) → output_config.format {type:'json_schema', name:'call_extraction', schema}, generated programmatically from the const arrays in schemas/vocabularies.ts (no zod — would break the single-source rule). Observation modeled as anyOf of per-type variants looped from OBSERVATION_TYPES: category enum on market-signal types, useCaseId enum (from registry) + direction/evidenceKind on use-case-evidence, nullable productArea/severity on triage types. Rules: additionalProperties:false everywhere, all props required, optionality via null-unions, 1–5 scales as integer enums, no recursion.
• extract.ts: keep .stream().finalMessage(), add output_config; replace regex fence-parse (lines 156–181) with direct JSON.parse; keep failed-raw dump + max_tokens guard.
• batch.ts: same output_config per request (supported on Batches); same parse simplification.
• schemas/validation.ts: demoted to safety net (registry drift, old/hand-edited files at index time, semantic warnings) — no rewrite.
• Fallback lever if strict enums hurt recall: loosen category to string, keep validator warnings.

W4 — Prompt caching, extraction path (~half day)

• Prompt assembly is already prefix-friendly: all volatile content is spliced at the ## Transcript heading, the last section of extract-call.md. Refactor pipeline/prompt.ts buildExtractionPrompt to return {staticPart, dynamicPart} (split on the marker instead of replace-then-splice).
• extract.ts: system: [{type:'text', text: staticPart, cache_control:{type:'ephemeral'}}], user message = dynamicPart. Static prefix ≈3.5–4.5K tokens — clears Sonnet-5's 1024-token minimum. 5-min TTL suffices for sequential runs.
• batch.ts: no cache_control — concurrent batch processing makes hit rates unreliable; 1.25× cache-writes with few reads could net-increase cost on the already-50%-discounted path.
• Verify: --force run of 2 calls with a one-line response.usage log; call 1 shows cache_creation_input_tokens ≳3000, call 2 cache_read_input_tokens ≈ same.

W5 — analyze.ts map-reduce reshape (~1–1.5 days; implements c8 D2.3)

• Delete stale pipeline/prompts/batch-themes.md. Extract shared company framing into pipeline/prompts/_qpoint-context.md injected via prompt.ts into extract + analyze prompts (kills the stale-pivot bug class permanently).
• Stage 0 (pure code): load data/processed/*.json; group observations per useCaseId and calls per company (with callId/date/company/sourceType context).
• Map A — per use case (analyze-use-case.md): momentum verdict, evidence rollup, explicit source-type agreement/disagreement, direction/evidenceKind balance, strongest verbatims, gaps, next test. Sonnet-5, max_tokens 8000, structured output → data/analysis/use-cases/{id}.json with sourceCallIds + generatedAt.
• Map B — per company (analyze-company.md): longitudinal trajectory, evolving signals, risks, quotes. Sonnet-5 → data/analysis/companies/{slug}.json.
• Reduce — overview (analyze-overview.md): consumes only Map A/B outputs + index.json stats (bounded input forever). Use-case scoreboard, cross-cutting patterns, positioning readout, source-type disagreements, recommendations. Opus-5 via ANALYZE_MODEL (adaptive thinking on by default), .stream().finalMessage(), max_tokens 32000, structured output → data/analysis.json (site path unchanged; keep recommendations: string[] + generatedAt shape).
• Incremental: skip a map task when its sourceCallIds set is unchanged; --force regenerates; reduce always reruns. No Batches/caching here (low volume, unique payloads).
• Verify: run on current 4-call corpus; inspect intermediates; site dashboard renders analysis; second run skips maps.

W6 — Site load-path small cut (user-approved, ~half day)

1. Call detail page (site/pages/calls/[id].vue:378) fetches /data/processed/{id}.json directly instead of finding in the full array — zero server config needed (nitro already serves data/ at /data).
2. Fix per-keystroke search: precompute per-call lowercase search blobs once at load (replacing rebuild at useCallData.ts:251-264).
3. Do not move aggregate views onto index.json — rollups need full observation objects. Type-duplication cleanup (useCallData vs schemas/call.ts) explicitly deferred.

Critical files

pipeline/model.ts, pipeline/extract.ts, pipeline/batch.ts, pipeline/analyze.ts, pipeline/prompt.ts, pipeline/prompts/* (new: _qpoint-context.md, analyze-use-case.md, analyze-company.md, analyze-overview.md), schemas/output-schema.ts (new), schemas/vocabularies.ts (read-only source), schemas/validation.ts, site/composables/useCallData.ts, site/pages/calls/[id].vue, package.json, .env.example, cycles/c10.api-modernization/* (new).

End-to-end verification

1. npx tsc --noEmit clean after each workstream.
2. --force re-extract 2 calls sequentially: schema-valid JSON with zero regex parsing, cache read confirmed via usage log, output semantically comparable to pre-change extraction.
3. One small batch run: results parse, truncation guard exercised in code review.
4. npm run analyze on the 4-call corpus: intermediates in data/analysis/, data/analysis.json written, site renders it.
5. Site: detail page loads a call via its own fetch; search stays responsive; npm run index + dev-server restart per known gotcha.

Cost note

Extraction stays Sonnet-5 (volume path, Batches discount retained). Opus-5 only on the analyze reduce step — runs rarely, bounded input, quality-critical; dollar impact is cents per run.