Thread `intents-pr5`: evaluation (2026-11-05)

From intents-pr5-prompt.md. It feeds intents-pr5-plan.md.

Evaluated against the lab as it stands today, uncommitted work included: the running :3000 pages, their code, and PR 5's counting rule run over the lab's world.

Update, same day (Mark): merging PR 5 wasn't meant to settle anything over the Nov 4 metric, so the "Replace" results below say what PR 5 asks for, not what's been decided. Mark's rule: posture reports only on what Qpoint can know and govern. Mark then ruled two metrics: a posture score of progress toward each policy's best possible mode, and a separate exceptions metric. Compliance Score stays as built, marked projected, and living reports stay over PDF. PR 6 merged later the same day; Mark ruled that its four reports replace our three, as living pages. See the plan's rulings.

The short version

Compliance and framework detail need rework, not a polish. PR 5 excludes what the 2026-11-04 posture-score thread built. That thread's Compliance Posture is subject-weighted and exception-adjusted, and PR 5 lists both as v1 exclusions. Compliance Score is out as well. The instinct behind the thread survives: configuration and what actually happened stay two separate readings.

Policies needs little. The frameworks strip has to go, which c126 and c123 already planned. PR 5 now makes it a requirement.

Reports is the largest change, but it's driven by PR 6, not PR 5. PR 6 is still open and unreviewed. It replaces our three live reports with four generated PDFs. PR 5 only forces the framework section to change.

One earlier ruling reverses. Tracking no longer stages policies at Observe (c118 A6 and Q10, c123 F8). Tracking is now a reporting preference outside the draft.

What changed in ../intents

PR 5 only adds files. The older observe_view-a-framework-details.md and observe_view-policies-and-frameworks.md are still on main and still describe coverage states and gaps. Treat the two new files as the current word.

PR 5's rules

1. Track / Untrack, Tracked / Untracked. Tracking is an organization reporting preference. It changes no policy, it saves outside the shared draft and publish, and each change is audited with actor, time and previous and new state.
2. One measure: "Policy enforcement posture". Off / Observe / Enforce counts and proportions, with the denominator visible.
3. The unit is a distinct policy-to-control mapping. Each one counts once, equally weighted, and takes the policy's published organization default mode. Pending changes don't move it.
4. Exceptions annotate; they don't count. Each bucket shows its distinct active published exception records (once per bucket, whatever the mapping count). Expired, revoked and pending ones are excluded.
5. Excluded in v1: a single weighted score, compliance scoring, and exception-adjusted, subject-weighted or device-weighted calculations. More sophisticated exception-aware calculations are named as future work.
6. No mapped policies reads "No mapped policies" with an unavailable ratio, never 0% or 100%. Unsupported requirements sit outside the denominator.
7. Framework detail has four parts: understand the framework, controls and mapped policies, posture, posture over time.
8. Vocabulary: "framework controls" is the umbrella term. Source terms stay: OWASP risks, NIST AI RMF outcomes, SOC 2 criteria. Native groupings stay. Short labels are told apart from official titles.
9. History is reconstructed from retained policy, exception and mapping history, including periods before or while untracked. Reconstructed ranges are marked, and missing history is unavailable rather than zero. Mapping changes are marked so a denominator change doesn't look like progress.
10. Other evidence (Sessions, Finding resolution, configuration history) can be described separately but adds no posture units.
11. Fixed v1 scope. No tailoring, custom frameworks or editable mappings. Future tailoring is described but not committed.

The two metrics against PR 5

This is the core conflict, so it comes first.

What survives from Mark's prompt: how we're configured and how it's going are still two readings, and each covers the other's blind spot. PR 6's Compliance Report is built the same way: posture, then Enforced / Observed Findings. Only the formula and the word "score" go.

The numbers: what the mock would show

PR 5's rule run over the lab's established world, with exceptions counted as live() counts them:

Three things this shows:

• The story changes. "50%" reads as half-done. "7% Enforce" reads as an org that's mostly observing, which is true of this world.
• The band has to change too. Framework detail's donut counts distinct policies (16: 2 / 10 / 4). PR 5 counts mappings (28: 2 / 21 / 5). If both appeared they would disagree, so the donut's basis moves.
• SOC 2 is empty. It's in PR 5's proposed catalog with nothing mapped in the lab, so most of its page can't be reviewed until mappings exist.

Two consequences of the rule are worth raising. Neither is a defect in the mock:

• Observe-only policies cap the ratio. 11 of OWASP LLM's 28 units sit on policies that can't enforce, so it can never pass 61% Enforce. The intent doesn't say so, and a reader will assume 100% is reachable.
• Pilots don't move the ratio. In the ?stage=adoption world, OWASP LLM reads 11 / 89 / 0 with 5 exceptions, while Engineering and Dev are enforcing. The adoption story's main progress shows only as an annotation. PR 6 asks for a tightening / loosening summary in the report; the framework page probably needs it too.

Surface by surface

Compliance listing (/mock/compliance)

Framework detail (/mock/compliance/detail)

Policies (/mock/policies, policy detail)

The policy model is untouched. PR 5 reaches Policies in three places.

There's no bulk route left for "turn on this framework's policies". PR 5's journey is: Manage Framework → Review Policy → Configure and Publish. Framework detail needs a way to go from "5 Off" to those policies. Today it only has per-pill links.

Reassuring: the lab's world is already live state, with the draft shown only under ?preview=draft. So "use published state" costs nothing, and the preview just needs labelling. Removing framework-track from the draft is close to one line, because OPERATIONAL in useGovernanceMock.ts already marks the kinds that apply live.

Reports (/mock/reports)

PR 6 is open, so this is a proposal, but the structure differs enough to weigh now.

What holds: R2's basis tags are PR 6's "selected period vs current snapshot". R6's hatching is its "unavailable, not zero". R3 is its "share definitions". R4 is its "read-only". The footer has the data cutoff. The document face and the rollups all carry over. The window picker stays because c128 D5 makes it the operator pages' control.

Where today's sections go if PR 6 lands:

Our own open questions, now answered

From CYCLE.md and the gap check in intents-redline.md §5.

CYCLE.md's objective needs matching edits: "no percentage" (PR 5 allows proportions, never a score or a % of controls satisfied), "non-policy evidence covers a requirement", the four working states, and F8.

The OWASP tour still uses Evidenced as a per-mitigation chip. That can stay as a description of what Qpoint does for one mitigation, as long as it isn't presented as a posture.

Worth raising with Peterson

Nothing has been sent; the memory rule stands that he revises the intents himself. These are for Mark to bring if he wants to.

1. Observe-only policies cap the Enforce ratio (above). An annotation would solve it without changing the calculation.
2. Pilots are invisible in the ratio (above). Should the framework page show tightening vs loosening, as PR 6's report does?
3. The catalog omits OWASP Agentic, which F4 ruled in on 2026-10-27.
4. Should an untracked framework show current posture? Expected Behavior says the catalog shows posture, and also that retained samples must not read as current. Read together, it's unclear.
5. "Qplane". PR 6 calls the app Qplane throughout, and enforce_publish-policy-configuration.md on main uses it too. Nothing on the design side does. Is it the product's name?
6. Leftover wording. PR 6 still says "enabled framework" and "disabled periods"; PR 5 keeps "framework toggles".
7. Superseded intents on main: the two older framework files, plus the four the redline lists.
8. Findings on a framework page. PR 5 doesn't say which window or basis applies. Report and page should agree.