Realism & the quality loop

The philosophy: show your work. Every graded run leaves an auditable row. Every score is reproducible. Every claim in this page is backed by data you can pull yourself from GET /api/v1/quality.

The 10 synthetic tells we grade against

Each was observed in real output before it entered the taxonomy — none are hypothetical:

Tell	What it looks like
`clone_flavor`	Two panelists share a catchphrase, backstory, or verbal fingerprint.
`convergence_theater`	Everyone lands the same verdict in slightly different words.
`uniform_register`	Every speaker is equally articulate, polished, and structured.
`uniform_polish`	No fragments, no trailing off, no "I don't know" — essay-speak.
`scripted_reactions`	Turns mirror the moderator's question structure or each other's arcs.
`stimulus_parroting`	Quoting copy that isn't in the stimulus, or reciting it as opinion.
`unverified_segment`	A persona claims another consumer type's defining story.
`incoherent_bio`	Attributes contradict the life story (a Millennial "retiree").
`shared_panel`	The same faces reappear across supposedly fresh groups.
`domain_leakage`	B2B jargon in consumer mouths; strategy language from shoppers.

Pass gradeRealism: true to focus_group. A JUDGE-tier model reads the full transcript + panel description and returns a 0–100 score, a verdict (passes_as_human / minor_tells / obviously_synthetic), and evidence-quoted tells — each with an engine-actionable fix suggestion.

Ledger

Every grade persists to the feedback ledger (a database row in prod). The ledger is the engine's institutional memory: score trends by engine era, recurring tells, judge-suggested fixes.

Repair

Recurring tells (≥3 occurrences or ≥2 high-severity) fold into repair tickets. Engine changes ship against those tickets — tested, committed, and re-measured with fresh graded runs. The engine never silently rewrites its own prompts.

What the loop has shipped (measured)

Engine change	Judge impact
Distinct-tuple panel draws + anti-clone dedupe	55 → 74
Detailed v2 persona prompts (Fable-optimized)	+2.5 vs v1 same-seed
Post-generation idiom dedupe (code-detected shared n-grams → banned-phrase regeneration)	residual shared idioms → 0–1/run
Semantic beat dedupe (no two panelists carry the same conversational beat)	peak 85
Model-realized panels (one frontier call casts the whole panel under a distinctness contract)	76 → 83.5 mean, now default

Free diversity telemetry on every run

Every focus_group response carries a diversity object — computed in code, no judge call, no extra cost:

"diversity": {
  "crossSpeakerLexicalOverlap": 0.002,   // 0 = fully distinct voices
  "turnLengthVariation": 0.33,           // higher = real length spread
  "openerDiversity": 1.0,                // 1 = every opener unique
  "sharedIdiomCount": 0,                 // phrase-clones that survived
  "lengthRangeRatio": 2.63               // longest / shortest turn
}

The scorecard endpoint

GET /api/v1/quality

{ "ok": true, "data": {
  "graded": 29,
  "recentMean": 79.4,
  "latest": { "toolName": "focus_group", "score": 82, "verdict": "minor_tells" },
  "trend": [ { "era": "panelRealizer-model", "runs": 2, "mean": 83.5, "min": 83, "max": 84 } ],
  "topTells": [ { "tell": "uniform_polish", "count": 3, "highSeverity": 0 } ],
  "repairQueue": [ { "tell": "clone_flavor", "occurrences": 8, "status": "open",
                     "suggestedFixes": ["Add a draw-time constraint that household-composition beats are panel-unique…"] } ],
  "methodology": "Scores are produced by a JUDGE-tier frontier model… ±5 single-run noise — trust 3+ run means."
} }

Honest limits

Single-run scores carry ±5 noise — evaluate 3+ run means, never one number. The judge shares a base-model family with the personas (a cross-model judge is env-switchable via MODEL_JUDGE), so self-grading may under-detect family-level idioms. Synthetic panels also read more critical than matched human panels — treat directional negatives as real, but size effects with quantitative follow-up. Synthetic evidence is directional (Tier 3): route go/no-go decisions through real-human quant.