Mavera DocsDrift & accuracy back-tests
How well do synthetic reads track real outcomes?
Drift — did it move between two runs?
{ "metric": "nps", "baselineRunId": "run_q1", "currentRunId": "run_q2", "threshold": { "sigma": 1 } }| Field | Meaning |
|---|---|
delta | Raw change (current − baseline). |
deltaZ | Magnitude in standard deviations vs the baseline CI. |
significance | p-value against the baseline confidence interval. |
direction | up · down · flat |
classification | stable (within noise) · moved (past threshold) · reversed (flipped sign) |
The threshold.sigma controls sensitivity — 1 flags a 1σ move, 2 only flags larger shifts. This is the canonical node inside scheduled trackers: wire it to a cron over a repeating study and it stays silent until something genuinely moves, then emits an alert-ready payload.
Accuracy — how right was the prediction?
This is the flywheel: once the real-world number lands, score the synthetic prediction against it.
{ "testType": "pricing", "feedBack": true }
# → correlation / MAE / bias of Mavera's pre-launch predictions vs the realized metric, per test type.| Metric | What it tells you |
|---|---|
| correlation | Did the synthetic ranking match reality's ranking? (did we pick the right winner) |
| MAE | Mean absolute error — how far off the level was, on average. |
| bias | Systematic over- or under-prediction (signed) — correctable. |
With feedBack: true, the realized outcome is folded back in to recalibrate future predictions of that test type.
The published track record
POST /api/v1/tools/get_parity_history { }
# → latest overall parity per study type + a stability band — your defensible "how much did these claims hold up" answer.And to score a single run against a named real-world benchmark on the same questions:
{ "studyId": "run_8842", "benchmark": "..." }
# → Pearson / Spearman against the benchmark.Hybrid validation with real humans
For a study you want to harden, route a subset to real panelists through a provider-abstracted hook and compare:
{ "studyId": "run_8842", "n": 50 }Back-testing older studies: the pattern is (1) compare_studies to see the trend, (2) detect_drift to confirm a real move, (3) measure_synthetic_accuracy / get_parity_history once reality lands to grade the original prediction. That sequence answers "did our past calls hold up" with numbers, not vibes.