Did the Change Help? Metrics, Baselines and a Trustworthy A/B Test

Two approaches, one comparison plan. AI-generated illustration; no real experiment or results shown.

A workflow produces drafts more quickly after a change. Has it improved? That depends on what the drafts are for. A shorter wait, a usable result and a worthwhile business outcome answer different questions. Before drawing a chart, write down the decision the measurement is supposed to support.

Start with the result you need

Imagine a team preparing source-backed editorial briefs. It proposes adding a source checklist to its draft template. The objective is to deliver briefs that a reviewer can accept without correcting the evidence. This is a fictional proposed change: no briefs have been assigned, no experiment has run and no improvement has been observed.

A metric is a defined quantitative measure. For this example, we will call the measure selected to judge the objective the key performance indicator, or KPI. The chosen KPI is the percentage of eligible briefs accepted without evidence correction. Drafting time remains a separate measure; it does not replace the acceptance question.

OpenTelemetry’s observability primer describes metrics as numeric aggregations over time, including signals such as error rates and request rates. Such operational signals help explain system behavior. A system can be available while delivering the wrong result. Likewise, a drafting process can be fast while producing briefs that require evidence repair; measuring speed alone would leave our objective unanswered.

Give the KPI a measurement contract

In the fictional proposal, “accepted without evidence correction” means the reviewer accepts the brief without replacing an unsupported factual statement or repairing a source mismatch. Style edits do not count as evidence corrections. A reviewer records the outcome against the assigned brief, rather than treating each revision as another successful brief.

The proposed formula is: accepted eligible briefs without evidence correction ÷ all eligible assigned briefs × 100. Its unit is percent of assigned briefs. Each brief contributes at most once to the numerator and denominator. A zero denominator leaves the percentage undefined; it is not a zero acceptance rate.

The follow-up deadline is still a design choice, not a hidden assumption. Under this proposal, a brief pending at the cutoff is not a success, even if accepted later. Both variants must use that rule. The team should keep the underlying outcome labels so a delay is distinguishable from an evidence failure.

Keep the baseline comparable

A baseline is the reference against which a result is compared. Google’s guidance on implementing a model recommends starting simply and using a simple model as a baseline before adding complexity. Applied to our workflow question, the existing template is a useful reference. A newer template earns no presumption of being better.

For the fictional historical baseline, the team would need records using the same acceptance definition, eligible work, denominator and follow-up rule. If older records count only completed briefs, they cannot be silently compared with all assigned briefs. The team must reconcile the definitions or state that the historical percentage is unavailable.

Even a compatible before-and-after comparison describes a change over time; it does not by itself identify the cause. Easier briefs, a different reviewer or a quieter week are alternative explanations in this imagined case. The proposed checklist needs a comparison designed around that causal question.

Compare variants through a planned A/B test

An A/B test compares variants on predefined outcomes, usually through random assignment of eligible units. Microsoft Research’s pre-experiment design guidance emphasizes a clear hypothesis, appropriate metrics and a suitable randomization unit. Randomization supports a causal comparison under a valid design; it does not guarantee one. Shared people or resources can let one variant affect the other. The analysis must account for the assignment unit and dependencies.

Sample planning depends on the effect worth detecting, outcome variability, eligible units and acceptable uncertainty. Repeated observations within one assigned unit do not automatically create independent samples. The observation window must allow relevant outcomes to mature. There is no universal number of briefs or test duration that makes this proposal trustworthy.

Here, variant A would retain the current template and variant B would add the checklist. Briefs are only a provisional assignment unit: the proposal is not ready to run until the team resolves whether reviewers learning from B would change their treatment of A. The allocation plan and analysis would then be specified together.

Once the design is qualified, eligible units would be randomly assigned to A or B over the same comparison period, retaining their assignments through follow-up.

Decide what the evidence could justify

Before starting, define the primary outcome, guardrails, allocation checks, analysis and stopping rules. Evaluate the estimated difference with its uncertainty. A favorable point estimate may still leave the decision unresolved. Do not declare a winner merely because an interim result looks encouraging.

For this fictional decision, the team would specify how much improvement matters and how much additional review delay it could tolerate. It would also require an explanation of missing outcome records before interpreting the comparison. None of those thresholds has been selected here; filling them in is part of planning, not a result to invent.

From AI Demo to Reliable Workflow provides the surrounding input and output contract and makes the next responsible role explicit. This measurement contract adds a narrower question: which recorded outcome would justify changing the process? Until the definitions, comparison and uncertainty support an answer, “we measured it” is a description of activity, not evidence that the change helped.