Performance Budgets and Regression Testing in CI

Performance regressions are easy to introduce and hard to notice in review. A new allocation appears harmless, a query gains one join, or a bundle imports a convenient dependency. Functional tests still pass because the output is correct. The cost emerges gradually, often after several individually reasonable changes have accumulated.

A CI performance budget makes cost part of the change contract. It does not promise that a shared runner can reproduce production latency, and it should not block a release whenever one stopwatch reading moves. Its job is narrower and more useful: run a controlled workload, compare like with like, and produce enough evidence to distinguish a material regression from ordinary noise. Good gates combine deterministic limits, repeated measurements, explicit uncertainty, and a review policy proportionate to the risk.

Define a Budget as a Product Constraint #

A budget is a limit connected to an outcome, not merely a metric that happens to be measurable. “The compressed initial script must remain below 180 KiB” protects download and parse cost. “Parsing the standard 50,000-record fixture must not regress by more than 8% against the accepted baseline” protects a known processing path. “Checkout p95 must stay below 400 ms” may be a production objective, but it is a poor unit-test threshold unless CI reproduces the relevant service topology.

Choose the layer at which the causal signal is strongest. Static artifact size is deterministic and belongs in ordinary CI. A pure algorithm can use an isolated benchmark. A database path may need a containerized database with fixed schema, statistics, and data. End-to-end latency under contention usually belongs in a controlled performance environment or scheduled pipeline rather than every pull request.

Each budget needs five fields:

Prefer metrics close to the mechanism when possible. An assertion that a repository lookup performs at most three SQL statements can be more stable than a 12 ms timing limit. A maximum allocation count may detect a copied buffer even when wall-clock noise masks it. Timing remains necessary for interactions the structural metrics cannot represent, but it should not be the only language of performance.

Do not create a budget merely by recording today’s result and rounding up. Start from the user or operational requirement, reserve margin for layers outside the test, then verify that the implementation has headroom. A baseline describes current behavior; a budget describes unacceptable behavior. They may begin near each other, but they serve different purposes.

Make the Workload Representative and Bounded #

A benchmark can be perfectly repeatable and irrelevant. Fixtures must preserve the properties that determine cost: cardinality, value distribution, object size, key skew, cache state, compression ratio, query selectivity, and concurrency. Random bytes are not a substitute for realistic JSON if parsing and compression are under test. Ten database rows will not expose a plan change that appears at one million rows.

Representative does not mean copying uncontrolled production data into CI. Build deterministic synthetic fixtures from documented distributions, or sanitize and freeze a legally approved sample. Give the fixture a version and checksum. Keep generation separate from measurement so setup work does not leak into timed regions. If the fixture changes, treat it as a baseline migration and compare old and new deliberately rather than accepting a sudden discontinuity.

Bound execution so the suite remains usable. A pull-request tier might run small, high-signal cases in under a few minutes. A merge or nightly tier can cover larger cardinalities, more runtimes, and concurrent load. The tiers should share metric names and result format so developers can move from a warning to deeper evidence without translating tools.

Warm state must be explicit. For a JIT runtime, run unmeasured iterations until the intended tier is reached, while recognizing that a separate JIT-benchmarking discipline is needed for compiler behavior itself. For database tests, state whether pages and prepared plans are cold or warm. For HTTP tests, decide whether connection establishment is included. Neither state is universally correct; ambiguity is the defect.

Also prevent the benchmark from being optimized away or measuring a mock. Consume outputs with a checksum or assertion. Confirm that the requested number of rows, bytes, or operations was processed. A fast benchmark that silently exercises an empty fixture is more dangerous than no benchmark because it creates false confidence.

Build a Stable Baseline Without Freezing the Past #

There are two common comparison models. A fixed baseline stores accepted measurements from a known commit and machine class. A paired baseline builds or runs the target branch and candidate branch in the same job. Fixed baselines are cheap but drift as runtimes, dependencies, and runners change. Paired baselines cost more but cancel much of the current host’s bias.

For timing-sensitive gates, prefer paired execution on dedicated or at least strongly isolated workers. Alternate baseline and candidate rounds rather than running all baseline rounds first; thermal state and background contention can drift over time. Use separate processes when global caches, heaps, or JIT state would let one variant contaminate the other. Pin runtime, compiler flags, CPU architecture, container limits, and important environment variables.

The comparison branch must be trustworthy. For a pull request, it is often the merge base or the target branch built with the same toolchain. If that baseline fails functional checks or cannot run the fixture, report an infrastructure or baseline error rather than declaring the candidate faster. Keep the raw measurements and identities of both commits.

A baseline should advance only through an explicit decision. Automatically replacing it whenever main changes can normalize a slow creep: ten accepted 2% regressions become a substantial loss while each individual gate passes. Retain an absolute product budget alongside relative comparison, and graph the metric over time. Relative gates catch abrupt changes; absolute limits and trends catch accumulation.

Some metrics need no empirical baseline. Compressed bundle size, generated file count, query count, and peak configured cache entries can often be calculated exactly. Use exact comparisons for exact data instead of wrapping everything in a statistical framework.

Work Through a Noise-Aware CI Gate #

Consider an illustrative command that converts a frozen event fixture into a compact report. Product profiling has already identified it as important; CI is not trying to rediscover the bottleneck. The accepted baseline has repeated process-level durations, and the candidate is measured in alternating rounds on the same worker.

Suppose the job records these illustrative medians after warm-up:

The paired ratios are consistently around 1.06. That pattern is stronger evidence than comparing the single fastest baseline run with the single fastest candidate run. The gate has three predeclared rules: report at 3%, require review at 5%, and block at 8%; additionally, block if the candidate exceeds the absolute 950 ms budget. This result requires review but does not automatically block.

The job should publish a machine-readable record rather than only colored console text:

{
  "benchmark": "event-report/50k-skewed-v3",
  "unit": "ms",
  "baselineCommit": "target-commit",
  "candidateCommit": "change-commit",
  "baselineMedian": 814,
  "candidateMedian": 858,
  "pairedMedianRatio": 1.059,
  "absoluteBudget": 950,
  "decision": "review",
  "rounds": 5
}

The values are illustrative, not claimed benchmark results. In a real harness, retain every sample, process exit status, warm-up count, host identity, runtime version, and fixture checksum. A summary without raw observations cannot be audited when a developer challenges a failure.

Review asks whether the effect is expected and justified. A change may intentionally spend 6% more CPU to eliminate an incorrect result or reduce memory substantially. The budget has still done its job by making the tradeoff visible. Approval should record the reason, owner, and whether the accepted baseline or absolute limit changes.

Treat Variance as Data, Not an Excuse #

Shared CI hosts introduce scheduler contention, frequency scaling, virtualization effects, filesystem neighbors, and network variability. Application behavior adds garbage collection, background compilation, and cache transitions. Repeating a noisy test does not automatically make it valid; first remove avoidable variance from the harness.

Inspect distributions and paired ratios. Median is robust against occasional extreme host pauses, while a trimmed mean can use more data if trimming rules are fixed in advance. For latency objectives that specifically concern a tail, collect enough events for the chosen percentile and use a workload environment capable of producing them. Five iterations cannot support a meaningful p99 gate.

Confidence intervals or resampling can quantify uncertainty, but statistical significance is not practical significance. With enough samples, a 0.3% difference may be statistically detectable and operationally irrelevant. Conversely, a noisy 15% estimate can be too risky to ignore even if a conventional significance test is inconclusive. Define a minimum material effect and an uncertainty policy before seeing the candidate.

Use reruns sparingly. “Rerun until green” selects favorable noise and corrupts the false-positive rate. An automatic confirmation run is reasonable when the first result enters a narrow ambiguous band, provided the policy combines all prescribed samples and permits only one confirmation. Infrastructure failures such as a throttled worker should be classified from host telemetry and retried separately from performance failures.

Control charts are useful for recurring benchmarks. Compare current results with recent central tendency and natural variation, while preserving the absolute budget. Sudden shifts, rising sequences, and widening variance can reveal environmental drift or an unstable implementation before a hard threshold is crossed.

Choose a Failure Policy Developers Can Trust #

Not every metric deserves a red build. A hard gate should have high consequence, low measurement ambiguity, a clear owner, and an actionable remediation path. Exact artifact-size limits often qualify. A microbenchmark on an oversubscribed public runner usually does not.

A practical policy has several outcomes:

• Pass: below absolute and relative thresholds with adequate evidence.
• Warn: a small material movement or trend change worth displaying in review.
• Review: a larger change requiring named approval and a written tradeoff.
• Block: an absolute contract breach, a severe relative regression, or a critical structural invariant failure.
• Inconclusive: insufficient samples, unstable host, broken baseline, or invalid fixture; this is not a pass.

Prevent silent budget inflation. Store budget definitions beside the code or fixture, require owner review for changes, and show old and new values in the pull request. An emergency override should expire or create follow-up work; otherwise temporary exceptions become permanent policy. Baseline updates need the same visibility as source changes.

Make failure output diagnostic. Name the benchmark and fixture, show baseline and candidate distributions, identify the violated rule, link the raw artifact, and print an exact local reproduction command when reproducibility permits. Do not ask every developer to become a statistician to learn that a bundle gained 40 KiB.

Budget ownership matters when multiple teams share a path. The team changing a common library may not know every consumer benchmark. Map metrics to owners, run affected suites through dependency metadata, and provide a documented escalation path. Unowned red gates are eventually disabled.

Use Trends to Catch Slow Regressions #

Pull-request comparisons are optimized for attribution: what did this change do? Trend systems answer a different question: where is the system heading? Store one canonical result from the main branch on a stable cadence, with commit, environment, fixture, and tool versions. Plot central value, variation, and absolute budget together.

Annotate runtime upgrades, runner migrations, fixture revisions, and known infrastructure incidents. A step change after moving CPU architecture may be real but should not be attributed to the nearest product commit. Preserve old series rather than rewriting history; begin a comparable segment when methodology changes.

Review trends by a fixed ownership cadence. Look for repeated warnings, variance growth, and metrics approaching the hard limit. A slope alert can open work before the budget is exhausted, but avoid extrapolating indefinitely from a short noisy series. Correlate related metrics: duration rising while operation count remains fixed suggests a different mechanism than both rising together.

Aggregate dashboards must not erase slices. One overall serialization benchmark can remain flat while the large-payload fixture degrades. Retain the cases tied to distinct complexity regimes and product risks. Remove benchmarks that no longer represent a supported path, but document that retirement rather than letting stale green checks accumulate.

Verify the Harness Before Trusting the Gate #

Test performance infrastructure like production code. Feed the comparator synthetic sample sets with known pass, review, block, and inconclusive outcomes. Verify boundary equality, unit conversion, missing samples, NaN values, process failure, and mismatched fixture versions. Ensure a lower-is-better metric and a higher-is-better throughput metric use opposite comparison directions.

Create a deliberately slower implementation or inject controlled delay to prove the gate turns red. Then restore it and prove the job recovers. For structural metrics, add a fixture that exceeds query or allocation limits. These mutation-style checks catch harnesses that collect a number but never apply policy.

Audit reproducibility from a clean checkout. The command should pin inputs, exclude setup from timed regions, consume outputs, and emit raw results. Compare local and CI behavior without demanding identical milliseconds; they should agree on workload identity and, for a sufficiently large injected regression, on direction. Monitor worker CPU throttling, memory pressure, and clock source so invalid environments become inconclusive rather than misleading.

Finally, periodically compare the CI signal with a production or controlled end-to-end indicator. A benchmark can lose relevance as architecture and traffic change. That does not make CI performance testing futile; it means budgets require maintenance like any other executable specification.

The durable approach is layered. Use deterministic cost invariants wherever possible, paired and repeated measurements for timing, absolute limits to prevent slow creep, and trends to expose gradual change. Couple every threshold to a fixture, owner, and decision policy. A trusted performance gate does not claim that CI is production. It gives reviewers consistent evidence that a change altered an important cost, early enough to make the tradeoff deliberately.