Profiling and Fixing Real Performance Bottlenecks

Performance work is experimental engineering. A slow endpoint does not prove that its most complicated function is slow, and a busy CPU graph does not prove that more CPU is the answer. The delay may come from queueing, garbage collection, a database round trip, filesystem contention, or one rare input that dominates the tail. Optimizing before separating those possibilities usually produces faster code that users never notice.

A useful investigation connects four things: a user-visible symptom, a repeatable workload, evidence about where time or memory goes, and a change whose effect survives realistic load. This chapter develops that workflow around a Node.js and TypeScript example, but the reasoning applies to most services.

Start with latency distributions, not a single average #

An average collapses a population into one number. Suppose 99 requests finish in 20 ms and one takes 2 seconds. The mean is about 40 ms, which describes neither the common request nor the unlucky one. Percentiles preserve more of the shape: p50 describes the median experience, p95 exposes a slower cohort, and p99 makes rare stalls visible.

For sorted observations x, a simple nearest-rank percentile uses:

Labels such as rank, p, N, and percentile_p are intentionally plain ASCII so they transfer cleanly into dashboards and test output. Also record throughput and errors; a system can improve p50 by rejecting expensive requests.

Segment before explaining. Split by route, status, payload size, tenant, cache state, database shard, and deployment version. A global p99 can hide that one report type is consistently bad. Compare equivalent windows and include warm-up: JIT compilation, connection establishment, filesystem caches, and autoscaling all distort cold measurements.

Build a baseline and test one hypothesis at a time #

Write the performance claim before touching code: “For reports over 100,000 events, repeated filtering consumes most CPU; grouping once will reduce p95 latency without increasing peak heap by more than 20%.” It identifies a workload, a suspected resource, a proposed mechanism, and guardrails.

The baseline should reproduce the symptom at the smallest realistic scale, not the smallest toy scale. Preserve representative cardinality, key distribution, object size, cache hit ratio, concurrency, and downstream latency. Run enough samples to see variance, then retain raw observations rather than only their summary.

Use paired comparisons when possible: alternate baseline and candidate runs on the same machine to reduce drift from temperature, background work, or shared infrastructure. Report a distribution across rounds. A useful effect-size label is:

speedup = 1.0 means no change; speedup = 2.0 means the candidate takes half as long. For noisy systems, include confidence intervals or at least min, median, p95, and sample count. Never select the best run from many attempts.

Instrumentation and sampling answer different questions. Instrumentation places timers or counters around known operations. It gives exact call counts and domain context, but changes code and can add overhead. Sampling periodically records the current stack. It has lower overhead and discovers unexpected hot paths, but short calls may be missed and stacks need statistical interpretation.

Benchmark and profile a Node.js/TypeScript report #

Consider a report that counts events for every requested user. The first implementation scans all events separately for each user. It looks harmless because filter is concise, but its work grows with both dimensions: O(user_count * event_count).

import { performance } from 'node:perf_hooks';

type Event = { userId: number; bytes: number };
type Row = { userId: number; events: number; bytes: number };

function repeatedScan(events: readonly Event[], userIds: readonly number[]): Row[] {
  return userIds.map((userId) => {
    const matches = events.filter((event) => event.userId === userId);
    return {
      userId,
      events: matches.length,
      bytes: matches.reduce((sum, event) => sum + event.bytes, 0),
    };
  });
}

function groupedScan(events: readonly Event[], userIds: readonly number[]): Row[] {
  const totals = new Map<number, { events: number; bytes: number }>();

  for (const event of events) {
    const current = totals.get(event.userId) ?? { events: 0, bytes: 0 };
    current.events += 1;
    current.bytes += event.bytes;
    totals.set(event.userId, current);
  }

  return userIds.map((userId) => ({
    userId,
    events: totals.get(userId)?.events ?? 0,
    bytes: totals.get(userId)?.bytes ?? 0,
  }));
}

const userIds = Array.from({ length: 2_000 }, (_, userId) => userId);
const events = Array.from({ length: 200_000 }, (_, index) => ({
  userId: (index * 17) % userIds.length,
  bytes: 200 + (index % 800),
}));

let checksum = 0;

function measure(name: string, report: () => Row[]): void {
  const samples: number[] = [];
  for (let round = 0; round < 12; round += 1) {
    const started = performance.now();
    const rows = report();
    const elapsed = performance.now() - started;
    checksum += rows[round % rows.length]!.bytes;
    if (round >= 2) samples.push(elapsed); // discard two warm-up rounds
  }

  samples.sort((left, right) => left - right);
  const p50 = samples[Math.floor(samples.length * 0.5)]!;
  const p95 = samples[Math.min(samples.length - 1, Math.floor(samples.length * 0.95))]!;
  console.log({ name, p50Ms: p50.toFixed(1), p95Ms: p95.toFixed(1) });
}

measure('repeated', () => repeatedScan(events, userIds));
measure('grouped', () => groupedScan(events, userIds));
console.log({ checksum });

Compile it with the project’s normal TypeScript build, then run both a benchmark and a CPU profile. Node writes a .cpuprofile file that Chrome DevTools and several editors can open:

node --cpu-prof --cpu-prof-name=report.cpuprofile dist/profile-report.js
node --heap-prof --heap-prof-name=report.heapprofile dist/profile-report.js

The checksum makes the result observable and guards against accidentally benchmarking work whose output is never consumed. Warm-up is explicit. Data is deterministic, so candidate and baseline see identical inputs. For a serious result, run each implementation in a separate process, randomize process order, pin runtime and CPU limits, and collect many process-level rounds; running them sequentially here is deliberately a compact diagnostic.

Read a flamegraph and classify the constrained resource #

A flamegraph aggregates sampled stacks. Width represents the number of samples containing a frame, approximately on-CPU time for a CPU profile. Vertical position is call depth, not chronology. Color is usually decorative unless the tool defines a meaning. Look for wide plateaus and their callers; do not automatically optimize the widest leaf, because it may be a runtime primitive doing necessary work.

A representative text rendering for the baseline might look like this:

(root)                                      100%
  main                                       97%
    measure                                  95%
      repeatedScan                           88%
        Array.map                            87%
          Array.filter                       63%
          Array.reduce                       19%
      groupedScan                             5%
        Map.get / Map.set                      3%

Interpretation: repeatedScan and its nested array passes dominate samples. The profile supports the repeated-work hypothesis; it does not prove that Array.filter is universally slow. Replacing filter with a hand-written inner loop would preserve the bad complexity. The meaningful fix changes data access from many full scans to one grouping pass.

CPU is only one resource class. Use evidence to choose the next profiler:

Memory has two distinct failure modes. Retention keeps objects reachable and makes the post-GC floor rise. Churn allocates short-lived objects rapidly; the heap may return to baseline, but collection consumes CPU and introduces pauses. process.memoryUsage() gives coarse trends, while heap snapshots answer what retains memory. Compare snapshots after equivalent full workload phases, not arbitrary timestamps.

For I/O, elapsed time belongs to both the caller and the dependency. Trace spans should separate connection-pool wait, DNS/TLS setup, server execution, response transfer, and local decoding. A Node CPU profile can look nearly empty while requests are slow because sleeping is not on-CPU work.

Load-test the system, including GC and the database #

Load testing should reveal a curve, not produce one heroic requests-per-second number. Increase offered concurrency in steps and record completed throughput, errors, percentiles, CPU, memory, event-loop delay, connection pools, and downstream saturation at each step. Stop before an uncontrolled incident in shared environments.

Little’s Law is a useful consistency check for a stable system:

Use compatible units: if throughput is requests per second, latency must be seconds. When throughput stops increasing but latency accelerates, a queue is growing around a saturated resource. Coordinated omission can hide this behavior when a load generator waits for each slow response before scheduling the next request. Prefer a tool that maintains the intended arrival schedule and reports missed or delayed starts.

GC deserves correlation, not superstition. Run representative tests with --trace-gc only in a controlled environment because logs are noisy. Correlate major collection timestamps with latency spikes and inspect allocation profiles. Raising the heap limit may reduce collection frequency while increasing worst-case pause and memory cost; it does not repair retention.

Database optimization begins at the query boundary. Capture normalized query latency, row counts, pool wait, and execution plans. Watch for N+1 queries, missing or ineffective indexes, stale statistics, lock waits, and fetching columns or rows that the application discards. Test plans with production-like cardinality: a query fast on 1,000 uniform rows may choose a disastrous plan on 100 million skewed rows.

The grouped report also demonstrates a tradeoff. It reduces CPU complexity to roughly O(event_count + user_count) but stores one aggregate per observed user. Measure peak heap before accepting it. If cardinality is unbounded, aggregate in the database, stream sorted input, partition the map, or enforce a report limit rather than moving the bottleneck from CPU to memory.

Ship safely, prevent regressions, and avoid optimization traps #

Production profiling must have an explicit overhead budget, duration, scope, access policy, and rollback. Start with existing metrics and traces. Enable higher-detail sampling on one canary instance or a small traffic slice, keep captures short, and treat profiles as sensitive: function arguments, URLs, SQL, and allocation contents can expose customer data. Avoid heap snapshots on a busy process unless you understand their stop-the-world and memory impact.

Deploy the candidate behind a flag or canary. Compare equivalent traffic by route and input shape. Watch latency percentiles, throughput, errors, CPU per request, memory, GC pauses, dependency load, and business correctness. A faster service that doubles database traffic has exported its cost. Keep the rollback mechanical.

After confirmation, turn the discovery into protection at the right level. A deterministic benchmark can detect an algorithmic regression, but timing thresholds are flaky on shared CI. Prefer generous budgets, repeated process runs, trend storage, or operation-count/cardinality assertions. Add a load test for the request-level service objective and production alerts for user-visible symptoms.

Common traps recur:

• Optimizing code because it looks inefficient without profiling the representative path.
• Comparing debug and release builds, different Node versions, or unmatched input data.
• Reporting only the fastest run or only average latency.
• Tuning p50 while p99, error rate, or throughput becomes worse.
• Adding caches without invalidation rules, cardinality limits, and hit-rate measurement.
• Replacing readable code with clever code for a change smaller than normal variance.
• Treating async syntax as parallel CPU execution; JavaScript still blocks its event-loop thread.
• Moving work to workers without including serialization, transfer, queueing, and operational cost.
• Raising timeouts, pool sizes, or heap limits to conceal saturation instead of removing its cause.

Takeaways #

Real optimization starts with a user-visible distribution and ends with production evidence. Measure percentiles, throughput, errors, and resource signals together. Reproduce the relevant data shape and concurrency, establish a baseline, then use sampling to discover hot stacks and instrumentation to explain known phases.

Interpret profiles according to the constrained resource: CPU width is not I/O waiting, a large heap is not automatically a leak, and GC pauses need allocation evidence. Load-test across a curve, include database and queue behavior, and evaluate memory or downstream costs introduced by the candidate.

Most importantly, optimize a falsifiable mechanism. In the report example, the win came from removing repeated scans, not from rewriting Array.filter. Canary the change, preserve correctness guardrails, and encode the result as a regression check. That discipline turns performance from folklore into an engineering loop.