dew.telemetry.benchmark
Wall-clock timing of calls: samples, which every benchmark tool in Dew measures independent
calls with, carried, which times calls that carry a state from one to the next, benchmark,
which lays out many calls’ timings as a table, and compared, which reads two calls against each
other by their paired ratio; and the process’s resident memory as the kernel reports it
(resident_memory), which the memory benchmarks read.
A run is timed from the call to the moment its output is ready on the devices
(jax.block_until_ready), so an asynchronous dispatch is timed as the work it
starts, and reading the output back to the host is not timed. The warm-up runs,
which compile what the call traces, are not kept. The caller passes what is
timed, a jitted function bound to its inputs: nothing is jitted here.
| Name | Summary |
|---|---|
Benchmark | What benchmark measured: each call’s timed runs in seconds, in order, and its last output. |
Comparison | What compared measured: each pair’s ratio of the first call’s seconds to the second’s, the median ratio and its interval (_median_rank), and the second call against itself (its A/A), whose interval is the placement’s own noise: a difference is one where interval lies apart from 1 and outside alone. |
benchmark | Time each of calls, in order, repeats times after warmup untimed runs (samples), and return the runs’ seconds and each call’s last output (Benchmark); no two of a call’s outputs are held at once. |
carried | count calls of step, each handed the state the last returned (a training step’s state, a decoder’s cache), and the seconds they took, with the state the last call returned. |
compared | first against second, each timed as samples times one call, in pairs interleaved pairs whose order a fair coin picks each pair (a call that runs second in a pair runs on whatever the first left), and second against itself in as many pairs, read as the median of each pair’s ratio of seconds and the interval its pairs’ order statistics give it at level (_median_rank): only paired ratios carry from one machine to another, and one-sided runs of an unchanged call moved -0.6% to +0.6% (the kernel challenge’s measurements, row 3616). |
samples | Seconds per call of fn(*args), one sample a synchronized group of calls calls, after warmup groups. |
Benchmark
Section titled “Benchmark”class Benchmark( seconds: Mapping[Hashable, tuple[float, ...]], outputs: Mapping[Hashable, T],)What benchmark measured: each call’s timed runs in seconds, in order, and its last output.
table() and a notebook’s display show each call’s median. Labels that are all pairs,
(row, column), lay out as a grid of rows by columns, a sampler against its step counts say.
Benchmark.median
Section titled “Benchmark.median”def median(label: Hashable) -> floatBenchmark.table
Section titled “Benchmark.table”def table() -> strThe medians as aligned plain text, one row a label (or a row of pair labels).
Comparison
Section titled “Comparison”class Comparison( ratios: tuple[float, ...], ratio: float, interval: tuple[float, float], alone: tuple[float, float],)What compared measured: each pair’s ratio of the first call’s seconds to the second’s, the median
ratio and its interval (_median_rank), and the second call against itself (its A/A), whose interval is
the placement’s own noise: a difference is one where interval lies apart from 1 and outside alone.
benchmark
Section titled “benchmark”def benchmark( calls: Mapping[Hashable, Callable[[], T]], *, warmup: int = 1, repeats: int = 3,) -> Benchmark[T]Time each of calls, in order, repeats times after warmup untimed runs (samples), and
return the runs’ seconds and each call’s last output (Benchmark); no two of a call’s outputs
are held at once. Raises ValueError for no timed run or a negative warm-up.
carried
Section titled “carried”def carried( step: Callable[[S], S], state: S, count: int, *, each: bool = False,) -> tuple[S, list[float]]count calls of step, each handed the state the last returned (a training step’s state, a
decoder’s cache), and the seconds they took, with the state the last call returned. With each,
one sample a call, synchronized before the next starts, which times every call alone; otherwise
one sample the window, its calls dispatched as a run dispatches them and synchronized once at its
end, which times their throughput. A carried state is never released between calls, so its
windows are timed apart from samples, whose calls are independent. Refuses no call.
compared
Section titled “compared”def compared( first: Callable[[], T], second: Callable[[], T], *, pairs: int = 400, warmup: int = 3, level: float = 0.95,) -> Comparisonfirst against second, each timed as samples times one call, in pairs interleaved pairs whose
order a fair coin picks each pair (a call that runs second in a pair runs on whatever the first left),
and second against itself in as many pairs, read as the median of each pair’s ratio of seconds and
the interval its pairs’ order statistics give it at level (_median_rank): only paired ratios carry
from one machine to another, and one-sided runs of an unchanged call moved -0.6% to +0.6% (the kernel
challenge’s measurements, row 3616). The coin makes the A/A’s ratios fall above and below 1 as
independent fair coins, however the timings depend on one another, so its interval holds 1 with chance
at least level, exactly; the pair’s holds its median ratio so where its pairs are independent. The
caller binds each call to fresh inputs where its time depends on its data (a MoE’s routing). Raises
ValueError where pairs reach no interval at level.
samples
Section titled “samples”def samples( fn: Callable[..., T], *args=(), warmup: int = 3, repeats: int = 10, calls: int = 1,) -> list[float]Seconds per call of fn(*args), one sample a synchronized group of
calls calls, after warmup groups. With one call a group each sample is
a call’s latency, and its result is released before the next call starts,
so no two calls’ outputs are ever held at once; more calls a group queue
their dispatches and synchronize once, which measures throughput. The
caller takes the statistic and units it reports.