Skip to content

dew.telemetry.benchmark

Wall-clock timing of calls: samples, which every benchmark tool in Dew measures independent calls with, carried, which times calls that carry a state from one to the next, benchmark, which lays out many calls’ timings as a table, and compared, which reads two calls against each other by their paired ratio; and the process’s resident memory as the kernel reports it (resident_memory), which the memory benchmarks read.

A run is timed from the call to the moment its output is ready on the devices (jax.block_until_ready), so an asynchronous dispatch is timed as the work it starts, and reading the output back to the host is not timed. The warm-up runs, which compile what the call traces, are not kept. The caller passes what is timed, a jitted function bound to its inputs: nothing is jitted here.

NameSummary
BenchmarkWhat benchmark measured: each call’s timed runs in seconds, in order, and its last output.
ComparisonWhat compared measured: each pair’s ratio of the first call’s seconds to the second’s, the median ratio and its interval (_median_rank), and the second call against itself (its A/A), whose interval is the placement’s own noise: a difference is one where interval lies apart from 1 and outside alone.
benchmarkTime each of calls, in order, repeats times after warmup untimed runs (samples), and return the runs’ seconds and each call’s last output (Benchmark); no two of a call’s outputs are held at once.
carriedcount calls of step, each handed the state the last returned (a training step’s state, a decoder’s cache), and the seconds they took, with the state the last call returned.
comparedfirst against second, each timed as samples times one call, in pairs interleaved pairs whose order a fair coin picks each pair (a call that runs second in a pair runs on whatever the first left), and second against itself in as many pairs, read as the median of each pair’s ratio of seconds and the interval its pairs’ order statistics give it at level (_median_rank): only paired ratios carry from one machine to another, and one-sided runs of an unchanged call moved -0.6% to +0.6% (the kernel challenge’s measurements, row 3616).
samplesSeconds per call of fn(*args), one sample a synchronized group of calls calls, after warmup groups.

dataclass source

class Benchmark(
seconds: Mapping[Hashable, tuple[float, ...]],
outputs: Mapping[Hashable, T],
)

What benchmark measured: each call’s timed runs in seconds, in order, and its last output.

table() and a notebook’s display show each call’s median. Labels that are all pairs, (row, column), lay out as a grid of rows by columns, a sampler against its step counts say.

def median(label: Hashable) -> float
def table() -> str

The medians as aligned plain text, one row a label (or a row of pair labels).

dataclass source

class Comparison(
ratios: tuple[float, ...],
ratio: float,
interval: tuple[float, float],
alone: tuple[float, float],
)

What compared measured: each pair’s ratio of the first call’s seconds to the second’s, the median ratio and its interval (_median_rank), and the second call against itself (its A/A), whose interval is the placement’s own noise: a difference is one where interval lies apart from 1 and outside alone.

function source

def benchmark(
calls: Mapping[Hashable, Callable[[], T]],
*,
warmup: int = 1,
repeats: int = 3,
) -> Benchmark[T]

Time each of calls, in order, repeats times after warmup untimed runs (samples), and return the runs’ seconds and each call’s last output (Benchmark); no two of a call’s outputs are held at once. Raises ValueError for no timed run or a negative warm-up.

function source

def carried(
step: Callable[[S], S],
state: S,
count: int,
*,
each: bool = False,
) -> tuple[S, list[float]]

count calls of step, each handed the state the last returned (a training step’s state, a decoder’s cache), and the seconds they took, with the state the last call returned. With each, one sample a call, synchronized before the next starts, which times every call alone; otherwise one sample the window, its calls dispatched as a run dispatches them and synchronized once at its end, which times their throughput. A carried state is never released between calls, so its windows are timed apart from samples, whose calls are independent. Refuses no call.

function source

def compared(
first: Callable[[], T],
second: Callable[[], T],
*,
pairs: int = 400,
warmup: int = 3,
level: float = 0.95,
) -> Comparison

first against second, each timed as samples times one call, in pairs interleaved pairs whose order a fair coin picks each pair (a call that runs second in a pair runs on whatever the first left), and second against itself in as many pairs, read as the median of each pair’s ratio of seconds and the interval its pairs’ order statistics give it at level (_median_rank): only paired ratios carry from one machine to another, and one-sided runs of an unchanged call moved -0.6% to +0.6% (the kernel challenge’s measurements, row 3616). The coin makes the A/A’s ratios fall above and below 1 as independent fair coins, however the timings depend on one another, so its interval holds 1 with chance at least level, exactly; the pair’s holds its median ratio so where its pairs are independent. The caller binds each call to fresh inputs where its time depends on its data (a MoE’s routing). Raises ValueError where pairs reach no interval at level.

function source

def samples(
fn: Callable[..., T],
*args=(),
warmup: int = 3,
repeats: int = 10,
calls: int = 1,
) -> list[float]

Seconds per call of fn(*args), one sample a synchronized group of calls calls, after warmup groups. With one call a group each sample is a call’s latency, and its result is released before the next call starts, so no two calls’ outputs are ever held at once; more calls a group queue their dispatches and synchronize once, which measures throughput. The caller takes the statistic and units it reports.