Skip to content

dew.telemetry.profile

Process-local JAX profile captures, exported as XProf’s own reports.

A capture runs only when you start a Profiler. Starting and stopping it wait for the default backend’s live arrays and effects, but they do not synchronize with other processes. When no capture is running, Dew installs no instrumentation hooks.

NameSummary
ProfilerCaptures a JAX profile of this process, as a context manager or through start and stop.
regionReturn a context manager that marks a named span in the running capture, or does nothing when no capture is running.

class source

class Profiler(
directory: str | os.PathLike[str] | None = None,
*,
options: jax.profiler.ProfileOptions | None = None,
reports: bool = True,
)

Captures a JAX profile of this process, as a context manager or through start and stop.

One profiler can capture more than once. Each start creates a new capture directory under directory. Without a directory, the first start creates a temporary one, which is kept after the run. A capture covers only this process, so other processes must start their own. Starting and stopping wait for the work on JAX’s default backend to finish, without copying array values to the host. Only one capture can run in a process at a time, and start raises RuntimeError while another is running.

A capture includes the HLO of every executable alive in the process, not only of the work it traced, and stop exports all of it. The op_profile, hlo_stats and roofline_model reports are built from that HLO and come out nearly empty without it. In a training run’s window this costs little. A long-lived process that has compiled many other programs pays for each of them, so there pass options with enable_hlo_proto off when the trace and kernel reports are enough.

On a GPU the reports also come out empty for work that already ran before the capture, as a Trainer’s profile window does: XLA runs a step’s kernels as CUDA command buffers, and their device time is attributed to their module, not their ops (on JAX 0.11.2 an RTX 4080’s warmed step named no op; 63 when its first run fell inside the capture; 166 with command buffers off). XLA means to run command buffers op by op while a profiling session is active (command_buffer_thunk.cc, ExecuteOnStream), and here does not; why is open, and tools/gpu_profile_check.py records whether it does on every GPU session. The manifest’s op_attribution then says so and a warning names the fix, profiling with XLA_FLAGS=--xla_gpu_enable_command_buffer=; time runs without that flag, since launching kernels one by one costs time. Where hlo_stats has no column naming the ops, op_attribution records that it could not tell.

A capture whose reader reads its trace alone (dew.telemetry.trace), as a benchmark’s device timeline or XLA’s profile-guided latency estimates do, takes reports=False: it needs no XProf, and its stop writes its manifest alone, not the reports, whose conversion takes minutes for a training window’s capture (row 2017).

captures: tuple[Path, ...]

The capture directories this profiler’s stops exported, in order: each holds its manifest and its reports/ (report_table, program_kernels). Where an export failed (the manifest’s capture_errors, export_status) reports/ may lack a report.

def region(name: str) -> AbstractContextManager[None]

Return a context manager that marks a named span in the trace while this profiler is capturing.

Wrap the work whose device and host time should show under name. Outside a capture the span does nothing, so the same code runs unprofiled without a branch at every call site.

def start() -> Self
def stop(*, wait: bool = True) -> None

Stop the capture and export its reports and manifest: at once, or where not wait, in a process of its own beside the caller’s work (dew.telemetry.export), the capture already stopped, which exported waits for. A training run’s window stops so, since converting a capture takes far longer than the steps it traced (row 2017: minutes for a 3-step window). A stop that failed exports at once, so its manifest records why, as it raises.

function source

def region(name: str) -> AbstractContextManager[None]

Return a context manager that marks a named span in the running capture, or does nothing when no capture is running.

The call site does not need to know whether a profiler is running or which one; with region("input.wait"): reads the same either way.