dew.telemetry.profile
Process-local JAX profile captures, exported as XProf’s own reports.
A capture runs only when you start a Profiler. Starting and stopping it
wait for the default backend’s live arrays and effects, but they do not
synchronize with other processes. When no capture is running, Dew installs
no instrumentation hooks.
| Name | Summary |
|---|---|
Profiler | Captures a JAX profile of this process, as a context manager or through start and stop. |
region | Return a context manager that marks a named span in the running capture, or does nothing when no capture is running. |
Profiler
Section titled “Profiler”class Profiler( directory: str | os.PathLike[str] | None = None, *, options: jax.profiler.ProfileOptions | None = None, reports: bool = True,)Captures a JAX profile of this process, as a context manager or through
start and stop.
One profiler can capture more than once. Each start creates a new
capture directory under directory. Without a directory, the first start
creates a temporary one, which is kept after the run. A capture covers
only this process, so other processes must start their own. Starting and
stopping wait for the work on JAX’s default backend to finish, without
copying array values to the host. Only one capture can run in a process
at a time, and start raises RuntimeError while another is running.
A capture includes the HLO of every executable alive in the process, not
only of the work it traced, and stop exports all of it. The op_profile,
hlo_stats and roofline_model reports are built from that HLO and come out
nearly empty without it. In a training run’s window this costs little.
A long-lived process that has compiled many other programs pays for each
of them, so there pass options with enable_hlo_proto off when the
trace and kernel reports are enough.
On a GPU the reports also come out empty for work that already ran before
the capture, as a Trainer’s profile window does: XLA runs a step’s
kernels as CUDA command buffers, and their device time is attributed to
their module, not their ops (on JAX 0.11.2 an RTX 4080’s warmed step
named no op; 63 when its first run fell inside the capture; 166 with
command buffers off). XLA means to run command buffers op by op while a
profiling session is active (command_buffer_thunk.cc, ExecuteOnStream),
and here does not; why is open, and tools/gpu_profile_check.py records
whether it does on every GPU session. The manifest’s op_attribution
then says so and a warning names the fix, profiling with
XLA_FLAGS=--xla_gpu_enable_command_buffer=; time runs without that
flag, since launching kernels one by one costs time. Where hlo_stats has
no column naming the ops, op_attribution records that it could not tell.
A capture whose reader reads its trace alone (dew.telemetry.trace), as a
benchmark’s device timeline or XLA’s profile-guided latency estimates do,
takes reports=False: it needs no XProf, and its stop writes its manifest
alone, not the reports, whose conversion takes minutes for a training
window’s capture (row 2017).
captures: tuple[Path, ...]-
The capture directories this profiler’s stops exported, in order: each holds its manifest and its
reports/(report_table,program_kernels). Where an export failed (the manifest’scapture_errors,export_status) reports/ may lack a report.
Profiler.region
Section titled “Profiler.region”def region(name: str) -> AbstractContextManager[None]Return a context manager that marks a named span in the trace while this profiler is capturing.
Wrap the work whose device and host time should show under name.
Outside a capture the span does nothing, so the same code runs
unprofiled without a branch at every call site.
Profiler.start
Section titled “Profiler.start”def start() -> SelfProfiler.stop
Section titled “Profiler.stop”def stop(*, wait: bool = True) -> NoneStop the capture and export its reports and manifest: at once, or where not wait, in a process
of its own beside the caller’s work (dew.telemetry.export), the capture already stopped, which
exported waits for. A training run’s window stops so, since converting a capture takes far longer
than the steps it traced (row 2017: minutes for a 3-step window). A stop that failed exports at
once, so its manifest records why, as it raises.
region
Section titled “region”def region(name: str) -> AbstractContextManager[None]Return a context manager that marks a named span in the running capture, or does nothing when no capture is running.
The call site does not need to know whether a profiler is running or
which one; with region("input.wait"): reads the same either way.