Skip to content

dew.training.runtime

Process setup that every recipe runs before it builds anything.

Every recipe needs the same setup: the file descriptor limit, the XLA flags, the compilation cache, the JAX distributed pool, and the environment variables that wandb and the tokenizers read. So the library does it in prepare_process, and each recipe calls that once at the top of main().

NameSummary
PreemptedRaised by Trainer.fit when it stops at a preemption notice.
PreemptionNoticeReports once a step whether a preemption notice has reached the run, from its creation until close.
prepare_processSet the environment variables and XLA flags, raise the soft file descriptor limit, and join the JAX process pool.
run_timestampReturn process 0’s wall clock as %Y-%m-%d_%H:%M:%S, on every process.
unpreparedWhich of prepare_process’s options a run states that this process does not carry: a flag of xla_flags XLA_FLAGS did not carry while the backend was closed (flags_set), a compilation cache JAX keeps elsewhere, a pool it has not joined (multi_host=True; False only says never to join one).

class source

class Preempted(step: int)

Raised by Trainer.fit when it stops at a preemption notice.

step is the step it stopped at, and fit has written that step’s checkpoint and data position before raising. Uncaught, it ends the program with exit status PREEMPTED_EXIT and no traceback, as SIGTERM itself would have. Run the same program again to resume from the checkpoint.

class source

class PreemptionNotice()

Reports once a step whether a preemption notice has reached the run, from its creation until close.

A scheduler stops a job by sending SIGTERM and then SIGKILL after a grace period, such as Slurm’s KillWait, Kubernetes’ termination grace period or a spot VM’s notice. In a process pool, JAX’s preemption service receives the SIGTERM; XLA’s notifier replaces the signal handler, so the process keeps running. The service shares the notice through the coordination service, and reached_preemption_sync_point picks one step that every process agrees on, so the checkpoint written there is complete. A process outside any pool has no such service, so the notice is SIGTERM itself, which this object catches until close. Only the main thread can set a signal handler, so a notice created on another thread outside a pool leaves SIGTERM’s default effect. A pool with the preemption service turned off (jax_enable_preemption_service) gets no notice, and SIGTERM ends it as usual.

def close() -> None

Put back the SIGTERM handler this notice replaced.

def reached() -> bool

Return whether to stop here, at this check.

In a pool, that is whether every process agreed to stop at this check, the sync point counting each check every process makes in step (each step’s, and each step of a cooldown branch), so it grows across a run however its steps run; a pool without the preemption service never stops. A process outside a pool stops once SIGTERM has arrived.

function source

def prepare_process(
wandb: Wandb | None = None,
multi_host: bool | None = None,
xla_flags: str | None = None,
compilation_cache_dir: str | None = None,
) -> None

Set the environment variables and XLA flags, raise the soft file descriptor limit, and join the JAX process pool.

wandb is the run’s dew.training.tracker.Wandb, or None for a run without a tracker. Only its offline switch is read, and it has to be read before wandb opens a run. multi_host=True requires a pool, False never joins one, and None joins the pool the environment describes, if any. compilation_cache_dir names the directory of JAX’s persistent compilation cache, which a pool across hosts must keep on storage they share; without one the process caches under the user’s cache directory where its pool shares it (dew.cache.persist_compilations), and JAX_ENABLE_COMPILATION_CACHE=false turns the cache off.

xla_flags reaches XLA through XLA_FLAGS, which XLA parses once per runtime it loads, at the first use of its debug options. Flags set here before the backend opens are written into XLA_FLAGS, as XLA may not have parsed them yet; where import dew has loaded jaxlib, it may have, and Dew warns that they are unproven (dew.devices.flags_hold). Setting XLA_FLAGS before the process starts makes them certain, as Dew’s command line does for a run’s trainer.xla_flags. Flags asked for after the backend opened are refused. A state kept on the CPU (Layout(host=("variables",))) needs one CPU device per local accelerator, which the trainer checks on its own mesh (dew.distributed.host.companion_mesh).

function source

def run_timestamp() -> str

Return process 0’s wall clock as %Y-%m-%d_%H:%M:%S, on every process.

The default run name includes it, and that name is the checkpoint directory every process writes into. So a process that read its own clock a second later would write into a different directory.

function source

def unprepared(
multi_host: bool | None,
xla_flags: str | None,
compilation_cache_dir: str | None,
) -> list[str]

Which of prepare_process’s options a run states that this process does not carry: a flag of xla_flags XLA_FLAGS did not carry while the backend was closed (flags_set), a compilation cache JAX keeps elsewhere, a pool it has not joined (multi_host=True; False only says never to join one). Each acts by the time JAX opens its backend, and prepare_process sets none after (_set_environment), so a run trained in a process no prepare_process set up with them would run without them.