dew.inference.banks
Build and place parameter banks for generation, one bank at a time.
A scanned run of identical layers reads each parameter as one array with
the layer axis in front; that stacked array is a bank, and one layer’s
slice of it is a row. (dew.nn.backbones.decoder_stack.StackView
adapts banks to the layers_N subtrees a checkpoint stores.) This module
builds the banked store directly. It reads, stacks and places each bank
separately and releases that bank’s rows before reading the next, so a
stack that does not fit in device memory is never held as separate layers
and as banks at the same time.
Layout.host_parameters puts banks in pinned host memory, and the stack
fetches them one layer at a time (run_stack). Everything else is placed
the way the trainer places a parameter. A LayerBanks source reads from a
run directory (CheckpointBanks), a tree in memory (HeldBanks) or local
HF safetensors (SafetensorsBanks). A source’s place puts whole banks on
the mesh. SafetensorsBanks.stream leaves the decoder weights on disk
instead, and the prefetch loop reads them during execution.
| Name | Summary |
|---|---|
BankedModel | |
CheckpointBanks | Serves banks by restoring a run’s published weights one bank at a time. |
DiskBankStats | Cumulative host-read statistics of a SafetensorsBanks source, as its stats() returns them. |
HeldBanks | Serves banks from a variables tree already held in memory. |
LayerBanks | Serves a banked store’s values, read one bank at a time. |
SafetensorsBanks | Reads a local HF decoder checkpoint without materializing its layer stack. |
StreamedBank | Fetches one run’s layers from a SafetensorsBanks source during execution. |
BankedModel
Section titled “BankedModel”class BankedModel(Protocol)BankedModel.bank_sites
Section titled “BankedModel.bank_sites”def bank_sites(store: Variables) -> tuple[DecoderBank, ...]CheckpointBanks
Section titled “CheckpointBanks”class CheckpointBanks(directory: str, step: int | None = None, ema: bool = False)Serves banks by restoring a run’s published weights one bank at a time.
With ema, the averaged copy is merged over the live weights, as
Checkpoints.variables does, so a bank holds what the run publishes.
step selects a checkpoint. When it is None, the source resolves it to
the latest checkpoint once, when the source is built, so every bank of
one load comes from the same checkpoint. A directory without a
checkpoint raises FileNotFoundError.
Rows are restored to device memory, where the stacking runs, and the stacked bank goes wherever the store places it. So a load stages one bank’s rows and one bank at a time, never the whole model.
stored: Variables-
The shape and dtype trees the checkpoint at
stepholds, read once when the source is built.
CheckpointBanks.shapes
Section titled “CheckpointBanks.shapes”def shapes() -> VariablesCheckpointBanks.entry
Section titled “CheckpointBanks.entry”def entry(placement: Placement) -> VariablesCheckpointBanks.bank
Section titled “CheckpointBanks.bank”def bank( layers: Sequence[int], placement: Placement, *, namespace: tuple[str, ...] = (),) -> VariablesDiskBankStats
Section titled “DiskBankStats”class DiskBankStats( cache_bytes: int, hits: int, misses: int, bytes_read: int, read_seconds: float, wait_seconds: float,)Cumulative host-read statistics of a SafetensorsBanks source, as its stats() returns them.
cache_bytes is the size of the layers cached now. hits and misses
count layer reads served from the cache and from disk, and bytes_read
is the bytes read on misses. read_seconds is the time spent reading,
which includes layout conversion and can overlap compute. wait_seconds
is the time callers of read waited for a row.
HeldBanks
Section titled “HeldBanks”class HeldBanks(variables: Variables)Serves banks from a variables tree already held in memory.
variables is the tree that model.init or a loader produced, with one
subtree per layer. The source borrows its arrays and never donates or
deletes them, so the caller’s tree stays usable. A load therefore holds
the source and the growing banked store at once, which costs two copies
of the model. Use CheckpointBanks for weights that do not fit twice.
HeldBanks.shapes
Section titled “HeldBanks.shapes”def shapes() -> VariablesHeldBanks.entry
Section titled “HeldBanks.entry”def entry(placement: Placement) -> VariablesHeldBanks.bank
Section titled “HeldBanks.bank”def bank( layers: Sequence[int], placement: Placement, *, namespace: tuple[str, ...] = (),) -> VariablesLayerBanks
Section titled “LayerBanks”class LayerBanks(Protocol)Serves a banked store’s values, read one bank at a time.
Paths are canonical below each collection, and a namespace selects a
decoder inside a wrapper model. The memory a source needs to hold one
bank depends on the source, and it bounds what a load costs. A source
subclasses this class to inherit place, which builds a model’s banked
store from the source.
LayerBanks.shapes
Section titled “LayerBanks.shapes”def shapes() -> VariablesReturn the whole store as jax.ShapeDtypeStruct leaves, one per layer.
LayerBanks.entry
Section titled “LayerBanks.entry”def entry(placement: Placement) -> VariablesRead the canonical leaves that placement selects, and only those.
LayerBanks.bank
Section titled “LayerBanks.bank”def bank( layers: Sequence[int], placement: Placement, *, namespace: tuple[str, ...] = (),) -> VariablesReturn the bank of one run of layers, on the shardings or Formats in placement.
The layers are stacked in order on a new leading axis; a run of one
layer returns that layer’s own subtree. A Format also sets a physical
layout, which the source must preserve. The returned collections are
local to namespace.
LayerBanks.place
Section titled “LayerBanks.place”def place( model: BankedModel, *, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None,) -> VariablesBuild, from these weights, the banked store that model’s runs read.
It waits for each bank’s copies to finish before reading the next bank, so a load has one bank’s transfers in flight and costs the store plus what this source holds. It does not donate or delete any of the source’s arrays.
layout.host_parameters may name only layers of the stack, because
the stack fetches each layer when it reaches it. An embedding or head
in host memory would be copied over whole and cost the device memory
it was meant to save. A run whose layers the layout’s patterns place
differently is refused as well, because a bank is one array with one
sharding. Both checks raise ValueError before anything is read.
SafetensorsBanks
Section titled “SafetensorsBanks”class SafetensorsBanks( directory: str | Path, *, cache_bytes: int = 0, param_dtype: DTypeLike | Literal['auto'] = 'auto', read_ahead: bool = True,)Reads a local HF decoder checkpoint without materializing its layer stack.
The checkpoint is a decoder, a media wrapper around one, or the native
backbone a release with a separately stored readout describes (a slot
decision release). It translates the weights as Pretrained.load does
(dew.interop.pretrained.native_backbone), with its SourceLeaf recipes
over read-only memory maps, including expert stacking, transposition and
the cast to param_dtype (by default the checkpoint’s dtype). Quantized
checkpoints are refused, because their codec may dequantize the entire
checkpoint before translation, which would break the memory bound below.
The cache keeps complete layers in read order until cache_bytes is
full, and holds them until close. A sequential decoder visits every
layer for each token, so an LRU cache would evict every layer once the
model is larger than the cache; keeping a fixed prefix of layers makes a
partial cache useful. One read-ahead slot holds the next layer, even
with a zero cache, unless read_ahead is False. Host memory for weights
is bounded by cache_bytes, two rows and one leaf’s conversion scratch.
Embeddings, heads, a media wrapper’s towers and projectors (read whole,
as the wrapper’s reader reads them) and the KV cache stay resident
outside that bound, and the bound does not count the kernel’s page cache.
Keep the source open and its files unchanged until all executions finish, because a compiled callback cannot read a closed source.
SafetensorsBanks.close
Section titled “SafetensorsBanks.close”def close() -> NoneSafetensorsBanks.stats
Section titled “SafetensorsBanks.stats”def stats() -> DiskBankStatsSafetensorsBanks.shapes
Section titled “SafetensorsBanks.shapes”def shapes() -> VariablesSafetensorsBanks.layer_bytes
Section titled “SafetensorsBanks.layer_bytes”def layer_bytes(index: int, *, namespace: tuple[str, ...] = ()) -> intSafetensorsBanks.entry
Section titled “SafetensorsBanks.entry”def entry(placement: Placement) -> VariablesSafetensorsBanks.bank
Section titled “SafetensorsBanks.bank”def bank( layers: Sequence[int], placement: Placement, *, namespace: tuple[str, ...] = (),) -> VariablesSafetensorsBanks.stream
Section titled “SafetensorsBanks.stream”def stream( model: BankedModel, *, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None,) -> VariablesReturn variables whose decoder layers the banked inference loop reads from disk, on one device.
Only the non-decoder leaves are loaded now. Each declared run gets a
static StreamedBank handle in the runtime-only streaming
collection, and the stack’s prefetch, which holds two rows at a time,
calls it during execution.
This works for single-device inference only. A mesh of more than one
device, host placement in layout, or a stack without scan_layers
raises ValueError before any weight is loaded. The host callback
needs the CPU backend beside the accelerator, so set
JAX_PLATFORMS=cuda,cpu if you set platforms explicitly. Keep this
source open until executions complete.
SafetensorsBanks.read
Section titled “SafetensorsBanks.read”def read( index: int, *, namespace: tuple[str, ...] = (), following: int | None = None,) -> VariablesRead the row of layer index and start reading the following layer before returning.
The read-ahead happens only when following is given and the
source’s read_ahead is on. Reads from executions that share this
source run one at a time. A pending read-ahead finishes before a
different request is served, so concurrent calls cannot build up an
unbounded queue of rows.
StreamedBank
Section titled “StreamedBank”class StreamedBank( source: SafetensorsBanks, first: int, count: int, namespace: tuple[str, ...], device: SingleDeviceSharding, following: int | None,)Fetches one run’s layers from a SafetensorsBanks source during execution.
It is a runtime-only handle, with no parameter buffer and no serialized model field. The handle is a static pytree node compared by identity, so changing the source retraces and changing the tokens does not. The callback has side effects, so JAX neither folds it at trace time nor skips it when an executable is reused.
StreamedBank.shapes
Section titled “StreamedBank.shapes”def shapes() -> VariablesStreamedBank.fetch
Section titled “StreamedBank.fetch”def fetch(index, dependency) -> Variables