Skip to content

dew.nn.diffusion_gemma

DiffusionGemma’s shared encoder and decoder, and its self-conditioning MLP.

The MLP follows the Transformers implementation: a scaled pre-norm, a gated feed-forward and a post-norm without scale. The previous step’s logits become soft embeddings through an fp32 softmax against the scaled embedding table. On the first inference step, an explicit self-conditioning mask zeros those embeddings. The official SFT objective supplies zero logits for its dropout branch instead, and zero logits give uniform soft embeddings, which are not a zero signal.

NameSummary
DiffusionGemmaReads one text parameter tree causally for the context and bidirectionally for canvases.
SelfConditioningFolds the previous step’s soft embeddings into the canvas embeddings: a GeGLU feed-forward (GatedMLP, its gate, up and down projections at this module’s own scope) of the normed signal, added to the embeddings and normed again.

class source

class DiffusionGemma(DecoderWrapper)

Reads one text parameter tree causally for the context and bidirectionally for canvases.

encode appends clean tokens to the cache. __call__ refines a canvas against that frozen cache and feeds the previous logits through self-conditioning. hidden_states and logits read clean tokens through the same causal encoder over the whole sequence, with no cache. Each method is a separate apply. The encoder and the decoder share one scope, so their parameters are identical and no second tree is stored. The training and cache hooks are its text model’s (DecoderWrapper), which the decoder clones: the decoder shares text’s scope, another reader, not another bank, and causal is the encoder’s, which setup holds causal.

def setup()

DiffusionGemma.with_trainable_layer_scalars

Section titled “DiffusionGemma.with_trainable_layer_scalars”
def with_trainable_layer_scalars() -> DiffusionGemma

Return this model with its layer scalars as parameters, the model the published SFT trains.

A source whose scalars are constants (“frozen”) is read by it through trainable_variables. A model without layer scalars is refused.

def trainable_variables(variables: VariableDict) -> VariableDict

Return this model’s variables as with_trainable_layer_scalars() reads them.

Google’s release makes each layer’s scalar a parameter, and Transformers declares the same tensor a buffer, which a “frozen” model keeps under constants. Each moves to its layer under params, constants it leaves empty are dropped, and no array is copied. A “trainable” model’s tree is returned as it is. A frozen model’s tree that already holds a trainable scalar is refused.

def init_cache(batch_size: int)
def encode(
tokens,
*,
coordinates: Coordinates | None = None,
image_indices=None,
image_groups=None,
block_groups=None,
rotary_positions=None,
attention_pairwise_mask=None,
attention_key_positions=None,
conditioning: Mapping[str, jax.Array] | None = None,
train: bool = False,
states: bool = False,
progress: Progress | None = None,
)

Append a clean prompt or committed canvas to the cache, evaluating media only when given.

It returns the logits, or with states the final normalized states before the head. A loss that scores the vocabulary a tile at a time reads the states, so the vocabulary-sized logits of a whole row never exist at once. The tokens are laid out by their coordinates, at a training step’s progress (dew.nn.progress; None at the run’s end).

def hidden_states(
tokens,
*,
train: bool = False,
image_indices=None,
conditioning: Mapping[str, jax.Array] | None = None,
**fields={},
)

Return the causal encoder’s final normalized states over clean tokens, writing no cache.

The encoder reads the text tree and fuses media as encode does, but over the whole sequence at once and without the cache encode fills for a canvas to attend to: the read a loss or a probe takes of clean text. fields go to CausalTransformer.hidden_states.

def logits(tokens, *, train: bool = False, **fields={})

Return the causal encoder’s fp32 logits over clean tokens, writing no cache (hidden_states).

def logits_from_hidden(hidden)

Return the logits of final states hidden, through the head the encoder and the decoder share.

def output_table() -> OutputTable | None

Return the shared head as the matrix final states contract, or None where none does (CausalTransformer.output_table).

def vocabulary_rows() -> jax.Array

The shared head’s rows (CausalTransformer.vocabulary_rows).

Flax module source

class SelfConditioning(
norm_eps: float = 1e-06,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
)

Folds the previous step’s soft embeddings into the canvas embeddings: a GeGLU feed-forward (GatedMLP, its gate, up and down projections at this module’s own scope) of the normed signal, added to the embeddings and normed again.

def setup()