dew.nn.autoencoders
Autoencoders a latent diffusion model reads and writes its samples through: the interface (AutoEncoder)
and the KL and Stable Diffusion VAEs. Each name is imported from its own module when first read, so a reader
of the interface alone reads neither VAE.
| Name | Summary |
|---|---|
AutoEncoder | Encodes images or video to the latents a latent diffusion model trains on, and decodes them back. |
AutoencoderKL | Encodes NHWC images to latents with a convolutional VAE, and decodes latents back to images. |
LatentAffine | The affine map an autoencoder’s latents are normalized by, as the arrays a trace holds (AutoEncoder.latent_affine): (z - shift) * factor and back as z / factor + shift, or, one that divides, (z - shift) / factor and back as z * factor + shift. |
LatentNormalization | A latent normalization, one description: the source values an autoencoder’s record keeps (record, as given) beside the shift and factor its autoencoder derived from them, written together once and read-only after. |
StableDiffusionVAE | Encodes and decodes images with a frozen native AutoencoderKL and its latent normalization. |
AutoEncoder
Section titled “AutoEncoder”class AutoEncoder(ABC)Encodes images or video to the latents a latent diffusion model trains on, and decodes them back.
A subclass encodes and decodes one batch of frames, [B, H, W, C] to
[B, h, w, c] and back. encode and decode here take video
[B, T, H, W, C] frame by frame, unless the subclass encodes time itself
(encode_video, decode_video, latent_shape), and they apply the
latent normalization (normalization, LatentNormalization). On the way
out, latents are normalized as (z - latent_shift) * latent_scale, and on
the way in the normalization is inverted, the SD3 convention.
latent_shift and latent_scale are each one number or one per latent
channel, read from normalization; one that divides by its factor (Qwen-
Image’s) rounds as a division does. The default is the identity; an
autoencoder normalized by the dataset’s own latent mean and 1/std, so the
diffusion model sees roughly unit-variance, zero-mean inputs, is
with_normalization’s.
The weights are an argument, as a ConditionEncoder’s are. params holds
what a run loaded, and every call takes the tree to use, so the layout
places the weights and the checkpoint stores them.
downscale_factor: int-
H / h, the spatial factor between a frame and its latent.
stochastic: bool-
Whether encoding samples its latent, so a training step draws each example’s
noise(encode_batch). A deterministic encoder (DC-AE, RAE) is False and refuses noise, raising TypeError naming itself. latent_channels: int-
c, the number of channels in a latent.
AutoEncoder.towers
Section titled “AutoEncoder.towers”def towers() -> tuple[nn.Module, ...]The networks this autoencoder runs, whose quantization the programs that call it compile for
(dew.nn.kernels.options.compiler_options): its model, none for one that computes without a
network (a pooling one).
AutoEncoder.record_arguments
Section titled “AutoEncoder.record_arguments”def record_arguments() -> dictThe subclass’s constructor arguments besides model and params, as it was given them (a
built-in’s compute dtype and latent normalization), which its record rebuilds it with
(dew.registry.to_record). A subclass a run records states them.
AutoEncoder.with_normalization
Section titled “AutoEncoder.with_normalization”def with_normalization(normalization: LatentNormalization) -> SelfThis autoencoder normalizing its latents by normalization instead: a copy, this one
unchanged.
AutoEncoder.recorded_fields
Section titled “AutoEncoder.recorded_fields”def recorded_fields() -> dictThe record a run keeps of this autoencoder, without its parameters (dew.registry.Recorded):
its module, whose record names its class, and record_arguments, an array as a list; bound
passes them back to its constructor with params.
AutoEncoder.bound
Section titled “AutoEncoder.bound”def bound(record, *, params: Variables) -> AutoEncoderRebuild the autoencoder a run recorded (dew.registry.to_record), around the checkpoint’s own
params, reading no source: its class found as every component’s is (dew.registry.component),
of Dew’s or of a package the run trusts, and its fields read against its constructor.
AutoEncoder.encode_batch
Section titled “AutoEncoder.encode_batch”def encode_batch( params, x: jnp.ndarray, key: jax.Array | None = None, noise: jnp.ndarray | None = None,) -> jnp.ndarrayEncode frames [B, H, W, C] to raw latents [B, h, w, c].
key draws a stochastic encoder’s sample, and None takes its mean.
noise, a standard normal draw of the latents’ shape, is the sample’s
in place of one from key (posterior_latent); a deterministic
encoder reads neither.
AutoEncoder.decode_batch
Section titled “AutoEncoder.decode_batch”def decode_batch(params, z: jnp.ndarray) -> jnp.ndarrayDecode raw latents [B, h, w, c] to frames [B, H, W, C].
AutoEncoder.moments
Section titled “AutoEncoder.moments”def moments(params, images: jnp.ndarray) -> jnp.ndarrayReturn the KL posterior of frames [B, H, W, C], its mean and log-variance stacked on channels.
A KL autoencoder’s latent is one draw from it normalized by
latent_shift and latent_scale alone, encode being
(posterior_latent(moments, key) - latent_shift) * latent_scale, so a
loss may train it through the posterior and renormalize the draw
(REPA-E). Any other autoencoder raises TypeError, naming itself.
AutoEncoder.decode_raw
Section titled “AutoEncoder.decode_raw”def decode_raw(params, latents: jnp.ndarray) -> jnp.ndarrayDecode a draw from moments’ posterior to frames; raises as moments does.
AutoEncoder.latent_shape
Section titled “AutoEncoder.latent_shape”def latent_shape(shape: tuple[int, ...]) -> tuple[int, ...]Return the latent shape for one example of shape.
shape is (H, W, C) or, for video, (T, H, W, C).
AutoEncoder.encode_video
Section titled “AutoEncoder.encode_video”def encode_video( params, x: jnp.ndarray, key: jax.Array | None = None, noise: jnp.ndarray | None = None,) -> jnp.ndarrayEncode video [B, T, H, W, C] to raw latents, frame by frame.
AutoEncoder.decode_video
Section titled “AutoEncoder.decode_video”def decode_video(params, z: jnp.ndarray) -> jnp.ndarrayDecode raw latents [B, t, h, w, c] to video, frame by frame.
AutoEncoder.latent_affine
Section titled “AutoEncoder.latent_affine”def latent_affine(params, dtype) -> LatentAffineThe map encode normalizes latents by and decode restores them by, in dtype, for the
weights params: this autoencoder’s configured normalization (normalization), whatever the
weights. A subclass whose normalization is data the variables hold reads it from params.
AutoEncoder.encode
Section titled “AutoEncoder.encode”def encode( params, x: jnp.ndarray, key: jax.Array | None = None, noise: jnp.ndarray | None = None,) -> jnp.ndarrayEncode images [B, H, W, C] or video [B, T, H, W, C] to normalized latents, a stochastic
encoder’s sample drawn from key or read off noise (encode_batch), by latent_affine.
AutoEncoder.decode
Section titled “AutoEncoder.decode”def decode(params, z: jnp.ndarray) -> jnp.ndarrayDecode normalized latents [B, h, w, c] or [B, t, h, w, c] back to images or video, by
latent_affine.
AutoencoderKL
Section titled “AutoencoderKL”class AutoencoderKL( channels: tuple[int, ...] = (128, 256, 512, 512), latent_channels: int = 4, image_channels: int = 3, blocks_per_level: int = 2, norm_groups: int = 32, quantize: bool = True, post_quantize: bool = True, decoder_channels: tuple[int, ...] | None = None, dtype: Dtype = jnp.float32,)Encodes NHWC images to latents with a convolutional VAE, and decodes latents back to images.
encode(image, key=None) returns the posterior mean, and passing a key
samples its diagonal Gaussian. decode(latents) returns unnormalized
image pixels. The latent scale and shift belong to AutoEncoder.
decoder_channels: tuple[int, ...] | None-
The decoder’s widths where they differ from the encoder’s, as a distilled decoder’s do; None uses
channels. downscale_factor: int-
The spatial factor between an image and its latent.
Every encoder level except the last halves each spatial axis, so the factor is 2 ** (len(channels) - 1).
AutoencoderKL.from_diffusers
Section titled “AutoencoderKL.from_diffusers”def from_diffusers(config: Mapping[str, object], dtype: Dtype) -> AutoencoderKLThe model a diffusers AutoencoderKL config describes, computing in dtype. The stack
computes SiLU, attention in the bottleneck, a decoder writing the image’s channels and the
plain encoder and decoder levels; a config asking for anything else is refused by name.
AutoencoderKL.setup
Section titled “AutoencoderKL.setup”def setup()AutoencoderKL.moments
Section titled “AutoencoderKL.moments”def moments(image)Return the posterior’s mean and log-variance, stacked on the channel axis.
AutoencoderKL.encode
Section titled “AutoencoderKL.encode”def encode(image, key=None, noise=None)AutoencoderKL.decode
Section titled “AutoencoderKL.decode”def decode(latents)LatentAffine
Section titled “LatentAffine”class LatentAffine(divides: bool = struct.field(pytree_node=False, default=False))The affine map an autoencoder’s latents are normalized by, as the arrays a trace holds
(AutoEncoder.latent_affine): (z - shift) * factor and back as z / factor + shift, or, one that
divides, (z - shift) / factor and back as z * factor + shift. A configured normalization gives
its host values (LatentNormalization.affine); a tuned run, the running statistics each call’s
variables hold. divides is static, so the map crosses a jit boundary as its two arrays.
LatentAffine.normalized
Section titled “LatentAffine.normalized”def normalized(latent: jnp.ndarray) -> jnp.ndarrayLatentAffine.restored
Section titled “LatentAffine.restored”def restored(z: jnp.ndarray) -> jnp.ndarrayLatentNormalization
Section titled “LatentNormalization”class LatentNormalization( record: Mapping[str, RecordValue], shift: float | np.ndarray, factor: float | np.ndarray, divides: bool = False,)A latent normalization, one description: the source values an autoencoder’s record keeps
(record, as given) beside the shift and factor its autoencoder derived from them, written
together once and read-only after. Latents are normalized as (z - shift) * factor and restored
as z / factor + shift; one that divides normalizes as (z - shift) / factor and restores as
z * factor + shift, as its source rounds.
scale: float | np.ndarray-
The factor a normalized latent is the shifted one times.
LatentNormalization.affine
Section titled “LatentNormalization.affine”def affine(dtype) -> LatentAffineThe map this normalization applies, its shift and factor in dtype.
StableDiffusionVAE
Section titled “StableDiffusionVAE”class StableDiffusionVAE( modelname='CompVis/stable-diffusion-v1-4', revision='bf16', dtype=jnp.bfloat16, latent_shift=None, latent_scale=None, params=None, model: AutoencoderKL | None = None,)Encodes and decodes images with a frozen native AutoencoderKL and its latent normalization.
StableDiffusionVAE.record_arguments
Section titled “StableDiffusionVAE.record_arguments”def record_arguments() -> dictStableDiffusionVAE.moments
Section titled “StableDiffusionVAE.moments”def moments(params, images)Return AutoencoderKL.moments of the frames, which encode_batch draws its latent from.
StableDiffusionVAE.decode_raw
Section titled “StableDiffusionVAE.decode_raw”def decode_raw(params, latents)Decode a posterior draw as decode_batch does, traced in the caller’s step, not jitted apart.