Skip to content

dew.data.images

Image record steps: decoding, filters, resizing, crops, captions and augmentation, on the host or on the accelerator.

NameSummary
import_opencvImport OpenCV in the thread that opens a loader, before its readers start.
resize_imageimage at size, a square side or (height, width); area interpolation where it shrinks along either side, cubic where it only grows.
fetchA record step (filter_map) that fetches the image at the record’s url field into to as bytes, in one attempt of at most timeout once its host is resolved (Fetch); a failed fetch is counted and dropped, a transient one fetched again on a later pass.
DecodeImageThe image in field as RGB uint8 pixels under to, including prepared ArrayRecord pixels, labels and captions; bytes that are no image are counted and dropped.
MinSideWhether the pixels in field, [..., H, W, C], are at least size on their shorter side.
MaxAspectWhether the pixels in field are at most ratio times as long as wide.
NotFlatWhether the pixels in field hold more than one value.
Interpolation
resized_toimage at height x width by OpenCV’s method interpolation; where None, area where it shrinks along either side and bicubic where it only grows.
ResizeThe pixels in field resized to size, a square side or (height, width), by OpenCV’s method interpolation, or by area down and bicubic up where None.
FitThe pixels in field fitted into a size square whole: the longer side resized to size (area down, bicubic up), the shorter in proportion, rounded, and the rest padded on both sides with fill, the odd pixel at the bottom or right.
CenterCropThe central size square of the pixels in field; a smaller image is first resized up so its shorter side is size.
RandomCropA crop of scale of the area resized to size, optionally flipped and colour-jittered.
BucketThe pixels in field resized to cover the (height, width) of sizes nearest their aspect ratio and cut to it at the centre, with that size’s index under to, so records of one bucket stack.
bucketA record step that resizes the pixels in field to cover the (height, width) of sizes nearest their aspect ratio and cuts that size from the centre, writing its index under to; a batch grouped by to stacks images of one shape.
RandomResizedCroptorchvision’s RandomResizedCrop: a crop of scale of the area and ratio of aspect, drawn from the record’s generator, resized to a size square; after ten draws that do not fit, the central crop of the nearest aspect.
RandomMaskAn inpainting mask for each image in field, drawn as Diffusers’ inpainting trainer draws one (examples/research_projects/dreambooth_inpaint/train_dreambooth_inpaint.py’s random_mask at 86ef420b1, its default ratio=1): a size drawn uniformly up to ratio of each side, a centre where half of it stays within the image, and by a coin a rectangle or an ellipse there, drawn by PIL as that script draws it.
FlipThe pixels in field mirrored left to right with probability p.
ColorJittertorchvision’s float ColorJitter of the pixels in field: brightness, contrast and saturation factors drawn from [1 - x, 1 + x], applied in a random order, rounded once.
ImageProcessorA Hugging Face image processor, as Processor calls it.
AugmentationAn albumentations pipeline (Compose), as Albumentations seeds and calls it.
ProcessorA Hugging Face image processor’s pixel_values for the image in field, under to.
AlbumentationsAn albumentations pipeline on the image in field, seeded from the record’s generator through Compose.set_random_seed.
TorchvisionA torchvision v2 transform on the image in field (HWC uint8 in and out), seeded from the record’s generator.
CAPTION_COLUMNSThe fields a record’s own caption is read from, the first it holds (Caption).
CaptionA record’s caption and int32 numeric labels, from columns or a class template.
RandomCaptionA caption template drawn from the record’s generator.
captionA record step that writes “caption”: template formatted with the record’s fields (“{label}” is the class name classes gives its label), or with none, the first of columns.
ConfigureCaption
AugmentationKeyThe key DeviceAugment augments the record with, drawn from the record’s generator.
CAPTIONSThe field a diffusion objective keeps each batch’s captions under as host text (EncodeCaptions), which a metric or a reward reads with its own tokenizer (dew.eval.images.CLIPScore).
EncodeCaptionsEach batch’s captions encoded by reader for the conditions that read them, or dropped without one; keep names a field to hold them beside as host text, which a metric reads against its own tokenizer (dew.eval.images.CLIPScore) and no compiled step takes (split_fields).
EncodeFieldsEach batch with the fields reader encodes from it (InputSpec.encode_fields), as EncodeCaptions does its captions.
DeviceAugmentCrop, flip and jitter each image of the training batch inside the compiled step, from its record’s own key (AugmentationKey).

function source

def import_opencv() -> None

Import OpenCV in the thread that opens a loader, before its readers start.

The reader threads reach their first decode together, so each would otherwise make the first import of cv2 at once. An import that fails in one of them leaves the others the half-built module, which surfaces as module 'cv2' has no attribute 'INTER_AREA' instead of the failure itself. Imported here, a broken install raises its own error when the loader opens. The import stays out of the module’s top so that reading text never loads OpenCV.

function source

def resize_image(image: np.ndarray, size: int | tuple[int, int]) -> np.ndarray

image at size, a square side or (height, width); area interpolation where it shrinks along either side, cubic where it only grows.

function source

def fetch(
url: str = 'url',
to: str = 'jpg',
*,
timeout: float = 10.0,
max_bytes: int = 64 * 2 ** 20,
threads: int | None = None,
) -> Fetch

A record step (filter_map) that fetches the image at the record’s url field into to as bytes, in one attempt of at most timeout once its host is resolved (Fetch); a failed fetch is counted and dropped, a transient one fetched again on a later pass. threads fetches that many at once on the stage’s own pool, in each loading worker, delivered in the order read; without it an indexed read runs Loading.threads fetches at once in each worker. The bytes are volatile: a published cache after the fetch keeps them for exact resume.

dataclass source

class DecodeImage(
field: str = 'jpg',
to: str = 'image',
*,
at_least: int | None = None,
rejects: bool = True,
)

The image in field as RGB uint8 pixels under to, including prepared ArrayRecord pixels, labels and captions; bytes that are no image are counted and dropped. With rejects=False, use map and fail on bad bytes. at_least bounds a reduced JPEG decode.

external: bool

An image a record names by path is read from storage (dew.data.pipeline.External).

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

RGB pixels under to, each image its own height and width, in place of the bytes in field.

def map(element: Batch | bytes) -> Batch | Rejected

dataclass source

class MinSide(size: int, field: str = 'image')

Whether the pixels in field, [..., H, W, C], are at least size on their shorter side.

dataclass source

class MaxAspect(ratio: float, field: str = 'image')

Whether the pixels in field are at most ratio times as long as wide.

dataclass source

class NotFlat(field: str = 'image')

Whether the pixels in field hold more than one value.

attribute source

function source

def resized_to(
image: np.ndarray,
height: int,
width: int,
method: Interpolation | None,
) -> np.ndarray

image at height x width by OpenCV’s method interpolation; where None, area where it shrinks along either side and bicubic where it only grows. The one resize every image and video step reads.

dataclass source

class Resize(
size: int | tuple[int, int],
method: Interpolation | None = None,
*,
field: str = 'image',
)

The pixels in field resized to size, a square side or (height, width), by OpenCV’s method interpolation, or by area down and bicubic up where None.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]
def map(element: Batch) -> Batch

dataclass source

class Fit(size: int, fill: int = 255, *, field: str = 'image')

The pixels in field fitted into a size square whole: the longer side resized to size (area down, bicubic up), the shorter in proportion, rounded, and the rest padded on both sides with fill, the odd pixel at the bottom or right. A web image keeps all it shows, as the retired online loader fitted it; Resize stretches it and CenterCrop cuts it instead.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]
def map(element: Batch) -> Batch

dataclass source

class CenterCrop(size: int, method: Interpolation = 'bicubic', *, field: str = 'image')

The central size square of the pixels in field; a smaller image is first resized up so its shorter side is size.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

The images size squares, as many channels as they held.

def map(element: Batch) -> Batch

dataclass source

class RandomCrop(
size: int,
*,
scale: tuple[float, float] = (0.8, 1.0),
flip: bool = False,
jitter: bool = False,
)

A crop of scale of the area resized to size, optionally flipped and colour-jittered.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]
def random_map(element: Batch, rng: np.random.Generator) -> Batch

dataclass source

class Bucket(
sizes: tuple[tuple[int, int], ...],
method: Interpolation = 'area',
field: str = 'image',
to: str = 'bucket',
)

The pixels in field resized to cover the (height, width) of sizes nearest their aspect ratio and cut to it at the centre, with that size’s index under to, so records of one bucket stack.

def map(element: Batch) -> Batch

function source

def bucket(
sizes: Sequence[tuple[int, int]],
method: Interpolation = 'area',
*,
field: str = 'image',
to: str = 'bucket',
) -> Bucket

A record step that resizes the pixels in field to cover the (height, width) of sizes nearest their aspect ratio and cuts that size from the centre, writing its index under to; a batch grouped by to stacks images of one shape.

dataclass source

class RandomResizedCrop(
size: int,
*,
scale: tuple[float, float] = (0.08, 1.0),
ratio: tuple[float, float] = (3 / 4, 4 / 3),
method: Interpolation = 'bilinear',
field: str = 'image',
)

torchvision’s RandomResizedCrop: a crop of scale of the area and ratio of aspect, drawn from the record’s generator, resized to a size square; after ten draws that do not fit, the central crop of the nearest aspect.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]
def random_map(element: Batch, rng: np.random.Generator) -> Batch

dataclass source

class RandomMask(ratio: float = 1.0, *, field: str = 'image', to: str = 'mask')

An inpainting mask for each image in field, drawn as Diffusers’ inpainting trainer draws one (examples/research_projects/dreambooth_inpaint/train_dreambooth_inpaint.py’s random_mask at 86ef420b1, its default ratio=1): a size drawn uniformly up to ratio of each side, a centre where half of it stays within the image, and by a coin a rectangle or an ellipse there, drawn by PIL as that script draws it. It is written to to as [height, width, 1] float32: 1 inside, the region a run paints, and 0 outside, the masked image’s pixels (dew.inputs.diffusion.latent_image_conditions). A diffusion run over an inpainting pipeline reads it (DiffusionRunConfig).

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

The facts with a mask of the images’ height and width beside them, where the facts size them.

def random_map(element: Batch, rng: np.random.Generator) -> Batch

dataclass source

class Flip(p: float = 0.5, *, field: str = 'image')

The pixels in field mirrored left to right with probability p.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

The images as their shape was: a mirror moves pixels, not the frame.

def random_map(element: Batch, rng: np.random.Generator) -> Batch

dataclass source

class ColorJitter(
brightness: float = JITTER.brightness,
contrast: float = JITTER.contrast,
saturation: float = JITTER.saturation,
*,
field: str = 'image',
)

torchvision’s float ColorJitter of the pixels in field: brightness, contrast and saturation factors drawn from [1 - x, 1 + x], applied in a random order, rounded once.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

The images as their shape was, uint8 still: the jitter changes their values alone.

def random_map(element: Batch, rng: np.random.Generator) -> Batch

class source

class ImageProcessor(Protocol)

A Hugging Face image processor, as Processor calls it.

class source

class Augmentation(Protocol)

An albumentations pipeline (Compose), as Albumentations seeds and calls it.

def set_random_seed(seed: int) -> None

dataclass source

class Processor(
processor: ImageProcessor,
*,
field: str = 'image',
to: str = 'pixel_values',
)

A Hugging Face image processor’s pixel_values for the image in field, under to.

def map(element: Batch) -> Batch

dataclass source

class Albumentations(transform: Augmentation, *, field: str = 'image')

An albumentations pipeline on the image in field, seeded from the record’s generator through Compose.set_random_seed. Each reading thread seeds and calls its own copy of the pipeline, so no other thread’s seed, from this stage or another holding the same pipeline, lands between the two, and a record’s draw is the same at any thread count and after a resume.

def random_map(element: Batch, rng: np.random.Generator) -> Batch

dataclass source

class Torchvision(
transform: Callable[[torch.Tensor], torch.Tensor],
*,
field: str = 'image',
)

A torchvision v2 transform on the image in field (HWC uint8 in and out), seeded from the record’s generator. torch draws from one generator a process, so the transform runs one record at a time in each process, the generator seeded inside that turn and restored after it.

def random_map(element: Batch, rng: np.random.Generator) -> Batch

attribute source

CAPTION_COLUMNS = ('caption', 'text')

The fields a record’s own caption is read from, the first it holds (Caption).

dataclass source

class Caption(
template: str | None = None,
classes: tuple[str, ...] = (),
columns: tuple[str, ...] = CAPTION_COLUMNS,
)

A record’s caption and int32 numeric labels, from columns or a class template.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

A caption, a string of no fixed shape, in place of any the record held, beside the fields as they were.

def map(element: Batch) -> Batch

dataclass source

class RandomCaption(
template: str | None = None,
classes: tuple[str, ...] = (),
columns: tuple[str, ...] = CAPTION_COLUMNS,
templates: tuple[str, ...] = (),
)

A caption template drawn from the record’s generator.

def random_map(element: Batch, rng: np.random.Generator) -> Batch

function source

def caption(
template: str | None = None,
*,
templates: tuple[str, ...] = (),
classes: tuple[str, ...] = (),
columns: tuple[str, ...] = CAPTION_COLUMNS,
) -> Caption | RandomCaption

A record step that writes “caption”: template formatted with the record’s fields (“{label}” is the class name classes gives its label), or with none, the first of columns. With templates, a random step instead draws one template per record; {} and {label} both name its class.

dataclass source

class ConfigureCaption(caption: Caption)
def declared(facts: Mapping[str, object]) -> Mapping[str, object]

A caption, a string of no fixed shape, in place of any the record held, beside the rest.

dataclass source

class AugmentationKey()

The key DeviceAugment augments the record with, drawn from the record’s generator.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]
def random_map(element: Batch, rng: np.random.Generator) -> Batch

attribute source

CAPTIONS = 'captions'

The field a diffusion objective keeps each batch’s captions under as host text (EncodeCaptions), which a metric or a reward reads with its own tokenizer (dew.eval.images.CLIPScore).

dataclass source

class EncodeCaptions(
reader: Callable[[Sequence[str]], Batch] | None = None,
keep: str | None = None,
)

Each batch’s captions encoded by reader for the conditions that read them, or dropped without one; keep names a field to hold them beside as host text, which a metric reads against its own tokenizer (dew.eval.images.CLIPScore) and no compiled step takes (split_fields). A reader that encodes a field of that name too is refused (ValueError), rather than one overwriting the other.

dataclass source

class EncodeFields(reader: Callable[[Batch], Batch])

Each batch with the fields reader encodes from it (InputSpec.encode_fields), as EncodeCaptions does its captions.

dataclass source

class DeviceAugment(
size: int,
*,
crop_scale: tuple[float, float] = (1.0, 1.0),
flip: bool = True,
jitter: bool = False,
)

Crop, flip and jitter each image of the training batch inside the compiled step, from its record’s own key (AugmentationKey).

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

Each image cropped to a size square, its key spent.