dew.data.steps.chat
Block-diffusion response rows of rendered conversations, and prompt record encodings.
| Name | Summary |
|---|---|
ResponseCanvases | One block-diffusion row per assistant turn of a rendered conversation. |
prompt_facts | The facts of rendered prompts: max_prompt_len ids and their length, and the reward fields reward_bytes wide where known, in place of the prompt’s columns. |
RenderPrompt | |
ConfigurePrompt |
ResponseCanvases
Section titled “ResponseCanvases”class ResponseCanvases(pad_id: int = 0)One block-diffusion row per assistant turn of a rendered conversation.
The row’s text is everything before the turn, pad_id-padded to
prompt_length, then the turn’s completion over count canvases of
canvas_size: eos_id fills the rest of its last canvas, pad_id the
canvases after it, and canvas_mask marks the filled canvases, as the
reference’s CanvasChunker lays a response out, and prompt_mask the
context’s own ids, so no id the vocabulary also pads with reads as padding (gemma bf0b49901,
diffusion/hackable_diffusion_adapter/data/data.py). A longer prompt or
completion keeps its first ids, as the reference’s Pad(truncate=True)
and CanvasChunker cut them (gm/data/_functional.py:83). Either cut
marks the row truncated.
ResponseCanvases.declared
Section titled “ResponseCanvases.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]A row for each assistant turn: its context and completion canvases as text, the
canvases filled as canvas_mask and the context’s own ids as prompt_mask.
ResponseCanvases.flat_map
Section titled “ResponseCanvases.flat_map”def flat_map(element: Batch) -> Sequence[Batch]prompt_facts
Section titled “prompt_facts”def prompt_facts( facts: Mapping[str, object], tokenizer: str, max_prompt_len: int, reward_bytes: int | None,) -> Mapping[str, object]The facts of rendered prompts: max_prompt_len ids and their length, and the reward fields
reward_bytes wide where known, in place of the prompt’s columns.
RenderPrompt
Section titled “RenderPrompt”class RenderPrompt( max_prompt_len: int = PROMPT_LEN, pad_id: int = PROMPT_PAD, reward_bytes: int = 1024, long: Literal['drop', 'truncate'] = PROMPT_LONG, thinking: bool | None = None,)vocabulary: ByteTokenizer | HFTokenizer-
The tokenizer and its template, resolved once as
Tokenize.vocabulary.
RenderPrompt.dependencies
Section titled “RenderPrompt.dependencies”def dependencies() -> Mapping[str, object]RenderPrompt.declared
Section titled “RenderPrompt.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]RenderPrompt.map
Section titled “RenderPrompt.map”def map(element: Batch) -> Batch | RejectedConfigurePrompt
Section titled “ConfigurePrompt”class ConfigurePrompt()ConfigurePrompt.declared
Section titled “ConfigurePrompt.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]What the step it binds makes: its reward width the facts state, else measured.