Skip to content

Train a language model

So far we have only generated images. But the same Trainer trains language models too, and in this notebook we shall train a small decoder-only transformer on FineWeb-Edu, a large collection of educational web pages, and then let it write some text of its own.

Next-token prediction on 'Plants need light to grow'. At each position the model reads the tokens so far and scores the candidates for the next one; training pushes up the token that actually came next.the model reads the tokens so farPlantsneedlighttogrowneednext?P(next token)needareandcanlightnext?P(next token)lightwatertothetonext?P(next token)toandfor.grownext?P(next token)growlivemakebetraining pushes the true one up

That’s the whole training signal: at every position, make the token that actually came next a little more likely.

The model is built from the same pieces current open models use: RMSNorm (Zhang and Sennrich, 2019), rotary position embeddings (Su et al., 2021), a gated MLP (Shazeer, 2020), and attention that can share its key and value heads across groups of query heads (Ainslie et al., 2023). Our small model keeps one key and value head per query head. Dew’s CausalTransformer builds it, LMObjective holds the next-token loss, and the same Trainer as in the diffusion notebooks runs the training.

Dew from GitHub with its cuda13 and streaming extras. Dew streams the text below from the Hub.

%pip install -q "dewml[cuda13,streaming] @ git+https://github.com/AshishKumar4/dew@71b43f2dc070cf289e1fbaf6177e484215d4ee1a" --extra-index-url https://pypi.dewml.dev/simple
install log
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
  Building wheel for dewml (pyproject.toml) ... done
TOKENIZER = "HuggingFaceTB/SmolLM2-135M"
SEQUENCE_LENGTH = 1024
BATCH_SIZE = 64
STEPS = 8000
STAGES = 5
LEARNING_RATE = 6e-4
EMB_FEATURES = 768
NUM_LAYERS = 12
NUM_HEADS = 12
MAX_NEW_TOKENS = 120
PROMPT = "Plants need light to grow because"
RUN_DIR = "runs/05-lm"
SEED = 0
import json
import jax
import jax.numpy as jnp
import numpy as np
from dew import figures
from dew.training import prepare_process
prepare_process() # Dew's setup for a training process, before the first JAX call
print(jax.devices())
[CudaDevice(id=0)]

A language model predicts the next token from the ones before it. We use the tokenizer of SmolLM2 (Allal et al., 2025), a byte-pair encoding: it splits text into pieces from a vocabulary of 49,152, whole words where they’re common and fragments of rarer ones, and it was trained on text much like ours.

We stream FineWeb-Edu’s 10-billion-token sample (Penedo et al., 2024) straight from the Hub’s Parquet shards, and hand it to Dew’s language-model preset, presets.lm. For validation we read another part of FineWeb-Edu, its CC-MAIN-2025-26 crawl, eight token windows a pass, through the same preset. The preset ends each document with the tokenizer’s end-of-text id, runs the documents on into one stream, and cuts that stream into rows without losing next-token targets (pack=None).

from dew.data import Hub, presets
from dew.data.compile import compile as compile_data
fineweb = Hub("HuggingFaceFW/fineweb-edu", name="sample-10BT")
crawl = Hub("HuggingFaceFW/fineweb-edu", name="CC-MAIN-2025-26")
windows = {"seq_len": SEQUENCE_LENGTH, "batch": BATCH_SIZE, "pack": None}
data = compile_data(fineweb.shuffle(seed=0, window=64).repeat().pipe(presets.lm, TOKENIZER, **windows),
validation=crawl.pipe(presets.lm, TOKENIZER, **windows, records=8),
read="streamed")
stderr
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
WARNING:absl:Warning: Average element size is zero. Defaulting to max threads.
from dew.data import HFTokenizer
tokenizer = HFTokenizer(TOKENIZER)
ids = tokenizer.encode(PROMPT, add_special_tokens=False)
print(ids)
print("vocabulary:", tokenizer.vocab_size)
pieces = [tokenizer.decode([token]) for token in ids]
figures.tokens([pieces], fills=[["tint1"] * len(pieces)])
[34246, 737, 1420, 288, 1075, 975]
vocabulary: 49152
token rowsPlants need light to grow because

Six words, six tokens. Each of these words is common enough to have a token of its own, and every one after the first carries its leading space. A rarer word would be split into pieces.

The preset produces rows of SEQUENCE_LENGTH + 1 token IDs. The first SEQUENCE_LENGTH IDs are the input; shifting by one gives the targets. The documents run on into each other, separated only by the end-of-text token, so attention can reach back across a document boundary. Validation reads only the other crawl’s documents.

from dew.data.dataset import DataPartition
batch = next(iter(data.train(DataPartition())))
print(batch["text"].shape)
print(repr(tokenizer.decode(batch["text"][0][:80])))
(64, 1025)
'Pacific lamprey (Entosphenus tridentatus) is an anadromous species in an ancient lineage of jawless fishes. The species is native to the North Pacific and its marine-accessible freshwater rivers and streams. Pacific lamprey are understudied relative to other anadromous fishes and has severely declined in abundance throughout the Columbia River Basin. Indigenous people of the Snake and Columbia River'

CausalTransformer from dew.nn.backbones is the decoder. max_seq_len sets the size of the KV cache, which only generation uses, so it has to cover the prompt plus every token we generate at the end. We compute in bfloat16 and keep the weights in float32.

We leave out dropout. It slows down memorisation, which matters most when a model sees the same text many times, and here the model reads each window of text only once.

from dew.nn.backbones import CausalTransformer
model = CausalTransformer(
vocab_size=tokenizer.vocab_size,
emb_features=EMB_FEATURES,
num_layers=NUM_LAYERS,
num_heads=NUM_HEADS,
max_seq_len=SEQUENCE_LENGTH,
dtype=jnp.bfloat16,
attention_impl="auto",
)

LMObjective is the next-token cross entropy. It shifts the batch into inputs and targets itself and computes the loss in float32. Like the diffusion objective, it keeps an EMA of the weights here, because we pass ema_decay=0.999; without that argument it keeps none.

The learning rate warms up over the first 10% of the steps and then decays along a cosine. eval_every runs the validation pass from the EMA weights, and Perplexity() turns it into one number, which the next section explains.

from dew import Checkpoints, LocalTracker, Trainer
from dew.config import OptimConfig
from dew.objectives.lm import LMObjective, Perplexity
from dew.training.optim import Cosine
objective = LMObjective(model, SEQUENCE_LENGTH, ema_decay=0.999)
schedule = Cosine(
peak=LEARNING_RATE,
warmup_steps=STEPS // 10, decay_steps=STEPS, end=LEARNING_RATE / 10)
trainer = Trainer(
objective, OptimConfig(schedule=schedule),
key=jax.random.key(SEED),
checkpoints=Checkpoints(RUN_DIR),
tracker=LocalTracker(f"{RUN_DIR}/tracking"),
)
variables = jax.eval_shape(objective.init, jax.random.key(0))
n_params = sum(x.size for x in jax.tree_util.tree_leaves(variables["params"]))
print(f"{n_params / 1e6:.1f}M parameters")
151.0M parameters

The same fit as before, except that we now also pass metrics=(Perplexity(),), so that every validation pass reports the validation perplexity. We train in STAGES stages, each call carrying on from the checkpoint the one before it wrote, and after each stage we ask the model what comes after PROMPT.

prompt = jnp.asarray([tokenizer.encode(PROMPT, add_special_tokens=False)], jnp.int32)
checked = []
for stage in range(1, STAGES + 1):
state = trainer.fit(data, steps=STEPS * stage // STAGES, log_every=STEPS // 10,
eval_every=STEPS // STAGES, checkpoint_every=STEPS,
metrics=(Perplexity(),))
logits = model.apply(state.averaged, prompt)[0, -1].astype(jnp.float32)
checked.append((int(state.step), np.asarray(jax.nn.softmax(logits))))
Training CausalTransformer from step 0 to 1600: 151.0M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
19:35:59 WARNING dew.nn.kernels.tuning: 8 kernel launches have no config measured on NVIDIA A100-SXM4-40GB, so they run the nearest measured shape's or their heuristic: measure them on a GPU to yourself with Trainer(tune_kernels=True) or `python tools/kernel_picks.py`
19:37:06 WARNING dew.training.planning: timing the plan against its batch whole at remat 'minimal', the head tiled
19:37:43 WARNING dew.training.trainer: the step fits the devices in the plan {'split': 2, 'rows': 32, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'probes': [{'split': 1, 'rows': 64, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'fits': False, 'headroom': -8563105846, 'temp': 46204800264, 'argument': 2643006516}, {'split': 64, 'rows': 1, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'fits': True, 'headroom': 36126465106, 'temp': 911165496, 'argument': 3247069244}, {'split': 8, 'rows': 8, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'fits': True, 'headroom': 31070413138, 'temp': 5967217464, 'argument': 3247069244}, {'split': 2, 'rows': 32, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'fits': True, 'headroom': 13736270674, 'temp': 23301359928, 'argument': 3247069244}, {'split': 1, 'rows': 64, 'head_tile': [4096, 8192], 'remat': 'minimal', 'xla_defaults': False, 'fits': True, 'headroom': 19918888250, 'temp': 17722806168, 'argument': 2643006516}], 'timed': [{'split': 2, 'rows': 32, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'seconds': 0.375743939700078}, {'split': 1, 'rows': 64, 'head_tile': [4096, 8192], 'remat': 'minimal', 'xla_defaults': False, 'seconds': 0.4013586002000011}]}
step 800/1600  loss 5.111  ce 5.104  perplexity 164.7  token_accuracy 23.0%  data_wait_ms 0.04193  grad_norm 0.8304  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 26.48  step_time_ms 382.2  samples_per_sec 167.5  tokens 65,536  tokens_per_s 171,482  mfu 52.9%  hlo_flops_util_lower_bound 52.9%  hfu_lower_bound 52.9%  unsized_kernels 244.0  accepted 100.0%  learning_rate 5.993e-04  accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 51,392  attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 51,392  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  0:05:07 left
step 1600/1600  loss 4.163  ce 4.148  perplexity 63.31  token_accuracy 30.8%  data_wait_ms 0.04213  grad_norm 0.4242  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 26.48  step_time_ms 382.8  samples_per_sec 167.2  tokens 65,536  tokens_per_s 171,191  mfu 52.8%  hlo_flops_util_lower_bound 52.8%  hfu_lower_bound 52.8%  unsized_kernels 244.0  accepted 100.0%  learning_rate 5.838e-04  accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 102,592  attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 102,592  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0
eval val (ema) at step 1600: perplexity 128.6   loss 4.856 (8 records in 10.63 s)
Trained 1600 steps in 0:13:35: first step after 154.17 s, then 2.6 step/s
76.1% of the wall time in steps, final loss 4.163
Resumed from step 1600 in /content/runs/05-lm
Training CausalTransformer from step 1600 to 3200: 151.0M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 2400/3200  loss 3.894  ce 3.903  perplexity 49.56  token_accuracy 32.7%  data_wait_ms 244.6  grad_norm 0.3212  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 27.55  step_time_ms 628.0  samples_per_sec 101.9  tokens 65,536  tokens_per_s 104,353  mfu 32.2%  hlo_flops_util_lower_bound 32.2%  hfu_lower_bound 32.2%  unsized_kernels 244.0  accepted 100.0%  learning_rate 5.369e-04  accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 153,792  attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 153,792  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  0:05:06 left
step 3200/3200  loss 3.651  ce 3.623  perplexity 37.46  token_accuracy 35.4%  data_wait_ms 0.04208  grad_norm 0.2834  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 27.55  step_time_ms 382.8  samples_per_sec 167.2  tokens 65,536  tokens_per_s 171,198  mfu 52.8%  hlo_flops_util_lower_bound 52.8%  hfu_lower_bound 52.8%  unsized_kernels 244.0  accepted 100.0%  learning_rate 4.651e-04  accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 204,992  attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 204,992  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0
eval val (ema) at step 3200: perplexity 43.69   loss 3.777 (8 records in 2.46 s)
WARNING:absl:Waiting for previous save to complete took 8.828419 seconds. If this number is high, consider checkpointing less frequently.
Trained 1600 steps in 0:15:15: first step after 67.42 s, then 2.0 step/s
89.3% of the wall time in steps, final loss 3.651
Resumed from step 3200 in /content/runs/05-lm
Training CausalTransformer from step 3200 to 4800: 151.0M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 4000/4800  loss 3.541  ce 3.514  perplexity 33.59  token_accuracy 38.8%  data_wait_ms 256.2  grad_norm 0.2565  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 27.61  step_time_ms 643.0  samples_per_sec 99.54  tokens 65,536  tokens_per_s 101,926  mfu 31.4%  hlo_flops_util_lower_bound 31.4%  hfu_lower_bound 31.4%  unsized_kernels 244.0  accepted 100.0%  learning_rate 3.770e-04  accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 256,192  attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 256,192  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  0:05:04 left
step 4800/4800  loss 3.770  ce 3.781  perplexity 43.88  token_accuracy 33.5%  data_wait_ms 0.04266  grad_norm 0.2614  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 27.61  step_time_ms 382.4  samples_per_sec 167.4  tokens 65,536  tokens_per_s 171,367  mfu 52.9%  hlo_flops_util_lower_bound 52.9%  hfu_lower_bound 52.9%  unsized_kernels 244.0  accepted 100.0%  learning_rate 2.832e-04  accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 307,392  attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 307,392  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0
eval val (ema) at step 4800: perplexity 33.79   loss 3.520 (8 records in 2.56 s)
WARNING:absl:Waiting for previous save to complete took 8.881389 seconds. If this number is high, consider checkpointing less frequently.
Trained 1600 steps in 0:15:28: first step after 68.64 s, then 1.9 step/s
89.3% of the wall time in steps, final loss 3.770
Resumed from step 4800 in /content/runs/05-lm
Training CausalTransformer from step 4800 to 6400: 151.0M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 5600/6400  loss 3.368  ce 3.304  perplexity 27.23  token_accuracy 39.8%  data_wait_ms 262.2  grad_norm 0.2422  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 27.66  step_time_ms 643.9  samples_per_sec 99.39  tokens 65,536  tokens_per_s 101,780  mfu 31.4%  hlo_flops_util_lower_bound 31.4%  hfu_lower_bound 31.4%  unsized_kernels 244.0  accepted 100.0%  learning_rate 1.951e-04  accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 358,592  attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 358,592  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  0:05:06 left
step 6400/6400  loss 3.693  ce 3.710  perplexity 40.87  token_accuracy 33.6%  data_wait_ms 0.04190  grad_norm 0.2342  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 27.66  step_time_ms 382.4  samples_per_sec 167.4  tokens 65,536  tokens_per_s 171,373  mfu 52.9%  hlo_flops_util_lower_bound 52.9%  hfu_lower_bound 52.9%  unsized_kernels 244.0  accepted 100.0%  learning_rate 1.232e-04  accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 409,792  attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 409,792  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0
eval val (ema) at step 6400: perplexity 29.56   loss 3.386 (8 records in 2.42 s)
WARNING:absl:Waiting for previous save to complete took 9.207719 seconds. If this number is high, consider checkpointing less frequently.
Trained 1600 steps in 0:15:27: first step after 67.07 s, then 1.9 step/s
89.5% of the wall time in steps, final loss 3.693
Resumed from step 6400 in /content/runs/05-lm
Training CausalTransformer from step 6400 to 8000: 151.0M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 7200/8000  loss 3.547  ce 3.519  perplexity 33.74  token_accuracy 35.5%  data_wait_ms 248.6  grad_norm 0.2370  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 27.71  step_time_ms 633.2  samples_per_sec 101.1  tokens 65,536  tokens_per_s 103,498  mfu 31.9%  hlo_flops_util_lower_bound 31.9%  hfu_lower_bound 31.9%  unsized_kernels 244.0  accepted 100.0%  learning_rate 7.632e-05  accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 460,992  attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 460,992  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  0:05:06 left
step 8000/8000  loss 3.519  ce 3.536  perplexity 34.33  token_accuracy 36.6%  data_wait_ms 0.04191  grad_norm 0.2393  compiles 1.000  cache_hits 0  cache_misses 0  peak_memory_gib 27.71  step_time_ms 382.4  samples_per_sec 167.4  tokens 65,536  tokens_per_s 171,403  mfu 52.9%  hlo_flops_util_lower_bound 52.9%  hfu_lower_bound 52.9%  unsized_kernels 244.0  accepted 100.0%  learning_rate 6.000e-05  accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 512,192  attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 512,192  chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0  truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0
eval val (ema) at step 8000: perplexity 28.10   loss 3.336 (8 records in 1.67 s)
WARNING:absl:Waiting for previous save to complete took 9.070493 seconds. If this number is high, consider checkpointing less frequently.
Trained 1600 steps in 0:15:19: first step after 67.31 s, then 1.9 step/s
89.3% of the wall time in steps, final loss 3.519

Perplexity is just the exponential of the cross entropy:

PPL=eH\text{PPL} = e^{H}

where:

  • HH is the average cross entropy per token, in nats.

Roughly, it tells us how many tokens the model is choosing between at each position. An untrained model scores about 49,152, one guess out of every token in the vocabulary. The plot shows the training loss and the validation loss, the logarithm of the perplexity, from the tracker’s journal.

rows = [json.loads(line) for line in open(f"{RUN_DIR}/tracking/scalars.jsonl")]
train = [(row["step"], row["scalars"]["train/loss"]) for row in rows
if "train/loss" in row["scalars"]]
val = [(row["step"], row["scalars"]["val/perplexity"]) for row in rows
if "val/perplexity" in row["scalars"]]
print("validation perplexity:", ", ".join(f"{ppl:.1f} at {step}" for step, ppl in val))
figures.curve({"training": [loss for _, loss in train]}, x=[step for step, _ in train],
x_label="step", y_label="loss (nats per token)")
validation perplexity: 128.6 at 1600, 43.7 at 3200, 33.8 at 4800, 29.6 at 6400, 28.1 at 8000
loss (nats per token) against step3.544.552000400060008000steploss (nats per token)3.52
figures.curve({"validation": [float(np.log(ppl)) for _, ppl in val]},
x=[step for step, _ in val], x_label="step", y_label="validation loss")
validation loss against step3.544.552000400060008000stepvalidation loss3.34

Validation perplexity falls from about 129 at step 1,600 to about 28 at the end: the model has narrowed each next token down from 49,152 choices to a few dozen. At step 1,600 validation perplexity is 129 against 63 in training: validation uses the averaged weights and text from the other crawl, while training reports the current weights on its latest batch. The training loss jumps around more than the validation loss because every batch is new text, while the validation pass reads the same eight windows each time.

What did the model learn along the way? Here are its guesses for the token after “Plants need light to grow because”, after each stage of training, for the eight tokens it likes best at the end.

final = checked[-1][1]
top = np.argsort(final)[::-1][:8]
names = [tokenizer.decode([token]).strip() or "·" for token in top]
for step, probs in checked:
shares = ", ".join(f"{name!r} {probs[t]:.3f}" for name, t in zip(names, top))
print(f"step {step}: {shares}")
figures.bars([probs[top] for _, probs in checked], labels=names,
captions=[f"step {step:,}" for step, _ in checked],
y_max=float(final[top[0]]), y_label="P(next token)")
step 1600: 'they' 0.013, 'of' 0.033, 'the' 0.116, 'it' 0.043, 'their' 0.003, 'plants' 0.001, 'sunlight' 0.000, 'light' 0.001
step 3200: 'they' 0.305, 'of' 0.185, 'the' 0.106, 'it' 0.082, 'their' 0.025, 'plants' 0.013, 'sunlight' 0.004, 'light' 0.002
step 4800: 'they' 0.309, 'of' 0.176, 'the' 0.114, 'it' 0.094, 'their' 0.039, 'plants' 0.009, 'sunlight' 0.014, 'light' 0.004
step 6400: 'they' 0.301, 'of' 0.194, 'the' 0.125, 'it' 0.086, 'their' 0.041, 'plants' 0.011, 'sunlight' 0.016, 'light' 0.010
step 8000: 'they' 0.305, 'of' 0.153, 'the' 0.135, 'it' 0.093, 'their' 0.041, 'plants' 0.016, 'sunlight' 0.014, 'light' 0.014
P(next token): they, of, the, it, their, plants, sunlight, lightP(next token)they0.010.310.310.300.31of0.030.190.180.190.15the0.120.110.110.130.14it0.040.080.090.090.09their0.000.030.040.040.04plants0.000.010.010.010.02sunlight0.000.000.010.020.01light0.000.000.000.010.01step 1,600step 3,200step 4,800step 6,400step 8,000

At step 1,600 the model is still spreading its bets: of the eight tokens shown, “the” leads at 0.12, and “they” gets only 0.01. By step 3,200 it has settled on “they” at about 0.3, and that barely moves afterwards. The tokens that would make the sentence about plants, “plants”, “sunlight” and “light”, climb slowly, from almost nothing to between 0.01 and 0.02 by the end.

A decoder writes one token at a time, and each new token attends to every token before it. Recomputing all of those every step would be really wasteful, so the KV cache stores the keys and values of every earlier position: the prompt runs through the model once, and each later step runs only the newest token.

generate runs that loop as compiled code. Sampling sets the temperature and top-k: temperature 0 always takes the most likely token, and higher temperatures pick more freely. We sample from the EMA weights, state.averaged.

Temperature divides the logits before the softmax. Here is the model’s real distribution for the next token after the prompt, at three temperatures: below 1 the favourite takes over, above 1 the others catch up.

logits = model.apply(state.averaged, prompt)[0, -1].astype(jnp.float32)
temperatures = [0.5, 1.0, 1.5]
figures.bars([np.asarray(jax.nn.softmax(logits / t))[top] for t in temperatures],
labels=names, captions=[f"temperature {t:g}" for t in temperatures],
y_label="P(next token)")
P(next token): they, of, the, it, their, plants, sunlight, lightP(next token)they0.630.310.09of0.160.150.05the0.120.140.05it0.060.090.04their0.010.040.02plants0.000.020.01sunlight0.000.010.01light0.000.010.01temperature 0.5temperature 1temperature 1.5
from dew.sampling import Sampling, generate
out = generate(model, state.averaged, prompt, max_new_tokens=MAX_NEW_TOKENS,
key=jax.random.key(1), sampling=Sampling(temperature=0.8, top_k=40))
print(tokenizer.decode(out.tokens[0]))
Plants need light to grow because the roots of the plant can also withstand UV rays. There is no way to get the nutrients they need. As a result, the plant is unable to get the roots of the plant and could easily become too drought.
The soil that is used to grow plants comes from water sources, so it’s important to water sources to prevent them from coming back. Soil can be very difficult to grow if you’ve been gardening and have a few plants or just one type of soil that would be ideal for your space.
Soil is the basis for most fruits and vegetables. You could grow large

At temperature 0.8 the model writes fluent English about plants, soil and gardening, and it stays on topic for a whole paragraph. It doesn’t make much sense, though: a plant that is “unable to get the roots of the plant”, soil that “can be very difficult to grow”. It has learned how this kind of text sounds, not yet what is true.

Greedy decoding (temperature 0) always picks the most likely token. It tends to fall into a loop, as it does below from the third line on, and its very first sentence just repeats the prompt: plants need light to grow because they need light to grow. That is why sampling is usually preferred for text like this.

greedy = generate(model, state.averaged, prompt, max_new_tokens=MAX_NEW_TOKENS,
key=jax.random.key(0), sampling=Sampling(temperature=0.0))
print(tokenizer.decode(greedy.tokens[0]))
Plants need light to grow because they need light to grow.
The plant is a very hardy plant that can grow up to 10 feet tall. It is also known as a “green plant” because it is a very hardy plant that can grow up to 10 feet tall.
The plant is also known as a “green plant” because it is a very hardy plant that can grow up to 10 feet tall.
The plant is also known as a “green plant” because it is a very hardy plant that can grow up to 10 feet tall.
The plant is also known as

fit wrote a checkpoint to RUN_DIR at the end of training: the weights, the EMA, the optimizer state, the step counters and the position in the data. A new Trainer over the same objective restores it with place(), and greedy decoding from the restored weights gives exactly the same text.

restored, _, _ = Trainer(objective, OptimConfig(schedule=schedule).build(STEPS), key=jax.random.key(SEED),
checkpoints=Checkpoints(RUN_DIR)).place()
again = generate(model, restored.averaged, prompt, max_new_tokens=MAX_NEW_TOKENS,
key=jax.random.key(0), sampling=Sampling(temperature=0.0))
print("restored step:", int(restored.step))
print("same greedy text:", bool(np.array_equal(again.tokens, greedy.tokens)))
Resumed from step 8000 in /content/runs/05-lm
restored step: 8000
same greedy text: True

Export the documents once with dew.data.text.export_tokens(dataset, store, TOKENIZER) to produce indexed token records and their metadata. TokenFiles(store).pipe(presets.lm, None, ...) reads that store without tokenizing it again. recipes/lm/train.py runs training from a recorded configuration. Notebook 07 scales this workflow to several devices; notebook 08 starts from a pretrained decoder.