Train a language model
So far we have only generated images. But the same Trainer trains language models too, and in this notebook we shall train a small decoder-only transformer on FineWeb-Edu, a large collection of educational web pages, and then let it write some text of its own.
That’s the whole training signal: at every position, make the token that actually came next a little more likely.
The model is built from the same pieces current open models use: RMSNorm (Zhang and Sennrich, 2019), rotary position embeddings (Su et al., 2021), a gated MLP (Shazeer, 2020), and attention that can share its key and value heads across groups of query heads (Ainslie et al., 2023). Our small model keeps one key and value head per query head. Dew’s CausalTransformer builds it, LMObjective holds the next-token loss, and the same Trainer as in the diffusion notebooks runs the training.
Install dependencies
Section titled “Install dependencies”Dew from GitHub with its cuda13 and streaming extras. Dew streams the text below from the Hub.
%pip install -q "dewml[cuda13,streaming] @ git+https://github.com/AshishKumar4/dew@71b43f2dc070cf289e1fbaf6177e484215d4ee1a" --extra-index-url https://pypi.dewml.dev/simpleinstall log
Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Building wheel for dewml (pyproject.toml) ... done
Settings
Section titled “Settings”TOKENIZER = "HuggingFaceTB/SmolLM2-135M"SEQUENCE_LENGTH = 1024BATCH_SIZE = 64STEPS = 8000STAGES = 5LEARNING_RATE = 6e-4EMB_FEATURES = 768NUM_LAYERS = 12NUM_HEADS = 12MAX_NEW_TOKENS = 120PROMPT = "Plants need light to grow because"RUN_DIR = "runs/05-lm"SEED = 0import json
import jaximport jax.numpy as jnpimport numpy as np
from dew import figuresfrom dew.training import prepare_process
prepare_process() # Dew's setup for a training process, before the first JAX callprint(jax.devices())[CudaDevice(id=0)]
From text to tokens
Section titled “From text to tokens”A language model predicts the next token from the ones before it. We use the tokenizer of SmolLM2 (Allal et al., 2025), a byte-pair encoding: it splits text into pieces from a vocabulary of 49,152, whole words where they’re common and fragments of rarer ones, and it was trained on text much like ours.
We stream FineWeb-Edu’s 10-billion-token sample (Penedo et al., 2024) straight from the Hub’s Parquet shards, and hand it to Dew’s language-model preset, presets.lm. For validation we read another part of FineWeb-Edu, its CC-MAIN-2025-26 crawl, eight token windows a pass, through the same preset. The preset ends each document with the tokenizer’s end-of-text id, runs the documents on into one stream, and cuts that stream into rows without losing next-token targets (pack=None).
from dew.data import Hub, presetsfrom dew.data.compile import compile as compile_data
fineweb = Hub("HuggingFaceFW/fineweb-edu", name="sample-10BT")crawl = Hub("HuggingFaceFW/fineweb-edu", name="CC-MAIN-2025-26")windows = {"seq_len": SEQUENCE_LENGTH, "batch": BATCH_SIZE, "pack": None}data = compile_data(fineweb.shuffle(seed=0, window=64).repeat().pipe(presets.lm, TOKENIZER, **windows), validation=crawl.pipe(presets.lm, TOKENIZER, **windows, records=8), read="streamed")stderr
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. WARNING:absl:Warning: Average element size is zero. Defaulting to max threads.
from dew.data import HFTokenizer
tokenizer = HFTokenizer(TOKENIZER)ids = tokenizer.encode(PROMPT, add_special_tokens=False)print(ids)print("vocabulary:", tokenizer.vocab_size)pieces = [tokenizer.decode([token]) for token in ids]figures.tokens([pieces], fills=[["tint1"] * len(pieces)])[34246, 737, 1420, 288, 1075, 975] vocabulary: 49152
Six words, six tokens. Each of these words is common enough to have a token of its own, and every one after the first carries its leading space. A rarer word would be split into pieces.
The preset produces rows of SEQUENCE_LENGTH + 1 token IDs. The first SEQUENCE_LENGTH IDs are the input; shifting by one gives the targets. The documents run on into each other, separated only by the end-of-text token, so attention can reach back across a document boundary. Validation reads only the other crawl’s documents.
from dew.data.dataset import DataPartition
batch = next(iter(data.train(DataPartition())))print(batch["text"].shape)print(repr(tokenizer.decode(batch["text"][0][:80])))(64, 1025) 'Pacific lamprey (Entosphenus tridentatus) is an anadromous species in an ancient lineage of jawless fishes. The species is native to the North Pacific and its marine-accessible freshwater rivers and streams. Pacific lamprey are understudied relative to other anadromous fishes and has severely declined in abundance throughout the Columbia River Basin. Indigenous people of the Snake and Columbia River'
The model
Section titled “The model”CausalTransformer from dew.nn.backbones is the decoder. max_seq_len sets the size of the KV cache, which only generation uses, so it has to cover the prompt plus every token we generate at the end. We compute in bfloat16 and keep the weights in float32.
We leave out dropout. It slows down memorisation, which matters most when a model sees the same text many times, and here the model reads each window of text only once.
from dew.nn.backbones import CausalTransformer
model = CausalTransformer( vocab_size=tokenizer.vocab_size, emb_features=EMB_FEATURES, num_layers=NUM_LAYERS, num_heads=NUM_HEADS, max_seq_len=SEQUENCE_LENGTH, dtype=jnp.bfloat16, attention_impl="auto",)The objective and the trainer
Section titled “The objective and the trainer”LMObjective is the next-token cross entropy. It shifts the batch into inputs and targets itself and computes the loss in float32. Like the diffusion objective, it keeps an EMA of the weights here, because we pass ema_decay=0.999; without that argument it keeps none.
The learning rate warms up over the first 10% of the steps and then decays along a cosine. eval_every runs the validation pass from the EMA weights, and Perplexity() turns it into one number, which the next section explains.
from dew import Checkpoints, LocalTracker, Trainerfrom dew.config import OptimConfigfrom dew.objectives.lm import LMObjective, Perplexityfrom dew.training.optim import Cosine
objective = LMObjective(model, SEQUENCE_LENGTH, ema_decay=0.999)schedule = Cosine( peak=LEARNING_RATE, warmup_steps=STEPS // 10, decay_steps=STEPS, end=LEARNING_RATE / 10)trainer = Trainer( objective, OptimConfig(schedule=schedule), key=jax.random.key(SEED), checkpoints=Checkpoints(RUN_DIR), tracker=LocalTracker(f"{RUN_DIR}/tracking"),)
variables = jax.eval_shape(objective.init, jax.random.key(0))n_params = sum(x.size for x in jax.tree_util.tree_leaves(variables["params"]))print(f"{n_params / 1e6:.1f}M parameters")151.0M parameters
Training
Section titled “Training”The same fit as before, except that we now also pass metrics=(Perplexity(),), so that every validation pass reports the validation perplexity. We train in STAGES stages, each call carrying on from the checkpoint the one before it wrote, and after each stage we ask the model what comes after PROMPT.
prompt = jnp.asarray([tokenizer.encode(PROMPT, add_special_tokens=False)], jnp.int32)checked = []for stage in range(1, STAGES + 1): state = trainer.fit(data, steps=STEPS * stage // STAGES, log_every=STEPS // 10, eval_every=STEPS // STAGES, checkpoint_every=STEPS, metrics=(Perplexity(),)) logits = model.apply(state.averaged, prompt)[0, -1].astype(jnp.float32) checked.append((int(state.step), np.asarray(jax.nn.softmax(logits))))Training CausalTransformer from step 0 to 1600: 151.0M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
19:35:59 WARNING dew.nn.kernels.tuning: 8 kernel launches have no config measured on NVIDIA A100-SXM4-40GB, so they run the nearest measured shape's or their heuristic: measure them on a GPU to yourself with Trainer(tune_kernels=True) or `python tools/kernel_picks.py`
19:37:06 WARNING dew.training.planning: timing the plan against its batch whole at remat 'minimal', the head tiled
19:37:43 WARNING dew.training.trainer: the step fits the devices in the plan {'split': 2, 'rows': 32, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'probes': [{'split': 1, 'rows': 64, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'fits': False, 'headroom': -8563105846, 'temp': 46204800264, 'argument': 2643006516}, {'split': 64, 'rows': 1, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'fits': True, 'headroom': 36126465106, 'temp': 911165496, 'argument': 3247069244}, {'split': 8, 'rows': 8, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'fits': True, 'headroom': 31070413138, 'temp': 5967217464, 'argument': 3247069244}, {'split': 2, 'rows': 32, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'fits': True, 'headroom': 13736270674, 'temp': 23301359928, 'argument': 3247069244}, {'split': 1, 'rows': 64, 'head_tile': [4096, 8192], 'remat': 'minimal', 'xla_defaults': False, 'fits': True, 'headroom': 19918888250, 'temp': 17722806168, 'argument': 2643006516}], 'timed': [{'split': 2, 'rows': 32, 'head_tile': None, 'remat': None, 'xla_defaults': False, 'seconds': 0.375743939700078}, {'split': 1, 'rows': 64, 'head_tile': [4096, 8192], 'remat': 'minimal', 'xla_defaults': False, 'seconds': 0.4013586002000011}]}
step 800/1600 loss 5.111 ce 5.104 perplexity 164.7 token_accuracy 23.0% data_wait_ms 0.04193 grad_norm 0.8304 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 26.48 step_time_ms 382.2 samples_per_sec 167.5 tokens 65,536 tokens_per_s 171,482 mfu 52.9% hlo_flops_util_lower_bound 52.9% hfu_lower_bound 52.9% unsized_kernels 244.0 accepted 100.0% learning_rate 5.993e-04 accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 51,392 attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 51,392 chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 0:05:07 left
step 1600/1600 loss 4.163 ce 4.148 perplexity 63.31 token_accuracy 30.8% data_wait_ms 0.04213 grad_norm 0.4242 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 26.48 step_time_ms 382.8 samples_per_sec 167.2 tokens 65,536 tokens_per_s 171,191 mfu 52.8% hlo_flops_util_lower_bound 52.8% hfu_lower_bound 52.8% unsized_kernels 244.0 accepted 100.0% learning_rate 5.838e-04 accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 102,592 attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 102,592 chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0
eval val (ema) at step 1600: perplexity 128.6 loss 4.856 (8 records in 10.63 s)
Trained 1600 steps in 0:13:35: first step after 154.17 s, then 2.6 step/s
76.1% of the wall time in steps, final loss 4.163
Resumed from step 1600 in /content/runs/05-lm
Training CausalTransformer from step 1600 to 3200: 151.0M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 2400/3200 loss 3.894 ce 3.903 perplexity 49.56 token_accuracy 32.7% data_wait_ms 244.6 grad_norm 0.3212 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 27.55 step_time_ms 628.0 samples_per_sec 101.9 tokens 65,536 tokens_per_s 104,353 mfu 32.2% hlo_flops_util_lower_bound 32.2% hfu_lower_bound 32.2% unsized_kernels 244.0 accepted 100.0% learning_rate 5.369e-04 accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 153,792 attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 153,792 chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 0:05:06 left
step 3200/3200 loss 3.651 ce 3.623 perplexity 37.46 token_accuracy 35.4% data_wait_ms 0.04208 grad_norm 0.2834 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 27.55 step_time_ms 382.8 samples_per_sec 167.2 tokens 65,536 tokens_per_s 171,198 mfu 52.8% hlo_flops_util_lower_bound 52.8% hfu_lower_bound 52.8% unsized_kernels 244.0 accepted 100.0% learning_rate 4.651e-04 accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 204,992 attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 204,992 chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0
eval val (ema) at step 3200: perplexity 43.69 loss 3.777 (8 records in 2.46 s)
WARNING:absl:Waiting for previous save to complete took 8.828419 seconds. If this number is high, consider checkpointing less frequently.
Trained 1600 steps in 0:15:15: first step after 67.42 s, then 2.0 step/s
89.3% of the wall time in steps, final loss 3.651
Resumed from step 3200 in /content/runs/05-lm
Training CausalTransformer from step 3200 to 4800: 151.0M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 4000/4800 loss 3.541 ce 3.514 perplexity 33.59 token_accuracy 38.8% data_wait_ms 256.2 grad_norm 0.2565 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 27.61 step_time_ms 643.0 samples_per_sec 99.54 tokens 65,536 tokens_per_s 101,926 mfu 31.4% hlo_flops_util_lower_bound 31.4% hfu_lower_bound 31.4% unsized_kernels 244.0 accepted 100.0% learning_rate 3.770e-04 accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 256,192 attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 256,192 chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 0:05:04 left
step 4800/4800 loss 3.770 ce 3.781 perplexity 43.88 token_accuracy 33.5% data_wait_ms 0.04266 grad_norm 0.2614 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 27.61 step_time_ms 382.4 samples_per_sec 167.4 tokens 65,536 tokens_per_s 171,367 mfu 52.9% hlo_flops_util_lower_bound 52.9% hfu_lower_bound 52.9% unsized_kernels 244.0 accepted 100.0% learning_rate 2.832e-04 accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 307,392 attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 307,392 chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0
eval val (ema) at step 4800: perplexity 33.79 loss 3.520 (8 records in 2.56 s)
WARNING:absl:Waiting for previous save to complete took 8.881389 seconds. If this number is high, consider checkpointing less frequently.
Trained 1600 steps in 0:15:28: first step after 68.64 s, then 1.9 step/s
89.3% of the wall time in steps, final loss 3.770
Resumed from step 4800 in /content/runs/05-lm
Training CausalTransformer from step 4800 to 6400: 151.0M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 5600/6400 loss 3.368 ce 3.304 perplexity 27.23 token_accuracy 39.8% data_wait_ms 262.2 grad_norm 0.2422 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 27.66 step_time_ms 643.9 samples_per_sec 99.39 tokens 65,536 tokens_per_s 101,780 mfu 31.4% hlo_flops_util_lower_bound 31.4% hfu_lower_bound 31.4% unsized_kernels 244.0 accepted 100.0% learning_rate 1.951e-04 accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 358,592 attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 358,592 chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 0:05:06 left
step 6400/6400 loss 3.693 ce 3.710 perplexity 40.87 token_accuracy 33.6% data_wait_ms 0.04190 grad_norm 0.2342 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 27.66 step_time_ms 382.4 samples_per_sec 167.4 tokens 65,536 tokens_per_s 171,373 mfu 52.9% hlo_flops_util_lower_bound 52.9% hfu_lower_bound 52.9% unsized_kernels 244.0 accepted 100.0% learning_rate 1.232e-04 accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 409,792 attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 409,792 chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0
eval val (ema) at step 6400: perplexity 29.56 loss 3.386 (8 records in 2.42 s)
WARNING:absl:Waiting for previous save to complete took 9.207719 seconds. If this number is high, consider checkpointing less frequently.
Trained 1600 steps in 0:15:27: first step after 67.07 s, then 1.9 step/s
89.5% of the wall time in steps, final loss 3.693
Resumed from step 6400 in /content/runs/05-lm
Training CausalTransformer from step 6400 to 8000: 151.0M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute
step 7200/8000 loss 3.547 ce 3.519 perplexity 33.74 token_accuracy 35.5% data_wait_ms 248.6 grad_norm 0.2370 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 27.71 step_time_ms 633.2 samples_per_sec 101.1 tokens 65,536 tokens_per_s 103,498 mfu 31.9% hlo_flops_util_lower_bound 31.9% hfu_lower_bound 31.9% unsized_kernels 244.0 accepted 100.0% learning_rate 7.632e-05 accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 460,992 attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 460,992 chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 0:05:06 left
step 8000/8000 loss 3.519 ce 3.536 perplexity 34.33 token_accuracy 36.6% data_wait_ms 0.04191 grad_norm 0.2393 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 27.71 step_time_ms 382.4 samples_per_sec 167.4 tokens 65,536 tokens_per_s 171,403 mfu 52.9% hlo_flops_util_lower_bound 52.9% hfu_lower_bound 52.9% unsized_kernels 244.0 accepted 100.0% learning_rate 6.000e-05 accepted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 512,192 attempted/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 512,192 chunks_per_record/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0 truncated/files:parquet:('hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0000.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0001.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0002.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0003.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0004.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0005.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0006.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0007.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0008.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0009.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0010.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0011.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0012.parquet', 'hf://datasets/HuggingFaceFW/fineweb-edu@92cece42bcce787ee4af4619ab449fe48d86230d/sample-10BT/train/0013.parquet') 0
eval val (ema) at step 8000: perplexity 28.10 loss 3.336 (8 records in 1.67 s)
WARNING:absl:Waiting for previous save to complete took 9.070493 seconds. If this number is high, consider checkpointing less frequently.
Trained 1600 steps in 0:15:19: first step after 67.31 s, then 1.9 step/s
89.3% of the wall time in steps, final loss 3.519Perplexity
Section titled “Perplexity”Perplexity is just the exponential of the cross entropy:
where:
- is the average cross entropy per token, in nats.
Roughly, it tells us how many tokens the model is choosing between at each position. An untrained model scores about 49,152, one guess out of every token in the vocabulary. The plot shows the training loss and the validation loss, the logarithm of the perplexity, from the tracker’s journal.
rows = [json.loads(line) for line in open(f"{RUN_DIR}/tracking/scalars.jsonl")]train = [(row["step"], row["scalars"]["train/loss"]) for row in rows if "train/loss" in row["scalars"]]val = [(row["step"], row["scalars"]["val/perplexity"]) for row in rows if "val/perplexity" in row["scalars"]]print("validation perplexity:", ", ".join(f"{ppl:.1f} at {step}" for step, ppl in val))figures.curve({"training": [loss for _, loss in train]}, x=[step for step, _ in train], x_label="step", y_label="loss (nats per token)")validation perplexity: 128.6 at 1600, 43.7 at 3200, 33.8 at 4800, 29.6 at 6400, 28.1 at 8000
figures.curve({"validation": [float(np.log(ppl)) for _, ppl in val]}, x=[step for step, _ in val], x_label="step", y_label="validation loss")Validation perplexity falls from about 129 at step 1,600 to about 28 at the end: the model has narrowed each next token down from 49,152 choices to a few dozen. At step 1,600 validation perplexity is 129 against 63 in training: validation uses the averaged weights and text from the other crawl, while training reports the current weights on its latest batch. The training loss jumps around more than the validation loss because every batch is new text, while the validation pass reads the same eight windows each time.
What did the model learn along the way? Here are its guesses for the token after “Plants need light to grow because”, after each stage of training, for the eight tokens it likes best at the end.
final = checked[-1][1]top = np.argsort(final)[::-1][:8]names = [tokenizer.decode([token]).strip() or "·" for token in top]for step, probs in checked: shares = ", ".join(f"{name!r} {probs[t]:.3f}" for name, t in zip(names, top)) print(f"step {step}: {shares}")figures.bars([probs[top] for _, probs in checked], labels=names, captions=[f"step {step:,}" for step, _ in checked], y_max=float(final[top[0]]), y_label="P(next token)")step 1600: 'they' 0.013, 'of' 0.033, 'the' 0.116, 'it' 0.043, 'their' 0.003, 'plants' 0.001, 'sunlight' 0.000, 'light' 0.001 step 3200: 'they' 0.305, 'of' 0.185, 'the' 0.106, 'it' 0.082, 'their' 0.025, 'plants' 0.013, 'sunlight' 0.004, 'light' 0.002 step 4800: 'they' 0.309, 'of' 0.176, 'the' 0.114, 'it' 0.094, 'their' 0.039, 'plants' 0.009, 'sunlight' 0.014, 'light' 0.004 step 6400: 'they' 0.301, 'of' 0.194, 'the' 0.125, 'it' 0.086, 'their' 0.041, 'plants' 0.011, 'sunlight' 0.016, 'light' 0.010 step 8000: 'they' 0.305, 'of' 0.153, 'the' 0.135, 'it' 0.093, 'their' 0.041, 'plants' 0.016, 'sunlight' 0.014, 'light' 0.014
At step 1,600 the model is still spreading its bets: of the eight tokens shown, “the” leads at 0.12, and “they” gets only 0.01. By step 3,200 it has settled on “they” at about 0.3, and that barely moves afterwards. The tokens that would make the sentence about plants, “plants”, “sunlight” and “light”, climb slowly, from almost nothing to between 0.01 and 0.02 by the end.
Generating text
Section titled “Generating text”A decoder writes one token at a time, and each new token attends to every token before it. Recomputing all of those every step would be really wasteful, so the KV cache stores the keys and values of every earlier position: the prompt runs through the model once, and each later step runs only the newest token.
generate runs that loop as compiled code. Sampling sets the temperature and top-k: temperature 0 always takes the most likely token, and higher temperatures pick more freely. We sample from the EMA weights, state.averaged.
Temperature divides the logits before the softmax. Here is the model’s real distribution for the next token after the prompt, at three temperatures: below 1 the favourite takes over, above 1 the others catch up.
logits = model.apply(state.averaged, prompt)[0, -1].astype(jnp.float32)temperatures = [0.5, 1.0, 1.5]figures.bars([np.asarray(jax.nn.softmax(logits / t))[top] for t in temperatures], labels=names, captions=[f"temperature {t:g}" for t in temperatures], y_label="P(next token)")from dew.sampling import Sampling, generate
out = generate(model, state.averaged, prompt, max_new_tokens=MAX_NEW_TOKENS, key=jax.random.key(1), sampling=Sampling(temperature=0.8, top_k=40))print(tokenizer.decode(out.tokens[0]))Plants need light to grow because the roots of the plant can also withstand UV rays. There is no way to get the nutrients they need. As a result, the plant is unable to get the roots of the plant and could easily become too drought. The soil that is used to grow plants comes from water sources, so it’s important to water sources to prevent them from coming back. Soil can be very difficult to grow if you’ve been gardening and have a few plants or just one type of soil that would be ideal for your space. Soil is the basis for most fruits and vegetables. You could grow large
At temperature 0.8 the model writes fluent English about plants, soil and gardening, and it stays on topic for a whole paragraph. It doesn’t make much sense, though: a plant that is “unable to get the roots of the plant”, soil that “can be very difficult to grow”. It has learned how this kind of text sounds, not yet what is true.
Greedy decoding (temperature 0) always picks the most likely token. It tends to fall into a loop, as it does below from the third line on, and its very first sentence just repeats the prompt: plants need light to grow because they need light to grow. That is why sampling is usually preferred for text like this.
greedy = generate(model, state.averaged, prompt, max_new_tokens=MAX_NEW_TOKENS, key=jax.random.key(0), sampling=Sampling(temperature=0.0))print(tokenizer.decode(greedy.tokens[0]))Plants need light to grow because they need light to grow. The plant is a very hardy plant that can grow up to 10 feet tall. It is also known as a “green plant” because it is a very hardy plant that can grow up to 10 feet tall. The plant is also known as a “green plant” because it is a very hardy plant that can grow up to 10 feet tall. The plant is also known as a “green plant” because it is a very hardy plant that can grow up to 10 feet tall. The plant is also known as
Reloading the checkpoint
Section titled “Reloading the checkpoint”fit wrote a checkpoint to RUN_DIR at the end of training: the weights, the EMA, the optimizer state, the step counters and the position in the data. A new Trainer over the same objective restores it with place(), and greedy decoding from the restored weights gives exactly the same text.
restored, _, _ = Trainer(objective, OptimConfig(schedule=schedule).build(STEPS), key=jax.random.key(SEED), checkpoints=Checkpoints(RUN_DIR)).place()again = generate(model, restored.averaged, prompt, max_new_tokens=MAX_NEW_TOKENS, key=jax.random.key(0), sampling=Sampling(temperature=0.0))print("restored step:", int(restored.step))print("same greedy text:", bool(np.array_equal(again.tokens, greedy.tokens)))Resumed from step 8000 in /content/runs/05-lm restored step: 8000 same greedy text: True
Where to go next
Section titled “Where to go next”Export the documents once with dew.data.text.export_tokens(dataset, store, TOKENIZER) to produce indexed token records and their metadata. TokenFiles(store).pipe(presets.lm, None, ...) reads that store without tokenizing it again. recipes/lm/train.py runs training from a recorded configuration. Notebook 07 scales this workflow to several devices; notebook 08 starts from a pretrained decoder.