HDF5 Tokenized Corpus
The downloaded corpus has to land in a layout the trainer can stream from at line speed. JSONL on disk does not survive 16 dataloader workers. HDF5 with a resizable, chunked integer dataset does. This lesson builds streaming tokenization into a resizable HDF5 dataset, sharded write across multiple files, memory-mapped read at training time, and a sliding-window dataloader that produces fixed-length sequences with the right packing.
Learning Objectives
- Stream documents into a resizable HDF5 integer dataset with deterministic chunking.
- Shard the write across multiple HDF5 files so failure is bounded and parallelism is possible.
- Read tokens back through HDF5's page-cache-backed chunked layout so the dataloader copies into batch buffers only at batch time.
- Implement a sliding-window dataloader that emits fixed-length training sequences with explicit packing rules.
The Problem
A modern language-model training run reads tokens at hundreds of thousands of samples per second across dozens of workers. JSONL on disk dies at the first cold-cache page fault: the JSON parser is slow, the document boundaries are not addressable, and seeking to "sample 4,217,884" requires scanning the file. Even Parquet, which compresses well, is a poor fit because the trainer does not want columns; it wants a flat token stream with $O(1)$ random access.
HDF5 fits because it offers a chunked, resizable, integer-only dataset whose chunks are page-cache friendly at read time. The trainer asks for a slice of tokens[3,200,000 : 3,200,8192] and HDF5 copies the requested hyperslab from the page cache into a freshly allocated NumPy array. The cost is one open file handle and a chunk-sized page-cache footprint per worker, which is negligible compared to the cost of decoding JSONL.
The build problem is making the write side honest. Resizable datasets are easy to misuse: write one document at a time and the HDF5 file is fragmented to the point of unusable. Write all documents in one resize and a process death loses the whole shard. The right discipline is buffer-then-extend, with a buffer size that matches the chunk size, and a sharded write that splits the workload across files so a crash loses at most one shard.
The Concept
Resizable HDF5 done right
The token dataset is created with maxshape=(None,) and a fixed chunks=(chunk_size,). Writing proceeds by buffering tokens in a NumPy array of length chunk_size. When the buffer fills, the dataset is resized by exactly chunk_size and the buffer is written into the new range. At end-of-shard the residual buffer is written into a final partial range. Every write is contiguous and chunk-aligned except the last one, which the reader is told to truncate at the recorded token_count in the shard's HDF5 attributes.
Sharded write
A single HDF5 file is a single point of failure. The pipeline writes shards in parallel: each input shard from Phase 19 lesson 42 produces one HDF5 output shard. A shards.json index records, per shard, the file path, the token count, the document count, and a sha256 over the tokens. The trainer reads shards.json to compute global offsets and to validate the corpus.
Memory-mapped read
At training time each worker opens its share of HDF5 files in swmr=True mode and asks for tokens[start:stop]. HDF5's chunk layout makes this a page-cache-backed read once the chunk is hot. The worker never materialises the whole file: the slice is copied into the dataloader's batch buffer, which the dataloader then copies into a pinned-memory training tensor at batch time. The hot path has one syscall per chunk transition; everything else is RAM access.
Sliding-window dataloader
The dataloader is the only stage that knows about training-sequence length. It picks a random start index in the global token stream, reads window_size + 1 tokens, and returns (input, target) = (tokens[:-1], tokens[1:]). Document boundaries are not enforced: a window may straddle two documents, with an explicit boundary_token_id between them so the model learns to use the separator. This is the standard packing rule; it is also the rule a beginner forgets, ending up with a corpus that is 8 percent training boundary tokens and 92 percent natural text.
Build It
code/main.py implements:
Tokenizer- a byte-level deterministic tokenizer good enough for the demo. The interface isencode(text) -> list[int]andvocab_size.HDF5ShardWriter- opens a resizable integer dataset, buffers tokens to chunk size, resizes and writes in fixed-size strides, recordstoken_countandsha256as HDF5 attributes on close.ShardedTokenizationPipeline- iterates input documents, routes them to a writer, and emits ashards.jsonindex.MmapTokenStore- opens shard files for memory-mapped reads, computes global offsets, exposes a singleget_slice(start, stop)API.SlidingWindowDataloader- picks random windows from the global stream and yields(input_ids, target_ids)NumPy arrays.
A demo at the bottom of the file builds a tiny in-memory corpus, tokenizes into two shards, opens them via memory map, runs the dataloader for 10 batches, and prints the per-batch shape and a checksum.
Run it:
python3 code/main.pyThe script exits zero and prints batch checksums.
Production Patterns
Four patterns scale this lesson to a real training run.
Chunk size equals the typical read. The trainer reads window_size + 1 tokens per sample. Set the HDF5 chunk to a multiple of window_size and reads are page-cache aligned. Mismatched chunks halve the throughput because every sample touches two chunks.
Token count in attributes, not in the dataset. The trailing slice of the dataset may be partially full because the chunk size does not divide the document boundary. Store the real token_count as an HDF5 attribute on the dataset and have the reader truncate at that value. Without this the reader walks off the end into zero-padded tokens and the model learns to predict zero.
Sharded sha256 with parallel verification. Each shard has its own sha256 over the token bytes. The trainer can verify all shards in parallel before training starts. A wrong sha256 fails the run early, not on epoch three after sixteen hours.
swmr=True on both sides, with libver="latest" on the writer. Single-Writer-Multiple-Reader mode requires the writer to open with libver="latest"create every dataset up front, then set file.swmr_mode = True. After that the writer must call dataset.flush() after each resize so reader workers (opened with swmr=True) see consistent data. Skipping libver="latest" or enabling SWMR after structural changes is a common source of "file is locked" failures.
Use It
Production patterns:
- One HDF5 per source shard. The downloader (lesson 42) emits one shard per URL; tokenization (this lesson) emits one HDF5 per source shard. The 1:1 mapping makes resume and partial-failure recovery trivial.
- Boundary token id. The boundary token is part of the tokenizer vocab and is the only token the dataloader injects. The training loss masks the boundary token if the model is supposed to ignore it; otherwise it learns to use it as a sequence separator.
shards.jsonas the source of truth. Adding a new shard means writing the HDF5, computing its sha256, and appending an entry. The trainer reads the file once at startup and never touches the directory listing.
Ship It
outputs/skill-hdf5-tokenized-corpus.md would, on a real project, describe which tokenizer feeds the pipeline, what chunk size matches the trainer's window, where shards.json lives in version control, and how dataloader workers are sharded across files. This lesson ships the engine.
Exercises
- Add a
--compression gzipflag to the HDF5 writer and measure the throughput cost on the demo corpus. Defend the chosen default. - Add a deterministic seed to the sliding-window dataloader and verify two runs with the same seed produce identical batches.
- Add a
--validatemode that reads every shard, recomputes the sha256 over its tokens, and compares againstshards.json. CI should run this before training starts. - Compare the dataloader throughput at chunk sizes equal to, half of, and twice the window size. Report the page-cache effect.
- Add a
--max-document-tokensflag that truncates very long documents at write time. Defend the trade-off against deciding at read time.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Resizable dataset | "Append-only" | An HDF5 dataset with maxshape=(None,) that grows via resize calls in chunk-sized strides |
| Chunked layout | "How HDF5 stores it" | Fixed-size on-disk pages that the kernel can memory-map and the dataloader can read contiguously |
swmr mode | "Read-while-write" | Single-Writer-Multiple-Reader mode that lets dataloader workers share the file safely |
| Shard index | "shards.json" | The durable index of all token shards with offsets and content hashes |
| Sliding window | "Training sample" | A fixed-length slice of the global token stream that the trainer pairs with its shift-by-one target |
Further Reading
- HDF5 chunking documentation - the chunked, resizable dataset layout this lesson uses
- h5py user guide - Python bindings for HDF5
- NumPy memory mapping - the read-side primitive HDF5 exposes through h5py
- Phase 19 · 42 - the downloader whose output this lesson tokenizes
- Phase 19 · 44 - the cosine schedule that consumes this dataloader
- Phase 19 · 45 - the AMP loop that wraps the training step
Check what you learned
6 questions. Answer each one to see why.
Why store the tokenized corpus in a resizable, chunked HDF5 dataset rather than JSONL?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Streaming tokenization into resizable, sharded HDF5 datasets with mmap reads.
Implements:
- A byte-level deterministic Tokenizer.
- An HDF5ShardWriter that buffers tokens to chunk size and resizes the dataset
in fixed-size strides, recording token_count and sha256 as dataset attributes.
- A ShardedTokenizationPipeline that emits one HDF5 per source shard and writes
a shards.json index.
- An MmapTokenStore that opens shard files in swmr mode for read access.
- A SlidingWindowDataloader that yields fixed-length (input, target) pairs.
The demo at the bottom builds an in-memory corpus, tokenizes into shards, opens
them via memory map, runs the dataloader for a few batches, and prints the
per-batch shape and a checksum. Run: python3 code/main.py
"""
from __future__ import annotations
import hashlib
import json
import random
import struct
import sys
import tempfile
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Iterable, Iterator
import numpy as np
try:
import h5py
except ImportError as exc:
raise SystemExit(
"h5py is required for this lesson. Install with: pip install h5py"
) from exc
DEFAULT_CHUNK_SIZE = 8192
DEFAULT_WINDOW_SIZE = 64
BOUNDARY_TOKEN_ID = 0
TOKEN_DTYPE = np.uint16
@dataclass
class ShardWriteResult:
"""Per-shard write outcome."""
shard_id: str
path: str
token_count: int
document_count: int
chunk_size: int
sha256: str
def to_dict(self) -> dict[str, object]:
return asdict(self)
@dataclass
class ShardIndexEntry:
"""Index row used by readers to locate a shard."""
shard_id: str
path: str
token_count: int
document_count: int
sha256: str
global_start: int
def to_dict(self) -> dict[str, object]:
return asdict(self)
class Tokenizer:
"""Byte-level deterministic tokenizer.
Vocabulary:
0 boundary token (separator injected by the dataloader)
1..256 raw byte tokens (offset by one so 0 is reserved)
Real tokenizers use BPE or SentencePiece; this implementation is enough to
drive the streaming-write story without pulling a third-party tokenizer.
"""
BOUNDARY_TOKEN = BOUNDARY_TOKEN_ID
BYTE_OFFSET = 1
def __init__(self) -> None:
self.vocab_size = 257
def encode(self, text: str) -> list[int]:
if not text:
return []
data = text.encode("utf-8")
return [self.BYTE_OFFSET + b for b in data]
def decode(self, ids: Iterable[int]) -> str:
byte_ids = [int(i) - self.BYTE_OFFSET for i in ids if int(i) >= self.BYTE_OFFSET]
return bytes(byte_ids).decode("utf-8", errors="replace")
class HDF5ShardWriter:
"""Stream tokens into a resizable HDF5 dataset with chunk-sized buffering.
Open in a `with` block to guarantee the residual buffer is flushed and the
closing attributes (token_count, sha256) are written.
"""
def __init__(
self,
path: Path,
chunk_size: int = DEFAULT_CHUNK_SIZE,
dataset_name: str = "tokens",
) -> None:
if chunk_size <= 0:
raise ValueError("chunk_size must be positive")
self.path = Path(path)
self.chunk_size = chunk_size
self.dataset_name = dataset_name
self._buffer: list[int] = []
self._token_count = 0
self._document_count = 0
self._hasher = hashlib.sha256()
self._file: h5py.File | None = None
self._dataset: h5py.Dataset | None = None
def __enter__(self) -> "HDF5ShardWriter":
self._file = h5py.File(self.path, "w", libver="latest")
self._dataset = self._file.create_dataset(
self.dataset_name,
shape=(0,),
maxshape=(None,),
chunks=(self.chunk_size,),
dtype=TOKEN_DTYPE,
)
self._file.swmr_mode = True
return self
def __exit__(self, exc_type, exc, tb) -> None:
try:
if self._dataset is not None and self._file is not None:
if self._buffer:
self._flush_buffer(final=True)
self._dataset.attrs["token_count"] = self._token_count
self._dataset.attrs["document_count"] = self._document_count
self._dataset.attrs["sha256"] = self._hasher.hexdigest()
finally:
if self._file is not None:
self._file.close()
self._file = None
self._dataset = None
def add_document(self, token_ids: Iterable[int]) -> None:
self._document_count += 1
for token in token_ids:
self._buffer.append(int(token))
if len(self._buffer) >= self.chunk_size:
self._flush_buffer(final=False)
def add_boundary(self) -> None:
"""Inject the separator token between documents."""
self._buffer.append(BOUNDARY_TOKEN_ID)
if len(self._buffer) >= self.chunk_size:
self._flush_buffer(final=False)
def _flush_buffer(self, final: bool) -> None:
if self._dataset is None:
raise RuntimeError("writer is not open")
if not self._buffer:
return
size = len(self._buffer) if final else self.chunk_size
chunk = np.asarray(self._buffer[:size], dtype=TOKEN_DTYPE)
new_total = self._token_count + size
self._dataset.resize((new_total,))
self._dataset[self._token_count : new_total] = chunk
self._dataset.flush()
self._hasher.update(chunk.tobytes())
self._token_count = new_total
self._buffer = self._buffer[size:]
if not final and len(self._buffer) >= self.chunk_size:
self._flush_buffer(final=False)
@property
def token_count(self) -> int:
return self._token_count
@property
def document_count(self) -> int:
return self._document_count
def result(self, shard_id: str) -> ShardWriteResult:
return ShardWriteResult(
shard_id=shard_id,
path=str(self.path),
token_count=self._token_count,
document_count=self._document_count,
chunk_size=self.chunk_size,
sha256=self._hasher.hexdigest(),
)
class ShardedTokenizationPipeline:
"""Tokenize iterable shard inputs into HDF5 files and write a shards.json."""
def __init__(
self,
tokenizer: Tokenizer,
output_dir: Path,
chunk_size: int = DEFAULT_CHUNK_SIZE,
) -> None:
self.tokenizer = tokenizer
self.output_dir = Path(output_dir)
self.output_dir.mkdir(parents=True, exist_ok=True)
self.chunk_size = chunk_size
def write_shard(self, shard_id: str, documents: Iterable[str]) -> ShardWriteResult:
shard_path = self.output_dir / f"{shard_id}.h5"
writer = HDF5ShardWriter(shard_path, chunk_size=self.chunk_size)
with writer:
for text in documents:
writer.add_document(self.tokenizer.encode(text))
writer.add_boundary()
return writer.result(shard_id)
def write_corpus(self, shards: dict[str, Iterable[str]]) -> list[ShardIndexEntry]:
entries: list[ShardIndexEntry] = []
running_offset = 0
for shard_id, documents in shards.items():
result = self.write_shard(shard_id, documents)
entries.append(
ShardIndexEntry(
shard_id=result.shard_id,
path=result.path,
token_count=result.token_count,
document_count=result.document_count,
sha256=result.sha256,
global_start=running_offset,
)
)
running_offset += result.token_count
index_path = self.output_dir / "shards.json"
body = {
"version": 1,
"chunk_size": self.chunk_size,
"total_tokens": running_offset,
"shards": [entry.to_dict() for entry in entries],
}
index_path.write_text(json.dumps(body, sort_keys=True, indent=2), encoding="utf-8")
return entries
class MmapTokenStore:
"""Memory-mapped read access to a sharded HDF5 token corpus.
The store opens each shard file once in SWMR mode. A request for
`get_slice(start, stop)` is routed across shards and the result is returned
as a flat NumPy uint16 array. Reads land in the page cache; the dataloader
pays one copy when it crosses into a training tensor.
"""
def __init__(self, shard_entries: list[ShardIndexEntry]) -> None:
if not shard_entries:
raise ValueError("at least one shard entry is required")
self._entries = shard_entries
self._files: list[h5py.File] = []
self._datasets: list[h5py.Dataset] = []
try:
for entry in shard_entries:
self._files.append(h5py.File(entry.path, "r", swmr=True))
self._datasets = [f["tokens"] for f in self._files]
except Exception:
for opened in self._files:
try:
opened.close()
except Exception:
pass
self._files = []
self._datasets = []
raise
self._total_tokens = sum(entry.token_count for entry in shard_entries)
@property
def total_tokens(self) -> int:
return self._total_tokens
def close(self) -> None:
for file in self._files:
try:
file.close()
except Exception:
pass
self._files = []
self._datasets = []
def __enter__(self) -> "MmapTokenStore":
return self
def __exit__(self, exc_type, exc, tb) -> None:
self.close()
def get_slice(self, start: int, stop: int) -> np.ndarray:
if start < 0 or stop < 0 or stop < start:
raise ValueError(f"bad slice: start={start} stop={stop}")
if stop > self._total_tokens:
raise ValueError(f"stop ({stop}) exceeds total tokens ({self._total_tokens})")
if stop == start:
return np.empty((0,), dtype=TOKEN_DTYPE)
out = np.empty((stop - start,), dtype=TOKEN_DTYPE)
cursor = 0
for entry, dataset in zip(self._entries, self._datasets):
shard_start = entry.global_start
shard_stop = shard_start + entry.token_count
if stop <= shard_start:
break
if start >= shard_stop:
continue
local_start = max(0, start - shard_start)
local_stop = min(entry.token_count, stop - shard_start)
length = local_stop - local_start
if length <= 0:
continue
out[cursor : cursor + length] = dataset[local_start:local_stop]
cursor += length
if cursor != stop - start:
raise RuntimeError(
f"slice read produced {cursor} tokens, expected {stop - start}"
)
return out
class SlidingWindowDataloader:
"""Random sliding-window sampler over a flat token stream."""
def __init__(
self,
store: MmapTokenStore,
window_size: int = DEFAULT_WINDOW_SIZE,
batch_size: int = 4,
seed: int = 0,
) -> None:
if window_size <= 1:
raise ValueError("window_size must be greater than 1")
if batch_size <= 0:
raise ValueError("batch_size must be positive")
if store.total_tokens <= window_size:
raise ValueError(
f"store has only {store.total_tokens} tokens; need more than {window_size}"
)
self.store = store
self.window_size = window_size
self.batch_size = batch_size
self._random = random.Random(seed)
self._max_start = store.total_tokens - window_size - 1
def _sample_window(self) -> tuple[np.ndarray, np.ndarray]:
start = self._random.randint(0, self._max_start)
chunk = self.store.get_slice(start, start + self.window_size + 1)
return chunk[:-1], chunk[1:]
def __iter__(self) -> Iterator[tuple[np.ndarray, np.ndarray]]:
while True:
inputs = np.empty((self.batch_size, self.window_size), dtype=TOKEN_DTYPE)
targets = np.empty((self.batch_size, self.window_size), dtype=TOKEN_DTYPE)
for row in range(self.batch_size):
inputs[row], targets[row] = self._sample_window()
yield inputs, targets
def take(self, num_batches: int) -> list[tuple[np.ndarray, np.ndarray]]:
iterator = iter(self)
return [next(iterator) for _ in range(num_batches)]
class JSONLSource:
"""Adapter that yields documents from a JSONL file with a configurable key.
The downloader (Phase 19 · 42) emits JSONL where each line is a JSON object
with a `text` field. This adapter pulls the text out and skips lines that
are malformed or missing the field. Real pipelines log the dropped lines;
this adapter counts them so callers can audit dropout rate.
"""
def __init__(self, path: Path, text_field: str = "text") -> None:
self.path = Path(path)
self.text_field = text_field
self.dropped_lines = 0
def __iter__(self) -> Iterator[str]:
with self.path.open("r", encoding="utf-8") as fh:
for line in fh:
line = line.rstrip("\n")
if not line:
continue
try:
record = json.loads(line)
except json.JSONDecodeError:
self.dropped_lines += 1
continue
if not isinstance(record, dict):
self.dropped_lines += 1
continue
value = record.get(self.text_field)
if not isinstance(value, str) or not value:
self.dropped_lines += 1
continue
yield value
def pack_documents(
tokenizer: Tokenizer,
documents: Iterable[str],
max_tokens: int,
) -> Iterator[list[int]]:
"""Pack tokenized documents into fixed-length groups with boundary tokens.
Yields lists of exactly max_tokens token ids. Long documents are split
across groups; short documents share a group separated by BOUNDARY_TOKEN_ID.
The final group may be shorter than max_tokens and is yielded as-is.
"""
if max_tokens <= 1:
raise ValueError("max_tokens must be greater than 1")
buffer: list[int] = []
for text in documents:
token_ids = tokenizer.encode(text)
if buffer:
buffer.append(BOUNDARY_TOKEN_ID)
buffer.extend(token_ids)
while len(buffer) >= max_tokens:
yield buffer[:max_tokens]
buffer = buffer[max_tokens:]
if buffer:
yield buffer
def tokenize_jsonl_path(
jsonl_path: Path,
output_dir: Path,
shard_id: str,
chunk_size: int = DEFAULT_CHUNK_SIZE,
text_field: str = "text",
) -> ShardWriteResult:
"""Convenience wrapper: tokenize one JSONL file into one HDF5 shard."""
tokenizer = Tokenizer()
pipeline = ShardedTokenizationPipeline(tokenizer, output_dir=output_dir, chunk_size=chunk_size)
source = JSONLSource(jsonl_path, text_field=text_field)
return pipeline.write_shard(shard_id, source)
def load_index(index_path: Path) -> list[ShardIndexEntry]:
"""Read shards.json and return ShardIndexEntry rows."""
data = json.loads(Path(index_path).read_text("utf-8"))
entries: list[ShardIndexEntry] = []
for row in data["shards"]:
entries.append(
ShardIndexEntry(
shard_id=str(row["shard_id"]),
path=str(row["path"]),
token_count=int(row["token_count"]),
document_count=int(row.get("document_count", 0)),
sha256=str(row["sha256"]),
global_start=int(row["global_start"]),
)
)
return entries
def validate_corpus(index_entries: list[ShardIndexEntry]) -> list[str]:
"""Recompute each shard's sha256 over its on-disk tokens and report mismatches."""
failures: list[str] = []
for entry in index_entries:
with h5py.File(entry.path, "r", swmr=True) as fh:
dataset = fh["tokens"]
recorded_count = int(dataset.attrs.get("token_count", entry.token_count))
tokens = np.asarray(dataset[:recorded_count], dtype=TOKEN_DTYPE)
recomputed = hashlib.sha256(tokens.tobytes()).hexdigest()
if recomputed != entry.sha256:
failures.append(entry.shard_id)
return failures
def build_demo_corpus() -> dict[str, list[str]]:
"""Two shards of synthetic documents long enough to exercise mmap reads."""
base = [
"the alignment problem is a story about reward functions and the things they fail to write down",
"attention scales better with sequence length so transformers replaced recurrent networks during the language modeling era",
"an evaluation harness keeps training honest by treating the test corpus as a contract that cannot drift",
"deduplication is upstream of tokenization because every duplicate token costs the trainer twice in compute",
"checkpoints record the optimizer state and the random seed so that a restart resumes exactly where it stopped",
]
long_repeat = " ".join(base * 4)
shards: dict[str, list[str]] = {
"shard-0000": [long_repeat, long_repeat, long_repeat],
"shard-0001": [long_repeat, long_repeat, long_repeat],
}
return shards
def run_demo() -> int:
"""Build a demo corpus, tokenize it, validate it, and run the dataloader.
Designed to be self-terminating: the pipeline writes into a temporary
directory and the dataloader takes a small fixed number of batches so the
script exits without external input.
"""
with tempfile.TemporaryDirectory() as tmp:
out = Path(tmp)
tokenizer = Tokenizer()
pipeline = ShardedTokenizationPipeline(tokenizer, output_dir=out, chunk_size=512)
shards = build_demo_corpus()
entries = pipeline.write_corpus(shards)
for entry in entries:
print(
f"[shard] {entry.shard_id} tokens={entry.token_count} "
f"sha256={entry.sha256[:12]} global_start={entry.global_start}"
)
validation_failures = validate_corpus(entries)
if validation_failures:
print(f"[validate] failed: {validation_failures}")
return 1
print(f"[validate] all {len(entries)} shards match recorded sha256")
with MmapTokenStore(entries) as store:
loader = SlidingWindowDataloader(store, window_size=64, batch_size=4, seed=7)
for batch_index, (inputs, targets) in enumerate(loader.take(10)):
checksum = int(hashlib.blake2b(inputs.tobytes(), digest_size=4).hexdigest(), 16)
print(
f"[batch] step={batch_index} shape={tuple(inputs.shape)} "
f"checksum={checksum:08x}"
)
return 0
if __name__ == "__main__":
sys.exit(run_demo())