Get Started

Tokenized Dataset with Sliding Window

A pretraining run is a function from token ids to gradients. This lesson builds the conveyor that feeds the ids in.

90 minutesBuildPythonFree lesson

Learning Objectives

  • Convert a raw corpus into a stream of token ids by calling the tokenizer once.
  • Slice the id stream into fixed-length windows with a configurable overlap stride.
  • Build a PyTorch Dataset that returns input and target tensors for next-token prediction.
  • Wrap the dataset in a DataLoader with a deterministic shuffle seeded per epoch.
  • Reason about the trade-off between stride, redundancy, and effective dataset size.

The frame

A pretraining run reads one batch of token ids at a time and updates the model. The shape of each batch is fixed by the training contract. For a causal language model, the batch holds (B, T) input ids and (B, T) target ids where the target is the input shifted left by one. The job of the data pipeline is to produce that contract on demand, in a deterministic and reproducible way, from a corpus that may be several gigabytes of raw text.

This lesson builds the pipeline. The tokenizer from the previous lesson turns text into a long flat list of ids. A sliding window slices that list into training examples. A custom Dataset exposes the examples as tensors. A DataLoader batches them and shuffles them with a known seed.

The shape contract

A causal LM consumes ids of shape (B, T) where B is the batch size and T is the context length. The target at position t is the input at position t+1. That means every training example covers T+1 raw ids. The window stride controls how much overlap exists between consecutive examples.

The slicer never overlaps with the boundary of the corpus. If the last window does not have enough ids to fill T+1 positions, the slicer drops it. Padding the tail with <|pad|> is also a valid choice but it complicates the loss mask. For this lesson we drop.

Why a sliding window

A pretraining corpus is one long stream of ids. If the model only saw non-overlapping windows, every training example would teach it the same T boundaries. Adjusting the stride moves those boundaries around so the model sees more diverse predict-next-token tasks.

A stride of T produces non-overlapping windows. A stride of T // 2 produces fifty-percent overlap and doubles the effective dataset. A stride of 1 produces maximum overlap and increases the dataset by a factor of T. The cost is more compute per epoch. The benefit is more boundary diversity. Most pretraining runs use a stride equal to the context length because the corpus is already much larger than the model can finish in one epoch, so the boundary diversity argument is weaker.

The Dataset class

A PyTorch Dataset has two required methods. __len__ returns the number of examples. __getitem__ returns one example as a pair of tensors. Our Dataset stores the encoded id stream and the stride. Indexing into it computes the start of the window on the fly so the memory cost is one copy of the id stream regardless of how many examples the stride produces.

The shift-by-one happens inside __getitem__. The Dataset returns (input, target) where input = window[:-1] and target = window[1:]. Both are PyTorch long tensors. The training loop treats them as ground truth.

Deterministic shuffle

A DataLoader with shuffle=True reads from a PyTorch random generator. By passing an explicit torch.Generator seeded per epoch, we get the same shuffle every time the run is restarted. That property matters when you want to compare two runs that differ only in a single hyperparameter. Without a seed, two runs see the data in different orders and the loss curves diverge for reasons unrelated to the change.

The seed contract in this lesson is simple. epoch_seed = base_seed + epoch_index. The base seed is passed at construction. The epoch index is incremented by the trainer at the top of each epoch. A re-run with the same base seed always sees the same order in every epoch.

Batch sampler

The default sampler in PyTorch picks indices uniformly at random with replacement disabled. That is what we want for pretraining. For finetuning on a small dataset the contract is the same. The DataLoader assembles a batch by calling __getitem__ B times and stacking the results. Because every example is the same length by construction, no padding logic is needed.

The lesson keeps num_workers=0 for simplicity. In a production run the workers parallelize the __getitem__ calls. With our pipeline that is mostly a no-op because the work is just a slice of an in-memory tensor, but the same Dataset API supports workers cleanly.

Counting examples

For an id stream of length Na context length Tand a stride Sthe number of examples is max(0, 1 + (N - (T + 1)) // S). The lesson exposes that calculation as a static method on the Dataset so the trainer can compute total steps per epoch without iterating.

What this lesson does not do

It does not stream from disk. The corpus is encoded fully in memory and held as a single tensor. For a corpus of a few million ids that is well under a hundred megabytes and is the right shape for the lesson. Disk streaming is a separate concern that plugs in by replacing the storage but keeps the Dataset contract.

It does not handle multiple documents. The corpus is treated as one continuous id stream. The next-document boundary is encoded by inserting <|endoftext|> ids when the corpus is built from multiple documents. The model learns to predict around the boundary.

How to read the code

main.py defines two classes and one helper. SlidingWindowDataset is the PyTorch Dataset. make_dataloader returns a configured DataLoader with a seeded generator. _encode_corpus_to_ids is the one-shot tokenizer call. The demo at the bottom builds a small tokenizer in-process, encodes a built-in corpus, constructs the dataset and dataloader, prints one batch, and asserts the shape contract. The tests in code/tests/test_dataset.py pin the window count formula, the shift-by-one property, the deterministic shuffle, and the stride trade-off.

Run the demo. Then change the context length from 16 to 32 and watch how the number of examples per epoch falls. That number is your steps-per-epoch budget.

Check what you learned

6 questions. Answer each one to see why.

Question 1 of 60 correct

What is the shape contract for a causal LM training batch?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Tokenized dataset with sliding window for next-token training.

Wraps a tokenizer-encoded id stream in a PyTorch Dataset and DataLoader so a
training loop can pull (B, T) input and (B, T) target batches.

The tokenizer is the small byte-level BPE from lesson 30, inlined here so
this lesson runs without inter-lesson imports.

Run: python3 code/main.py
"""

from __future__ import annotations

import re
from collections import Counter
from dataclasses import dataclass, field
from typing import Iterable

import torch
from torch.utils.data import DataLoader, Dataset


BYTE_ALPHABET_SIZE = 256
DEFAULT_SPECIALS = ("<|endoftext|>", "<|pad|>")
WORD_SPLIT_RE = re.compile(r"\S+|\s+")


@dataclass
class MiniBPE:
    """Inline byte-level BPE tokenizer (same contract as lesson 30)."""

    vocab: dict[int, bytes] = field(default_factory=dict)
    inv_vocab: dict[bytes, int] = field(default_factory=dict)
    merges: dict[tuple[int, int], int] = field(default_factory=dict)
    special_to_id: dict[str, int] = field(default_factory=dict)
    id_to_special: dict[int, str] = field(default_factory=dict)

    @property
    def vocab_size(self) -> int:
        return len(self.vocab)

    def initialize(self, specials: Iterable[str] = DEFAULT_SPECIALS) -> None:
        self.vocab.clear()
        self.inv_vocab.clear()
        self.merges.clear()
        self.special_to_id.clear()
        self.id_to_special.clear()
        for i in range(BYTE_ALPHABET_SIZE):
            self.vocab[i] = bytes([i])
            self.inv_vocab[bytes([i])] = i
        for s in specials:
            token_id = len(self.vocab)
            self.vocab[token_id] = s.encode("utf-8")
            self.inv_vocab[s.encode("utf-8")] = token_id
            self.special_to_id[s] = token_id
            self.id_to_special[token_id] = s


def _pretokenize(text: str) -> list[str]:
    return WORD_SPLIT_RE.findall(text)


def _count_pairs(units: dict[tuple[int, ...], int]) -> Counter:
    pairs: Counter = Counter()
    for symbols, count in units.items():
        for i in range(len(symbols) - 1):
            pairs[(symbols[i], symbols[i + 1])] += count
    return pairs


def _apply_merge_to_corpus(
    units: dict[tuple[int, ...], int],
    pair: tuple[int, int],
    new_id: int,
) -> dict[tuple[int, ...], int]:
    new_units: dict[tuple[int, ...], int] = {}
    for symbols, count in units.items():
        if len(symbols) < 2:
            new_units[symbols] = new_units.get(symbols, 0) + count
            continue
        out: list[int] = []
        i = 0
        a, b = pair
        while i < len(symbols):
            if i < len(symbols) - 1 and symbols[i] == a and symbols[i + 1] == b:
                out.append(new_id)
                i += 2
            else:
                out.append(symbols[i])
                i += 1
        merged = tuple(out)
        new_units[merged] = new_units.get(merged, 0) + count
    return new_units


def train_bpe(tokenizer: MiniBPE, corpus: str, target_vocab_size: int) -> None:
    min_vocab_size = BYTE_ALPHABET_SIZE + len(DEFAULT_SPECIALS)
    if target_vocab_size < min_vocab_size:
        raise ValueError(
            f"target_vocab_size must be >= {min_vocab_size}, got {target_vocab_size}"
        )
    tokenizer.initialize(DEFAULT_SPECIALS)
    chunks = _pretokenize(corpus)
    units: dict[tuple[int, ...], int] = {}
    for chunk in chunks:
        symbols = tuple(chunk.encode("utf-8"))
        units[symbols] = units.get(symbols, 0) + 1
    while tokenizer.vocab_size < target_vocab_size:
        pairs = _count_pairs(units)
        if not pairs:
            break
        max_count = max(pairs.values())
        candidates = sorted(p for p, c in pairs.items() if c == max_count)
        best = candidates[0]
        if pairs[best] < 2:
            break
        new_id = len(tokenizer.vocab)
        merged_bytes = tokenizer.vocab[best[0]] + tokenizer.vocab[best[1]]
        tokenizer.vocab[new_id] = merged_bytes
        tokenizer.inv_vocab[merged_bytes] = new_id
        tokenizer.merges[best] = new_id
        units = _apply_merge_to_corpus(units, best, new_id)


def encode_text(tokenizer: MiniBPE, text: str) -> list[int]:
    ranked = {pair: rank for rank, pair in enumerate(tokenizer.merges.keys())}
    out: list[int] = []
    for chunk in _pretokenize(text):
        symbols: list[int] = list(chunk.encode("utf-8"))
        while len(symbols) >= 2:
            best_rank = None
            best_index = -1
            best_pair: tuple[int, int] | None = None
            for i in range(len(symbols) - 1):
                pair = (symbols[i], symbols[i + 1])
                rank = ranked.get(pair)
                if rank is None:
                    continue
                if best_rank is None or rank < best_rank:
                    best_rank = rank
                    best_index = i
                    best_pair = pair
            if best_pair is None:
                break
            new_id = tokenizer.merges[best_pair]
            symbols = symbols[:best_index] + [new_id] + symbols[best_index + 2:]
        out.extend(symbols)
    return out


class SlidingWindowDataset(Dataset):
    """PyTorch Dataset over a flat id stream.

    Each example is a window of size T+1. __getitem__ returns
    (input_ids, target_ids) where target = input shifted left by one.
    """

    def __init__(
        self,
        ids: list[int],
        context_length: int,
        stride: int | None = None,
    ) -> None:
        if context_length < 1:
            raise ValueError(f"context_length must be >= 1, got {context_length}")
        if not ids:
            raise ValueError("ids must be non-empty")
        if stride is None:
            stride = context_length
        if stride < 1:
            raise ValueError(f"stride must be >= 1, got {stride}")
        self.ids = torch.tensor(ids, dtype=torch.long)
        self.context_length = context_length
        self.stride = stride

    @staticmethod
    def count_windows(num_ids: int, context_length: int, stride: int) -> int:
        usable = num_ids - (context_length + 1)
        if usable < 0:
            return 0
        return 1 + usable // stride

    def __len__(self) -> int:
        return self.count_windows(self.ids.numel(), self.context_length, self.stride)

    def __getitem__(self, index: int) -> tuple[torch.Tensor, torch.Tensor]:
        if index < 0:
            index += len(self)
        if index < 0 or index >= len(self):
            raise IndexError(f"window index {index} out of range")
        start = index * self.stride
        end = start + self.context_length + 1
        window = self.ids[start:end]
        return window[:-1].clone(), window[1:].clone()


def make_dataloader(
    dataset: SlidingWindowDataset,
    batch_size: int,
    shuffle: bool = True,
    base_seed: int = 0,
    epoch: int = 0,
    drop_last: bool = True,
) -> DataLoader:
    """Build a DataLoader with a deterministic per-epoch shuffle."""
    generator = torch.Generator()
    generator.manual_seed(base_seed + epoch)
    return DataLoader(
        dataset,
        batch_size=batch_size,
        shuffle=shuffle,
        drop_last=drop_last,
        generator=generator if shuffle else None,
        num_workers=0,
    )


def _encode_corpus_to_ids(tokenizer: MiniBPE, corpus: str, target_vocab: int) -> list[int]:
    train_bpe(tokenizer, corpus, target_vocab_size=target_vocab)
    return encode_text(tokenizer, corpus)


DEMO_CORPUS = """\
the quick brown fox jumps over the lazy dog
a journey of a thousand miles begins with a single step
the only way to do great work is to love what you do
the best time to plant a tree was twenty years ago
the second best time is now
practice is the bridge between intention and skill
small daily actions compound into large outcomes
read more than you write write more than you talk
the map is not the territory and the menu is not the meal
what gets measured gets managed if the measurement is honest
the quick brown fox runs across the meadow at dawn
a small step today is better than a perfect plan tomorrow
courage is not the absence of fear it is action despite fear
the lazy dog sleeps under the old oak tree
every expert was once a beginner who refused to quit
focus is saying no to a hundred good ideas
the river that you cannot cross today will be easier tomorrow
practice the basics until the basics become invisible
""" * 8


def _print_section(title: str) -> None:
    bar = "-" * len(title)
    print(f"\n{title}\n{bar}")


def main() -> int:
    target_vocab = 320
    context_length = 16
    stride = 8
    batch_size = 4
    base_seed = 7

    tokenizer = MiniBPE()
    ids = _encode_corpus_to_ids(tokenizer, DEMO_CORPUS, target_vocab)

    _print_section("Corpus and tokenizer")
    print(f"corpus chars      : {len(DEMO_CORPUS)}")
    print(f"vocab size        : {tokenizer.vocab_size}")
    print(f"total ids         : {len(ids)}")

    dataset = SlidingWindowDataset(ids, context_length=context_length, stride=stride)
    print(f"context length    : {context_length}")
    print(f"stride            : {stride}")
    print(f"num windows       : {len(dataset)}")
    expected = SlidingWindowDataset.count_windows(len(ids), context_length, stride)
    assert len(dataset) == expected, "len(dataset) must equal count_windows"

    _print_section("Inspect one example")
    input_ids, target_ids = dataset[0]
    print(f"input shape       : {tuple(input_ids.shape)}")
    print(f"target shape      : {tuple(target_ids.shape)}")
    assert input_ids.shape == target_ids.shape, "shapes must match"
    assert torch.equal(input_ids[1:], target_ids[:-1]), "target must be input shifted by one"

    _print_section("Pull a batch from the DataLoader")
    loader = make_dataloader(dataset, batch_size=batch_size, base_seed=base_seed, epoch=0)
    inputs, targets = next(iter(loader))
    print(f"inputs            : {tuple(inputs.shape)}")
    print(f"targets           : {tuple(targets.shape)}")
    print(f"first input row   : {inputs[0].tolist()}")
    print(f"first target row  : {targets[0].tolist()}")
    assert inputs.shape == (batch_size, context_length)
    assert targets.shape == (batch_size, context_length)

    _print_section("Shuffle is seeded")
    loader_a = make_dataloader(dataset, batch_size=batch_size, base_seed=base_seed, epoch=0)
    loader_b = make_dataloader(dataset, batch_size=batch_size, base_seed=base_seed, epoch=0)
    batch_a = next(iter(loader_a))
    batch_b = next(iter(loader_b))
    assert torch.equal(batch_a[0], batch_b[0]), "same seed must produce same first batch"
    print("same seed -> same first batch: OK")

    loader_c = make_dataloader(dataset, batch_size=batch_size, base_seed=base_seed, epoch=1)
    batch_c = next(iter(loader_c))
    assert not torch.equal(batch_a[0], batch_c[0]), "different epoch must change order"
    print("different epoch -> different order: OK")

    _print_section("Stride trade-off")
    for s in (4, 8, 16):
        ds = SlidingWindowDataset(ids, context_length=context_length, stride=s)
        print(f"  stride {s:>2}: {len(ds):>4} windows")

    print("\nDemo OK.")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())