Tag: tiny-gpt

Train a Tiny GPT, Part 5: Fine-Tuning

A 5.8M-parameter GPT scores 1.626 bits per byte on Holmes. Gemma 4 E4B scores 0.953 untrained, and a 34-minute QLoRA run on an 8 GB GPU gets it to 0.865.

Train a Tiny GPT, Part 4: Inference

The KV cache sped a 5.8M-parameter GPT up 4.4x on a laptop CPU and did almost nothing for the RTX 3070 at batch 1. Then sampling settings decide how it reads.

Train a Tiny GPT, Part 3: Training

Train a 5.8M-parameter GPT on Sherlock Holmes: 59 seconds on an RTX 3070 vs 48 minutes on its CPU, and the overfitting that starts near step 1,250 of training.

Train a Tiny GPT, Part 2: Attention

Build causal self-attention and RoPE in plain PyTorch, then measure naive vs fused attention on an 8GB GPU: 6400 MiB vs 8.3 MiB of VRAM at 16k tokens.

Train a Tiny GPT, Part 1: Tokenizers

Build a byte-level BPE tokenizer in about 100 lines of Python, then compare it to tiktoken's gpt2, cl100k_base and o200k_base on prose, code, YAML and German.