From ca4f6d09b8ba449bdb1b3a1e651f074c168dbab4 Mon Sep 17 00:00:00 2001 From: Varun Nuthalapati Date: Tue, 14 Apr 2026 08:18:48 -0700 Subject: [PATCH] docs(pretrain): add TinyStories pretraining section Add a dedicated 'Pretrain on TinyStories' section to tutorials/pretrain.md explaining how to use the TinyStories data module for quick end-to-end pretraining experiments. Covers both the debug.yaml config path and manual CLI configuration. Also fix a typo: 'scenarioes' -> 'scenarios'. Closes #1082 --- tutorials/pretrain.md | 51 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 50 insertions(+), 1 deletion(-) diff --git a/tutorials/pretrain.md b/tutorials/pretrain.md index a02c74b932..2dbc920917 100644 --- a/tutorials/pretrain.md +++ b/tutorials/pretrain.md @@ -97,7 +97,7 @@ litgpt pretrain pythia-14m \ Often, it makes sense to adopt an existing pretrained model and further pretrain it on our own custom data. The existing pretrained model can be either our own pretrained model or a model downloaded from a model hub. -The following subsections illustrate three typical scenarioes: +The following subsections illustrate three typical scenarios: 1. Starting from a downloaded base model 2. Continuing the pretraining after interruption @@ -179,6 +179,55 @@ litgpt pretrain pythia-160m \ ``` +  +## Pretrain on TinyStories + +[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) is a small dataset of short stories written in simple English. At ~2 GB it is ideal for quickly testing a pretraining setup end-to-end without requiring large compute resources. + +  + +> [!TIP] +> TinyStories is a great starting point if you want to verify your training pipeline before committing to a full-scale run on a larger dataset. + +  + +**Step 1 – Download the tokenizer** + +```bash +litgpt download EleutherAI/pythia-14m \ + --tokenizer_only true +``` + +**Step 2 – Pretrain using the built-in `debug.yaml` config** + +The [debug.yaml](https://github.com/Lightning-AI/litgpt/blob/main/config_hub/pretrain/debug.yaml) config pretrains a small `pythia-14m` model on TinyStories. Data preprocessing (download + tokenization) is handled automatically on the first run. + +```bash +litgpt pretrain pythia-14m \ + --config https://raw.githubusercontent.com/Lightning-AI/litgpt/main/config_hub/pretrain/debug.yaml +``` + +The config trains for 100 million tokens with a global batch size of 125 and a learning-rate warmup of 100 steps. Checkpoints are saved to `out/pretrain/debug/` by default. + +**Step 2 (alternative) – Configure manually** + +You can also specify the `TinyStories` data module directly without using the config file: + +```bash +litgpt pretrain pythia-14m \ + --tokenizer_dir checkpoints/EleutherAI/pythia-14m \ + --data TinyStories \ + --train.max_tokens 100_000_000 \ + --train.global_batch_size 125 \ + --train.micro_batch_size 5 +``` + +  +> [!TIP] +> Use `litgpt pretrain --data.help TinyStories` to list all available dataset options such as `data_path`, `seed`, and `num_workers`. +  + +   ## Pretrain a 1.1B TinyLlama model