A command-line toolkit to extract text content and category data from Wikipedia dump files
English | 日本語
# Install
gem install wp2txt
# Extract text from English Wikipedia (auto-download)
wp2txt --lang=en -o ./output
# Extract specific articles
wp2txt --lang=en --articles="Tokyo,Kyoto" -o ./articles
# Extract articles from a category
wp2txt --lang=en --from-category="Cities in Japan" -o ./citiesWP2TXT extracts plain text and category information from Wikipedia dump files. It processes XML dumps (compressed with bzip2), removes MediaWiki markup, and outputs clean text suitable for corpus linguistics, text mining, and other research purposes.
- Auto-download - Automatically download dumps by language code
- Article extraction by title - Extract specific articles without downloading full dumps
- Category-based extraction - Extract all articles from a specific Wikipedia category
- Category metadata extraction - Preserves article category information in output
- Template expansion - Expands common templates (dates, units, coordinates) to readable text
- Multilingual support - Category and redirect detection for 350+ Wikipedia languages
- Streaming processing - Process large dumps without intermediate files
- JSON output - Machine-readable JSONL format for data pipelines
- Offline research indexes - Exhaustive, version-pinned queries over categories, section headings, and full text; interlanguage links for cross-edition SQL
- MCP server - Expose a local dump to LLM agents (Claude, ChatGPT, Gemini, local models) for grounded, reproducible corpus work
wp2txt is particularly suited for:
- Building domain-specific corpora using category information (e.g., "the plot sections of all film articles")
- Comparative linguistic research across topic areas
- Extracting Wikipedia text with metadata for NLP tasks
- Cross-linguistic studies using parallel category structures
- Version-pinned RAG knowledge bases and LLM evaluation datasets
- Exhaustive claims about a Wikipedia edition ("no article in this category mentions X") that web search cannot make
wp2txt uses official Wikipedia dump files, the recommended method for bulk data access. This approach respects Wikimedia's infrastructure guidelines.
$ gem install wp2txt
WP2TXT requires one of the following commands to decompress bz2 files:
lbzip2(recommended - uses multiple CPU cores)pbzip2bzip2(pre-installed on most systems)
On macOS with Homebrew:
$ brew install lbzip2
On Windows: Install Bzip2 for Windows and add to PATH.
The Docker image bundles everything — Ruby, decompression tools, and the MCP server — so no local Ruby setup is needed.
Interactive / batch CLI:
docker run -it -v /path/to/localdata:/data ghcr.io/yohasebe/wp2txtThe wp2txt command is available inside the container. Use /data for input/output files.
Images are published to GitHub Container Registry (ghcr.io/yohasebe/wp2txt); Docker Hub (yohasebe/wp2txt) is maintained as a mirror.
MCP server (no Ruby required on the host):
# Build the indexes once (cached in a named volume so they persist)
docker run -it -v wp2txt:/root/.wp2txt ghcr.io/yohasebe/wp2txt wp2txt --build-index --fulltext -L ja
# Register with an MCP client, e.g. Claude Code
claude mcp add wp2txt -- docker run -i --rm -v wp2txt:/root/.wp2txt ghcr.io/yohasebe/wp2txt wp2txt-mcp -L jaNote: use -i (not -it) when running the MCP server — a TTY would corrupt the JSON-RPC stream. The named volume (wp2txt:) holds downloaded dumps and indexes; without it they are lost when the container exits.
$ wp2txt --lang=en -o ./text
This automatically downloads the English Wikipedia dump and extracts plain text. Downloads are cached in ~/.wp2txt/cache/.
$ wp2txt --lang=en --articles="Cognitive linguistics,Generative grammar" -o ./articles
Only the index file and necessary data streams are downloaded, making it much faster than processing the full dump.
$ wp2txt --lang=en --from-category="Cities in Japan" -o ./cities
Include subcategories with --depth:
$ wp2txt --lang=en --from-category="Cities in Japan" --depth=2 -o ./cities
Preview without downloading (shows article counts):
$ wp2txt --lang=en --from-category="Cities in Japan" --dry-run
$ wp2txt -i ./enwiki-20220801-pages-articles.xml.bz2 -o ./text
# Category info only (title + categories)
$ wp2txt -g --lang=en -o ./category
# Summary only (title + categories + opening paragraphs)
$ wp2txt -s --lang=en -o ./summary
# Metadata only (title + section headings + categories)
$ wp2txt -M --lang=en --format json -o ./metadata
# Extract specific sections from particular articles (fast)
# Section names are case-insensitive; alias matching is enabled by default
$ wp2txt --lang=en --articles="Tokyo" --sections="summary,history,geography" --format json -o ./sections
# Extract specific sections from a category (moderate)
$ wp2txt --lang=en --from-category="Cities in Japan" --sections="summary,history" --format json -o ./sections
# Extract specific sections from full dump (slow - processes all articles)
$ wp2txt --lang=en --sections="summary,plot,reception" --format json -o ./sections
# Section heading statistics (useful for discovering section names before extraction)
$ wp2txt --lang=en --section-stats -o ./stats
# JSON/JSONL output
$ wp2txt --format json --lang=en -o ./json
[[Article Title]]
Article content goes here with sections and paragraphs...
CATEGORIES: Category1, Category2, Category3
Each line contains one JSON object:
{"title": "Article Title", "categories": ["Cat1", "Cat2"], "text": "...", "redirect": null}For redirect articles:
{"title": "NYC", "categories": [], "text": "", "redirect": "New York City"}$ wp2txt --cache-status # Show cache status
$ wp2txt --cache-clear # Clear all cache
$ wp2txt --cache-clear --lang=en # Clear cache for English only
$ wp2txt --update-cache # Force fresh download
When cache exceeds the expiry period (default: 30 days), wp2txt displays a warning but allows using cached data.
Special content is replaced with marker placeholders by default:
Inline markers (appear within sentences):
| Marker | Content Type |
|---|---|
[MATH] |
Mathematical formulas |
[CODE] |
Inline code |
[CHEM] |
Chemical formulas |
[IPA] |
IPA phonetic notation |
Block markers (standalone content):
| Marker | Content Type |
|---|---|
[CODEBLOCK] |
Source code blocks |
[TABLE] |
Wiki tables |
[INFOBOX] |
Information boxes |
[NAVBOX] |
Navigation boxes |
[GALLERY] |
Image galleries |
[REFERENCES] |
Reference lists |
[SCORE] |
Musical scores |
[TIMELINE] |
Timeline graphics |
[GRAPH] |
Graphs/charts |
[SIDEBAR] |
Sidebar templates |
[MAPFRAME] |
Interactive maps |
[IMAGEMAP] |
Clickable image maps |
Configure with --markers:
$ wp2txt --lang=en --markers=all -o ./text # All markers (default)
$ wp2txt --lang=en --markers=math,code -o ./text # Only MATH and CODE
Note: --markers=none is deprecated as removing special content can make surrounding text nonsensical.
Common MediaWiki templates are automatically expanded (enabled by default):
| Template | Output |
|---|---|
{{birth date|1990|5|15}} |
May 15, 1990 |
{{convert|100|km|mi}} |
100 km (62 mi) |
{{coord|35|41|N|139|41|E}} |
35°41′N 139°41′E |
{{lang|ja|日本語}} |
日本語 |
{{nihongo|Tokyo|東京|Tōkyō}} |
Tokyo (東京, Tōkyō) |
{{frac|1|2}} |
1/2 |
{{circa|1900}} |
c. 1900 |
Supported: date/age templates, unit conversion, coordinates, language tags, quotes, fractions, and more. Parser functions ({{#if:}}, {{#switch:}}) and magic words ({{PAGENAME}}, {{CURRENTYEAR}}) are also supported.
Disable with --no-expand-templates.
By default, citation templates are removed. Use --extract-citations to extract formatted citations:
$ wp2txt --lang=en --extract-citations -o ./text
Supported: {{cite book}}, {{cite web}}, {{cite news}}, {{cite journal}}, {{Citation}}, etc.
Beyond text extraction, wp2txt can turn a dump into a local, version-pinned research database: SQLite indexes over categories, section headings, redirects, and (optionally) the full article text, plus interlanguage links for cross-edition comparison — all queryable offline, exhaustively, and exposed to LLM agents via an MCP server.
$ wp2txt --build-index --fulltext --lang=ja # build the indexes
$ wp2txt --find-articles --in-category "映画作品" -D 3 --has-section "あらすじ" --lang=ja
$ wp2txt --search "タイムループ" --in-category "映画作品" -D 3 --lang=ja
$ wp2txt --import-langlinks -L ja --langlinks-langs en,de,fr,zh,ko
$ wp2txt-mcp --lang=ja # stdio MCP server for LLM agentsUnlike web/API access, these queries scan every article (a 0 matches result is a
verifiable absence claim for that dump version) and are reproducible: extractions record
the dump version and query in a .meta.json sidecar.
→ See the Research Infrastructure Guide for index building, exhaustive queries, full-text search, interlanguage links, cross-language SQL, the full MCP tool list, and design principles.
Usage: wp2txt [options]
Input source (one of --input or --lang required):
-i, --input=<s> Path to compressed file (bz2) or XML file
-L, --lang=<s> Wikipedia language code (e.g., ja, en, de)
-A, --articles=<s> Specific article titles (comma-separated)
-G, --from-category=<s> Extract articles from Wikipedia category
-D, --depth=<i> Subcategory recursion depth (default: 0)
-y, --yes Skip confirmation prompt
--dry-run Preview category extraction
-U, --update-cache Force refresh of cached files
Output options:
-o, --output-dir=<s> Output directory (default: current)
-j, --format=<s> Output format: text or json (default: text)
-f, --file-size=<i> Output file size in MB (default: 10, 0=single)
Cache management:
--cache-dir=<s> Cache directory (default: ~/.wp2txt/cache)
--cache-status Show cache status and exit
--cache-clear Clear cache and exit
Configuration:
--config-init Create default config (~/.wp2txt/config.yml)
--config-path=<s> Path to configuration file
Extraction modes (mutually exclusive):
-g, --category-only Extract only title and categories
-s, --summary-only Extract title, categories, and summary
-M, --metadata-only Extract only title, headings, and categories
Section extraction:
-S, --sections=<s> Extract specific sections (comma-separated, case-insensitive)
--section-output=<s> Output mode: structured or combined (default: structured)
--min-section-length=<i> Minimum section length in characters (default: 0)
--skip-empty Skip articles with no matching sections
--alias-file=<s> Custom section alias definitions file (YAML)
--no-section-aliases Disable section alias matching (exact match only)
--section-stats Collect and output section heading statistics (JSON)
--show-matched-sections Include matched_sections field in JSON output
Content filtering:
-a, --category, --no-category Show category info (default: true)
-t, --title, --no-title Keep page titles (default: true)
-d, --heading, --no-heading Keep section titles (default: true)
-l, --list Keep list items (default: false)
--table Keep wiki table content (default: false)
-p, --pre Keep preformatted text blocks (default: false)
-r, --ref Keep references as [ref]...[/ref] (default: false)
--multiline Keep multi-line templates (default: false)
-e, --redirect Show redirect destination (default: false)
-m, --marker, --no-marker Show list markers (default: true)
-k, --markers=<s> Content markers (default: all)
-C, --extract-citations Extract formatted citations
-E, --expand-templates Expand templates (default: true)
--no-expand-templates Disable template expansion
Performance:
-n, --num-procs=<i> Parallel processes (default: auto)
--no-turbo Disable turbo mode (saves disk space, slower)
-R, --ractor Use Ractor parallelism (Ruby 4.0+, streaming only)
-b, --bz2-gem Use bzip2-ruby gem instead of system command
Output control:
-q, --quiet Suppress progress output (errors only)
--no-color Disable colored output
Info:
-v, --version Print version
-h, --help Show help
--build-index Build the metadata index (add --fulltext for FTS)
--find-articles / --search Exhaustive offline queries (see docs/RESEARCH.md)
--import-langlinks Import interlanguage links (version-matched)
--fts-optimize Optimize an existing full-text index
See the Research Infrastructure Guide for details.
Create persistent settings with:
$ wp2txt --config-init
This creates ~/.wp2txt/config.yml:
cache:
dump_expiry_days: 30 # Days before dumps are stale (1-365)
category_expiry_days: 7 # Category cache expiry (1-90)
directory: ~/.wp2txt/cache
defaults:
format: text # Default output format
depth: 0 # Default subcategory depthCommand-line options override configuration file settings.
Benchmark results on MacBook Air M4 (7 parallel processes, turbo mode, excluding download time):
| Wikipedia | Dump Size | Articles | Processing Time | Output |
|---|---|---|---|---|
| Japanese | 4.37 GB | 1,485,937 | ~27 min | 463 files (4.5 GB) |
| English | 24.2 GB | ~6.8M | ~2 hours | 2,000 files (20 GB) |
Turbo mode (default) splits bz2 into XML chunks first, then processes in parallel. Use --no-turbo to save disk space at the cost of slower processing.
- Special content (math, code, etc.) is marked with placeholders by default.
- Some text may not be extracted correctly due to markup variations or language-specific formatting.
See CHANGELOG.md for detailed release notes.
v2.3.0 (July 2026): Interlanguage links import, cross-language SQL (multi-dump ATTACH), explicit-title extraction, SQL results to file with reproducibility sidecars, GHCR image publishing.
v2.2.0 (July 2026): Offline metadata index, FTS5 full-text search, MCP server, query_sql escape hatch, extraction jobs.
v2.1.0 (February 2026): SQLite caching, Ractor parallelism (Ruby 4.0+), template expansion, content markers, Docker image update.
v2.0.0 (January 2026): Auto-download mode, category-based extraction, article extraction by title, JSON output, streaming processing, Ruby 4.0 support.
- Yoichiro Hasebe (yohasebe@gmail.com)
The author will appreciate your mentioning one of these in your research.
- Yoichiro HASEBE. 2006. Method for using Wikipedia as Japanese corpus. Doshisha Studies in Language and Culture 9(2), 373-403.
- 長谷部陽一郎. 2006. Wikipedia日本語版をコーパスとして用いた言語研究の手法. 『言語文化』9(2), 373-403.
BibTeX:
@misc{wp2txt_2026,
author = {Yoichiro Hasebe},
title = {WP2TXT: A command-line toolkit to extract text content and category data from Wikipedia dump files},
url = {https://github.com/yohasebe/wp2txt},
year = {2026}
}
This software is distributed under the MIT License. Please see the LICENSE file.