This project explores the effect of different tokenization methods for three target languages: language pairs: Finnish - Hungarian Finnish - North Sami Estonian - Hungarian Estonian - North Sami Russian - Komi Zyrian
- High-resource: Finnish
- Low-resource: Estonian
The goal is to analyze how different tokenizers influence sentence segmentation and representation for languages with varying data availability and morphological complexity.
- BPE (Byte Pair Encoding)
- OBPE (Overlap BPE): https://github.com/Vaidehi99/OBPE
- Unigram
Each tokenizer will be applied to the same dataset to ensure fair comparison of token-level segmentation behavior and downstream translation performance.
The pilot experiments use the ELRC-EMEA corpus — a bilingual dataset derived from the European Medicines Agency’s PDF documents:
European Medicines Agency (EMEA)
https://www.ema.europa.eu
February 2020 (EN–MT)
├── models/ # tokenization and POS tagger models
├── tagger_scripts/ # POS tagging and training scripts
├── tokenizer_scripts/ # tokenization and data processing scripts
├── vocab # bpe & unigram vocab
├── README.md
A flexible tokenizer script tokenizer_bpe_unigram.py can train and apply both BPE and Unigram tokenizers. Users can specify input files, output files, model prefix, vocabulary size, and tokenizer type directly via command-line arguments. This allows seamless experimentation without modifying the script for different datasets.
Usage Examples:
# Unigram tokenization
python tokenizer_scripts/tokenizer_bpe_unigram.py \
--tokenizer unigram \
--input pilot_data/wmt24pp.en-et.et \
--output pilot_data/wmt24pp.unigram.en-et \
--model_prefix unigram_model \
--vocab_size 5000
# BPE tokenization
python tokenizer_scripts/tokenizer_bpe_unigram.py \
--tokenizer bpe \
--input pilot_data/wmt24pp.en-fi.fi \
--output pilot_data/wmt24pp.bpe.en-fi \
--model_prefix bpe_model \
--vocab_size 5000This repository contains publicly available datasets:
- ELRC-2709-EMEA: Bilingual corpus from the European Medicines Agency (EN–FI / EN–ET).
- WMT24++ (en-et / en-fi): Publicly available parallel datasets from the WMT24 News Translation task.
Ensure that you comply with the respective data-sharing and citation guidelines when using, redistributing, or publishing results based on these datasets.
- For ELRC-EMEA, see ELRC-SHARE.
- For WMT24++, refer to the WMT24 dataset license and terms.
No proprietary data from this repository may be redistributed beyond the usage permissions granted by the original dataset licenses.