diff --git a/docs/en/training_guides/cpt-npu-tutorial.ipynb b/docs/en/training_guides/cpt-npu-tutorial.ipynb new file mode 100644 index 0000000..30fe501 --- /dev/null +++ b/docs/en/training_guides/cpt-npu-tutorial.ipynb @@ -0,0 +1,355 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0-title", + "metadata": {}, + "source": [ + "# Continued Pre-training (CPT) on Ascend NPU\n", + "\n", + "This notebook runs **Continued Pre-training** — plain next-token prediction over a raw-text corpus — on a Huawei Ascend NPU (e.g. `Ascend910B4`) using only mainline `transformers` + `torch_npu`. No CUDA, no `bitsandbytes`, no `training_hub`.\n", + "\n", + "> Why a separate NPU notebook? `training_hub` (the library `cpt-comprehensive-tutorial.ipynb` uses) targets NVIDIA GPUs via `torch.cuda`, `bitsandbytes`, and `unsloth`. None of those run on Ascend. The recipe here reproduces the same *training* using `transformers.Trainer` on `torch_npu`, so the resulting checkpoint is a drop-in HF-format model just like the CUDA path produces.\n", + "\n", + "**Runtime prerequisites**\n", + "\n", + "- Ascend driver + CANN runtime installed on the node (matching the workbench image's CANN version).\n", + "- Workbench image: `PyTorch CANN` (Python 3.12, CANN 8.5.0, PyTorch 2.9.0, `torch_npu 2.9.0`) or equivalent.\n", + "- At least 1 NPU visible via the Kubernetes device plugin (`huawei.com/Ascend910B4` or your cluster's resource name).\n", + "- Persistent storage for the base model + checkpoints (`/opt/app-root/src/...`)." + ] + }, + { + "cell_type": "markdown", + "id": "1-when", + "metadata": {}, + "source": [ + "## When to use CPT\n", + "\n", + "CPT keeps the original **causal-LM** objective and updates *all* weights, so it's the tool for **injecting domain knowledge** before instruction tuning. Typical pipeline:\n", + "\n", + "```\n", + "raw corpus → CPT (this notebook) → SFT / LoRA → alignment\n", + "```\n", + "\n", + "CPT catastrophically forgets general-domain skills if you run it too long on a narrow corpus. Mitigations: keep learning rate small (`1e-5`–`5e-6`), mix general-domain text into the corpus, and always follow up with SFT/OSFT." + ] + }, + { + "cell_type": "markdown", + "id": "2-env", + "metadata": {}, + "source": [ + "## Step 1 — Environment check\n", + "\n", + "Confirm `torch_npu` is importable and at least one NPU is visible. `torch_npu` monkey-patches `torch` on import so `torch.npu.*` becomes available." + ] + }, + { + "cell_type": "code", + "id": "2-env-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "# NPU allocator tuning — expandable segments avoid fragmentation on Ascend.\n", + "os.environ.setdefault(\"PYTORCH_NPU_ALLOC_CONF\", \"expandable_segments:True\")\n", + "\n", + "import torch\n", + "import torch_npu # noqa: F401 — side-effect import registers torch.npu\n", + "import transformers\n", + "\n", + "print(\"torch:\", torch.__version__)\n", + "print(\"torch_npu:\", torch_npu.__version__)\n", + "print(\"transformers:\", transformers.__version__)\n", + "print(\"NPU available:\", torch.npu.is_available())\n", + "print(\"NPU count:\", torch.npu.device_count())\n", + "if torch.npu.is_available():\n", + " torch.npu.set_device(0)\n", + " print(\"device 0:\", torch.npu.get_device_name(0))\n", + " # bf16 support is standard on Ascend910B — quick sanity check.\n", + " x = torch.randn(64, 64, dtype=torch.bfloat16, device=\"npu\")\n", + " y = (x @ x.T).float().mean().item()\n", + " print(f\"bf16 matmul ok, mean={y:.4f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "3-params", + "metadata": {}, + "source": [ + "## Step 2 — Parameters\n", + "\n", + "Edit these to point at your model and corpus. Defaults use a small base and short training so the notebook completes as a smoke test in a few minutes on a single NPU." + ] + }, + { + "cell_type": "code", + "id": "3-params-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- model & corpus ---\n", + "MODEL_PATH = os.environ.get(\"HF_MODEL_DIR\", \"/opt/app-root/src/models/Qwen2.5-0.5B\")\n", + "CORPUS_PATH = os.environ.get(\"CORPUS_PATH\", \"/opt/app-root/src/datasets/cpt/corpus.jsonl\")\n", + "OUTPUT_DIR = os.environ.get(\"OUTPUT_DIR\", \"/opt/app-root/src/cpt-npu-output\")\n", + "\n", + "# --- training ---\n", + "BLOCK_SIZE = 512 # chunk length used to build training samples\n", + "NUM_EPOCHS = 1\n", + "PER_DEVICE_BATCH_SIZE = 2\n", + "GRAD_ACC_STEPS = 4 # effective batch = 2 * 4 = 8 tokens/step\n", + "LEARNING_RATE = 5e-6 # keep small — CPT is easy to overfit / forget\n", + "WARMUP_STEPS = 20\n", + "SAVE_STEPS = 200\n", + "LOGGING_STEPS = 10\n", + "\n", + "# --- compute ---\n", + "USE_BF16 = True # Ascend910B supports bf16 natively\n", + "SEED = 42\n", + "\n", + "os.makedirs(OUTPUT_DIR, exist_ok=True)\n", + "print(\"model :\", MODEL_PATH)\n", + "print(\"corpus:\", CORPUS_PATH)\n", + "print(\"output:\", OUTPUT_DIR)" + ] + }, + { + "cell_type": "markdown", + "id": "4-data-format", + "metadata": {}, + "source": [ + "## Step 3 — Data format\n", + "\n", + "CPT expects **raw text**, not chat turns. Each line of the JSONL is one document under a `text` field:\n", + "\n", + "```json\n", + "{\"text\": \"Alauda AI is an MLOps platform that runs training and inference on Kubernetes.\"}\n", + "{\"text\": \"Continued pre-training adapts a base model to a new domain using unlabeled text.\"}\n", + "```\n", + "\n", + "The next cell falls back to a tiny **built-in synthetic corpus** if `CORPUS_PATH` doesn't exist, so the notebook can be run end-to-end without any dataset upload — useful as a smoke test." + ] + }, + { + "cell_type": "code", + "id": "4-data-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "from pathlib import Path\n", + "\n", + "if not Path(CORPUS_PATH).exists():\n", + " fallback = Path(OUTPUT_DIR) / \"synthetic_corpus.jsonl\"\n", + " docs = [\n", + " \"Alauda AI is an MLOps platform that runs fine-tuning and inference on Kubernetes.\",\n", + " \"Continued pre-training keeps the causal-LM objective and updates every weight.\",\n", + " \"Huawei Ascend NPUs are programmed through the CANN runtime and torch_npu.\",\n", + " \"A CPT run should be followed by SFT to restore instruction-following behaviour.\",\n", + " \"transformers.Trainer works on torch_npu out of the box once torch_npu is imported.\",\n", + " ] * 20\n", + " with fallback.open(\"w\") as f:\n", + " for d in docs:\n", + " json.dump({\"text\": d}, f); f.write(\"\\n\")\n", + " CORPUS_PATH = str(fallback)\n", + " print(f\"corpus not found; wrote synthetic fallback to {CORPUS_PATH}\")\n", + "print(\"corpus:\", CORPUS_PATH)" + ] + }, + { + "cell_type": "markdown", + "id": "5-tokenize", + "metadata": {}, + "source": [ + "## Step 4 — Load model + tokenizer, tokenize corpus\n", + "\n", + "Load the base model in `bf16` (Ascend-native), then tokenize the corpus and pack it into fixed-length chunks of `BLOCK_SIZE`. For CPT, `labels == input_ids` — every token contributes to the loss." + ] + }, + { + "cell_type": "code", + "id": "5-tokenize-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import AutoModelForCausalLM, AutoTokenizer\n", + "from datasets import load_dataset\n", + "\n", + "tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, use_fast=True)\n", + "if tokenizer.pad_token is None:\n", + " tokenizer.pad_token = tokenizer.eos_token\n", + "\n", + "model = AutoModelForCausalLM.from_pretrained(\n", + " MODEL_PATH,\n", + " torch_dtype=torch.bfloat16 if USE_BF16 else torch.float32,\n", + " attn_implementation=\"eager\", # SDPA works on NPU; eager is the safe default across CANN versions\n", + ")\n", + "model.to(\"npu\")\n", + "model.gradient_checkpointing_enable()\n", + "\n", + "raw = load_dataset(\"json\", data_files=CORPUS_PATH, split=\"train\")\n", + "print(\"raw documents:\", len(raw))\n", + "\n", + "def tokenize(batch):\n", + " return tokenizer(batch[\"text\"], add_special_tokens=False)\n", + "\n", + "tokenized = raw.map(tokenize, batched=True, remove_columns=raw.column_names)\n", + "\n", + "def group_texts(examples):\n", + " concatenated = {k: sum(examples[k], []) for k in examples.keys()}\n", + " total_len = (len(concatenated[\"input_ids\"]) // BLOCK_SIZE) * BLOCK_SIZE\n", + " result = {\n", + " k: [t[i : i + BLOCK_SIZE] for i in range(0, total_len, BLOCK_SIZE)]\n", + " for k, t in concatenated.items()\n", + " }\n", + " result[\"labels\"] = [ids.copy() for ids in result[\"input_ids\"]]\n", + " return result\n", + "\n", + "packed = tokenized.map(group_texts, batched=True)\n", + "print(\"packed chunks:\", len(packed), \"of\", BLOCK_SIZE, \"tokens each\")" + ] + }, + { + "cell_type": "markdown", + "id": "6-train", + "metadata": {}, + "source": [ + "## Step 5 — Train\n", + "\n", + "Use plain `Trainer` with `bf16=True`. **Do not** set `fp16=True` — Ascend prefers bf16, and mixing fp16 with older CANN builds triggers dynamic-shape recompilation.\n", + "\n", + "The optimizer is `adamw_torch` (mainline PyTorch). Do not use `adamw_bnb_8bit` — bitsandbytes is not upstream on NPU (see the QLoRA-NPU notebook for how to bring in the community `bitsandbytes-npu` fork if you need it)." + ] + }, + { + "cell_type": "code", + "id": "6-train-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling\n", + "\n", + "collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)\n", + "\n", + "args = TrainingArguments(\n", + " output_dir=OUTPUT_DIR,\n", + " num_train_epochs=NUM_EPOCHS,\n", + " per_device_train_batch_size=PER_DEVICE_BATCH_SIZE,\n", + " gradient_accumulation_steps=GRAD_ACC_STEPS,\n", + " learning_rate=LEARNING_RATE,\n", + " warmup_steps=WARMUP_STEPS,\n", + " logging_steps=LOGGING_STEPS,\n", + " save_steps=SAVE_STEPS,\n", + " save_total_limit=2,\n", + " bf16=USE_BF16,\n", + " fp16=False,\n", + " optim=\"adamw_torch\",\n", + " lr_scheduler_type=\"cosine\",\n", + " seed=SEED,\n", + " report_to=[],\n", + " remove_unused_columns=False,\n", + " dataloader_pin_memory=False, # pinned memory is not supported on NPU\n", + ")\n", + "\n", + "trainer = Trainer(\n", + " model=model,\n", + " args=args,\n", + " train_dataset=packed,\n", + " data_collator=collator,\n", + ")\n", + "\n", + "train_result = trainer.train()\n", + "print(train_result.metrics)" + ] + }, + { + "cell_type": "markdown", + "id": "7-save", + "metadata": {}, + "source": [ + "## Step 6 — Save the HF-format checkpoint\n", + "\n", + "Save the tuned weights and tokenizer under `OUTPUT_DIR/hf_format` so downstream SFT / inference recipes can load it with `AutoModelForCausalLM.from_pretrained(...)`." + ] + }, + { + "cell_type": "code", + "id": "7-save-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "hf_dir = os.path.join(OUTPUT_DIR, \"hf_format\")\n", + "trainer.save_model(hf_dir)\n", + "tokenizer.save_pretrained(hf_dir)\n", + "print(\"saved:\", hf_dir)\n", + "print(\"contents:\", sorted(os.listdir(hf_dir)))" + ] + }, + { + "cell_type": "markdown", + "id": "8-inference", + "metadata": {}, + "source": [ + "## Step 7 — Quick inference smoke\n", + "\n", + "Load the freshly-saved checkpoint back onto the NPU and generate a few tokens. This is a *sanity* check — the base model is small and the training corpus is tiny, so don't judge quality here." + ] + }, + { + "cell_type": "code", + "id": "8-infer-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import AutoModelForCausalLM, AutoTokenizer\n", + "\n", + "tok = AutoTokenizer.from_pretrained(hf_dir)\n", + "mdl = AutoModelForCausalLM.from_pretrained(hf_dir, torch_dtype=torch.bfloat16).to(\"npu\")\n", + "mdl.eval()\n", + "\n", + "prompt = \"Alauda AI is\"\n", + "inputs = tok(prompt, return_tensors=\"pt\").to(\"npu\")\n", + "with torch.no_grad():\n", + " out = mdl.generate(**inputs, max_new_tokens=32, do_sample=False)\n", + "print(tok.decode(out[0], skip_special_tokens=True))" + ] + }, + { + "cell_type": "markdown", + "id": "9-notes", + "metadata": {}, + "source": [ + "## Notes and gotchas on NPU\n", + "\n", + "- **Import order.** `import torch_npu` **before** the first `torch.npu.*` call — it registers the backend.\n", + "- **Precision.** Use `bf16`. Avoid `fp16` unless you know your CANN build supports it end-to-end; mixing fp16 with older builds triggers dynamic-shape retracing that stalls training.\n", + "- **Optimizer.** Stick to `adamw_torch`. `adamw_bnb_8bit` requires `bitsandbytes` — see the [QLoRA-NPU tutorial](./qlora-npu-tutorial.ipynb) for how to install the Ascend fork if you need paged optimizers.\n", + "- **Attention backend.** `attn_implementation=\"eager\"` is the portable default. `sdpa` works on most CANN 8.x builds but occasionally hits an unsupported-op path on prerelease combinations. `flash_attention_2` is CUDA-only.\n", + "- **Data loader.** Set `dataloader_pin_memory=False`; NPU does not use CUDA pinned host memory and the flag can trip guard checks.\n", + "- **Multi-NPU.** For >1 NPU on a single node, launch this notebook's script form under `torch.distributed` via `torchrun --nproc_per_node=`. `Trainer` picks up `torch.distributed` transparently on NPU thanks to `hccl`.\n", + "- **Catastrophic forgetting.** Keep `learning_rate` in the `1e-5` – `5e-6` range for CPT and always follow up with SFT — CPT alone breaks instruction-following." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/en/training_guides/qlora-npu-tutorial.ipynb b/docs/en/training_guides/qlora-npu-tutorial.ipynb new file mode 100644 index 0000000..70df73b --- /dev/null +++ b/docs/en/training_guides/qlora-npu-tutorial.ipynb @@ -0,0 +1,398 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0-title", + "metadata": {}, + "source": [ + "# QLoRA on Ascend NPU\n", + "\n", + "This notebook fine-tunes a chat model with **QLoRA — 4-bit NF4 base + LoRA adapters** — on a Huawei Ascend NPU (e.g. `Ascend910B4`) using `transformers` + `peft` + `torch_npu` + the community `bitsandbytes-npu-beta` fork. It is the NPU companion of [`qlora-comprehensive-tutorial.ipynb`](./qlora-comprehensive-tutorial.ipynb), which uses `training_hub` on CUDA.\n", + "\n", + "> **Why a separate NPU notebook?** `training_hub` targets NVIDIA GPUs via `torch.cuda`, `bitsandbytes` (CUDA build), and `unsloth`. None of those run on Ascend today. Mainline `bitsandbytes` also removed its Ascend backend from the v0.46+ install docs — upstream re-add is tracked in [`bitsandbytes-foundation/bitsandbytes#1695`](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1695). Until that merges, the practical Ascend path is the community fork [`SlightwindSec/bitsandbytes`](https://github.com/SlightwindSec/bitsandbytes), published as `bitsandbytes-npu-beta` on PyPI.\n", + "\n", + "**Support caveat.** `bitsandbytes-npu-beta==0.45.3` (2025-03-18) was built against roughly `torch_npu 2.1 – 2.4` / `CANN 8.0.RC3 – 8.1`. The Alauda `PyTorch CANN` workbench image ships `torch_npu 2.9.0` / `CANN 8.5.0`. `torch_npu` keeps loose ABI compatibility across `2.x`, but this specific combination is **not officially validated** by the fork's author. If import fails with `undefined symbol` or a dtype mismatch, pin the workbench to an older `PyTorch CANN` image (CANN 8.1 / torch_npu 2.4 range) instead.\n", + "\n", + "**Runtime prerequisites**\n", + "\n", + "- Ascend driver + CANN runtime installed on the node.\n", + "- Workbench image: `PyTorch CANN` (Python 3.12, `torch_npu`, `transformers`, `peft`) on an **aarch64** node — `bitsandbytes-npu-beta` ships only a `manylinux2014_aarch64` wheel.\n", + "- At least 1 NPU exposed as `huawei.com/Ascend910B4` (or your cluster's resource name) with ≥ 8 GB HBM available.\n", + "- Persistent storage for base model + adapter checkpoints." + ] + }, + { + "cell_type": "markdown", + "id": "1-when", + "metadata": {}, + "source": [ + "## When to use QLoRA on NPU\n", + "\n", + "- Base model too large to hold in bf16 on the NPUs you have (e.g. Qwen2.5-14B on a single 32 GB Ascend910B).\n", + "- You want a small, cheap adapter you can ship alongside the frozen base model.\n", + "- You're okay trading a little quality for a large memory saving — QLoRA is memory-efficient LoRA, not a full-parameter tune.\n", + "\n", + "If the base model already fits in bf16, use plain LoRA (`load_in_4bit=False`) — you skip the fork dependency entirely and stay on upstream `transformers + peft`." + ] + }, + { + "cell_type": "markdown", + "id": "2-install", + "metadata": {}, + "source": [ + "## Step 1 — Install `bitsandbytes-npu-beta`\n", + "\n", + "One-off install into the workbench venv. Everything else (`transformers`, `peft`, `accelerate`, `trl`, `datasets`) comes from the bundled runtime.\n", + "\n", + "```bash\n", + "pip install --no-cache-dir bitsandbytes-npu-beta==0.45.3\n", + "```\n", + "\n", + "The version pin is deliberate — `0.45.3` is the last (as of Jan 2026) published tag of the community fork. Don't upgrade past it without checking [the PyPI release page](https://pypi.org/project/bitsandbytes-npu-beta/); a newer release may or may not exist." + ] + }, + { + "cell_type": "markdown", + "id": "3-env", + "metadata": {}, + "source": [ + "## Step 2 — Environment check\n", + "\n", + "**Import order matters.** `torch_npu` must be imported **before** `bitsandbytes` so the fork detects the NPU backend." + ] + }, + { + "cell_type": "code", + "id": "3-env-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "# Cuts fragmentation on LoRA / QLoRA workloads.\n", + "os.environ.setdefault(\"PYTORCH_NPU_ALLOC_CONF\", \"expandable_segments:True\")\n", + "\n", + "import torch\n", + "import torch_npu # noqa: F401 — must precede bitsandbytes import\n", + "import bitsandbytes as bnb\n", + "import transformers, peft\n", + "\n", + "print(\"torch :\", torch.__version__)\n", + "print(\"torch_npu :\", torch_npu.__version__)\n", + "print(\"bitsandbytes :\", bnb.__version__)\n", + "print(\"transformers :\", transformers.__version__)\n", + "print(\"peft :\", peft.__version__)\n", + "assert torch.npu.is_available(), \"no NPU visible — check the device plugin and resource limits\"\n", + "torch.npu.set_device(0)\n", + "print(\"device 0 :\", torch.npu.get_device_name(0))" + ] + }, + { + "cell_type": "markdown", + "id": "4-params", + "metadata": {}, + "source": [ + "## Step 3 — Parameters\n", + "\n", + "Defaults use a small base and a short training so the notebook completes in a few minutes on a single NPU. Bump `MODEL_PATH`, `NUM_EPOCHS`, and `MAX_SEQ_LEN` for real runs." + ] + }, + { + "cell_type": "code", + "id": "4-params-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "MODEL_PATH = os.environ.get(\"HF_MODEL_DIR\", \"/opt/app-root/src/models/Qwen2.5-0.5B-Instruct\")\n", + "DATA_PATH = os.environ.get(\"DATA_PATH\", \"/opt/app-root/src/datasets/chat/train.jsonl\")\n", + "OUTPUT_DIR = os.environ.get(\"OUTPUT_DIR\", \"/opt/app-root/src/qlora-npu-output\")\n", + "\n", + "# LoRA\n", + "LORA_R = 8\n", + "LORA_ALPHA = 16\n", + "LORA_DROPOUT = 0.05\n", + "LORA_TARGETS = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"]\n", + "\n", + "# Training\n", + "NUM_EPOCHS = 1\n", + "PER_DEVICE_BATCH_SIZE = 2\n", + "GRAD_ACC_STEPS = 4 # effective batch = 8\n", + "LEARNING_RATE = 2e-4 # QLoRA can use LR much higher than full fine-tune\n", + "MAX_SEQ_LEN = 512\n", + "WARMUP_STEPS = 20\n", + "LOGGING_STEPS = 10\n", + "SAVE_STEPS = 200\n", + "SEED = 42\n", + "\n", + "os.makedirs(OUTPUT_DIR, exist_ok=True)" + ] + }, + { + "cell_type": "markdown", + "id": "5-data", + "metadata": {}, + "source": [ + "## Step 4 — Data format\n", + "\n", + "Chat JSONL — one conversation per line under `messages`:\n", + "\n", + "```json\n", + "{\"messages\": [\n", + " {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n", + " {\"role\": \"user\", \"content\": \"What is machine learning?\"},\n", + " {\"role\": \"assistant\", \"content\": \"Machine learning is ...\"}\n", + "]}\n", + "```\n", + "\n", + "The next cell falls back to a tiny built-in synthetic dataset if `DATA_PATH` doesn't exist — the notebook stays runnable as a smoke test." + ] + }, + { + "cell_type": "code", + "id": "5-data-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "from pathlib import Path\n", + "\n", + "if not Path(DATA_PATH).exists():\n", + " fallback = Path(OUTPUT_DIR) / \"synthetic_chat.jsonl\"\n", + " with fallback.open(\"w\") as f:\n", + " for _ in range(32):\n", + " json.dump({\"messages\": [\n", + " {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n", + " {\"role\": \"user\", \"content\": \"Hello, how are you?\"},\n", + " {\"role\": \"assistant\", \"content\": \"I am well — how can I help?\"},\n", + " ]}, f); f.write(\"\\n\")\n", + " DATA_PATH = str(fallback)\n", + " print(f\"data not found; wrote synthetic fallback to {DATA_PATH}\")\n", + "print(\"data:\", DATA_PATH)" + ] + }, + { + "cell_type": "markdown", + "id": "6-load-quantized", + "metadata": {}, + "source": [ + "## Step 5 — Load the base model in 4-bit\n", + "\n", + "`BitsAndBytesConfig` works unchanged against `bitsandbytes-npu-beta` — no `bnb.enable_ascend()` bootstrap needed. Route the model to `npu:0` via `device_map`." + ] + }, + { + "cell_type": "code", + "id": "6-load-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n", + "from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model\n", + "\n", + "tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, use_fast=True)\n", + "if tokenizer.pad_token is None:\n", + " tokenizer.pad_token = tokenizer.eos_token\n", + "\n", + "bnb_config = BitsAndBytesConfig(\n", + " load_in_4bit=True,\n", + " bnb_4bit_quant_type=\"nf4\",\n", + " bnb_4bit_compute_dtype=torch.bfloat16,\n", + " bnb_4bit_use_double_quant=True,\n", + ")\n", + "\n", + "model = AutoModelForCausalLM.from_pretrained(\n", + " MODEL_PATH,\n", + " quantization_config=bnb_config,\n", + " device_map={\"\": \"npu:0\"},\n", + " torch_dtype=torch.bfloat16,\n", + " attn_implementation=\"eager\", # sdpa works on most CANN 8.x builds; eager is the portable default\n", + ")\n", + "model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)\n", + "\n", + "lora_config = LoraConfig(\n", + " r=LORA_R, lora_alpha=LORA_ALPHA, lora_dropout=LORA_DROPOUT,\n", + " target_modules=LORA_TARGETS, bias=\"none\", task_type=\"CAUSAL_LM\",\n", + ")\n", + "model = get_peft_model(model, lora_config)\n", + "model.print_trainable_parameters()" + ] + }, + { + "cell_type": "markdown", + "id": "7-tokenize", + "metadata": {}, + "source": [ + "## Step 6 — Tokenize the chat corpus\n", + "\n", + "Apply the tokenizer's chat template and tokenize each conversation to `input_ids` + `labels`. For QLoRA-style SFT, the whole conversation contributes to loss (Alauda's `training_hub` default is assistant-only masking — the recipe here uses full-turn loss for portability; adapt if you need instruction-only masking)." + ] + }, + { + "cell_type": "code", + "id": "7-tok-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from datasets import load_dataset\n", + "\n", + "raw = load_dataset(\"json\", data_files=DATA_PATH, split=\"train\")\n", + "\n", + "def render(example):\n", + " text = tokenizer.apply_chat_template(example[\"messages\"], tokenize=False)\n", + " ids = tokenizer(text, truncation=True, max_length=MAX_SEQ_LEN,\n", + " padding=\"max_length\", add_special_tokens=False)\n", + " ids[\"labels\"] = ids[\"input_ids\"].copy()\n", + " return ids\n", + "\n", + "packed = raw.map(render, remove_columns=raw.column_names)\n", + "print(\"samples:\", len(packed), \"of length\", MAX_SEQ_LEN)" + ] + }, + { + "cell_type": "markdown", + "id": "8-train", + "metadata": {}, + "source": [ + "## Step 7 — Train\n", + "\n", + "Use plain `Trainer` with `bf16=True` and `optim=\"adamw_torch\"`. Do **not** use `paged_adamw_8bit` — that path relies on CUDA kernels not shipped in the NPU fork." + ] + }, + { + "cell_type": "code", + "id": "8-train-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling\n", + "\n", + "collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)\n", + "\n", + "args = TrainingArguments(\n", + " output_dir=OUTPUT_DIR,\n", + " num_train_epochs=NUM_EPOCHS,\n", + " per_device_train_batch_size=PER_DEVICE_BATCH_SIZE,\n", + " gradient_accumulation_steps=GRAD_ACC_STEPS,\n", + " learning_rate=LEARNING_RATE,\n", + " warmup_steps=WARMUP_STEPS,\n", + " logging_steps=LOGGING_STEPS,\n", + " save_steps=SAVE_STEPS,\n", + " save_total_limit=2,\n", + " bf16=True, fp16=False,\n", + " optim=\"adamw_torch\",\n", + " lr_scheduler_type=\"cosine\",\n", + " seed=SEED,\n", + " report_to=[],\n", + " remove_unused_columns=False,\n", + " dataloader_pin_memory=False, # NPU has no CUDA pinned host memory\n", + ")\n", + "\n", + "trainer = Trainer(\n", + " model=model,\n", + " args=args,\n", + " train_dataset=packed,\n", + " data_collator=collator,\n", + ")\n", + "\n", + "train_result = trainer.train()\n", + "print(train_result.metrics)" + ] + }, + { + "cell_type": "markdown", + "id": "9-save", + "metadata": {}, + "source": [ + "## Step 8 — Save the LoRA adapter\n", + "\n", + "Save only the adapter weights (a few MB) — the 4-bit base is unchanged and lives in the base-model directory. To serve the tuned model, re-attach the adapter to the fp16/bf16 base at inference time." + ] + }, + { + "cell_type": "code", + "id": "9-save-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "adapter_dir = os.path.join(OUTPUT_DIR, \"adapter\")\n", + "model.save_pretrained(adapter_dir)\n", + "tokenizer.save_pretrained(adapter_dir)\n", + "print(\"saved:\", adapter_dir)\n", + "print(\"contents:\", sorted(os.listdir(adapter_dir)))" + ] + }, + { + "cell_type": "markdown", + "id": "10-merge", + "metadata": {}, + "source": [ + "## Step 9 (optional) — Reload adapter for inference\n", + "\n", + "Load the frozen bf16 base and attach the trained adapter for a quick generation smoke. For real inference workflows you'd typically merge the adapter (`PeftModel.merge_and_unload()`) once, save the merged model, and serve the merged HF checkpoint." + ] + }, + { + "cell_type": "code", + "id": "10-merge-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import AutoModelForCausalLM, AutoTokenizer\n", + "from peft import PeftModel\n", + "\n", + "tok = AutoTokenizer.from_pretrained(MODEL_PATH)\n", + "base = AutoModelForCausalLM.from_pretrained(MODEL_PATH, torch_dtype=torch.bfloat16).to(\"npu\")\n", + "tuned = PeftModel.from_pretrained(base, adapter_dir).to(\"npu\")\n", + "tuned.eval()\n", + "\n", + "prompt = tok.apply_chat_template(\n", + " [{\"role\": \"user\", \"content\": \"Hello, how are you?\"}],\n", + " tokenize=False, add_generation_prompt=True,\n", + ")\n", + "inputs = tok(prompt, return_tensors=\"pt\").to(\"npu\")\n", + "with torch.no_grad():\n", + " out = tuned.generate(**inputs, max_new_tokens=32, do_sample=False)\n", + "print(tok.decode(out[0], skip_special_tokens=True))" + ] + }, + { + "cell_type": "markdown", + "id": "11-notes", + "metadata": {}, + "source": [ + "## Notes and gotchas on NPU QLoRA\n", + "\n", + "- **Import order.** `import torch_npu` **before** `import bitsandbytes`. If bnb is imported first, it won't register the NPU backend and 4-bit loading will fall over.\n", + "- **`bitsandbytes-npu-beta` version pinning.** `0.45.3` (2025-03-18) is the last known-good published tag. `pip install bitsandbytes` (unpinned) resolves to the upstream CUDA build — always specify the fork's PyPI name.\n", + "- **CANN / torch_npu compatibility.** The fork's wheel was built against `torch_npu 2.1 – 2.4`. If import fails on newer stacks (e.g. `torch_npu 2.9` / CANN 8.5), pin the workbench to a CANN 8.1-era `PyTorch CANN` image.\n", + "- **Optimizer.** Use `adamw_torch`. `paged_adamw_8bit` and other bnb-specific optimizers rely on CUDA kernels not present in the NPU fork.\n", + "- **CPU offload.** Do **not** set `llm_int8_enable_fp32_cpu_offload=True` — the fork does not document offload as working.\n", + "- **Attention backend.** `attn_implementation=\"eager\"` is the portable default. `sdpa` usually works on CANN 8.x. `flash_attention_2` is CUDA-only.\n", + "- **Precision.** Use `bf16`. Avoid `fp16` on the compute path — Ascend910B prefers bf16 and mixing it with the fork's dequant kernels has been reported to trip recompilation.\n", + "- **aarch64 only.** The fork ships only a `manylinux2014_aarch64` wheel; there is no x86 wheel.\n", + "- **Data loader.** Set `dataloader_pin_memory=False`; NPU has no CUDA pinned host memory.\n", + "- **Path to native support.** Track [`bitsandbytes-foundation/bitsandbytes#1695`](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1695) — once merged upstream, the fork can be retired and mainline `bitsandbytes` with `--index-url` for a matching wheel becomes the recommended path." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/en/training_guides/training-hub-fine-tuning.mdx b/docs/en/training_guides/training-hub-fine-tuning.mdx index 42f5fe5..997ecd8 100644 --- a/docs/en/training_guides/training-hub-fine-tuning.mdx +++ b/docs/en/training_guides/training-hub-fine-tuning.mdx @@ -63,7 +63,9 @@ Download into your workbench and execute cell-by-cell: | SFT comprehensive tutorial | SFT | [`sft-comprehensive-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/sft-comprehensive-tutorial.ipynb) | | OSFT comprehensive tutorial | OSFT | [`osft-comprehensive-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/osft-comprehensive-tutorial.ipynb) | | QLoRA comprehensive tutorial | QLoRA | [`qlora-comprehensive-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/qlora-comprehensive-tutorial.ipynb) | +| QLoRA on Ascend NPU | QLoRA (NPU) | [`qlora-npu-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/qlora-npu-tutorial.ipynb) | | CPT comprehensive tutorial | CPT | [`cpt-comprehensive-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/cpt-comprehensive-tutorial.ipynb) | +| CPT on Ascend NPU | CPT (NPU) | [`cpt-npu-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/cpt-npu-tutorial.ipynb) | Install and configure: @@ -199,9 +201,19 @@ The output is a **LoRA adapter**, not a full checkpoint. To serve, load base + a `trl`, `peft`, and `bitsandbytes`. The default `lora_sft` backend is `unsloth`; for full control you can also drive `peft` + `bitsandbytes` directly on the same runtime image. -**NPU:** bitsandbytes 4-bit is not available on Huawei Ascend. Use LoRA (without 4-bit) on -the `llamafactory0.9-cann8.5-arm64` runtime (`finetuning_type: lora`) as the -parameter-efficient path — see [Fine-tune and Pretrain on Ascend NPU](./fine-tune-and-pretrain-llms-on-ascend-npu.mdx). +**NPU:** `training_hub` and mainline `bitsandbytes` do not target Huawei Ascend. Two paths on +NPU: + +- **True 4-bit NF4 QLoRA** via the community `bitsandbytes-npu-beta` fork + ([SlightwindSec/bitsandbytes](https://github.com/SlightwindSec/bitsandbytes), tracked + upstream in [PR #1695](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1695)) — + see [`qlora-npu-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/qlora-npu-tutorial.ipynb). Uses standard + `transformers` + `peft` + `torch_npu`; the API is drop-in, but the wheel is only tested + against `torch_npu 2.1–2.4` / CANN 8.0–8.1 and compatibility with newer CANN builds is + best-effort. +- **Plain LoRA (no 4-bit)** on the `llamafactory0.9-cann8.5-arm64` runtime + (`finetuning_type: lora`) — see [Fine-tune and Pretrain on Ascend NPU](./fine-tune-and-pretrain-llms-on-ascend-npu.mdx). + Slightly higher memory footprint than QLoRA, but no non-mainline dependencies. ::: ## Continued pre-training (CPT) \{#cpt} @@ -254,10 +266,15 @@ general-domain text, and by following up with an SFT/OSFT pass (point `model_pat checkpoint). If forgetting is the primary concern, prefer **OSFT**. :::note -**NPU:** Full-parameter continued pre-training also runs on Huawei Ascend via the -`MindSpeed-LLM` runtime (`pretrain_gpt.py`). See -[Fine-tune and Pretrain on Ascend NPU](./fine-tune-and-pretrain-llms-on-ascend-npu.mdx) and -the `qwen25_pretrain_verify.ipynb` recipe. +**NPU:** Two paths for CPT on Ascend: + +- **Light-weight, single-node** CPT with `transformers.Trainer` on `torch_npu` — + see [`cpt-npu-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/cpt-npu-tutorial.ipynb). Uses only mainline + `transformers` + `torch_npu` (no `MindSpeed-LLM`, no HF↔MCore conversion). Best when the + model fits in bf16 on the NPUs you have and you don't need TP/PP. +- **Distributed, TP/PP-aware** CPT via the `MindSpeed-LLM` runtime (`pretrain_gpt.py`) — + see [Fine-tune and Pretrain on Ascend NPU](./fine-tune-and-pretrain-llms-on-ascend-npu.mdx) + and the `qwen25_pretrain_verify.ipynb` recipe. ::: ## Multi-node diff --git a/e2e/cases/c15_qlora_npu.sh b/e2e/cases/c15_qlora_npu.sh new file mode 100755 index 0000000..ab7da92 --- /dev/null +++ b/e2e/cases/c15_qlora_npu.sh @@ -0,0 +1,339 @@ +#!/usr/bin/env bash +# C15 — QLoRA (4-bit NF4 + LoRA adapters) on a Huawei Ascend NPU using +# transformers + peft + torch_npu + the community bitsandbytes-npu-beta +# fork. Companion of C13 (which uses training_hub on CUDA): +# training_hub / mainline bitsandbytes do not run on Ascend, so the recipe +# uses `BitsAndBytesConfig(load_in_4bit=True, ...)` with the +# `bitsandbytes-npu-beta` PyPI package. Generates a tiny synthetic Qwen2 base +# model + a synthetic chat JSONL in-Pod so it needs no external download. +# +# Image: the same PyTorch CANN workbench image C7/C8/C16 use. CANN env is +# sourced per the C8 pattern. NPU is requested via ${NPU_RESOURCE_NAME} (e.g. +# huawei.com/Ascend910B4), quantity ${NPU_RESOURCE_VALUE:-1}. +# +# SKIP (rc=77) conditions: +# * NPU_NAMESPACE or NPU_RESOURCE_NAME not set (opt-in — see run_all.sh); +# * the NPU slice cannot be scheduled (captured scheduler event); +# * the in-Pod NPU sanity check fails (torch.npu.is_available() False); +# * `import bitsandbytes` fails after `pip install bitsandbytes-npu-beta` — +# the fork is torch_npu 2.1–2.4 / CANN 8.0–8.1 era, and CANN 8.5+ compat +# is not officially validated by the fork's author (see +# `qlora-npu-tutorial.ipynb` for the compatibility caveat). +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +source "${HERE}/../lib.sh" + +require_env NPU_NAMESPACE "namespace for NPU e2e resources" +require_env NPU_RESOURCE_NAME "extended resource name for one NPU, for example huawei.com/Ascend910B4" +NS="${NPU_NAMESPACE}" +JOB_NAME="c15-qlora-npu-$(printf '%05x' $$)" +IMAGE="${C15_IMAGE:-docker.io/alaudadockerhub/alauda-workbench-jupyter-pytorch-cann-py312-ubi9:v0.1.7}" +IMAGE_PULL_SECRET="${C15_IMAGE_PULL_SECRET:-${E2E_IMAGE_PULL_SECRET:-}}" +NPU_RESOURCE_VALUE="${NPU_RESOURCE_VALUE:-1}" +NPU_MEMORY_RESOURCE_NAME="${NPU_MEMORY_RESOURCE_NAME:-}" +NPU_MEMORY_RESOURCE_VALUE="${NPU_MEMORY_RESOURCE_VALUE:-8192}" +NPU_RUNTIME_CLASS="${NPU_RUNTIME_CLASS:-}" +# The fork's PyPI name & pinned version. See qlora-npu-tutorial.ipynb. +BNB_NPU_PIN="${BNB_NPU_PIN:-bitsandbytes-npu-beta==0.45.3}" +PIP_INDEX_URL="${PIP_INDEX_URL:-}" + +cleanup() { + npu_kc -n "${NS}" delete job "${JOB_NAME}" --ignore-not-found --wait=false || true +} +trap cleanup EXIT + +log "C15: submitting Job ${JOB_NAME} (image=${IMAGE}, bnb=${BNB_NPU_PIN})" + +cat <&2 + exit 77 + fi + python - <<'PY' + import sys + try: + import torch_npu # MUST be first + import bitsandbytes as bnb + print("bitsandbytes:", bnb.__version__) + except Exception as e: + print(f"E2E-SKIP: bitsandbytes-npu-beta import failed: " + f"{type(e).__name__}: {e}", file=sys.stderr) + sys.exit(77) + PY + + # 1) Tiny synthetic Qwen2-style base model + tokenizer (offline). + python - <<'PY' + import os, json, torch + from transformers import GPT2TokenizerFast, Qwen2Config, Qwen2ForCausalLM + from tokenizers import ByteLevelBPETokenizer + + MODEL_DIR = "/workspace/tiny-qwen2" + os.makedirs(MODEL_DIR, exist_ok=True) + torch.manual_seed(0) + bpe = ByteLevelBPETokenizer() + bpe.train_from_iterator( + ["hello world", "the quick brown fox", "alauda ai qlora on ascend"], + vocab_size=512, min_frequency=1, + special_tokens=["", "", "", ""], + ) + bpe.save_model(MODEL_DIR) + tok = GPT2TokenizerFast( + vocab_file=f"{MODEL_DIR}/vocab.json", + merges_file=f"{MODEL_DIR}/merges.txt", + unk_token="", bos_token="", eos_token="", pad_token="", + ) + tok.chat_template = "{% for m in messages %}{{ m.role }}: {{ m.content }}\n{% endfor %}" + tok.save_pretrained(MODEL_DIR) + cfg = Qwen2Config( + vocab_size=tok.vocab_size + 4, + hidden_size=64, num_hidden_layers=2, num_attention_heads=4, + num_key_value_heads=4, intermediate_size=128, max_position_embeddings=256, + rope_theta=10000.0, tie_word_embeddings=True, + ) + Qwen2ForCausalLM(cfg).save_pretrained(MODEL_DIR) + + data_path = "/workspace/test_qlora_data.jsonl" + with open(data_path, "w") as f: + for _ in range(16): + json.dump({"messages": [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Hello, how are you?"}, + {"role": "assistant", "content": "I am well - how can I help?"}, + ]}, f); f.write("\n") + print(f"prepared {MODEL_DIR} and {data_path}") + PY + + # 2) QLoRA: BitsAndBytesConfig(load_in_4bit=...) + peft LoRA. + # Order: torch_npu first, then bitsandbytes. + python - <<'PY' + import os, sys, time, glob, torch + import torch_npu # noqa: F401 — must precede bitsandbytes + try: + import bitsandbytes as bnb # noqa: F401 + except Exception as e: + print(f"E2E-SKIP: bitsandbytes-npu-beta unusable at runtime: " + f"{type(e).__name__}: {e}", file=sys.stderr) + sys.exit(77) + + from transformers import ( + AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, + DataCollatorForLanguageModeling, Trainer, TrainingArguments, + ) + from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model + from datasets import load_dataset + + MODEL_DIR = "/workspace/tiny-qwen2" + DATA = "/workspace/test_qlora_data.jsonl" + OUT = "/workspace/ckpt" + + tok = AutoTokenizer.from_pretrained(MODEL_DIR) + if tok.pad_token is None: + tok.pad_token = tok.eos_token + + bnb_cfg = BitsAndBytesConfig( + load_in_4bit=True, bnb_4bit_quant_type="nf4", + bnb_4bit_compute_dtype=torch.bfloat16, + bnb_4bit_use_double_quant=True, + ) + try: + model = AutoModelForCausalLM.from_pretrained( + MODEL_DIR, quantization_config=bnb_cfg, + device_map={"": "npu:0"}, + torch_dtype=torch.bfloat16, + attn_implementation="eager", + ) + except Exception as e: + print(f"E2E-SKIP: 4-bit load on NPU failed " + f"({type(e).__name__}: {e})", file=sys.stderr) + sys.exit(77) + model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True) + peft_cfg = LoraConfig( + r=8, lora_alpha=16, lora_dropout=0.05, bias="none", + target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], + task_type="CAUSAL_LM", + ) + model = get_peft_model(model, peft_cfg) + model.print_trainable_parameters() + + raw = load_dataset("json", data_files=DATA, split="train") + def render(ex): + text = tok.apply_chat_template(ex["messages"], tokenize=False) + out = tok(text, truncation=True, max_length=128, + padding="max_length", add_special_tokens=False) + out["labels"] = out["input_ids"].copy() + return out + packed = raw.map(render, remove_columns=raw.column_names) + + args = TrainingArguments( + output_dir=OUT, num_train_epochs=1, + per_device_train_batch_size=2, gradient_accumulation_steps=1, + learning_rate=2e-4, warmup_steps=0, + logging_steps=1, save_steps=1000, save_total_limit=1, + bf16=True, fp16=False, optim="adamw_torch", + lr_scheduler_type="constant", seed=42, + report_to=[], remove_unused_columns=False, + dataloader_pin_memory=False, + max_steps=5, + ) + collator = DataCollatorForLanguageModeling(tokenizer=tok, mlm=False) + trainer = Trainer(model=model, args=args, train_dataset=packed, data_collator=collator) + t0 = time.time() + trainer.train() + print(f"QLoRA-NPU finished in {time.time()-t0:.1f}s") + + adapter_dir = os.path.join(OUT, "adapter") + model.save_pretrained(adapter_dir) + tok.save_pretrained(adapter_dir) + hits = glob.glob(os.path.join(adapter_dir, "adapter_model*")) + assert hits, f"no adapter under {adapter_dir}: {sorted(os.listdir(adapter_dir))}" + print(f"QLoRA adapter: {sorted(os.listdir(adapter_dir))}") + PY +YAML + +log "C15: waiting for pod to appear..." +deadline=$((SECONDS + 120)) +POD="" +while [ "${SECONDS}" -lt "${deadline}" ]; do + POD="$(npu_kc -n "${NS}" get pod -l "job-name=${JOB_NAME}" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)" + [ -n "${POD}" ] && break + sleep 5 +done +log "C15: pod=${POD}" + +# Scheduling SKIP: no schedulable NPU slice. +sched_deadline=$((SECONDS + 300)) +phase="" +while [ "${SECONDS}" -lt "${sched_deadline}" ]; do + phase="$(npu_kc -n "${NS}" get pod "${POD}" -o jsonpath='{.status.phase}' 2>/dev/null || true)" + case "${phase}" in Running|Succeeded|Failed) break ;; esac + sleep 5 +done +if [ "${phase}" = "Pending" ] || [ -z "${phase}" ]; then + EVT="$(npu_kc -n "${NS}" get event --field-selector "involvedObject.name=${POD}" \ + -o jsonpath='{range .items[*]}{.reason}: {.message}{"\n"}{end}' 2>/dev/null | tail -3)" + log "C15: SKIP — pod ${POD} still ${phase:-} (no schedulable NPU slice):" + echo "${EVT}" | sed 's/^/ /' + exit "${E2E_SKIP_RC}" +fi + +if [ -n "${POD}" ]; then + npu_kc -n "${NS}" logs -f "${POD}" 2>&1 & + LOGS_PID=$! +fi +deadline=$((SECONDS + 2400)) +status="" +while [ "${SECONDS}" -lt "${deadline}" ]; do + status="$(npu_kc -n "${NS}" get job "${JOB_NAME}" -o jsonpath='{.status.conditions[?(@.status=="True")].type}' 2>/dev/null || true)" + case "${status}" in *Complete*) break ;; *Failed*) break ;; esac + sleep 15 +done +reap_logs "${LOGS_PID:-}" +log "C15: job status=${status}" + +# Runtime SKIP: in-pod guard exits 77 for missing NPU or unusable bnb-npu wheel. +EXIT_CODE="$(npu_kc -n "${NS}" get pod "${POD}" -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}' 2>/dev/null || true)" +if [ "${EXIT_CODE}" = "77" ]; then + log "C15: SKIP — in-pod guard signalled unsupported runtime:" + npu_kc -n "${NS}" logs "${POD}" --tail=30 2>&1 | grep -i 'E2E-SKIP' | sed 's/^/ /' || true + exit "${E2E_SKIP_RC}" +fi + +if [[ "${status}" != *Complete* ]]; then + log "C15: ==== pod final state ====" + npu_kc -n "${NS}" get pod -l "job-name=${JOB_NAME}" -o wide 2>&1 | tail -5 || true + log "C15: ==== container logs ====" + npu_kc -n "${NS}" logs -l "job-name=${JOB_NAME}" --tail=200 2>&1 || true + log "C15: ==== pod describe ====" + npu_kc -n "${NS}" describe pod -l "job-name=${JOB_NAME}" 2>&1 | tail -30 || true +fi +[[ "${status}" == *Complete* ]] diff --git a/e2e/cases/c16_cpt_npu.sh b/e2e/cases/c16_cpt_npu.sh new file mode 100755 index 0000000..1c5f829 --- /dev/null +++ b/e2e/cases/c16_cpt_npu.sh @@ -0,0 +1,288 @@ +#!/usr/bin/env bash +# C16 — Continued pre-training (CPT) on a Huawei Ascend NPU using ONLY mainline +# transformers + torch_npu. Companion of C14 (which uses training_hub on CUDA): +# training_hub doesn't run on Ascend, so the recipe uses transformers.Trainer +# with bf16 + adamw_torch on `torch.npu`. Generates a tiny synthetic Qwen2 base +# model + a synthetic raw-text corpus in-Pod so it needs no external download. +# +# Image: the same PyTorch CANN workbench image C7/C8 use. CANN env is sourced +# per the C8 pattern. NPU is requested via ${NPU_RESOURCE_NAME} (e.g. +# huawei.com/Ascend910B4), quantity ${NPU_RESOURCE_VALUE:-1}. +# +# SKIP (rc=77) conditions: +# * NPU_NAMESPACE or NPU_RESOURCE_NAME not set (opt-in — see run_all.sh); +# * the NPU slice cannot be scheduled (captured scheduler event); +# * the in-Pod NPU sanity check fails (torch.npu.is_available() is False). +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +source "${HERE}/../lib.sh" + +require_env NPU_NAMESPACE "namespace for NPU e2e resources" +require_env NPU_RESOURCE_NAME "extended resource name for one NPU, for example huawei.com/Ascend910B4" +NS="${NPU_NAMESPACE}" +JOB_NAME="c16-cpt-npu-$(printf '%05x' $$)" +IMAGE="${C16_IMAGE:-docker.io/alaudadockerhub/alauda-workbench-jupyter-pytorch-cann-py312-ubi9:v0.1.7}" +IMAGE_PULL_SECRET="${C16_IMAGE_PULL_SECRET:-${E2E_IMAGE_PULL_SECRET:-}}" +NPU_RESOURCE_VALUE="${NPU_RESOURCE_VALUE:-1}" +NPU_MEMORY_RESOURCE_NAME="${NPU_MEMORY_RESOURCE_NAME:-}" +NPU_MEMORY_RESOURCE_VALUE="${NPU_MEMORY_RESOURCE_VALUE:-8192}" +NPU_RUNTIME_CLASS="${NPU_RUNTIME_CLASS:-}" + +cleanup() { + npu_kc -n "${NS}" delete job "${JOB_NAME}" --ignore-not-found --wait=false || true +} +trap cleanup EXIT + +log "C16: submitting Job ${JOB_NAME} (image=${IMAGE})" + +cat <", "", "", ""], + ) + bpe.save_model(MODEL_DIR) + tok = GPT2TokenizerFast( + vocab_file=f"{MODEL_DIR}/vocab.json", + merges_file=f"{MODEL_DIR}/merges.txt", + unk_token="", bos_token="", eos_token="", pad_token="", + ) + tok.save_pretrained(MODEL_DIR) + cfg = Qwen2Config( + vocab_size=tok.vocab_size + 4, + hidden_size=64, num_hidden_layers=2, num_attention_heads=4, + num_key_value_heads=4, intermediate_size=128, max_position_embeddings=256, + rope_theta=10000.0, tie_word_embeddings=True, + ) + Qwen2ForCausalLM(cfg).save_pretrained(MODEL_DIR) + + # 2) Synthetic RAW-TEXT corpus — one document per line under "text". + data_path = "/workspace/test_cpt_data.jsonl" + docs = [ + "Alauda AI is an MLOps platform that runs fine-tuning and inference on Kubernetes.", + "Continued pre-training keeps the causal-LM objective and updates every weight.", + "Huawei Ascend NPUs are programmed through the CANN runtime and torch_npu.", + "A CPT run should be followed by SFT to restore instruction-following behaviour.", + ] * 8 + with open(data_path, "w") as f: + for d in docs: + json.dump({"text": d}, f); f.write("\n") + print(f"prepared base model {MODEL_DIR} and raw-text corpus {data_path}") + PY + + # 3) Continued pre-training via transformers.Trainer on torch_npu. + python - <<'PY' + import os, time, torch + import torch_npu # noqa: F401 + from transformers import ( + AutoModelForCausalLM, AutoTokenizer, + DataCollatorForLanguageModeling, + Trainer, TrainingArguments, + ) + from datasets import load_dataset + + MODEL_DIR = "/workspace/tiny-qwen2" + DATA = "/workspace/test_cpt_data.jsonl" + OUT = "/workspace/ckpt" + BLOCK = 128 + + tok = AutoTokenizer.from_pretrained(MODEL_DIR) + if tok.pad_token is None: + tok.pad_token = tok.eos_token + model = AutoModelForCausalLM.from_pretrained( + MODEL_DIR, torch_dtype=torch.bfloat16, attn_implementation="eager", + ).to("npu") + model.gradient_checkpointing_enable() + + raw = load_dataset("json", data_files=DATA, split="train") + def tokenize(batch): + return tok(batch["text"], add_special_tokens=False) + tokenized = raw.map(tokenize, batched=True, remove_columns=raw.column_names) + def group_texts(examples): + concat = {k: sum(examples[k], []) for k in examples.keys()} + n = (len(concat["input_ids"]) // BLOCK) * BLOCK + out = {k: [t[i:i+BLOCK] for i in range(0, n, BLOCK)] + for k, t in concat.items()} + out["labels"] = [ids.copy() for ids in out["input_ids"]] + return out + packed = tokenized.map(group_texts, batched=True) + print("packed chunks:", len(packed), "of", BLOCK, "tokens") + + args = TrainingArguments( + output_dir=OUT, num_train_epochs=1, + per_device_train_batch_size=2, gradient_accumulation_steps=1, + learning_rate=5e-6, warmup_steps=0, + logging_steps=1, save_steps=1000, save_total_limit=1, + bf16=True, fp16=False, optim="adamw_torch", + lr_scheduler_type="constant", seed=42, + report_to=[], remove_unused_columns=False, + dataloader_pin_memory=False, + max_steps=5, + ) + collator = DataCollatorForLanguageModeling(tokenizer=tok, mlm=False) + trainer = Trainer(model=model, args=args, train_dataset=packed, data_collator=collator) + t0 = time.time() + trainer.train() + print(f"CPT-NPU finished in {time.time()-t0:.1f}s") + + hf_dir = os.path.join(OUT, "hf_format") + trainer.save_model(hf_dir) + tok.save_pretrained(hf_dir) + files = sorted(os.listdir(hf_dir)) + print("saved:", hf_dir, files) + # Any of these is a valid HF checkpoint marker; be liberal so a + # transformers rename doesn't break the assertion. + wanted = {"model.safetensors", "pytorch_model.bin", "adapter_model.safetensors"} + assert wanted & set(files), f"no HF checkpoint under {hf_dir}: {files}" + PY +YAML + +log "C16: waiting for pod to appear..." +deadline=$((SECONDS + 120)) +POD="" +while [ "${SECONDS}" -lt "${deadline}" ]; do + POD="$(npu_kc -n "${NS}" get pod -l "job-name=${JOB_NAME}" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)" + [ -n "${POD}" ] && break + sleep 5 +done +log "C16: pod=${POD}" + +# Scheduling SKIP: no schedulable NPU slice. +sched_deadline=$((SECONDS + 300)) +phase="" +while [ "${SECONDS}" -lt "${sched_deadline}" ]; do + phase="$(npu_kc -n "${NS}" get pod "${POD}" -o jsonpath='{.status.phase}' 2>/dev/null || true)" + case "${phase}" in Running|Succeeded|Failed) break ;; esac + sleep 5 +done +if [ "${phase}" = "Pending" ] || [ -z "${phase}" ]; then + EVT="$(npu_kc -n "${NS}" get event --field-selector "involvedObject.name=${POD}" \ + -o jsonpath='{range .items[*]}{.reason}: {.message}{"\n"}{end}' 2>/dev/null | tail -3)" + log "C16: SKIP — pod ${POD} still ${phase:-} (no schedulable NPU slice):" + echo "${EVT}" | sed 's/^/ /' + exit "${E2E_SKIP_RC}" +fi + +if [ -n "${POD}" ]; then + npu_kc -n "${NS}" logs -f "${POD}" 2>&1 & + LOGS_PID=$! +fi +deadline=$((SECONDS + 2400)) +status="" +while [ "${SECONDS}" -lt "${deadline}" ]; do + status="$(npu_kc -n "${NS}" get job "${JOB_NAME}" -o jsonpath='{.status.conditions[?(@.status=="True")].type}' 2>/dev/null || true)" + case "${status}" in *Complete*) break ;; *Failed*) break ;; esac + sleep 15 +done +reap_logs "${LOGS_PID:-}" +log "C16: job status=${status}" + +# Runtime SKIP: in-pod guard exits 77 if the NPU is not visible. +EXIT_CODE="$(npu_kc -n "${NS}" get pod "${POD}" -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}' 2>/dev/null || true)" +if [ "${EXIT_CODE}" = "77" ]; then + log "C16: SKIP — in-pod guard signalled no NPU:" + npu_kc -n "${NS}" logs "${POD}" --tail=20 2>&1 | grep -i 'E2E-SKIP' | sed 's/^/ /' || true + exit "${E2E_SKIP_RC}" +fi + +if [[ "${status}" != *Complete* ]]; then + log "C16: ==== pod final state ====" + npu_kc -n "${NS}" get pod -l "job-name=${JOB_NAME}" -o wide 2>&1 | tail -5 || true + log "C16: ==== container logs ====" + npu_kc -n "${NS}" logs -l "job-name=${JOB_NAME}" --tail=200 2>&1 || true + log "C16: ==== pod describe ====" + npu_kc -n "${NS}" describe pod -l "job-name=${JOB_NAME}" 2>&1 | tail -30 || true +fi +[[ "${status}" == *Complete* ]] diff --git a/e2e/run_all.sh b/e2e/run_all.sh index 64fe194..f655e8f 100755 --- a/e2e/run_all.sh +++ b/e2e/run_all.sh @@ -41,6 +41,19 @@ CASES=( # (A30) is reserved by the persistent inference workload and no slice frees up; # the orchestrator controls A30 capacity. Same build-harbor image as C13. "C14:GPU:cases/c14_traininghub_cpt.sh" + # C15 — QLoRA (4-bit NF4 + LoRA) on Ascend NPU using transformers + peft + + # torch_npu + the community bitsandbytes-npu-beta fork (SlightwindSec). + # Self-contained (synthetic Qwen2 + chat JSONL). Opt-in via NPU_NAMESPACE + + # NPU_RESOURCE_NAME. SKIPs (rc=77) if NPU slice is unschedulable, no PyPI + # egress to install bitsandbytes-npu-beta, or the fork wheel is incompatible + # with the workbench's CANN / torch_npu combination. + "C15:NPU:cases/c15_qlora_npu.sh" + # C16 — continued pre-training (CPT) on Ascend NPU using transformers.Trainer + # on torch_npu (no MindSpeed-LLM, no HF↔MCore conversion). Self-contained + # (synthetic Qwen2 + raw-text corpus). Opt-in via NPU_NAMESPACE + + # NPU_RESOURCE_NAME. SKIPs (rc=77) if NPU slice is unschedulable or the + # in-Pod NPU sanity check fails. + "C16:NPU:cases/c16_cpt_npu.sh" ) want=( "$@" )