From 561c9168e1786626c14aadca635c11bdc54bac57 Mon Sep 17 00:00:00 2001 From: yiwu Date: Wed, 8 Jul 2026 14:06:24 +0000 Subject: [PATCH] docs(training-hub): add QLoRA & CPT tutorials for Ascend NPU MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit training_hub / bitsandbytes / unsloth all target CUDA and don't run on Huawei Ascend, so the existing QLoRA and CPT tutorials don't cover NPU users. This PR adds two companion notebooks that reach the same result on Ascend using only mainline transformers + torch_npu (+ the community bitsandbytes-npu-beta fork for QLoRA), plus matching e2e cases. Docs - docs/en/training_guides/qlora-npu-tutorial.ipynb — true 4-bit NF4 QLoRA via BitsAndBytesConfig(load_in_4bit=True, ...) and peft LoRA on torch_npu. Uses the community bitsandbytes-npu-beta==0.45.3 fork (SlightwindSec/bitsandbytes), pinned deliberately; upstream re-add tracked in bitsandbytes-foundation/bitsandbytes#1695. Documents known compat caveats (fork built for torch_npu 2.1–2.4 / CANN 8.0–8.1, aarch64-only wheel), import-order requirement (torch_npu before bitsandbytes), NPU-specific gotchas (adamw_torch not paged_adamw_8bit, no CPU offload, dataloader_pin_memory=False, PYTORCH_NPU_ALLOC_CONF). - docs/en/training_guides/cpt-npu-tutorial.ipynb — continued pre-training with transformers.Trainer on torch_npu (no MindSpeed-LLM, no HF↔MCore conversion needed). Bf16 + adamw_torch; pack raw text to BLOCK_SIZE with labels=input_ids. Positioned as the light-weight single-node alternative to the MindSpeed-LLM TP/PP recipe. - training-hub-fine-tuning.mdx — link the two new notebooks; replace the "not available on NPU" QLoRA note with the community-fork path; add the light-weight vs distributed CPT-on-NPU split. E2E - e2e/cases/c15_qlora_npu.sh — NPU counterpart of C13. Same in-Pod synthetic Qwen2 + chat JSONL pattern; installs bitsandbytes-npu-beta at runtime and runs QLoRA. Skips (rc=77) on no schedulable NPU slice, no PyPI egress, or a bnb-npu wheel that can't load against the workbench's CANN/torch_npu combo. - e2e/cases/c16_cpt_npu.sh — NPU counterpart of C14. Same in-Pod synthetic base model + raw-text corpus pattern; runs CPT via transformers.Trainer on torch_npu. Skips (rc=77) on no schedulable NPU slice or failed NPU sanity check. - run_all.sh — C15/C16 wired in with NPU: prefix so they're gated by SKIP_NPU=1 and the require_env NPU_NAMESPACE inside each case. Caveats - E2E for C15/C16 was not run this session — the dedicated NPU cluster is offline (join token expired 2026-06-18) and the fallback cluster's 8× Ascend910 are fully held by another team's serving workload. Both cases are self-contained and follow the same skip/scheduling contract as C7/C8, so they will skip gracefully until an NPU host becomes available. - bitsandbytes-npu-beta on CANN 8.5.0 + torch_npu 2.9.0 is untested by the fork's author; the notebook and e2e both flag this and skip cleanly on import failure so it is discoverable but non-fatal. Co-Authored-By: Claude Opus 4.7 --- .../en/training_guides/cpt-npu-tutorial.ipynb | 355 ++++++++++++++++ .../training_guides/qlora-npu-tutorial.ipynb | 398 ++++++++++++++++++ .../training-hub-fine-tuning.mdx | 31 +- e2e/cases/c15_qlora_npu.sh | 339 +++++++++++++++ e2e/cases/c16_cpt_npu.sh | 288 +++++++++++++ e2e/run_all.sh | 13 + 6 files changed, 1417 insertions(+), 7 deletions(-) create mode 100644 docs/en/training_guides/cpt-npu-tutorial.ipynb create mode 100644 docs/en/training_guides/qlora-npu-tutorial.ipynb create mode 100755 e2e/cases/c15_qlora_npu.sh create mode 100755 e2e/cases/c16_cpt_npu.sh diff --git a/docs/en/training_guides/cpt-npu-tutorial.ipynb b/docs/en/training_guides/cpt-npu-tutorial.ipynb new file mode 100644 index 0000000..30fe501 --- /dev/null +++ b/docs/en/training_guides/cpt-npu-tutorial.ipynb @@ -0,0 +1,355 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0-title", + "metadata": {}, + "source": [ + "# Continued Pre-training (CPT) on Ascend NPU\n", + "\n", + "This notebook runs **Continued Pre-training** — plain next-token prediction over a raw-text corpus — on a Huawei Ascend NPU (e.g. `Ascend910B4`) using only mainline `transformers` + `torch_npu`. No CUDA, no `bitsandbytes`, no `training_hub`.\n", + "\n", + "> Why a separate NPU notebook? `training_hub` (the library `cpt-comprehensive-tutorial.ipynb` uses) targets NVIDIA GPUs via `torch.cuda`, `bitsandbytes`, and `unsloth`. None of those run on Ascend. The recipe here reproduces the same *training* using `transformers.Trainer` on `torch_npu`, so the resulting checkpoint is a drop-in HF-format model just like the CUDA path produces.\n", + "\n", + "**Runtime prerequisites**\n", + "\n", + "- Ascend driver + CANN runtime installed on the node (matching the workbench image's CANN version).\n", + "- Workbench image: `PyTorch CANN` (Python 3.12, CANN 8.5.0, PyTorch 2.9.0, `torch_npu 2.9.0`) or equivalent.\n", + "- At least 1 NPU visible via the Kubernetes device plugin (`huawei.com/Ascend910B4` or your cluster's resource name).\n", + "- Persistent storage for the base model + checkpoints (`/opt/app-root/src/...`)." + ] + }, + { + "cell_type": "markdown", + "id": "1-when", + "metadata": {}, + "source": [ + "## When to use CPT\n", + "\n", + "CPT keeps the original **causal-LM** objective and updates *all* weights, so it's the tool for **injecting domain knowledge** before instruction tuning. Typical pipeline:\n", + "\n", + "```\n", + "raw corpus → CPT (this notebook) → SFT / LoRA → alignment\n", + "```\n", + "\n", + "CPT catastrophically forgets general-domain skills if you run it too long on a narrow corpus. Mitigations: keep learning rate small (`1e-5`–`5e-6`), mix general-domain text into the corpus, and always follow up with SFT/OSFT." + ] + }, + { + "cell_type": "markdown", + "id": "2-env", + "metadata": {}, + "source": [ + "## Step 1 — Environment check\n", + "\n", + "Confirm `torch_npu` is importable and at least one NPU is visible. `torch_npu` monkey-patches `torch` on import so `torch.npu.*` becomes available." + ] + }, + { + "cell_type": "code", + "id": "2-env-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "# NPU allocator tuning — expandable segments avoid fragmentation on Ascend.\n", + "os.environ.setdefault(\"PYTORCH_NPU_ALLOC_CONF\", \"expandable_segments:True\")\n", + "\n", + "import torch\n", + "import torch_npu # noqa: F401 — side-effect import registers torch.npu\n", + "import transformers\n", + "\n", + "print(\"torch:\", torch.__version__)\n", + "print(\"torch_npu:\", torch_npu.__version__)\n", + "print(\"transformers:\", transformers.__version__)\n", + "print(\"NPU available:\", torch.npu.is_available())\n", + "print(\"NPU count:\", torch.npu.device_count())\n", + "if torch.npu.is_available():\n", + " torch.npu.set_device(0)\n", + " print(\"device 0:\", torch.npu.get_device_name(0))\n", + " # bf16 support is standard on Ascend910B — quick sanity check.\n", + " x = torch.randn(64, 64, dtype=torch.bfloat16, device=\"npu\")\n", + " y = (x @ x.T).float().mean().item()\n", + " print(f\"bf16 matmul ok, mean={y:.4f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "3-params", + "metadata": {}, + "source": [ + "## Step 2 — Parameters\n", + "\n", + "Edit these to point at your model and corpus. Defaults use a small base and short training so the notebook completes as a smoke test in a few minutes on a single NPU." + ] + }, + { + "cell_type": "code", + "id": "3-params-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# --- model & corpus ---\n", + "MODEL_PATH = os.environ.get(\"HF_MODEL_DIR\", \"/opt/app-root/src/models/Qwen2.5-0.5B\")\n", + "CORPUS_PATH = os.environ.get(\"CORPUS_PATH\", \"/opt/app-root/src/datasets/cpt/corpus.jsonl\")\n", + "OUTPUT_DIR = os.environ.get(\"OUTPUT_DIR\", \"/opt/app-root/src/cpt-npu-output\")\n", + "\n", + "# --- training ---\n", + "BLOCK_SIZE = 512 # chunk length used to build training samples\n", + "NUM_EPOCHS = 1\n", + "PER_DEVICE_BATCH_SIZE = 2\n", + "GRAD_ACC_STEPS = 4 # effective batch = 2 * 4 = 8 tokens/step\n", + "LEARNING_RATE = 5e-6 # keep small — CPT is easy to overfit / forget\n", + "WARMUP_STEPS = 20\n", + "SAVE_STEPS = 200\n", + "LOGGING_STEPS = 10\n", + "\n", + "# --- compute ---\n", + "USE_BF16 = True # Ascend910B supports bf16 natively\n", + "SEED = 42\n", + "\n", + "os.makedirs(OUTPUT_DIR, exist_ok=True)\n", + "print(\"model :\", MODEL_PATH)\n", + "print(\"corpus:\", CORPUS_PATH)\n", + "print(\"output:\", OUTPUT_DIR)" + ] + }, + { + "cell_type": "markdown", + "id": "4-data-format", + "metadata": {}, + "source": [ + "## Step 3 — Data format\n", + "\n", + "CPT expects **raw text**, not chat turns. Each line of the JSONL is one document under a `text` field:\n", + "\n", + "```json\n", + "{\"text\": \"Alauda AI is an MLOps platform that runs training and inference on Kubernetes.\"}\n", + "{\"text\": \"Continued pre-training adapts a base model to a new domain using unlabeled text.\"}\n", + "```\n", + "\n", + "The next cell falls back to a tiny **built-in synthetic corpus** if `CORPUS_PATH` doesn't exist, so the notebook can be run end-to-end without any dataset upload — useful as a smoke test." + ] + }, + { + "cell_type": "code", + "id": "4-data-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "from pathlib import Path\n", + "\n", + "if not Path(CORPUS_PATH).exists():\n", + " fallback = Path(OUTPUT_DIR) / \"synthetic_corpus.jsonl\"\n", + " docs = [\n", + " \"Alauda AI is an MLOps platform that runs fine-tuning and inference on Kubernetes.\",\n", + " \"Continued pre-training keeps the causal-LM objective and updates every weight.\",\n", + " \"Huawei Ascend NPUs are programmed through the CANN runtime and torch_npu.\",\n", + " \"A CPT run should be followed by SFT to restore instruction-following behaviour.\",\n", + " \"transformers.Trainer works on torch_npu out of the box once torch_npu is imported.\",\n", + " ] * 20\n", + " with fallback.open(\"w\") as f:\n", + " for d in docs:\n", + " json.dump({\"text\": d}, f); f.write(\"\\n\")\n", + " CORPUS_PATH = str(fallback)\n", + " print(f\"corpus not found; wrote synthetic fallback to {CORPUS_PATH}\")\n", + "print(\"corpus:\", CORPUS_PATH)" + ] + }, + { + "cell_type": "markdown", + "id": "5-tokenize", + "metadata": {}, + "source": [ + "## Step 4 — Load model + tokenizer, tokenize corpus\n", + "\n", + "Load the base model in `bf16` (Ascend-native), then tokenize the corpus and pack it into fixed-length chunks of `BLOCK_SIZE`. For CPT, `labels == input_ids` — every token contributes to the loss." + ] + }, + { + "cell_type": "code", + "id": "5-tokenize-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import AutoModelForCausalLM, AutoTokenizer\n", + "from datasets import load_dataset\n", + "\n", + "tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, use_fast=True)\n", + "if tokenizer.pad_token is None:\n", + " tokenizer.pad_token = tokenizer.eos_token\n", + "\n", + "model = AutoModelForCausalLM.from_pretrained(\n", + " MODEL_PATH,\n", + " torch_dtype=torch.bfloat16 if USE_BF16 else torch.float32,\n", + " attn_implementation=\"eager\", # SDPA works on NPU; eager is the safe default across CANN versions\n", + ")\n", + "model.to(\"npu\")\n", + "model.gradient_checkpointing_enable()\n", + "\n", + "raw = load_dataset(\"json\", data_files=CORPUS_PATH, split=\"train\")\n", + "print(\"raw documents:\", len(raw))\n", + "\n", + "def tokenize(batch):\n", + " return tokenizer(batch[\"text\"], add_special_tokens=False)\n", + "\n", + "tokenized = raw.map(tokenize, batched=True, remove_columns=raw.column_names)\n", + "\n", + "def group_texts(examples):\n", + " concatenated = {k: sum(examples[k], []) for k in examples.keys()}\n", + " total_len = (len(concatenated[\"input_ids\"]) // BLOCK_SIZE) * BLOCK_SIZE\n", + " result = {\n", + " k: [t[i : i + BLOCK_SIZE] for i in range(0, total_len, BLOCK_SIZE)]\n", + " for k, t in concatenated.items()\n", + " }\n", + " result[\"labels\"] = [ids.copy() for ids in result[\"input_ids\"]]\n", + " return result\n", + "\n", + "packed = tokenized.map(group_texts, batched=True)\n", + "print(\"packed chunks:\", len(packed), \"of\", BLOCK_SIZE, \"tokens each\")" + ] + }, + { + "cell_type": "markdown", + "id": "6-train", + "metadata": {}, + "source": [ + "## Step 5 — Train\n", + "\n", + "Use plain `Trainer` with `bf16=True`. **Do not** set `fp16=True` — Ascend prefers bf16, and mixing fp16 with older CANN builds triggers dynamic-shape recompilation.\n", + "\n", + "The optimizer is `adamw_torch` (mainline PyTorch). Do not use `adamw_bnb_8bit` — bitsandbytes is not upstream on NPU (see the QLoRA-NPU notebook for how to bring in the community `bitsandbytes-npu` fork if you need it)." + ] + }, + { + "cell_type": "code", + "id": "6-train-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling\n", + "\n", + "collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)\n", + "\n", + "args = TrainingArguments(\n", + " output_dir=OUTPUT_DIR,\n", + " num_train_epochs=NUM_EPOCHS,\n", + " per_device_train_batch_size=PER_DEVICE_BATCH_SIZE,\n", + " gradient_accumulation_steps=GRAD_ACC_STEPS,\n", + " learning_rate=LEARNING_RATE,\n", + " warmup_steps=WARMUP_STEPS,\n", + " logging_steps=LOGGING_STEPS,\n", + " save_steps=SAVE_STEPS,\n", + " save_total_limit=2,\n", + " bf16=USE_BF16,\n", + " fp16=False,\n", + " optim=\"adamw_torch\",\n", + " lr_scheduler_type=\"cosine\",\n", + " seed=SEED,\n", + " report_to=[],\n", + " remove_unused_columns=False,\n", + " dataloader_pin_memory=False, # pinned memory is not supported on NPU\n", + ")\n", + "\n", + "trainer = Trainer(\n", + " model=model,\n", + " args=args,\n", + " train_dataset=packed,\n", + " data_collator=collator,\n", + ")\n", + "\n", + "train_result = trainer.train()\n", + "print(train_result.metrics)" + ] + }, + { + "cell_type": "markdown", + "id": "7-save", + "metadata": {}, + "source": [ + "## Step 6 — Save the HF-format checkpoint\n", + "\n", + "Save the tuned weights and tokenizer under `OUTPUT_DIR/hf_format` so downstream SFT / inference recipes can load it with `AutoModelForCausalLM.from_pretrained(...)`." + ] + }, + { + "cell_type": "code", + "id": "7-save-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "hf_dir = os.path.join(OUTPUT_DIR, \"hf_format\")\n", + "trainer.save_model(hf_dir)\n", + "tokenizer.save_pretrained(hf_dir)\n", + "print(\"saved:\", hf_dir)\n", + "print(\"contents:\", sorted(os.listdir(hf_dir)))" + ] + }, + { + "cell_type": "markdown", + "id": "8-inference", + "metadata": {}, + "source": [ + "## Step 7 — Quick inference smoke\n", + "\n", + "Load the freshly-saved checkpoint back onto the NPU and generate a few tokens. This is a *sanity* check — the base model is small and the training corpus is tiny, so don't judge quality here." + ] + }, + { + "cell_type": "code", + "id": "8-infer-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import AutoModelForCausalLM, AutoTokenizer\n", + "\n", + "tok = AutoTokenizer.from_pretrained(hf_dir)\n", + "mdl = AutoModelForCausalLM.from_pretrained(hf_dir, torch_dtype=torch.bfloat16).to(\"npu\")\n", + "mdl.eval()\n", + "\n", + "prompt = \"Alauda AI is\"\n", + "inputs = tok(prompt, return_tensors=\"pt\").to(\"npu\")\n", + "with torch.no_grad():\n", + " out = mdl.generate(**inputs, max_new_tokens=32, do_sample=False)\n", + "print(tok.decode(out[0], skip_special_tokens=True))" + ] + }, + { + "cell_type": "markdown", + "id": "9-notes", + "metadata": {}, + "source": [ + "## Notes and gotchas on NPU\n", + "\n", + "- **Import order.** `import torch_npu` **before** the first `torch.npu.*` call — it registers the backend.\n", + "- **Precision.** Use `bf16`. Avoid `fp16` unless you know your CANN build supports it end-to-end; mixing fp16 with older builds triggers dynamic-shape retracing that stalls training.\n", + "- **Optimizer.** Stick to `adamw_torch`. `adamw_bnb_8bit` requires `bitsandbytes` — see the [QLoRA-NPU tutorial](./qlora-npu-tutorial.ipynb) for how to install the Ascend fork if you need paged optimizers.\n", + "- **Attention backend.** `attn_implementation=\"eager\"` is the portable default. `sdpa` works on most CANN 8.x builds but occasionally hits an unsupported-op path on prerelease combinations. `flash_attention_2` is CUDA-only.\n", + "- **Data loader.** Set `dataloader_pin_memory=False`; NPU does not use CUDA pinned host memory and the flag can trip guard checks.\n", + "- **Multi-NPU.** For >1 NPU on a single node, launch this notebook's script form under `torch.distributed` via `torchrun --nproc_per_node=`. `Trainer` picks up `torch.distributed` transparently on NPU thanks to `hccl`.\n", + "- **Catastrophic forgetting.** Keep `learning_rate` in the `1e-5` – `5e-6` range for CPT and always follow up with SFT — CPT alone breaks instruction-following." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/en/training_guides/qlora-npu-tutorial.ipynb b/docs/en/training_guides/qlora-npu-tutorial.ipynb new file mode 100644 index 0000000..70df73b --- /dev/null +++ b/docs/en/training_guides/qlora-npu-tutorial.ipynb @@ -0,0 +1,398 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0-title", + "metadata": {}, + "source": [ + "# QLoRA on Ascend NPU\n", + "\n", + "This notebook fine-tunes a chat model with **QLoRA — 4-bit NF4 base + LoRA adapters** — on a Huawei Ascend NPU (e.g. `Ascend910B4`) using `transformers` + `peft` + `torch_npu` + the community `bitsandbytes-npu-beta` fork. It is the NPU companion of [`qlora-comprehensive-tutorial.ipynb`](./qlora-comprehensive-tutorial.ipynb), which uses `training_hub` on CUDA.\n", + "\n", + "> **Why a separate NPU notebook?** `training_hub` targets NVIDIA GPUs via `torch.cuda`, `bitsandbytes` (CUDA build), and `unsloth`. None of those run on Ascend today. Mainline `bitsandbytes` also removed its Ascend backend from the v0.46+ install docs — upstream re-add is tracked in [`bitsandbytes-foundation/bitsandbytes#1695`](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1695). Until that merges, the practical Ascend path is the community fork [`SlightwindSec/bitsandbytes`](https://github.com/SlightwindSec/bitsandbytes), published as `bitsandbytes-npu-beta` on PyPI.\n", + "\n", + "**Support caveat.** `bitsandbytes-npu-beta==0.45.3` (2025-03-18) was built against roughly `torch_npu 2.1 – 2.4` / `CANN 8.0.RC3 – 8.1`. The Alauda `PyTorch CANN` workbench image ships `torch_npu 2.9.0` / `CANN 8.5.0`. `torch_npu` keeps loose ABI compatibility across `2.x`, but this specific combination is **not officially validated** by the fork's author. If import fails with `undefined symbol` or a dtype mismatch, pin the workbench to an older `PyTorch CANN` image (CANN 8.1 / torch_npu 2.4 range) instead.\n", + "\n", + "**Runtime prerequisites**\n", + "\n", + "- Ascend driver + CANN runtime installed on the node.\n", + "- Workbench image: `PyTorch CANN` (Python 3.12, `torch_npu`, `transformers`, `peft`) on an **aarch64** node — `bitsandbytes-npu-beta` ships only a `manylinux2014_aarch64` wheel.\n", + "- At least 1 NPU exposed as `huawei.com/Ascend910B4` (or your cluster's resource name) with ≥ 8 GB HBM available.\n", + "- Persistent storage for base model + adapter checkpoints." + ] + }, + { + "cell_type": "markdown", + "id": "1-when", + "metadata": {}, + "source": [ + "## When to use QLoRA on NPU\n", + "\n", + "- Base model too large to hold in bf16 on the NPUs you have (e.g. Qwen2.5-14B on a single 32 GB Ascend910B).\n", + "- You want a small, cheap adapter you can ship alongside the frozen base model.\n", + "- You're okay trading a little quality for a large memory saving — QLoRA is memory-efficient LoRA, not a full-parameter tune.\n", + "\n", + "If the base model already fits in bf16, use plain LoRA (`load_in_4bit=False`) — you skip the fork dependency entirely and stay on upstream `transformers + peft`." + ] + }, + { + "cell_type": "markdown", + "id": "2-install", + "metadata": {}, + "source": [ + "## Step 1 — Install `bitsandbytes-npu-beta`\n", + "\n", + "One-off install into the workbench venv. Everything else (`transformers`, `peft`, `accelerate`, `trl`, `datasets`) comes from the bundled runtime.\n", + "\n", + "```bash\n", + "pip install --no-cache-dir bitsandbytes-npu-beta==0.45.3\n", + "```\n", + "\n", + "The version pin is deliberate — `0.45.3` is the last (as of Jan 2026) published tag of the community fork. Don't upgrade past it without checking [the PyPI release page](https://pypi.org/project/bitsandbytes-npu-beta/); a newer release may or may not exist." + ] + }, + { + "cell_type": "markdown", + "id": "3-env", + "metadata": {}, + "source": [ + "## Step 2 — Environment check\n", + "\n", + "**Import order matters.** `torch_npu` must be imported **before** `bitsandbytes` so the fork detects the NPU backend." + ] + }, + { + "cell_type": "code", + "id": "3-env-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "# Cuts fragmentation on LoRA / QLoRA workloads.\n", + "os.environ.setdefault(\"PYTORCH_NPU_ALLOC_CONF\", \"expandable_segments:True\")\n", + "\n", + "import torch\n", + "import torch_npu # noqa: F401 — must precede bitsandbytes import\n", + "import bitsandbytes as bnb\n", + "import transformers, peft\n", + "\n", + "print(\"torch :\", torch.__version__)\n", + "print(\"torch_npu :\", torch_npu.__version__)\n", + "print(\"bitsandbytes :\", bnb.__version__)\n", + "print(\"transformers :\", transformers.__version__)\n", + "print(\"peft :\", peft.__version__)\n", + "assert torch.npu.is_available(), \"no NPU visible — check the device plugin and resource limits\"\n", + "torch.npu.set_device(0)\n", + "print(\"device 0 :\", torch.npu.get_device_name(0))" + ] + }, + { + "cell_type": "markdown", + "id": "4-params", + "metadata": {}, + "source": [ + "## Step 3 — Parameters\n", + "\n", + "Defaults use a small base and a short training so the notebook completes in a few minutes on a single NPU. Bump `MODEL_PATH`, `NUM_EPOCHS`, and `MAX_SEQ_LEN` for real runs." + ] + }, + { + "cell_type": "code", + "id": "4-params-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "MODEL_PATH = os.environ.get(\"HF_MODEL_DIR\", \"/opt/app-root/src/models/Qwen2.5-0.5B-Instruct\")\n", + "DATA_PATH = os.environ.get(\"DATA_PATH\", \"/opt/app-root/src/datasets/chat/train.jsonl\")\n", + "OUTPUT_DIR = os.environ.get(\"OUTPUT_DIR\", \"/opt/app-root/src/qlora-npu-output\")\n", + "\n", + "# LoRA\n", + "LORA_R = 8\n", + "LORA_ALPHA = 16\n", + "LORA_DROPOUT = 0.05\n", + "LORA_TARGETS = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"]\n", + "\n", + "# Training\n", + "NUM_EPOCHS = 1\n", + "PER_DEVICE_BATCH_SIZE = 2\n", + "GRAD_ACC_STEPS = 4 # effective batch = 8\n", + "LEARNING_RATE = 2e-4 # QLoRA can use LR much higher than full fine-tune\n", + "MAX_SEQ_LEN = 512\n", + "WARMUP_STEPS = 20\n", + "LOGGING_STEPS = 10\n", + "SAVE_STEPS = 200\n", + "SEED = 42\n", + "\n", + "os.makedirs(OUTPUT_DIR, exist_ok=True)" + ] + }, + { + "cell_type": "markdown", + "id": "5-data", + "metadata": {}, + "source": [ + "## Step 4 — Data format\n", + "\n", + "Chat JSONL — one conversation per line under `messages`:\n", + "\n", + "```json\n", + "{\"messages\": [\n", + " {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n", + " {\"role\": \"user\", \"content\": \"What is machine learning?\"},\n", + " {\"role\": \"assistant\", \"content\": \"Machine learning is ...\"}\n", + "]}\n", + "```\n", + "\n", + "The next cell falls back to a tiny built-in synthetic dataset if `DATA_PATH` doesn't exist — the notebook stays runnable as a smoke test." + ] + }, + { + "cell_type": "code", + "id": "5-data-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "from pathlib import Path\n", + "\n", + "if not Path(DATA_PATH).exists():\n", + " fallback = Path(OUTPUT_DIR) / \"synthetic_chat.jsonl\"\n", + " with fallback.open(\"w\") as f:\n", + " for _ in range(32):\n", + " json.dump({\"messages\": [\n", + " {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n", + " {\"role\": \"user\", \"content\": \"Hello, how are you?\"},\n", + " {\"role\": \"assistant\", \"content\": \"I am well — how can I help?\"},\n", + " ]}, f); f.write(\"\\n\")\n", + " DATA_PATH = str(fallback)\n", + " print(f\"data not found; wrote synthetic fallback to {DATA_PATH}\")\n", + "print(\"data:\", DATA_PATH)" + ] + }, + { + "cell_type": "markdown", + "id": "6-load-quantized", + "metadata": {}, + "source": [ + "## Step 5 — Load the base model in 4-bit\n", + "\n", + "`BitsAndBytesConfig` works unchanged against `bitsandbytes-npu-beta` — no `bnb.enable_ascend()` bootstrap needed. Route the model to `npu:0` via `device_map`." + ] + }, + { + "cell_type": "code", + "id": "6-load-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n", + "from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model\n", + "\n", + "tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, use_fast=True)\n", + "if tokenizer.pad_token is None:\n", + " tokenizer.pad_token = tokenizer.eos_token\n", + "\n", + "bnb_config = BitsAndBytesConfig(\n", + " load_in_4bit=True,\n", + " bnb_4bit_quant_type=\"nf4\",\n", + " bnb_4bit_compute_dtype=torch.bfloat16,\n", + " bnb_4bit_use_double_quant=True,\n", + ")\n", + "\n", + "model = AutoModelForCausalLM.from_pretrained(\n", + " MODEL_PATH,\n", + " quantization_config=bnb_config,\n", + " device_map={\"\": \"npu:0\"},\n", + " torch_dtype=torch.bfloat16,\n", + " attn_implementation=\"eager\", # sdpa works on most CANN 8.x builds; eager is the portable default\n", + ")\n", + "model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)\n", + "\n", + "lora_config = LoraConfig(\n", + " r=LORA_R, lora_alpha=LORA_ALPHA, lora_dropout=LORA_DROPOUT,\n", + " target_modules=LORA_TARGETS, bias=\"none\", task_type=\"CAUSAL_LM\",\n", + ")\n", + "model = get_peft_model(model, lora_config)\n", + "model.print_trainable_parameters()" + ] + }, + { + "cell_type": "markdown", + "id": "7-tokenize", + "metadata": {}, + "source": [ + "## Step 6 — Tokenize the chat corpus\n", + "\n", + "Apply the tokenizer's chat template and tokenize each conversation to `input_ids` + `labels`. For QLoRA-style SFT, the whole conversation contributes to loss (Alauda's `training_hub` default is assistant-only masking — the recipe here uses full-turn loss for portability; adapt if you need instruction-only masking)." + ] + }, + { + "cell_type": "code", + "id": "7-tok-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from datasets import load_dataset\n", + "\n", + "raw = load_dataset(\"json\", data_files=DATA_PATH, split=\"train\")\n", + "\n", + "def render(example):\n", + " text = tokenizer.apply_chat_template(example[\"messages\"], tokenize=False)\n", + " ids = tokenizer(text, truncation=True, max_length=MAX_SEQ_LEN,\n", + " padding=\"max_length\", add_special_tokens=False)\n", + " ids[\"labels\"] = ids[\"input_ids\"].copy()\n", + " return ids\n", + "\n", + "packed = raw.map(render, remove_columns=raw.column_names)\n", + "print(\"samples:\", len(packed), \"of length\", MAX_SEQ_LEN)" + ] + }, + { + "cell_type": "markdown", + "id": "8-train", + "metadata": {}, + "source": [ + "## Step 7 — Train\n", + "\n", + "Use plain `Trainer` with `bf16=True` and `optim=\"adamw_torch\"`. Do **not** use `paged_adamw_8bit` — that path relies on CUDA kernels not shipped in the NPU fork." + ] + }, + { + "cell_type": "code", + "id": "8-train-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling\n", + "\n", + "collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)\n", + "\n", + "args = TrainingArguments(\n", + " output_dir=OUTPUT_DIR,\n", + " num_train_epochs=NUM_EPOCHS,\n", + " per_device_train_batch_size=PER_DEVICE_BATCH_SIZE,\n", + " gradient_accumulation_steps=GRAD_ACC_STEPS,\n", + " learning_rate=LEARNING_RATE,\n", + " warmup_steps=WARMUP_STEPS,\n", + " logging_steps=LOGGING_STEPS,\n", + " save_steps=SAVE_STEPS,\n", + " save_total_limit=2,\n", + " bf16=True, fp16=False,\n", + " optim=\"adamw_torch\",\n", + " lr_scheduler_type=\"cosine\",\n", + " seed=SEED,\n", + " report_to=[],\n", + " remove_unused_columns=False,\n", + " dataloader_pin_memory=False, # NPU has no CUDA pinned host memory\n", + ")\n", + "\n", + "trainer = Trainer(\n", + " model=model,\n", + " args=args,\n", + " train_dataset=packed,\n", + " data_collator=collator,\n", + ")\n", + "\n", + "train_result = trainer.train()\n", + "print(train_result.metrics)" + ] + }, + { + "cell_type": "markdown", + "id": "9-save", + "metadata": {}, + "source": [ + "## Step 8 — Save the LoRA adapter\n", + "\n", + "Save only the adapter weights (a few MB) — the 4-bit base is unchanged and lives in the base-model directory. To serve the tuned model, re-attach the adapter to the fp16/bf16 base at inference time." + ] + }, + { + "cell_type": "code", + "id": "9-save-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "adapter_dir = os.path.join(OUTPUT_DIR, \"adapter\")\n", + "model.save_pretrained(adapter_dir)\n", + "tokenizer.save_pretrained(adapter_dir)\n", + "print(\"saved:\", adapter_dir)\n", + "print(\"contents:\", sorted(os.listdir(adapter_dir)))" + ] + }, + { + "cell_type": "markdown", + "id": "10-merge", + "metadata": {}, + "source": [ + "## Step 9 (optional) — Reload adapter for inference\n", + "\n", + "Load the frozen bf16 base and attach the trained adapter for a quick generation smoke. For real inference workflows you'd typically merge the adapter (`PeftModel.merge_and_unload()`) once, save the merged model, and serve the merged HF checkpoint." + ] + }, + { + "cell_type": "code", + "id": "10-merge-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import AutoModelForCausalLM, AutoTokenizer\n", + "from peft import PeftModel\n", + "\n", + "tok = AutoTokenizer.from_pretrained(MODEL_PATH)\n", + "base = AutoModelForCausalLM.from_pretrained(MODEL_PATH, torch_dtype=torch.bfloat16).to(\"npu\")\n", + "tuned = PeftModel.from_pretrained(base, adapter_dir).to(\"npu\")\n", + "tuned.eval()\n", + "\n", + "prompt = tok.apply_chat_template(\n", + " [{\"role\": \"user\", \"content\": \"Hello, how are you?\"}],\n", + " tokenize=False, add_generation_prompt=True,\n", + ")\n", + "inputs = tok(prompt, return_tensors=\"pt\").to(\"npu\")\n", + "with torch.no_grad():\n", + " out = tuned.generate(**inputs, max_new_tokens=32, do_sample=False)\n", + "print(tok.decode(out[0], skip_special_tokens=True))" + ] + }, + { + "cell_type": "markdown", + "id": "11-notes", + "metadata": {}, + "source": [ + "## Notes and gotchas on NPU QLoRA\n", + "\n", + "- **Import order.** `import torch_npu` **before** `import bitsandbytes`. If bnb is imported first, it won't register the NPU backend and 4-bit loading will fall over.\n", + "- **`bitsandbytes-npu-beta` version pinning.** `0.45.3` (2025-03-18) is the last known-good published tag. `pip install bitsandbytes` (unpinned) resolves to the upstream CUDA build — always specify the fork's PyPI name.\n", + "- **CANN / torch_npu compatibility.** The fork's wheel was built against `torch_npu 2.1 – 2.4`. If import fails on newer stacks (e.g. `torch_npu 2.9` / CANN 8.5), pin the workbench to a CANN 8.1-era `PyTorch CANN` image.\n", + "- **Optimizer.** Use `adamw_torch`. `paged_adamw_8bit` and other bnb-specific optimizers rely on CUDA kernels not present in the NPU fork.\n", + "- **CPU offload.** Do **not** set `llm_int8_enable_fp32_cpu_offload=True` — the fork does not document offload as working.\n", + "- **Attention backend.** `attn_implementation=\"eager\"` is the portable default. `sdpa` usually works on CANN 8.x. `flash_attention_2` is CUDA-only.\n", + "- **Precision.** Use `bf16`. Avoid `fp16` on the compute path — Ascend910B prefers bf16 and mixing it with the fork's dequant kernels has been reported to trip recompilation.\n", + "- **aarch64 only.** The fork ships only a `manylinux2014_aarch64` wheel; there is no x86 wheel.\n", + "- **Data loader.** Set `dataloader_pin_memory=False`; NPU has no CUDA pinned host memory.\n", + "- **Path to native support.** Track [`bitsandbytes-foundation/bitsandbytes#1695`](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1695) — once merged upstream, the fork can be retired and mainline `bitsandbytes` with `--index-url` for a matching wheel becomes the recommended path." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/en/training_guides/training-hub-fine-tuning.mdx b/docs/en/training_guides/training-hub-fine-tuning.mdx index 42f5fe5..997ecd8 100644 --- a/docs/en/training_guides/training-hub-fine-tuning.mdx +++ b/docs/en/training_guides/training-hub-fine-tuning.mdx @@ -63,7 +63,9 @@ Download into your workbench and execute cell-by-cell: | SFT comprehensive tutorial | SFT | [`sft-comprehensive-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/sft-comprehensive-tutorial.ipynb) | | OSFT comprehensive tutorial | OSFT | [`osft-comprehensive-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/osft-comprehensive-tutorial.ipynb) | | QLoRA comprehensive tutorial | QLoRA | [`qlora-comprehensive-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/qlora-comprehensive-tutorial.ipynb) | +| QLoRA on Ascend NPU | QLoRA (NPU) | [`qlora-npu-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/qlora-npu-tutorial.ipynb) | | CPT comprehensive tutorial | CPT | [`cpt-comprehensive-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/cpt-comprehensive-tutorial.ipynb) | +| CPT on Ascend NPU | CPT (NPU) | [`cpt-npu-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/cpt-npu-tutorial.ipynb) | Install and configure: @@ -199,9 +201,19 @@ The output is a **LoRA adapter**, not a full checkpoint. To serve, load base + a `trl`, `peft`, and `bitsandbytes`. The default `lora_sft` backend is `unsloth`; for full control you can also drive `peft` + `bitsandbytes` directly on the same runtime image. -**NPU:** bitsandbytes 4-bit is not available on Huawei Ascend. Use LoRA (without 4-bit) on -the `llamafactory0.9-cann8.5-arm64` runtime (`finetuning_type: lora`) as the -parameter-efficient path — see [Fine-tune and Pretrain on Ascend NPU](./fine-tune-and-pretrain-llms-on-ascend-npu.mdx). +**NPU:** `training_hub` and mainline `bitsandbytes` do not target Huawei Ascend. Two paths on +NPU: + +- **True 4-bit NF4 QLoRA** via the community `bitsandbytes-npu-beta` fork + ([SlightwindSec/bitsandbytes](https://github.com/SlightwindSec/bitsandbytes), tracked + upstream in [PR #1695](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1695)) — + see [`qlora-npu-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/qlora-npu-tutorial.ipynb). Uses standard + `transformers` + `peft` + `torch_npu`; the API is drop-in, but the wheel is only tested + against `torch_npu 2.1–2.4` / CANN 8.0–8.1 and compatibility with newer CANN builds is + best-effort. +- **Plain LoRA (no 4-bit)** on the `llamafactory0.9-cann8.5-arm64` runtime + (`finetuning_type: lora`) — see [Fine-tune and Pretrain on Ascend NPU](./fine-tune-and-pretrain-llms-on-ascend-npu.mdx). + Slightly higher memory footprint than QLoRA, but no non-mainline dependencies. ::: ## Continued pre-training (CPT) \{#cpt} @@ -254,10 +266,15 @@ general-domain text, and by following up with an SFT/OSFT pass (point `model_pat checkpoint). If forgetting is the primary concern, prefer **OSFT**. :::note -**NPU:** Full-parameter continued pre-training also runs on Huawei Ascend via the -`MindSpeed-LLM` runtime (`pretrain_gpt.py`). See -[Fine-tune and Pretrain on Ascend NPU](./fine-tune-and-pretrain-llms-on-ascend-npu.mdx) and -the `qwen25_pretrain_verify.ipynb` recipe. +**NPU:** Two paths for CPT on Ascend: + +- **Light-weight, single-node** CPT with `transformers.Trainer` on `torch_npu` — + see [`cpt-npu-tutorial.ipynb`](https://github.com/alauda/aml-docs/tree/master/docs/en/training_guides/cpt-npu-tutorial.ipynb). Uses only mainline + `transformers` + `torch_npu` (no `MindSpeed-LLM`, no HF↔MCore conversion). Best when the + model fits in bf16 on the NPUs you have and you don't need TP/PP. +- **Distributed, TP/PP-aware** CPT via the `MindSpeed-LLM` runtime (`pretrain_gpt.py`) — + see [Fine-tune and Pretrain on Ascend NPU](./fine-tune-and-pretrain-llms-on-ascend-npu.mdx) + and the `qwen25_pretrain_verify.ipynb` recipe. ::: ## Multi-node diff --git a/e2e/cases/c15_qlora_npu.sh b/e2e/cases/c15_qlora_npu.sh new file mode 100755 index 0000000..ab7da92 --- /dev/null +++ b/e2e/cases/c15_qlora_npu.sh @@ -0,0 +1,339 @@ +#!/usr/bin/env bash +# C15 — QLoRA (4-bit NF4 + LoRA adapters) on a Huawei Ascend NPU using +# transformers + peft + torch_npu + the community bitsandbytes-npu-beta +# fork. Companion of C13 (which uses training_hub on CUDA): +# training_hub / mainline bitsandbytes do not run on Ascend, so the recipe +# uses `BitsAndBytesConfig(load_in_4bit=True, ...)` with the +# `bitsandbytes-npu-beta` PyPI package. Generates a tiny synthetic Qwen2 base +# model + a synthetic chat JSONL in-Pod so it needs no external download. +# +# Image: the same PyTorch CANN workbench image C7/C8/C16 use. CANN env is +# sourced per the C8 pattern. NPU is requested via ${NPU_RESOURCE_NAME} (e.g. +# huawei.com/Ascend910B4), quantity ${NPU_RESOURCE_VALUE:-1}. +# +# SKIP (rc=77) conditions: +# * NPU_NAMESPACE or NPU_RESOURCE_NAME not set (opt-in — see run_all.sh); +# * the NPU slice cannot be scheduled (captured scheduler event); +# * the in-Pod NPU sanity check fails (torch.npu.is_available() False); +# * `import bitsandbytes` fails after `pip install bitsandbytes-npu-beta` — +# the fork is torch_npu 2.1–2.4 / CANN 8.0–8.1 era, and CANN 8.5+ compat +# is not officially validated by the fork's author (see +# `qlora-npu-tutorial.ipynb` for the compatibility caveat). +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +source "${HERE}/../lib.sh" + +require_env NPU_NAMESPACE "namespace for NPU e2e resources" +require_env NPU_RESOURCE_NAME "extended resource name for one NPU, for example huawei.com/Ascend910B4" +NS="${NPU_NAMESPACE}" +JOB_NAME="c15-qlora-npu-$(printf '%05x' $$)" +IMAGE="${C15_IMAGE:-docker.io/alaudadockerhub/alauda-workbench-jupyter-pytorch-cann-py312-ubi9:v0.1.7}" +IMAGE_PULL_SECRET="${C15_IMAGE_PULL_SECRET:-${E2E_IMAGE_PULL_SECRET:-}}" +NPU_RESOURCE_VALUE="${NPU_RESOURCE_VALUE:-1}" +NPU_MEMORY_RESOURCE_NAME="${NPU_MEMORY_RESOURCE_NAME:-}" +NPU_MEMORY_RESOURCE_VALUE="${NPU_MEMORY_RESOURCE_VALUE:-8192}" +NPU_RUNTIME_CLASS="${NPU_RUNTIME_CLASS:-}" +# The fork's PyPI name & pinned version. See qlora-npu-tutorial.ipynb. +BNB_NPU_PIN="${BNB_NPU_PIN:-bitsandbytes-npu-beta==0.45.3}" +PIP_INDEX_URL="${PIP_INDEX_URL:-}" + +cleanup() { + npu_kc -n "${NS}" delete job "${JOB_NAME}" --ignore-not-found --wait=false || true +} +trap cleanup EXIT + +log "C15: submitting Job ${JOB_NAME} (image=${IMAGE}, bnb=${BNB_NPU_PIN})" + +cat <&2 + exit 77 + fi + python - <<'PY' + import sys + try: + import torch_npu # MUST be first + import bitsandbytes as bnb + print("bitsandbytes:", bnb.__version__) + except Exception as e: + print(f"E2E-SKIP: bitsandbytes-npu-beta import failed: " + f"{type(e).__name__}: {e}", file=sys.stderr) + sys.exit(77) + PY + + # 1) Tiny synthetic Qwen2-style base model + tokenizer (offline). + python - <<'PY' + import os, json, torch + from transformers import GPT2TokenizerFast, Qwen2Config, Qwen2ForCausalLM + from tokenizers import ByteLevelBPETokenizer + + MODEL_DIR = "/workspace/tiny-qwen2" + os.makedirs(MODEL_DIR, exist_ok=True) + torch.manual_seed(0) + bpe = ByteLevelBPETokenizer() + bpe.train_from_iterator( + ["hello world", "the quick brown fox", "alauda ai qlora on ascend"], + vocab_size=512, min_frequency=1, + special_tokens=["", "", "", ""], + ) + bpe.save_model(MODEL_DIR) + tok = GPT2TokenizerFast( + vocab_file=f"{MODEL_DIR}/vocab.json", + merges_file=f"{MODEL_DIR}/merges.txt", + unk_token="", bos_token="", eos_token="", pad_token="", + ) + tok.chat_template = "{% for m in messages %}{{ m.role }}: {{ m.content }}\n{% endfor %}" + tok.save_pretrained(MODEL_DIR) + cfg = Qwen2Config( + vocab_size=tok.vocab_size + 4, + hidden_size=64, num_hidden_layers=2, num_attention_heads=4, + num_key_value_heads=4, intermediate_size=128, max_position_embeddings=256, + rope_theta=10000.0, tie_word_embeddings=True, + ) + Qwen2ForCausalLM(cfg).save_pretrained(MODEL_DIR) + + data_path = "/workspace/test_qlora_data.jsonl" + with open(data_path, "w") as f: + for _ in range(16): + json.dump({"messages": [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Hello, how are you?"}, + {"role": "assistant", "content": "I am well - how can I help?"}, + ]}, f); f.write("\n") + print(f"prepared {MODEL_DIR} and {data_path}") + PY + + # 2) QLoRA: BitsAndBytesConfig(load_in_4bit=...) + peft LoRA. + # Order: torch_npu first, then bitsandbytes. + python - <<'PY' + import os, sys, time, glob, torch + import torch_npu # noqa: F401 — must precede bitsandbytes + try: + import bitsandbytes as bnb # noqa: F401 + except Exception as e: + print(f"E2E-SKIP: bitsandbytes-npu-beta unusable at runtime: " + f"{type(e).__name__}: {e}", file=sys.stderr) + sys.exit(77) + + from transformers import ( + AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, + DataCollatorForLanguageModeling, Trainer, TrainingArguments, + ) + from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model + from datasets import load_dataset + + MODEL_DIR = "/workspace/tiny-qwen2" + DATA = "/workspace/test_qlora_data.jsonl" + OUT = "/workspace/ckpt" + + tok = AutoTokenizer.from_pretrained(MODEL_DIR) + if tok.pad_token is None: + tok.pad_token = tok.eos_token + + bnb_cfg = BitsAndBytesConfig( + load_in_4bit=True, bnb_4bit_quant_type="nf4", + bnb_4bit_compute_dtype=torch.bfloat16, + bnb_4bit_use_double_quant=True, + ) + try: + model = AutoModelForCausalLM.from_pretrained( + MODEL_DIR, quantization_config=bnb_cfg, + device_map={"": "npu:0"}, + torch_dtype=torch.bfloat16, + attn_implementation="eager", + ) + except Exception as e: + print(f"E2E-SKIP: 4-bit load on NPU failed " + f"({type(e).__name__}: {e})", file=sys.stderr) + sys.exit(77) + model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True) + peft_cfg = LoraConfig( + r=8, lora_alpha=16, lora_dropout=0.05, bias="none", + target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], + task_type="CAUSAL_LM", + ) + model = get_peft_model(model, peft_cfg) + model.print_trainable_parameters() + + raw = load_dataset("json", data_files=DATA, split="train") + def render(ex): + text = tok.apply_chat_template(ex["messages"], tokenize=False) + out = tok(text, truncation=True, max_length=128, + padding="max_length", add_special_tokens=False) + out["labels"] = out["input_ids"].copy() + return out + packed = raw.map(render, remove_columns=raw.column_names) + + args = TrainingArguments( + output_dir=OUT, num_train_epochs=1, + per_device_train_batch_size=2, gradient_accumulation_steps=1, + learning_rate=2e-4, warmup_steps=0, + logging_steps=1, save_steps=1000, save_total_limit=1, + bf16=True, fp16=False, optim="adamw_torch", + lr_scheduler_type="constant", seed=42, + report_to=[], remove_unused_columns=False, + dataloader_pin_memory=False, + max_steps=5, + ) + collator = DataCollatorForLanguageModeling(tokenizer=tok, mlm=False) + trainer = Trainer(model=model, args=args, train_dataset=packed, data_collator=collator) + t0 = time.time() + trainer.train() + print(f"QLoRA-NPU finished in {time.time()-t0:.1f}s") + + adapter_dir = os.path.join(OUT, "adapter") + model.save_pretrained(adapter_dir) + tok.save_pretrained(adapter_dir) + hits = glob.glob(os.path.join(adapter_dir, "adapter_model*")) + assert hits, f"no adapter under {adapter_dir}: {sorted(os.listdir(adapter_dir))}" + print(f"QLoRA adapter: {sorted(os.listdir(adapter_dir))}") + PY +YAML + +log "C15: waiting for pod to appear..." +deadline=$((SECONDS + 120)) +POD="" +while [ "${SECONDS}" -lt "${deadline}" ]; do + POD="$(npu_kc -n "${NS}" get pod -l "job-name=${JOB_NAME}" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)" + [ -n "${POD}" ] && break + sleep 5 +done +log "C15: pod=${POD}" + +# Scheduling SKIP: no schedulable NPU slice. +sched_deadline=$((SECONDS + 300)) +phase="" +while [ "${SECONDS}" -lt "${sched_deadline}" ]; do + phase="$(npu_kc -n "${NS}" get pod "${POD}" -o jsonpath='{.status.phase}' 2>/dev/null || true)" + case "${phase}" in Running|Succeeded|Failed) break ;; esac + sleep 5 +done +if [ "${phase}" = "Pending" ] || [ -z "${phase}" ]; then + EVT="$(npu_kc -n "${NS}" get event --field-selector "involvedObject.name=${POD}" \ + -o jsonpath='{range .items[*]}{.reason}: {.message}{"\n"}{end}' 2>/dev/null | tail -3)" + log "C15: SKIP — pod ${POD} still ${phase:-} (no schedulable NPU slice):" + echo "${EVT}" | sed 's/^/ /' + exit "${E2E_SKIP_RC}" +fi + +if [ -n "${POD}" ]; then + npu_kc -n "${NS}" logs -f "${POD}" 2>&1 & + LOGS_PID=$! +fi +deadline=$((SECONDS + 2400)) +status="" +while [ "${SECONDS}" -lt "${deadline}" ]; do + status="$(npu_kc -n "${NS}" get job "${JOB_NAME}" -o jsonpath='{.status.conditions[?(@.status=="True")].type}' 2>/dev/null || true)" + case "${status}" in *Complete*) break ;; *Failed*) break ;; esac + sleep 15 +done +reap_logs "${LOGS_PID:-}" +log "C15: job status=${status}" + +# Runtime SKIP: in-pod guard exits 77 for missing NPU or unusable bnb-npu wheel. +EXIT_CODE="$(npu_kc -n "${NS}" get pod "${POD}" -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}' 2>/dev/null || true)" +if [ "${EXIT_CODE}" = "77" ]; then + log "C15: SKIP — in-pod guard signalled unsupported runtime:" + npu_kc -n "${NS}" logs "${POD}" --tail=30 2>&1 | grep -i 'E2E-SKIP' | sed 's/^/ /' || true + exit "${E2E_SKIP_RC}" +fi + +if [[ "${status}" != *Complete* ]]; then + log "C15: ==== pod final state ====" + npu_kc -n "${NS}" get pod -l "job-name=${JOB_NAME}" -o wide 2>&1 | tail -5 || true + log "C15: ==== container logs ====" + npu_kc -n "${NS}" logs -l "job-name=${JOB_NAME}" --tail=200 2>&1 || true + log "C15: ==== pod describe ====" + npu_kc -n "${NS}" describe pod -l "job-name=${JOB_NAME}" 2>&1 | tail -30 || true +fi +[[ "${status}" == *Complete* ]] diff --git a/e2e/cases/c16_cpt_npu.sh b/e2e/cases/c16_cpt_npu.sh new file mode 100755 index 0000000..1c5f829 --- /dev/null +++ b/e2e/cases/c16_cpt_npu.sh @@ -0,0 +1,288 @@ +#!/usr/bin/env bash +# C16 — Continued pre-training (CPT) on a Huawei Ascend NPU using ONLY mainline +# transformers + torch_npu. Companion of C14 (which uses training_hub on CUDA): +# training_hub doesn't run on Ascend, so the recipe uses transformers.Trainer +# with bf16 + adamw_torch on `torch.npu`. Generates a tiny synthetic Qwen2 base +# model + a synthetic raw-text corpus in-Pod so it needs no external download. +# +# Image: the same PyTorch CANN workbench image C7/C8 use. CANN env is sourced +# per the C8 pattern. NPU is requested via ${NPU_RESOURCE_NAME} (e.g. +# huawei.com/Ascend910B4), quantity ${NPU_RESOURCE_VALUE:-1}. +# +# SKIP (rc=77) conditions: +# * NPU_NAMESPACE or NPU_RESOURCE_NAME not set (opt-in — see run_all.sh); +# * the NPU slice cannot be scheduled (captured scheduler event); +# * the in-Pod NPU sanity check fails (torch.npu.is_available() is False). +set -euo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +source "${HERE}/../lib.sh" + +require_env NPU_NAMESPACE "namespace for NPU e2e resources" +require_env NPU_RESOURCE_NAME "extended resource name for one NPU, for example huawei.com/Ascend910B4" +NS="${NPU_NAMESPACE}" +JOB_NAME="c16-cpt-npu-$(printf '%05x' $$)" +IMAGE="${C16_IMAGE:-docker.io/alaudadockerhub/alauda-workbench-jupyter-pytorch-cann-py312-ubi9:v0.1.7}" +IMAGE_PULL_SECRET="${C16_IMAGE_PULL_SECRET:-${E2E_IMAGE_PULL_SECRET:-}}" +NPU_RESOURCE_VALUE="${NPU_RESOURCE_VALUE:-1}" +NPU_MEMORY_RESOURCE_NAME="${NPU_MEMORY_RESOURCE_NAME:-}" +NPU_MEMORY_RESOURCE_VALUE="${NPU_MEMORY_RESOURCE_VALUE:-8192}" +NPU_RUNTIME_CLASS="${NPU_RUNTIME_CLASS:-}" + +cleanup() { + npu_kc -n "${NS}" delete job "${JOB_NAME}" --ignore-not-found --wait=false || true +} +trap cleanup EXIT + +log "C16: submitting Job ${JOB_NAME} (image=${IMAGE})" + +cat <", "", "", ""], + ) + bpe.save_model(MODEL_DIR) + tok = GPT2TokenizerFast( + vocab_file=f"{MODEL_DIR}/vocab.json", + merges_file=f"{MODEL_DIR}/merges.txt", + unk_token="", bos_token="", eos_token="", pad_token="", + ) + tok.save_pretrained(MODEL_DIR) + cfg = Qwen2Config( + vocab_size=tok.vocab_size + 4, + hidden_size=64, num_hidden_layers=2, num_attention_heads=4, + num_key_value_heads=4, intermediate_size=128, max_position_embeddings=256, + rope_theta=10000.0, tie_word_embeddings=True, + ) + Qwen2ForCausalLM(cfg).save_pretrained(MODEL_DIR) + + # 2) Synthetic RAW-TEXT corpus — one document per line under "text". + data_path = "/workspace/test_cpt_data.jsonl" + docs = [ + "Alauda AI is an MLOps platform that runs fine-tuning and inference on Kubernetes.", + "Continued pre-training keeps the causal-LM objective and updates every weight.", + "Huawei Ascend NPUs are programmed through the CANN runtime and torch_npu.", + "A CPT run should be followed by SFT to restore instruction-following behaviour.", + ] * 8 + with open(data_path, "w") as f: + for d in docs: + json.dump({"text": d}, f); f.write("\n") + print(f"prepared base model {MODEL_DIR} and raw-text corpus {data_path}") + PY + + # 3) Continued pre-training via transformers.Trainer on torch_npu. + python - <<'PY' + import os, time, torch + import torch_npu # noqa: F401 + from transformers import ( + AutoModelForCausalLM, AutoTokenizer, + DataCollatorForLanguageModeling, + Trainer, TrainingArguments, + ) + from datasets import load_dataset + + MODEL_DIR = "/workspace/tiny-qwen2" + DATA = "/workspace/test_cpt_data.jsonl" + OUT = "/workspace/ckpt" + BLOCK = 128 + + tok = AutoTokenizer.from_pretrained(MODEL_DIR) + if tok.pad_token is None: + tok.pad_token = tok.eos_token + model = AutoModelForCausalLM.from_pretrained( + MODEL_DIR, torch_dtype=torch.bfloat16, attn_implementation="eager", + ).to("npu") + model.gradient_checkpointing_enable() + + raw = load_dataset("json", data_files=DATA, split="train") + def tokenize(batch): + return tok(batch["text"], add_special_tokens=False) + tokenized = raw.map(tokenize, batched=True, remove_columns=raw.column_names) + def group_texts(examples): + concat = {k: sum(examples[k], []) for k in examples.keys()} + n = (len(concat["input_ids"]) // BLOCK) * BLOCK + out = {k: [t[i:i+BLOCK] for i in range(0, n, BLOCK)] + for k, t in concat.items()} + out["labels"] = [ids.copy() for ids in out["input_ids"]] + return out + packed = tokenized.map(group_texts, batched=True) + print("packed chunks:", len(packed), "of", BLOCK, "tokens") + + args = TrainingArguments( + output_dir=OUT, num_train_epochs=1, + per_device_train_batch_size=2, gradient_accumulation_steps=1, + learning_rate=5e-6, warmup_steps=0, + logging_steps=1, save_steps=1000, save_total_limit=1, + bf16=True, fp16=False, optim="adamw_torch", + lr_scheduler_type="constant", seed=42, + report_to=[], remove_unused_columns=False, + dataloader_pin_memory=False, + max_steps=5, + ) + collator = DataCollatorForLanguageModeling(tokenizer=tok, mlm=False) + trainer = Trainer(model=model, args=args, train_dataset=packed, data_collator=collator) + t0 = time.time() + trainer.train() + print(f"CPT-NPU finished in {time.time()-t0:.1f}s") + + hf_dir = os.path.join(OUT, "hf_format") + trainer.save_model(hf_dir) + tok.save_pretrained(hf_dir) + files = sorted(os.listdir(hf_dir)) + print("saved:", hf_dir, files) + # Any of these is a valid HF checkpoint marker; be liberal so a + # transformers rename doesn't break the assertion. + wanted = {"model.safetensors", "pytorch_model.bin", "adapter_model.safetensors"} + assert wanted & set(files), f"no HF checkpoint under {hf_dir}: {files}" + PY +YAML + +log "C16: waiting for pod to appear..." +deadline=$((SECONDS + 120)) +POD="" +while [ "${SECONDS}" -lt "${deadline}" ]; do + POD="$(npu_kc -n "${NS}" get pod -l "job-name=${JOB_NAME}" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)" + [ -n "${POD}" ] && break + sleep 5 +done +log "C16: pod=${POD}" + +# Scheduling SKIP: no schedulable NPU slice. +sched_deadline=$((SECONDS + 300)) +phase="" +while [ "${SECONDS}" -lt "${sched_deadline}" ]; do + phase="$(npu_kc -n "${NS}" get pod "${POD}" -o jsonpath='{.status.phase}' 2>/dev/null || true)" + case "${phase}" in Running|Succeeded|Failed) break ;; esac + sleep 5 +done +if [ "${phase}" = "Pending" ] || [ -z "${phase}" ]; then + EVT="$(npu_kc -n "${NS}" get event --field-selector "involvedObject.name=${POD}" \ + -o jsonpath='{range .items[*]}{.reason}: {.message}{"\n"}{end}' 2>/dev/null | tail -3)" + log "C16: SKIP — pod ${POD} still ${phase:-} (no schedulable NPU slice):" + echo "${EVT}" | sed 's/^/ /' + exit "${E2E_SKIP_RC}" +fi + +if [ -n "${POD}" ]; then + npu_kc -n "${NS}" logs -f "${POD}" 2>&1 & + LOGS_PID=$! +fi +deadline=$((SECONDS + 2400)) +status="" +while [ "${SECONDS}" -lt "${deadline}" ]; do + status="$(npu_kc -n "${NS}" get job "${JOB_NAME}" -o jsonpath='{.status.conditions[?(@.status=="True")].type}' 2>/dev/null || true)" + case "${status}" in *Complete*) break ;; *Failed*) break ;; esac + sleep 15 +done +reap_logs "${LOGS_PID:-}" +log "C16: job status=${status}" + +# Runtime SKIP: in-pod guard exits 77 if the NPU is not visible. +EXIT_CODE="$(npu_kc -n "${NS}" get pod "${POD}" -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}' 2>/dev/null || true)" +if [ "${EXIT_CODE}" = "77" ]; then + log "C16: SKIP — in-pod guard signalled no NPU:" + npu_kc -n "${NS}" logs "${POD}" --tail=20 2>&1 | grep -i 'E2E-SKIP' | sed 's/^/ /' || true + exit "${E2E_SKIP_RC}" +fi + +if [[ "${status}" != *Complete* ]]; then + log "C16: ==== pod final state ====" + npu_kc -n "${NS}" get pod -l "job-name=${JOB_NAME}" -o wide 2>&1 | tail -5 || true + log "C16: ==== container logs ====" + npu_kc -n "${NS}" logs -l "job-name=${JOB_NAME}" --tail=200 2>&1 || true + log "C16: ==== pod describe ====" + npu_kc -n "${NS}" describe pod -l "job-name=${JOB_NAME}" 2>&1 | tail -30 || true +fi +[[ "${status}" == *Complete* ]] diff --git a/e2e/run_all.sh b/e2e/run_all.sh index 64fe194..f655e8f 100755 --- a/e2e/run_all.sh +++ b/e2e/run_all.sh @@ -41,6 +41,19 @@ CASES=( # (A30) is reserved by the persistent inference workload and no slice frees up; # the orchestrator controls A30 capacity. Same build-harbor image as C13. "C14:GPU:cases/c14_traininghub_cpt.sh" + # C15 — QLoRA (4-bit NF4 + LoRA) on Ascend NPU using transformers + peft + + # torch_npu + the community bitsandbytes-npu-beta fork (SlightwindSec). + # Self-contained (synthetic Qwen2 + chat JSONL). Opt-in via NPU_NAMESPACE + + # NPU_RESOURCE_NAME. SKIPs (rc=77) if NPU slice is unschedulable, no PyPI + # egress to install bitsandbytes-npu-beta, or the fork wheel is incompatible + # with the workbench's CANN / torch_npu combination. + "C15:NPU:cases/c15_qlora_npu.sh" + # C16 — continued pre-training (CPT) on Ascend NPU using transformers.Trainer + # on torch_npu (no MindSpeed-LLM, no HF↔MCore conversion). Self-contained + # (synthetic Qwen2 + raw-text corpus). Opt-in via NPU_NAMESPACE + + # NPU_RESOURCE_NAME. SKIPs (rc=77) if NPU slice is unschedulable or the + # in-Pod NPU sanity check fails. + "C16:NPU:cases/c16_cpt_npu.sh" ) want=( "$@" )