Skip to content

feat(rwkv): replace RWKV-4 with RWKV-7 - #12

Merged
Cai-z-us merged 7 commits into
mainfrom
feat/rwkv7
Aug 28, 2026
Merged

feat(rwkv): replace RWKV-4 with RWKV-7#12
Cai-z-us merged 7 commits into
mainfrom
feat/rwkv7

Conversation

@Cai-z-us

@Cai-z-us Cai-z-us commented Aug 28, 2026

Copy link
Copy Markdown

CPU CI GPU run-slow

What does this PR do?

Replaces the legacy RWKV-4 implementation under the existing rwkv model identity with a clean RWKV-7 integration based on the current upstream Transformers main.

Model and runtime

  • adds native RWKV-7 configuration, TimeMix, ChannelMix, recurrent cache, generation, stateful/stateless training, gradient checkpointing, and standard Transformers outputs
  • uses the public FlashRWKV2 training and inference operators and pins FlashRWKV2==0.1.0a9
  • keeps BF16 training, FP16 Albatross-style inference, and FP32 WKV state as explicit runtime contracts
  • supports the intended vanilla PEFT LoRA inference boundary and fails closed for unsupported runtime/provider combinations
  • fixes fused layer-zero dispatch, ChannelMix input ordering, residual-stream preservation, placeholder reuse, and converter shard planning

Tokenizer, conversion, and docs

  • adds the native RwkvTokenizer backed by tokenizers-rwkv
  • adds the canonical bot/assistant/tool-use Jinja chat template
  • replaces the RWKV-4 converter with a memory-bounded RWKV-7 .pth to sharded Safetensors converter
  • updates RWKV documentation, dependency metadata, AutoTokenizer registration, and focused model/tokenizer tests

This intentionally does not preserve the former RWKV-4 Python/state/checkpoint API and does not add CPU or FLA fallbacks.

An upstream RWKV-7 PR already exists at huggingface#47780, so this PR targets the project integration branch only and does not create duplicate upstream work.

Validation

  • make style
  • python -m pytest -q tests/models/rwkv/test_modeling_rwkv_mock_provider.py tests/models/rwkv/test_tokenization_rwkv.py — 13 passed
  • git diff --check
  • converted and published the 1.5B, 2.9B, 7.2B, and 13.3B g1i release artifacts after remote CUDA validation

Code Agent Policy

This PR is opened in the project fork with explicit maintainer authorization. It is not submitted to huggingface/transformers.

Before submitting

  • Documentation is updated.
  • Necessary focused tests are included.
  • Duplicate open PRs and issue ownership were checked for this repository.

@github-actions

Copy link
Copy Markdown

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

  • The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
  • A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/ — this covers files the PR deletes or renames, not only the ones it edits
  • A new high-severity security issue was detected in the changed Python files (Bandit check)
  • The PR touches a path this repository protects from untrusted PRs, such as the file that decides who reviews it — a maintainer must make that change in a separate PR

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

@github-actions

Copy link
Copy Markdown

CI recap

Dashboard: View test results in Grafana
Latest run: 33168091507
Result: failure | Grafana metrics are not available yet.

Code quality check failed: test jobs were skipped. Fix the code quality issues and push again to run tests.

@github-actions

Copy link
Copy Markdown

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, rwkv

@Cai-z-us
Cai-z-us merged commit c41eae7 into main Aug 28, 2026
28 of 34 checks passed
@Cai-z-us
Cai-z-us deleted the feat/rwkv7 branch August 28, 2026 13:02
Cai-z-us added a commit that referenced this pull request Aug 28, 2026
Replace the legacy RWKV-4 integration with RWKV-7, FlashRWKV2 a9, tokenizer, converter, tests, and documentation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant