-
Notifications
You must be signed in to change notification settings - Fork 0
docs: remove the simd feature and document table-driven packed encoding #505
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -454,6 +454,8 @@ Location: `wacore/binary/src/encoder.rs:227-237` | |||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ### Packed Encoding | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| Nibble and hex packing share one code path, `write_packed_bytes`: an ASCII→nibble lookup table (`NIBBLE_ENC` or `HEX_ENC`, picked by `data_type`) maps each input byte to its packed nibble, pairs are packed two at a time into a stack buffer, and validity is checked once via an OR accumulator (`seen`) instead of per pair — that is what lets the pair loop unroll. | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| #### Nibble packing (numeric strings) | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| Strings containing only digits, dash, and dot are packed into 4 bits per character: | ||||||||||||||||||||||||||
|
|
@@ -466,58 +468,55 @@ Strings containing only digits, dash, and dot are packed into 4 bits per charact | |||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| pub const PACKED_MAX: u8 = 127; // Max length for packed/token strings | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| fn pack_nibble(value: u8) -> u8 { | ||||||||||||||||||||||||||
| match value { | ||||||||||||||||||||||||||
| b'-' => 10, | ||||||||||||||||||||||||||
| b'.' => 11, | ||||||||||||||||||||||||||
| 0 => 15, // Padding | ||||||||||||||||||||||||||
| c if c.is_ascii_digit() => c - b'0', | ||||||||||||||||||||||||||
| _ => panic!("Invalid nibble"), | ||||||||||||||||||||||||||
| /// ASCII to nibble for `NIBBLE_8`: digits plus the two punctuation characters | ||||||||||||||||||||||||||
| /// a phone number can carry. | ||||||||||||||||||||||||||
| static NIBBLE_ENC: [u8; 256] = { | ||||||||||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: The two lookup-table excerpts are internally inconsistent: the comment claims digits/A-F map to 0-15, but the shown initializers never set those entries, so any reader copying the snippet gets a table where digits resolve to PACK_INVALID — contradicting the "'1' → 1" / "Packed: 0x12" example in the same block. Add the digit and A-F mapping loops (or drop the misleading comments) to both NIBBLE_ENC and HEX_ENC. Prompt for AI agents
Suggested change
|
||||||||||||||||||||||||||
| let mut table = [PACK_INVALID; 256]; | ||||||||||||||||||||||||||
| let mut c = b'0'; | ||||||||||||||||||||||||||
| while c <= b'9' { | ||||||||||||||||||||||||||
| table[c as usize] = c - b'0'; | ||||||||||||||||||||||||||
| c += 1; | ||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||
| table[b'-' as usize] = 10; | ||||||||||||||||||||||||||
| table[b'.' as usize] = 11; | ||||||||||||||||||||||||||
| table[0] = 15; // padding for an odd-length string | ||||||||||||||||||||||||||
| table | ||||||||||||||||||||||||||
|
coderabbitai[bot] marked this conversation as resolved.
|
||||||||||||||||||||||||||
| }; | ||||||||||||||||||||||||||
| ``` | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| Location: `wacore/binary/src/encoder.rs:769-777` | ||||||||||||||||||||||||||
| Location: `wacore/binary/src/encoder.rs:32-43` | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| #### Hex Packing | ||||||||||||||||||||||||||
| #### Hex packing | ||||||||||||||||||||||||||
|
cubic-dev-ai[bot] marked this conversation as resolved.
|
||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| Uppercase hex strings (0-9, A-F) are packed into 4 bits per character: | ||||||||||||||||||||||||||
| Uppercase hex strings (0-9, A-F) are packed into 4 bits per character, against `HEX_ENC` instead of `NIBBLE_ENC`: | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ```rust | ||||||||||||||||||||||||||
| // Input: "DEADBEEF" | ||||||||||||||||||||||||||
| // Packed: 0xDE, 0xAD, 0xBE, 0xEF | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| fn pack_hex(value: u8) -> u8 { | ||||||||||||||||||||||||||
| match value { | ||||||||||||||||||||||||||
| c if c.is_ascii_digit() => c - b'0', | ||||||||||||||||||||||||||
| c if (b'A'..=b'F').contains(&c) => 10 + (c - b'A'), | ||||||||||||||||||||||||||
| 0 => 15, // Padding | ||||||||||||||||||||||||||
| _ => panic!("Invalid hex"), | ||||||||||||||||||||||||||
| /// ASCII to nibble, the inverse of the decoder's `HEX_PAIRS`. | ||||||||||||||||||||||||||
| static HEX_ENC: [u8; 256] = { | ||||||||||||||||||||||||||
| let mut table = [PACK_INVALID; 256]; | ||||||||||||||||||||||||||
| let mut c = b'0'; | ||||||||||||||||||||||||||
| while c <= b'9' { | ||||||||||||||||||||||||||
| table[c as usize] = c - b'0'; | ||||||||||||||||||||||||||
| c += 1; | ||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||
| let mut c = b'A'; | ||||||||||||||||||||||||||
| while c <= b'F' { | ||||||||||||||||||||||||||
| table[c as usize] = 10 + (c - b'A'); | ||||||||||||||||||||||||||
| c += 1; | ||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||
| table[0] = 15; // padding for an odd-length string | ||||||||||||||||||||||||||
| table | ||||||||||||||||||||||||||
| }; | ||||||||||||||||||||||||||
| ``` | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| Location: `wacore/binary/src/encoder.rs:780-787` | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| #### SIMD Optimization | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| The encoder uses SIMD instructions for fast packing of long strings: | ||||||||||||||||||||||||||
| Location: `wacore/binary/src/encoder.rs:14-28` | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ```rust | ||||||||||||||||||||||||||
| while input_bytes.len() >= 16 { | ||||||||||||||||||||||||||
| let input = u8x16::from_slice(chunk); | ||||||||||||||||||||||||||
| let indices = input.saturating_sub(nibble_base); | ||||||||||||||||||||||||||
| let nibbles = lookup.swizzle_dyn(indices); | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| let (evens, odds) = nibbles.deinterleave( | ||||||||||||||||||||||||||
| nibbles.rotate_elements_left::<1>() | ||||||||||||||||||||||||||
| ); | ||||||||||||||||||||||||||
| let packed = (evens << Simd::splat(4)) | odds; | ||||||||||||||||||||||||||
| self.write_raw_bytes(&packed.to_array()[..8])?; | ||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||
| ``` | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| Location: `wacore/binary/src/encoder.rs:809-824` | ||||||||||||||||||||||||||
| <Note> | ||||||||||||||||||||||||||
| Packing used to run through per-character `match` ladders (`pack_nibble`/`pack_hex`) reached via a `fn` pointer, and for a while carried a `portable_simd` fast path for long strings on top of that. Both are gone: measurement showed the vector path only ever beat the `match` ladders, not a lookup table, and lost to the table at every string length tested — `HEX_PAIRS[byte]` (the decoder's mirror-image table) is one 2-byte load, and a shuffle/interleave/store sequence doesn't beat that. The two lookup tables above replaced all three call paths; an exhaustive test (`encode_tables_match_the_ladders_they_replaced`) checks every one of the 256 byte values against the original `match` ladders so the tables can't silently drift from the encoding they replaced. | ||||||||||||||||||||||||||
| </Note> | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ### JID Encoding | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
|
|
@@ -850,7 +849,7 @@ fn unpack_nibble(packed: u8, position: u8) -> u8 { | |||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||
| ``` | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| Unpacking a run of packed values dispatches through a 256-entry byte-pair table — one lookup and a 2-byte store per input byte, rather than the two-shift/two-lookup scalar walk this snippet shows. This is an implementation detail of `read_packed`; the value mapping above still describes what each nibble decodes to for `NIBBLE_8`. `HEX_8` unpacking is the mirror of [`pack_hex`](#hex-packing) shown earlier — 0–9 to digits, 10–15 to `A`–`F` — not the nibble table above. | ||||||||||||||||||||||||||
| Unpacking a run of packed values dispatches through a 256-entry byte-pair table — one lookup and a 2-byte store per input byte, rather than the two-shift/two-lookup scalar walk this snippet shows. This is an implementation detail of `read_packed`; the value mapping above still describes what each nibble decodes to for `NIBBLE_8`. `HEX_8` unpacking is the mirror of [`HEX_ENC`](#hex-packing) shown earlier — 0–9 to digits, 10–15 to `A`–`F` — not the nibble table above. | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| Location: `wacore/binary/src/decoder.rs:400-450` | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,8 +7,7 @@ description: Add whatsapp-rust to your Rust project | |
|
|
||
| Before installing whatsapp-rust, ensure you have: | ||
|
|
||
| - **Rust nightly** (default) — required only by the default `simd` feature, which uses the unstable `portable_simd` API. The project pins `nightly-2026-06-16` via `rust-toolchain.toml`. | ||
| - **Rust 1.94 or newer** — the workspace MSRV. Edition 2024 itself needs no nightly, so stable works once `simd` is off. See [Using stable Rust](#using-stable-rust). | ||
| - **Rust 1.94 or newer** — the workspace MSRV, and all default features build on **stable Rust**. See [Using stable Rust](#using-stable-rust). | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win Split the long documentation statements. Each location combines multiple independent facts. Use concise sentences. Address the reader as “you” or “você”. In Portuguese, change
As per coding guidelines, use active voice and second person, and keep one idea per sentence. 📍 Affects 2 files
🤖 Prompt for AI AgentsSource: Coding guidelines |
||
| - **Cargo** package manager | ||
|
|
||
| <Note> | ||
|
|
@@ -55,9 +54,7 @@ tokio = { version = "1.53", features = ["macros", "rt-multi-thread"] } | |
|
|
||
| If you need to declare a sibling crate explicitly (for example to enable a crate-specific feature flag), you can still add it individually. The full multi-crate form: | ||
|
|
||
| <CodeGroup> | ||
|
|
||
| ```toml Nightly (default) | ||
| ```toml Cargo.toml | ||
| [dependencies] | ||
| whatsapp-rust = "0.7" | ||
| # Only needed for crate-specific features not exposed through whatsapp-rust: | ||
|
|
@@ -69,22 +66,6 @@ whatsapp-rust = "0.7" | |
| tokio = { version = "1.53", features = ["macros", "rt-multi-thread"] } | ||
| ``` | ||
|
|
||
| ```toml Stable Rust | ||
| [dependencies] | ||
| whatsapp-rust = { version = "0.7", default-features = false, features = [ | ||
| "sqlite-storage", | ||
| "tokio-transport", | ||
| "tokio-runtime", | ||
| "ureq-client", | ||
| "tokio-native", | ||
| "signal", | ||
| ] } | ||
| wacore = { version = "0.7", default-features = false } | ||
| tokio = { version = "1.53", features = ["macros", "rt-multi-thread"] } | ||
| ``` | ||
|
|
||
| </CodeGroup> | ||
|
|
||
| ## Feature flags | ||
|
|
||
| whatsapp-rust supports several optional features: | ||
|
|
@@ -96,7 +77,6 @@ whatsapp-rust supports several optional features: | |
| | `tokio-transport` | Tokio WebSocket transport | ✅ Yes | | ||
| | `ureq-client` | Ureq HTTP client | ✅ Yes | | ||
| | `sqlite-storage` | SQLite storage backend | ✅ Yes | | ||
| | `simd` | SIMD-optimized binary protocol encoding/decoding (**requires nightly Rust**) | ✅ Yes | | ||
| | `signal` | Unix signal handling (graceful shutdown on SIGTERM/Ctrl+C) | ✅ Yes | | ||
| | `tracing` | Emit `tracing` spans/events across connect, send, receive, IQ, app state, pairing, media, and session flows. See [Observability](/advanced/observability) | ❌ No | | ||
| | `tracing-pii` | Render raw phone numbers in `tracing` fields instead of redacted `pn#<token>`. Local debugging only — never enable in production | ❌ No | | ||
|
|
@@ -192,38 +172,11 @@ See [Chat Store](/api/chat-store) for the full API. | |
|
|
||
| ## Using stable Rust | ||
|
|
||
| whatsapp-rust uses Rust **edition 2024** and declares an MSRV of **1.94**, both of which stable Rust supports. The one nightly-only piece is the default `simd` feature, which uses Rust's unstable `portable_simd` API for optimized binary protocol encoding/decoding. The project itself pins `nightly-2026-06-16` via `rust-toolchain.toml`. | ||
|
|
||
| To compile on **stable Rust**, disable the `simd` feature by setting `default-features = false`. You must do this on **both** `whatsapp-rust` and `wacore` — otherwise Cargo's [feature unification](https://doc.rust-lang.org/cargo/reference/features.html#feature-unification) will re-enable SIMD through the `wacore` dependency: | ||
| whatsapp-rust uses Rust **edition 2024** and declares an MSRV of **1.94**, both of which stable Rust supports, and the default feature set has no nightly-only dependency — `cargo build`/`cargo add whatsapp-rust` works with **stable Rust** out of the box, no feature flags to disable. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
For readers following the crates.io snippet above ( Useful? React with 👍 / 👎.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fair catch on the mechanics — That said, I'm leaving this as-is rather than adding a "still on 0.7.0" caveat or restoring the opt-out instructions: this docs site's established pattern (see e.g. #504, #503, #502, #499, #500, all merged the same day as their corresponding Generated by Claude Code |
||
|
|
||
| ```toml Cargo.toml | ||
| [dependencies] | ||
| # Disable defaults (removes `simd`), then re-enable everything else | ||
| whatsapp-rust = { version = "0.7", default-features = false, features = [ | ||
| "sqlite-storage", | ||
| "tokio-transport", | ||
| "tokio-runtime", | ||
| "ureq-client", | ||
| "tokio-native", | ||
| "signal", | ||
| ] } | ||
| # wacore also needs default-features = false to prevent feature unification | ||
| # from re-enabling simd | ||
| wacore = { version = "0.7", default-features = false } | ||
|
|
||
| # These crates have no SIMD dependency — no changes needed | ||
| whatsapp-rust-sqlite-storage = "0.7" | ||
| whatsapp-rust-tokio-transport = "0.7" | ||
| whatsapp-rust-ureq-http-client = "0.7" | ||
| waproto = "0.7" | ||
| tokio = { version = "1.53", features = ["macros", "rt-multi-thread"] } | ||
| ``` | ||
|
|
||
| <Warning> | ||
| Setting `default-features = false` only on `whatsapp-rust` is **not enough** if you also depend on `wacore` directly. The direct `wacore` dependency enables `simd` by default, and Cargo merges features across all dependents. Both must opt out. | ||
| </Warning> | ||
|
|
||
| The encoder/decoder automatically falls back to scalar code paths when SIMD is disabled. There is no functional difference — only a minor performance difference in binary protocol operations. | ||
| <Note> | ||
| The project's own `rust-toolchain.toml` still pins a nightly compiler (`nightly-2026-06-16`), but only for internal, binary-size-focused build flags — `-Zshare-generics` and lld/ICF linking, set workspace-wide in `.cargo/config.toml`, plus `-Zbuild-std` for the Docker image build only (see [Docker deployment](#docker-deployment) below) — not for any language feature the published crates need. That pin governs building the whatsapp-rust workspace itself; it has no effect on your project when you depend on whatsapp-rust from crates.io or git. | ||
| </Note> | ||
|
|
||
| ## 32-bit target support | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Split the overloaded documentation sentences.
Each sentence combines several independent implementation claims. Split the claims so readers can identify the encoding flow and the historical performance rationale.
advanced/binary-protocol.mdx#L457-L458: separate table selection, pair packing, stack buffering, and validity checking.advanced/binary-protocol.mdx#L504-L505: separate removed paths, benchmark result, decoder-table detail, and exhaustive-test coverage.As per coding guidelines, "Keep sentences concise — one idea per sentence in documentation."
📍 Affects 1 file
advanced/binary-protocol.mdx#L457-L458(this comment)advanced/binary-protocol.mdx#L504-L505🤖 Prompt for AI Agents
Source: Coding guidelines