Recompute a silent payment's output scripts, rather than trust them

[fametrano]

Closes #760, the follow-up to
#641 — which landed BIP375's six
psbt fields and deliberately stopped at the codec.

btclib.psbt.silent_payments is the other half. A silent payment output
script is derived, not signed: get it wrong and the transaction is
still consensus-valid, so it confirms and the money is gone. The shares and
the BIP374 proofs are what make that derivation checkable by somebody
holding none of the keys, and the Transaction Extractor is where the check
has to happen — it is the last party before the bytes go on the wire.

The Signer writes

set_input_share	one input's ECDH share and its BIP374 proof
set_global_share	the single pair standing for every eligible input
set_output_scripts	the scripts, and the modifiable flags cleared with them

Both writers refuse a private key that is not the one they would be proving
against: every reader of the psbt would reject such a share, so the signer
would have published a proof of its own mistake.

The Extractor reads

assert_as_valid is the four checks BIP375's own validator publishes, in
its order — assert_shares_as_valid for the proofs,
assert_eligibility_as_valid for the inputs a silent payment forbids (a
witness version above 1, any sighash type but SIGHASH_ALL),
assert_output_scripts_as_valid for the derivation — each naming what
failed rather than "invalid".

input_pub_key is what all of it stands on, and is the piece that was
missing: btclib.silent_payments.pub_key_from_input reads a signed
input's key out of the witness or the scriptSig, where an unsigned one has
neither and BIP375 asks an Updater for PSBT_IN_BIP32_DERIVATION instead.

The vectors are now answered in full

All 22 invalid psbts refused, all 19 valid ones accepted, where the
codec alone refused five. Each case is held to the check its own category
names, so a psbt refused for the wrong reason fails rather than counting as
a pass.

One rule where the BIP and its own vectors disagree

This is the part worth your eye. BIP375 says the codes of one scan key are
sorted lexicographically to order k. The vectors' scripts are the ones
output index order derives.

The case that decides it is published as valid — "two sp outputs - output
0 uses label=3 / output 1 uses label=1" — and its two spend keys are in
descending order, so the two rules assign k the other way round and only
one reproduces the scripts the file carries. I measured both readings of
"the codes": sorting the 66-byte info fields and sorting the bech32m
address strings order that pair the same wrong way. Upstream's own
bip-0375/validator/validate_psbt.py walks index order too, so two of its
three artefacts agree and the prose is the outlier.

Index order is therefore what is implemented, and
test_the_k_ordering_is_the_output_index asserts it in both directions
— index order reproduces both scripts, the lexicographic order reproduces
neither — so a revision settling it the other way fails here rather than
passing quietly.

A related finding: the two invalid vectors named after ordering turn out
not to decide it. Their candidate orderings all agree, and their scripts
match no k assignment at all, so they are refused whatever rule you pick.
test_the_two_ordering_vectors_are_refused_whatever_the_order records that,
because a reader of the descriptions would reasonably expect otherwise.

Happy to write this up for bitcoin/bips if you want it reported upstream.

Also

btclib.silent_payments gains output_key, the last step of BIP352's
derivation: the psbt path reaches it from an ECDH share rather than from a
private key, so it is what the two paths share instead of output_keys.
output_keys is now written in terms of it.

Gates

• uv run pytest — 26514 passed, coverage 100.00%
• uv run pre-commit run --all-files — exit 0
• sphinx-build -W --keep-going — build succeeded

Summary by Sourcery

Add BIP375 silent payment Signer and Transaction Extractor roles over PSBT, ensuring output scripts are recomputed and validated rather than trusted, and wire the new module into the PSBT package and test suite.

New Features:

• Introduce btclib.psbt.silent_payments module implementing BIP375 Signer helpers to write ECDH shares, proofs, and derive silent payment output scripts.
• Provide Transaction Extractor validation routines to enforce BIP375’s share coverage, input eligibility, and recomputed output scripts for PSBTs.
• Export the silent_payments role module from btclib.psbt and document its availability in README and PSBT docs.

Enhancements:

• Refactor core silent payments derivation to expose an output_key helper for computing a single recipient taproot output key and reuse it in output_keys.
• Clarify and codify the effective ordering rule for k values (output index order) where BIP375 prose conflicts with its own test vectors.
• Extend changelog and test data README to describe full BIP375 vector coverage and the rationale behind the k-ordering choice.

Tests:

• Add a dedicated tests/psbt/silent_payments_test.py suite validating all BIP375 role checks against upstream test vectors, including k-ordering behavior and error categorization.
• Update tests/all_test.py to treat silent_payments as a published PSBT group and ensure it is exported from the btclib.psbt namespace.

[sourcery-ai]

Sorry @fametrano, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[sourcery-ai]

Reviewer's Guide

Implements BIP375’s Signer and Transaction Extractor roles for silent payments in PSBTs, adds shared derivation helpers, and wires the new functionality into the public psbt API, docs, and test suite while resolving a spec vs. vector discrepancy in k-ordering.

Sequence diagram for BIP375 Signer and Transaction Extractor over silent payments PSBT

sequenceDiagram
    actor Signer
    participant Psbt
    participant psbt_silent_payments as btclib_psbt_silent_payments
    actor Extractor

    Signer->>Psbt: construct PSBT with silent payment outputs
    Signer->>psbt_silent_payments: set_input_share(psbt, vin_i, prv_key, aux)
    Signer->>psbt_silent_payments: set_global_share(psbt, prv_keys, aux)
    Signer->>psbt_silent_payments: set_output_scripts(psbt)
    psbt_silent_payments->>Psbt: write script_pub_key, clear tx_modifiable

    Extractor->>psbt_silent_payments: assert_as_valid(psbt)
    psbt_silent_payments->>Psbt: psbt.assert_valid()
    psbt_silent_payments->>psbt_silent_payments: _assert_modifiable_cleared(psbt)
    psbt_silent_payments->>psbt_silent_payments: assert_shares_as_valid(psbt)
    psbt_silent_payments->>psbt_silent_payments: assert_eligibility_as_valid(psbt)
    psbt_silent_payments->>psbt_silent_payments: assert_output_scripts_as_valid(psbt)
    psbt_silent_payments->>Psbt: PSBT accepted or BTClibValueError raised

Loading

File-Level Changes

Change	Details	Files
Introduce btclib.psbt.silent_payments module implementing BIP375 Signer and Transaction Extractor roles over PSBT silent payment fields.	Define helpers to read eligible input pubkeys from PSBT (including taproot and BIP32 derivation), compute shared secrets from ECDH shares, and order silent payment outputs for k assignment. Implement Signer-side APIs set_input_share, set_global_share, and set_output_scripts that write ECDH shares/proofs, derive output scripts, and clear modifiable flags, with validation that keys match inputs and that all outputs are derivable. Implement Extractor-side APIs assert_shares_as_valid, assert_eligibility_as_valid, assert_output_scripts_as_valid, and assert_as_valid, enforcing share/proof pairing, input eligibility (witness version and SIGHASH_ALL), coverage, modifiable flags, and exact script derivation. Implement internal validation helpers for witness version detection, scan key collection, share coverage checks, and global/per-input DLEQ proof verification.	btclib/psbt/silent_payments.py
Extend core silent payments derivation with an output_key helper and refactor output_keys to use it.	Add output_key(secret, B_m, k) to derive an x-only taproot output key from a shared secret and spend key as the last step of BIP352 derivation. Refactor output_keys to use output_key per recipient instead of re-implementing the tweak/add/x-only logic inline, and tighten typing of the keys list.	btclib/silent_payments.py
Expose the new silent_payments PSBT role module in the public psbt package API and documentation.	Update btclib.psbt.init to import and re-export the silent_payments module alongside musig2. Adjust tests/all_test.py to treat silent_payments as a published psbt group and assert it is exported from btclib.psbt. Extend btclib.psbt docs and top-level README to describe BIP375 support including Signer and Transaction Extractor roles and recomputation of silent payment output scripts.	btclib/psbt/__init__.py tests/all_test.py docs/source/btclib.psbt.rst README.md
Update changelog and test data README to reflect full BIP375 coverage and document the k-ordering rule discrepancy and resolution.	Add a detailed CHANGELOG entry describing the new btclib.psbt.silent_payments module, the Signer/Extractor responsibilities, and the decision to implement index-based k-ordering over lexicographic ordering to match vectors and upstream validator. Revise tests/_data/README.md to note that all 22 invalid and 19 valid BIP375 test vectors are now covered, explaining how cases are categorized and how the k-ordering disagreement is pinned by tests.	CHANGELOG.md tests/_data/README.md
Add comprehensive tests for BIP375 roles, k-ordering behavior, and edge cases around eligibility, shares, proofs, and modifiable flags.	Create tests/psbt/silent_payments_test.py to drive btclib.psbt.silent_payments against bip375_test_vectors.json, asserting that all valid vectors pass and all invalid ones fail for the correct reason/category. Add tests for k-ordering decision (output index vs. lexicographic), ordering vectors that are invalid regardless of ordering, input_pub_key sourcing (BIP32 derivation vs. script), ineligible inputs, share/proof pairing and forged proofs, shared secret vs. raw share distinction, signer correctness for per-input and global shares, modifiable flag handling, and behavior when no silent payment outputs are present. Test witness version detection based purely on script shape and that global shares without eligible inputs, or psbts still under construction, are handled with appropriate errors or no-ops.	tests/psbt/silent_payments_test.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#760	Implement BIP375 Signer and Transaction Extractor roles for silent payments in PSBT, including input public key/eligibility reading, ECDH share and BIP374 proof verification, coverage rules, eligibility rules, recomputation of silent payment output scripts from shares, and enforcement of PSBT_GLOBAL_TX_MODIFIABLE flags.	✅
#760	Update tests so that bip375_test_vectors.json is fully enforced: all 22 invalid PSBT vectors are refused, all 19 valid vectors accepted, each held to the specific BIP375 check category (structure, ECDH coverage, input eligibility, output scripts) it is meant to exercise.	✅
#760	Resolve and codify the k-ordering rule for multiple outputs sharing a scan key by measuring behavior against BIP375 test vectors and upstream validator, and implementing the interoperable ordering (output-index order) with explicit tests documenting the discrepancy with the BIP prose.	✅

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[fametrano]

Head moved: rebased onto origin/main after #767 landed. No new work.

The five failing checks on the previous head were not this pull request's: main was red on exactly the same five since #754, because tests/build_system_test.py imported tomllib unconditionally and that is stdlib only from 3.11 — an import error at collection, so the four 3.10 cells never ran a test. #767 fixed it (regex on 3.10) together with the wheel smoke-test, and this branch now sits on top of that.

Gates re-run after the rebase:

• uv run pytest — 26514 passed, coverage 100.00%
• uv run pre-commit run --all-files — exit 0
• sphinx-build -W --keep-going — build succeeded

The BIP375 k-ordering discrepancy this pull request works around is now tracked on its own: #768, with the three citations pinned to the tip commit of each path.