diff --git a/README.mediawiki b/README.mediawiki index 140eed3076..19ae427fbe 100644 --- a/README.mediawiki +++ b/README.mediawiki @@ -1513,6 +1513,27 @@ users (see also: [https://en.bitcoin.it/wiki/Economic_majority economic majority | bubb1es, haris | Specification | Draft +|- +| [[bip-0455.md|455]] +| Peer Services +| Peer sharing of block spent coins +| Robert Netzke, Ruben Somsen +| Specification +| Draft +|- +| [[bip-0456.md|456]] +| Peer Services +| Hints for unspent coins +| Robert Netzke, Ruben Somsen +| Specification +| Draft +|- +| [[bip-0457.md|457]] +| Applications +| SwiftSync initial block download +| Robert Netzke, Ruben Somsen +| Specification +| Draft |} diff --git a/bip-0455.md b/bip-0455.md new file mode 100644 index 0000000000..231c1dfa55 --- /dev/null +++ b/bip-0455.md @@ -0,0 +1,284 @@ +``` + BIP: 455 + Layer: Peer Services + Title: Peer sharing of block spent coins + Authors: Robert Netzke + Ruben Somsen + Deputies: Edil Medeiros + Status: Draft + Type: Specification + Assigned: 2026-07-22 + License: BSD-3-Clause + Discussion: https://groups.google.com/g/bitcoindev/c/FpSWUxItXQs/m/pnfjP6rFCgAJ +``` + +## Abstract + +Inputs of a Bitcoin block are referenced by the outpoint data structure. This commonly poses a limitation during initial +block download (IBD), such that a client must process blocks sequentially to validate the chain history. The SwiftSync +protocol allows blocks to be evaluated in arbitrary order, however additional data is required that must be served over +the peer-to-peer network. This document describes how to share this data over the peer to peer network. + +## Motivation + +A common approach to IBD is to process blocks sequentially to ensure the existence of input data when validating a +block. Metadata corresponding to an input, such as the amount, must be present in a local cache to validate the block, +hence sequential validation is a natural choice. This is a result of the height, coinbase flag, input script, and amount +of the block inputs being omitted from the data committed to by proof of work in the current block, and, thus, this data +cannot be trusted if received over the wire naively. Using the SwiftSync protocol, a client is able to verify the +correctness of this data, even if served by a potentially untrusted party. This allows a significant improvement in IBD +performance, as block validation may be done in parallel. + +## Specification + +In Bitcoin Core, to roll-back the chain state in the event of a block reorganization, the height, coinbase flag, script +and amount metadata for each spend transaction output of a block are stored in a data structure known colloquially as +"undo data". This terminology stems from its use to "undo" the effect of a block by repopulating the UTXO set with the +coins that were spent by the reorganized block. To remain general in language, this data will be referred as "spent +coins." + +Bitcoin Core full archival nodes store spent coins for all blocks. This is useful in the context of SwiftSync, as no +additional index must be created or maintained to serve this data to peers. There are, however, some discrepancies +between how this data is serialized on disk in Bitcoin Core and how this proposal seeks to serialize this data over the +peer-to-peer protocol, which are detailed in the rationale section. + +This section defines how to request and serve block spent coins over the peer-to-peer protocol, as well as signaling +support of this feature to peers. + +### Definitions + +- `[]byte`: arbitrary sequence of bytes with no fixed length +- ``: byte vector of size N, where N is specified inline. N is fixed length and known at compile time (e.g. + \<32 bytes>) +- `vector`: vector of arbitrary length of elements of type Foo +- `CompactSize`: encoding of unsigned integers defined in peer-to-peer messages, as defined in the Function Appendix + section +- `CompressAmount`: compression function for integer amounts, as defined in the Function Appendix section + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and +"OPTIONAL" in this document are to be interpreted as described in RFC 2119. + +### Data structures + +#### Height and Coinbase Flag Code + +When validating a block, a client must confirm coinbase outputs are mature, which is given by the height of the coin. +The height and coinbase flag are encoded as a 32 bit integer[^1]. To encode the height and flag, binary left shift the +height one bit, treat the coinbase flag as a bit, insert it into the newly opened bit position. To decode the height, +binary right shift the code. To decode the coinbase flag, mask the first bit position of the header code and interpret +the bit as a boolean. + +Take an 8-bit example of a height with binary encoding `0010 0111`. To encode a coinbase output at this height, one +begins with a left shift: `0100 1110`, and places the coinbase flag in the least significant bit: `0100 1111`. + +#### Reconstructable Script Format + +When validating historical data, common script types may be represented more concise than the usual encoding. Bare +scripts and future output types are not compressed, however this format is extensible. The `Expansion` column is the +usual representation of the `Script` column and the `Format` column shows the compressed form. Scripts are serialized in +this format by concatenating the `Prefix` and `Format` fields specified below. + +| Prefix | Script | Format | Expansion | +| :----- | :------- | :------------------------------------ | :----------------------------------------------------------- | +| `0x00` | Unknown | `CompactSize(Len([]bytes)) + []bytes` | `[]bytes` | +| `0x01` | `P2PKH` | `<20 bytes>` | `OP_DUP OP_HASH160 20 <20 bytes> OP_EQUALVERIFY OP_CHECKSIG` | +| `0x02` | `P2PK` | `<32-byte public key (0x02 parity)>` | `33 0x02 <32 byte public key> OP_CHECKSIG` | +| `0x03` | `P2PK` | `<32-byte public key (0x03 parity)>` | `33 0x03 <32 byte public key> OP_CHECKSIG` | +| `0x04` | `P2PK` | `<64 byte public key>` | `65 0x04 <64 byte public key> OP_CHECKSIG` | +| `0x05` | `P2SH` | `<20 bytes>` | `OP_HASH160 20 <20 bytes> OP_EQUAL` | +| `0x06` | `P2WSH` | `<32 bytes>` | `OP_0 32 <32 bytes>` | +| `0x07` | `P2WPKH` | `<20 bytes>` | `OP_0 20 <20 bytes>` | +| `0x08` | `P2TR` | `<32-byte X-only public key>` | `OP_1 32 <32 bytes>` | + +#### Amount Format + +The 64 bit unsigned integers representing amounts are compressed by first using the `CompressAmount` function defined +below, and serializing the result with `CompactSize`. + +#### Coin + +| Field | Type | Serialization | Description | +| :----------------------- | :---------------------------- | :---------------------------------- | :------------------------------------------------ | +| Input index | 32-bit unsigned integer | Little endian | The index in the block inputs, coinbase excluded. | +| Height and coinbase flag | Height + Coinbase Flag Code | Defined above | — | +| Script | Reconstructable script format | Defined above | — | +| Amount | 64-bit unsigned integer | `CompressAmount` then `CompactSize` | Satoshi-denominated value. | + +### Messages + +#### MSG_GET_SPENT_COINS + +`MSG_GET_SPENT_COINS` defines a request for the inputs of a block. + +Define `cmdString` as `getbspent`. Define BIP-324 message type as ???. + +| Field | Type | Description | +| :---------- | :---------------------- | :------------------------------------------------------------------- | +| `blockhash` | `<32 bytes>` | Hash of the block for which inputs are requested. | +| `cutoff` | 32-bit unsigned integer | If greater than zero, include only coins created before this height. | + +Rationale of the `cutoff` field is detailed in the rationale section below. + +#### MSG_SPENT_COINS + +`MSG_SPENT_COINS` defines the data structure for inputs of a block. + +Define `cmdString` as `bspent`. Define BIP-324 message type as ???. + +| Field | Type | Description | +| :---------- | :------------------------------- | :------------------------------------------------ | +| `blockhash` | `<32 bytes>` | Block hash these coins are spent from. | +| `len` | `CompactSize(Len(vector))` | Length of the coins vector. | +| `coins` | `vector` | Coins spent, after filtering on request `cutoff`. | + +A client supporting the `bspent` MUST include coins created _before_ the `cutoff` field in `getbspent` requests. A +client receiving a `bspent` message with un-requested or missing coins MUST disconnect from the serving peer. + +## Signaling + +Support for serving historical block spent coins is advertised by a feature message, introduced by +[BIP-434](https://github.com/bitcoin/bips/blob/master/bip-0434.md). + +| featureid | featuredata | +| :------------------ | :---------- | +| `blockspentcoinsv1` | `0x00` | + +A client advertising this feature SHOULD respond to `getbspent` messages, subject to rate-limiting and bandwidth +limiting. + +## Rationale + +The lifetime, or interval between creation and spending height, of the coins on the Bitcoin blockchain demonstrate an +empirical phenomena that the majority of coins are spent within 100 blocks. In fact, approximately 41 percent of coins +are spent within 10 blocks at the time of writing[^2]. Clients may leverage this to reduce the bandwidth required to +fetch spent coins by using an in-memory cache. For example, a client may store coins that were created in a 5 block +window, and request only coins that are older than this height via the `cutoff` filter. This results in a significant +bandwidth reduction at the cost of a cache that can be set dynamically by the client depending on available memory. + +Beyond the use of a dynamic coin height filter, there are additional reasons to not simply read the spent coins from +disk and send it over the wire. Legacy fields (`nVersion`) are set to `0x00` when writing and reading this data to +maintain compatibility of disk format with old clients. Furthermore, using the amount compression specified above, an +11gb reduction in bandwidth is achieved. `CompactSize`, which is commonly used in P2P messages to describe collection +lengths, was selected over `VARINT`, which is used internally within Bitcoin Core to represent variable length integers +The application of `VARINT` as opposed to `CompactSize` offers a further reduction of 4gb, however the `VARINT` +primitive is currently a Bitcoin Core implementation detail and has not been included in a P2P message. Reusing existing +network primitives results in the majority of savings, so this specification opts to lower implementation burden for +clients. With respect to reconstructable script, utilizing this format results in a savings of around 12gb. The scheme +is loss-less, and may be upgradable by appending script variants. For reference, the naive encoding of block spent coins +is 118gb as of block 930,000[^2][^3][^4]. + +## Function Appendix + +Bitcoin Core utilizes a technique to remove trailing zeros from the representation of amounts. This technique offers a +significant size reduction in amount serialization. These functions are duplicated from the +[test framework](https://github.com/bitcoin/bitcoin/blob/master/test/functional/test_framework/compressor.py). + +### Compress Amount + +```python +def compress_amount(n): + if n == 0: + return 0 + e = 0 + while ((n % 10) == 0) and (e < 9): + n //= 10 + e += 1 + if e < 9: + d = n % 10 + assert (d >= 1 and d <= 9) + n //= 10 + return 1 + (n*9 + d - 1)*10 + e + else: + return 1 + (n - 1)*10 + 9 +``` + +## Decompress Amount + +```python +def decompress_amount(x): + if x == 0: + return 0 + x -= 1 + e = x % 10 + x //= 10 + n = 0 + if e < 9: + d = (x % 9) + 1 + x //= 9 + n = x * 10 + d + else: + n = x + 1 + while e > 0: + n *= 10 + e -= 1 + return n +``` + +`CompactSize` is commonly used to represent the size of collections in peer-to-peer messages. + +## Encode Compact Size + +```python +def encode_compactsize(n): + if n < 0xfd: + return bytes([n]) + elif n <= 0xffff: + return b"\xfd" + n.to_bytes(2, "little") + elif n <= 0xffffffff: + return b"\xfe" + n.to_bytes(4, "little") + else: + return b"\xff" + n.to_bytes(8, "little") +``` + +## Decode Compact Size + +```python +def decode_compactsize(b): + prefix = b[0] + if prefix < 0xfd: + return prefix + elif prefix == 0xfd: + return int.from_bytes(b[1:3], "little") + elif prefix == 0xfe: + return int.from_bytes(b[1:5], "little") + else: + return int.from_bytes(b[1:9], "little") +``` + +## Compatibility + +Clients seeking to perform fully-validating SwiftSync require peers that serve undo data. Serving data requires no +additional index and may be enabled via advertising the `feature` message. + +## Reference Implementation and Test Vectors + +### Reference Implementation + +- [Bitcoin Core](https://github.com/rustaceanrob/bitcoin/tree/bip-block-undo) + +### Test Vectors + +- [Reconstructable script](bip-0455/test_vectors/reconstructable_script.json) +- [Compressed Amount](bip-0455/test_vectors/compressed_amount.json) + +In order: +`P2PKH, P2SH, P2TR, P2WPKH, P2WSH, P2PK (odd), P2PK (even), P2PK (uncompressed), OP_RETURN (unspendable/unknown)` + +## Copyright + +This BIP is licensed under the 3-clause BSD license. + +[^1]: When representing objects in memory, programming languages will align the bytes of the fields of an object. A +boolean is commonly padded to 4 or 8 bytes, but only requires a bit. Further, if the coinbase flag was a separate field +as represented in the message, it would require at least one byte. Losing one bit of precision in the block height still +allows for valid encodings of heights up to 2,147,483,647. +[^2]: Relevant statistics may be generated via binaries in +the [`swiftsync-research`](https://github.com/rustaceanrob/swiftsync-research) repository +[^3]: Reconstructable +scripts are borrowed from [UTREEXO](https://github.com/bitcoin/bips/pull/1923) which is subsequently borrowed from Cory +Field's UHS proposal +[^4]: Astute readers may notice uncompressed public keys may be compressed before they are sent +and decompressed by the receiving client. Although this would slightly reduce bandwidth, it would increase the +complexity of client code, as a `secp256k1` context would be required to decode the message, which is not currently a +requirement. As of height 936,212 the number of uncompressed public keys spent in blocks is 853,515. This represents a +very modest savings in bandwidth, around 30MB. As such, this technique is omitted for implementation simplicity. diff --git a/bip-0455/test_vectors/compressed_amount.json b/bip-0455/test_vectors/compressed_amount.json new file mode 100644 index 0000000000..a6797d68d6 --- /dev/null +++ b/bip-0455/test_vectors/compressed_amount.json @@ -0,0 +1,8 @@ +[ + [0, "0x0"], + [1, "0x1"], + [1000000, "0x7"], + [100000000, "0x9"], + [5000000000, "0x32"], + [2100000000000000, "0x1406f40"] +] diff --git a/bip-0455/test_vectors/reconstructable_script.json b/bip-0455/test_vectors/reconstructable_script.json new file mode 100644 index 0000000000..0dccdd151f --- /dev/null +++ b/bip-0455/test_vectors/reconstructable_script.json @@ -0,0 +1,11 @@ +[ + ["76a9142365e46227cc171083ea275f45ea8646c61d1fbb88ac", "012365e46227cc171083ea275f45ea8646c61d1fbb"], + ["a914b472a266d0bd89c13706a4132ccfb16f7c3b9fcb87", "05b472a266d0bd89c13706a4132ccfb16f7c3b9fcb"], + ["5120720b1ffb2c63684973c5e9898b188c9d367fa2bc1ce76b8ea02872b5e3ffe705", "08720b1ffb2c63684973c5e9898b188c9d367fa2bc1ce76b8ea02872b5e3ffe705"], + ["00146262b97a514ea54d12f51e0a4fe4c09fb74ff7bd", "076262b97a514ea54d12f51e0a4fe4c09fb74ff7bd"], + ["00200000000000000000000000000000000000000000000000000000000000000000", "060000000000000000000000000000000000000000000000000000000000000000"], + ["210334ed84e3c579d5ff9122fb4215210ec5aaad51c3f60bf971d939db1c5b56a9fbac", "0334ed84e3c579d5ff9122fb4215210ec5aaad51c3f60bf971d939db1c5b56a9fb"], + ["210299745a46d9f42b4f578e32d5582120a4688b4224f7e20081f781efc198d11edeac", "0299745a46d9f42b4f578e32d5582120a4688b4224f7e20081f781efc198d11ede"], + ["410441a5367189b64cc1601c2a708556e37ade94ec808be746e45e35d86d2ee0cb9cd3b2e65ee51baf285cda78589605c3a59ba0492d577349ad3f0afaac862aa59eac", "0441a5367189b64cc1601c2a708556e37ade94ec808be746e45e35d86d2ee0cb9cd3b2e65ee51baf285cda78589605c3a59ba0492d577349ad3f0afaac862aa59e"], + ["6a", "00016a"] +] diff --git a/bip-0456.md b/bip-0456.md new file mode 100644 index 0000000000..ca91c4c64c --- /dev/null +++ b/bip-0456.md @@ -0,0 +1,254 @@ +``` + BIP: 456 + Layer: Peer Services + Title: Hints for unspent coins + Authors: Robert Netzke + Ruben Somsen + Deputies: Edil Medeiros + Status: Draft + Type: Specification + Assigned: 2026-07-22 + License: BSD-3-Clause + Discussion: https://groups.google.com/g/bitcoindev/c/FpSWUxItXQs/m/pnfjP6rFCgAJ +``` + +## Abstract + +This document specifies version `0x00` of the hintsfile for the SwiftSync initial block download process specified in +BIP 457. The hintsfile stores a compact representation of the UTXO set at a particular height by encoding which +transaction outputs remain unspent in each block. + +## Motivation + +SwiftSync clients can avoid constructing every intermediate UTXO set state during initial block download if they can +determine, while processing historical blocks, which outputs must be retained for the target UTXO set. A hintsfile +provides that information in a form that can be generated, transmitted, stored, and consumed independently of the blocks +themselves. + +This specification defines a compact encoding for that hints information so implementations can exchange and interpret +hintsfiles interoperably. For each block up to a target height, the hintsfile identifies the outputs that remain unspent +at that height. This allows a client to add only those outputs to the final UTXO set and aggregate the remaining spent +outputs for later verification, reducing disk I/O and enabling faster IBD. + +The hintsfile format is versioned. Later versions may define different encodings for the same hints information without +changing the role of the hintsfile in the SwiftSync protocol. Because the resulting UTXO set is verified by SwiftSync, +an incorrect hintsfile cannot cause acceptance of an invalid chain state. + +## Specification + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and +"OPTIONAL" in this document are to be interpreted as described in RFC 2119. + +A hintsfile encodes the transaction outputs that remain unspent at a target block height. For each block from genesis +through the target height, the file contains the output indices from that block that are members of the target UTXO set. + +### Hintsfile + +A _hintsfile_ is composed of the following fields: + +| Field | Value | Description | +| :------ | :------------------------- | :------------------------------- | +| Magic | `0x55 0x54 0x58 0x4f` | File identifier | +| Version | `0x00` | File version | +| Height | `<4 bytes, little endian>` | Block height | +| Hints | `vector` | Unspent output indices per block | + +The `Hints` field is a `vector`, with one element per block from height 0 through the indicated `Height`, +inclusive, i.e., the element at position `h` encodes the output indices for block height `h`. For each block, output +indices are assigned by iterating over transaction outputs in block order, excluding unspendable outputs as defined in +BIP 457. The first counted output has index 0. + +### Elias-Fano Serialization + +The serialization of an `EliasFano` object is: + +| Field | Type | Serialization | Description | +| :---- | :---------------------- | :------------ | :------------------------ | +| `N` | 32-bit unsigned integer | `CompactSize` | Number of unspent indices | +| `M` | 32-bit unsigned integer | `CompactSize` | Greatest encoded index | +| `L` | `vector` | Raw bytes | Low-bit bitset | +| `H` | `vector` | Raw bytes | Unary-encoded high bits | + +With the exception that, for `N = 0x00`, the serialization consists only of the byte `0x00`. + +### Elias-Fano Encoding + +An `EliasFano` value object encodes a strictly increasing sequence `S` of `N` unsigned integer indices: + +$$ +S = [i_0, i_1, \cdots, i_{N-1}] +$$ + +where each index satisfies: + +$$ +0 \leq i_0 < i_1 < \cdots < i_{N-1} = M +$$ + +For `N = 0`, the sequence is empty and the value is serialized as the single byte `0x00`. + +For `N > 0`, let `M` be the greatest index in `S`, i.e. $M = i_{N-1}$, and define the number of low bits as: + +$$ +\ell = \left\lfloor \log_2\left(\frac{M + 1}{N}\right)\right\rfloor +$$ + +Each index $i_j$ is split into a high part and a low part: + +$$ +\text{low}_j = i_j \bmod 2^{\ell} +$$ + +$$ +\text{high}_j = \left\lfloor \frac{i_j}{2^\ell} \right\rfloor +$$ + +The low parts are encoded by concatenating each $\text{low}_j$ as an $\ell$-bit unsigned integer in sequence order. The +resulting bitstring is called `L`. + +The high parts are encoded as a unary bitvector `H`. Let: + +$$ +k = N + \left\lfloor \frac{M}{2^\ell} \right\rfloor +$$ + +Initialize `H` as a bitvector of $k$ zero bits. For each sequence element $i_j$, set the bit at zero-based position +$\text{high}_j + j$ to `1`. All other bits remain `0`. Equivalently, `H` encodes the gaps between consecutive high parts +in unary: append $\text{high}_0$ zero bits followed by `1`, then for each subsequent $j$, append + +$$ +\text{high}_j - \text{high}_{j - 1} +$$ + +zero bits followed by `1`. + +### Deserialization + +To recover `L` and `H`, the client decodes `N` and `M`, then computes the number of low bits $\ell$. The number of bytes +to interpret as `L` is $\left\lceil \frac{N \cdot \ell}{8} \right\rceil$ and the number of bytes to interpret as `H` is +$\left\lceil \frac{N + \lfloor M / 2^\ell \rfloor}{8} \right\rceil$. + +### Worked Example + +Suppose we want to encode a block with 17 total outputs, of which only indices 3, 7, and 12 remain unspent. Thus, the +block will be represented by the sequence $S = [3, 7, 12]$. Then `N = 3` and `M = 12`. Numbers up to 15 are encoded in +four bits in binary. + +We start by computing $\ell$: + +$\ell = \left\lfloor \log_2\left(\frac{M+1}{N}\right) \right\rfloor = \left\lfloor +\log_2\left(\frac{13}{3}\right) \right\rfloor = \lfloor \log_2(4.33) \rfloor = \lfloor 2.11 \rfloor = 2$ + +We write out the binary representation for all elements in the sequence $S$ and split them into upper and lower bits: + +| Element | Binary | Upper (`value >> 2`) | Lower (`value & 0b11`) | +| :------ | :----- | :------------------- | :--------------------- | +| 3 | `0011` | `00` → 0 | `11` | +| 7 | `0111` | `01` → 1 | `11` | +| 12 | `1100` | `11` → 3 | `00` | + + +Next, we build `L` by concatenating the lower bits. In the example, we have `11 11 00`, which corresponds to `L = +111100`. + +The upper values are $0, 1, 3$. We encode the gaps between consecutive upper values in unary. Since the first +element does not have a previous upper value, it is encoded as the difference to $0$. + +| Element | Upper value | Gap from previous | Unary | +| :------ | ----------: | :---------------- | :---- | +| 3 | 0 | 0 (start) | `1` | +| 7 | 1 | 1 | `01` | +| 12 | 3 | 2 | `001` | + +Then we will build `H` by concatenating the unary. In the example, we have `1 01 001`, which corresponds to `H = +101001`. + +Final encoding of the block will be `0000 0011 0000 1100 1111 0010 1001` or `0x020CF29`. + +Now we would like to recover an element from the encoding. Suppose we are fetching `S[2] = 12`. We may use a +combination of bit shifts and bitwise _OR_ to recover it. + +To get the lower bits, we read $\ell$ bits from `L` with offset $2\ell$. In our case, `L = 111100`, so we read `00`. + +To get the upper bits, we find the position of the third termination bit `1`. In our example , `H = 101001`, and we see +that the third `1` is at position 5. To retrieve the high bits we stored, we may take the position and subtract the +index, so $5 - 2 = 3$. The position minus the index is equivalent to counting the numbers of `0`s in `H` up to the +desired index. + +Finally, we combine the low bits and high bits with a bitwise _OR_ and left shift operation: `S[2] = 3 << 2 | 0 = 12 | 0 = 12`. + +### Interpreting the Hintsfile + +After deserializing the hints from file as described above, a client may find the indices of interest by recovering +integers $i_0, \cdots, i_{N-1}$ using the fetch algorithm. Once the block is received, the client may then iterate over +the outputs, adding the outputs with a matching index to the UTXO set. When iterating over the block outputs, a client +MUST ignore _unspendable outputs_. A client interpreting the hintsfile MUST fail decoding if the number of `EliasFano` +objects does not match the height recorded in the file header. + +## Rationale + +Our goal is to indicate which outputs in each block will remain unspent at a particular chain height. A simple way to +represent this information would be a [bitset](https://en.cppreference.com/w/cpp/utility/bitset.html) with one bit per +output: `0` if the output is spent by the target height and `1` if it remains unspent. For example, for a block with 8 +outputs, the bitset `1000 0010` would indicate that outputs 0 and 6 remain unspent. + +This representation is inefficient because it uses space for every output position, while most historical outputs are +spent. In practice, blocks commonly contain thousands of outputs, and a maximally packed historical block could contain +more than 32,000 outputs. A bitset would therefore require thousands of bits per block regardless of how many outputs +remain unspent at the target height. For older blocks, where only a small fraction of outputs may remain unspent, this +results in a bitstring that is mostly zeroes. + +Encoding only those indices of outputs that remain unspent is therefore more compact. In the example above, the same +information can be represented by the indices `0, 6`. This reduces the encoding problem to representing a small count +`N` of random, strictly increasing indices in the range $[0, M]$. This formulation has a known theoretical optimum of +$\log_2 \binom{M}{N}$ bits. Elias-Fano encoding was selected because it is close to this optimum while remaining concise +and simple to implement without third-party dependencies. As of block height 930,000, the UTXO set can be represented in +119 MB with this method, which is within Bitcoin Core's 450 MB `dbcache` requirement and reasonable for most clients to +hold directly in memory. This encoding represents elements in $2N + N \lceil \log_2(M/N) \rceil$ bits, which is within a +reasonable bound of the theoretical optimum. + +Partitioning hints by block was chosen for simplicity. Groupings of multiple blocks were explored, but did not reduce +the file size. Keeping one `EliasFano` object per block also makes the relationship between a block and its retained +outputs direct for clients interpreting the hintsfile. + +Intrablock spends are still included in the output index counter. Excluding them would reduce the reference file size by +approximately 4 MB, but would require client code that is adding coins to also account for whether those coins are spent +later in the same block. This added complexity is not justified by the size reduction and would be especially awkward in +implementations whose coin-adding logic is separated from spend handling, such as Bitcoin Core's `AddCoin` method. + +## Distribution + +A malicious hintsfile distributor cannot lead a client to accepting an invalid UTXO set state, but they can expend time +and resources of a client attempting to perform IBD with SwiftSync. Clients SHOULD obtain a hintsfile from a party that +has a moral, financial, or otherwise incentive-aligned reason to provide truthful data. + +## Reference Implementation and Test Vectors + +### Reference Implementation(s) + +- [Bitcoin Core](https://github.com/rustaceanrob/bitcoin/tree/hintsfile-v1) +- [`hintsfile crate`](https://github.com/rustaceanrob/hintsfile/tree/master) + +### Test Vectors + +- [Elias-Fano encoding](bip-0456/test_vectors/elias_fano.json) + +## Compatibility + +There are no previous versions of the hintsfile format. This document does not introduce compatibility issues with +existing Bitcoin protocol behavior. + +## Acknowledgements + +Thank you to l0rinc for challenging and reviewing the many iterations of this file, and thank you to Eliam for exploring +alternatives and giving insights. + +## References + +- [Research repository](https://github.com/rustaceanrob/swiftsync-research) +- [Explanatory article](https://t.holmium.no/dia/elias-fano/#_representation_1_fixed_size_compact_encoding) +- [Paper](https://drops.dagstuhl.de/storage/00lipics/lipics-vol078-cpm2017/LIPIcs.CPM.2017.30/LIPIcs.CPM.2017.30.pdf) + +## Copyright + +This BIP is licensed under the 3-clause BSD license. diff --git a/bip-0456/test_vectors/README.md b/bip-0456/test_vectors/README.md new file mode 100644 index 0000000000..58386025ae --- /dev/null +++ b/bip-0456/test_vectors/README.md @@ -0,0 +1,3 @@ +# Hintsfile test vectors + +Additional test vectors may be found at https://github.com/rustaceanrob/hintsfiles diff --git a/bip-0456/test_vectors/elias_fano.json b/bip-0456/test_vectors/elias_fano.json new file mode 100644 index 0000000000..5c748109cb --- /dev/null +++ b/bip-0456/test_vectors/elias_fano.json @@ -0,0 +1,5 @@ +[ + [[13, 16, 19, 22, 25, 28, 31, 34, 37, 40], "0a288d8d8016ad50"], + [[5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89, 96, 103, 110, 117], "11758d8d8d8d804a4949292524"], + [[17, 20, 23, 26, 29, 32, 35, 38, 41, 44, 47, 50, 53, 56, 59, 62, 65, 68, 71, 74, 77, 80, 83, 86, 89, 92, 95, 98, 101, 104, 107, 110, 113, 116, 119, 122, 125, 128, 131, 134, 137, 140, 143, 146, 149, 152, 155, 158, 161, 164], "32a4aaaaaaaaaaaa800094a5294a5294a5294a5294a5294a5290"] +] diff --git a/bip-0457.md b/bip-0457.md new file mode 100644 index 0000000000..0ba00e96d1 --- /dev/null +++ b/bip-0457.md @@ -0,0 +1,166 @@ +``` + BIP: 457 + Layer: Applications + Title: SwiftSync initial block download + Authors: Robert Netzke + Ruben Somsen + Deputies: Edil Medeiros + Status: Draft + Type: Specification + Assigned: 2026-07-22 + License: BSD-3-Clause + Discussion: https://groups.google.com/g/bitcoindev/c/FpSWUxItXQs/m/pnfjP6rFCgAJ + Requires: BIP 455, BIP 456 +``` + +## Abstract + +_SwiftSync_ is a protocol to accelerate initial block download (IBD) using existing cryptographic primitives and minimal +state. The protocol is comprised of a hash aggregate for a set of elements and a "hintsfile" to indicate the spent-ness +of outputs in the chain history. Using these hints, clients may perform IBD in parallel, which maximizes the use of +existing system resources and shifts the performance limitations to internet quality. + +## Motivation + +For fully-validating Bitcoin clients, initial block download is the first user experience, and, moreover, is a +bootstrapping cost for second layer protocols. Improvements to this process benefit end-users and scaling protocols +alike. In the typical approach of keeping unspent transaction outputs in a cache, IBD faces two limitations. First, +although the lifetime of coins demonstrates an empirical distribution, cache misses occur for coins that are deleted. +This creates unnecessary disk I/O and database compaction. Secondly, given the structure of a block, coins that are +spent are indexed by their outpoint. This creates a requirement for clients to maintain a cache to fetch coin metadata +associated with an outpoint. _SwiftSync_ alleviates both of these limitations, allowing for IBD in as fast as a client +can download blocks and verify signatures. + +## Specification + +### Definitions + +- $H$: A hashing function +- $\text{Hintsfile}\_{n}$: Defined in BIP 456 +- $\text{UTXO}_{n}$: Unspent outputs at block height $n$ + +_SwiftSync_ builds on a common observation in math and computing, that _verification_ is often orders of magnitude more +performant than _computation_. What a client seeks to verify when performing _SwiftSync_ is that a unspent transaction +output (UTXO) set indeed corresponds to the chain history downloaded from peers. This creates a speedup, as all intermediate +UTXO set states are not computed, only the state at the target height. + +A key invariant is that a UTXO set state at height $n$ is equivalent to all of the outputs created in the chain history, +less all of the inputs: + +$$ \text{Outputs} - \text{UTXO}_{n} = \text{Inputs} $$ + +Given this relationship between the two sets, a client uses hints to $\text{UTXO}_{n}$ to verify the chain history +they have received is correct. This document describes the fully-validating client, however this protocol may be easily +extended to assume-valid assumptions. + +Some coins are impossible to spend. We define any of the following as _unspendable outputs_: + +- An output with script length over 10,000 +- An output beginning with `OP_RETURN` +- A [BIP-30](https://github.com/bitcoin/bips/blob/master/bip-0030.mediawiki) unspendable coinbase output + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and +"OPTIONAL" in this document are to be interpreted as described in RFC 2119. + +### Aggregation + +A client must compare $\text{Outputs} - \text{UTXO}_{n}$ with $\text{Inputs}$ in a succinct way. Rather than say, +comparing the lists, a client may compute two aggregates, and compare the values at the end of the protocol. The +_aggregate_ is defined over the data structure of the elements and a hashing function. It is recommended, but not +strictly required, that a client uses the typical SHA-256 hash, with an additional randomly generated salt value[^1]. +The elements of the set a client compares are _coins_, which are defined as the following: + +``` +Coin = CreationHeight || CoinbaseFlag || OutputScript || Amount || Outpoint +``` + +Each field of the `Coin` is serialized before hashing as follows: + +| Element | Serialization | +| :------------------------------------ | :----------------------------------------------- | +| `Creation Height` and `Coinbase Flag` | `HeightAndCoinbaseFlagCode` described in BIP 455 | +| `Output Script` | as in the block representation | +| `Amount` | unsigned 64-bit integer, little endian | +| `Outpoint` | as in the block representation | + +To add an element to an aggregate, a client computes the $H(\text{Coin})$ and interprets that number as an unsigned +256-bit integer. To update the state of the aggregate, a client simply adds this element to the previous state, modulo +256 bits. Thus, for a set of $\text{Coin}$, the _aggregate_ is defined as: + +$\text{Agg} = H(\text{Coin}\_{0}) + ... + H(\text{Coin}\_{i})\\mod2^{256}$ + +### Protocol + +Block validation when performing _SwiftSync_ is nearly the same, only with a few additional steps. First, a client +requires a $\text{Hintsfile}\_{n}$ and initializes two aggregates with the same value (i.e. zero), one for block inputs +$\text{Agg}\_{inputs}$, the other for block outputs $\text{Agg}\_{outputs}$ + +When downloading blocks, _SwiftSync_ clients will do the following: + +1. Download the required spent coins data defined in BIP 455. +1. Using the undo data, validate the block. If the block is invalid, fail. +1. For all inputs, excluding coinbase inputs, add the hashes to $\text{Agg}\_{inputs}$ using the spent coin data. +1. For all outputs: + 1. If the output is _unspendable_, continue + 1. If the output is in $\text{Hintsfile}\_{n}$, add the output to $\text{UTXO}\_{n}$. Else, add the hash of the + output coin to $\text{Agg}\_{outputs}$. + +Notice here that a client does not have to download blocks in any particular order, and may download blocks from +multiple peers at a time. A client then verifies $\text{Agg}\_{outputs} = \text{Agg}\_{inputs}$ once they have arrived +at height $n$[^2]. If the verification succeeds, the client accepts $\text{UTXO}\_{n}$ as valid. In the failure case, +the client rejects $\text{UTXO}\_{n}$ and attempts to recover using a reindex in the usual serial case. + +## Rationale + +While there are hash functions, such as siphash, that may offer a performance improvement compared to SHA-256, consensus +is already dependent on the cryptographic assumptions of SHA. Thus, SHA-256 is recommended to circumvent adding new +cryptographic assumptions to IBD. On griefing, a malicious peer may construct spent coins that is valid for the given +block, yet it is not spending the correct coins in the history for the chain of most work. Take, for example, a +trivially spendable output. The malicious peer may use any script when serving the block inputs, which will alter the +output of $H\_{salt}(\text{Coin})$, ultimately causing the final verification to fail. To mitigate this, it is +encouraged to commit to the hashes of the block inputs, either within the binary or file from a semi-trusted source. + +## Note on BIP-30 and BIP-34 + +During the period between the genesis block and BIP-34 activation, a _SwiftSync_ client must check for duplicate +coinbase outputs. A cache of these outputs is modest in memory footprint, and may be easily added and queried for the +fixed block range. More information on this caveat is detailed in +[this article](https://gist.github.com/RubenSomsen/a02b9071bf81b922dcc9edea7d810b7c). + +## Extension to Assume Valid + +The protocol may be easily extended, and rather simplified, with similar assumptions to _assume valid_. Rather than +hashing the entirety of coin data, a client may take $H(\text{Outpoint})$ and add these results to the aggregates. This +removes the need to download block spent coins from peers, and is compatible with the current peer-to-peer protocol. + +There are, however, a number of drawbacks with this valid approach. At the end of the protocol, the client must also +check the total monetary value introduced in the system is less than the expected value for the height $n$. An +equivalency check is not possible, as there are instances where coinbase subsidies were under-claimed. Further, the +number of signature operations cannot be checked for input scripts, and the order of transactions cannot be checked as +well. + +Finally, the spent coins for the block cannot be computed locally with this approach. This poses a data availability +problem, as fully-validating SwiftSync clients will need their connections to serve spent coins. + +## Reference Implementation + +- [`swiftsync`](https://github.com/2140-dev/swiftsync) Rust implementation + +## References + +- [Original proposal](https://gist.github.com/RubenSomsen/a61a37d14182ccd78760e477c78133cd) + +## Compatibility + +There are no previous versions of this document. Any implementation seeking to perform initial block download is +compatible. + +## Copyright + +This BIP is licensed under the 3-clause BSD license. + +[^1]: Any hash function that is resistant to targeted collision attacks is suitable. MuHash serves as another example. +[^2]: For the aggregate construction, one may observe that, rather than using two aggregates, adding H(Coin) when it +is created and subtracting the coin hash when it was spent should result in an aggregate state of $0$ at the end of the +protocol. While this is certainly more elegant, separating the aggregates allows for implementation flexibility in +client code