By Ruben Somsen
Source: https://gist.github.com/RubenSomsen/a02b9071bf81b922dcc9edea7d810b7c
The Bitcoin mailing list post is here , and the podcast discussion is here .
introduction
During my recent research on " SwiftSync " ( Chinese translation ), I became aware of an unresolved consensus bug in BIP30 . It appears that this bug cannot be triggered without a blockchain reorg as deep as 2010, so its severity is debatable. Our current Bitcoin client includes checkpoints dating back to 2013, so it is protected against such deep reorgs. However, if all checkpoints are removed , the bug becomes theoretically exploitable.
(Translator's note: "Checkpointing" is a special method of maintaining consensus in a blockchain network: the client software enforces that the block hash value at a certain height should be a certain value. If the resulting block hash value is not this value, it is rejected. In a blockchain network based on proof of work (PoW), this method can be used to prevent new nodes joining the network from being lured into a malicious blockchain that can be created at low cost (as the power of mining machines increases, it becomes increasingly cheaper to create blocks with the same difficulty as older historical blocks). Despite this positive reason, people tend to believe that allowing this mechanism to exist may give client developers too much power (worrying that it would allow developers to essentially decide which blockchain users choose). Therefore, there have been efforts to eliminate this mechanism in clients.)
From a consensus checking perspective, BIP30 is also somewhat peculiar, requiring it to iterate over the entire UTXO set, even if they are unrelated to the inputs spent by the transaction. This is inefficient and significantly complicates the implementation of alternative verification schemes (such as utreexo, SwiftSync, and likely also impacting zero-knowledge proof-based systems like ZeroSync). It would be a good thing if BIP30 could be retired entirely.
Rather than necessarily advocating for action (the status quo seems pretty good), I hope to propose solutions and start a discussion for both.
1. Consensus bug
BIP30 treats two overlapping transactions as exceptions. The latter transaction is the coinbase transaction at block height 91880. When this transaction was processed, the coinbase transaction at block height 91722 was overwritten. Another such case occurred in blocks 91812 and 91842.
The problem arises when the blockchain reorganizes between blocks 91880 and 91772. When we roll back the blockchain, the output created in block 91880 is removed from the UTXO set. Consequently, the overwritten output disappears completely from the UTXO set. However, another node that never witnessed the reorganization still retains this output in its set (because it was never overwritten). Once this UTXO is spent, a chain fork occurs.
Solution A
We can enforce that the reorg cannot occur exactly between blocks 91722 and 91880—it must be either deep before block 91722 or shallow just after block 91880. This ensures that neither the blocks that experienced the reorg nor new nodes will have this problematic output in their UTXO sets. Considering that this is about 160 blocks at the mining difficulty of 2010, this is not a significant limitation.
Solution B
When discussing my findings with Sjors Provoost, he pointed out that the currently considered removal of checkpoints (which could itself be considered a hard fork) is also an opportunity to change the consensus rules before the checkpoints—we could fix this bug by not removing the coinbase transaction when hitting the reorgs at blocks 91880 and 91842. Furthermore, Sjors's observation raises the question of whether we might want to change other consensus rules that existed before 2013.
2. Make BIP30 UTXO set checks obsolete
Currently, BIP30 applies from the Genesis Block until the activation of BIP34 (block height 227931, March 2013). If this block is reorganized, BIP30 will remain applicable indefinitely. BIP34 also has its own issues, which are addressed in the " Consensus Cleanup BIP "—you can read about it yourself; I won't elaborate here.
Technically, BIP30 is simply designed to prevent duplicate unspent outputs. It does this by checking every entry in the existing UTXO set, and if there are any duplicates, refusing to create a duplicate block (which is why it's so inefficient). The two duplicate transactions that occurred in 2010 were hard-coded as an exception. Under these rules, spending an output and then recreating the same output is not a violation. However, this does not appear to have ever occurred.
The final question to address is why BIP34 is deprecated if block 227931 is reorganized. The reason is that if it is not deprecated, it is possible to create outputs that violate BIP34's rules (ensuring coinbase transaction uniqueness) before BIP34 is activated (this is exactly the problem that the consensus cleanup proposal attempts to solve).
Ideally, it would be a good thing to have the BIP30 check completely decommissioned (ensuring that it is no longer needed even if a block reorg occurs).
Solution
Given that we have no overlapping transactions except for those two exceptions, we can replace the inefficient BIP30 UTXO set check with a coinbase transaction uniqueness check. We simply cache the coinbase transaction's TXID and ensure that there are no overlapping transactions in the coinbase transaction. This check, which persists until block 227931, only requires about 7MB of cache. However, because BIP34 may be deprecated, the BIP30 check may continue indefinitely, resulting in a continuously growing cache. The solution is as follows.
In addition to checking the uniqueness of the coinbase transaction, we also check that it does not conflict with any future coinbase transactions (i.e., it does not conflict with BIP34 and the consensus cleanup BIP). This ensures that BIP34 can be activated at block height 227931, regardless of whether a reorg occurs.
Conclusion
The above are some of the issues with BIP30 and possible solutions. This article serves as a reference for whether or not we take action. Thanks to Antoine Poinsot, Pieter Wuille, and Sjors Provoost for discussions before this article was published.
