Bitcoin Development Philosophy: Upgrade

Author: Kalle Rosenbaum & Linnéa Rosenbaum

Source: https://bitcoindevphilosophy.com/#upgrading

See the previous article here.

Upgrading Bitcoin safely is extremely difficult. Such changes (even if the content is definitive) can take many years to roll out. In this chapter, we'll learn about common terms surrounding Bitcoin upgrades, examine several historical upgrade cases, and the lessons we can learn from them. Finally, we'll discuss "blockchain splits" and the associated costs and risks.

To better understand this chapter, you should start with David Harding's article on " Harmony and Disharmony ":

Bitcoin experts often use the word "consensus," which is rather abstract and difficult to explain in a few words. However, the etymology of "consensus" is the Latin word "concentus," which means "to sing in harmony" [1]. Therefore, we can discuss "the harmony of Bitcoin" instead of "the consensus of Bitcoin."

Harmony is what makes Bitcoin work. Thousands of full nodes work independently: verifying the validity of the transactions they receive, thus generating a harmonious consensus on the state of the Bitcoin ledger, without any one node operator trusting another. It's like a choir where every member sings the same song at the same time, producing a more beautiful melody than any single member singing alone.

The harmonious result of Bitcoin is a system in which it is secure, not only against thieves (as long as you keep your private keys safe), but also against endless inflation, large-scale and targeted confiscations, and free from the bureaucratic quagmire of the traditional financial system.

— David Harding, Harmony and Disharmony

This chapter discusses how Bitcoin can be upgraded without causing discord. Maintaining harmony, or consensus, is actually one of the biggest challenges in Bitcoin's development. The upgrade mechanism has many details, and it's best understood by studying real-world examples of past upgrades. For this reason, this section devotes considerable space to historical cases; to that end, we'll begin by learning some useful terminology.

5.1 Words

According to Wikipedia, " forward compatibility" refers to the ability of older software to process data generated by newer software and ignore parts that the newer software cannot understand.

If a product compiled using an earlier version can "gracefully" handle input designed for later versions of the same standard, ignoring parts it cannot understand, we say that the standard supports "forward compatibility".

— Forward compatibility, Wikipedia

Conversely, " backward compatibility " means that data from older software is also available on newer software. If a change is both forward-compatible and backward-compatible, we say that the change is "fully compatible".

If a change to Bitcoin's consensus rules is fully compatible, we call it a " soft fork ." This is the most common way to upgrade Bitcoin, and we will discuss many of the reasons for this in later sections of this chapter. If a change to Bitcoin's consensus rules is backward compatible but not forward compatible, we call it a " hard fork ."

For a technical overview of soft and hard forks, please read Chapter 11 of *Grokking Bitcoin* . This chapter explains the terminology and delves into the upgrade mechanism. I recommend readers skim this chapter before proceeding, although it's not strictly necessary.

5.2 Historical Upgrades

Today's Bitcoin (protocol) is different from when the first block was created. There have been numerous upgrades over the years. In 2017, Eric Lombrozo, at the Breaking Bitcoin conference , explained the different upgrade mechanisms of Bitcoin, pointing out that these mechanisms have also undergone many changes. He even explained how Satoshi Nakamoto upgraded Bitcoin through a hard fork once.

In fact, Satoshi Nakamoto did use a hard fork to upgrade Bitcoin once—and we would never do that again because it's a terrible approach. Look at the git commit description [ 757f076 ], which says this commit was to roll back makefile.unix wx-config to version 0.3.6. That's it. He didn't mention at all that it was a destructive change. He basically hid it away. He also posted on the bitcoin talk forum , urging all users to upgrade to version 0.3.6 as soon as possible; if you can't upgrade immediately, it's best to shut down your Bitcoin node first. Based on the above information, I don't know why he did this—he simultaneously decided to add some optimizations, bug fixes, and other improvements to the same code.

— Eric Lombrozo, "Changing the Consensus Rules Without Breaking Bitcoin", Breaking Bitcoin Conference (2017)

Lombrozo also points out that, whether intentional or unintentional, this hard fork created opportunities for future soft forks because it introduced script opcodes OP_NOP1 ~ OP_NOP10 . We will learn more about this code change in Section 9.2.1 . So far, these opcodes have been used in two soft forks: BIP65 (OP_CHECKLOCKTIMEVERIFY) and BIP113 (OP_SEQUENCEVERIFY). (Translator's note: These two are used to verify absolute and relative time locks in scripts, respectively.)

Lombrozo also briefly outlines the changes to the upgrade mechanism over the years (up to 2017). Since then, only one major upgrade, "Taproot," has been deployed (see section 5.2.3 for details). This lengthy and somewhat convoluted activation process has helped us gain more insights into the Bitcoin upgrade mechanism.

5.2.1 Upgrade of "Isolation Witnessing"

While all upgrades before the Segwit upgrade could be described as painless to some extent, this one was different. When the Segwit activation code was released (October 2016), it seemed that the overwhelming majority of Bitcoin users supported it, but for some reason, miners did not express support for the upgrade, which caused activation to stall, and there seemed to be no solution.

In his article " The Long Road to Segregated Witness, " published on the Bitcoin Magazine website, Aaron van Wirdum describes this eventful journey. He first explains what the Segregated Witness upgrade is and how it became embroiled in the debate over changing the Bitcoin block size limit. Van Wirdum then outlines the pivotal events that ultimately led to its activation. A key element is an upgrade mechanism called a " User-Activated Soft Fork (UASF)," proposed by user "Schaolinfry."

Shaolinfry proposed an alternative: User-Activated Soft Fork (UASF). Instead of letting hash power take the lead, it designs an "activation signal day" where users enforce it from a predetermined future point in time. If such a UASF is enforced by the majority of economic actors, it should convince most miners to follow (or activate) the soft fork.

— Aaron van Wirdum, “The Long Road to Isolated Witnessing,” Bitcoin Magazine (2017)

Furthermore, Van Wirdum cited an email posted by Shaolinfry in the Bitcoin-dev mailing list. There, Shaolinfry expressed opposition to miner-activated soft forks and listed numerous problems with this activation method.

First, it requires that the hash power be trusted to verify honestly after activation. However, in the BIP66 soft fork, although 95% of the hash power indicated that it was ready, about half of it did not actually verify the upgraded rules and mistakenly mined an invalid block [1].

Secondly, miner signals have a natural veto mechanism; a small amount of hash power can veto a node's intention to activate an upgrade, preventing anyone from upgrading. So far, soft forks have utilized a relatively centralized mining ecosystem, with only a small number of mining pools constructing valid blocks. As we move towards a more decentralized mining ecosystem, we may increasingly be hampered by "upgrade inertia," which can veto the vast majority of upgrades.

—— Shaolinfry, Bitcoin-dev mailing list (2017)

Shaolinfry also points out a common misinterpretation of miner signals: There's a widespread perception that they are a tool for miners to decide whether or not to upgrade the protocol, rather than an action to help coordinate upgrades. Because of this misunderstanding, miners may also feel obligated to publicly express their views on a soft fork, as if this adds credibility to their proposals.

The UASF proposal, simply put, is a "signal day" where nodes begin enforcing specific new rules. This way, miners don't need to take collective action to coordinate upgrades; instead, they can activate the upgrade before the signal day, provided enough blocks have already indicated their readiness.

My suggestion is to combine the strengths of both. Since user-activated soft forks require a relatively long preparation period before activation, we can combine BIP9 to offer an option for hash-coordinated activation (which would be faster), and also add a slower signaling day activation. In either case, we can leverage the warning system in BIP9. This change is relatively simple; just add an "Activation Time" parameter, which will change the BIP9 state of the soft fork to "LOCKED_IN" before the BIP9 deployment timeout date.

—— Shaolinfry, Bitcoin-dev mailing list (2017)

This idea attracted considerable interest, but it didn't seem to garner universal support, raising concerns that it could potentially cause a split in the blockchain network. Aaron van Wirdum's article explains how these controversies were ultimately resolved with BIP91 , a Bitcoin upgrade proposal (BIP) put forward by James Hilliard.

Hilliard proposed a more complex but cleverer solution that makes everything compatible: the Segregated Witness upgrade proposed by the Bitcoin Core development team, the BIP148 user-activated soft fork, and the "New York Agreement" activation mechanism. His BIP91 can keep Bitcoin from splitting—at least during Segregated Witness activation.

— Aaron van Wirdum, “The Long Road to Isolated Witnessing,” Bitcoin Magazine (2017)

This involves some more complex factors (such as the so-called "New York Consensus"), which the BIP must also consider. I encourage readers to read van Wirdum's article in its entirety to understand many of the interesting details of this story.

5.2.2 Discussion after the upgrade of Segregated Witness

Following the activation of the Segregated Witness upgrade, discussions regarding the activation mechanism continue. As Eric Lombrozo pointed out in his presentation at Breaking Bitcoin and Shaolinfry noted (see section 5.2.1 above), miner-activated soft forks are not an ideal upgrade mechanism.

Sometimes we want to add more features to the Bitcoin protocol. This is a huge philosophical question we need to ask ourselves. Should we use UASF for the next upgrade? Or a hybrid approach? Miner self-activation (this method) is outdated. We will no longer use BIP9.

— Eric Lombrozo, "Changing the Consensus Rules Without Breaking Bitcoin", Breaking Bitcoin Conference (2017)

In January 2020, Matt Corallo sent an email to the Bitcoin-dev mailing list, initiating a discussion about future soft fork deployment mechanisms. He listed five goals that he considered crucial to the upgrade. David Harding summarized these in a Bitcoin Optech weekly report as follows:

1. If the proposed changes to the consensus rules encounter serious opposition, the company has the capability to abandon the upgrade.
2. There is sufficient coordination time before the upgraded software is released, allowing the vast majority of economic nodes to upgrade to enforce these rules.
3. The network's hashrate is expected to remain roughly the same before and after the change (and during any transitions).
4. To the greatest extent possible, prevent the creation of blocks that are invalid under the new rules, as such invalid blocks will cause errors in block confirmation statistics for nodes that do not upgrade and SPV clients.
5. Ensure that the abandonment mechanism is not abused by troublemakers or used to delay anticipated escalations without known problems.

——David Harding, Bitcoin Optech newsletter #80 (2020)

Corallo proposed a mechanism that combines miner-activated soft forks and user-activated soft forks:

Therefore, to be more specific, I believe that an activation method that can serve as a correct precedent and properly balance the above objectives should be as follows:

1) A standard BIP 9 deployment process, with a one-year time window for activation, requires 95% of miners to be ready;

2) In cases where activation fails within one year, a 6-month quiet period will be set, during which the community can analyze and discuss the reasons for non-activation;

3) Where reasonable, provide a simple command-line command/configuration parameter in the earliest deployment software releases that allows users to select a BIP 8 deployment process with a 24-month time window to facilitate Signal Day activation (at the same time, new Bitcoin Core releases automatically enable this tag).

This provides a very long time window for more standardized activation methods; however, to ensure that objective 3 is met while still satisfying objective 5, a significantly longer time window is needed. Developing Bitcoin is not a race. If necessary, waiting 42 months ensures we are not setting a negative precedent that we will regret later.

— Matt Corallo, “Modern Soft Fork Activation Approaches”, Bitcoin-dev Mailing List (2020)

Chinese translation

5.2.3 “Taproot” Upgrade and “Speedy Trial”

When Taproot was ready for deployment in October 2020—that is, when all the technical details surrounding its consensus rules had been implemented and gained widespread acceptance within the community—discussions about how to deploy it began to take hold. Prior to this, such discussions had been relatively low-key.

Numerous proposals regarding activation mechanisms began to emerge, and David Harding summarized these proposals in the Bitcoin Wiki . In his article, he explained some properties of BIP8, which underwent changes at that time to make it more flexible:

When this document was created, BIP8 was based on lessons learned in 2017. A major change that followed BIP 9+148 was that the time metric for forced activation was block height, rather than the median time past; the second major change was that forced activation was a boolean parameter that could be selected when setting the activation parameters for a soft fork during the initial deployment, and could also be updated in subsequent deployments.

BIP8, without mandatory activation, is very similar to BIP9 (which has a version bit mechanism with timeout and delay). The only major difference is that BIP8 uses the block height, while BIP9 uses the median of past times. This setting allows upgrade attempts to fail (but can be tried again).

BIP8 with mandatory activation will eventually have a mandatory signaling period during which all blocks produced must adhere to its rules—they must indicate that they are ready to activate the soft fork—which will enable the same soft fork to activate in earlier deployments without mandatory activation. In other words, if an already released software version x does not have a mandatory activation mechanism, while a subsequent released version y does, and successfully forces miners to begin indicating readiness within the same time window, then both versions of the software will begin enforcing the new consensus rules at the same time.

The revised BIP8 proposal is flexible enough to express other ideas that appear to be BIP8. This provides a common element that distinguishes many different proposals.

— David Harding, Taproot Activation Proposal, Bitcoin Wiki (2020)

From this point onward, the discussion became very heated, especially regarding whether lockinontimeout should be set to true (thus enabling user-activated soft forks, or what Harding calls "BIP8 with forced activation") or false (thus enabling miner-activated soft forks, or what Harding calls "BIP8 without forced activation").

Among the listed proposals was one called "Let's see what happens." For several reasons, this proposal didn't receive much attention until several months later.

For seven months, the discussion continued, and it seemed impossible to reach a broad consensus on which deployment mechanism to use. People were divided into two main camps: some favored lockinontimeout=true (UASF camp), while others favored lockinontimeout=true ("try it and see if it fails, then figure something else out" camp). Because neither option had overwhelming support, the debate spun in circles and seemed unable to move forward. Some of the discussion took place in an online chat room (IRC), in a channel called "##taproot-activation"; however, on March 5, 2021 , things changed:

 06:42 < harding> roconnor: 有人提议BIP8(3m, false)（为期3 个月，不设强制激活的BIP8）吗？我之前提过这个问题，但我没看到有人回答。 [...]06:43 < willcl_ark_> 应该有，我自己就在想，与此相比，隔离见证的激活机制是非常直接的：直接是LOT=false ，要是没通过，那就来一次UASF 。06:43 < maybehuman> 有趣。如果我没有记错，在这个频道刚刚创建的时候，“Let's see what happens” （就是false，3m）是一个热门的选择。06:44 < roconnor> harding: 我支持。我不知道这有没有价值。根据我对其他人的顾虑的理解，基本上，我认为这会是一个能够得到广泛接受的选择。06:44 < willcl_ark_> maybehuman: 因为每个人都想要这个升级，甚至矿工们也说自己能在大约两周内升级（至少f2pool 是这么说的）06:44 < roconnor> harding: BIP8(3m,false) 加上一个延长的锁定期。06:45 < harding> roconnor: 哦，不错。从我7 个月前笫一次在wiki 上总结这些选项以来，这是我最喜欢的选项。06:45 <@michaelfolkson> UASF 无法发布(true,3m)，但是Core 可以发布(false, 3m)06:45 < willcl_ark_> harding: 对我来说它真的像是一个好方法。*如果* 它失败了，那我们可以先想想这是为什么，就不用浪费太多时间。

—— #taproot-activation IRC log

This "Let's see what happens" approach ultimately seemed to capture people's hearts. This process, later described as a "Speedy Trial" because of its short period of expression, was explained to the wider community by David Harding in an email posted in the Bitcoin-Dev mailing list .

An earlier version of this proposal was written more than 200 days ago [3], while the underlying code for taproot was merged into the Bitcoin Core software more than 140 days ago [4]. If we were to "quickly" start from the same point in time when the code was merged (which is unrealistic), we would either have activated taproot in less than two months or turned to trying to activate it again more than a month ago.

However, we have been debating since we started discussing activation schemes after Segregated Witness over a year ago, and it doesn’t seem to be any closer to what I think is a widely acceptable solution [5]. I think a “quick fix” is a way to move forward quickly, either to end the debate (which would end the debate so far if activation is successful) or to give us some real data to base future taproot activation proposals.

— David Harding, Bitcoin-dev mailing list

This activation mechanism underwent two months of improvements and was then released in Bitcoin Core version 0.21.1 . Miners quickly began to indicate their readiness for the upgrade, thus changing their deployment status to LOCKED_IN ; after a period of calm, the "Taproot" rule was activated in mid-November 2021, starting from block height 709632 .

5.2.4 Future Upgrade Mechanism

Given the issues that have arisen with recent soft forks (SegWit and Taproot), it is unclear how future upgrades will be deployed. Speedy Trial was used to deploy Taproot, but it was used because it was able to bridge the divide between UASF and MASF factions, not because it is the best deployment mechanism we know.

5.3 Risks

At any fork—whether soft or hard, user-activated or miner-activated—there is a persistent risk of a blockchain network split during the activation period. A split lasting several blocks can severely damage sentiment surrounding Bitcoin and its price. But most importantly, it creates immense confusion about "what Bitcoin is." Is Bitcoin this chain, or that chain?

The risk of users activating a soft fork is that the new rules can be activated even if most of the hash power does not support them. This could lead to a prolonged chain split until the vast majority of hash power adopts the new rules. If miners have already mined blocks on the chain using the old rules after the split, it could be difficult to migrate them to the chain using the new rules. However, it's worth mentioning an impressive event (see section 9.2.3 ): In March 2013, the Bitcoin network experienced a prolonged blockchain split due to an unexpected hard fork, but contrary to this incentive, the two major mining pools decided to abandon the branches they had already mined in order to rebuild consensus.

On the other hand, the risk of miners activating soft forks is that they might send incorrect signals, meaning the actual percentage of hash power supporting the change might be smaller than it appears. If the actual hash power supporting the change doesn't constitute a majority, we could potentially see a long-term chain split similar to those described in previous chapters. This problem, or at least a similar one, did occur during the deployment of BIP66 (see section 9.2.4 ), but was resolved within six blocks.

5.3.1 The Cost of Division

Jimmy Sone discussed the costs associated with hard forks at the Breaking Bitcoin conference in Paris, but many of the costs he mentioned also apply to chain splits caused by failed soft forks. He spoke of " negative externalities ," defined as the costs that others have to pay for your actions.

A classic example of a “negative externality” is a factory. It may be productive—an oil refinery—producing a good that benefits the economy, but it also generates negative externalities, such as pollution. It’s not just something everyone has to pay the price for—either clean it up or endure it. It also has second- and third-order follow-up effects, such as increased traffic flow as more workers are needed at the factory. There may also be wildlife around the factory. Sometimes, however, not everyone bears the cost; only a portion of the population—such as those who previously used the surrounding roads and the wildlife living around the factory—suffers the burden.

— Jimmy Song, “The Social Cost of a Hard Fork,” Breaking Bitcoin Conference (2017)

In the context of Bitcoin, he will use Bitcoin Cash (Bcash) as an example of negative externalities; Bcash is a hard fork of Bitcoin, created shortly before the Breaking Bitcoin conference in 2017. Song also distinguishes between one-time costs and perpetual costs of hard forks.

In many examples of one-off costs, he also mentioned costs incurred due to transactions.

There are many exchanges in the market, and they all have to bear significant one-off costs. First and foremost, the deposit and withdrawal windows must be suspended for one or two days because they are unsure what will happen. Many such exchanges have had to utilize cold storage due to user demand for Bcash. This is their fiduciary duty, and they must do so. You also have to audit new software. This is what we at itbit had to do. We wanted to spend Bcash—but how? Did we have to download electron cash software? Would there be malware? Therefore, an audit was necessary. It might take us 10 days to figure out if it actually works. Then, again, we had to decide whether to allow only one-time withdrawals or to list the currency. For exchanges, getting merchants to trade a new currency is not a simple matter; it involves the entire process of cold storage, signing, deposits, and withdrawals. Alternatively, you could have a one-time event to return Bcash to users, which would solve the problem. But that also has its issues. Finally, no matter what you do—let users withdraw or list a new currency—you'll need new infrastructure to operate this new currency, even if you're only offering one-time withdrawals. There has to be a way to return them to users. And you'll need to issue a temporary notification, right? Time is short and it's urgent.

— Jimmy Song, “The Social Cost of a Hard Fork,” Breaking Bitcoin Conference (2017)

He also listed one-time costs incurred by merchants, payment processors, wallet software, miners, and users, as well as perpetual costs such as privacy losses and higher chain reorganization risks.

In practice, when a blockchain split occurs, if the chain with looser rules is stronger than the chain with stricter rules, a chain reorganization will happen. This will have a severe impact on all transactions on the damaged branch. For these reasons, avoiding chain splits is extremely important at all times.

5.4 Conclusion

Bitcoin has grown and evolved over time. During this period, various upgrade mechanisms have been adopted, and its learning curve is steep. As we learn more about the network's responses, increasingly sophisticated and robust methods are constantly being invented.

To maintain harmony in Bitcoin, soft forks have proven to be the way forward. However, one major problem remains unresolved: how can we safely deploy soft forks without causing discord?