Note: This article is from @BlazingKevin_Twitter , MarsBit organizes it as follows:
1/ Arweave version 2.6 has been released, and the hard fork upgrade will be completed on March 6. The theme revolves around how to reduce energy consumption and increase storage incentives.
The last important upgrade of Arweave dates back to version 2.4 in February 21, which completed the upgrade from SPoA to SPoRA, encouraging miners to increase access speed to data.
As Arweave ushers in another major upgrade, this article will review the previous upgrades so that everyone can have a better understanding
2/ Arweave 1.5 (mainnet launch in October 2018)
The weave size when it was launched was only 177Mi. In this part, I will introduce some unique nouns in the Arweave network. Here we encountered the first noun called block weave (block weave).
The block square is the block structure of the Arweave network, which is different from the common block chain connection method.
3/ The data structure of Arweave is not strictly a single-linked list but a more complex graph structure. Each block in the structure is related to three blocks, the previous block and the next block of the current block and a random point to a previous block. The block is called the recall block (recall block/recall chunk), which constitutes the block spinning (excerpted from the Arweave yellow paper).
4/ The memory block is determined according to the hash and height of the previous block. The principle of cryptography ensures that the memory block is both certain and unpredictable when it is selected.
At this time, the performance of Arweave is as follows:
5/ Different from the mining mechanism of PoW and PoS, the mining mechanism of Arweave is Proof of Access. In the early days of Arweave, Proof of Access means that in order for a miner to obtain the right to package a new block, he must prove that he can access the data of the historical block, which means that the miner must store the historical data.
6/ In actual work, whenever a new block is generated, PoA will randomly select a historical block as the memory block, and ask the miners to put the memory block into the new block.
7/ Therefore, there is no concept of full nodes and light nodes in the Arweave network. After new miners join the network, they only need to start by storing new blocks and meeting blocks, instead of storing all historical blocks. The number of copies (recall blocks) stored between miners is also different.
8/ The PoA consensus requires miners to store memory blocks, but does not require miners to store all historical data. In the previous figure, we saw that the block time of Arweave is 2 minutes. In these two minutes, there is also a fierce PoW competition, and yes, the workload proof is included in the PoA consensus in Arweave.
9/ The two-minute mining process can be divided into two parts. The first part we have already understood is the proof of access. When a new block arrives, within 2 minutes, the PoA consensus will randomly select a historical block as a memory block, and only miners who have an access path to the memory block are eligible to enter the second part of the proof-of-work competition. If a miner does not store memory blocks, he can send requests to nearby miners.
10/ Instantly store this memory block and enter the proof-of-work with lagging speed. It can be seen from this that the more historical blocks a miner synchronizes and the more rare blocks they synchronize, the greater the possibility of passing the first part of the requirements. Arweave has cleverly changed the storage of historical data from a requirement to an incentive. In the first part, miners who own recalled blocks enter proof-of-work.
11/ In version 1.5 of Arweave, the proof of work is a pure hash power competition: consume resources, accumulate hardware. The miners frantically calculate toward the two-minute finish line, and when the two-minute time expires, the miner with the most calculations wins. Competition for the next block restarts.
12/ There is a problem with this PoA design. Since miners can download memory blocks to nearby miners in the first part, many miners choose not to store historical blocks, but pile up a large number of gpu hardware for parallel calculation. In this way, even if you enter the second part slightly behind, you can still rely on a large amount of computing power to turn defeat into victory.
13/ This strategy has gradually become the mainstream. Miners give up the storage of historical data and fast access to copies, and choose to accumulate computing power. As a result of this strategy, historical data will gradually be centralized, and the resource consumption of the entire network will explode.
The original Arweave network design has flaws, there is no limit to the stacking of GPUs. In June 2019, Arweave launched version 1.7.
14/ Arweave 1.7 (Random X, June 2019)
In order to limit the behavior of miners crazily piling up GPUs, in version 1.7, Arweave launched Random X. Random X is a hashing equation, which is characterized by being difficult to run on GPUs or Asics. After the miners cannot pile up GPUs for mining, they can only rely on a single CPU to complete the proof of work, reducing energy consumption.
15/ There is no concept of a full node on Arweave, which means that miners do not have to maintain consensus, so when a user sends a transaction, will the miners share and synchronize with each other? The answer is yes, imagine that I am a miner, I must hope to get the maximum benefit when I dig out a new block, when I receive a user transaction, I will choose to propagate the transaction to other miners in the network, so that , other miners will also synchronize their received transactions
16/ We can think of it as an incentive measure. If miners do not share transactions with each other, the number of transactions in blocks packaged by winning miners will decrease. After the user experience declines, the network will fall into a death spiral. For profit, miners actively synchronize transactions.
17/ Arweave is a storage network, unlike ordinary blockchains, a transaction is up to 5.8MB. Miners need to synchronize all transactions within two minutes and complete the proof of work, which puts forward requirements on the transmission speed of transactions.
18/ At the same time, the scalability of the Arweave network is also limited. Miners need to ensure that the access or download of the recalled block is completed in the first part, and the user's new transaction is synchronized, otherwise in the second part of the workload proof , miners cannot be on the same starting line. Arweave's network development encountered transmission bottlenecks.
19/ Arweave 1.8 (the maximum transaction size doubled to 10MB in October 2019)
The size of 5.8MB is more than enough for normal transactions, but it is still too small for the storage network, and sometimes it can't even fit a picture, let alone audio and video. In October 2019, Arweave increased the maximum size of a single transaction to 10MB.
20/ However, the bottleneck of transaction transmission is still not resolved. The ultimate goal of Arweave is to store human history and become a modern library of Alexandria. To reach this endpoint, Arweave had to address scalability issues. Arweave is a distributed network architecture with miners scattered in different regions of the world. Objectively, the hardware used by miners and the network conditions they have are different and cannot be unified.
21/ The performance of the Arweave network actually depends on the average transmission speed of all nodes. Increasing the number of miners in the network will increase the transmission speed requirements of synchronous data per unit time. Blindly increasing miners or increasing the transaction and block size to expand the network will As a result, user transactions are lost, so at this stage, Arweave cannot improve scalability, and the growth rate of historical data in the network is at a low speed.
22/ Arweave 2.0 (SPoA March 2020)
In order to break the transmission bottleneck, Arweave introduced two concepts in version 2.0: succinct proof (succinct proof) and format 2 transaction (new transaction format).
Succinct proofs use a common technique, the Merkle tree, when miners package a new block, they must prove that they have a copy/access of the recalled block and include the recalled block in the new block.
23/ This makes miners who have not stored the recalled block first consume bandwidth to download the entire recalled block from nearby miners. As the capacity of a single transaction increases, the block size also increases, and the bandwidth requirements for transmitting the recalled block also increase. Further improve. In order to maximize profits, miners must wait for transactions from other miners as much as possible, and miners who have not stored recalled blocks will start sharing transactions after synchronizing recalled blocks.
24/ The increase in block size shortens the time for miners to share transactions, and some miners have to start proof of work without synchronizing all transactions. In order to solve this problem, the succinct proof allows the miner to package the memory block according to the sorting method of the Merkle tree, and finally generates a root proof, which is also called a succinct proof, which is used to prove that the miner can store the historical block.
25/ The concise proof replaces the memory block, which can be synchronized by miners or put into a new block, saving block space and block transmission costs.
format 2 transaction is a new transaction format introduced by Arweave, which helps it increase the transaction volume that can be accommodated in a block without an upper limit. Old transaction data includes header and data, which cannot be separated.
26/ The header and data of the transaction in format 2 can be split. Why can splitting the transaction increase the block transaction volume without limit? Let's go back to the 2-minute block time of version 2.0. In the first part, miners will synchronize concise proofs, quickly establish and recall block connections, and at the same time, the transaction format of the old version will not be deleted.
27/ In the first part, format 1 transactions will still be fully synchronized between miners, and header and data will not be divided. The difference is that only the header of the format 2 transaction sent by the user is put into the new block, and only the header is synchronized between miners.
28/ The introduction of concise proofs and format 2 transactions greatly reduces the amount of data synchronized between miners in the first part and improves the scalability of Arweave. When a new block is generated, its block composition is: a concise proof for recalling the block, the complete data of the format 1 transaction, and the header of the format 2 transaction.
29/ It can be imagined that the block at this time is a huge Lego model, but many parts in the middle are blank, and these blank parts are the data part of the format 2 transaction. The data data will be synchronized when the next block proceeds to the second part, because the workload proof does not occupy the bandwidth, and it is completely possible to use the bandwidth to synchronize the untransmitted data data of the previous block when the CPU performs hash calculations.
30/ There is no problem with only the transaction header when a new block is generated, because the verification can also be completed through the header, but the data must be completed later, because the current block will be used as a memory block at some point in the future, and the memory block can survive concisely Proof, but only if the data is complete.
31/ At this point, the scalability of Arweave has been released, but new problems have arisen. Different from the strategy of stacking GPUs mentioned above, the introduction of SPoA has led miners to another path. We know that the introduction of succinct proofs allows miners to quickly recall block information synchronously. Instead of storing historical blocks, miners choose to wait for other miners to synchronize succinct proofs and shift the cost to mining hardware.
32/ Arweave 2.4 (SPoRA February 2021)
PoA can only guarantee permanent storage, not access speed. Without a competitive advantage in data retrieval, miners benefit from using remote storage pools rather than maintaining separate, decentralized nodes. In order to solve the problem that miners no longer store copies of history, Arweave upgraded SPoA to SPoRA (succinct proof of random access)
33/ Let's go back to 2 minutes to the block time again. In the first part, there is no change. Miners can qualify for the second part by synchronizing concise proofs. Most miners choose to use remote storage pools. With increased bandwidth speeds, fast syncing and access to remote memory, building a storage pool yourself isn't an option.
34/ The introduction of SPoRA makes the second part different. The hash calculation proposed by SPoRA will require miners to calculate the hash value for a certain transaction in the memory block, generate a lightweight and concise proof, and when performing the second part, Proofs of succinctness cannot be passed between miners.
35/ For miners who have not built their own storage pool, it can be imagined that they are forced to return to the first part, find the hash value in the remote storage pool, and then enter the second part of the workload calculation. However, SPoRA is random and uninterrupted in recalling hash calculations in blocks, and miners who do not have a personal storage pool will continue to be returned to the first part.
36/ Such a design requires miners to maintain their own personal storage pools. SPoRA reduces the weight of previous miners' block generation probability and adds consideration of data access speed.
37/ SPoRA creates a more decentralized and efficient block weaving by suppressing the resource pool between CPUs, allowing miners to focus on maintaining local hardware and nodes, achieving geographical diversification and decentralization, thereby incentivizing miners Replicate data more efficiently and quickly. (Excerpt from the Arweave Yellow Paper)
38/ The new design has brought new problems. Now we can ensure that miners store as many copies of history as possible on their personal hard drives. However, as mentioned above, SPoRA has added data access speed considerations. If miners cannot quickly find the transaction data required by SPoRA in the hard disk, and cannot generate light and concise proofs in time, they will not be able to obtain mining rewards.
39/ SPoRA allows miners to pursue hard disk reading speed, to retrieve data faster, and calculate hash value faster means higher rewards. To add, the price of the hard disk is basically proportional to the reading speed. The reading speed of the 70-knife mechanical hard disk is 750MB/s, while the reading speed of the 700-knife solid-state hard disk is 7300MB/s.
40/ Miners need to pay more if they want higher income. This is a reasonable mechanism, but there are better solutions. SPoRA does not consider the entry threshold of Arweave miners, which limits the scalability of Arweave in disguise.
The above is for @ArweaveEco A review of previous upgrades of @ArweaveNewsCN , the article is a bit long, thank you for reading here. Next
I will understand Arweave2.6 from my own perspective, thank you
