Author: Kam
Compiled by: Luffy, Foresight News
This letter is written to the Hyperliquid development team, hoping that the team can take the time to review this feedback on the management of the Hyperliquid blockchain.
Key points:
Due to the closed-source code, lack of documentation, and dependence on a centralized application programming interface (API), validators face numerous major challenges, resulting in frequent jailing and unstable performance.
The testnet incentive measures have spawned a HYPE token black market, which benefits large holders in making deals rather than fairly selecting validators.
The validator rewards on the mainnet are too low to meet the high self-staking requirements, and the degree of decentralization is limited, as 81% of the staking shares are controlled by the foundation nodes.
To compete with mainstream L1 blockchains, Hyperliquid must increase transparency, reduce staking centralization, implement a fair validator selection process, and strengthen interaction with external validators.
I started engaging with Hyperliquid in December 2023, and it is an amazing crypto application. It is easy to use, has an excellent user experience, and offers some unique features like the vault and the famous HLP. Currently, HLP manages over $350 million in assets, and anyone can participate in Hyperliquid in a passive way.
Seeing this platform perform so well, and understanding that Hyperliquid is running as its own L1 blockchain, I hope that Chorus One (a staking solution) can participate as an operator on the Hyperliquid chain. I am an employee of Chorus One, one of the largest node operators in this field. Chorus One has been very active in the Proof-of-Stake industry since 2018.
After being whitelisted on October 17th last year, Chorus One joined the Hyperliquid testnet. I would like to share our overall experience on the testnet with the Hyperliquid engineering team, as even though we have been on the testnet for nearly 3 months, we have not had the opportunity to engage with them.
During this time, we have witnessed one of the most successful token launches of 2024: the launch of the HYPE token. At the same time, we have experienced an interesting and challenging testnet environment. I would like to mention some of the issues we have observed, in the hope that they can be addressed in the coming days, weeks, or months.
Testnet Experience
So far, the testnet experience has been extremely challenging. Operators have almost no knowledge of how to run nodes, and the resources available are very limited. In addition, we have been essentially groping in the dark, and we have discovered several issues, including the following:
Frequent jailing with unclear reasons
Initially, we were jailed multiple times, but we did not understand the reason. Due to the closed-source code, we were unable to accurately assess the cause. The only thing we could do was to communicate with other validators on Discord and speculate together. After talking to several validators, we found that everyone was frequently jailed and did not fully understand the reason.
Node location
Later, we found that the jailing issue might be because we did not run our node in Tokyo. Moving the node to Tokyo might help. Unfortunately, the team never explicitly stated this, and we only discovered it after encountering numerous problems.
After moving the node to Tokyo, the situation improved somewhat. This is likely because many of the testnet nodes with large staked tokens are also located in Tokyo, so our node can miss fewer blocks and keep up with the pace. However, even after relocating the node, we still encountered jailing issues constantly, and we still did not know the exact reason. This is mainly due to the closed-source code.
Reliance on automatic unjailing scripts
We realized that maintaining good uptime on the Hyperliquid testnet depends on the speed of scripts automatically unjailing the nodes. The only way to improve uptime is to rely on scripts that can quickly unjail the nodes automatically. Validators cannot fully understand or solve the root problem and can only automatically unjail the nodes without delving into the details.
The centralized Hyperliquid API as a single point of failure
There were a few times when our unjailing attempts failed because the Hyperliquid API was malfunctioning. If the API fails, validators cannot unjail themselves, as they must send requests to the Hyperliquid server to be unjailed.
The team may be aware of this, but this design needs to be reconsidered, as it makes the API a major single point of failure for the network. If the goal is to build a Byzantine fault-tolerant system, there should be no nodes with special privileges, such as those dependent on a centralized API.
Mainnet Validator Selection
Hyperliquid recently went through a process of decentralizing its validator set, selecting around 16 validators. Previously, Hyperliquid was managed by 4 validators from the core team, which drew a lot of criticism. Recently, Hyperliquid took a significant step by expanding the number of validators from 4 to 16.
Regarding the selection of validators, 4 validators were announced through the following post on Discord:
These 4 validators are Validao, Bharvest, Hypurrstake, and Prrposefulnode. They were selected based on their uptime, maintaining over 90% uptime in the past 7 or 30 days.
This is a remarkable achievement, for several reasons. The main reason is that the validators' performance is also affected by external factors, such as Hyperliquid API failures, jailing issues, and continuous binary crashes, all of which have a significant impact on performance.
In addition to these 4 validators selected based on testnet performance, the Hyperliquid Foundation also has 5 validators running on the mainnet. Furthermore, 7 other validators have been selected to join the mainnet, but the reasons for their selection have not been publicly disclosed.
Subsequently, a black market for the HYPE testnet tokens emerged.
The Hyperliquid testnet initially had 50 validators. Initially, specific entities were whitelisted to join the testnet, but on December 12th, the validators were opened up completely.
The requirements were simple: 10,000 HYPE testnet tokens were needed to register as a validator. However, to become an active validator, one must be among the top 50, otherwise the validator will be in an inactive state.
This led to a surge in the price of the HYPE testnet tokens. Initially, the price reached over 3,000 testnet USDC, and a few days later, it even reached over 28,000 testnet USDC. At the time of writing, the current token price is around 700 testnet USDC.
Unfortunately, the faucet only distributes 100 testnet USDC every 4 hours. To be among the top 50 active validators on the testnet, it currently requires over 528,747 HYPE testnet tokens. Assuming a token price of 700 testnet USDC per token, and relying solely on the faucet to release USDC, the calculation is as follows:
Days required = (528747×700)÷(100×6) = 616871.5 days
This means that relying solely on the faucet to acquire the required HYPE testnet tokens to become an active validator on Hyperliquid would take approximately 616,871.5 days, or 1,690 years.
However, those who received HYPE airdrops on the mainnet also received the same amount of tokens on the testnet. This created an opportunity for validators to collaborate with these community members, allowing the validators to occupy a spot in the active set by having the testnet HYPE tokens staked to them.
At the same time, this situation has also given those holding testnet HYPE tokens other ideas. Given the fierce competition to join the testnet validator set, many validators are eager to acquire as many HYPE testnet tokens as possible. As a result, a black market has emerged, with large holders of testnet HYPE tokens selling them to validators in exchange for real USDC on the mainnet.
I have never seen such a chaotic situation. Although the Hyperliquid team clearly does not condone these practices, they have the ability to solve this problem. One possible solution is to implement a proper validator selection process on the testnet.
In most other Proof-of-Stake networks, the core team usually shares a spreadsheet that any validator can fill out to express their willingness to run the chain. The team then reviews these applications and performs an initial screening based on various criteria, such as the validator's experience in running nodes, past contributions, community involvement, or other factors.
This group of initially selected validators can contribute to the testnet, closely collaborate with the engineering team to provide feedback, and ensure smooth operation. We have tried multiple times to provide feedback, but have not been successful so far.
Mainnet and Decentralization
As mentioned earlier, the current validator set on the Hyperliquid mainnet consists of 16 validators, which can be viewed at: https://app.hyperliquid.xyz/staking.
5 of them are from the Hyperliquid Foundation. 4 were selected based on their performance on the testnet, maintaining over 90% uptime. 7 were selected by the Hyperliquid team.
Out of the 404,495,250 HYPE tokens staked, approximately 329,578,724 HYPE tokens are staked on the Foundation nodes, accounting for around 81.4% of the total staked amount. We don't know much about HyperBFT, but assuming it operates as a Byzantine Fault Tolerant system, the core assumption of most BFT systems is that no more than 33% of the voting power exhibits malicious behavior. If a single entity controls one-third of the staked amount, they can stop the blockchain from functioning. If they control two-thirds, they can completely control the network.
The Hyperliquid Foundation initially staked 60 million HYPE tokens for each of their Foundation nodes. However, many HYPE holders have also chosen to stake on the Foundation nodes, which is very detrimental to decentralization. The team should interact more with the community and encourage a more distributed staking distribution.
There are three potential solutions:
Promote the importance of staking with external validators to improve the chain's security and decentralization.
The Foundation nodes should implement a 100% commission rate to incentivize users to stake with external validators, promoting decentralization.
Reallocate the Foundation's staked amount to external validators, which is the practice of most chains.
Distributing the staked amount to external validators would also help them be more economically sustainable. Hyperliquid is a blockchain focused on high throughput, and the infrastructure cost of running nodes (especially in Tokyo) can be quite high. Currently, the bottom-tier validators earn between $3,000 to $5,000 per year, which is not enough to cover their costs. Particularly challenging is the requirement to stake 10,000 HYPE tokens (around $250,000 at the current price) on the mainnet to participate in validation.
Currently, users interact with Hyperliquid by cross-chaining USDC from Arbitrum to the Hyperliquid chain. Reviewing the cross-chain bridge contract, it appears that the cross-chain bridge is still managed by 4 validators. These validators seem to have no association with the chain's consensus mechanism or the 16 validators on the mainnet.
Hyperliquid has a great product, but the team still needs to improve several aspects of their infrastructure to truly compete with mainstream L1 blockchains.
Some improvements could include:
Listening to the validators. While the team's current approach of working on their own and not interacting with many external parties has been effective in building their Perp product, validators are the pillars of an L1 blockchain. Listening to their input is equally important to ensure smooth operation.
Open-sourcing the code. This would help validators better understand the challenges of running nodes on the Hyperliquid chain and also build trust with users. Open-sourcing the code would also allow validators to gain more insight into the architecture and consensus algorithm. Currently, there is very limited information about HyperBFT, and open-sourcing could provide the much-needed transparency and understanding.
Creating a proper validator selection process to prevent a black market for HYPE testnet tokens. Selecting validators based on uptime is a fair method, but obtaining good uptime should also be fair. It should not depend on factors like having connections to acquire testnet tokens, buying testnet tokens, or relying on the uptime of Hyperliquid's APIs.
Overall, Hyperliquid does not need to make major changes to compete with mainstream L1 blockchains. The main focus should be on interacting more with external parties and considering their input. I look forward to seeing how the situation develops in the coming weeks and months, and our team is ready to provide assistance and feedback at any time. Wishing the Hyperliquid team the best of luck!
