How Can Trading Subnets Do Better?

In Bittensor, there are two subnets that are strongly related to transactions, one is Subnet 8 Propriety Trading Network, and the other is Subnet 28 Foundry S&P 500 Oracle. Today, the ratio of TAO Emissions of the former is about 3.82%, and that of the latter is about 1.79%. So, do their current outputs match their incentives, and what can be optimized in the future?

Subnet 8 Propriety Trading Network (PTN)

Emission: 3.82% (2024–07–15)

Github: https://github.com/taoshidev/proprietary-trading-network/tree/main

SN8 provides a simulated trading system, and the available trading targets include foreign exchange, crypto assets (currently only BTC and ETH) and indices. Traders can trade these targets according to established rules and build investment portfolios. For specific rules, please refer to: official documents .

Miners act as Traders and submit Long/Short transaction instructions in the network; Validators are responsible for processing these instructions and tracking the performance of each Miner's portfolio in real time. Miners are ranked according to a set of scoring criteria. Only Miners ranked in the top 25 and not punished will receive TAO Emissions incentives.

How do the scoring and penalty mechanisms work?

The score is calculated based on the return rate, Omega ratio and Sortino ratio of the Miner's portfolio. The proportion of TAO Emissions each Miner receives is determined by the proportion of the score.

However, even if the comprehensive score is high, the punished miners will not be able to obtain incentives. Miners will be punished if the following two situations occur.

First, the consistency penalty . If Miners cannot maintain relatively stable trading performance for 30 consecutive days, they will be punished. Stable trading performance includes two aspects: first, Miners must remain active and have traded for at least 18 days in 30 days; second, the net value of Miners' portfolio cannot fluctuate too much in a short period of time ( the fluctuation within 6 hours is higher than 30 times the standard ).

Second, the drawdown penalty , the maximum drawdown range is limited to 0.25% to 5%, if the miner's maximum drawdown is lower than 0.25% or higher than 5%, he will be punished. In addition, the maximum drawdown refers to the entire historical situation and is not limited to the 30-day performance window period.

For example, Miner-5GhRddUNcwWSaaa8o5ipcYr4HLCYMg1WwH3rUWdF6RHgE581 has the highest return rate, but his maximum drawdown exceeded 16%, and he was punished with a drawdown, resulting in almost no incentives being allocated.

Obviously, PTN encourages a sound and relatively conservative investment strategy.

First, in terms of the scoring mechanism, we choose to refer to the Omega ratio and Sortino ratio, two indicators that focus on tail risk and downside risk, rather than just considering the rate of return. In addition, the concept of diversified investment is also reflected in the long-term goals of PTN. PTN is not just for training a specific trading model, but hopes to screen out several outstanding trading models by maintaining a highly competitive simulated trading ranking competition. The investment portfolios of these models are weighted averaged according to the ranking to obtain an aggregated investment portfolio, thereby reducing the risk of relying on one model.

Such a simulation system seems to be highly competitive. The model must not only guarantee a certain rate of return, but also maintain a low drawdown. But can the winning model be truly put into large-scale actual transactions?

Unfortunately, due to defects in the simulation system design, the winning model may not perform as well in the real market as in the simulation market.

There are many unreasonable aspects in the design of trading rules:

1. It ignores market liquidity and slippage during transactions : all trading instructions can be fully executed according to the quotes in the simulation system, there is no transaction delay or change in the actual transaction price, which is obviously inconsistent with the real market.
2. Ignoring the possibility of liquidation : The simulation system does not take into account the situation of insufficient margin and has no mechanism for forced liquidation.
3. Exaggerated capital utilization : Although the maximum leverage ratio for each trading pair is limited, no reasonable leverage ratio limit is set for the overall position of Miners; and, it is assumed that all positions can share margin, which is significantly different from the actual trading system.
4. Fixed borrowing rates and funding rates are not realistic : in reality, these transaction costs vary with market fluctuations; fixed rates may underestimate these costs and thus exaggerate the return on investment.
5. The supported order types are too limited : The simulation system essentially only accepts orders of the market price full execution type, and does not support the most basic Stop-loss or Take-profit type orders, which limits the flexibility of the strategy.
6. Excessive restrictions on trading frequency and holding period : The fastest trading frequency is limited to one order every 10 seconds, and the shortest holding period must be more than 15 minutes, which also limits the flexibility of the strategy.

The inherent flaws of the simulation system itself will also exacerbate the failure of the winning model to adapt to the real market:

1. Ignoring the impact of transactions on the market and the competitive pressure in the real market: whether the transactions in the simulation system can be completed does not refer to whether the same real orders are executed in reality, nor does it consider the impact of these transactions on the market, ignoring the reflexivity of transactions.
2. The tail risk of winning miners may be underestimated: Although the scoring mechanism includes indicators for measuring tail risk and downside risk, defects in the design of trading rules may underestimate the actual transaction costs and overestimate the capital utilization rate, thereby causing the model's rate of return to be overestimated. These indicators may not be accurate.

So, are there people who are actually trading based on Miners’ strategies? And what are their actual performances?

Even though there is already a product on the market, it is still difficult to draw conclusions about the actual performance of these strategies.

Dale is a robot that trades based on signals from Tarvis (the ninth-ranked Miner in PTN). It has been traded on Bybit for 45 days and currently has 168 users copying trades, AUM exceeding 400,000 USDT, and total profit close to 20,000 USDT.

For Bittensor, Dale is a worthy attempt, and it is an example of real users enjoying the output of Bittensor. Since its launch, a total of 838 users have copied orders, of which 217 users have made profits, 305 users have broken even, and 316 users have lost money. The user with the highest profit used 130,556 USDT, copied orders for 33 days, earned 3,871 USDT, and had a return rate of 2.96%; the user with the highest loss used 135,755 USDT, copied orders for 7 days, lost 7,503 USDT, and had a return rate of -5.52%.

However, since Tarvis’ strategy includes many foreign exchange transactions, Dale will only copy Tarvis’s transactions on BTC and ETH with 10x leverage, which can only partially reflect Tarvis’s actual performance.

Moreover, although the performance is good in terms of yield (+25.98%) and trading success rate (72%), considering that the overall online time is only 45 days and the vast majority of trading profits come from the week of June 11 to June 18, it is difficult to say that this is a stable and continuously profitable strategy as advertised by PTN.

In addition, it should be clarified that the profit of nearly 20,000 USDT is the total profit of all copycat users and Dale, and cannot be simply understood as the income of SN 8. Even the top validators may confuse this.

Validators should think more seriously about how to assign weights to Subnets. Whether or not revenue is generated should not be the only indicator. Current output and future potential should also be considered to be worthy of the current incentive ratio. If the 11.83% Emissions are maintained, it means that 851.76 $TAO are allocated to SN 8 every day, which is equivalent to more than $250,000 in incentives. Using $250,000 of release per day to reward a total of $20,000 in trading profits is obviously not a good deal.

Taking a step back, even at the current Emissions of about 3.82%, it means that 275.04 $TAO are allocated to SN 8 every day. With so many incentives, sn8 should also do better.

Subnet 28 Foundry S&P 500 Oracle

Emission: 1.79% (2024–07–15)

Github: https://github.com/foundryservices/snpOracle

SN28 built a network to predict the price of the S&P 500 index. Validators are responsible for sending future timestamps to Miners, and Miners are required to provide the S&P 500 prices for six consecutive 5-minute periods after the timestamp. Validators record these predictions and score Miners based on how close the predictions are to the actual results.

How does the scoring system work?

SN 28 uses root mean square error and directional accuracy to evaluate miners, with a 50–50 weighting of the two.

1. Root mean square error: The square root of the mean of the sum of squares of the differences between each prediction and the actual value of Miners. The specific formula is as follows:

The smaller the RMSE value is, the closer the model's predicted value is to the actual value, and the higher the model's prediction accuracy is.

2. Directional accuracy: Even if Miners are unable to accurately predict specific values, as long as the predicted direction of change (up or down) is correct, it is considered to be directionally correct.

So, how accurate are the predictions of Miners on SN 28?

The backtest data of the past 30 days is not satisfactory. The green line represents the actual trend of S&P 500, and the other lines represent the predictions of Miners. It can be seen intuitively from the figure that the gap between the predicted value and the actual value is not small, and the direction is not always correct.

Worse still, SN 28 is hardly a subnet that encourages competition.

The difference in incentives received by different miners is very small, and none of them are outstanding. Currently, there are 312 miners in the network, and the ratio of incentives allocated to the top miner is 0.485%, while the ratio of 234 miners is above 0.4%. This reflects that the prediction accuracy of most miners is similar, and none of them can be called accurate.

Given SN 28's current performance, such a result does not seem to have any practical use.

After understanding the actual operation of these two subnets, let’s answer the questions raised at the beginning.

Are incentives overestimated given current outputs?

Both SN 8 and SN 28 should do better to be worthy of the current incentives.

For SN 8, as one of the top 5 subnets in TAO Emissions, it is difficult to convince the public by relying only on a simulated trading system that still has many defects. These defects may cause the winning strategies in the simulation to be unsuitable for actual applications. In the simulation system, the cost of transactions may be underestimated and the impact of transactions on the market may be ignored, making some objective indicators unable to accurately evaluate the actual performance of miners. The winning model from PTN may not be widely used in real trading.

For SN 28, the discontinuous and poorly accurate price predictions are even further away from practical application. Due to the lack of a mechanism to stimulate effective competition among Miners, even the predictions given by the top-ranked Miners are unreliable, not to mention being used to guide transactions.

What areas can be optimized in the future?

For SN 8, in addition to fixing the loopholes in the simulation system, it is also necessary to consider incorporating the actual performance of the model into the scoring indicators. Because the difference between the simulation system and the real market is inevitable, even a small difference may cause the actual performance to be far from the simulation results. In addition, considering the actual performance will also encourage miners to develop more products like Dale, accelerating the process of Bittensor output being widely used by real users.

For SN 28, the most urgent task is to develop a more complete scoring mechanism to encourage effective competition among Miners and improve the accuracy of prediction results. In addition, we should also find practical application scenarios for Miners' output. If we just predict for the sake of prediction, there is no need to waste TAO Emissions on the "lottery game" among Miners.

Reference

1. https://github.com/taoshidev/proprietary-trading-network/tree/main
2. https://docs.taoshi.io/ptn/miner/overview/
3. https://dashboard.taoshi.io/miner/5GhCxfBcA7Ur5iiAS343xwvrYHTUfBjBi4JimiL5LhujRT9t
4. https://dashboard.taoshi.io/miner/5G3ys2356ovgUivX3endMP7f37LPEjRkzDAM3Km8CxQnErCw
5. https://www.bybit.com/copyTrade/trade-center/detail?leaderMark=TwqtPCVsAiXw/1F21f1byQ==&ref=NNBM3N&inviteUuid=2NDbnUXx+LO/7FrPoz5bKm0zT3hZuoOJVO646IKNUbKB038yNU1VuPD25xgDiFnA&af_xp=custom&pid= copy_trade&is_retargeting=true&c=copy_trade-web_to_app&af_force_deeplink=true
6. https://github.com/foundryservices/snpOracle
7. https://bittensor.foundrydigital.com/
8. https://x.com/fish_datura/status/1806801342645583960?s=46&t=sfxHJI4f3g5nVyB50vFXPw