The response to the first half was good, but the real dry goods are in the second half.
It took several days to write the second half of the content, because the post introducing the system is not easy to write. The writing is too esoteric and most people don’t have the foundation to understand it at all, and the writing is too simple and everyone thinks it’s nothing more than that. You can only hold back a little bit, and write whatever you want.
If the retail hand-rolling party belongs to the farming society, the script studio should be regarded as the industrial revolution, and our automated hair-rolling system can be regarded as a nuclear weapon.
Someone left a message saying that I sell the system. Do you think my structure is so low? In fact, I am here to dissuade you farmers, haha.
1. Needs and Pain Points
The most important thing to develop a system is not technology but a full understanding of needs and pain points. During the development of the entire system, I also consulted and learned from many industry leaders, learned about the operation modes and pain points of various teams, and learned a lot of good strategies.
Pain points of handjob studio:
1. High cost of manpower and equipment
2. High management costs, including training for new projects, acceptance of daily tasks, etc.
3. Fund security issues. What to do if it is stolen, and manual misoperation may also cause losses. Although it can be solved by castrating the little fox's wallet, etc., it is not a cure for the symptoms, so I dare not invest in large funds.
4. Homogenization of operation mode, concentration of operation time, etc.
Pain points of Script Studio:
1. It is also a matter of financial security. Most of the private keys are stored in plain text or simply encrypted, and are directly called by scripts. The security is very low, and large funds are not dared to be invested.
2. The script is relatively homogeneous, not flexible enough, and easy to use.
3. Without a perfect management system, operation and management are more difficult.
4. Most of the front-end clicks are simulated through selenium. The script development efficiency is low, and each project has to be redeveloped. And it eats more servers and bandwidth
To sum up, the main needs and pain points are: improving the security of funds, reducing operation management and labor costs, reducing server consumption and costs, and improving the richness of strategies (making each wallet unique, like an independent person) , Simplify the development difficulty of scripts and strategies
Combining with the actual needs of our team, we feel that security is the highest priority, because we all have a boutique account strategy and a large number of accounts. Take a team of 5 people and each project with 1,000 accounts per person as an example. For zksync, even if we keep a minimum of 0.2E for each account, a project with 5,000 accounts will get 1,000E. These are stored on the server for a long time and must be secured.
The second is to reduce labor costs, and it can be fully automatic and unattended.
The last is to simplify the development difficulty of strategies and scripts, achieve more than 90% configurability, and fast development speed is the core competitiveness in the later stage.
2. System architecture
Based on the above requirements and pain points, the system architecture we designed is as follows, mainly divided into several modules:
The following is a simple architecture diagram
1. Private key center:
Considering the security of our large funds, and of course the cost of servers, all our servers except the private key central database are placed on self-built servers in the office, with an average of 5,000 yuan per two-way server, divided by EXSI 10 virtual machines (linux and windows) can run at least 10,000 scripts at the same time, which is extremely cost-effective.
All private keys are double-encrypted and stored in Alibaba Cloud's private key server. The script server can only use it after connecting to the private key server to decrypt it every time it executes a task, and only the person knows the password.
There are also security measures such as a whitelists between the script server and the private key server. Whether the script server or the private key server is hacked alone, it will not leak the secrets. The security level is basically equivalent to some MPC wallets outside.
2. Asset Management Center:
The assets here refer to information including IP, Twitter, DC, email, recharge address, etc. Each Lumao account corresponds to a completely independent set of assets, which are allocated according to different tasks to achieve 100% isolation and witch prevention.
3. Network management center:
The network is also where we spend a lot of time exploring and optimizing. No matter how much you spend on an IP purchased from outside, in fact, the connection speed in China is generally not fast. Everyone who has bought an IP knows that it takes several seconds to connect and open a web page in the fingerprint browser.
And we bought the cheapest one with 1000 IPs only 100U a month. So we made a powerful routing architecture. The enterprise broadband in the office is connected to a load-balanced self-built airport cluster, and then connected to these IPs. The speed and stability are greatly improved, and the cost is extremely low.
Now that we have launched two or three projects, the daily traffic will be more than 100G, and the subsequent projects will have to continue to expand.
4. Task scheduling center:
The task scheduling center is also one of the cores of this system. Most of the scripts on behalf of the script, the so-called witch prevention is to add some random variables and random waits between the scripts running, the logic is very simple and rough, and people need to be on duty when running.
Based on a set of open-source task scheduling frameworks on the Internet, we have developed a scheduling center that is suitable for mao. It can run unattended 7*24 hours. Any error has functions such as breakpoint redo, and supports parallel tasks and There are two modes of serial tasks .
Take 1000 wallets as an example:
The so-called parallel task means that these 1,000 wallets are configured to execute 5 chain interactions each week, which is completely random and automatically executed at the appropriate time of the week, and can be executed when the GAS is the lowest (such as setting the Mainnet GAS Do below 20, otherwise wait).
Serial tasks are suitable for large amounts of funds. For example, each wallet transfers 100E of funds once a month. If parallel tasks require 1000*100 E, it is obviously impossible to have so much money. Then you can use the serial mode, the first wallet withdraws coins from the exchange, Cross-chain, a series of interactions on the chain, Cross-chain back, recharge back to the exchange, and then continue to execute the next one. Any problems in the middle will be abnormal processing mechanism.
Combining the parallel mode and the serial mode, a very rich interaction strategy can be combined.
Moreover, based on the development of the basic script, the combination of strategies only needs to configure the json file and the corresponding parameters, which can achieve zero development amount in the later stage.
5. On-chain script development framework
I have done research, and more than 90% of the scripting studios realize the interaction through the fingerprint browser to simulate the front-end interaction, which is the kind of dozens of windows that you see many people posting on the Internet.
Compared with directly invoking contract interaction, each project needs to develop a set of front-end scripts, the speed of front-end interaction is also very slow, the probability of error is also high, and it is very server-intensive, and the Gas fee is much higher.
The remaining 10% can only write some simple contract interaction scripts, and their understanding of web3 is very limited.
And I have been playing DEFI for so many years, so it can be said that I am very familiar with the contract codes of various projects. We have also done the price difference monitoring of the whole chain DEX very early, and we can just take that framework and change it.
In fact, the projects on the chain are mainly divided into three categories: DEX, lending, and MINT NFT.
There are mainly two bottom layers of DEX, one is based on uniswap V2, and the other is based on uniswap V3
The corresponding Farm functions are also divided into two types: the early popular sushi swap Masterchef mode, and the now popular solidly VE(3,3) mode.
There are also two main types of lending, one is compound-based projects, such as Venus. One is an AAVE based project such as Radiant.
MINT NFT or domain name is even simpler, much the same.
So you only need to abstract these functional modules into components, and you can develop them as quickly as building Lego.
At present, the EVM chain or EVM-like chain, we only need 5 minutes to do some parameter configuration to develop a new project, and then spend 10 minutes to test it.
Just send a piece of code for you to take a look, 22 lines of code can realize the configuration of a lending platform.
6. Front-end (selenium) script development framework:
For most web3 projects, in fact, on-chain interaction is enough. But now more and more project parties are constantly doing various tasks on various platforms such as galxe in the pua masturbation party. Like this time, linea released more than 30 tasks a week, let alone masturbation, most teams didn't even have time to develop scripts. Because various Twitter, DC and other tasks are involved, the front-end selenium simulation has to be used.
Therefore, we have also developed a set of front-end frameworks simultaneously, and made various functional modules into components, such as Twitter module, DC module, etc., combined with interactive scripts on the chain, to make the task platform configurable. It can do a galxe task, and only need to configure a json file to realize automatic execution.
At present, the completion rate of this area is about 60%. Each module has a lot of development, but after the development is completed, it will be done once and for all.
After writing so much, there is no way to fully explain the idea of system design, but I believe that students with basic knowledge should also be able to gain a lot.
If you don’t understand students, don’t ask me, I’m not a teacher and I don’t have time to teach you, it’s impossible to provide source code, and I don’t sell the system.
Welcome everyone to correct and communicate.
