On April 17, the 12th IOSG Ventures Old Friends Reunion was held as scheduled. The theme of this event was "Singularity: AI x Crypto Convergence", so we also invited outstanding representatives who are emerging in the industry. The purpose of this gathering is to allow participants to discuss the convergence of artificial intelligence and cryptocurrency fields and the impact of this convergence on the future. In such an event, participants have the opportunity to share their insights, experiences and ideas, thereby promoting cooperation and innovation within the industry.
Next is one of the keynotes of this event, which will be presented by Illia Polosukhin, co-founder of NEAR Protocol, a portfolio of IOSG Ventures , on "Why AI Needs to be Open – Why AI Needs Web3"
Why AI Needs to be Open
Let's talk about "Why AI needs to be open". My background is in Machine Learning, and I have been working on various machine learning projects for about ten years of my career. But before getting involved in Crypto, natural language understanding, and founding NEAR, I worked at Google. We now develop the framework that drives most modern AI, called Transformer. After leaving Google, I started a Machine Learning company so that we could teach machines to program and change how we interact with computers. But we didn't do it in 2017 or 18, it was too early, and there was no computing power and data to do it.
What we did was to attract people from all over the world to do the work of labeling data for us, mostly students. They were in China, Asia, and Eastern Europe. Many of them did not have bank accounts in these countries. The United States is not very willing to send money easily, so we started to think about using blockchain as a solution to our problem. We want to make it easier to pay people around the world in a programmatic way, no matter where they are. By the way, the current challenge of Crypto is that although NEAR solves a lot of problems now, usually you need to buy some Crypto first before you can trade on the blockchain to earn it, which is the opposite of the process.
Just like businesses, they'll say, hey, first you need to buy some equity in the company to use it. This is one of the many problems we're solving at NEAR. Now let's go a little deeper into the AI side. Language models are not new, they've been around since the 50s. It's a statistical tool that's been widely used in natural language tools. For a long time, starting in 2013 with the re-launch of deep learning, a new innovation started. The innovation is that you can match words, add them to multi-dimensional vectors and convert them into mathematical form. This works well with deep learning models, which are just a lot of matrix multiplications and activation functions.
That allowed us to start doing advanced deep learning and training models to do a lot of interesting things. Now, looking back, what we were doing at the time were neural networks, and they were very much modeled after humans, and we could read one word at a time. So that was very slow, right? If you were trying to show something to a user on Google.com, no one is going to wait to read Wikipedia , say, five minutes later to give you an answer, but you want the answer right away. So the Transformers model, which is the model that drives ChatGPT, Midjourney, and all the recent progress, all came from this same idea of having something that can process data in parallel, can reason, and can give you an answer right away.
So one of the major innovations of this idea here is that every word, every token, every image patch is processed in parallel, taking advantage of our GPUs and other accelerators that have highly parallel computing capabilities. And by doing that, we are able to do inference on it at scale. And that scale enables scaling up training to handle automatic training data. And so after that, we saw Dopamine, which did amazing work in a short period of time, enabling explosive training. It has a lot of text and is starting to achieve amazing results in reasoning and understanding the world's language.
The direction now is to accelerate innovation in AI, which was previously a tool that data scientists, machine learning engineers would use, and then somehow explain in their products or be able to go and discuss the data with decision makers. Now we have this model of AI communicating directly with people. You may not even know that you are communicating with the model because it is actually hidden behind the product. So we have experienced this transformation from those who previously understood how AI works to understanding and being able to use it.
So just to give you some context here, when we say we are using GPUs to train our models, this is not the kind of gaming GPUs that we use on our desktops to play video games.
Each machine typically has eight GPUs, and they're all connected to each other through a motherboard, and then stacked into racks, with about 16 machines in each rack. Now, all of these racks are also connected to each other through dedicated network cables to ensure that information can be transferred directly between the GPUs at a very high speed. So the information doesn't fit on the CPU. In fact, you don't process it on the CPU at all. All the computation happens on the GPU. So this is a supercomputer setup. Again, this is not a traditional "Hey, this is a GPU thing." So a model of the size of GPU4 used 10,000 H100s to train in about three months, and it cost $64 million. You understand the scale of the current costs and how much it costs to train some modern models.
The important thing is, when I say the systems are connected, the current connection speed of the H100, the previous generation, is 900GB per second, and the connection speed between the CPU and the RAM inside the computer is 200GB per second, which is local to the computer. So the speed of sending data from one GPU to another GPU in the same data center is faster than your computer. Your computer can basically communicate with itself in the box. And the new generation is basically 1.8TB per second. From a developer's perspective, this is not an individual computing unit. These are supercomputers with a huge memory and computing power that provide you with extremely large-scale computing.
Now, this leads us to the problem that these large companies have the resources and the ability to build these models that are now almost provided as a service to us, and I don't know how much work is actually in it, right? So that's an example, right? You go to a completely centralized corporate provider and you put in a query. What happens is that there are several teams that are not software engineering teams that decide how the results are going to appear, right? You have a team that decides what data goes into the data set.
For example, if you just scrape data off the internet, the number of times Barack Obama was born in Kenya is exactly the same as the number of times Barack Obama was born in Hawaii, because people love to speculate on controversy. So you decide what to train on. You decide to filter out some information because you don't believe it's true. So if individuals like this have made decisions about what data is going to be used and that data exists, those decisions are largely influenced by the people who made them. You have a legal team that decides what content we can't view because it's copyrighted and what's illegal. We have an "ethics team" that decides what's unethical and what content we shouldn't show.
So in a way, there's a lot of this filtering and manipulation going on. These models are statistical models. They pick out of the data. If something isn't in the data, they don't know the answer. If something is in the data, they're likely to treat it as fact. Now, that can be worrisome when you get an answer from an AI. Right. Now, you're supposedly getting an answer from the model, but there's no guarantee. You don't know how the result was generated. A company could potentially sell your specific session to the highest bidder to actually change the result. Imagine you ask which car to buy, and Toyota decides that it thinks it should favor Toyota, and Toyota will pay this company 10 cents to do that.
So even if you use these models as a knowledge base that's supposed to be neutral and represent the data, actually before you get the results, there are a lot of things that happen that can bias the results in a very specific way. And this has raised a lot of questions, right? This is basically a week of different legal battles between big companies and the media. The SEC, pretty much everybody is trying to sue each other right now because these models bring so much uncertainty and power. And, if you look forward, the problem is that big tech companies will always have an incentive to continue to increase revenue, right? Like, if you're a public company, you need to report revenue, you need to continue to grow.
To achieve this, if you already have a target market, let's say you already have 2 billion users. There aren't that many new users on the internet anymore. You don't have a lot of options other than to maximize average revenue, which means you need to extract more value from users who may not have much value at all, or you need to change their behavior. Generative AI is very good at manipulating and changing user behavior, especially if people think of it as all-knowledge intelligence. So we have this very dangerous situation where there's a lot of pressure to regulate, and regulators don't fully understand how this technology works. We have very little protection for users from manipulation.
Manipulative content, misleading content, even without ads, you can just take a screenshot of something, change the caption, post it on Twitter, and people go crazy. You have economic incentives that lead you to constantly maximize revenue. And, it's not actually like inside Google you're doing evil, right? When you decide which model to launch, you do an A or B test to see which one brings in more revenue. So you're constantly maximizing revenue by extracting more value from the user. And, the users and the community don't have any input into what the model is, what data is used, and what goals are actually trying to achieve. That's what the users of the app are. It's a moderation.
This is why we are constantly promoting the integration of WEB 3 and AI. Web 3 can be an important tool that allows us to have new incentives and to motivate us to produce better software and products in a decentralized manner. This is the general direction of the entire web 3 AI development. Now, in order to help understand the details, I will briefly talk about the specific parts. The first part is Content Reputation.
Again, this is not a pure AI problem, although language models bring tremendous power and scale to how people manipulate and exploit information. What you want is a kind of cryptographic reputation that is traceable and traceable that shows up when you look at different pieces of content. So imagine you have some community nodes that are actually encrypted and found on every page on every website. Now, if you go beyond that, all of these distribution platforms are going to be disrupted because these models are now going to be reading almost all of this content and giving you a personalized summary and a personalized output.
So we actually have an opportunity to create new creative content, rather than trying to reinvent, let's add blockchain and NFTs to existing content. A new creator economy around model training and inference time, where the data that people create, whether it's a new publication, a photo, a YouTube, or the music that you create, will go into a network based on how much it contributes to the training of the model. So based on that, there can be some compensation globally based on the content. So we move from the current economy of eyeballs driven by ad networks to an economy that actually brings innovation and interesting information.
One important thing I want to mention is that a lot of the uncertainty comes from floating point operations. All of these models involve a lot of floating point operations and multiplications. These are non-deterministic operations.
Now, if you multiply them on GPUs of different architectures. So you take an A100 and an H100, the results are going to be different. So a lot of approaches that rely on determinism, like cryptoeconomics and optimism, are actually going to have a lot of difficulty and it's going to take a lot of innovation to make this happen. And finally, there's this interesting idea that we've been building programmable money and programmable assets, but if you could imagine that you add this intelligence to them, you can have smart assets that are now defined not by code, but by the ability to interact with the world in natural language, right? That's where we can have a lot of interesting yield optimization, DeFi, we can do trading strategies inside the world.
Now the challenge is that all current events don't have strong robust behavior. They're not trained to be adversarially robust because the purpose of the training is to predict the next token. So it's easier to convince a model to give you all the money. It's really important to actually solve this problem before we go on. So I'll just leave you with this idea that we're at a crossroads, right? There's a closed AI ecosystem that has extreme incentives and flywheels because when they launch a product, they generate a lot of revenue and then they plow that revenue back into building the product. But the product is inherently designed to maximize the company's revenue and therefore maximize the value extracted from the user. Or we have this open, user-owned approach where the user is in control.
These models actually work in your favor, trying to maximize your profits. They provide you with a way to actually protect you from a lot of the dangers on the Internet. So that's why we need more development and applications of AI x Crypto. Thank you everyone.
