With the rapid evolution of AI models, how to efficiently infer these large models has become a crucial issue that the industry cannot avoid. The open-source project vLLM from UC Berkeley not only directly addresses this technical challenge but has also gradually built its own community and ecosystem, even giving rise to Inferact, a startup focused on inference infrastructure. This article will take you deep into the origins of vLLM, its technological breakthroughs, the development of its open-source community, and how Inferact aims to create a "general-purpose engine for AI inference."
From Academic Experiment to GitHub Star Project: The Birth of vLLM
vLLM originated from a doctoral research project at UC Berkeley aimed at addressing the inefficiency of inference in large language models (LLMs). At the time, Meta open-sourced the OPT model, and Woosuk Kwon, one of vLLM's early contributors, attempted to optimize its demo service, discovering that it represented an unsolved inference system problem. "We thought it would only take a few weeks, but it opened up a completely new path for research and development," Kwon recalled.
Bottom-up Challenge: Why does LLM inference differ from traditional ML?
vLLM targets autoregressive language models, whose inference process is dynamic, asynchronous, and cannot be batch-processed, making them very different from traditional image or speech models. The input length of these models can range from a single sentence to hundreds of pages of documents, requiring precise allocation of GPU memory, and making token-level scheduling and key-value cache handling extremely complex.
One of the key technological breakthroughs of vLLM is "Page Attention," a design that helps the system manage memory more effectively and handle diverse requests and long sequences of output.
More Than Just Programming: A pivotal moment in transitioning from campus to the open-source community
In 2023, the vLLM team held its first open-source meetup in Silicon Valley. They originally thought only a dozen people would attend, but the number of registrants far exceeded expectations, and the venue was packed, becoming a turning point in the community's development.
Since then, the vLLM community has grown rapidly, now boasting over 50 regular contributors and more than 2,000 GitHub contributors, making it one of the fastest-growing open-source projects today, and receiving support from Meta, Red Hat, NVIDIA, AMD, AWS, Google, and many other parties.
Multiple forces compete on the same stage: creating an "AI-powered operating system".
One of the keys to vLLM's success is that it has created a common platform for model developers, chip manufacturers, and application developers. They do not need to interface with each other; they only need to interface with vLLM to achieve maximum compatibility between the model and the hardware.
This also means that vLLM is trying to create an "AI operating system": allowing all models and all hardware to run on the same general inference engine.
Inferences are becoming increasingly difficult? The triple pressures of scale, hardware, and agent intelligence.
The challenges of reasoning are constantly escalating, including:
The scale of models has exploded: from the initial tens of billions of parameters to today's trillion-level models, such as Kim K2, the computing resources required for inference have also increased accordingly.
Model and hardware diversity: Although the Transformer architecture is consistent, the internal details are becoming increasingly divergent, with various variants such as sparse attention and linear attention emerging one after another.
The rise of agent systems: Models no longer just answer one round, but participate in continuous dialogues, call external tools, execute Python scripts, etc. The inference layer needs to maintain state for a long time and handle asynchronous input, further raising the technical threshold.
Real-world application: Case studies of large-scale vLLM deployment
vLLM is not just an academic toy; it has already been deployed on major platforms such as Amazon, LinkedIn, and Character AI. For example, Amazon's intelligent assistant "Rufus" is powered by vLLM, serving as the inference engine behind shopping searches.
There were even engineers who deployed a vLLM feature directly to hundreds of GPUs while it was still under development, demonstrating the high level of trust it enjoys within the community.
The Company Behind vLLM: Inferact's Role and Vision
To further advance vLLM, the core developers founded Inferact and secured investment from multiple sources. Unlike typical commercial companies, Inferact prioritizes open source. One of its founders, Simon Mo, stated, "Our company exists to make vLLM the global standard inference engine." Inferact's business model revolves around maintaining and expanding the vLLM ecosystem while providing enterprise-level deployment and support, creating a dual-track approach of commercial and open source development.
Inferact is actively recruiting engineers with experience in machine learning infrastructure, particularly those skilled in large-scale model inference, distributed systems, and hardware acceleration. This presents an opportunity for developers seeking technical challenges and deep system optimization to participate in the next generation of AI infrastructure.
The team aims to create an "abstraction layer" similar to an operating system or database, enabling AI models to run seamlessly across diverse hardware and application scenarios.
This article, "Building a Universal AI Inference Layer! How Does the vLLM Open Source Project Aim to Become a Global Inference Engine?", first appeared on ABMedia ABMedia .
