What exactly is Polkadot’s future upgrade direction “JAM”?

To support the development of JAM and ensure that it is built in a true decentralized spirit, Gavin announced the establishment of the JAM bonus in conjunction with the Web3 Foundation during his speech. A total of 10 million DOTs will be used to incentivize the development of other implementations of JAM .

In conjunction with his speech, Gavin also released a technical graypaper. If you want to dive deeper into the vision and technical details of the project, you can find the paper on the new JAM Graypaper website: https://graypaper.com/.

Gavin and Polkadot have collectively led the creation of innovative technologies to realize the vision of creating a free and open web. JAM is the next chapter in the evolving Polkadot story.

Here’s the latest introduction to JAM from the Polkadot Wiki, written by Gavin Wood.

JAM stands for Join-Accumulate Machine, a new design that is planned to replace the existing relay chain. The name JAM is derived from CoreJAM, which stands for Collect Refine Join Accumulate, describing the computational model embodied by the machine, which was originally described by Gavin Wood in an RFC. However, in the actual chain, only Join and Accumulate are executed, while the Collect and Refine processes occur off-chain.

Unlike the current iterative approach, JAM will be introduced as a comprehensive single upgrade. There are several reasons for this including:

• Uniform upgrades can precisely constrain the operations after the upgrade, which is difficult to do in iterative methods.

• It reduces the number of small upgrades and major changes that often occur regularly over a period of weeks or months.

While this transition requires significant breaking changes, we will work to minimize its impact to manageable levels. It is a better option to combine multiple smaller breaking changes into a single transition that introduces a new blockchain concept that integrates various existing ideas.

JAM will be a domain-specific chain that is designed to handle domain-specific problems. In this case, it's rollups, which the Ethereum community calls optimistic roll-ups. JAM's rollups are very limited in terms of security. This is what Polkadot has been doing for the last five years, and it's already a highly domain-specific rollup chain. JAM essentially makes it less prescriptive and more general.

The JAM chain accepts rollup outputs, and more generally bits of computation done elsewhere, and incorporates the outputs into the shared state, similar to how the Polkadot relay chain functions.

The JAM chain's job is to provide the necessary equipment to ensure that the output correctly reflects the input after it has been transformed.

JAM has several similarities to smart contract chains:

• The JAM Chain itself directly executes permissionless code.

• The state of a JAM chain is organized into different packages.

• In addition to the encapsulation of the state, it also includes the encapsulation of the code and the balance.

These encapsulations of state are called services. Therefore, the state of JAM is divided into services. Creating new services is permissionless, similar to deploying smart contracts on a smart contract chain. Therefore, adding new services to the JAM chain does not require any authoritative approval or compliance with governance mechanisms, unlike Substrate-based chains, which require governance approval to add new pallets. Services include code, balances, and certain state components, similar to the structure commonly seen on smart contract chains.

The code for the JAM service is divided into three different entry points:

• Refine is the function that performs the most stateless computation. It defines the transformations for a rollup of a specific service.

• The Accumulate function takes the output and folds it into the overall state of the service

• OnTransfer processes information from other services.

Work packages are the inputs to a service. A work package can contain many work items. Each work item is associated with a service, and it reflects the actual input to that service. For parachain services, this is where transactions and all blockchain inputs come in.

The JAM safety device includes a two-stage process, where the Refine function accepts work items as input and produces work results as output, and then enters the Accumulate function (Refine first, then Accumulate). Work items are refined into work results, so a work package containing many work items is refined into a work report, which is the result corresponding to multiple work items. A work package can be assigned to use a core within a specific period of time (usually 6 seconds).

JAM is distinguished from smart contract chains by its transactionless operation. There are no transactions in JAM; all actions are permissionless and initially go through the Refine phase. In this phase, the service pre-refines the input data and converts it into a work report containing the work results. These work results are then transmitted to the chain.

Even though there are no transactions, JAM still accepts external messages in a specific format. There are five types of external messages:

1. Guarantees

2. Assurances

3. Judgments

4. Preimages

5. Tickets

The first three types are part of the JAM Chain security framework. "Guarantees" and "Assurances" involve the validators collectively proving that a work result, after being transformed by the service's "Refine" function, accurately reflects the result of its corresponding work item.

Judgment occurs when the integrity of a work result is in question, at which point a large number of validators attest to its validity or lack thereof. In this case, invalid work items may have been integrated into the state of the service, and a rollback may be necessary. Judgment must occur within one hour of submitting a work report to the chain, during which time block finality is temporarily suspended.

Preimages are a feature that JAM chains provide to Refine functions. While Refine functions are usually stateless, they can perform one stateful operation: finding the preimage of a hash. This feature is the only aspect of Refine functions that has a predefined preference.

Tickets enter the block production mechanism as anonymous entries. They are not a direct requirement for block production; instead, the system operates two epochs in advance. This mechanism is part of the SAFROL algorithm, a refined version of the original SASSAFRAS algorithm.

In JAM, the Refine processing phase can accept up to 15 MB of data per time period, with each time period lasting 6 seconds. However, the maximum amount of data produced by Refine is 90 kB, and due to the distributed nature of the availability system, a lot of data compression is required. For example, in the context of a parachain, 15 MB of data represents a proof of validity (PoV), while 90 kB of data corresponds to a candidate receipt.

Refine can use up to 6 seconds of PVM Gas, equivalent to a full block period of the relay chain. This extended execution time, compared to PVF’s current two-second limit, is achieved through security metering and other optimizations.

Refine can also perform preimage lookups. If a hash and its associated preimage are believed to be available on the JAM chain, the preimage can be requested by providing the hash. This capability enables efficient storage and retrieval of code, for example by storing parachain code on the JAM chain and referencing its hash in work packages.

Refine is the main processing workhorse, handling tasks that are mostly stateless operations.

The Accumulate function is responsible for integrating the output generated by the Refine function into the chain state. Accumulate can accept multiple outputs from Refine, all of which originate from the same service. Both Refine and Accumulate serve as entry points from a specific service code block.

The execution time per output of Accumulate is much shorter than that of Refine, typically only 10 milliseconds at most. However, the duration depends on the number of work items in the work package. If the work package contains multiple items, the available time is divided among them.

Unlike Refine, Accumulate is stateful and has access to the state of the JAM chain. It can read storage from any service, write to its key-value store, transfer funds, and include memos when funds are transferred. In addition, Accumulate can create new services, upgrade their code, and request the availability of preimages, etc.

Additionally, Refine can call child instances of PVM. This allows the creation of child instances or virtual machines where code and data can be deployed, memory and stack configurations customized, and computations performed within a flexible framework.

The onTransfer function in the JAM system is also stateful, enabling it to modify the state of a service. It has the ability to check the state of other services and modify its own state. This functionality facilitates communication between services, albeit in an asynchronous manner.

Unlike smart contract platforms where many interactions occur synchronously, in JAM, interactions between encapsulated components (such as smart contracts or services in this case) occur asynchronously. Messages and tokens are sent together, and at some point during the same six-second execution cycle, the receiving service processes them. There is no immediate return path; if a return path is required, the sending service must initiate another transfer or modify its state in a way that the receiving service can later interpret.

Both Accumulate and onTransfer are designed to be executed in parallel, allowing accumulates and transfers for different services to occur simultaneously. This design opens the possibility for future enhancements, such as allocating gas inputs beyond the current 10 milliseconds. In theory, a secondary core could be used to execute certain accumulates, giving them more gas to take advantage of.

As stated in the original Polkadot whitepaper, Polkadot is primarily tailored for a specific service profile: providing parachains. To enable this service, Polkadot has developed two important subcomponents:

• Distributed Data Availability System

• A system that provides audits and guarantees for computation (i.e. an optimistic rollup system with strong security guarantees)

Compared to Polkadot, JAM has fewer preset preferences and provides a higher level of abstraction and generalization. This helps to more easily utilize the underlying components according to personal preferences.

JAM operates in a permissionless manner, similar to a smart contract chain, allowing individuals to upload and anticipate the execution of code. In addition, it hosts data, enables preimage lookups, and manages state, similar to a key-value system. Since JAM lacks a mechanism to directly accept transactions, JAM's genesis block includes a service to facilitate the creation of new services.

Services within JAM have no predefined limits on the amount of code, data, or state. Their capacity is determined by cryptoeconomic factors; the more DOT tokens deposited, the greater the capacity for data and state. For example, parachain services on JAM combine all Polkadot 1.1 functionality into a single service, and other services can also take advantage of Polkadot's distributed availability system, as well as its auditing and assurance systems for computation.

The design of PVM is based on the RISC-V instruction set architecture (ISA), which is known for its simplicity and versatility. The RISC-V ISA provides several advantages:

1. Easy to translate into common hardware formats such as x86, x64 and ARM.

2. Well supported by tools like LLVM.

The PVM embodies simplicity and security, with the ability to be sandboxed and provide various execution guarantees. It is deterministic, consensus-sensitive, and easy to meter. Compared to other virtual machines, the PVM lacks complexity and excessive preset preferences.

WASM, while optimized for web use cases, also brings challenges to stack management, especially in terms of processing contiguity. RISC-V solves this problem by placing the stack in memory, naturally facilitating contiguity without adding complexity.

Furthermore, PVM demonstrates superior execution speed when running on traditional hardware, especially on X64 and ARM, providing advantages such as free metering, which is advantageous over WASM.

The continuity of support for RISC-V will establish a new standard for scalable coding across multi-core platforms such as JAM. Asynchronous parallel architectures are increasingly important for the scalability of hardware and software platforms, a trend that is expected to extend to blockchains and consensus algorithms.

SAFROLE is a block production algorithm that simplifies SASSAFRAS. It excludes some components that may be useful for parachains. So parachains may stick to SASSAFRAS instead of SAFROLE. SAFROLE will be as simple as possible:

• Make sure to minimize preset preferences to maximize potential future use cases.

• Following in the footsteps of the Ethereum Yellow Paper and really trying to get as many implementations out there as possible to try and spread the expertise.

Understanding how Polkadot 1.0 works end-to-end is challenging. With JAM, anyone who can read and understand the yellow paper should be able to read and understand how JAM works pretty quickly. So simplicity is critical.

SAFROLE is a SNARK-based block production algorithm. It uses SNARKs because of their anonymity properties. And it provides almost completely forkless constant-time block production. The few instances where forks can occur basically only happen if there is a network split or someone deliberately acts maliciously. The great value of anonymity is not to keep the identity of the validators secret; in fact, when they produce a block, they will reveal their identity anyway, so this is actually to ensure the security of the block production mechanism itself, basically to avoid spam attacks.

JAM's network uses the QUIC protocol. This allows direct peer-to-peer connections between a large number of validators. As a result, more than 1,000 validators on Polkadot can maintain persistent connections with each other without having to worry about possible socket issues. Since JAM does not handle transactions, gossip is basically not required. In cases where distribution is required that is not peer-to-peer or among a very small subset of validators, mesh diffusion is used, where validators are arranged in a grid, packets are sent in rows, and then each node sends it to all members of its column.

In state-based blockchains like Ethereum, the header of a block usually contains a post-state root that summarizes the state after all block computations. Therefore, the block header cannot be sent until all computations are complete. However, some computations can be performed before sending the block header, as their results determine the validity of the block.

However, JAM takes a different approach by placing the pre-state root in the block header instead of the post-state root. This means that the state root shown in the header is delayed by one block. Therefore, lightweight computations of about 5% of the block work or execution time can be performed and the block can be dispatched immediately. The remaining 95% of the block computation, mainly the Accumulate task, can be completed later. This allows the next block to be started before the current block is executed.

This approach allows for more efficient use of the time between blocks. In traditional settings, such as Polkadot’s six-second block time, the post-state root must be provided and can only be computed in part of that time. However, through pipelining in JAM, the entire block time can be effectively utilized for computation, maximizing efficiency.

While using the entire block time for computation may not be ideal as it may result in perpetual catch-up and delayed block imports, JAM’s approach can significantly extend computation time compared to traditional setups. This means that effective block computation times of around three to three and a half seconds can be achieved, which is a big improvement over current setups.

One architectural difference between JAM and the Relay Chain is the degree of fixed functionality. While the Relay Chain fixes some elements, such as the language used to define the protocol (WASM), JAM goes further in this regard. For example, it specifies the types used to encode block headers and hashing schemes, making it difficult to change these aspects.

However, JAM retains flexibility comparable to that achieved by the WebAssembly meta-protocol in the Relay Chain through its service model. In this model, the upgradeability responsibility is shifted to the services, freeing the chain itself from the burden of upgrading. Three main reasons supporting this decision include:

1. Prioritize simplicity. Maintaining a non-upgradeable chain significantly reduces complexity.

2. The relay chain tends to introduce complex features without removing old ones, which complicates things. Since upgrades are easy to implement, there is little incentive to simplify the Substrate SDK. Therefore, copying Polkadot becomes impractical.

3. The fixed parameters of the JAM provide optimization potential. By clearly understanding the specific tasks that the JAM chain must perform and being able to set fixed parameters, optimizations in areas such as network topology and timing become feasible. This is in contrast to the highly upgradeable nature of the relay chain, where these optimizations are more complex due to the frequent changes that may be made with each upgrade.

Despite these differences, JAM maintains flexibility in application-level functionality, such as core time sales, staking, and governance, all of which are managed within the service. Additionally, JAM introduces a new concept by associating token balances with services, providing opportunities for economic model adjustments that are not easily possible in purely upgradeable chains such as the Relay Chain.

To ensure that JAM lives up to its original expectations, an ongoing effort includes building a comprehensive testing environment for the JAM Chain. Unlike running a small-scale test network on unreliable hardware to manage cloud computing costs, this initiative involves significant investment.

This is where JAM Toaster comes in, essentially a testbed for large-scale experimentation and performance evaluation. This addresses challenges previously encountered during the development of the Polkadot relay chain, when we learned that the emergent effects and dynamics of operating a network at such scale proved difficult. Previous attempts were limited to a few dozen nodes on the test network and the Kusama network, which lacked comprehensive monitoring capabilities due to the lack of access to validating nodes. In contrast, small-scale test networks fail to accurately simulate the network dynamics of large-scale deployments.

JAM Toaster aims to bridge this gap by providing in-depth research on the entire JAM network, including 1023 nodes. The platform facilitates the study of network behavior and performance characteristics, providing developers with valuable insights into the expected performance of parachains.

In JAM, benchmarking or performance testing can be optional. While benchmarking may still be required in some cases, JAM's metering system can generally avoid the need for frequent benchmarking. JAM runs on a metering system that allows users to evaluate computational work after completion. In addition, there is a theoretical mechanism to control computations as blocks are built.

However, benchmarking is still necessary in some cases. For example, benchmarking may be necessary to improve performance when metering is too conservative for certain use cases. Also, benchmarking is useful for tasks that require extended execution time to ensure that they do not run too long.

Next up is Cross-Chain Messaging (XCMP), for which JAM requires full XCMP support. This is because in the relay chain, if all parachains always transmit all data, more data can be passed through candidate receipts. JAM strictly adheres to the rules, even for parachain services, including restrictions on data transfer between the “Refine” and “Accumulate” phases. Currently, with Horizontal Relay Chain Messaging (HRMP), all messages traverse the relay chain, limiting data payloads to 4 kB or less, which may not be realistic. Therefore, XCMP only relays message headers through the chain, while the actual message data is transferred off-chain, which is a necessary and long-overdue improvement.

Accords essentially encapsulate state and logic, similar to smart contracts, with multiple instances per parachain. They facilitate message exchange between instances and enable synchronous interaction with parachains. The protocol is useful in scenarios where there is a lack of trust between parachains, such as token transfers. Unlike existing approaches that involve reserve intermediaries, Accords simplify direct token transfers between parachains, eliminating the need for trust-compromising intermediaries. Additionally, Accords can act as an XCM forwarding mechanism, ensuring message integrity even when relayed through a third-party intermediary, eliminating the need for explicit source marking.

Finally, JAM’s broader, less preconceived approach to leveraging the underlying consensus mechanism facilitates the implementation of more innovative solutions. For example, for complex tasks like zero-knowledge proofs, distributed availability becomes more practical and efficient. In addition, JAM supports a hybrid resource consumption model where work packages can contain both compute-intensive tasks and data-intensive operations. By pairing services that require heavy computation with services that require high data availability, JAM optimizes the utilization of validator resources, thereby reducing costs. This flexible approach makes it cheaper to combine workloads such as distributed availability and SNARK verification with parachain workloads while increasing efficiency.

JAM's design prioritizes compatibility with existing Polkadot 1 parachains. While it maintains compatibility with the Polkadot SDK, the Polkadot Validator Function (PVF) has been retargeted. It will target the Polkadot Virtual Machine (PVM) instead of WebAssembly. This transition is made possible by the fact that PVM is a slight modification of RISC-V, which has been formalized as a target for LLVM. Therefore, PVM will likely become an official LLVM target before JAM is deployed.

In addition to being a host for parachains, JAM introduces significant enhancements. It offers the potential to simplify benchmarking efforts and alleviate future benchmarking needs. In addition, JAM introduces the concept of protocol, multi-instance, multi-shard smart contracts for managing and enforcing specific interaction protocols between parachains. Furthermore, comprehensive cross-chain messaging (XCMP) support is critical, making it possible to remove limitations on information transfer between parachains, which is typically facilitated through cross-chain messages (XCM).

Regarding Agile Coretime, JAM maintains compatibility with existing setups. However, it introduces the ability to target coretime not only on parachains, but on arbitrary groups of work packages. This flexibility enhances the versatility and efficiency of resource allocation within the JAM ecosystem.