A step-by-step roadmap for scaling rollups with calldata expansion and sharding

Author: Vitalik Buterin, co-founder of Ethereum

Compiled by: Nan Feng, Unitimes

For Ethereum, Rollups are the only trustless scalability solution in the short to medium term, and possibly long term. Transaction fees on Ethereum L1 have been high for several months now, making it more urgent to take all necessary actions to help drive the entire ecosystem towards Rollups. Rollups have significantly reduced costs for many Ethereum users: the l2fees.info website often shows that the fees on the Optimism and Arbitrum networks are about 3-8 times lower than those on the Ethereum base layer itself, while zk-Rollups have better data compression and can avoid including signatures, resulting in fees that are approximately 40-100 times lower than those on the Ethereum base layer.

However, for many users, even these (Rollup) fees are still too expensive. For a long time, data sharding has been considered a long-term solution to the current shortcomings of Rollups, with the potential to add about 1-2MB/s of dedicated data space for Rollups on the Ethereum chain. This article outlines a practical path to implement this solution, unlocking data space for Rollups as quickly as possible while increasing additional space and security over time.

Step 1: Expand transaction calldata

Currently, existing Rollups utilize transaction calldata. Therefore, if we want to increase the capacity of Rollups and reduce costs in the short term without requiring any extra work from individual Rollup teams, we should lower the Gas costs for transaction calldata. The current average block size is still far from threatening the stability of the Ethereum network, so this is likely to be safe, although it may require some additional logic to prevent very unsafe edge cases.

See the EIP-4488 proposal, or another (simpler but milder) EIP-4490 proposal.

• EIP-4488:

https://github.com/ethereum/EIPs/pull/4488

• EIP-4490:

https://github.com/ethereum/EIPs/pull/4490

EIP 4488 should theoretically increase the data space available for Rollups per slot to about 1 MB and reduce costs on Rollups by about 5 times. This can be implemented faster than the subsequent steps.

Step 2: A few shards

Meanwhile, we can start working on launching "proper" sharding. Achieving sharding in its full (functional) form will take a long time, but what we can do is implement it step by step and benefit from each step. The first step is naturally to implement the "business logic" of the sharding specification, but we need to keep the number of initially launched shards very low (for example, 4 shards) to avoid most of the difficulties surrounding the sharded network. Each shard will broadcast in its own subnet. By default, validators will trust the committee, but if they wish, they can choose to verify in each subnet, but only if they have seen the full data of any shard block confirmed by the beacon block.

The sharding specification itself is not particularly difficult; it involves a code change of a similar scale to the recently released Altair hard fork (the Altair beacon change specification document is 728 lines, while the sharding beacon change specification document is 888 lines), so it can reasonably be expected to be implemented within a similar timeframe to that of Altair's implementation and deployment.

To make sharded data actually usable by Rollups, Rollups will need to be able to make their proofs enter into the sharded data. There are two options:

1. Add a BEACONBLOCKROOT opcode; Rollups will add code to verify Merkle proofs rooted in historical beacon chain block roots;

2. Add future state and historical access precompiles, so that when the commitment scheme changes in the future, Rollups do not need to change their code.

This will increase the Rollup data space per slot to about 2 MB (250 kB per shard * 4 shards, plus the expanded calldata from Step 1 above).

Step 3: N shards, protected by a committee

Increase the number of active shards from 4 to 64. At this point, sharded data will enter the subnetwork, so the P2P layer at that time must be sufficiently robust to make it feasible to split it into a larger number of subnetworks. The security of data availability will be based on the honest majority (validators) assumption, relying on the security of the committee.

This will increase the Rollup data space per slot to about 16 MB (250 kB per shard * 64 shards); we assume that by this time Rollups will have migrated out of the Ethereum execution chain.

Step 4: Data availability sampling (DAS)

Add data availability sampling (DAS) to ensure a higher level of security, so that even in cases of attacks where most (validators) are dishonest, users can still be protected. Data availability sampling can be implemented in phases: first, allow the network to test it in a non-binding manner, then make it a prerequisite for receiving beacon blocks, and possibly even test it on certain clients first.

Once data availability sampling is fully introduced, the rollout of sharding is complete.

Sharded Optimistic Rollups and ZK Rollups

A major difference between the current Ethereum and Ethereum after implementing sharding is that, in a sharded world, Rollup data cannot actually become part of the transactions submitting Rollup blocks to smart contracts. Instead, the publication of Rollup data and the submission of Rollup blocks must be separate: first, data publication will put the data on-chain (i.e., into the shard chain), and then block submission will submit the block header along with a proof pointing to the underlying data.

Optimism and Arbitrum have already used a two-step design for submitting Rollup blocks, so this will be a minor code change for both.

For ZK Rollups, things are a bit tricky because submitting a transaction requires providing a proof that directly operates on the data. They can prove that the data in the shard matches the commitment on the beacon chain through ZK-SNARK, but this operation is very expensive. Fortunately, there are cheaper alternatives.

If the ZK-SNARK is a PLONK proof based on BLS12-381, then they can simply package the shard data commitment as input. The BLS12-381 shard data commitment is a KZG commitment, the same type of commitment used in PLONK, and can be passed directly as a public input to the proof.

If the ZK-SNARK uses some different mechanism (or even BLS12-381 PLONK but with a larger trusted setup), then it can include its own data commitment and use a proof of equivalence to verify that the commitments in the proof correspond to the same data as those in the beacon chain.

In a sharded world, who will store historical data?

A necessary condition for increasing data space is to remove the property of the Ethereum core protocol that is responsible for permanently maintaining all data that has reached consensus. Because the volume of this data is too large. For example:

1. The maximum chain size theoretically brought by EIP-4488 is about 1,262,861 bytes per 12-second slot, or about 3.0 TB per year, but in reality, it is more likely to be about 250-1000 GB per year, especially in the early stages.

2. 4 shards (1 MB per slot) will add approximately 2.5 TB per year.

3. 64 shards (16 MB per slot) will bring a total of about 40 TB of storage per year.

Most users' hard drives range from 256 GB to 2 TB, with 1 TB seeming to be the median. The following image shows the results of an internal survey conducted among a group of blockchain researchers regarding how large computer hard drives are:

This means that users can currently run a node, but if any part of this roadmap is implemented without modification, users will not be able to run a node. Of course, larger drives are available, but users would have to go to great lengths to purchase them, significantly increasing the complexity of running a node. The current main solution is EIP-4444, which eliminates the responsibility of node operators to store blocks or receipts older than 1 year. In the case of sharding, this 1-year duration will likely be further shortened, and nodes will only need to be responsible for the shards on the subnetwork they actively participate in.

This raises the question: if the Ethereum core protocol does not store this data, who will store it?

First, it is important to remember that even with sharding, the volume of data will not be that large. Yes, 40 TB per year does exceed the capacity of an individual running "default" consumer hardware (in fact, even 1 TB per year does). However, for someone willing to invest some resources and find ways to store this data, it is within their acceptable range. Currently, a 48 TB HDD (hard disk drive) costs $1,729, while a 14 TB one is about $420. For someone running 1 validator slot with 32 ETH, they might be willing to pay and store the entire chain after the implementation of sharding for the staking rewards. Therefore, the situation of "no one will store some historical data of a shard to the extent that this data is completely lost" seems unlikely.

So who will store this data? Here are some of my thoughts:

• Individual and institutional volunteers;
• Block explorers (etherchain.org, etherscan.io, amberdata.io, etc.) will definitely store all data, as providing data to users is their business model.
• Rollup DAOs designate and pay participants to store and provide historical data related to their Rollup.
• Historical data can be uploaded and shared via torrents.
• Clients can voluntarily choose to randomly store 0.05% of the historical data of the blockchain (using erasure coding so that only a small amount of data will be lost if many clients go offline at the same time).
• Clients in the Portal Network can randomly store a portion of the blockchain's historical data, and the Portal Network will automatically direct data requests to nodes that store that data.
• The protocol can incentivize the storage of historical data.
• Protocols like The Graph can create incentive markets where clients pay servers for historical data and Merkle proofs that prove its correctness. This incentivizes individuals and institutions to run servers that store historical data and provide this data on demand.

Some of these solutions (individual and institutional volunteers, block explorers) are already available. The current P2P torrent scene is a great example of an ecosystem primarily driven by volunteers storing a large amount of content. Other protocol-based solutions are more powerful because they provide incentive mechanisms, but they may take longer to develop. In the long run, accessing historical data through these L2 protocols may be more efficient than through the current Ethereum protocol.