Author: Jiawei

Editor: Olivia, IOSG Ventures

This article is for industry learning and communication purposes only and does not constitute any investment advice.

• If "The Merge" progresses smoothly, sharding will become the main focus of Ethereum's development in 2023 and beyond, and since sharding was proposed in 2015, its implications have changed significantly.

• After Vitalik proposed the "Rollup-Centric Ethereum Roadmap" and Ethereum's "Endgame," there has been a de facto shift in Ethereum's direction—"taking a back seat," serving as a security guarantee and data availability layer for Rollups.

• Danksharding and Proto-Danksharding are a series of technical combinations, manifested in a set of combinations that "identify problems" and introduce or propose new technologies to "solve problems."

• Looking ahead to the next few years, the overall value of Rollups will increase: a multi-Rollup development pattern on Ethereum, highly developed cross-Rollup infrastructure, and a thriving Rollup ecosystem—potentially even surpassing Ethereum itself.

  

Introduction

Image source: https://consensys.net/blog/blockchain-explained/the-roadmap-to-serenity-2/

As we reach the halfway point of 2022, we can look back at the Serenity Roadmap proposed by Vitalik in his 2018 Devcon speech and easily see that Ethereum's development path has undergone several changes—compared to the current roadmap, sharding has been given new meanings, and eWASM is rarely mentioned.

To avoid potential fraud and user misguidance, at the end of January this year, the Ethereum Foundation announced the abandonment of the term "ETH2," instead referring to the current Ethereum mainnet as the "execution layer" for processing transactions and execution, while the original ETH2 terminology was changed to the "consensus layer" for coordinating and processing PoS.

Currently, Ethereum's official roadmap covers three parts: the Beacon Chain, the Merge, and sharding.

The Beacon Chain serves as the preparatory work for Ethereum's migration to PoS and as the coordinating network for the consensus layer, having been activated on December 1, 2020, and has been running for nearly 20 months.

The Merge refers to the final merging of the current Ethereum mainnet with the Beacon Chain, unifying the execution layer and the consensus layer, marking Ethereum's official migration to PoS. In IOSG's article "Dawn Approaches: The Ethereum Merge is Near," we introduced important developments related to the Merge: the current Ethereum Ropsten and Sepolia testnets have successfully completed the Merge, followed by the Goerli Merge; if all goes well, it means we are not far from the mainnet Merge.

Image source: https://medium.com/decipher-media/blockchain-scaling-solutions-4-1-ethereum-sharding-e88e8cacdfc

In this article, we will focus on sharding. The reasons are:

First, assuming the mainnet Merge can be successfully achieved within the year, then sharding will follow closely as the main focus of Ethereum's development in 2023.

Second, the concept of Ethereum sharding was first proposed by Vitalik at Devcon 1 in 2015, and subsequently, the Sharding FAQ on GitHub outlined six development stages of sharding (as shown in the above image). However, with the updates to Ethereum's roadmap and the promotion of related EIPs, the meaning and priority of sharding have changed significantly. When discussing sharding, we need to ensure a common understanding of its implications.

Given these two points, it is important to clarify the origins and developments of sharding. This article will focus on the origins, progress, and future roadmap of Ethereum's original sharding, Danksharding, and Proto-Danksharding, rather than delving into every technical detail. For detailed content on Danksharding and Proto-Danksharding, please refer to IOSG's previous articles: “Will the Scaling Killer Danksharding be the Future of Ethereum Sharding?”, “EIP4844: The Upcoming Launch of the Predictable Dip Effect on L2 Transaction Fees”.

Quick Review

This article will frequently mention Rollups, data availability, and sharding.

Let’s quickly review the basic concepts of the three.

Currently, mainstream Rollups are divided into zkRollup and Optimistic Rollup. The former is based on validity proofs, meaning transactions are executed in batches, relying on cryptographic proofs (SNARKs) to ensure the correctness of state transitions; the latter "optimistically" assumes that all state transitions are correct unless proven otherwise; it requires a time window to ensure that erroneous state transitions can be detected.

Data availability is crucial for both zkRollup and Optimistic Rollup. For the former, users can reconstruct all Layer 2 transactions based on data availability to ensure censorship resistance; for the latter, all Layer 2 data must be published, with no transactions hidden. The current bottlenecks in data availability and corresponding solutions will be mentioned later.

Image source: https://www.web3.university/article/ethereum-sharding-an-introduction-to-blockchain-sharding

Ethereum full nodes store the complete state of the EVM and participate in all transaction validations, ensuring decentralization and security, but this comes with scalability issues: transactions are executed linearly, and each node must confirm each transaction individually, which is undoubtedly inefficient.

Moreover, as time goes on, the data on the Ethereum network continues to accumulate (currently reaching 786GB), leading to increasing hardware requirements for running full nodes. A decrease in the number of full nodes could lead to potential single points of failure and weaken the degree of decentralization.

Intuitively, sharding is akin to division of labor, grouping all nodes so that each transaction only needs to be validated by a single group of nodes, which periodically submits transaction records to the main chain, thus achieving parallel processing of transactions (for example, if there are 1000 nodes, originally each transaction must be validated by every node; if they are divided into 10 groups of 100 nodes each to validate transactions, the efficiency is clearly greatly improved). Adopting sharding allows for increased scalability while also reducing the hardware requirements for individual groups of nodes, thereby addressing the two aforementioned issues.

Original Sharding

In the original Ethereum plan, there are 64 shards, each with its own independent proposer and committee. The proposer is a randomly selected validator who collects and orders transactions; the committee is a group of validators (composed of at least 128 validators) that is randomly assigned to various shards at regular intervals to validate the validity of transactions. If 2/3 of the committee votes in favor, the validator management contract (VMC) submits the transaction record to the Beacon Chain. Unlike the "data sharding" mentioned below, this type of sharding is also referred to as "execution sharding."

Background

Before discussing Danksharding, let’s take some time to understand its background. Personally, I speculate that the community atmosphere leading to the introduction of Danksharding mainly stems from two articles by Vitalik. These two articles set the tone for Ethereum's future development direction.

First, in October 2020, Vitalik published the "Rollup-Centric Ethereum Roadmap," proposing that Ethereum needs to provide concentrated support for Rollups in the medium to short term. Firstly, Ethereum's base layer scaling will focus on expanding the data capacity of blocks rather than improving the efficiency of on-chain computation or IO operations. That is, Ethereum sharding aims to provide more space for data blobs (rather than transactions), and Ethereum does not need to interpret this data, only ensuring its availability. Secondly, Ethereum's infrastructure will be adjusted to support Rollups (such as L2 support for ENS, L2 integration for wallets, and cross-L2 asset transfers). In the long run, Ethereum's future should be a single execution shard with high security that everyone can handle, along with a scalable data availability layer.

Image source: https://vitalik.ca/general/2021/12/06/endgame.html

Subsequently, in December 2021, Vitalik described Ethereum's ultimate vision in "Endgame": block production is centralized, but block validation is achieved in a trustless and highly decentralized manner while ensuring censorship resistance. The underlying chain guarantees the data availability of blocks, while Rollups guarantee the validity of blocks (in zkRollup, this is achieved through SNARKs; in Optimistic Rollup, it only requires one honest participant to run a fraud-proof node).

Similar to the multi-chain ecosystem of Cosmos, Ethereum's future will be one of multiple Rollups coexisting—each relying on Ethereum for data availability and shared security. Users rely on bridges to operate across different Rollups without incurring high fees on the main chain.

The above two articles essentially determine Ethereum's development direction: optimizing the construction of Ethereum's base layer to serve Rollups. This argument may be based on the view that since Rollups have been validated as effective and have gained good adoption, "instead of spending years waiting for an uncertain and complex scaling solution (referring to original sharding), it is better to focus on Rollup-based solutions."

After this, Dankrad proposed the new sharding scheme, Danksharding. Below, we will break down the specific technical components of Danksharding for understanding.

Proto-Danksharding

Image source: https://l2fees.info/

The background for Proto-Danksharding is that while Rollup solutions significantly reduce transaction fees compared to the Ethereum main chain, they have not yet reached an ideal low level. This is due to the fact that the CALLDATA providing data availability on the Ethereum main chain still incurs significant costs (16 gas/byte). In the original proposal, Ethereum suggested providing each block with 16MB of dedicated data space for Rollups in data sharding, but the actual implementation of data sharding is still a long way off.

On February 25 of this year, Vitalik and DankRad proposed the EIP-4844 (Shard Blob Transactions) proposal, also known as Proto-Danksharding, aimed at expanding Ethereum's data availability in a simple and forward-compatible manner, ensuring its usability after Danksharding is launched. The changes proposed only occur at the consensus layer, requiring no additional adaptation work from execution layer clients, users, and Rollup developers.

Proto-Danksharding does not actually implement sharding but introduces a transaction format called "Blob-carrying Transactions" for future sharding. This transaction format differs from ordinary transactions in that it carries an additional data block called a blob (approximately 125kB), effectively enlarging the block and providing cheaper data availability compared to CALLDATA (approximately 10kB).

However, the general issue with "large blocks" is the increasing demand for disk space, with Proto-Danksharding potentially adding an extra 2.5TB of storage to Ethereum each year (currently, the total network data is only 986GB). Therefore, Proto-Danksharding sets a time window (e.g., 30 days) after which blob data will be deleted, allowing users or protocols to back up blob data during this period.

In this way, Ethereum's consensus layer acts merely as a highly secure "real-time bulletin board," ensuring that this data is available for a sufficiently long time and allowing other users or protocols enough time to back up the data, rather than having Ethereum permanently retain all historical blob data.

The reason for this approach is that while 2.5TB added annually is manageable for storage, it poses a significant burden on Ethereum nodes. As for the potential trust assumption issue, it essentially requires only one data storage party to be honest (1 of N), and the system can function normally without needing the real-time participation of a validating node set (N/2 of N) to store this historical data.

So, is there an incentive to encourage third parties to store this data? I have not yet discovered any incentive schemes, but Vitalik himself proposed several potential data storage parties:

1. Protocols related to applications (e.g., Rollups). They can require nodes to store historical data related to the application, and if historical data is lost, it poses risks to that application, thus providing them with motivation to store it;

2. BitTorrent;

3. Ethereum's Portal Network, a platform providing lightweight access to protocols;

4. Blockchain explorers, API providers, or other data service providers;

5. Individual enthusiasts or scholars engaged in data analysis;

6. Third-party indexing protocols like The Graph.

Danksharding Data Availability Sampling (DAS)

Image source: https://notes.ethereum.org/@hww/workshopfeb2022

In Proto-Danksharding, we mentioned that the new transaction format effectively enlarges the block, and Rollups accumulate a large amount of data, requiring nodes to download this data to ensure data availability.

The idea behind DAS is: if data can be divided into N blocks, and each node randomly downloads K blocks, it can verify whether all data is available without downloading all data, significantly reducing the burden on nodes. But what if a data block is lost? Simply downloading K blocks randomly makes it difficult to detect if a block is missing.

To implement DAS, erasure coding technology is introduced. Erasure coding is a fault-tolerant encoding technique where data is segmented, certain checks are added, and relationships are established between the various data segments, allowing complete data reconstruction through algorithms even if some data segments are lost.

If the redundancy ratio of erasure coding is set to 50%, it means that only 50% of the block data needs to be available for anyone in the network to reconstruct all block data and broadcast it. If an attacker wants to deceive nodes, they must hide more than 50% of the blocks, but as long as multiple random samples are taken, this situation is highly unlikely to occur. (For example, if 30 random samples are taken from the blocks, the probability that all those blocks are hidden by the attacker is )

Since nodes do not download all data but rely on erasure coding to reconstruct data, it is essential to ensure that the erasure coding is correctly encoded; otherwise, incorrect erasure coding cannot reconstruct the data.

Thus, KZG polynomial commitments are further introduced, which are a simplified form that "represents" a polynomial, used to prove that the value of the polynomial at a specific position matches a specified value without including all the data of that polynomial. In Danksharding, KZG commitments are used to verify erasure coding.

If we could place all data in a single KZG commitment, it would certainly be convenient, but constructing this KZG commitment, or reconstructing this data once some data becomes unavailable—both require significant resources. (In fact, a single block's data requires multiple KZG commitments for assurance.) To further reduce the burden on nodes and avoid centralization, Danksharding proposes a two-dimensional KZG commitment framework.

Once we address the above issues, relying on DAS, nodes or light clients only need to randomly download K data blocks to verify that all data is available; thus, even after introducing "large blocks," the burden on nodes will not increase significantly.

(Note: Specifically, the erasure coding algorithm used in Danksharding is Reed-Solomon coding; KZG commitments are a polynomial commitment scheme published by Kate, Zaverucha, and Goldberg. This will not be elaborated here; readers interested in the algorithm principles can explore further. Additionally, a solution to ensure the correctness of erasure coding is the fraud-proof used in Celestia.)

Separation of Block Proposers and Builders (PBS)

In the current situation, PoW miners and PoS validators are both block builders and block proposers—validators in PoS can use MEV profits to acquire more new validator seats, thus having more opportunities to realize MEV; moreover, large validator pools clearly have a stronger MEV capture capability than ordinary validators, leading to serious centralization issues. Therefore, PBS proposes to separate Builders and Proposers.

The idea behind PBS is as follows: Builders construct a sorted list of transactions and submit bids to Proposers. Proposers only need to accept the transaction list with the highest bid, and no one can know the specific content of the transaction list before the auction winner is selected.

This separation and auction mechanism introduces "involution" between the game and Builders: after all, each Builder's ability to capture MEV varies, and Builders need to weigh the potential MEV profits against the auction bids, which effectively reduces the net income from MEV; regardless of whether the block submitted by the Builder is successfully produced, they must pay the bidding fee to the Proposer. In this way, the Proposer (broadly speaking, all validators collectively, randomly reselected within a certain time) effectively shares a portion of the MEV income, reducing the degree of MEV centralization.

The above describes the advantages of PBS in addressing MEV, and there is another reason for introducing PBS. In Danksharding, the requirement for Builders is to compute a KZG proof for 32MB of data in about 1 second, which requires a 32-64 core CPU; and to broadcast 64MB of data in a P2P manner within a certain time frame, which requires 2.5Gbit/s bandwidth. Clearly, validators cannot meet such requirements.

Thus, PBS separates the two roles, with Proposers still acting as general validator nodes, responsible for selecting transaction lists and broadcasting block headers; while Builders take on a specialized role, responsible for the aforementioned tasks and constructing transaction lists.

Image source: https://ethresear.ch/t/two-slot-proposer-builder-separation/10980

In October last year, Vitalik proposed a dual-slot PBS scheme (note: each slot is 12 seconds, which is the time unit of the Beacon Chain), but the specific PBS scheme is still under discussion.

Censorship Resistance List (crList)

Image source: https://notes.ethereum.org/@hww/workshopfeb2022

However, PBS also brings a problem: if a Builder always bids the highest price (even at an economic loss) to win the auction, they effectively gain the ability to censor transactions by selectively excluding certain transactions from the block.

To address this, Danksharding further introduces the Censorship Resistance List (crList), which allows Proposers to specify a transaction list that must be included by Builders; after winning the auction, Builders must prove that all transactions in the crList have been included (or that the block is full), otherwise, the block will be deemed invalid.

Summary

Image source: https://notes.ethereum.org/@hww/workshopfeb2022

By combining the above data availability sampling (DAS), separation of block builders and proposers (PBS), and the censorship resistance list (crList), we arrive at a complete picture of Danksharding. We find that the concept of "sharding" has actually been downplayed; although the term Sharding is retained, the actual focus has shifted to supporting data availability.

So, what advantages does Danksharding have compared to original sharding?

(Dankrad himself lists 10 advantages of Danksharding here; we will select two for specific explanation.)

In original sharding, each individual shard has its own proposer and committee, which vote on the validation of transactions within the shard, and the proposer of the Beacon Chain collects all voting results. This task is difficult to complete within a single slot. In Danksharding, however, there is only a committee on the Beacon Chain (a broadly defined validator set, randomly reselected within a certain time), which verifies the Beacon Chain blocks and shard data. This simplifies the original 64 groups of proposers and committees into one group, greatly reducing both theoretical and engineering implementation complexity.

Another advantage of Danksharding is that synchronous calls between the Ethereum main chain and zkRollups may be realized. As mentioned earlier, in original sharding, the Beacon Chain needs to collect all shard voting results, which introduces confirmation delays. In Danksharding, the blocks and shard data of the Beacon Chain are uniformly authenticated by the Beacon Chain's committee, meaning that transactions in the same Beacon block can access shard data instantly. This opens up more imaginative possibilities for composability: for example, the distributed AMM (dAMM) proposed by StarkWare can conduct swaps or share liquidity across L1/L2, thus addressing the issue of liquidity fragmentation.

Once Danksharding is implemented, Ethereum will become a unified settlement layer and data availability layer for Rollups.

Closing Thoughts

In the above image, we summarize Danksharding.

In summary, we can see that the directional focus of Ethereum's roadmap is very clear over the next 2 to 3 years—centered around serving Rollups. Although it remains uncertain whether the roadmap will change during this process: Danksharding is expected to be realized within the next 18-24 months, while Proto-Danksharding is expected to be realized within 6-9 months. At the very least, we have clarified that Rollups, as the foundation for Ethereum's scalability, occupy a certain dominant position.

Based on Vitalik's outlook, we also propose some predictive thoughts and speculations:

• First, similar to the multi-chain ecosystem of Cosmos, there will be a competitive landscape of multiple Rollups on Ethereum in the future, with Ethereum providing security and data availability guarantees for them.

• Second, cross-L1/Rollup infrastructure will become a necessity. Cross-domain MEV will bring more complex arbitrage combinations, similar to the aforementioned dAMM providing richer composability.

• Third, the ecological applications of multiple Rollups will surpass Ethereum itself. As Ethereum's positioning becomes secondary, serving as the data availability layer for Rollups, we speculate that more applications will migrate to Rollups (if the second point holds); or at least applications will be developed simultaneously on both Ethereum and Rollups.

References:

https://consensys.net/blog/blockchain-explained/the-roadmap-to-serenity-2/

https://www.web3.university/article/ethereum-sharding-an-introduction-to-blockchain-sharding

https://ethereum-magicians.org/t/a-rollup-centric-ethereum-roadmap/4698

https://vitalik.ca/general/2021/12/06/endgame.html

https://notes.ethereum.org/@vbuterin/protodankshardingfaq

https://twitter.com/pseudotheos/status/1504457560396468231

https://ethos.dev/beacon-chain/

https://notes.ethereum.org/@vbuterin/pbscensorshipresistance#How-does-proposerbuilder-separation-PBS-work

https://notes.ethereum.org/@fradamt/H1ZqdtrBF

https://cloud.tencent.com/developer/article/1829995

https://medium.com/coinmonks/builder-proposer-separation-for-ethereum-explained-884c8f45f8dd

https://dankradfeist.de/ethereum/2021/10/13/kate-polynomial-commitments-mandarin.html

https://members.delphidigital.io/reports/the-hitchhikers-guide-to-ethereum

https://vitalik.ca/general/2019/09/22/plonk.html