In-depth Interpretation of EIP-4844: A Small Step for Sharding, a Big Step for Ethereum Scalability

Written by: Yihan Xu, Foresight Research

In this report, you can learn about :

EIP-4844;

Proto-danksharding;

Danksharding;

KZG Commitment;

KZG Ceremony;

Dynamic state sharding;

…

Recently, the KZG Ceremony with excessive community participation has been a hot topic. What exactly is this ceremony doing?

In short, the KZG Ceremony is an essential initialization step for implementing EIP-4844, which is a preliminary version in the process of achieving Ethereum sharding.

I. Sharding: The Long-term Solution for Ethereum Scalability

• Sharding expands from the perspective of data availability, while rollup primarily expands from the execution layer, together alleviating the congestion issue on the mainnet; I believe sharding may be an attempt to break through the blockchain trilemma.
• The following chart of Ethereum block size trends illustrates the necessity of expanding at the data level from another angle. Since the genesis block, Ethereum has been rapidly iterating from the underlying architecture to upper-layer applications, yet the average block size remains around 90Kb, with no qualitative breakthrough at its peak. Although Rollup addresses Ethereum's congestion issue from the computation layer, its performance is still constrained by the data storage capacity of layer 1.

• Due to the complexity of the implementation process and security considerations, the Ethereum development team has divided sharding into multiple phases, including the recently mentioned proto-danksharding and danksharding. The entire process will be an update spanning several years;
• Under the current data storage model of Ethereum, only a few specific high-configuration machines can join the network as nodes. After full sharding, Ethereum will no longer require every node to store all data, thereby reducing the data storage costs on the mainnet while enhancing security (the threshold for becoming a node is lowered, further increasing decentralization and reducing the risk of attacks).

II. EIP-4844: Short-term High Returns, A Streamlined Version of Sharding

EIP-4844 = Proto-Danksharding; Proto comes from the name of Ethereum researchers.

Due to the high complexity of implementing danksharding (which will be analyzed in the next section), the development cycle is measured in years. Therefore, proto-danksharding serves as a scaling solution for Ethereum prior to implementing danksharding, primarily realizing the transaction format, precompile, and other designs within danksharding;

1. Overview of Proto-danksharding

Proto-Danksharding mainly introduces a new transaction type, namely blob-carrying transaction. Thus, Rollup's data is transmitted to layer 1 in the form of blobs at lower costs and stored non-permanently. Meanwhile, blobs are significantly larger than the current calldata, better supporting high TPS on rollups.

About blobs:

• Each transaction can carry up to 2 blobs;
• Ideally, each block contains 8 blobs, approximately 1MB, and can contain up to 16 blobs, approximately 2MB;
• Blobs do not need to be permanently stored as history logs like calldata;
• Compared to danksharding, nodes still need to verify the complete DA.

2. Interpretation of Blob-carrying Transactions

Attaching blobs (data units) to transactions

Function

Referring to the diagram provided by Vitalik in the proposal, data blobs are similar to current calldata, allowing rollups to upload transaction, proof, and other data to layer 1 via blobs to ensure data availability.

Cost

The design of data blobs aims to support rollup's high-throughput transactions. Compared to calldata of the same size (using on-chain storage), the cost of blobs will be significantly lower (no need for permanent storage). Therefore, the gas spent by rollups to maintain data availability will be significantly reduced compared to before.

Capacity

Each blob is approximately 125kB (fact: the current average block size is only ~90kB).

3. The Value and Challenges of Blob-carrying Transactions

Value

Blobs can be seen as a form of cache, where the transaction data submitted by rollups exists in a cached form. This reduces the requirements for storage hardware, provides additional data expansion for Ethereum, and lowers gas costs.

Challenges: Hardware Performance Requirements for Ethereum Nodes

The current average block size of Ethereum is only ~90kB, but a single blob is ~125kB.

According to the design of EIP-4844, each slot is normally about 1 MB, so the annual increase in data volume is:

1 MB/block * 5 block/min * 43200 min/month * 12 month/year = 2.47 TB per year

The annual increase in data size far exceeds the total data volume of Ethereum, making such a storage solution evidently inefficient.

Solutions

From the perspective of short-term scalability, since each node still needs to store the full historical data, while implementing EIP-4844, blobs outside a certain time window (the specific limit time has not yet been finalized, possibly 1 month or 1 year) will be automatically deleted;

From the long-term benefits of sharding, implementing EIP-4444 means that nodes do not need to store all historical data but only need to store data after a specific time based on history expiry;

These two solutions alleviate the tradeoff of storage space for blob-carrying transactions to varying degrees.

4. KZG Commitment

KZG Commitment is the polynomial commitment scheme adopted in EIP-4844.

Analysis of KZG Commitment

KZG is an abbreviation of the surnames of authors Aniket Kate, Gregory M. Zaverucha, and Ian Goldberg, who published the polynomial commitment scheme paper "Constant-Size Commitments to Polynomials and Their Applications" in 2010. This scheme has widespread applications in plonk-style zk-snark protocols.

Referring to the diagram in Dankrad Feist's presentation, the KZG root is similar to a Merkle root, with the distinction that the KZG root commits to a polynomial, meaning all positions are on this polynomial. In the context of proto-danksharding, the KZG root commits to a set of data, any of which can be verified as belonging to this whole.

This is also why KZG commitment is more compatible with subsequent implementations of DAS.

The process of KZG commitment is as follows:

• Prover: Provides proof, calculates the commitment of the data, the prover cannot change the given polynomial, and the commitment used for proof is only valid for this specific polynomial;
• Verifier: Receives the commitment value sent by the prover and verifies it to ensure that the prover has provided valid proof.

Advantages of KZG Commitment

I believe the main considerations regarding cost and security can be summarized but are not limited to the following points:

Cost

• KZG commitment features fast verification, relatively lower complexity, and simplicity;
• It does not require submitting additional proofs, thus reducing costs and saving bandwidth;
• The Point evaluation precompile required for data access can achieve lower costs.

Security

• In the event of a failure, it will only affect the data in the commitment corresponding to the blob, without other far-reaching impacts.

Greater Compatibility

• Overall, KZG commitment is compatible with DAS schemes in sharding, avoiding the costs of redundant development.

5. KZG Ceremony (trusted setup)

KZG Ceremony provides the trust setup for KZG Commitment, currently attracting over 20,000 participants and contributions, making it the largest trust setup in history.

The recent enthusiastic participation in the KZG Ceremony by the community is to provide the trust setup for the KZG commitment adopted in EIP-4844;

Process of KZG Ceremony

Referring to Vitalik's flowchart, anyone can participate and contribute secrets, mixing them with previous results to generate a new result, and so on, obtaining the final SRS through a nested process, assisting in completing the trust setup for KZG commitment.

Trust Setup

• EIP-4844 adopts a common multi-participant trust setup, namely powers-of-tau;
• Following a 1-of-N trusted model, regardless of how many people participate in generating the setup, as long as one person does not leak their generation method, the trusted initialization is valid;

Necessity

• The trust setup for KZG commitment can be simply understood as generating a parameter that needs to be relied upon each time a cryptographic protocol is executed, similar to the trusted initialization required for zk-snark;
• When the prover provides proof, KZG commitment C = f(s)g1. Here, f is the evaluation function, and s is the final secret obtained from the KZG trusted setup;
• It can be seen that the final secret is the core parameter for generating polynomial commitments, and as a trusted process for obtaining this core parameter, this KZG Ceremony is crucial for the overall implementation of sharding.

6. Changes Brought by EIP-4844

Rollup

Referring to the diagram provided on ethresear, rollups need to include the state delta and the versioned hash of the KZG commitment in the calldata for submission (zk-rollup also needs to submit zk proof).

The difference is that calldata only contains some small amounts of data, such as state delta and KZG commitment, while the transaction batch containing a large amount of transaction data is placed in the blob.

• Effectively reduces costs, as placing it in calldata is expensive;
• Reduces the occupation of block space.

Security

• Data availability: Blobs are stored on the beacon chain, equivalent to the security of layer 1;
• Historical data: Nodes will not only store blobs for a limited time; layer 2 rollups need to perform permanent data storage, thus security relies on rollups.

Cost

Proto-Danksharding introduces a new transaction type, and the addition of the low-cost data format blob will undoubtedly further reduce the costs of rollups. Depending on the actual on-chain applications and implementation progress, the costs of rollups may decrease by a factor of 10 or even 50;

At the same time, EIP-4844 introduces blob fees;

• Gas and blobs will have adjustable gas prices and limits;
• The charging unit for blobs is still gas, and the gas amount will fluctuate with traffic to maintain the goal of averaging 8 blobs per block (limiting the additional increase in data volume).

Implementation of Precompile

The data in the blobs cannot be directly accessed; the EVM can only obtain the commitment of the data blob. Therefore, rollups need to provide precompile to verify the validity of the commitment.

Below, we analyze the two precompile algorithms mentioned in EIP-4844.

Point Evaluation Precompile (for those interested in the mathematical principles, refer to Vitalik's analysis)

• Proves that multiple commitments point to the same data;
• Mainly targets zk-rollups, where rollups need to provide two types of commitments: 1. KZG commitment; 2. The commitment of the zk-rollup itself;
• For optimistic rollups, most have already adopted a multi-round fraud proof mechanism, where the data volume required for the final round fraud proof is relatively small. Therefore, using point evaluation precompile can achieve lower costs.

Blob Verification Precompile

• Proves that the versioned hash and blob correspond effectively;
• Optimistic rollups need to access the full data when submitting fraud proofs, so they first verify the legality of the versioned hash and blob before proceeding with fraud proof verification.

III. Danksharding: An Important Step Towards Full Sharding

Danksharding is named after Ethereum researcher Dankrad Feist.

1. Scalability: Further Expanding the Cache Space for Rollups

After the implementation of proto-danksharding, due to the introduction of the new transaction format with blobs, each block can gain an average of an additional 1MB of cache space. After the implementation of Danksharding, each block can gain an additional 16MB, with a maximum of 32MB allowed.

2. Data Availability: More Efficient Storage and Verification Strategies

Compared to proto-danksharding, which requires full nodes to download all data, after the implementation of Danksharding, Ethereum nodes only need to sample the blobs. The sampled data will be distributed among all nodes in the network and can form complete data blobs.

DAS: Efficient Sampling Checks

• Through erasure coding, it helps all nodes in the network more easily detect the probability of original data loss when downloading partial data, thus enhancing security.

3. Security: Essentially Unchanged

Since each node no longer stores all historical data, from the perspectives of data availability, backup, and sampling checks, security shifts from requiring at least one node to store all data to multiple nodes storing partial data and ultimately restoring complete data.

Although at first glance, reliance on a single point appears to be much more secure than reliance on multiple points, the number of nodes in the Ethereum network is sufficient to ensure the need for data backup, so security will not change significantly.

4. New Challenges: Increased Requirements for Block Builders

Although validators no longer need to download and retain all historical data, alleviating the requirements for bandwidth and storage hardware, block creators still need to upload blob data containing all transaction data. Here is a brief introduction to PBS (proposer/builder separation), referring to the diagram provided by Dankrad showing the application of PBS in the Danksharding scheme:

• The role of the block proposer is split into proposer and builder. Initially proposed for anti-MEV design, this separation in the design of Danksharding aims to reduce the bandwidth requirements during block creation.

IV. Other Sharding Solutions: Shardeum's Dynamic State Sharding

Shardeum is an EVM-compatible layer 1 public chain. Unlike Ethereum's static sharding solution, Shardeum enhances underlying scalability and security through dynamic state sharding, while naturally ensuring a high degree of decentralization;

1. Dynamic State Sharding

Advantages

The most intuitive advantage of dynamic state sharding is linear scaling, where nodes joining the network can be efficiently grouped by the sharding algorithm and respond quickly, enhancing the TPS of the blockchain network. In the design of dynamic state sharding, each node will cover different ranges of addresses, with redundancy designed to ensure efficient sharding and security.

Implementation in Scenarios

Setting aside the complexity of the ecosystem, from the perspective of the difficulty of implementing the two technologies, dynamic state sharding is more challenging than static sharding. It is evident that Shardeum's technical team has deep expertise in the technical aspects of sharding, and their previous research on Shardus technology has made significant contributions to the underlying technology of this public chain, even showcasing the linear scaling brought by dynamic state sharding in the early stages of the project.

2. Summary of Shardeum

Product

Nodes are divided into different groups, following the divide and conquer approach, to offload the computational and storage workloads, allowing for a higher degree of parallel processing. Thus, more nodes can join, further enhancing the throughput and decentralization of the public chain.

Team

The team has rich market experience and strong narrative capabilities, with deep research into dynamic sharding.

Technology

They have designed a suitable sharding solution (dynamic state sharding) and consensus design (Proof of Stake + Proof of Quorum) for their scenario, prioritizing scalability while ensuring a higher degree of decentralization and security.

Progress

They plan to launch the betanet on 2023-02-02, which is worth paying attention to.

V. Thoughts and Outlook on Sharding

• Sharding is a long-term solution for Ethereum scalability and a long-term plan of great value and significance. During the process of implementing sharding, all existing solutions may be continuously iterated, including the currently mentioned proto-danksharding, danksharding, etc., which are worth ongoing attention;
• Understanding the general direction of sharding is important, but the technical solutions adopted in each proposal during the process of achieving full sharding (PBS, DAS, multidimensional fee market) are equally worthy of attention, and I believe many excellent teams and projects related to this will emerge;
• Sharding is a term that encompasses a type of scalability technology, but there is not just one specific solution for its implementation. It is essential to recognize that different public chains will have sharding solutions suitable for their own scenarios. For example, some designs in danksharding are only suitable for the Ethereum network, and the tradeoff of security requires a large number of nodes to offset;
• A reasonable combination of sharding and other scalability solutions can enhance scalability 1+1>2. The current Danksharding is not a standalone scalability solution but complements other implementations in the Ethereum ecosystem. For instance, Danksharding and rollups together achieve better results for Ethereum scalability.

Reference

https://notes.ethereum.org/@dankrad/kzgcommitmentsin_proofs

https://notes.ethereum.org/@dankrad/new_sharding

https://vitalik.ca/general/2022/03/14/trustedsetup.html

https://notes.ethereum.org/@vbuterin/protodankshardingfaq#Why-use-the-hash-of-the-KZG-instead-of-the-KZG-directly

https://ethresear.ch/t/easy-proof-of-equivalence-between-multiple-polynomial-commitment-schemes-to-the-same-data/8188

https://dankradfeist.de/ethereum/2020/06/16/kate-polynomial-commitments.html

https://notes.ethereum.org/@dankrad/new_sharding

https://eips.ethereum.org/EIPS/eip-4844

https://www.eip4844.com/

https://biquanlibai.notion.site/Data-Availability-caa896aae59d489b98f2448f17b01640

https://docs.google.com/presentation/d/1-pe9TMF1ld185GL-5HSWMAsaZLbEqgfU1sYsHGdD0Vw/edit#slide=id.g1150d91b32e0474

https://ethresear.ch/t/a-design-of-decentralized-zk-rollups-based-on-eip-4844/12434