Ethereum's next milestone upgrade? Vitalik explains what Danksharding really is

Author: Vitalik Buterin

Compiled by: DeFi Dao

Ethereum founder Vitalik Buterin recently addressed questions related to Proto-danksharding (also known as EIP-4844). What exactly can this new sharding design bring to Ethereum?

What is Danksharding?

Danksharding is a new sharding design proposed for Ethereum, which introduces some significant simplifications compared to previous designs.

The main difference between all recent Ethereum sharding proposals since 2020 (including Danksharding and earlier versions) and most non-Ethereum sharding proposals is Ethereum's rollup-centric roadmap: Ethereum sharding does not provide more space for transactions but rather for data, which the Ethereum protocol itself does not attempt to interpret.

Verifying a blob only requires checking whether the blob is available, i.e., whether it can be downloaded from the network. The data space in these blobs is expected to be utilized by Layer 2 rollup protocols that support high-throughput transactions.

The main innovation introduced by Danksharding is the merged fee market: there are no longer a fixed number of shards, each with different blocks and different block proposers. In Danksharding, there is only one proposer that selects all transactions and all data entering that slot.

To avoid imposing high system requirements on validators due to this design, we introduced proposer/builder separation (PBS): a special class of participants called block builders bids for the right to choose slots, and proposers only need to select the highest bidding valid header.

Only block builders need to handle the entire block (here, decentralized oracle protocols can also be used to achieve distributed block building); all other validators and users can very efficiently verify blocks through data availability sampling (remember: the "large" part of the block is just data).

What is proto-danksharding (also known as EIP-4844)?

Proto-danksharding (also known as EIP-4844) is an Ethereum Improvement Proposal (EIP) aimed at implementing most of the logic and "scaffolding" (such as transaction formats and validation rules) that constitute the complete Danksharding specification, but it has not yet actually implemented any sharding. In the proto-danksharding implementation, all validators and users still need to directly verify the availability of the complete data.

The main feature introduced by proto-danksharding is a new transaction type, which we refer to as a transaction with a blob. Transactions with blobs are similar to regular transactions, except they also carry additional data called blobs. The blobs are very large (~125 kB) and are much cheaper than a similar amount of calldata. However, EVM execution cannot access the blob data; the EVM can only see the commitment to the blob.

Because validators and clients still need to download the full blob content, the data bandwidth target in proto-danksharding is 1 MB per slot, rather than the full 16 MB. However, since this data does not compete with the gas usage of existing Ethereum transactions, there are still significant scalability benefits.

Why is it acceptable to add 1 MB of data to blocks that everyone must download, rather than making calldata 10 times cheaper?

This relates to the difference between average load and worst-case load. Today, we have encountered average block sizes of about 90 kB, but the theoretically possible maximum block size (if all 30M gas in the block is used for calldata) is about 1.8 MB. The blocks that the Ethereum network has processed in the past are close to this maximum.

However, if we simply reduce the calldata gas cost by 10 times, then although the average block size would increase to still acceptable levels, the worst-case scenario would become 18 MB, which is simply too much for the Ethereum network.

The current gas pricing scheme cannot separate these two factors: the ratio between average load and worst-case load depends on how much gas users spend on calldata versus other resources, meaning that gas prices must be set based on the worst-case possibilities, leading to average loads unnecessarily lower than what the system can handle.

However, if we change the gas pricing to more explicitly create a multi-dimensional fee market, we can avoid the mismatch between average and worst-case loads and include data close to the maximum amount we can safely handle in each block. Proto-danksharding and EIP-4488 are two proposals that do just that.

How does proto-danksharding compare to EIP-4488?

EIP-4488 is an earlier and simpler attempt to address the same average/worst-case load mismatch issue. EIP-4488 uses two simple rules to achieve this:

• The calldata gas cost is reduced from 16 gas per byte to 3 gas per byte.
• A limit of 1 MB per block plus an additional 300 bytes per transaction (theoretical maximum: ~1.4 MB).

The hard limit is the simplest way to ensure that a significant increase in average load does not lead to an increase in worst-case load. The reduction in gas costs will greatly increase the use of rollups, potentially increasing average block sizes to hundreds of KB, but the hard limit will directly prevent the worst-case possibility of including a single block of 10 MB. In fact, the worst-case block size would be smaller than it is now (1.4 MB vs. 1.8 MB).

Proto-danksharding, on the other hand, creates a separate transaction type that can store cheaper data in large fixed-size blobs and limits how many blobs each block can contain. These blobs cannot be accessed from the EVM (only commitments to the blobs), and the blobs are stored by the consensus layer (beacon chain) rather than the execution layer.

The main practical difference between EIP-4488 and proto-danksharding is that EIP-4488 attempts to minimize the changes required today, while proto-danksharding makes many changes today, so future upgrades to full sharding will require very few changes.

While implementing full sharding (using data availability sampling, etc.) is a complex task and remains complex after proto-danksharding, this complexity is contained within the consensus layer. Once proto-danksharding is launched, execution layer client teams, rollup developers, and users will not need to do further work to complete the transition to full sharding.

Note that the choice between the two is not binary: we can implement EIP-4488 as soon as possible and then follow up with proto-danksharding six months later.

What parts of complete danksharding does proto-danksharding implement? What still needs to be implemented?

According to EIP-4844, the completed work in this EIP includes:

• A new transaction type whose format is exactly what needs to exist in "full sharding."
• All execution layer logic required for full sharding.
• All execution/consensus cross-validation logic required for full sharding.
• Layer separation between BeaconBlock validation and data availability sampling blobs.
• Most of the BeaconBlock logic required for full sharding.
• Self-adjusting independent gas price for blobs.

The work that still needs to be completed to implement full sharding includes:

• Low-level expansion of blob_kzg in the consensus layer to allow for 2D sampling.
• The actual implementation of data availability sampling.
• PBS (proposer/builder separation) to avoid requiring a single validator to handle 32 MB of data in one slot.
• Hosting proofs or similar protocol requirements for each validator to verify specific portions of sharded data in each block.

Note that all remaining work is consensus layer changes and does not require any additional work from execution client teams, users, or rollup developers.

What about the increased disk space requirements from all these very large blocks?

Both EIP-4488 and proto-danksharding lead to a long-term maximum usage of about 1 MB per slot (12 seconds). This amounts to approximately 2.5 TB per year, which is far higher than the growth rate currently required by Ethereum.

In the case of EIP-4488, addressing this issue requires a historical record expiration scheme (EIP-4444), where clients are no longer required to store historical records beyond a certain time period (proposed durations range from 1 month to 1 year).

In the case of proto-danksharding, regardless of whether EIP-4444 is implemented, the consensus layer can implement separate logic to automatically delete blob data after a period (e.g., 30 days). However, regardless of what short-term data expansion solution is adopted, it is strongly recommended to implement EIP-4444 as soon as possible.

Both strategies will limit the additional disk load on consensus clients to a few hundred GB at most. In the long run, adopting some historical expiration mechanism is essentially mandatory: full sharding will add about 40 TB of historical blob data each year, so users will realistically only be able to store a small portion of it for a period of time. Therefore, it is worth setting expectations for this early on.

How will users access old blobs if the data is deleted after 30 days?

The purpose of the Ethereum consensus protocol is not to guarantee the permanent storage of all historical data. Instead, the goal is to provide a highly secure real-time bulletin board and leave room for other decentralized protocols to provide more long-term storage.

The existence of the bulletin board is to ensure that the data published on it is available long enough for any user wanting that data or any long-term protocol backing up the data to have sufficient time to retrieve it and import it into their other applications or protocols.

In general, long-term historical storage is easy. While 2.5 TB per year is too much for regular nodes, it is very manageable for dedicated users: you can purchase very large hard drives for about $20 per TB, which is perfectly adequate for hobbyists.

Unlike consensus with an N/2-of-N trust model, historical storage has a 1-of-N trust model: you only need one of the data storers to be honest. Therefore, each piece of historical data only needs to be stored a few hundred times, rather than the thousands of nodes that are doing real-time consensus validation.

Some practical ways to store complete historical records and make them easily accessible include:

• Application-specific protocols (e.g., Rollup) may require their nodes to store portions of historical records relevant to their applications. Lost historical data poses no risk to the protocol, only to individual applications, so it makes sense for applications to bear the burden of storing data relevant to themselves.
• Storing historical data in BitTorrent, e.g., automatically generating and distributing a 7 GB file daily that contains blob data from blocks.
• Ethereum portal networks (currently under development) can easily scale to store history.
• Block explorers, API providers, and other data services may store complete historical records.
• Individual enthusiasts and scholars engaged in data analysis may store complete historical records. In the latter case, storing history locally provides them with significant value, as it makes direct computation on it much easier.
• Third-party indexing protocols like TheGraph may store complete historical records.

At higher levels of historical storage (e.g., 500 TB per year), the risk of some data being forgotten becomes higher (in addition, the data availability verification system becomes more strained). This could be the true limit of scalability for sharded blockchains. However, all proposed parameters are currently far from this point.

What is the format of blob data, and how is it submitted?

A blob is a vector containing 4096 field elements, with numbers in the range:

0 <= x < 52435875175126190479447740508185965837690552500527637822603658699938581184513

Mathematically, a blob is viewed as representing a polynomial of degree < 4096 over a finite field with the above modulus, where the field element at position i in the blob is the evaluation of that polynomial at wi. w is a constant satisfying w=1.

The commitment to the blob is a KZG commitment to that polynomial. However, from an implementation perspective, focusing on the mathematical details of the polynomial is not important. Instead, there will simply be a vector of elliptic curve points (based on a trusted setup for Lagrange), and the KZG commitment to the blob will just be a linear combination. Referencing the code from EIP-4844:

def blobtokzg(blob: Vector[BLSFieldElement, 4096]) -> KZGCommitment: computedkzg = bls.Z1 for value, pointkzg in zip(tx.blob, KZGSETUPLAGRANGE): assert value < BLSMODULUS computedkzg = bls.add( computedkzg, bls.multiply(pointkzg, value) ) return computed_kzg

BLSMODULUS is the modulus mentioned above, and KZGSETUP_LAGRANGE is a vector of elliptic curve points based on a trusted setup for Lagrange. For implementers, it is reasonable to simply treat it as a black-box specialized hash function.

Why use KZG's hash instead of using KZG directly?

EIP-4844 does not use KZG to directly represent blobs but instead uses a versioned hash: a single 0x01 byte (indicating this version) followed by the last 31 bytes of the SHA256 hash of the KZG.

This is done for EVM compatibility and future compatibility: KZG commitments are 48 bytes, while EVM more naturally uses 32-byte values. If we switch from KZG to something else (for example, for quantum resistance reasons), KZG commitments can still remain at 32 bytes.

What are the two precompiles introduced in proto-danksharding?

Proto-danksharding introduces two precompiles: blob verification precompile and point evaluation precompile.

The blob verification precompile is self-explanatory: it takes the versioned hash and the blob as inputs and verifies that the provided versioned hash is indeed a valid versioned hash of the blob. This precompile is intended for use by Optimistic Rollups. Referencing EIP-4844:

Optimistic Rollups only need to actually provide the underlying data when submitting fraud proofs. The fraud proof submission function will require the entire content of the fraud blob to be submitted as part of calldata. It will use the blob verification function to validate the data against the previously submitted versioned hash, and then perform fraud proof validation on that data as it does today.

The point evaluation precompile takes the versioned hash, x coordinate, y coordinate, and proof (KZG commitment of the blob and KZG evaluation proof) as inputs. It verifies the proof to check P(x) = y, where P is the polynomial represented by the blob with the given versioned hash. This precompile is intended for use by ZK Rollups. Referencing EIP-4844:

ZK Rollups will provide two commitments for their transaction or state incremental data: KZG in the blob and some commitments from any proof system used internally by the ZK Rollup. They will use the commitment proofs of the equivalent protocol, using the point evaluation precompile, to prove kzg (the protocol ensures it points to available data) and the ZK Rollup's own commitment referencing the same data.

Note that most major Optimistic Rollup designs use multi-round fraud-proof schemes, where the last round only requires a small amount of data. Therefore, it is conceivable that Optimistic Rollups could also use the point evaluation precompile instead of the blob verification precompile, and doing so would be cheaper.

What does a KZG trusted setup look like?

See:

A general description of how trusted setups work

https://vitalik.ca/general/2022/03/14/trustedsetup.html on powers-of-tau

Examples of implementations of all important trusted setup-related computations

https://github.com/ethereum/research/blob/master/trusted_setup/trusted_setup.py

Specifically, in our case, the current plan is to run four ceremonies in parallel of sizes (n1=4096,n2=16), (n1=8192,n2=16), (n1=16834,n2=16), and (n1=32768,n2=16) (with different secrets). Theoretically, only the first is needed, but running more larger sizes allows us to increase the blob size for future applicability. We cannot just have one larger setup because we want to have a hard limit on the polynomial degrees that can be effectively submitted, which is equal to the blob size.

A possible practical approach is to start with the Filecoin setup and then run a ceremony to extend it. Multiple implementations, including browser implementations, will allow many people to participate.

Can't we use some other commitment scheme without a trusted setup?

Unfortunately, using anything other than KZG (such as IPA or SHA256) would make the sharding roadmap much more difficult. There are several reasons for this:

Non-arithmetic commitments (such as hash functions) are incompatible with data availability sampling, so if we use such a scheme, we would have to switch to KZG anyway when we move to full sharding.

IPA may be compatible with data availability sampling, but it leads to more complex schemes with weaker properties (e.g., self-healing and distributed block building become more difficult).

Neither hashes nor IPA are compatible with cheap implementations of point evaluation precompiles. Therefore, implementations based on hashes or IPA would not effectively enable cheap fraud proofs in ZK Rollups or support multi-round Optimistic Rollups.

Thus, unfortunately, the functional loss and increased complexity of using anything other than KZG far outweigh the risks of KZG itself. Moreover, any risks associated with KZG are contained: a KZG failure would only affect Rollups and other applications relying on blob data, without impacting the rest of the system.

How "complex" and "new" is KZG?

KZG commitments were introduced in a paper in 2010 and have been widely used in PLONK-type ZK-SNARK protocols since 2019. However, the underlying mathematics of KZG commitments is relatively simple arithmetic built on top of the mathematics of elliptic curve operations and pairings.

The specific curve used is BLS12-381, which was invented by Barreto-Lynn-Scott in 2002. Elliptic curve pairings, which are necessary for verifying KZG commitments, are very complex mathematics, but they were invented in the 1940s and have been applied in cryptography since the 1990s. By 2001, many cryptographic algorithms using pairings had been proposed.

From the perspective of implementation complexity, KZG is not harder to implement than IPA: the function to compute the commitment (see above) is exactly the same as in the case of IPA, just using a different set of elliptic curve point constants. The point verification precompile is more complex because it involves pairing evaluations, but the mathematics is the same as that already completed in the implementation of EIP-2537 (BLS12-381 precompile) and is very similar to the bn128 pairing precompile (see also: optimized Python implementation). Therefore, implementing KZG verification does not require complex "new work."

What are the different software components implemented in proto-danksharding?

There are four main components:

1. Changes to consensus in the execution layer (see EIP):

• A new transaction type that includes blobs.
• An opcode that outputs the versioned hash of the i-th blob in the current transaction.
• Blob verification precompile.
• Point evaluation precompile.

1. Changes to consensus in the consensus layer (see this folder in the repo):

• A list of blob KZG in BeaconBlockBody.
• A "sidecar" mechanism where the complete blob content is passed along with a separate object from the BeaconBlock.
• Cross-checking between the blob versioned hash in the execution layer and the blob KZG in the consensus layer.

1. Memory pool

• BlobTransactionNetworkWrapper (see the network section of the EIP).
• Stronger anti-DoS protection to compensate for the large blob sizes.

1. Block building logic

• Accepting transaction wrappers from the memory pool, placing transactions into ExecutionPayload, and putting KZG into the beacon block and sidecar body.
• Responding to the two-dimensional fee market.

Note that for the minimal implementation, we do not need a memory pool at all (we can rely on Layer 2 transaction bundling markets); we only need a client to implement block building logic. Only consensus changes in the execution layer and consensus layer require extensive consensus testing, which is relatively lightweight. Anything in between such a minimal implementation and a "full" deployment where all clients support block production and memory pools is possible.

What does the multi-dimensional fee market in proto-danksharding look like?

Proto-danksharding introduces a multi-dimensional EIP-1559 fee market, where there are two resources, gas and blobs, each with separate floating gas prices and separate limits.

That is, there are two variables and four constants:

Blob fees are charged in gas, but it is a variable amount of gas that will be adjusted so that, in the long run, the average number of blobs per block actually equals the target number.

The two-dimensional nature means that block builders will face a more challenging problem: rather than simply accepting transactions with the highest priority fees until they run out of transactions or reach the block gas limit, they will have to avoid hitting two different limits simultaneously.

Here's an example. Suppose the gas limit is 70, and the blob limit is 40. The mempool has enough transactions to fill the block, with two types (tx gas includes per-blob gas):

• Priority fee 5 per gas, 4 blobs, 4 total gas.
• Priority fee 3 per gas, 1 blob, 2 total gas.

A miner following a naive "lower priority fee" algorithm would fill the entire block with 10 transactions of the first type (40 gas) and earn 5 * 40 = 200 gas in revenue. Because these 10 transactions completely filled the blob limit, they would not be able to include more transactions. But the optimal strategy is to take 3 transactions of the first type and 28 transactions of the second type. This gives you a block containing 40 blobs and 68 gas, with revenue of 5 * 12 + 3 * 56 = 228.

Do execution clients now have to implement complex multi-dimensional knapsack problem algorithms to optimize their block production? No, for several reasons:

• EIP-1559 ensures that most blocks will not hit either limit, so in practice, only a few blocks face the multi-dimensional optimization problem. In the usual case where the mempool does not have enough (sufficiently paid) transactions to hit either limit, any miner can achieve optimal revenue by including every transaction they see.
• In practice, relatively simple heuristic methods can get close to optimal. For some data on this, see Ansgar's analysis of EIP-4488.
• Multi-dimensional pricing is not even the largest source of revenue from specialization—MEV is. Specialized MEV revenue extracted from on-chain DEX arbitrage, liquidations, front-running NFT sales, etc., constitutes a large portion of the total "extractable revenue" (i.e., priority fees): specialized MEV revenue seems to average about 0.025 ETH per block, while total priority fees are typically around 0.1 ETH per block.
• Proposer/builder separation is designed around highly specialized block production. PBS transforms the block building process into an auction where specialized participants can bid for the privilege of creating blocks. Regular validators only need to accept the highest bid. This is to prevent MEV-driven economies of scale from spreading to validator centralization, but it addresses all the potential issues that could make optimizing block building more difficult.

For these reasons, the more complex fee market dynamics will not significantly increase centralization or risk; in fact, the more broadly applied principles may actually reduce the risk of DoS attacks!

How does the exponential EIP-1559 blob fee adjustment mechanism work?

Today's EIP-1559 adjusts the base fee b to reach a specific target gas usage level t as follows:

where b(n) is the base fee of the current block, b(n+1) is the base fee of the next block, t is the target, and u is the gas used.

One major issue with this mechanism is that it does not actually target t. Suppose we get two blocks, the first u=0, and the next u=2t. We get:

Although the average usage equals t, the base fee drops by 63/64. So the base fee only stabilizes when the usage rate is slightly above t; in practice, this is clearly about 3% higher, although the exact number depends on variance.

A better formula is exponential adjustment:

exp(x) is the exponential function e^x, where e≈2.71828. For small x values, exp(x)≈1+x. However, it has the convenient property of being independent of transaction replacement: multi-step adjustments

only depend on the total u1+…+u/n, and not on the distribution. To understand why, we can do some math:

Thus, the same transactions included will lead to the same final base fee, regardless of how they are distributed across different blocks.

The fact that the current base fee equals

clearly indicates that the excess cannot exceed a very narrow range: if it exceeds 8t∗60, then the base fee becomes e^60, which is absurdly high and no one can pay it. If it is below 0, then resources are essentially free, and the chain will be spammed until the excess returns above zero.

The adjustment mechanism works entirely in these terms: it tracks the actual total (u1+u2+…+u/n) and calculates the targeted total (nt), and sets the price based on the difference exponentially. To simplify the calculation, we do not use e^x but instead use 2^x; in fact, we use an approximation of 2^x: the fake_exponential function in the EIP. The fake exponential is almost always within 0.3% of the actual value.

To prevent prolonged underutilization from leading to prolonged 2x full blocks, we add an additional feature: we will not let the excess blocks drop below zero. If actualtotal is below targetedtotal, we simply set actualtotal to equal targetedtotal. In extreme cases (if blob gas drops to zero), this does indeed break the invariance of transaction ordering, but the increased safety makes this an acceptable trade-off.

Also, note an interesting result of this multi-dimensional market: when proto-danksharding is initially introduced, there may initially be very few users, so for a while, the cost of a blob will almost certainly be very cheap, even as "regular" Ethereum blockchain activity remains expensive.

The author believes that this fee adjustment mechanism is better than the current approach, so ultimately all parts of the EIP-1559 fee market should shift to using it.

For a longer and more detailed explanation, see Dankrad's post.

How does fake_exponential work?

For convenience, here is the code for fake_exponential:

def fake_exponential(numerator: int, denominator: int) -> int: cofactor = 2 ** (numerator // denominator) fractional = numerator % denominator return cofactor + ( fractional * cofactor * 2 + (fractional ** 2 * cofactor) // denominator ) // (denominator * 3)

Here is the core mechanism re-expressed mathematically, removing rounding:

The goal is to stitch together many instances of (QX), where one is appropriately shifted and scaled for each range [2^k,2^(k+1)]. Q(x) itself is an approximation of 2^x for 0≤x≤1, chosen for the following properties:

• Simplicity (it is a quadratic equation).
• Correctness at the left edge (Q(0)=2^0=1).
• Correctness at the right edge (Q(1)=2^1=2).
• Smooth slope (we ensure Q'(1)=2∗Q'(0), so the slope of each shifted + scaled copy of Q at its right edge matches the slope of the next copy at its left edge).

The last three requirements yield three linear equations for the three unknown coefficients, and the Q(x) given above provides the unique solution.

The approximation is surprisingly good; for all inputs except the smallest, the answer given by fake_exponential is within 0.3% of the actual value of 2^x:

What issues in proto-danksharding are still under debate?

Note: This section is easily outdated. Do not trust it to provide the latest thoughts on any specific issue.

• How capable is the chain in handling persistent long-term 1 MB+ blocks? If the risk is too great, should the target blob count be reduced from the start?
• Should blobs be priced in gas or ETH (which is burned)? Should there be other adjustments to the fee market?
• Should the new transaction type be treated as a blob or an SSZ object, in which case would require changing ExecutionPayload to a union type? (This is a trade-off between "doing more work now" and "doing more work later.")
• The exact details of the trusted setup implementation (technically beyond the scope of the EIP itself, as for implementers, this setup is "just a constant," but it still needs to be completed).