The First Five Years of MEV: The Past and the Future

Original Title: 《MEV: The First Five Years》

Author: James Prestwich, Founder and CTO of Nomad

Compiled by: Qianwen, ChainCatcher

Preface

Five years ago, I wrote a blog post titled 《Miners Aren't Your Friends》, introducing MEV into the Ethereum narrative. At that time, we didn't call it MEV. Phil Daian and his collaborators named it "Miner Extractable Value" in their groundbreaking paper "Flash Boys 2.0" more than a year later. We had planned to publish a follow-up article introducing Python snippets for generating front-running trades on EtherDelta and other mainnet DEXs. Later, I got involved in building HTLCs and other cross-chain tools, and the follow-up article was postponed and ultimately shelved.

Today, five years later, I sit down to write this article about MEV, feeling a bit surreal. This feeling is exactly the same as last time. We are once again at the beginning of a long bear market, deep in the aftermath of the last speculative frenzy. The familiar feeling has quietly returned. Just like last time, I am convinced that the next time will be different. Next time, we will learn from our mistakes, and we will build something better. Maybe.

Author's Note: This is not an introductory reading on MEV. Many people have written on related topics, and most have done so better than I have. This article assumes you are already familiar with MEV; if not, please read some blogs before coming back to this.

This is a narrative history of MEV. It is not entirely neutral and carries my own judgments and emotional tones. In this story, I am (at best) a supporting character, but this is my blog, so I can tell it from my perspective.

MEV: The First Five Years

MEV is foundational to protocol design. Every serious practitioner knows about MEV, plans for it, and has strong opinions about it. From our current position, it is easy to forget that our understanding of MEV has developed only in recent years. It has origins as early as cryptocurrency itself, but the related terminology, expressions, and tools are still evolving. The past two years have fundamentally changed our understanding of MEV. We easily forget that MEV emerged completely from nothing.

Before MEV

The discourse surrounding MEV can be traced back to early 2010s Bitcoin research on "Fee Sniping." Fee sniping was later adapted to the EVM state model and formally became the "time bandit" attacks described in the "Flash Boys 2.0" paper. Bitcoin implemented the first (unintentional) consensus layer MEV mitigation measure (the 100-block coinbase maturity rule) at launch, and in December 2014, added a one-block time lock for transactions using wallet nodes, thereby purposefully implementing the first client-layer MEV defense.

Fee sniping defense flowchart, from 0xB10C

The competition for state and the ambiguously committed state transitions led to the emergence of MEV. Bitcoin essentially lacks shared state, and thus there is no competition, and Bitcoin's state transitions are strictly regulated, so MEV is generally limited to fee sniping and other types of double spending. In other words, without directly attacking the consensus mechanism, the MEV that Bitcoin miners can extract is minimal. This makes Bitcoin unsuitable for MEV research, and we won't spend much time on Bitcoin or similar chains.

The Birth of MEV (2018-2019)

Like any good relationship, MEV needs to do two things: compete and commit. First, it needs people to compete for control of public state. MEV requires a state that is desirable and for which people are willing to pay. Secondly, it needs commitment (before execution). For a time, MEV hoped that people would commit to updating the state (which is desirable and contested), but that commitment had not yet been executed. When people sign and broadcast transactions, they commit to competing for control of that state. However, the lag between commitment and execution allows MEV to intervene, touching that state both before and after. It pulls on user intent, pushing their commitments toward the worst acceptable outcome, allowing extractors to pocket the arbitrage.

DEXs are an excellent place for MEV to be implemented. They have a perfect competition scheme: everyone wants to trade in the market, and every trade can compete with others. For example, the self-matching order book DEX design EtherDelta, launched in 2016, allowed miners to gain priority in trades, but no one really exploited it for anything. Some on-chain automated matching CLOBs (Central Limit Order Books) seemed crazy at the time, and looking back, they are still quite strange. While we knew that fees cost money, by mid-2018, the price of a unit of fee was about 1/400 of today’s price, and 1/10000 of last year’s. It wasn't until 2020 that we had an intuitive understanding of the fee market.

The launch of DAI (now SAI) in 2017 introduced liquidations to DeFi. Liquidations brought a significant but infrequent amount of MEV (Spike MEV). Over the past few years, we have observed oracles cheating to win liquidations in their own protocols and winning Spike MEV through extreme fee auctions. Spike MEV can also arise from hacks, NFT drops, and other special events. Because it cannot be divided into smaller packets, Spike MEV has a significant impact on protocol operations. Protocol design must account for infrequent but potentially consequential malicious incentives.

On the other hand, DEXs tend to change with external market fluctuations, generating Flow MEV from the movement of their markets. It is characterized by more frequent and smaller MEV packets. The purpose of AMMs is closely tied to MEV; their existence is to track external markets, achieving this by granting value to extractors who push constant functions toward "fair" pricing. In reality, they maintain efficiency through MEV-extracting "arbitrageurs." The first AMMs in late 2018 (first Bancor, then Uniswap) began to provide liquidity on order books. This led to a corresponding increase in MEV and drew significant attention to the issue. Of course, in 2018, Uniswap's liquidity pools were in single digits and had far fewer assets.

At this time, MEV research was almost nonexistent; of course, we didn't even call it MEV. We mostly referred to it as front-running, borrowing terms like "DEX front-running" or "liquidation front-running" from TradFi exchanges. We knew it wasn't just about inserting orders. After theoretically understanding backrunning and sandwich attacks, we realized that if extractors were profitable, trades could be delayed or rolled back.

Past DeFi may have unknowingly established the necessary environment for creating MEV. However, no one was really concerned about the issue. The state at that time was not ideal enough, so the returns did not exist. Because no one had practical experience in this area, we were not even clear about what would happen when extraction arrived.

The Formation of Flashbots (2019-2020)

With the publication of "Flash Boys 2.0" in April 2019, MEV research began to gain popularity. I recommend you reread this excellent paper to understand each author. It is a foundational paper for protocol and mechanism designers and will forever be part of the standard curriculum. Without Tina, I think its development might have stagnated long ago.

I met Tina in early 2018 in a VC's office. We were connected through a mutual friend, and we were both at the same pitch event. I was promoting a variant of Summa's HTLC project, which was ultimately halted due to our realization of the limitations of cross-chain protocols requiring two online parties (and the market's tolerance for complexity). Tina was pitching a social blockchain game about milking cows (not the later CoW Swap). We shared a common interest in derivatives mining and had some ideas on how to enter the space. We would talk at every hackathon during the bear market. She had a knack for identifying forward-thinking ideas and assembling teams around them. Her hackathon project later became CarboClan, a game based on derivatives mining related to honey lemon.

In the latter half of 2019, Tina began organizing events. She built defii.wtf in just a week and hosted it alongside DevCon Osaka. Just three weeks later, she hosted macro.wtf. Shortly thereafter, Pirate Ships was formed, a name derived from a joke we made about flash loans and flash mints at EthDenver in 2019. Pirate Ships were salon-style gatherings hosted by Tina in San Francisco, New York, and elsewhere. You could find a diverse group of researchers, engineers, operators, scammers, and other cryptocurrency practitioners here. These "ships" (salons) sometimes invited researchers, sometimes had specific themes, and sometimes were casual meetups with locals. This organization ran from the end of 2019 through the entire pandemic, attracting the most dedicated supporters.

The birth of Flashbots owes much more to Pirate Ships than to the paper "Flash Boys 2.0." It was born as a permanent "ship." MEV Ship was established in mid-2020 and expanded from a one-person space to an online digital collective. Other founders and stewards of Flashbots—Stephen, Phil, Alex—were pulled into this "ship" (you can still find some stories in the Flashbots docs). Eventually, the "ship" was formally established and renamed "Flashbots," complete with a cute robot emoji, quietly opening an immortal chapter.

That whole year was a long wait for breakthroughs. Flashbots Research (i.e., MEV Ship) made its debut, bringing MEV to the center of Ethereum discourse. The summer of DeFi had begun, but we still didn't know what impact it would have on us. The release of Mev-explore and Mev-inspect provided rough tools for available MEV chains, and suddenly this became feasible—it was no longer just a theory. The arrival of Flashbots auctions that autumn heralded that MEV extraction tools were just around the corner. Money was everywhere, and the race to "pick up money" had begun.

MEV Ship laid the foundation for Flashbots and established a culture of collaborative research and experimentation, driving the organization's development. The clear mission derived from this culture has defined the narrative of MEV. MEV extraction will inevitably be productized, but the culture and spirit of Flashbots will not.

Specialization (2021-Present)

Of course, the aura of MEV Ship and Flashbots attracted a lot of technical talent. The Flashbots auction (mev-geth and flashbots relays) launched in January 2021—less than three months after Flashbots announced—at a perfect timing. Ethereum experienced its largest price surge in history, and the amount of MEV generated correspondingly increased; at such a moment, Mev-geth was launched.

Before the Flashbots auction, some brave searchers extracted MEV by broadcasting transactions and specific fees to the transaction pool. This was unreliable and highly inefficient. It required searchers to see transactions in the pool, simulate, extract, and broadcast within the block. The regulations on fees before EIP 1559 and the general unreliability of the transaction pool complicated this process. Assuming blocks were sorted by fees, front-running would select fees to execute immediately before the target. Spike MEV trades would pay shockingly high fees. Because searchers used public transaction pools, fierce real-time auctions occurred. This led to a chaotic yet interesting situation.

The relative simplicity of Flashbots auctions drove their adoption. It was simplified to a single RPC endpoint, allowing searchers to submit bundles, which should be included in the block exactly as specified, while reverting would discard the bundle. This created a clean interface for specialization. Searchers hunt and capture MEV, ensuring it is bundled and passed to miners for inclusion on-chain. Flashbots auctions quickly defined extraction to the extent that other systems became almost meaningless.

The subsequent development of Flashbots auctions is what we see in mev-boost, separating "builders" from "producers" (a term for miners and stakers). I first heard about proposer-builder separation ("PBS") at EthBerlin in 2019. Will was working on eth2 at the time and taught me a lot about state witness production and updates in "stateless Ethereum" design. I don't remember what we called it back then, but it was conceptualized as a specialized node defined by the protocol, with additional responsibilities. "Stateless Ethereum" was terminated due to the pandemic. MEV PBS is an instance in mev-boost that exists outside the protocol, based on market specialization rather than protocol-defined specialization.

Today, we have a relatively mature MEV supply chain. Searchers mine MEV from transaction flows, competing for price increases and traffic. Searching requires skill and extreme specialization. They operate in secrecy to maintain an advantage over other searchers. Builders accept bundles from searchers and construct them into blocks. Searchers and builders are symbiotic. Builders rely on searchers for extraction, while searchers depend on builders to honestly include the bundles.

Builders purchase the right to include their blocks in the main chain from proposers (a term for miners or stakers). Proposers have the rights granted by the protocol to add blocks to the chain and choose the builder block that pays the most MEV. Users create MEV, which is extracted by searchers. Searchers pay builders; builders pay proposers. The gears turn, blocks are assembled, and transactions enter the chain.

Since we are on this topic, let's briefly mention it. Bundles have an unexpected side effect: they cause searchers to inadvertently subsidize the transaction fees of the extraction targets. When searchers include user transactions in bundled transactions, those user transactions are confirmed earlier and more reliably than in other cases. The MEV extracted is partially paid to builders and proposers along the supply chain. This effectively converts part of the extracted MEV into transaction fees for bundled transactions. Essentially, the extracted MEV pays proposers a "shadow fee." The priority of transactions is first determined by their MEV, and second by their protocol fees.

Extraction unexpectedly undermined EIP-1559. Non-bundled transactions charge gas fees beyond the base fee (calculated by gas units). Bundled transactions, however, still operate as a first-price auction, charging a uniform tip on top of the base fee rather than calculating by gas. This means there are now two interdependent fee markets: one is the shadow market for MEV transactions, and the other is the regular market. MEV can purchase priority. We predicted the emergence of these shadow fee markets before EIP 1559 (and indeed before all other fee mechanisms). What we did not anticipate was the market situation using MEV. Because extraction operates behind the scenes, users benefit from the shadow market without even realizing its existence.

From a mechanism design perspective, MEV-driven PBS is unlikely to be "safe" or "incentive-compatible." So far, Flashbots has relied on honest assumptions to fill the gaps in incentive mechanisms. They then established these assumptions as market normative behavior. Why wouldn't builders break apart bundles and take MEV instead? Why wouldn't builders include reverted bundles? Because violating these market norms would cause searchers to stop using the relays. We believe that letting the game iterate is more profitable than ending the game and directly seizing the MEV from bundles. I am uncertain whether this mechanism can support Spike MEV and vertical integration in the long run.

The incompatibility of auction incentives can be safely ignored for now. The current market norms are sufficient. Mechanism design has bowed to market pressures. The compression of profit margins is the only relevant issue in today's MEV supply chain. Since proposers have exclusive rights to choose blocks, builders must compete on price. They are forced to give up an increasing share of MEV to proposers and obtain more shares from searchers for payment. As proposers capture the majority of the share, the profit margins for MEV have collapsed.

Fundamentally, proposers extract rent. Builders and searchers cannot buy from others or refuse to buy. Proposers are granted exclusive rights by the protocol to choose the next block and can exercise this right without supervision. To paraphrase a rather famous politician, Rod R. Blagojevich, "Proposers have something as valuable as gold. They won't give it up for nothing." Proposers are granted significant power by the protocol, which is not a moral or natural right. It is granted by the rules of the protocol and can be artificially changed. In any case, proposers are rewarded for extending the blockchain, and then builders pay proposers to extend the blockchain in a specific way, effectively receiving two rewards.

Before MEV extraction, ordering played a less significant role; inflationary block subsidies were the primary incentive. Therefore, rights were freely distributed. This is why proposers did not pay for this right. It had no value in the past. But now, ordering includes rights to a significant amount of MEV, and this asymmetric power dynamic distorts the MEV supply chain. Research in this area is ongoing. In the future, we may see protocol-internal mechanisms proposed to address this issue.

Sunny advocates for encrypting ongoing transactions. Encrypted transactions can prevent proposers from knowing the MEV value before transactions are ordered. Personally, I am interested in points including designing MEV into fork choice rules, selling ordering rights to the highest bidder, and retaining most of the extracted value for protocol redistribution.

MEV: The First Five Years

So, where do we go from here? Five years ago, MEV originated from curiosity. It was the result of researchers having a few drinks after discussing serious matters at dinner. MEV awaited the right combination of DeFi activity and practical experience. As we entered the bull market of 2021, Flashbots rode the wave of MEV research.

Now, we hold MEV roundtables, bringing searchers and developers together to discuss protocol design related to MEV. The market structure has determined the configuration of these three roles and is beginning to test its limits. While people still respect the ideals of Pirate Ship, extraction on Ethereum has shown signs of specialization. In the future, MEV will not be defined by any spirit or research. The story of MEV will be written by profit margins.

Throughout its history, people (including myself) have viewed MEV as either dangerous or evil. The word "theft" has been jokingly mentioned more than a few times. But in recent days, I have begun to think that this value judgment is unreasonable. MEV is a fact. No amount of philosophy, research, or mechanical combinations will change that. This is why I am pleased to witness the emergence of a robust MEV extraction supply chain. Specialized MEV is predictable MEV; predictable MEV is useful MEV. We are not clever enough to plan the market in advance, so let’s make the most of everything we can utilize.