Collective Intelligence in the Security Domain: Community-Driven Bounty and Audit Markets

Author: Ray, IOSG Ventures

Introduction

Blockchain, as a large-scale computer system, has become significantly more complex than it was five years ago. The modularity of the infrastructure has become more refined, the logic of smart contracts at the application layer has become increasingly rich, and interactions between contracts are very frequent. More importantly, blockchain systems now host a vast amount of assets. Therefore, discussions about security cycles have recently increased within the blockchain security community (similar to 2017, when discussions about security primarily revolved around developers writing contracts and showing them to friends at the Ethereum Foundation for basic testing).

Throughout the security lifecycle of blockchain programs (from testing, inviting third-party audits to post-monitoring and updating audits), the bug bounty community acts as a safety net, attracting white hats to conduct a final review of the project’s code through game theory and collaborative work. Some smart contract security workers feel that bug bounties are more like the last line of defense, but I believe that bug bounties and audit competitions have the potential to play a greater role in enhancing the overall security of the system throughout the entire security lifecycle.

Of course, there are also bug bounty programs in the traditional cybersecurity field (Bug Bounty or Vulnerability Rewards). Major tech companies like Facebook, Google, and Microsoft deploy bounty programs for their in-house security teams and product lines. Additionally, around 2015, third-party bug bounty platforms represented by HackerOne and Bugcrowd emerged. Currently, these two leading security companies can generate annual revenues of nearly $50 million and $20 million, respectively, by taking a commission from the bounties they distribute. In the blockchain world, bounties are a more interesting topic frequently discussed in the security circle, primarily because the open-source nature of blockchain code lowers the cost for hackers to attack and enhance their attack strategies. Coupled with the crypto world’s strong advocacy for collaborative work and open contribution models in creator and ownership economies, this creates a more valuable open white hat economy model.

What are bug bounties and audit competitions? Why do we need them?

Security is a dynamic game between attackers and defenders. As computer security expert and cryptographer Bruce Schneier said, "Security is a process, not a product. It is a mindset that must permeate every aspect of the software development process." In the dark forest of the blockchain world, where all code is open-source and transparent, a blockchain project that wants to survive long-term will inevitably have an eternal demand for the security of its products/contracts. Most blockchain products have some financial attributes, and the most important asset in finance is trust, which users only grant once.

So, what are the shortcomings and issues of traditional audits? What advantages do community-driven bug bounties and audit competitions have to address these issues?

Developers using audit services often find that:

• Even after purchasing services from third-party audit companies, issues can still arise in the code post-audit. Although the reasons for these issues vary (technical and non-technical), it ultimately indicates that relying on a single audit company may not be entirely reliable. The quality of code audits still depends on the auditor's level, and clients often lack the ability to discern "who is better."
• Bug bounty platforms and audit competitions serve as a more open "sandbox," allowing white hats to freely review project code, with no restrictions on background (participants may come from professional audit companies or be freelance security analysts), and no restrictions on tools. Clients only need to set reasonable bounties and pay white hats for their contributions when they find issues.
• Typically, clients will first submit the code they need white hats to review, define the security level of vulnerabilities (usually related to potential economic losses, with vulnerabilities that can easily lead to economic losses being rated more severely), set a bounty budget, and outline the scope of the code to be tested, even specifying testing steps.

What is the market size?

The business model of bounty platforms and audit competitions typically involves taking a portion of the bounties paid by clients or the total bounty pool as the platform's service fee. Clients (project parties) in need of code security audits will publish their plans on bounty platforms based on their needs (which code needs to be audited, how to define the severity of vulnerabilities, and how much bounty they are willing to pay), while white hats will look for vulnerabilities based on the project party's requirements. Once a vulnerability is found by a white hat and meets the project party's needs, the bounty will be awarded to the white hats, and the bounty platform will take a cut as a service fee.

In the traditional Web2 cybersecurity field, bug bounty platforms are also a relatively young direction (emerging after 2012). Currently, the largest bug bounty platforms are HackerOne and Bugcrowd. In 2022, HackerOne's annual revenue reached $58 million, with a company valuation of around $500 million, having historically paid out $230 million in bounties (with $150 million paid out in 2021 and 2022), discovering over 65,000 software vulnerabilities, and having over 1 million registered hackers, with more than 1,000 clients using HackerOne services each month. Its competitor Bugcrowd, on the other hand, exceeded $20 million in revenue in 2022.

In the Web3 security field, in 2022, all web3 bug bounty and audit competition platforms collectively awarded $50 million in bounties to white hat hackers, with the average fee level for such platforms around 10% to 30%. Therefore, conservatively estimating the current market size is around $5 million to $15 million, still a very emerging market.

Another interesting point is that an increasing number of clients are willing to directly use code audit services provided by decentralized security communities. The most famous example is Opensea, which, before launching their new platform Seaport, did not directly seek third-party audit companies as usual, but instead chose the currently largest decentralized audit competition platform Code4Rena and set up a $1 million bounty pool. In today's increasingly competitive traditional security audit market (competing for human resources, technical tools, and market BD), could decentralized security services be an important increment for this market? (Currently, there are 56 audit companies in the market, with leading companies having annual revenues ranging from $10 million to $40 million. I believe the potential for the decentralized security market is very large).

Bug Bounty Platforms vs. Audit Competition Platforms

Although bug bounty platforms have a decade-long development history in Web2, audit competition platforms are a fresh concept native to Web3. The target audience for audit competition services is project parties that are about to launch products or certain new features, leveraging the power of decentralized communities to help them complete audit services within a specific timeframe (more than two weeks). From this perspective, audit competitions pose a significant commercial threat to traditional audit companies.

Below, I will illustrate the differences between these two types of platforms in terms of participation methods, reward structures, and testing coverage.

Participation Methods

Bug bounty platforms (like Immunefi) typically feature open projects, allowing anyone to participate. Participants usually explore independently and report vulnerabilities in exchange for rewards. If two people discover the same vulnerability, the first to submit the report receives the reward based on a first-come, first-served principle.

Community-driven audit competition platforms (such as Code4rena and Sherlock) usually have time constraints, with participants competing to find and report vulnerabilities within a specific timeframe. Compared to bounty platforms, there is some team collaboration (for example, each project will have a clearly assigned Lead Senior Auditor and Lead Judge who will review and summarize all audit results into an audit report submitted to the client, and these two leaders also follow the decentralized principles of community election and competition). Additionally, if two audit competitors discover duplicate vulnerabilities within the specified time, both can receive rewards.

Reward Structures

The rewards actually issued by both will primarily consider the severity of the discovered vulnerabilities.

The only difference is that community-driven audit competition platforms like Code4Rena allocate a portion of each project's bounty pool (5% to 10%) to the Lead Senior Auditor and Lead Judge, as they essentially take on the role of project leaders in traditional audit companies.

Another interesting point is that project parties on bug bounty platforms sometimes offer project tokens as rewards, but I have also seen some white hat hackers in the community prefer to receive stablecoins like USDC or USDT rather than project tokens that fluctuate in price.

Scope and Focus

Bug bounty platform projects typically have a broad scope, while projects on audit competitions usually have a more focused scope, targeting specific functions or aspects of the software, requiring white hats to concentrate their efforts within a shorter timeframe.

Projects Focused on Audit Competitions

Code4Rena - A community-driven audit competition platform similar to esports

Code4Rena has three types of roles:

1. Auditors (Wardens) review the code. Anyone from professional security engineers to novice developers seeking more experience can register as auditors to participate in public audit competitions.

2. Judges are typically the best engineers in the C4 community. They determine the severity, validity, and quality of vulnerabilities and assess the performance of auditors.

3. Sponsors are project parties, such as Opensea, Blur, ENS, Chainlink, etc., who create bounty pools to attract auditors to audit their project code. Sponsors can also choose to hold private competitions by invitation only to enhance privacy.

The most interesting aspect is the culture that Code4Rena is building: encouraging collaboration and teamwork. Unlike traditional bug bounty programs, Code4Rena pays all auditors who report valid vulnerabilities, meaning that the vulnerability has already been reported. This method encourages healthy competition among auditors, as they are motivated to find high-severity and common vulnerabilities. On this platform, some auditor groups may form temporary teams to collaboratively search for vulnerabilities.

Business Model:

Any project can initiate an audit competition program on Code4rena and provide USDC or ETH to establish a base bounty pool (usually ranging from $40,000 to $100,000). Code4rena will take 20% from the base bounty pool as service revenue for organizing the competition, providing reviews, and compiling audit reports. Project parties can also offer project tokens on top of the base bounty pool to set up additional pools, from which Code4rena will take a 40% fee.

Sherlock - Community-driven audits with smart contract insurance coverage

Similar to Code4rena, Sherlock also has auditors, sponsors, and judges, but its uniqueness lies in the insurance services provided by the platform. Anyone can invest in the insurance pool on the Sherlock platform, depositing USDC into the insurance pool, allowing protocol clients to purchase services to hedge against the risk of smart contracts being hacked. Insurance investors' income sources include: premiums paid by protocol clients + interest earned from depositing insurance pool funds into other DeFi pools (like Aave, Compound, etc.) + Sherlock token incentives. However, investors bear the risk of repaying policies while earning returns.

Another point that differs from Code4rena is the distribution mechanism of audit service revenues provided by the platform. Compared to Code4rena, Sherlock has rules that allow the Chief Senior Security Auditor and Chief Judge to receive fixed amounts (5% to 10%) from the bounty pool to appropriately compensate and incentivize full-time senior auditors. Additionally, there are selection and competition systems to elect leadership roles.

How to Build a Hacker Community? What Do Web3 White Hats Care About?

After observing different decentralized security communities (ImmuneFi, Hats Finance, Code4Rena, Sherlock, etc.) and discussing with some security entrepreneurs, we believe that all decentralized platforms aim to build a healthier and more efficient communication and collaboration platform. Bounty platforms act as a marketplace between hackers and projects, considering their needs from the hacker's perspective (as shown in the table below), while also considering what project parties care about most (audit quality).

Source: "Bug Hunters' Perspectives on the Challenges and Benefits of the Bug Bounty Ecosystem"

In addition to some common needs, I also observed some interesting topics in the Immunefi white hat community (the most active white hat Discord community).

For example:

A white hat named Rappie wanted to publicly disclose some vulnerabilities he/she had previously discovered in projects and inquired about any community rules that needed to be followed. (1. Only disclose vulnerabilities that have been fixed. 2. Ensure that any public information does not negatively impact the protocol or its users. For example, after they have fixed your SQL injection vulnerability, do not publish information about their entire database. 3. Ensure that you need to DM the project team before making it public).

A white hat named Noam Yakov questioned the definition of a bounty project (this often happens because usually only serious security vulnerabilities can earn bounties, and how project parties define the severity of vulnerabilities is something white hats are very concerned about, leading to frequent disputes in the community). He expressed confusion over the classification of MEV impact as a serious security vulnerability in the Uniwhales bounty project, and ultimately, the discussion concluded that such descriptions do not apply to all MEV situations. For instance, cases where toxic order flow can drain the protocol's pool assets would certainly be considered serious security incidents (thus, defining a set of security level frameworks is often insufficient; typically, roles similar to arbitrators in the platform need to be involved in different actual cases).

Regarding the interesting topic of "What are your demands and expectations for bounty platforms like Immunefi?" a white hat named ckksec provided his answer: 1) Help clarify the legal aspects of income for these anonymous crypto white hats, such as issuing invoices. 2) The platform should not only have a rating system for white hats but also rate the quality of projects, as white hats often need to spend time discerning the quality of projects. 3) For white hats willing to open their profiles, the platform can showcase their workflows, and it would also be best if the platform could transparently display the security analysis report information received by project parties.

What Tools Can Help White Hats?

With the rise of LLMs like GPT, I have recently heard frequent discussions about whether security audits can also be replaced by AI. Experienced security professionals I have spoken with generally believe that GPT is unlikely to directly replace human intelligence. Some low-hanging fruit (easily discoverable issues) may be detected by language models, but medium to high-risk issues still require expert involvement. For instance, according to feedback from a seasoned security expert, more complex tests like data analysis and dynamic analysis require human involvement to specifically combine the protocol's actual business logic for security analysis testing and to define expected target attributes in advance. The most challenging part is writing good attributes and defining the correct testing domains. Based on their experiments with GPT, they believe that GPT cannot fully replace humans at this stage.

Of course, there are also optimistic results showing that LLMs can greatly enhance the analysis efficiency of security analysis tools and reduce false positive rates https://twitter.com/HatforceSec/status/1671758690808913922

https://www.researchgate.net/publication/371758506Doyoustillneedamanualsmartcontract_audit.

From another interesting non-technical perspective on this topic, the dynamic game between security attackers and defenders is ever-evolving; could AI also provide assistance to security attackers?

Human-Centric Security

People tend to think of software as cold, mechanical, and logical entities, believing that enhancing system security only requires improving analytical techniques and system defense levels. However, people lack the perspective of considering security issues from economic incentives and human nature. In the dark forest of open-source code, we need a distribution system that aligns with rational human assumptions to build positive economic incentives that attract more individuals willing to contribute their wisdom to the security of blockchain systems over the long term.

Currently, the traditional security audit market is stable, with brand reputation being the most important intangible asset for companies in this field. Over time, the influence of leading security brands and the trust of clients steadily increase. However, traditional security audits also have their own issues (a relatively singular business model reliant on human resources, making it difficult to scale growth, and leading companies needing to balance growth and audit quality; some companies have already encountered such bottlenecks, even affecting their brand value).

Community-driven security audit competitions represent an innovative business model. Currently, the two major platforms have exceeded 300 clients and have gradually found product-market fit (PMF). Bounty platforms serve as a good complement to the security lifecycle. Although these decentralized platforms have yet to find particularly effective token models, we are very optimistic about the potential for scaled growth in this market (as collective intelligence aligns well with the attack-defense scenarios in the security market).

Will community-driven audit platforms pose a threat to centralized audit companies? We believe they will have a mutually beneficial competitive and complementary relationship. In the short term, as platforms like Code4rena form a certain network effect and demonstrate a good track record (with a low percentage of audited projects being hacked), they may indeed create competitive pressure for some mid-tier centralized companies. However, in the long term, this may also compel centralized audit platforms to form commercial collaborations with community-driven platforms, as this can broaden the client base of centralized security audit platforms and enhance audit quality (similar to how previously independent security bounty projects operated by large Web2 companies later formed collaborations with third-party platforms like HackerOne).

Although community-driven security platforms aim for a more DAO-like direction (Forta can also be categorized here), they still face challenges in actual project operations, such as how to make workflow and economic distribution processes more transparent and public, how to balance project parties' considerations of privacy and security, how to clearly define the relationship between team collaboration and individual contributions, and how to resolve disputes over interests in a relatively fair and professional manner. These are all challenges that security DAOs need to confront.

Reference:

1. "HackerOne Year Book"

2. "Bounty Everything - Hackers and the Making of the Global Bug Marketplace"

3. "An empirical study of vulnerability rewards programs"

4. "The 2022 Hacker Report"

5. "Productivity and Patterns of Activity in Bug Bounty Programs"

6. https://immunefi.com

7. https://bugrap.io/

8. https://hackenproof.com/

9. https://hats.finance/

10. https://code4rena.com/

11. https://www.sherlock.xyz/