AIxDePIN: What new opportunities will emerge from the collision of this hot track?

Authors: Cynic, Shigeru

With the power of algorithms, computing power, and data, the advancement of AI technology is redefining the boundaries of data processing and intelligent decision-making. Meanwhile, DePIN represents a paradigm shift from centralized infrastructure to decentralized, blockchain-based networks.

As the world accelerates its pace towards digital transformation, AI and DePIN (Decentralized Physical Infrastructure) have become foundational technologies driving change across various industries. The integration of AI and DePIN not only facilitates rapid technological iteration and widespread application but also opens up safer, more transparent, and efficient service models, bringing profound changes to the global economy.

DePIN: The Pillar of the Digital Economy Transitioning from Virtual to Physical

DePIN is an abbreviation for Decentralized Physical Infrastructure. In a narrow sense, DePIN mainly refers to distributed networks of traditional physical infrastructure supported by distributed ledger technology, such as power networks, communication networks, and positioning networks. In a broader sense, any distributed network supported by physical devices can be referred to as DePIN, such as storage networks and computing networks.

from: Messari

If Crypto has brought about a decentralized transformation at the financial level, then DePIN is the decentralized solution in the real economy. It can be said that PoW mining machines are a form of DePIN. From day one, DePIN has been a core pillar of Web3.

The Three Elements of AI—Algorithms, Computing Power, and Data; DePIN Dominates Two

The development of artificial intelligence is generally considered to rely on three key elements: algorithms, computing power, and data. Algorithms refer to the mathematical models and program logic that drive AI systems, computing power refers to the computational resources required to execute these algorithms, and data is the foundation for training and optimizing AI models.

Which of the three elements is the most important? Before the emergence of chatGPT, people generally believed it was algorithms; otherwise, academic conferences and journal papers would not be filled with one algorithm tweak after another. However, when chatGPT and the large language model (LLM) supporting its intelligence appeared, people began to realize the importance of the latter two. Massive computing power is a prerequisite for the birth of models, and the quality and diversity of data are crucial for building robust and efficient AI systems. In contrast, the demands on algorithms are no longer as stringent as before.

In the era of large models, AI has shifted from meticulous refinement to large-scale brute force, with increasing demands for computing power and data, which DePIN can precisely provide. Token incentives can leverage the long-tail market, and vast amounts of consumer-grade computing power and storage will become the best nourishment for large models.

Decentralization of AI is Not Optional, but Essential

Of course, some may ask, computing power and data are available in AWS data centers, and they surpass DePIN in terms of stability and user experience. Why choose DePIN over centralized services?

This argument has its merits; after all, almost all large models today are developed directly or indirectly by large internet companies. ChatGPT is backed by Microsoft, Gemini by Google, and almost every major internet company in China has its own large model. Why? Because only large internet companies have enough high-quality data and substantial financial resources to support computing power. But this is incorrect; people no longer want to be manipulated by internet giants.

On one hand, centralized AI poses risks to data privacy and security and may be subject to censorship and control. On the other hand, AI created by internet giants can further increase dependency and lead to market concentration, raising barriers to innovation.

from: https://www.gensyn.ai/

Humanity should not need a Martin Luther for an AI era; people should have the right to communicate directly with the divine.

From a Business Perspective: Cost Reduction and Efficiency Improvement are Key

Even setting aside the ideological debate between decentralization and centralization, there are still advantages to using DePIN for AI from a business perspective.

First, we need to clearly recognize that although internet giants possess a large amount of high-end GPU resources, the consumer-grade GPUs scattered among the public can also form a considerable computing power network, demonstrating the long-tail effect of computing power. The idle rate of these consumer-grade GPUs is actually very high. As long as the incentives provided by DePIN exceed electricity costs, users will be motivated to contribute computing power to the network. At the same time, all physical facilities are managed by the users themselves, and the DePIN network does not need to bear the operational costs that centralized suppliers cannot avoid, focusing solely on the protocol design itself.

Regarding data, the DePIN network can release the potential usability of data and reduce transmission costs through edge computing and other methods. Additionally, most distributed storage networks have automatic deduplication features, reducing the workload of cleaning AI training data.

Finally, the crypto-economics brought by DePIN enhances the system's fault tolerance, promising a win-win situation for providers, consumers, and platforms.

from: UCLA

In case you don't believe it, a recent study from UCLA shows that using decentralized computing achieves 2.75 times the performance compared to traditional GPU clusters at the same cost, specifically being 1.22 times faster and 4.83 times cheaper.

The Road Ahead: What Challenges Will AIxDePIN Encounter?

We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard.

------John Fitzgerald Kennedy

Building AI models using DePIN's distributed storage and distributed computing without trust still presents many challenges.

Work Verification

Essentially, computing deep learning models and PoW mining are both forms of general computation, with the underlying signals changing between logic gates. From a macro perspective, PoW mining is "useless computation," attempting to derive a hash value with a prefix of n zeros through countless random number generations and hash function calculations; while deep learning computation is "useful computation," calculating the parameter values of each layer in deep learning through forward and backward propagation to build an efficient AI model.

The fact is that "useless computation" like PoW mining uses hash functions, where it is easy to compute the image from the pre-image but hard to compute the pre-image from the image, allowing anyone to easily and quickly verify the validity of the computation. In contrast, for deep learning model computations, due to the hierarchical structure where the output of each layer serves as the input for the next layer, verifying the validity of the computation requires executing all previous work, making it difficult to verify simply and effectively.

from: AWS

Work verification is crucial; otherwise, the provider of the computation could simply submit a randomly generated result without performing any calculations.

One idea is to have different servers perform the same computational task and verify the validity of the work by checking if the results are the same through repeated execution. However, the vast majority of model computations are non-deterministic, meaning that even in identical computing environments, the same results cannot be reproduced, and only similar results can be achieved in a statistical sense. Additionally, repeated computations would lead to rapidly rising costs, which contradicts the key goal of DePIN to reduce costs and improve efficiency.

Another idea is the Optimistic mechanism, which initially assumes that the results are the product of valid computations while allowing anyone to verify the results. If errors are found, a Fraud Proof can be submitted, and the protocol will penalize the fraudster and reward the whistleblower.

Parallelization

As mentioned earlier, DePIN primarily taps into the long-tail consumer-grade computing power market, which means that the computing power provided by a single device is relatively limited. For large AI models, training on a single device would take a very long time, necessitating the use of parallelization to shorten the training time.

The main difficulty in parallelizing deep learning training lies in the dependencies between tasks. This dependency can make parallelization challenging.

Currently, deep learning training parallelization is mainly divided into data parallelism and model parallelism.

Data parallelism refers to distributing data across multiple machines, where each machine holds all the parameters of a model and uses local data for training, ultimately aggregating the parameters from each machine. Data parallelism works well when the data volume is large but requires synchronous communication to aggregate parameters.

Model parallelism is used when the model size is too large to fit into a single machine, allowing the model to be split across multiple machines, with each machine holding a portion of the model's parameters. Communication between different machines is required during forward and backward propagation. Model parallelism has advantages when the model is large, but the communication overhead during forward and backward propagation is significant.

For gradient information between different layers, it can be further divided into synchronous updates and asynchronous updates. Synchronous updates are simple and direct but increase waiting time; asynchronous updates have shorter waiting times but introduce stability issues.

from: Stanford University, Parallel and Distributed Deep Learning

Privacy

There is a global movement to protect personal privacy, with governments strengthening the protection of personal data privacy and security. Although AI extensively uses public datasets, what truly distinguishes different AI models is the proprietary user data of various companies.

How can we benefit from proprietary data during training without exposing privacy? How can we ensure that the parameters of the constructed AI model are not leaked?

These are two aspects of privacy: data privacy protects users, while model privacy protects the organizations that build the models. In the current context, data privacy is far more important than model privacy.

Various solutions are being attempted to address privacy issues. Federated learning trains at the source of the data, keeping data local while transmitting model parameters to ensure data privacy; zero-knowledge proofs may emerge as a new contender.

Case Study: What Quality Projects Are Available in the Market?

Gensyn

Gensyn is a distributed computing network for training AI models. The network uses a layer-1 blockchain based on Polkadot to verify whether deep learning tasks have been executed correctly and triggers payments via commands. Founded in 2020, it disclosed a $43 million Series A funding round in June 2023, led by a16z.

Gensyn constructs certificates for the executed work using metadata from gradient-based optimization processes, and it is executed consistently through multi-granularity, graph-based precise protocols and cross-evaluators to allow for re-running verification work and comparing consistency, ultimately confirmed by the chain itself to ensure the validity of the computation. To further enhance the reliability of work verification, Gensyn introduces staking to create incentives.

There are four types of participants in the system: submitters, solvers, validators, and whistleblowers.

• Submitters are the end users of the system, providing tasks to be computed and paying for completed work units.
• Solvers are the primary workers of the system, executing model training and generating proofs for validators to check.
• Validators are the key that links the non-deterministic training process with deterministic linear computation, replicating part of the solver's proof and comparing the distance with the expected threshold.
• Whistleblowers are the last line of defense, checking the work of validators and raising challenges, receiving rewards upon successful challenges.

Solvers need to stake, and whistleblowers verify the work of solvers. If wrongdoing is found, they can challenge, and upon successful challenges, the tokens staked by the solver are forfeited, and the whistleblower is rewarded.

According to Gensyn's predictions, this solution is expected to reduce training costs to one-fifth of centralized providers.

from: Gensyn

FedML

FedML is a decentralized collaborative machine learning platform for decentralized and collaborative AI anywhere at any scale. More specifically, FedML provides an MLOps ecosystem that can train, deploy, monitor, and continuously improve machine learning models while collaborating on combined data, models, and computing resources in a privacy-preserving manner. Founded in 2022, FedML disclosed a $6 million seed round in March 2023.

FedML consists of two key components: FedML-API and FedML-core, representing high-level APIs and underlying APIs, respectively.

FedML-core includes two independent modules: distributed communication and model training. The communication module is responsible for the underlying communication between different workers/clients, based on MPI; the model training module is based on PyTorch.

FedML-API is built on top of FedML-core. With FedML-core, new distributed algorithms can be easily implemented using client-oriented programming interfaces.

In the latest work by the FedML team, it was demonstrated that using FedML Nexus AI for AI model inference on consumer-grade GPU RTX 4090 is 20 times cheaper and 1.88 times faster than A100.

from: FedML

Future Outlook: DePIN Brings AI Democratization

One day, as AI further evolves into AGI, computing power will become the de facto universal currency, and DePIN will accelerate this process.

The integration of AI and DePIN opens up a new technological growth point, providing tremendous opportunities for the development of artificial intelligence. DePIN offers vast amounts of distributed computing power and data for AI, aiding in the training of larger models and achieving stronger intelligence. At the same time, DePIN also drives AI towards a more open, secure, and reliable direction, reducing dependence on a single centralized infrastructure.

Looking ahead, AI and DePIN will continue to develop in synergy. Distributed networks will provide a strong foundation for training ultra-large models, which will play important roles in the applications of DePIN. While protecting privacy and security, AI will also assist in optimizing DePIN network protocols and algorithms. We look forward to AI and DePIN bringing a more efficient, fair, and trustworthy digital world.