Detailed Explanation of the CESS Mechanism (3): Storage, Content Distribution Network, and Multi-Replica Recoverable Storage Proof

In the previous article, we detailed the design philosophy of CESS in the consensus mechanism and blockchain layer: ensuring fairness and efficiency in consensus through a randomly selected rotating consensus node mechanism (R²S), while avoiding excessive influence of storage miners. The blockchain itself, in addition to basic information such as transactions, proofs, and contracts, also contains metadata of user-uploaded content, making it one of the few public chains that implement on-chain recording of metadata. This is thanks to CESS's efficient on-chain transaction processing capabilities, which make it possible to handle data in a more decentralized manner.

This article will focus on the design of CESS in the storage and content distribution layers, as well as CESS's unique multi-replica recoverable storage proof mechanism. We will separately present the mining mechanism and distribution rules related to storage mining, which include consensus, storage, and content distribution.

1. Storage Network

CESS currently combines the storage layer and content distribution layer into one network, but we will still look at the functions of both separately.

As the most important functional carrier of a decentralized storage network, the storage network is the "heart" of the entire CESS network. The CESS storage network consists of storage miners, addressing the issue that current decentralized storage cannot provide elastic and scalable cloud storage capabilities. The CESS storage network pools storage resources through virtualization technology, providing a unified, on-demand storage method for top-level or external applications, shielding the instability caused by differences in underlying hardware. Miners provide effective storage space and available storage space to the entire CESS network through a series of on-chain proofs, including valid storage space, amount of stored data, traffic contribution, and operational performance. CESS does not facilitate point-to-point transactions by matching demand-side and storage service providers, but rather integrates the storage resources provided by storage miners and allocates demand according to required capacity, bandwidth, etc., to the corresponding storage service providers (i.e., virtualization technology, which abstracts the specific services provided by miners into "virtual" storage services), meaning that when users have data storage needs, it provides storage services for applications.

In terms of resource utilization, CESS maximizes resource efficiency through "pooling" technology. "Pooling" means treating all storage resources as a whole storage resource pool rather than individual miners, with user-uploaded data being randomly allocated to storage miners that meet the storage conditions.

Specifically, when users upload data, consensus nodes first encrypt the data (using trusted execution environments for encryption), shard it, and perform redundancy and other preprocessing (decentralized proxy re-encryption mechanism). The processed data will select miners that meet the conditions for storage based on the user's storage requirements (e.g., storage time, etc.). Importantly, CESS does not choose one or a few miners to complete the storage task, but randomly distributes the sharded data segments to miners that meet the requirements. This avoids the potential monopoly situation that may arise in Filecoin's point-to-point storage model.

On the other hand, this model also maximizes the utilization of storage resources. In existing storage networks, when receiving large-scale data storage tasks (e.g., data volumes exceeding 5TB), it may be impossible for some home miners to achieve overall storage, thus losing their competitive ability. This issue has not been well addressed in most existing storage networks, and as storage networks develop to a certain extent, Filecoin and Arweave will inevitably move towards storage centralization. For example, when there are miner A (3TB storage capacity) and miner B (1TB storage capacity) in the network, for existing storage networks, 2TB of data can only be stored by miner A, while CESS can achieve 1TB storage by both A and B, thus maximizing utilization.

In addition to improving utilization, this model also lowers the hardware threshold for storage facilities. On one hand, miners only need to perform storage tasks without engaging in complex and specialized matters such as "taking orders" and running nodes; on the other hand, miners will randomly receive data segments, which does not depend on the scale of the miners themselves.

In this way, CESS's storage layer truly realizes the vision of decentralized storage and maximized usage efficiency, effectively utilizing idle resources rather than simply increasing storage resources for profit.

2. Content Distribution Network

The content distribution network in CESS actually serves the function of a traditional CDN in the cloud.

For decentralized storage networks, one of the biggest problems is not the storage itself, but "data uploading." For miners, especially domestic miners, downloading the data that users need to store is not difficult, but the network cost required for uploading when users need to use it is relatively high. This leads to many miners being unwilling to upload data to users after ensuring data persistence through storage proofs, resulting in actual network unavailability.

Not only for decentralized storage networks, but even for traditional clouds, they cannot bear the high instantaneous concurrency and traffic generated when users directly retrieve data from cloud data centers, which is one of the necessities for the existence of CDNs.
CESS has designed caching and retrieval miners in the content distribution network to help the network operate more efficiently, where caching miners cache popular data to achieve faster access speeds, while retrieval miners help applications quickly locate the required data.

3. Multi-Replica Recoverable Storage Proof (PoDR²)

The data preprocessing and distribution to miners mentioned above belong to CESS's innovative multi-replica recoverable storage proof (PoDR²) mechanism. The multi-replica recoverable storage proof (PoDR²) mechanism ensures that the CESS platform effectively stores the customized replicas of user-uploaded data. Once any data is uploaded to the CESS system, several data replicas (default three, customizable) will be automatically created, and auxiliary verification metadata necessary for generating recoverable proofs will be created for each data replica, which will be stored in the blockchain system.

At this point, CESS can distribute the processed data to various storage miners, and within an effective period, miners need to report the data they store, allowing the CESS system to confirm whether the data is damaged.

It is worth mentioning that the PoDR² mechanism will statistically and monitor all data segments that make up a single file (including all replicas) as a whole. Once a certain data segment is identified as damaged, CESS will automatically generate a new data segment as a supplement and send it to a new storage miner, ensuring the recoverability of replicas and enhancing the robustness of the system's data storage.

It is evident that the biggest difference in CESS's storage mechanism lies in: compared to designs that rely on miners to perform redundancy and other protective operations on data, CESS implements encryption, redundancy, and other protective strategies at the system's underlying level, with miners only needing to store the processed data segments and ensure the validity of the storage. Furthermore, even if some miners lose data, the system can restore the original data through other data segments. This greatly reduces the possibility of single points of failure and enhances the security of data in decentralized storage networks.

4. Conclusion

In the design of the storage mechanism, CESS, on one hand, protects data privacy and security through the multi-replica recoverable storage proof (PoDR²) mechanism and has designed disaster recovery measures; on the other hand, it maximizes the utilization of storage resources through "pooling" and addresses the ability to truly call "limited resources," making outstanding contributions to the innovation and progress of decentralized storage in terms of mechanism and design philosophy.