CESS's storage pallet developed for Substrate has been completed, and teams using Substrate for their projects can directly call CESS's storage services after the CESS mainnet goes live. Let's review the thoughts of CESS's R&D Director Swowk on why the storage pallet was developed.

1. Background

As a multifunctional blockchain framework, Substrate offers multiple modules (also known as pallets) for developers to use. From resource management such as accounts and assets to utilities like random number generators and schedulers, these existing pallets can meet the needs of most developer application scenarios. However, there is still room for improvement.

Recently, we had a requirement to implement a data storage service on Substrate, but after examining all existing pallets, we did not find one that met our needs. Therefore, we wanted to develop a custom pallet to address this issue.

We are not discussing niche things here; in reality, applications continuously consume and generate various data during their operation, whether system, user, or temporary data, which is a common scenario. Many DApps have numerous scenarios that require off-chain data storage services, such as NFTs. The quality of the chosen storage service will directly impact the performance and reliability of the entire application.

Thus, we hope to provide a pallet specifically designed for storage services compatible with the current Substrate API for the Substrate/Polkadot community, allowing developers to leverage CESS's stable and secure data storage with minimal code changes. We believe this will further enhance the development experience when using Substrate and enrich the Polkadot ecosystem.

2. Deficiencies of Current Solutions

Currently, there is only one pallet related to data storage in the existing Substrate FRAME, namely the Transaction Storage Pallet. It supports running IPFS nodes in parallel with Substrate and allows data to be retrieved after being placed into Substrate storage. However, due to its inherent characteristics and several deficiencies, its application scope is greatly limited:

• Data needs to be uploaded to the blockchain network. Although this data is not actually stored on-chain, it still incurs additional gas costs and congestion, making it unsuitable for large file storage;
• All validator nodes need to establish their own IPFS services, which is subject to many restrictions;
• High development difficulty, requiring significant modifications to Substrate-based code;
• This pallet only supports file uploads on the Substrate side, and viewers need to retrieve it through an IPFS client.

3. CESS's New Solution

We have designed and implemented a data storage service based on Substrate. On one hand, validator nodes do not need to start additional services, nor do they need to make significant modifications to Substrate-based code. Therefore, whether for new chains or existing chains, developers can easily integrate our storage service.

On the other hand, by customizing the storage REST component, users can conveniently upload and download data without needing to install additional client programs.

4. Design Architecture

Our proposed architecture is shown in the diagram below, consisting of the Data Storage Pallet and a custom Storage Sidecar, inspired by the Substrate API Sidecar (https://github.com/paritytech/substrate-api-sidecar).

Data Storage Pallet: Implements the recording and management of stored data. This pallet implements functionalities related to metadata, such as root data management, data owner management, and data classification regarding stored data.

Custom Storage Sidecar: Provides RESTful services to interact with the Data Storage Pallet. Unlike the Substrate API Sidecar, the Storage Sidecar not only encapsulates the basic functionalities for interacting with Substrate-based blockchains but also includes storage-related APIs, including data storage and data retrieval. The data transmitted by users will ultimately be stored in the CESS storage system through this interface.

5. Core Functions

Data Storage

• Users call the data storage API of the custom Storage Sidecar to upload data files;
• Calls the encapsulated CESS API to forward the data to CESS;
• Once the data is confirmed to be written, the custom Storage Sidecar will call Extrinsic to record the relevant information of the on-chain data file;
• The CESS storage system maintains the integrity and privacy of the data throughout its entire lifecycle.

Data Retrieval

• Users call the storage API of the custom Storage Sidecar to obtain the target data;
• Using the custom Storage Sidecar, query the on-chain data routing information;
• Use the routing information to call the CESS data retrieval API;
• Retrieve and return the target data from the CESS storage system;
• Return the target data to the custom Storage Sidecar;
• The custom Storage Sidecar updates on-chain information if necessary;
• Return the target data to the user.

6. Usage Instructions

In fact, developers can experience CESS storage services in just two steps.

• Step 1 Integrate the Data Store Pallet into FRAME

Open the runtime/Cargo.toml configuration file from your code in a text editor;

Import the pallet-data-store crate by adding it to the dependencies list to make it available for runtime;

pallet-data-store = { default-features = false, git = "https://github.com/paritytech/substrate.git", branch = "main" }

Add the following code to the features module, allowing the pallet-data-store crate to be included in the std feature; >

[features]
default = ['std']
std = [
    ...
    'pallet-aura/std',
    'pallet-balances/std',
    # Add this line
    "pallet-data-store/std",
    ...
]

Open the runtime/src/lib.rs file in a text editor;

After the last line of the Balances code, add the following code for the Data Store pallet:

parameter_types! {
    pub const StringLimit: u32 = 1024;
}
/// Configure the pallet-data-store.
impl pallet_data_store::Config for Runtime { 
    type Event = Event;
    type StringLimit = StringLimit;
    type WeightInfo =      pallet_data_store::weights::DataStoreWeight<Runtime>;
}

Add Data Store to the construct_runtime! macro

// Create the runtime by composing the FRAME pallets that were previously configured.

Compile the node in release mode by running the following command

cargo build --release

• Step 2 Use the custom Storage Sidecar to store data

Once the Runtime FRAME integration is complete, you can now interact with your blockchain network through our Data Store Sidecar (https://github.com/CESSProject/data-store-sidecar) while using CESS storage services. Here’s a tutorial on using CESS storage features in the Sidecar:

First, please complete the installation and startup according to the content in this link (https://github.com/CESSProject/data-store-sidecar#source-code-installation-and-usage).

Of course, we also support quick deployment using Docker (https://github.com/CESSProject/data-store-sidecar#docker).

Once the service is successfully started, you can call it directly. For user guidance, please refer to the API documentation (https://example-datastore.cess.cloud/docs/).