Detailed Explanation of FWS: Filecoin's Open Source Alternative to AWS

Original Title: Filecoin Launches FWS (Filecoin Web Service)

Author: Filecoin Network

For our engineering team, this is an important release day.

The engineering team in the ecosystem has reached a significant milestone with the Filecoin Web Services project—the release of FWS means that open-source alternatives are beginning to provide alternatives to services like AWS, GCP, and Azure. This platform leverages the potential of Filecoin, the world's largest decentralized storage network.

The announcement marks an important step towards providing practical cloud service alternatives to developers worldwide.

What is FWS?

FWS (Filecoin Web Services) is a suite of computing and storage technologies built on the Filecoin network. The Filecoin network is a decentralized network with built-in revenue generation and incentive mechanisms. We recognize that cloud services provide a great developer experience, so one of our ambitions is to offer a cloud-level developer experience on top of the transparent, open-source incentive layer that a decentralized network can provide.

This tech stack encompasses various components, such as computing, storage, and network resources, which developers can use to build decentralized applications. The components offered by Filecoin Web Services are designed for scalability, flexibility, and security from the outset, making them suitable for a wide range of use cases.

FWS Product Line

Estuary

https://estuary.tech

Estuary consists of various components, such as the Estuary Node, which is an easy-to-deploy and manage Filecoin storage node. Additionally, the Estuary Dashboard provides a user-friendly interface for monitoring and managing storage deal proposals on this network. Estuary also offers various developer tools and APIs (such as the Estuary API) that allow developers to build storage applications on Filecoin. Furthermore, there is the Estuary SDK, which provides a range of libraries and tools for integration needs with this network.

Delta Technology Stack

https://delta.store

The Delta Technology Stack is an open-source toolkit for creating and managing storage deal proposals on the Filecoin network. This includes the Deltac storage deal proposal negotiation engine (Deal Making Engine), the Delta Dataset Manager for large datasets, and the Edge-UR Gateway for providing storage and external access services for unsealed copies.

In-depth on Delta

Delta simplifies the process of preparing and managing storage deal proposals on the Filecoin network. Delta can be used to manage data (including version management, metadata management, and tracking the status of storage deal proposals), making it easier for users to publish their data on the Filecoin network.

The storage deal proposal negotiation engine allows users to specify the terms of the storage deal proposal, such as the required storage space, price, and transaction conditions. Subsequently, this platform will use smart contracts on the Filecoin network to enforce the terms of the storage deal proposal and facilitate the transaction.

Delta encapsulates the complex process of preparing and executing storage deal proposals on the Filecoin network, providing a seamless experience for users who want to store their data copies on the Filecoin decentralized storage network.

Features:

• Preparation and announcement of storage deal proposal plans

• Storage piece submission computation and optimization

• E2E/import support (online/offline)

• Automatic/manual repair and update of storage deal proposals

• Wallet allocation

• Storage provider allocation and selection

• Blockstore cleanup

• Storage deal proposal status management

• Copy management

Delta Dataset Manager

The Delta Dataset Manager (Delta DM) tool provides storage deal proposal copy management tracking services for datasets uploaded to the Filecoin network through out-of-band processes. This offers a solution for quickly reaching storage deal proposals for large amounts of data, as such scenarios are best accelerated using out-of-band processes.

The Delta DM platform provides an intuitive web interface that simplifies the process of managing datasets. It tracks copies and DataCap quotas, offering self-service to Client Service Providers through a storage deal proposal pull mechanism.

This tool can operate independently from the data preparation and data transfer processes and can run outside of Boost and Lotus, allowing Delta DM + Delta to function as an independent solution.

Delta Importer

The Delta Importer is a standalone binary program designed to run on top of a storage provider's infrastructure. It automates the process of importing CAR files from the file system into Boost. By integrating with Delta DM to request storage deal proposals from the self-service API, the Delta Importer facilitates an automated pipeline for storage deal negotiation and proposal inclusion. Delta DM has multiple operational modes covering a range of different data onboarding strategies.

Upcoming:

• Automatic repair of expired storage deal proposals

• Retrieval-SP------specifying a single location for unsealed copies.

• SP Metadata------detailed copy parameters

Edge-UR

Edge-UR is a dedicated node that allows users to upload and retrieve content on Estuary using the same API key. Each user is assigned their own node, which can seamlessly switch upload protocols. While data still needs to be uploaded via HTTP, Edge-UR nodes can help transfer files to Delta nodes to facilitate storage deal proposals.

Overall, Edge-UR simplifies the process of uploading and retrieving data to Estuary while ensuring the security and control of user content.

Features:

• Lightweight IPFS node that can connect with IPFS nodes.

• Upload and retrieve------users can upload data and immediately retrieve it afterward.

• Achieve asynchronous storage deal proposals using Delta.

• Track content and its related storage deal proposals.

Ptolemy

Ptolemy is designed as an orchestration engine to handle large datasets that are typically confined to S3 buckets or similar file systems. Ptolemy can encapsulate the entire file system architecture within CAR files for achieving storage deal proposals. The goal of Ptolemy is to remove the cumbersome data preparation work for storage providers and end users.

FWI

Filecoin Web Infrastructure (FWI) marks an important step towards decentralization. It serves as a blueprint for the storage deal proposal achievement engine and other Filecoin technologies and services, depicting a low-cost, reliable computing and storage solution that pushes the Filecoin network forward. It is a scalable computing cluster storage system and visualization cluster that includes high availability design and Kubernetes orchestration tools.

FWI is an open-source automation tool and is provided as infrastructure in code form, allowing you to build your own "FWI-alikes" applications, running your enterprise-level client infrastructure on your own hardware and under your own conditions. It can match different hardware and software, with all components being pluggable and open-source.

FWI also offers many benefits—one of which is enabling Customer Provisioned Infrastructure, allowing engineering teams to run infrastructure on behalf of customers and receive FIL in return—this is a significant breakthrough. Details of this work will be discussed in the coming weeks.

Storage Market

https://storage.market/

Storage.market serves as an information hub for data storage products, providing users with analytical data and the latest news related to the storage market. This service is particularly useful for businesses and individuals who want to stay informed about industry trends and make informed decisions regarding storage products and services.

User Onboarding

To upload data to the Filecoin network, users need to use specific tools and applications that allow them to interact with the network and perform different operational processes (such as uploading, retrieving, and managing data). This is the segment that the Filecoin ecosystem aims to serve: our engineering team is working with several clients who are using our team's expertise and tools to build solutions for uploading data to the Filecoin network.

Collaborating with multiple clients on these projects is challenging. Each client has their own needs, use cases, and varying technical capabilities. However, through our understanding of the Filecoin network architecture, protocols, and features, as well as our insight into client needs and expectations, we can develop customized solutions that meet their requirements and help them leverage the benefits of decentralized storage.

Collaboration with Radiant Earth

Radiant Earth is an innovative organization aimed at driving positive global impact through the use of machine learning and Earth observation (EO) technologies. They provide a range of high-quality open resources that are trusted by professionals and researchers worldwide.

One of Radiant Earth's core missions is to cultivate a community of practice to develop relevant standards for machine learning in Earth observation tasks. This involves gathering experts from machine learning, Earth observation, and related fields to collaborate, share knowledge, and develop best practice standards for the responsible and efficient use of these technologies.

As part of our collaboration with Radiant Earth, we uploaded a small portion of their machine learning and Earth observation data to the Filecoin network using our proprietary tech stack. We understood the necessary steps to accomplish this task and ensured that their data was properly formatted, securely stored, and easily accessible through our tools.

Tools we used:

• Ptolemy

• Delta Dataset Manager

• Delta Deal Making Engine

• Delta Importer

Learn more about Radiant Earth at https://mlhub.earth/

Collaboration with SUCHO

The outstanding SUCHO initiative is driven by 1,500 volunteers worldwide who are collaborating online to digitize and preserve Ukraine's cultural heritage. Amid ongoing conflict, SUCHO has taken action to preserve over 5,000 websites and ~40TB of data belonging to Ukrainian cultural institutions, utilizing five different storage providers globally to prevent these valuable resources from being lost or taken offline.

Tools we used:

• Ptolemy

• Delta Dataset Manager

• Delta Deal Making Engine

• Delta Importer

Learn more about SUCHO at https://www.sucho.org/

Collaboration with Encloud

Encloud focuses on data security and recognizes the importance of using Filecoin to further enhance its security features while providing users with free access.

We are collaborating with Encloud in this regard by opening our online Edge-UR nodes to them for testing their proof-of-concept model.

Tools we used:

• Delta

• Edge-UR