GITBOOK-30: Market docs

documentation/en/SUMMARY.md

@@ -14,7 +14,18 @@
   * [Listen Address](configuration/listen-address.md)
   * [Alert Manager](configuration/alert-manager.md)
   * [Default Curio Configuration](configuration/default-curio-configuration.md)
-* [Enabling market](enabling-market.md)
+* [Enabling market (Boost) (deprecated)](enabling-market.md)
+* [Curio Market](curio-market/README.md)
+  * [Storage Market](curio-market/storage-market.md)
+  * [Deal filters](curio-market/deal-filters.md)
+  * [Curio HTTP Server](curio-market/curio-http-server.md)
+  * [Curio libp2p Server](curio-market/curio-libp2p-server/README.md)
+    * [libp2p Protocols](curio-market/curio-libp2p-server/libp2p-protocols.md)
+  * [Indexing](curio-market/indexing.md)
+  * [IPNI (Interplanetary Network Indexer) Provider](curio-market/ipni-interplanetary-network-indexer-provider.md)
+  * [Market UI](curio-market/market-ui.md)
+  * [Retrievals](curio-market/retrievals.md)
+  * [Migrating From Boost](curio-market/migrating-from-boost.md)
 * [Snap Deals](snap-deals.md)
 * [Batch Sealing with SupraSeal](supraseal.md)
 * [Scaling Curio cluster](scaling-curio-cluster.md)

documentation/en/curio-market/README.md

@@ -0,0 +1,47 @@
+---
+description: An overview of the Curio market module
+---
+
+# Curio Market
+
+The Curio Storage Market is a comprehensive framework designed to manage Filecoin storage deals using a flexible and scalable architecture that supports multiple protocols, currently including MK1.2, with future protocols planned for seamless integration. The market handles deal-making through both online and offline processes, providing efficient tools for each step of the deal lifecycle.
+
+* **Protocols:** The market is designed to support multiple deal protocols. Currently, MK1.2 is implemented, but the system can easily expand to accommodate future protocols without major reconfiguration.
+* **Deal Flow:** The market manages both online and offline deals, ensuring smooth operations for various storage needs. Offline deals can utilize a PieceLocator configuration with HTTP servers to retrieve pieces when requested.
+* **libp2p and Networking:** Curio’s market integrates with libp2p for peer-to-peer communication, allowing for decentralized and secure data transmission between storage clients and providers.
+* **HTTP Server:** The HTTP server plays a vital role in managing tasks like offline deal data retrieval through URLs and handling client requests for deal processing.
+* **Retrieval and Indexing:** Deals are efficiently indexed and can be retrieved for future use, ensuring that stored data remains accessible and manageable throughout its lifecycle.
+
+The Curio Storage Market provides a highly configurable and extensible system for deal management, optimized for current and future decentralized storage protocols.
+
+## Curio vs Boost
+
+How is Curio market different from Boost?
+
+### Deal Processing
+
+In Curio’s deal processing, the task-based approach introduces modularity and flexibility in handling different stages of a deal's lifecycle. Each step, from data preparation to sealing, is managed as an individual task. This approach breaks down the complexity of the process by distributing specific responsibilities, such as data validation, publishing deals, and commP calculation, across multiple independent tasks.
+
+Each task is monitored, retried on failure, and orchestrated through the Harmony task system, ensuring better resource utilization and scalability. This structure allows for parallel processing of deals and ensures that failures in one part do not impact the entire deal flow. Additionally, it provides flexibility for incorporating new protocols as they are introduced, seamlessly integrating them into the task orchestration.
+
+This method ensures that Curio’s deal processing remains future-proof, adaptable to different protocols, and scalable across large volumes of data and interactions.
+
+### Offline Deals
+
+In Boost, offline deals were initiated by manually importing data on the storage provider's side. However, Curio simplifies this process by allowing users to [add a URL for offline deal data directly into the database](storage-market.md#add-data-url-for-offline-deals) or use the [`PieceLocator` configuration](storage-market.md#piecelocator-configuration) to point to a remote server that can serve the deal’s piece. Since Curio operates as a cluster rather than a single node, data might be needed on different nodes depending on task scheduling and execution. The remote read design in Curio provides flexibility by enabling any node to fetch the required data dynamically during task execution, ensuring efficient processing across the cluster.
+
+### Optional CommP
+
+In Boost, the CommP (Commitment to Piece) calculation was a mandatory step in the deal processing workflow. However, in Curio, CommP is optional and can be skipped if desired. This flexibility allows users to streamline deal processing by bypassing the CommP check when necessary, while still ensuring that deals can progress smoothly without compromising efficiency in specific use cases.
+
+### HTTP only retrievals
+
+Curio’s retrieval mechanism is HTTP-only, allowing serving deal data from multiple nodes, each running an HTTP server. These nodes can operate under different domain names, providing flexibility and scalability. This distributed retrieval design ensures that data can be fetched from any node within the cluster, based on availability and proximity, optimizing retrieval speed. By supporting multiple nodes and domains for HTTP-based retrieval, Curio enhances resilience and performance, allowing clients to access deal data efficiently without relying on a single centralized source, all within the HTTP protocol.
+
+### IPNI Sync
+
+Curio creates dedicated peerID for each miner ID which are used to identify the provider on IPNI. This peer ID is different from the peer ID on chain.
+
+Curio announces per piece instead of per deal to IPNI nodes. This removes unnecessary overhead of mapping deals to ads and instead takes a simpler approach of advertising only for the piece we have regardless of how many times they are onboarded.
+
+The chunking in Curio process leverages a database to ensure fast and efficient chunk creation and reconstruction. By storing the first CID of each chunk along with metadata like offsets and chunk numbers, the system can quickly locate and retrieve chunks. Sorted entries minimize redundancy by eliminating duplicates and allow for efficient querying and retrieval. Additionally, caching mechanisms and database indexing further enhance speed, making the system optimized for rapid chunking and reconstruction. This makes IPNI sync much faster in Curio compared to Boost.

documentation/en/curio-market/curio-http-server.md

@@ -0,0 +1,131 @@
+---
+description: >-
+  This page provides an overview of the Curio HTTP Server's key features,
+  including HTTPS support, security, middleware, routing capabilities, and
+  instructions for attaching custom service routes.
+---
+
+# Curio HTTP Server
+
+The Curio HTTP Server is a secure, flexible, and high-performance HTTP server designed for use with Curio cluster. It comes with built-in support for HTTPS using Let's Encrypt certificates, advanced middleware features like logging and compression, and the ability to integrate various Curio services, including IPNI, retrieval providers, and LibP2P.
+
+## Key Features
+
+### 1. **HTTPS with Let's Encrypt**
+
+* Automatic certificate management using Let's Encrypt, ensuring that all traffic is encrypted.
+* Supports automatic renewal of certificates and domain validation through the `autocert.Manager`.
+
+### 2. **Security-First Design**
+
+* **Strict Security Headers**: Adds essential security headers like Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, and XSS protection to mitigate common vulnerabilities.
+* Protects from clickjacking, content-type sniffing attacks, and more by enforcing best practices via HTTP headers.
+
+### 3. **Flexible Routing with Chi**
+
+* Uses the `chi` router for lightweight, flexible routing. Easily extend routes or handle custom paths by attaching new service modules.
+* Out-of-the-box support for path handling and easy extension for future service needs.
+
+### 4. **Built-in Middleware**
+
+* **Compression**: Utilizes the `httpcompression` package to support GZIP, Brotli, and Deflate, optimizing bandwidth usage based on configurable compression levels.
+* **Logging**: Logs every incoming request with details such as request method, path, and duration, aiding in easier debugging and monitoring.
+* **CORS Support**: Conditional CORS support based on configuration for handling cross-origin requests securely.
+
+### 5. **WebSocket and LibP2P Support**
+
+* Specialized handling for WebSocket upgrade requests, with support for forwarding them to the `/libp2p` endpoint.
+* Facilitates smooth communication between nodes and services in distributed Curio cluster running a single LibP2P.
+
+### 6. **Health Check Endpoint**
+
+* A built-in `/health` endpoint provides an easy way to check the status of the HTTP server, ensuring it is running smoothly.
+
+## Performance and Scalability
+
+* The server is optimized for handling large numbers of concurrent requests efficiently with appropriate timeouts (read, write, idle) to prevent overload.
+* Integrated compression ensures minimal bandwidth usage, even for large data exchanges.
+
+## Database Integration for TLS Certificate Cache
+
+The server stores Let's Encrypt certificates and cache information in `Harmonydb`. This ensures persistence and fast access to TLS certificates in a Curio cluster with multiple HTTP servers.
+
+### Database Operations
+
+* **Get**: Retrieves TLS certificates from the `autocert_cache` table.
+* **Put**: Inserts or updates certificates in the database.
+* **Delete**: Removes expired or invalid certificates from the cache.
+
+## Attaching Routes to Extend Server Functionality
+
+Curio HTTP Server is designed to allow easy integration of various services by attaching custom routes. The `attachRouters` function enables the attachment of specific service routes to the Chi router. Examples include:
+
+1. **Retrieval Provider**: Attaches routes to handle data retrieval using the `retrieval` module.
+   * Creates a `RetrievalProvider` and registers the necessary HTTP endpoints.
+2. **IPNI (Interplanetary Network Indexer)**: Integrates IPNI-specific routes.
+   * The IPNI provider is instantiated, and routes are attached for IPNI services to handle data advertisement publishing.
+3. **LibP2P Redirector**: Handles WebSocket connections for LibP2P communication.
+   * Redirects WebSocket upgrade requests from `/` to `/libp2p` for seamless peer-to-peer communication.
+
+Here’s how the routes are attached:
+
+```go
+attachRouters(ctx context.Context, r *chi.Mux, d *deps.Deps) (*chi.Mux, error) {
+   // Attach retrievals
+   rp := retrieval.NewRetrievalProvider(ctx, d.DB, d.IndexStore, d.CachedPieceReader)
+   retrieval.Router(r, rp)
+
+   // Attach IPNI
+   ipp, err := ipni_provider.NewProvider(d)
+   if err != nil {
+      return nil, xerrors.Errorf("failed to create new IPNI provider: %w", err)
+   }
+   ipni_provider.Routes(r, ipp)
+
+   go ipp.StartPublishing(ctx)
+
+   // Attach LibP2P redirector
+   rd := libp2p.NewRedirector(d.DB)
+   libp2p.Router(r, rd)
+
+   return r, nil
+}
+```
+
+This flexibility allows the server to be easily extended with new services without modifying the core server logic.
+
+## Configuration
+
+The Curio HTTP Server can be customized using the `HTTPConfig` structure, which allows you to configure timeouts, compression levels, and security settings to suit your application. Below are the key configuration options along with their default values and explanations of their impact.
+
+### **HTTPConfig**
+
+* **Enable**: This boolean flag enables or disables the HTTP server. Default: `true`.
+* **DomainName**: The domain name used by the server to handle HTTP requests. It must be a valid domain name and cannot be an IP address. A correct `DomainName` is required for Let's Encrypt to issue SSL certificates, enabling HTTPS.\
+  Default: `""` (empty), meaning no domain is specified. This needs to be set for production environments using HTTPS.
+* **ListenAddress**: Defines the IP address and port on which the server listens for incoming requests.\
+  Default: `"0.0.0.0:12310"` — This binds the server to all available network interfaces on port 12310.
+* **DelegateTLS: It** allows the server to delegate TLS to a reverse proxy. When enabled the listen address will serve HTTP and the reverse proxy will handle TLS termination.
+* **ReadTimeout**: This sets the maximum duration to read a full request from the client, including the request body.\
+  Default: `10 seconds` — A reasonable timeout for handling most requests. Reducing this value can prevent slow clients from hanging the server, while increasing it might help in cases where the request size is larger or the connection is slower.
+* **WriteTimeout**: The maximum time allowed for writing the response to the client.\
+  Default: `10 seconds` — Ensures that the server does not take too long to send responses, especially useful for API calls or small files. For large files or slower clients, you may need to increase this value.
+* **IdleTimeout**: The duration after which an idle connection is closed if no activity is detected.\
+  Default: `2 minutes` — Prevents resources from being consumed by idle connections. If your application expects longer periods of inactivity, such as in long polling or WebSocket connections, this value should be adjusted accordingly.
+* **ReadHeaderTimeout**: The time allowed to read the request headers from the client.\
+  Default: `5 seconds` — Prevents slow clients from keeping connections open without sending complete headers. For standard web traffic, this value is sufficient, but it may need adjustment for certain client environments.
+* **EnableCORS**: A boolean flag to enable or disable Cross-Origin Resource Sharing (CORS).\
+  Default: `true` — This allows cross-origin requests, which is important for web applications that might make API calls from different domains.
+* **CompressionLevels**: Defines the compression levels for GZIP, Brotli, and Deflate, which are used to optimize the response size. The defaults balance performance and bandwidth savings:
+  * **GzipLevel**: Default: `6` — A moderate compression level that balances speed and compression ratio, suitable for general-purpose use.
+  * **BrotliLevel**: Default: `4` — A moderate Brotli compression level, which provides better compression than GZIP but is more CPU-intensive. This level is good for text-heavy responses like HTML or JSON.
+  * **DeflateLevel**: Default: `6` — Similar to GZIP in terms of performance and compression, this default level provides a good balance for most responses.
+
+### **Impact of Compression Levels**
+
+The compression levels directly affect server performance and bandwidth usage:
+
+* Higher compression levels (e.g., `GzipLevel 9`, `BrotliLevel 11`) reduce the size of responses, which can save bandwidth but require more CPU processing time, especially for large responses.
+* Lower levels (e.g., `GzipLevel 1`) are faster but provide less compression, meaning higher bandwidth usage but reduced server load.
+
+For most applications, the default values of `6` for GZIP and Deflate, and `4` for Brotli provide a good trade-off between compression efficiency and CPU load, especially for responses that contain text or JSON.

documentation/en/curio-market/curio-libp2p-server/README.md

@@ -0,0 +1,48 @@
+---
+description: >-
+  This page outlines the main features of the Libp2p server, including its
+  configuration and components.
+---
+
+# Curio libp2p Server
+
+## Overview
+
+The Curio Libp2p server facilitates network communication between nodes and clients using the Libp2p protocol. It primarily handles deal proposals, deal status updates, and miner information in the Filecoin network. This server is tightly integrated with Curio's HTTP server and ensures the network operates smoothly for Filecoin deal processing.
+
+### Key Components:
+
+* **Host Setup**: The libp2p host is initialized with necessary configuration details, including listening addresses, identity keys, and a peerstore for managing peer connections.
+* **Deal Handling**: The libp2p server manages deal proposals and responses using the MK12 market protocol. It processes incoming streams for deal proposals, status checks, and query requests.
+* **WebSocket Proxy**: WebSocket connections from clients are proxied through the Curio HTTP server at the `/libp2p` path, directing traffic to the libp2p node's listening address for secure P2P communication.
+* **Miner Information Updates**: The libp2p provider also ensures that miner information, such as peer IDs and multi-addresses, is kept up-to-date on-chain, allowing for smooth communication during deal negotiations.
+
+## libp2p Host
+
+A **Libp2p host** is initialized with randomized ports for listening to incoming connections. Unlike other networks where the listen address might be broadcasted, in Curio's implementation, the Libp2p node listens on a random port and does not advertise its address to other peers.
+
+The setup involves creating a peer identity, configuring listen addresses, and ensuring that the node is ready to handle connections and requests. Once the node is operational, its local listen address is stored in the database, which the WebSocket proxy uses for connecting clients.
+
+## Network Deal Management
+
+Curio libp2p server plays a critical role in handling incoming deal proposals over the network. Deals are initiated using the `DealProtocolv120ID` and `DealProtocolv121ID` for handling storage market deals, and the `DealStatusV12ProtocolID` for tracking deal statuses.
+
+The deal provider continuously listens for deal requests, processes them, and sends responses back to the clients.
+
+### Deal Proposal Handling
+
+When a deal proposal is received, the libp2p handler send the request to MK12(INSERT LINK HERE) provider to validates the request, checks miner permissions, and processes the deal. If the deal is valid, MK12 provider moves forward with execution, handling the storage and sealing process on the storage provider's side.
+
+### Deal Status Tracking
+
+The libp2p server supports querying the status of a deal using the `DealStatusV12ProtocolID`. This allows clients to monitor the current state of their proposals, such as whether the deal is sealed, indexed, or still in progress.
+
+## WebSocket Proxy
+
+The Curio libp2p server leverages the HTTP server's `/libp2p` path to proxy WebSocket connections between the client and the libp2p node. This mechanism ensures secure and direct P2P communication through WebSocket channels. By connecting via the WebSocket proxy, peers can securely exchange deal information, status updates, and more.
+
+This proxy system enables seamless data flow between Curio nodes and external clients or peers, facilitating decentralized communication across the network without exposing local node details directly to clients.
+
+### Failover and Node Coordination
+
+To ensure proper coordination between multiple nodes, the libp2p server periodically updates the database with the node's status, such as the `running_on` field, and performs keep-alive checks. If the node fails to update its status in time, another node may take over, ensuring that there is no interruption in the decentralized network.