# Mooncake Store Deployment & Tunnig Guide This guide covers minimal deployment, and operational tuning of Mooncake Store. ## Architecture Overview ![architecture](../image/mooncake-store-preview.png) **Master Service** (`mooncake_master`): The central coordinator. It manages cluster membership, allocates object storage across client nodes, and enforces eviction/placement policies. Runs as a standalone process. **Client Node**: Each node contributes DRAM (and optionally VRAM/SSD) to form the distributed cache pool. Clients communicate with the master over RPC for control operations (`Put`/`Get`/`Remove`), but transfer actual data directly between each other via the Transfer Engine — the master is never in the data path. **Metadata Service**: A separate service (etcd, Redis, or HTTP) used by the Transfer Engine for peer discovery and configuration. The master's embedded HTTP metadata server can replace an external etcd/Redis for simple deployments. We also provide a P2P handshake mechanism (`P2PHANDSHAKE`) that enables decentralized metadata management by storing metadata locally on each node, eliminating the need for a centralized service — this is the simplest metadata handshake method and the recommended starting point (see [Quick Start](#quick-start)). For a detailed design discussion, see the [Mooncake Store Design](../design/mooncake-store.md). --- ## Quick Start Deploy a minimal single-node Mooncake Store in three steps. ### 1. Start the Metadata Service Choose one option: ```bash # Option A: P2P handshake — the simplest metadata handshake method (recommended) # Nothing to start here. There is no metadata service to deploy: each node # exchanges and stores metadata locally during connection setup. Just set the # client's metadata_server to the literal string "P2PHANDSHAKE" (see step 3). # Option B: Embed HTTP metadata server in the master # (configured in step 2 via --enable_http_metadata_server) # Option C: External etcd etcd --listen-client-urls http://0.0.0.0:2379 \ --advertise-client-urls http://localhost:2379 ``` > **Tip:** P2P handshake is the easiest way to get started — it is decentralized > and requires no etcd/Redis/HTTP metadata service. Prefer it for development and > simple deployments; use an external etcd/Redis for large, long-lived clusters. ### 2. Start the Master Service ```bash mooncake_master \ --enable_http_metadata_server=true \ --http_metadata_server_host=0.0.0.0 \ --http_metadata_server_port=8080 ``` On success the log shows: ``` Starting Mooncake Master Service Port: 50051 Max threads: 4 Master service listening on 0.0.0.0:50051 ``` The master's default RPC port is `50051`. The HTTP metadata server serves on port `8080` in this example. ### 3. Start a Store Client Use the Python sample program to bring up a client that contributes 3.2 GB of DRAM to the cluster: ```python # stress_cluster_benchmark.py import os from distributed_object_store import DistributedObjectStore store = DistributedObjectStore() store.setup( local_hostname=os.getenv("LOCAL_HOSTNAME", "localhost"), metadata_server=os.getenv("METADATA_ADDR", "http://127.0.0.1:8080/metadata"), global_segment_size=3200 * 1024 * 1024, # DRAM contributed to the cluster local_buffer_size=512 * 1024 * 1024, # Transfer Engine buffer protocol=os.getenv("PROTOCOL", "tcp"), device_name=os.getenv("DEVICE_NAME", ""), master_server_address=os.getenv("MASTER_SERVER", "127.0.0.1:50051"), ) ``` ```bash python3 stress_cluster_benchmark.py ``` To use **P2P handshake** instead of an HTTP/etcd metadata service, pass the literal string `P2PHANDSHAKE` as `metadata_server` — no other change is needed: ```python store.setup( local_hostname=os.getenv("LOCAL_HOSTNAME", "localhost"), metadata_server="P2PHANDSHAKE", # decentralized, no metadata service global_segment_size=3200 * 1024 * 1024, local_buffer_size=512 * 1024 * 1024, protocol=os.getenv("PROTOCOL", "tcp"), device_name=os.getenv("DEVICE_NAME", ""), master_server_address=os.getenv("MASTER_SERVER", "127.0.0.1:50051"), ) ``` The standalone store service accepts the same value: ```bash python -m mooncake.mooncake_store_service \ --local_hostname=localhost \ --metadata_server=P2PHANDSHAKE \ --master_server=127.0.0.1:50051 ``` **What just happened:** 1. The client registered itself with the master via RPC. 2. The master allocated a 3.2 GB segment on this node and added it to the cluster's memory pool. 3. The client is now ready to serve `Put`/`Get`/`Remove` requests. You can also deploy a standalone store service process that hosts memory/SSD without an inference application: ```bash python -m mooncake.mooncake_store_service \ --local_hostname=localhost \ --metadata_server=http://127.0.0.1:8080/metadata \ --master_server=127.0.0.1:50051 ``` ### Verify ```bash # Health check — master metrics endpoint curl -s http://localhost:9003/metrics/summary # List registered clients # (exposed through the store's Python API or RPC) ``` --- ## Deployment Scenarios ### Single-Node (TCP) — Development / Quick Evaluation The simplest deployment, as shown in [Quick Start](#quick-start). A single `mooncake_master` orchestrates clients over TCP. Suitable for development, testing, and single-host evaluation. ```bash mooncake_master \ --enable_http_metadata_server=true \ --http_metadata_server_host=0.0.0.0 \ --http_metadata_server_port=8080 ``` Limitation: the master is a single point of failure. If it crashes, cluster operations pause until it is restored. --- ### High-Availability (etcd) — Production HA Runs a cluster of master instances coordinated through etcd. If the leader fails, the remaining instances elect a new leader automatically. ```bash # Start each master instance with: mooncake_master \ --enable-ha=true \ --etcd-endpoints="10.0.0.1:2379;10.0.0.2:2379;10.0.0.3:2379" \ --rpc-address=10.0.0.1 ``` Each instance must specify its own reachable `--rpc-address`. The etcd cluster used for HA can be shared with or separate from the Transfer Engine's metadata etcd. --- ### High-Availability (Redis) — Alternative HA Backend Same HA semantics but using Redis instead of etcd for leader election: ```bash mooncake_master \ --enable-ha=true \ --ha_backend_type=redis \ --ha_backend_connstring="redis://127.0.0.1:6379" \ --rpc-address=10.0.0.1 ``` --- ### Snapshot & Restore — Backup / Disaster Recovery ```{caution} Metadata Snapshot And Restore is experimental feature. ``` Periodically persist master metadata to local disk or S3, enabling recovery from a recent snapshot after a crash. ```bash export MOONCAKE_SNAPSHOT_LOCAL_PATH=/data/mooncake_snapshots mooncake_master \ --enable_snapshot=true \ --snapshot_interval_seconds=300 \ --snapshot_retention_count=5 \ --snapshot_object_store_type=local \ --enable_snapshot_restore=true ``` --- ### Tiered Storage with SSD Offload — Cost-Effective Capacity Extends the cache pool from DRAM to SSD while keeping normal reads and writes on the distributed memory path. With `--enable_offload=true`, completed memory writes are queued for asynchronous SSD persistence through the master control plane. Set `--offload_on_evict=true` to defer that SSD write until the memory eviction path selects an object for reclamation. When `--promotion_on_hit=true`, SSD-only objects can be promoted back to DRAM after repeated reads; admission is gated by `--promotion_admission_threshold`. ```bash mooncake_master \ --enable_offload=true \ --offload_on_evict=true \ --promotion_on_hit=true \ --promotion_admission_threshold=2 \ --root_fs_dir=/mnt/ssd_cache \ --enable_http_metadata_server=true \ --http_metadata_server_port=8080 ``` --- ### CXL-Aware Allocation — Memory Tiering When the host has CXL-attached memory, the master can preferentially allocate new objects on the CXL tier, reserving local DRAM for latency-sensitive operations. ```bash mooncake_master \ --enable_cxl=true \ --cxl_path=/dev/dax0.0 \ --cxl_size=17179869184 \ --allocation_strategy=cxl ``` --- ### Container / Dynamic Network Interface When the master runs in a container with a dynamic IP, use `--rpc_interface` to resolve the RPC address from a stable interface name: ```bash mooncake_master \ --rpc_interface=eth0 \ --enable_http_metadata_server=true \ --http_metadata_server_host=0.0.0.0 \ --http_metadata_server_port=8080 ``` The master resolves the current IPv4 address of `eth0` at startup and uses it as the advertised RPC address. --- ## Metrics Endpoints The master exposes Prometheus-style metrics on `--metrics_port`: ```bash # Prometheus format curl -s http://:9003/metrics # Human-readable summary curl -s http://:9003/metrics/summary ``` --- ## Quick Tips - Scale `--rpc_thread_num` with available CPU cores and workload. - Start with default eviction settings; adjust `--eviction_high_watermark_ratio` and `--eviction_ratio` based on memory pressure and object churn. - Use `/metrics/summary` during bring-up; integrate `/metrics` with Prometheus/Grafana for production. - For detailed SSD offload configuration (storage backends, eviction policies, io_uring), see the [SSD Offload guide](ssd-offload). - For NVMe-oF SSD pool configuration see the [NVMe-oF SSD Pool Deployment Guide](nvmf-ssd-deployment-guide) - For experimental 3FS (USRBIO) integration as a persistent storage backend, see the [3FS USRBIO Plugin guide](../getting_started/plugin-usage/3FS-USRBIO-Plugin). - For detailed monitoring and observation see [Observability](../getting_started/observability) :::{toctree} :maxdepth: 1 :hidden: ssd-offload NvMe-Of SSD Pool HF3FS Plugin (Experimental)<../getting_started/plugin-usage/3FS-USRBIO-Plugin> ../getting_started/observability ::: --- ## Reference: Master Startup Flags ### RPC | Flag | Default | Description | |------|---------|-------------| | `--rpc_port` | `50051` | RPC listen port | | `--rpc_thread_num` | `min(4, CPU cores)` | RPC worker threads | | `--rpc_address` | `0.0.0.0` | RPC bind address | | `--rpc_interface` | empty | Network interface to resolve RPC address at startup (overrides `--rpc_address`) | | `--rpc_conn_timeout_seconds` | `0` | Idle connection timeout; `0` disables | | `--rpc_enable_tcp_no_delay` | `true` | Enable TCP_NODELAY | ### Logging The master uses glog. When `--log_dir` is set, all severities are merged into a single journal file in that directory (`mooncake_master.INFO.-