[RFC]: Replica copy and move support

[zhongzhouTan-coder]

Changes proposed

Introduction

This RFC proposes the introduction of explicit ReplicaCopy and ReplicaMove primitives within the Mooncake Store. These primitives provide the fundamental mechanisms required to migrate data replicas between different storage nodes. By exposing these operations, we enable external orchestration systems (such as gateways or schedulers) to implement cache-aware scheduling policies. Furthermore, this lays the groundwork for future internal features like dynamic replica management.

Motivation

Currently, Mooncake Store's replica placement is largely determined at creation time. As access patterns change or cluster topology evolves, the initial placement may become suboptimal. There is a growing need to adjust replica placement dynamically to optimize performance and resource utilization.

The primary drivers for this feature are:

1. External Cache-Aware Scheduling: Upper-layer applications or gateways (e.g., LMCache) often have global visibility into request patterns and compute scheduling. They need a way to proactively move data to where compute will happen (data locality) or replicate hot data to multiple nodes to spread the load. Without explicit copy/move APIs, these external systems cannot leverage Mooncake for optimized data placement.
2. Dynamic Replica Management: To support future features like auto-scaling of hot keys (creating more replicas for frequently accessed data) or rebalancing storage usage (moving cold data), the underlying store must support atomic copy and move operations.
3. Resource Optimization: Moving replicas allows for better utilization of cluster resources by rebalancing data distribution across nodes, preventing storage hotspots.

In Scope

1. Design and implement ReplicaCopy and ReplicaMove operations in the Master service for DRAM-based replicas.
2. Design and implement the corresponding data transfer logic in the Client service.
3. Expose APIs for external systems to invoke these operations.
4. Add new basic copy/move python api in the client side.
5. Add logging and metrics to monitor replica movements.
6. Testing and validation of the new features.

Out of Scope

1. Support for Disk or SSD storage media (DRAM only).
2. Cross-tier migration (e.g., Memory to Disk).

Proposal

In the previous RFC, we introduced a simple high level design for the ReplicaCopy and ReplicaMove operations. But the previous design use master initiated data transfer, which may introduce extra overhead for the current system architecture. And thanks to the PR, we resue the basic idea and introduce a new design which change the transfer task initialzie from master push to client poll.

Architecture

Component Diagram (Client Poll Model)

Pros and cons (Master push vs Client poll)

1. Master push

   • Pros:
     • Real-Time Latency: trigger a transfer immediately.
     • Traffic Efficiency: No empty checks from clients.
   • Cons:
     • Master and client complexity: introduce a new communication mechanism which will increase system complexity.

2. Client poll (Recommended)

   • Pros:
     • Simplicity & Stability: consistent with existing architecture.
     • Natural Flow Control：the Master doesn't need to implement complex backpressure logic.
   • Cons:
     • Latency：there is a delay up to the polling interval.

Sequence Diagram

1. Replica Copy

   

2. Replica Move

   

Core components

1. External RPC Interface (Master Service)

   These APIs are exposed to external systems (e.g., Gateways) to trigger operations.

   struct ReplicaCopyRequest {
       std::string key;
       std::List<std::string> targets; // localhost_name or client_id?
   };
   
   struct ReplicaMoveRequest {
       std::string key;
       std::string source; // localhost_name or client_id?
       std::string target; // localhost_name or client_id?
   };
   
   // RPC Methods
   tl::expected<UUID, ErrorCode> Copy(ReplicaCopyRequest req);
   tl::expected<UUID, ErrorCode> Move(ReplicaMoveRequest req);

2. Task queue and definition

   enum class TaskType {
       REPLICA_COPY,
       REPLICA_MOVE,
   };
   
   enum class TaskStatus {
       PENDING,
       RUNNING,
       SUCCESS,
       FAILED
   };
   
   struct Task {
       uint64_t task_id;
       TaskType type;
       TaskStatus status;
       std::string payload; // Serialized arguments based on type
   };
   
   struct ReplicaCopyPayload {
       std::string key;
       std::string source;
       std::List<std::string> targets;
   };
   
   struct ReplicaMovePayload {
       std::string key;
       std::string source;
       std::string target;
   };
   
   // Task Queue Management
   class ClientTaskManager {
       std::mutex client_tasks_mutex_;
       // the key can be client_id or host_name
       std::unordered_map<std::string, std::vector<Task>> client_tasks_;
   
       // use host_name or client_id
       void AddTask(const std::string& client_host_name, Task task);
   };
   
   // RPC Method
   tl::expected<std::vector<Task>, ErrorCode> PollTasks(const std::string& client_id); // client_id or host_name

3. Internal RPC Interface

   These APIs are used by the mooncake client to complete the replica copy/move tasks.

   struct AllocationConfig {
       std::vector<std::string> segment_names;
   };
   
   // RPC Methods
   // Copy
   tl::expected<std::vector<Replica::Descriptor>, ErrorCode> CopyStart(const std::string& key, const std::vector<size_t>& slice_lengths, const AllocationConfig& config);
   tl::expected<void, ErrorCode> CopyEnd(const std::string& key);
   // Move
   tl::expected<std::vector<Replica::Descriptor>, ErrorCode> MoveStart(const std::string& key, const std::vector<size_t>& slice_lengths, const AllocationConfig& config);
   tl::expected<void, ErrorCode> MoveEnd(const std::string& key);

4. Task executor (client)

   The task executor component in the client is responsible for executing the replica copy/move tasks received from the master. It handles the data transfer and communicates the status back to the master.

   class ReplicaCopyExecutor {
   public:
       void Execute(const ReplicaCopyPayload& payload) {
           // 1. Read data from local buffer(preferably) or other node
           auto query_result = client_->Query(key);
           client_->Get(key, query_result.value(), slices);
   
           // 2. Allocate Target
           AllocationConfig config = { payload.targets };
           auto target_descs = master_client_->CopyStart(payload.key, GetLengths(slices), config);
           
           // 3. Transfer Data
           for (const auto& replica : target_descs.value()) {
               TransferWrite(replica, slices);
           }
   
           // add rollback logic here if needed
   
           // 4. Commit
           master_client_->CopyEnd(payload.key);
       }
   };
   
   class ReplicaMoveExecutor {
   public:
       void Execute(const ReplicaMovePayload& payload) {
           // 1. Read data from local buffer(preferably) or other node
           auto query_result = client_->Query(key);
           client_->Get(key, query_result.value(), slices);
   
           // 2. Allocate Target
           AllocationConfig config = { {payload.target} };
           auto target_descs = master_client_->MoveStart(payload.key, GetLengths(slices), config);
   
           // 3. Transfer Data
           TransferWrite(target_descs[0], slices);
           
           // add rollback logic here if needed
   
           // 4. Commit
           master_client_->MoveEnd(payload.key);
       }
   };

5. Task Status Monitor (nice to have)

   The master service should provide an external API for external systems to monitor the status of ongoing replica copy/move tasks.

Before submitting a new issue...

• Make sure you already searched for relevant issues and read the documentation

[zhongzhouTan-coder]

@ykwd @stmatengss @nickyc975 please review the current implementation, welcome give your suggestions~

[ykwd]

Thanks for the proposal. The overall design looks good to me. Here are a few comments:

1. It seems that this is a framework for performing copy and move operations, and the decision-making logic—such as which data needs to be moved and where it should be moved—is not included in this PR. Is my understanding correct?

2. Regarding what the source and target fields in ReplicaCopyRequest and ReplicaMoveRequest should contain (e.g., localhost_name, client_id, or something else), I think it depends on what information is already determined at the time the request is generated.

For example, if the destination client has already been decided, then the request could include the segment name on that client (if a client has multiple segments, they share the same segment name but have different uuid). This would allow CopyStart and MoveStart to directly allocate the corresponding space on that segment. From the sequence diagram, it looks like this is the intended approach.

However, this raises a question: what should we do if, during CopyStart or MoveStart, the segment on that client is full and allocation fails?

[zhongzhouTan-coder]

@ykwd Thanks for your comments~
If the client is full and allocation fails, I think we can introduce two mechanisms:

1. handle in client side with retry
   • client temporally add the task to local failed queue, and try later
   • if reach the max try, we should report the task as failed
2. handle in master side with local kv cache eviction
   • if client is full, we can trigger a local segment kv cache eviction and try reallocated again

we can consider this in the detailed implementation~

[ykwd]

@ykwd Thanks for your comments~ If the client is full and allocation fails, I think we can introduce two mechanisms:

1. handle in client side with retry

   • client temporally add the task to local failed queue, and try later
   • if reach the max try, we should report the task as failed

2. handle in master side with local kv cache eviction

   • if client is full, we can trigger a local segment kv cache eviction and try reallocated again

we can consider this in the detailed implementation~

While both approaches are correct, I personally feel that the first one may be easier to implement, since we currently only support global eviction and do not have per-client eviction.

[zhongzhouTan-coder]

Yes, for simplicity, we can introduce the retry mechanism with time or max try limit for the current implementation. And do some optimization in the future~

[stmatengss]

For CopyStart and MoveStart, could we rename them to be simple and easy to understand? On the other hand, it would be better to list Python APIs here.

[zhongzhouTan-coder]

@stmatengss Currently the exposure api name only move and copy, and the CopyStart/MoveStart/CopyEnd/MoveEnd just the internal api that is aimed to complete the whole copy/move work.

And here I have one question about the python API, we current assume that the external system like cache aware gateway will directly call our master server by RPC call (HTTP may be exposure in the future) and not need to deploy a mooncake client, so we do not need to exposure the python api.

I think this determined by how the external will use our system, we can also exposure the cpoy/move to python api if needed~

[stmatengss]

@stmatengss Currently the exposure api name only move and copy, and the CopyStart/MoveStart/CopyEnd/MoveEnd just the internal api that is aimed to complete the whole copy/move work.

And here I have one question about the python API, we current assume that the external system like cache aware gateway will directly call our master server by RPC call (HTTP may be exposure in the future) and not need to deploy a mooncake client, so we do not need to exposure the python api.

I think this determined by how the external will use our system, we can also exposure the cpoy/move to python api if needed~

These Python APIs enable Mooncake clients to perform data migrations, especially for requests bypassing the gateway.
Eg., one client is shut down, we want to migrate KVCache to another client.

[zhongzhouTan-coder]

I think for this situation, we need to clarify that who should call the api to start kv cache migration, may be the client itself.

If the client itself needs to migrate the kv cache to another client, we can exposure a move/copy api in the client, and because we already have basic copystart/movestart... api to complete the move/copy logic, the implementation logic will be simple.

But for the copy/move exposure in the client side, the parameters depend on the client how to use it when its shutdown. I think this can be done in another feature request such as graceful shutdown, because the selection and placement of migration replica is another complex work. And also welcome to discuss and give your suggestions~

[stmatengss]

I think for this situation, we need to clarify that who should call the api to start kv cache migration, may be the client itself.

If the client itself needs to migrate the kv cache to another client, we can exposure a move/copy api in the client, and because we already have basic copystart/movestart... api to complete the move/copy logic, the implementation logic will be simple.

But for the copy/move exposure in the client side, the parameters depend on the client how to use it when its shutdown. I think this can be done in another feature request such as graceful shutdown, because the selection and placement of migration replica is another complex work. And also welcome to discuss and give your suggestions~

Agree. Graceful shutdown is another important feature. Providing a similar Python API on the client side would enhance its flexibility.

Other features, like migration data or load balancing, rely on this. We can't restrict users to our conductor for these features.

[zhongzhouTan-coder]

Thanks, I think we can provide a basic python api in the client side that is similar as the exteranl copy/move api. If there have any other requirements, we can do some extension in the future.

[stmatengss]

Thanks, I think we can provide a basic python api in the client side that is similar as the exteranl copy/move api. If there have any other requirements, we can do some extension in the future.

Cool! Thx alot. Looking forward to this PR!

[zhongzhouTan-coder]

@stmatengss Thanks, currently @nickyc975, @Vincent-Bo-ali and me are currently collaborated to work on this feature, and hope the pr will be available quickly~

[stmatengss]

Thank a lot. Let's work together!