How to use Mooncake-Conductor on vLLM

[yejj710]

@mich45208 we are trying to test kv-aware routing of XpYd on vllm-router for performance based on the conductor with vllm, lmcache, and mooncake.

[yejj710]

Minimal usage of mooncake_conductor:

1. compile code: cd mooncake-conductor/conductor-ctrl/ && go mod tidy && go build -o mooncake_conductor main.go
2. run mooncake-conductor: ./mooncake_conductor (will start a http-server on port 13333)
3. register kv-publihser(via /register api) details can be see in [RFC]: Mooncake KV-Store Indexer API Standardization #1403
4. query kv-hit status (via /query api)

[mich45208]

thanks @yejj710
I tried to recompile whole mooncake stack on this branch and got 2 compile errors:

/tmp/Mooncake/mooncake-store/src/master_service.cpp:1726:13: error:
 'MetadataAccessor' was not declared in this scope; did you mean 'MetadataAccessorRW'?
 1726 |             MetadataAccessor accessor(this, key);
      |             ^~~~~~~~~~~~~~~~
      |             MetadataAccessorRW

/tmp/Mooncake/mooncake-store/include/master_service.h:725:13: error:
 'metadata' was not declared in this scope
  725 |             metadata.VisitReplicas(
      |             ^~~~~~~~

I was able to fix and compile with patch:

diff --git a/mooncake-store/include/master_service.h b/mooncake-store/include/master_service.h
index 20c5dd5..b2f0e62 100644
--- a/mooncake-store/include/master_service.h
+++ b/mooncake-store/include/master_service.h
@@ -722,7 +722,7 @@ class MasterService {

             std::vector<Replica::Descriptor> replica_list;
             replica_list.reserve(replicas_.size());
-            metadata.VisitReplicas(
+            VisitReplicas(
                 &Replica::fn_is_completed, [&replica_list](const Replica& replica) {
                     replica_list.emplace_back(replica.get_descriptor());
                 });
diff --git a/mooncake-store/src/master_service.cpp b/mooncake-store/src/master_service.cpp
index 1adcb03..826c11b 100644
--- a/mooncake-store/src/master_service.cpp
+++ b/mooncake-store/src/master_service.cpp
@@ -1723,7 +1723,7 @@ auto MasterService::NotifyOffloadSuccess(
         auto res = AddReplica(client_id, key, replica);

         if (publisher) {
-            MetadataAccessor accessor(this, key);
+            MetadataAccessorRW accessor(this, key);
             if (!accessor.Exists()) {
                 LOG(ERROR) << "key=" << key << ", error=object_not_found";
                 return tl::make_unexpected(ErrorCode::OBJECT_NOT_FOUND);

Lmk if you want me to add that change

[mich45208]

also it looks like conductor requires tokenized input and the test uses specific python proxy to send requests to tokenizer before sending to conductor, which adds more hops and overhead. Is there any plans for optimization, or a router with internal tokenizer is required to have best performance with conductor.