Kakao's LLM inference platform team reports that it reduced processing time for a fixed workload from 121.7 minutes to 47.2 minutes without changing GPUs or models. The team operates the inference platform used by internal services, covering everything from loading models onto GPUs to the gateway that distributes requests. In a post on Kakao Tech, the engineers describe two changes: allowing already-computed results to be stored outside the GPU, and routing the next request that asks about the same document to the server that holds those results.
The experiment used the same 32 GPUs and the same 3,456 requests. The team frames the result as a way to handle more AI requests on the same hardware by retaining computation results longer and making subsequent requests find them.
Why it matters
LLM inference platforms are typically constrained by GPU capacity, and operators often face demand that outpaces available accelerators. The Kakao result is notable because the gain came from scheduling and memory placement rather than from new hardware or a different model. If the approach generalizes, it suggests that part of the headroom in existing inference clusters lies in how cached computation is stored and how requests are matched to it.
The post describes the platform team's scope as spanning model deployment on GPUs and the request-routing gateway. That end-to-end ownership is relevant to the result, since the two changes touch both the caching layer and the routing layer.
Technical details
The first change is KV cache offloading. In transformer inference, the key-value cache holds intermediate attention state for tokens already processed, and it normally lives in GPU memory. Kakao's team enabled storage of these computed results outside the GPU, so the cache is not confined to accelerator memory.
The second change is EPP routing. Rather than distributing requests without regard to where prior computation resides, the gateway sends a request that references the same document to the server that already holds the corresponding cached results. The post summarizes the combination as keeping computed results around longer and having the next request go find them.
The reported measurement is a reduction from 121.7 minutes to 47.2 minutes for 3,456 requests across 32 GPUs. The source does not provide per-request latency figures, cache hit rates, the size of the offloaded cache, the models involved, or the specific documents used in the workload, so those details cannot be assessed from the material available.
What to watch
The result points to cache locality as a routing concern. Operators evaluating similar changes will want to know how often requests in their own traffic reference the same documents, since the routing benefit depends on reuse. They will also want to understand the cost of moving cached state between storage and GPU memory, which the Kakao post does not quantify in the excerpt available.
The team's framing is deliberately narrow: same GPUs, same model, same request count. That makes the comparison clean but leaves open how the approach behaves under different traffic mixes or with other models. For platform engineers, the practical takeaway is that the gateway and the cache are a single design surface, and that request affinity to cached computation is worth measuring alongside raw GPU utilization.





