Google Cloud and Anyscale boost Ray Serve LLM performance on GKE

Google Cloud and Anyscale have introduced performance optimizations for Ray Serve LLM on Google Kubernetes Engine (GKE), addressing long-standing trade-offs between scalability and latency in large language model (LLM) inference. The collaboration focuses on three key architectural changes that collectively improve throughput and reduce response times for production workloads.

What changed

The updates center on three technical enhancements to Ray Serve LLM's infrastructure. First, the integration of HAProxy directly into Ray Serve replaces external load-balancing components, reducing proxy overhead and preventing Python runtime saturation during high-traffic periods. Second, a new direct token streaming architecture separates the initial request path from the response stream, allowing tokens to bypass the ingress router entirely. Third, the v2 Ray executor backend for vLLM moves Ray out of the data plane, enabling asynchronous scheduling and aligning performance with native vLLM executors.

Benchmark tests conducted on GKE clusters using Google Cloud's A4 VMs with NVIDIA HGX B200 hardware demonstrated the impact of these changes. Using the Gemma 4 E2B model, the updated Ray Serve LLM achieved up to 5x higher throughput and 8x lower latency compared to previous versions. Performance on an eight-replica serving cluster now approaches that of native vLLM setups while retaining Ray Serve's flexibility for model development and deployment.

Why the improvements matter

The performance gains address a critical challenge for organizations deploying LLMs in production: maintaining low latency and high throughput without sacrificing the developer experience. Previous versions of Ray Serve required teams to choose between ease of use and performance, often forcing compromises in either model serving efficiency or operational simplicity. The new optimizations eliminate this trade-off, allowing developers to use familiar Python-native APIs while meeting the demands of state-of-the-art inference workloads.

For infrastructure teams, the updates reduce the operational complexity of scaling LLM serving clusters. The HAProxy integration and token streaming architecture minimize bottlenecks that previously required manual tuning or additional infrastructure components. The v2 Ray executor backend also ensures compatibility with future optimizations in vLLM, reducing the maintenance burden for teams managing multiple model serving frameworks.

What to watch

The collaboration between Google Cloud and Anyscale signals a broader industry trend toward optimizing LLM serving infrastructure for production environments. Future updates may focus on further reducing overhead in multi-model serving scenarios or improving support for emerging hardware accelerators. Organizations evaluating model serving platforms should monitor how these optimizations translate to cost savings and performance improvements in their specific workloads.