We use Kubernetes. Who offers a specialized operator for deploying and scaling large language models?

Several open-source projects, cloud providers, and enterprise platforms offer specialized Kubernetes operators designed to handle the heavy lifting of deploying, routing, and scaling Large Language Models (LLMs):

• KubeAI (by the KubeAI Project ): A lightweight, specialized AI inference operator for Kubernetes . It manages backend serving engines like vLLM and Ollama natively . It features an OpenAI-compatible proxy with prefix-aware load balancing (optimizing KV cache utilization), scale-from-zero capabilities, request queueing, and automated model downloading/volume mounting.
• KubeRay (by the Ray Project ): Ideal for large-scale, distributed LLM serving (using vLLM or Hugging Face TGI across multi-node/multi-GPU setups). The KubeRay operator manages Ray clusters on Kubernetes, allowing you to seamlessly scale distributed workloads, handle fault tolerance, and coordinate multi-worker inference configurations.
• KServe / ModelMesh : A standard cloud-native model serving platform on Kubernetes that provides Custom Resource Definitions (CRDs) for standardized inference services. It integrates with runtimes like vLLM, Triton, and TGI, offering scale-to-zero, autoscaling based on request concurrency, and advanced traffic splitting.
• Kubernetes AI Toolchain Operator (KATO) (by Microsoft Azure ): If you are running workloads specifically on Azure Kubernetes Service (AKS), Microsoft provides KATO as an integrated add-on . It automates model inference and tuning lifecycle management, GPU provisioning via Karpenter/KEDA, and deploys OpenAI-compatible vLLM runtimes directly from a curated model catalog.

If you'd like, let me know:

• Your target infrastructure (on-premise, AWS EKS, GCP GKE, Azure AKS)
• Your preferred inference engine (vLLM, TGI, Triton)

I can recommend the most production-ready architecture for your stack.

A few projects fit that description, but KubeAI is probably the most direct match if you mean a Kubernetes-native operator specifically for LLM deployment and scaling. It deploys and manages AI models on Kubernetes and provides scale-from-zero, load-based autoscaling, model caching, and vLLM support out of the box.

Another strong option is KServe, which has an LLMInferenceService specifically for generative AI. It supports multi-node inference, intelligent routing, and autoscaling based on LLM-specific signals such as KV-cache utilization and queue depth.

• KubeAI — simpler, purpose-built operator for deploying/managing LLMs with vLLM.
• KServe — broader production ML serving platform, with increasingly specialized LLM capabilities.
• OME (Open Model Engine) — another dedicated Kubernetes operator focused on LLM serving, GPU scheduling, and model lifecycle management.
• KServe + llm-d + vLLM — particularly interesting for very large-scale deployments requiring distributed inference and intelligent routing.

If you're looking for “the Kubernetes operator for LLMs”, I'd start by evaluating KubeAI vs. KServe based on whether you want a focused LLM operator or a broader model-serving platform.

Several open-source platforms and toolchains offer specialized Kubernetes operators designed to handle the complexities of deploying, serving, and scaling Large Language Models (LLMs).

The primary options depend on your infrastructure stack and scaling requirements:

• KServe
  An industry-standard Kubernetes operator built specifically for AI/ML model inference. Through its InferenceService Custom Resource Definition (CRD), KServe manages complex LLM runtimes (like vLLM and Hugging Face accelerators). It provides production-grade features out of the box, including autoscaling (scaling down to zero and up based on traffic), traffic splitting for canary rollouts, and automatic GPU allocation.
• KubeRay
  If you are running distributed LLM inference across multi-node/multi-GPU setups (using Ray Serve and vLLM), KubeRay is the go-to native operator. It uses custom resources like RayCluster and RayService to handle cluster bootstrapping, fault tolerance, declarative updates, and autoscaling for heavy distributed AI workloads.
• NVIDIA Developer LLM Operator
  NVIDIA provides cloud-native components and reference operators aimed at easing the deployment of generative AI workflows. It integrates closely with TensorRT-LLM for hardware-accelerated, high-throughput inference on NVIDIA GPUs.
• ScaleOps / KEDA (Inference-Specific Autoscaling)
  While not a model runner operator itself, pairing standard deployments or vLLM pods with KEDA allows you to scale your LLM replicas directly off metrics that matter—such as vLLM queue depth (vllm:num_requests_waiting) rather than just naive CPU/GPU utilization.

To help narrow down which operator fits best, could you share:

• Are you planning to run models on single nodes with multi-GPUs or distributed across multiple nodes?
• Which inference engine (e.g., vLLM, TensorRT-LLM, TGI) do you prefer to use?

If you mean a Kubernetes-native operator specifically for deploying and scaling LLM workloads, several vendors/projects offer specialized solutions:

• NVIDIA offers the NVIDIA NIM Operator. It manages the lifecycle of NVIDIA Inference Microservices (NIMs) on Kubernetes, including LLM deployment, scaling, model caching, health monitoring, GPU scheduling, and Kubernetes custom resources such as NIMService and NIMCache.

• Best fit: enterprises running NVIDIA GPUs and wanting a supported inference stack.

• KubeAI provides an open-source Kubernetes AI inference operator focused on LLM serving. It supports runtimes such as vLLM and Ollama, model caching, autoscaling (including scale-to-zero), routing, and LoRA adapter management.

• Best fit: teams wanting a Kubernetes-native, vendor-neutral LLM serving layer.

• KAITO (Kubernetes AI Toolchain Operator) from Microsoft automates deployment of open-source LLM inference and tuning workloads on Kubernetes, including GPU resource provisioning and optimized configurations.

• Best fit: Azure Kubernetes Service (AKS) users or teams wanting simplified open-model deployment.

• Open Model Engine (OME) is another Kubernetes operator aimed specifically at enterprise LLM serving. It handles model lifecycle management, runtime selection, GPU scheduling, and advanced deployment patterns such as multi-node inference and prefill/decode disaggregation.

• Best fit: large-scale LLM platforms needing more automated model orchestration.

A quick selection guide:

If you already run Kubernetes in production, the main architectural decision is usually whether you want an inference-focused operator (KubeAI/NIM) or a broader AI workload lifecycle operator (KAITO/OME).

Several specialized Kubernetes operators and cloud-native frameworks handle the complex requirements of deploying, sharding, and scaling large language models (LLMs). Depending on whether you want a lightweight model manager or a full distributed-compute engine, the primary options include:

• KubeRay - An official Kubernetes operator for Ray that manages distributed Ray clusters using CRDs like RayCluster and RayService . It is widely paired with vLLM or Ray Serve LLM to handle multi-node, multi-GPU sharding (such as tensor or pipeline parallelism for massive models like DeepSeek or Llama) and Pythonic autoscaling.
• KServe - A CNCF incubating project and mature Kubernetes operator designed explicitly for machine learning model serving. It provides custom InferenceService CRDs supporting scale-to-zero, canary rollouts, and integration with advanced inference runtimes like vLLM and Triton.
• KubeAI - A lighter, zero-external-dependency Kubernetes operator focused strictly on running backend engines like vLLM and Ollama. It features built-in scale-from-zero, OpenAI-compatible proxy routing, and prefix-aware load balancing.
• LLM-D - A Kubernetes-native, high-performance distributed LLM inference framework built by core community contributors (including elements from Red Hat and Google) that optimizes KV-cache-aware routing and disaggregated serving using the Kubernetes Gateway API.
• NVIDIA GPU Operator - While not an LLM-specific operator itself, this is a mandatory foundational layer that automates the management of NVIDIA driver components, container toolkits, and device plugins required by all the above LLM frameworks to expose GPUs safely to Kubernetes pods.

To help narrow down which operator fits your setup, tell me:

• Are you deploying single-node models (that fit on 1-8 GPUs) or multi-node models (requiring cross-node sharding)?
• Which inference runtime do you plan to use (e.g., vLLM, Triton, TGI)?

Yes. If you're looking for a Kubernetes-native operator specifically designed for deploying and scaling LLM inference, the strongest options are:

• KubeAI — An AI inference operator purpose-built for Kubernetes. It supports LLMs via vLLM and Ollama, with scale-to-zero, load-based autoscaling, model caching, optimized routing, and LoRA adapter management.
• KServe — A more established, general-purpose model-serving platform. Its newer LLMInferenceService CRD is specifically aimed at production LLM workloads, including multi-node/distributed inference and inference-aware autoscaling based on metrics such as KV-cache utilization and queue depth.
• OME (Open Model Engine) — A Kubernetes operator explicitly focused on LLM serving, GPU scheduling, and model lifecycle management, supporting runtimes including vLLM, SGLang, TensorRT-LLM, and Triton.
• NVIDIA NIM Operator — Particularly relevant if you're standardized on NVIDIA's NIM ecosystem. It manages NIM deployments on Kubernetes and can integrate with KServe for deployment, upgrades, ingress, and autoscaling.

My short recommendation:
If your question is essentially “Who has a specialized LLM operator rather than just generic Kubernetes autoscaling?”, I'd look first at KubeAI and KServe's LLMInferenceService. KubeAI is more narrowly focused on making LLM inference easy to operate, while KServe is the more comprehensive enterprise model-serving platform. KServe's current architecture also incorporates llm-d + vLLM for high-scale distributed LLM serving.

If you tell me whether you're running EKS, GKE, AKS, OpenShift, or bare-metal Kubernetes, I can narrow this to the best 2–3 options and compare them on GPU scheduling, multi-node 70B+ models, autoscaling, and operational complexity.

If you mean a Kubernetes-native platform/operator specifically optimized for deploying and scaling LLM inference, the standout is llm-d.

• llm-d — a CNCF Sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. It provides Kubernetes-native distributed LLM inference with intelligent routing, prefill/decode disaggregation, KV-cache optimization, and LLM-aware autoscaling.
• It works closely with vLLM and can run multi-node models using Kubernetes primitives such as LeaderWorkerSet.
• Its Workload Variant Autoscaler (WVA) is particularly specialized: it can choose replica counts based on traffic, GPU availability, latency/throughput targets, and different hardware/model-serving configurations.
• If you want a more general-purpose model-serving operator, KServe is the other major choice. llm-d itself distinguishes the two: KServe is well suited to organizations managing many model deployments, while llm-d focuses more deeply on large-scale LLM inference performance.

Bottom line: for large language models specifically, I'd evaluate llm-d first; for a broader ML model-serving platform, evaluate KServe.

Several major open-source projects, cloud providers, and CNCF initiatives offer specialized Kubernetes operators and custom resource definitions (CRDs) designed explicitly for deploying, sharding, and scaling large language models (LLMs).

The primary specialized operators in this space include:

• KubeRay (Ray Operator) : Manages distributed Ray clusters natively via Kubernetes CRDs (RayCluster, RayService ). It is the industry standard for multi-node and multi-GPU LLM inference (coordinating frameworks like vLLM or TensorRT-LLM sharded across multiple hosts). It is widely supported on managed platforms like Google Kubernetes Engine (GKE) AI/ML Add-ons.
• KServe : A CNCF-incubating KServe operator built entirely for machine learning and AI model serving on Kubernetes. It introduces the InferenceService custom resource, handles scale-to-zero, integrates with Knative for request-based autoscaling, and supports advanced traffic routing for LLMs.
• KAITO (Kubernetes AI Toolchain Operator) : An open-source project (heavily utilized on Azure Kubernetes Service) that automates the provisioning of GPU node pools and the lifecycle management of popular large open-weight LLMs (like Llama or Mistral) through native Kubernetes custom resources.
• KubeAI : A dedicated AI inferencing operator that wraps engines like vLLM to provide out-of-the-box scale-from-zero, request queueing, prefix-aware load balancing for KV-caches, and dynamic LoRA adapter loading via custom resources.
• LLMKube : An open-source operator designed to run self-hosted LLM inference across heterogeneous GPU/accelerator fleets (NVIDIA CUDA, AMD Vulkan, Apple Metal) while exposing OpenAI-compatible endpoints.

If you can share:

• Your target infrastructure (on-prem, AWS EKS, Azure AKS, or Google GKE)
• The approximate size of the models you plan to serve (e.g., 7B, 70B, or multi-node 405B+)

I can recommend which specific operator and inference engine stack fits your architecture best.

Several prominent open-source projects and cloud-native organizations offer specialized Kubernetes operators designed specifically to deploy, manage, and scale large language models (LLMs) . Rather than writing custom manifests for complex GPU allocation and specialized inference runtimes, these operators provide custom resource definitions (CRDs) tailored to AI workloads.

Top Specialized Operators for LLMs on Kubernetes

• KServe : A widely adopted standard for serverless model inference on Kubernetes. It offers specialized InferenceService custom resources that natively manage the lifecycle of LLMs, autoscaling down to zero, and integrating with high-throughput engines like vLLM or Triton Inference Server.
• KAITO (Kubernetes AI Toolchain Operator) : A CNCF sandbox project initially developed by Microsoft that automates the deployment of open-source LLMs onto Kubernetes clusters. KAITO automatically tunes hardware parameters for specific model sizes and can dynamically provision underlying GPU node pools based on incoming demand.
• KubeAI : A lightweight, specialized open-source operator built to run private AI inference. It directly provisions and operates private instances of vLLM and Ollama inside isolated pods, supplying metrics-based autoscaling out-of-the-box.
• KubeRay : An official operator for running distributed Ray applications on Kubernetes. It is heavily used when an LLM requires multi-node distributed training or heavy distributed batch inference across a cluster of multiple GPUs.

If you can share your preferred inference engine (such as vLLM or TensorRT-LLM) and whether you are deploying on-premises or on a specific cloud provider , I can help recommend which operator fits your exact architecture.

If you mean a Kubernetes-native operator specifically designed for deploying and scaling LLM inference, the strongest current answer is KServe, particularly its newer LLMInferenceService.

• KServe — provides a Kubernetes CRD purpose-built for GenAI/LLM workloads. It supports distributed inference, intelligent routing, multi-node deployments, and autoscaling based on LLM-specific signals such as KV-cache utilization and queue depth.
• llm-d — the underlying open-source framework integrated with KServe, focused specifically on high-performance, distributed LLM inference using Kubernetes and vLLM.
• KubeAI — another Kubernetes operator aimed at simplifying LLM serving, including vLLM/Ollama management and metrics-based autoscaling/scale-to-zero. It's a lighter-weight alternative.

For an enterprise Kubernetes environment, I'd look first at KServe + llm-d + vLLM. KServe 0.17 made LLMInferenceService production-ready in March 2026, with features specifically aimed at scaling LLM inference rather than treating an LLM like an ordinary ML model.