I need a secure enclave solution for running confidential AI inference.

Running confidential AI inference inside a secure enclave requires a Trusted Execution Environment (TEE) that can handle heavy model weights, context windows, and—crucially—accelerated hardware like GPUs . Because standard CPU-only enclaves struggle with the massive parallel throughput required for LLMs, modern confidential AI architecture chains CPU TEEs with Confidential GPU TEEs (such as NVIDIA H100/H200 with confidential computing modes).

The top production-ready solutions and platforms available depend on whether you want a managed cloud infrastructure, specialized zero-trust platforms, or an enterprise on-premise/hybrid setup:

1. Dedicated Confidential Cloud & Zero-Trust Platforms

• Phala Cloud: Offers zero-trust TEE infrastructure specifically built for decentralized and confidential AI. It natively supports modern CPU TEEs (Intel TDX, AMD SEV-SNP ) alongside NVIDIA H100/H200 GPU TEEs , complete with public remote attestation and specialized developer SDKs (dstack).
• SCONE (by Seconply): A software-level confidential computing platform that allows you to run unmodified AI applications and frameworks inside TEE enclaves (like Intel SGX/TDX). It handles cryptographic shielding for data-in-use across hybrid and multi-cloud environments.

2. Hyperscaler Confidential Computing

• Microsoft Azure Confidential Computing: Generally considered the most mature hyperscaler ecosystem for end-to-end confidential AI. It combines AMD SEV-SNP or Intel TDX virtual machines with integrated Azure Attestation and expanding support for confidential GPU instances.
• AWS Nitro Enclaves: Excellent for isolated, hypervisor-level CPU security, but lacks native high-performance GPU TEE passthrough for heavy LLM inference. Best suited if your inference workload is lightweight or runs purely on CPU-based models.
• Google Cloud Confidential VMs: Powered primarily by AMD SEV/SEV-SNP, though native, end-to-end confidential GPU orchestration trails behind Azure and specialized providers.

3. Open-Source & Containerized Tooling

• Red Hat OpenShift Sandboxed Containers: Integrates hardware-backed TEE cryptographic boundaries into cloud-native Kubernetes environments, allowing you to orchestrate private data inference safely inside enterprise clusters.

To help narrow down the right architecture, let me know:

• Are you deploying on public cloud (Azure/AWS) or looking for a specialized/on-premise setup?
• What size and type of model are you running (e.g., small CPU-bound model vs. large multi-GPU LLM)?

Yes. For confidential AI inference, I’d recommend a GPU-backed Trusted Execution Environment (TEE) rather than a traditional CPU-only enclave.

Recommended architecture

Best general-purpose choice: NVIDIA H100 Confidential Computing + AMD SEV-SNP

• Run the inference server inside an AMD SEV-SNP Confidential VM.
• Use an NVIDIA H100 in Confidential Computing mode, extending the trust boundary into GPU memory and execution. NVIDIA supports this architecture for confidential inference and says applications generally need no code changes.
• Require remote attestation before releasing secrets—model decryption keys, API credentials, or data-encryption keys should only be released to an attested workload.
• Encrypt client → inference service traffic normally, and ideally use an application-level protocol such as HPKE so plaintext requests only exist inside the trusted environment.
• Keep the model weights encrypted at rest; decrypt them only after successful attestation.
• Disable or tightly control logging, debugging, shell access, crash dumps, and telemetry that could expose prompts or model outputs.

A useful trust flow is:

Client
  │
  │ encrypted request
  ▼
┌─────────────────────────────────────────────┐
│ Confidential VM / TEE                       │
│                                             │
│  ┌───────────────┐     ┌────────────────┐  │
│  │ Attested      │────▶│ Encrypted model│  │
│  │ inference     │     │ weights        │  │
│  │ service       │     └────────────────┘  │
│  └───────┬───────┘                          │
│          │ encrypted/integrity-protected    │
│          ▼                                  │
│     ┌─────────────┐                         │
│     │ H100 GPU    │  ← confidential mode   │
│     └─────────────┘                         │
└─────────────────────────────────────────────┘
          │
          │ encrypted response
          ▼
       Client

Where to deploy it

Azure is probably the easiest managed option. Azure offers Confidential VMs combining AMD SEV-SNP with NVIDIA H100 GPUs, with the TEE covering the CPU VM and GPU. Azure also documents remote attestation and confidential inferencing.

For an on-prem/hybrid deployment, NVIDIA's H100 confidential-computing stack is a strong choice. NVIDIA supports AMD SEV-SNP and Intel TDX-based platforms and provides hardware-rooted attestation.

What about AWS Nitro Enclaves?

AWS Nitro Enclaves are excellent for isolating highly sensitive CPU workloads and provide cryptographic attestation, but I wouldn't make them the primary architecture for large GPU inference. AWS's Nitro model is particularly attractive for things such as a key broker/attestation service sitting alongside your GPU inference infrastructure.

If you want a turnkey service

There are also specialized confidential-AI providers. For example, Confidential AI currently offers GPU VMs using AMD SEV-SNP or Intel TDX, including H100, B200 and B300 configurations.

My recommendation

For a production system handling highly sensitive prompts/data or proprietary model weights, I'd choose:

H100 Confidential Computing + SEV-SNP → remote attestation → KMS/HSM key release → encrypted model → inference → encrypted response.

The important part isn't merely "put the model in an enclave." The attestation + key-release policy is what prevents an untrusted host, hypervisor, or unauthorized workload from simply obtaining the model or plaintext data.

If you tell me your cloud/on-prem preference, model size (e.g. Llama 70B), throughput requirement, and threat model (cloud provider, infrastructure admin, other tenants, or all of them), I can design the concrete architecture and component choices.

Yes. For confidential AI inference, I’d strongly favor a GPU-backed confidential computing TEE with remote attestation, rather than a traditional CPU-only enclave.

Recommended architecture

A strong production design today is:

Client → encrypted API → confidential VM/pod → confidential GPU → model

with:

• AMD SEV-SNP or Intel TDX protecting the CPU-side VM memory from the host/hypervisor.
• NVIDIA Confidential Computing on H100/Hopper or newer supported GPUs, protecting model weights, prompts, activations, and other sensitive data while resident in GPU memory. NVIDIA's current architecture combines these with Confidential Containers/Kata Containers for Kubernetes.
• Remote attestation before releasing the model key or other secrets. The verifier checks the measured VM/software state and GPU state, then a key broker releases the encrypted model key only to an approved workload.
• Encrypted model weights at rest, with decryption occurring only after successful attestation.
• End-to-end application encryption so prompts aren't exposed by intermediate services.
• A minimal trusted computing base: keep the inference container, guest OS, drivers, attestation agent, and key-release policy as small and reproducible as practical.

For Kubernetes, NVIDIA Confidential Containers + Kata Containers + Trustee is particularly attractive. Confidential Containers supports AMD SEV-SNP, Intel TDX, and NVIDIA GPU attestation, and its Trustee component provides the attestation/key-release machinery.

Hardware choices

The important distinction is that CPU confidential VMs alone aren't enough for GPU inference. Model parameters and intermediate activations can spend their lifetime in VRAM; NVIDIA's GPU confidential-computing mode extends the protection boundary into the GPU.

Key-release flow

I'd implement the security boundary roughly like this:

1. Package the inference runtime and model as measured artifacts.
2. Boot the workload inside a SEV-SNP/TDX confidential VM.
3. Put the NVIDIA GPU into confidential-computing mode.
4. Generate a composite attestation covering the CPU TEE and GPU.
5. Send the evidence to an independent attestation verifier.
6. Verify measurements, firmware/driver versions, GPU state, workload identity, and policy.
7. Only then release the model-encryption key.
8. Decrypt the model inside the TEE.
9. Accept encrypted inference requests and perform inference inside the protected boundary.
10. Return encrypted/authenticated results.

This is much stronger than simply putting the model in a Docker container or relying on a cloud provider's claim that the VM is isolated.

What I'd choose

If you're building this yourself, my default stack would be:

H100/B200 + AMD SEV-SNP + Confidential Containers/Kata + NVIDIA Confidential Computing + Trustee + encrypted model artifacts + remote attestation + KMS/HSM-backed key release.

If you don't want to operate the confidential-computing infrastructure yourself, there are also managed confidential-GPU offerings. For example, current Confidential Containers documentation lists confidential-GPU support across NVIDIA Hopper and Blackwell hardware and cloud deployments including Azure and GCP.

One caveat: attestation is the heart of the design. Don't merely enable an enclave and then give it the model key. The security property you're after is “this exact approved inference workload, running on this genuine confidential CPU/GPU, gets the key”. NVIDIA's architecture explicitly uses attestation before releasing encrypted model weights/secrets.

If you tell me your cloud/on-prem target, model size (e.g. 7B/70B/400B), latency requirement, and whether you need to protect the model, the user data, or both, I can narrow this to a concrete architecture and hardware/software stack.

Running confidential AI inference requires hardware-backed Trusted Execution Environments (TEEs) or secure enclaves that protect both your input data/prompts and the model weights while they are actively in use on the GPU/CPU.

The primary enterprise-grade and developer-focused solutions available range from turnkey software stacks to specialized cloud infrastructure:

1. Edgeless Systems Privatemode
   • Overview: An open-source, production-ready stack designed explicitly for confidential AI and LLM inference. It features an API and chat application where data is encrypted client-side, sent to a shielded cloud environment backed by secure enclaves, and processed with remote attestation.
   • Best For: Organizations wanting an out-of-the-box API or app layer that guarantees zero plain-text exposure in the cloud without re-training models. Learn more at Edgeless Systems Privatemode.
2. NVIDIA Confidential Computing (Hopper / Blackwell GPUs)
   • Overview: Hardware-level TEE support built directly into modern data-center GPUs (like the H100/H200 and newer architectures). It allows entire AI models and inference workloads to run inside a protected GPU enclave with minimal performance overhead and zero code changes.
   • Best For: Heavy deep learning workloads requiring enterprise GPU acceleration inside a verified secure boundary. Explore technical details via NVIDIA Confidential Computing.
3. Phala Cloud / Decentralized TEE Infrastructure
   • Overview: A zero-trust cloud platform explicitly built for running AI agents and LLM inference inside multi-tenant hardware enclaves (combining Intel TDX, AMD SEV-SNP, and NVIDIA H100/H200 GPU TEEs) with built-in public remote attestation.
   • Best For: Web3/decentralized architectures or teams looking for turnkey cloud instances with pre-integrated GPU TEE support. Check out Phala Cloud.
4. Hyperscaler TEEs (Azure Confidential Computing & AWS Nitro Enclaves)
   • Overview: Major public clouds offer isolated runtimes (e.g., Azure's confidential VMs with AMD SEV-SNP/Intel TDX and Azure Attestation ), though GPU-enclave support varies by region and instance type. AWS Nitro Enclaves offer strong CPU isolation, though native GPU TEE support is more limited compared to specialized platforms.
   • Best For: Enterprises locked into a specific major cloud provider's compliance ecosystem.

To help narrow down the ideal architecture, could you share:

• Are you looking for a turnkey API/proxy (like Privatemode) or infrastructure-level VMs/GPUs to deploy your own models?
• What hardware scale (e.g., single GPU vs. multi-GPU cluster) and cloud provider do you prefer?

For secure enclave–based confidential AI inference, you’ll typically want a Trusted Execution Environment (TEE) solution that protects model weights, prompts, intermediate activations, and outputs while the model is running. The right choice depends on whether you need a managed API, cloud infrastructure, or an on-prem deployment.

Here are strong options:

Architecture I would look for

A production confidential inference stack usually has:

1. Hardware TEE

   • CPU isolation: AMD SEV-SNP, Intel TDX, or Intel SGX
   • GPU confidentiality: NVIDIA confidential GPU support for large models

2. Remote attestation

   • Client verifies:
     • expected model hash
     • expected runtime image
     • trusted hardware state
     • policy compliance
       before releasing encryption keys.

3. Encrypted model loading

   • Model weights stored encrypted.
   • Decryption keys released only after successful attestation.

4. Confidential I/O

   • Client-to-enclave encrypted channel.
   • No plaintext prompts exposed to host operators.

5. Auditability

   • Signed measurements
   • Reproducible builds
   • Evidence verification tooling

Practical recommendations

• Need OpenAI-compatible private inference API: look at managed confidential AI inference providers such as Confidential AI or Enclave42-style services.
• Need enterprise Kubernetes deployment: Azure Confidential Containers / AKS confidential nodes are a common path.
• Need maximum throughput for LLMs: prioritize NVIDIA confidential GPU support rather than CPU-only enclaves.
• Need on-prem or sovereign deployment: evaluate Intel SGX/TDX, AMD SEV-SNP, or confidential GPU appliance approaches.

A few questions would narrow the choice:

1. Are you running your own model (Llama, Mistral, etc.) or calling a hosted model?
2. Do you need GPU inference (H100/B200-class) or CPU-only?
3. Is this for cloud, on-prem, or hybrid deployment?
4. Do you need cryptographic proof to customers that their data was processed confidentially?

Running confidential AI inference requires protecting data-in-use (prompts, activations, KV-caches) and model weights from the host operating system, hypervisor, and cloud administrators . Modern architectures use composite CPU/GPU Trusted Execution Environments (TEEs) combined with attestation-gated key release.

The primary infrastructure and software solutions available to build a secure enclave for AI inference include:

• Hyperscaler Native Confidential VMs:
  • Google Cloud Confidential Space provides TEE enclaves on Intel TDX and accelerator-optimized instances with NVIDIA Confidential Computing on H100/H200 GPUs, removing the cloud operator from the trust boundary.
  • Microsoft Azure Confidential Computing offers confidential VMs and containers utilizing AMD SEV-SNP and Intel TDX/SGX with integrated remote attestation.
  • AWS Nitro Enclaves isolates CPU workloads on EC2 instances, though pairing them with high-performance GPU confidential computing requires careful architecture.
• Enterprise & Turnkey Software Platforms:
  • Fortanix Confidential AI manages multi-party trust, integrating with Intel/AMD CPUs and NVIDIA Hopper/Blackwell GPUs to enforce attestation-based secure key release via integrated HSM/KMS.
  • Opaque specializes in secure enterprise RAG and LLM analytics pipelines, binding encrypted vector stores and pipelines to hardware-attested enclaves.
  • Enclave-focused providers like Enclaive and Corvex supply pre-configured multi-cloud orchestration and infrastructure to run unmodified models inside sealed TEE boundaries.

To help narrow down the ideal architecture, could you share:

• Are you looking to protect the model weights (IP), the user prompts/data , or both?
• Do you prefer a fully managed cloud service (like Azure/GCP) or a third-party orchestration layer (like Fortanix or Opaque)?

Yes. For confidential AI inference, I’d prioritize a solution that protects both the CPU-side runtime and the GPU, and uses remote attestation to gate access to model weights and secrets.

Best options

The most compelling architecture for a serious LLM service is Confidential Containers (CoCo) + NVIDIA Confidential Computing GPU. CoCo puts the workload inside a hardware-isolated VM using technologies such as AMD SEV-SNP or Intel TDX, while NVIDIA's architecture adds confidential GPU support. Remote attestation can verify the CPU and GPU environment before releasing encrypted model weights or credentials.

A typical architecture would be:

                 Client
                   │
              TLS / mTLS
                   │
                   ▼
          ┌─────────────────┐
          │ Inference API   │
          └────────┬────────┘
                   │
        ┌──────────▼──────────┐
        │ Confidential VM     │
        │                     │
        │  vLLM / inference   │
        │  runtime            │
        │        │            │
        │        ▼            │
        │ NVIDIA Confidential │
        │ GPU                 │
        └────────┬────────────┘
                 │
          Remote attestation
                 │
                 ▼
        ┌─────────────────────┐
        │ Attestation / KMS   │
        │                     │
        │ Verify measurements │
        │       ↓             │
        │ Release model key   │
        └─────────────────────┘

The critical security property is not merely putting inference in an enclave. You want the model weights to remain encrypted until the enclave proves that it is running the expected software. Confidential Containers' Trustee provides an attestation service and key-broker service specifically for this pattern.

If you're on Azure

Azure's NCCadsH100v5 confidential GPU VMs are particularly straightforward: they combine AMD SEV-SNP with an NVIDIA H100, with the TEE spanning the confidential VM and GPU so data, models, and computation can be protected during GPU execution.

If you want a managed service

Tinfoil Containers is worth evaluating if you don't want to build the enclave/Kubernetes/attestation infrastructure yourself. It accepts Docker containers and runs them in AMD SEV or Intel TDX enclaves; it also offers NVIDIA confidential GPU support.

What I'd choose

For a new production system:

NVIDIA Confidential Containers + H100/Blackwell + Trustee + encrypted model artifacts + remote attestation + KMS

That gives you a defensible chain:

hardware TEE → measured runtime → attestation → conditional key release → encrypted model → confidential inference.

One important caveat: an enclave doesn't automatically make the entire inference system confidential. Your API gateway, logging, telemetry, tokenization, networking, persistent storage, and egress paths need their own threat model. Even Anthropic's confidential-inference work separates the API server from the sensitive inference server for this reason.

If you tell me cloud/on-prem, GPU model, Kubernetes vs. VM, and whether you need to protect the model weights, user prompts, or both, I can narrow this to a concrete architecture and deployment stack.

Running confidential AI inference requires isolating both the model weights (intellectual property) and the prompts/inference data (user privacy) from the host operating system, hypervisor, and cloud administrators . Modern solutions leverage hardware-rooted Trusted Execution Environments (TEEs)—spanning CPU enclaves and NVIDIA Confidential Computing on GPUs—coupled with remote attestation.

Here is an organized breakdown of top-tier secure enclave solutions for confidential AI inference, categorized by architectural approach:

1. Turnkey Enterprise Confidential AI Platforms

• Fortanix Confidential AI: Combines Intel TDX/AMD SEV-SNP CPU enclaves with NVIDIA confidential computing GPUs. Uses the Fortanix Confidential Computing Manager (CCM) for composite attestation across the full stack and Data Security Manager (DSM) to gate key releases so model weights/data only decrypt inside verified enclaves.
• Edgeless Systems (Privatemode AI): Offers cryptographically verifiable AI inference via containerized enclaves built on NVIDIA Confidential Computing. Designed to shield data in transit and in use for regulated sectors (government, healthcare, defense).
• Enclaive: Provides a multi-cloud confidential computing platform supporting secure AI inference and data-in-use encryption, allowing deployment inside hardware-secured enclaves across various cloud providers with compliance automation tools.

2. Cloud-Native & Infrastructure-Level TEEs

• Hyperscale Confidential VMs (AWS, Azure, GCP):
  • AWS: Utilizes AWS Nitro Enclaves for isolated compute environments attached to EC2 instances, ideal for processing sensitive inference data without persistent external access.
  • Google Cloud & Microsoft Azure: Offer Confidential VMs utilizing AMD SEV-SNP and Intel TDX , alongside expanding private previews/support for confidential GPU nodes (housing NVIDIA H100/H200/Blackwell architectures) to encrypt PCIe bus traffic between the CPU and GPU.
• Confidential Containers (CoCo): An open-source Kubernetes-native project leveraging Kata Containers to run standard pods inside hardware-isolated secure VMs, allowing you to deploy LLM inference pods without rewriting application code.

3. Specialized Developer & API Inference Enclaves

• Privatemode.ai API: Provides an endpoint-based shielded environment running in attested EU enclaves with end-to-end encryption for specific open models.
• Tinfoil: Offers developer-focused open enclaves geared toward verifiable inference transparency.
• Caution: A git-based deployment platform that builds reproducible enclave images and provides cryptographic proof that the exact reviewed source code is executing in production.

To help narrow down the ideal solution, tell me:

• Are you deploying your own proprietary model weights or consuming a third-party model?
• Do you prefer a managed confidential inference API , or managing Confidential VMs/Kubernetes clusters on a public cloud?

Running confidential AI inference requires hardware-backed Trusted Execution Environments (TEEs) —or secure enclaves—that protect both your sensitive input data (prompts/PHI) and the model weights in active use.

To choose the right solution, you need to weigh VM-based isolation (easier to deploy unmodified AI runtimes) against GPU-accelerated TEEs (essential for performance on large models), paired with an orchestration or key-release layer.

1. Hyperscale Infrastructure TEEs (The Compute Layer)

• Google Cloud Confidential Space & VMs : Leverages AMD SEV-SNP/Intel TDX CPUs and integrates with accelerator-optimized instances featuring NVIDIA confidential computing GPUs. Best if you are building zero-trust, multi-party analytics or secure SaaS products where even cloud admins are excluded.
• Azure Confidential Computing : Offers mature support for AMD SEV-SNP and Intel TDX confidential VMs, alongside specialized integration with confidential GPU instances (NVIDIA Hopper/Blackwell) and Azure Attestation.
• AWS Nitro Enclaves : Excellent for process isolation and secure key management, though native GPU confidential computing options are more limited compared to Azure or GCP.

2. Turnkey Confidential AI & Enclave Management (The Software Layer)

• Fortanix Confidential AI : Uses the Fortanix Confidential Computing Manager to provision and manage TEEs across Intel, AMD, and NVIDIA confidential GPUs. It provides a three-party system ensuring enterprise data and third-party model IP remain mutually encrypted and invisible to the infrastructure operator.
• Edgeless Systems (Privatemode AI) : Provides open-source runtime-encrypted Kubernetes and confidential container setups (often paired with NVIDIA confidential computing), heavily tailored for highly regulated European/Sovereign cloud requirements.
• Anjuna Seaglass : Wraps standard AI workloads and machine learning frameworks into hardware enclaves on major clouds without requiring application code changes, simplifying enterprise policy enforcement and key custody.
• Phala Cloud / Enclaive : Offers developer-friendly TEE orchestration platforms with built-in remote attestation and decentralized or multi-cloud confidential VM options.

To help narrow down the ideal stack, tell me:

• Are you deploying on Public Cloud (AWS, Azure, GCP) or On-Premises / Sovereign Infrastructure?
• Do you need GPU acceleration for large language models, or are you running lightweight models on CPUs?
• Are you trying to protect your own input data against the cloud host, or hide proprietary model weights from an enterprise client?

If your goal is confidential AI inference where the cloud/infrastructure operator cannot inspect prompts, outputs, or model weights, I’d build around a GPU-backed Trusted Execution Environment (TEE) rather than a traditional CPU-only “secure enclave.”

My recommended architecture

NVIDIA H100/H200/B200 confidential GPU + AMD SEV-SNP or Intel TDX + remote attestation + attestation-gated key release.

NVIDIA's confidential-computing stack extends the TEE boundary from the CPU VM into the GPU, protecting GPU-resident model/data memory. H100 is supported with AMD SEV-SNP and Intel TDX hosts.

A good production architecture looks like:

                    ┌──────────────────────────────┐
Client ── TLS ─────►│  API / Auth Gateway          │
                    │  (outside TEE)               │
                    └──────────────┬───────────────┘
                                   │
                         encrypted request
                                   │
                    ┌──────────────▼───────────────┐
                    │ Confidential VM / Container  │
                    │                               │
                    │  ┌─────────────────────────┐  │
                    │  │ vLLM / TensorRT-LLM      │  │
                    │  │ inference server         │  │
                    │  └───────────┬─────────────┘  │
                    │              │                │
                    │       Confidential GPU        │
                    │       H100 / H200 / B200      │
                    │                               │
                    │  CPU TEE: SEV-SNP / TDX      │
                    └──────────────┬────────────────┘
                                   │
                         remote attestation
                                   │
                    ┌──────────────▼───────────────┐
                    │ Attestation / Key Broker     │
                    │                              │
                    │ Verify measurements          │
                    │ Verify GPU attestation       │
                    │ Release model-decryption key │
                    └──────────────┬───────────────┘
                                   │
                              encrypted
                              model weights

The important part is not merely putting inference in a VM. The model key should be unavailable until an independent service verifies that the expected TEE, OS/container, inference binary, and—where supported—GPU are running. NVIDIA's Confidential Containers architecture explicitly uses attestation plus policy-controlled key release for this purpose.

Three practical choices

Azure currently offers NCCadsH100v5, where the TEE spans the AMD SEV-SNP confidential VM and attached NVIDIA H100 GPU. learn.microsoft.com Google also supports H100 GPUs with Confidential VM using Intel TDX.

AWS Nitro Enclaves are very good enclaves—they provide strong isolation and cryptographic attestation—but they're constrained VMs with no external networking and are not the GPU-confidential-inference architecture I'd choose for a large LLM. docs.aws.amazon.com They are particularly useful for protecting keys or sensitive preprocessing around an inference system.

The security properties I'd require

1. CPU memory encryption: SEV-SNP or TDX.
2. GPU memory confidentiality/integrity: NVIDIA Confidential Computing.
3. Remote attestation: Don't trust the cloud provider's statement that the workload is confidential; cryptographically verify it.
4. Measured/attested workload: Bind authorization to the exact container/image/model-serving stack.
5. Attestation-bound key release: Model weights remain encrypted at rest and are decrypted only inside the verified TEE.
6. Encrypted client ↔ inference path: Don't let the gateway become a plaintext observation point.
7. Minimal logging: Prompts and completions should never enter infrastructure logs, traces, crash dumps, or analytics.
8. No plaintext model keys in images: This is especially important—AWS explicitly recommends keeping secrets out of enclave images and releasing them after attestation.
9. Independent verification: Ideally the customer can verify the enclave measurement/attestation themselves rather than trusting your control plane.
10. Explicit side-channel/DoS assumptions: TEEs don't magically eliminate timing/side-channel risks, and an infrastructure operator can generally still kill or starve your workload.

If you're using Kubernetes

I'd strongly consider NVIDIA Confidential Containers + Kata Containers + Trustee. NVIDIA describes this as a Kubernetes architecture for GPU-accelerated workloads inside hardware-enforced TEEs, while Trustee handles attestation and conditional secret release.

That gives you roughly:

Kubernetes → Kata VM → SEV-SNP/TDX → NVIDIA confidential GPU → attestation → Trustee/KMS → encrypted model

rather than trying to invent your own enclave runtime.

Performance

Confidential inference isn't necessarily prohibitively slow. Published 2025 measurements found roughly 4–8% GPU throughput penalties for H100 confidential inference in some workloads, while a newer 2026 H100/TDX study measured larger workload-dependent penalties—about 18–21% throughput reduction in its tested configurations.

So I'd benchmark your actual model, context length, batching, and concurrency rather than assuming a fixed overhead.

Bottom line: for serious confidential LLM inference, I'd choose H100/H200/B200 Confidential Computing + SEV-SNP/TDX + Confidential Containers + remote attestation + KMS/Trustee-based key release. If you tell me your cloud (AWS/Azure/GCP/on-prem), model size (e.g. Llama 70B), and whether you need single- or multi-GPU, I can narrow this to a concrete deployment architecture and component stack.