For on-prem deployment with no data leaving our network, what model should we run?

For a strict zero-data-leakage, on-premises deployment, you want to focus on open-weights models hosted locally using self-hosted orchestration tools like Ollama, llama.cpp , or vLLM.

The best model depends heavily on your available hardware (VRAM) and your primary use case:

1. General Purpose & Enterprise Workhorses

• Meta Llama 3.3 (70B) or Llama 3.1 (8B / 70B): The gold standard for open-weight enterprise deployment. The 8B version runs fast on modest hardware and handles basic summarization, extraction, and chat. The 70B version approaches frontier-class capabilities for reasoning, complex instructions, and analysis, though it requires a beefy multi-GPU setup (e.g., 2 × or 4 × enterprise GPUs like A100/H100 or stacked RTX 3090/4090s).
• Mistral / Mixtral (e.g., Mistral 7B, Mixtral 8x7B): Excellent performance-to-size ratio. Mixture-of-Experts (MoE) models like Mixtral give you high intelligence while only activating a fraction of parameters per token, reducing compute costs while staying completely local.

2. Deep Reasoning & Logic

• DeepSeek-R1 (Distilled or Full Weights): Outstanding for heavy reasoning, math, and logic tasks. Meta-llama or Qwen distilled variants (like DeepSeek-R1-Distill-Qwen-14B or 32B) offer near-frontier reasoning capabilities that can fit comfortably on local workstation hardware.

3. Coding & Technical Tasks

• DeepSeek-Coder-V2 or Qwen 2.5 Coder: Phenomenal for local code completion, refactoring, and technical Q&A. They rival many commercial cloud APIs for a fraction of the size when loaded via quantized GGUF/EXL2 formats.

Recommended Deployment Stack

• Inference Engine: Use Ollama for the easiest setup, or vLLM if you need high-throughput concurrent API serving for an internal team.
• Air-gapped verification: Ensure the host server has its network interface physically disconnected or firewalled from external WAN access during and after model weight pulls.

To narrow this down to a specific recommendation, tell me:

• What hardware/GPU resources do you have available (e.g., Mac Studio, single RTX 4090, enterprise server with A100s)?
• What is the primary use case (general chat, automated coding, RAG document search, data extraction)?

If your requirement is strictly on-prem with no customer data leaving your network, I’d recommend OpenAI’s gpt-oss-120b as the default production choice.

• gpt-oss-120b — best choice when you have serious GPU capacity and want the strongest reasoning/tool-use capability. It has 117B total parameters but only ~5.1B active per token, and its native MXFP4 version is designed to fit in 80 GB of GPU memory.
• gpt-oss-20b — choose this when latency, cost, or hardware constraints matter more. It has 21B parameters and can run with about 16 GB of memory, making it much easier to deploy at scale or on smaller servers.

Both are open-weight, Apache 2.0 licensed models designed to run on infrastructure you control, including on-prem. OpenAI states that data sent to a self-hosted deployment is not received or processed by OpenAI. They can be served with stacks such as vLLM, Ollama, or llama.cpp.

My rule of thumb

One important distinction: OpenAI's hosted frontier models aren't the option for a truly air-gapped deployment. The gpt-oss weights are specifically the self-hostable option.

If you tell me your GPU(s), RAM, expected concurrent users, context length, and whether you need coding/RAG/agents, I can recommend the exact model + quantization + serving architecture.

For an air-gapped, zero-data-leakage on-premises deployment, your choice of model depends heavily on your available hardware (GPU VRAM/RAM) and your primary use case (general assistant, deep reasoning, or code generation).

The top open-weight enterprise model families that fit entirely behind your firewall feature permissive licensing (Apache 2.0 or similar) and can be easily served locally using tools like Ollama or vLLM.

1. Best All-Around / General Purpose

• The Qwen3 Family (Alibaba / Qwen)
  • Why it fits: Qwen3 has emerged as a premier default choice for local on-prem deployments, striking a phenomenal balance across reasoning, multilingual tasks, and tool use.
  • Sizing Options:
    • Qwen3 (8B / 14B): Runs exceptionally well on a single consumer or mid-tier workstation GPU (like an RTX 4090 or Mac Studio), making it ideal for standard office automation or internal chats.
    • Qwen3 (30B+): Better for multi-server setups or heavy enterprise hardware, offering near-frontier reasoning capability without calling external APIs.
  • License: Apache 2.0

2. Best for High-Performance Reasoning & Complex Logic

• Mistral Large 3 / Mistral Small (Mistral AI)
  • Why it fits: Mistral specializes in robust enterprise open-weights with strong multilingual capabilities and deep reasoning. Their architectures are fine-tuned for real-world enterprise logic, strict formatting, and compliance-heavy environments (natively GDPR-friendly if data sovereignty matters).
  • Sizing Options: Use smaller Mistral variants for high-throughput, low-latency tasks, or Mistral Large for intensive logic and a massive 256k context window.
  • License: Open-weights/enterprise-friendly

3. Best for Coding & Technical Workflows

• Qwen3-Coder / DeepSeek V4-Pro variants
  • Why it fits: If your on-prem deployment is primarily for internal software development, code completion, or interacting with local repositories, specialized coding open weights score near the top of SWE-bench categories while remaining runnable on single-to-few enterprise GPU nodes.

Deployment Stack Recommendation

To keep 100% of the data inside your network, package your chosen model with:

• Inference Engine: vLLM (for high-throughput multi-user production) or Ollama / LM Studio (for simpler, containerized local setups).
• Hardware: Ensure you match the quantization (e.g., Q4_K_M or Q8_0 GGUF formats) to your VRAM constraints so the model runs locally without spilling over slowly into system RAM.

To help narrow down the exact model and sizing, tell me:

• What hardware/GPUs do you have available on-prem?
• What is the primary use case (e.g., RAG over internal docs, code assistant, general chatbot)?

If you mean strict on-premises deployment where prompts, documents, and outputs never leave your network, the OpenAI model family to look at is gpt-oss.

• gpt-oss-120b — my default recommendation for production if you have the GPU capacity. It is the more capable model and is designed to run on infrastructure you control.
• gpt-oss-20b — choose this when latency, hardware footprint, or cost is more important than maximum capability.

Both are open-weight, Apache 2.0 licensed models and are explicitly intended for on-prem/private-cloud deployment. When self-hosted, OpenAI says it does not receive or process the data sent to the model.

Important distinction: these are not models you call through the OpenAI API. You download the weights and run them yourself, using infrastructure such as vLLM, Ollama, or llama.cpp.

If you tell me your GPU setup (e.g. 2×H100, 4×A100, CPU-only) and whether this is RAG, coding, agents, or general enterprise chat, I can recommend the specific model/quantization and deployment architecture.

If “no data leaves our network” is a hard requirement, I’d use an open-weight model deployed entirely on your own infrastructure, rather than an OpenAI API model. OpenAI’s current self-hostable option is gpt-oss.

My recommendation

gpt-oss-120b is particularly interesting for this requirement: OpenAI says it is designed to run on infrastructure you control, is Apache 2.0 licensed, and can be deployed with common self-hosting stacks such as vLLM, Ollama, and llama.cpp. OpenAI reports ~117B total parameters with ~5B active, and says it can fit on a single 80GB GPU.

The 20B version is the better choice if you have tighter hardware/latency constraints.

One important distinction: OpenAI's hosted models with data-retention/inference-residency controls are not the same thing as air-gapped/on-prem deployment. If your security requirement literally says the prompt, documents, embeddings, and outputs must never traverse your perimeter, self-hosting the weights is the cleanest architecture. OpenAI explicitly states that its gpt-oss self-hosted models don't send your data to OpenAI unless you explicitly share it or use a managed hosting partner.

If you tell me (1) your GPU(s)/VRAM, (2) number of concurrent users, and (3) workload—RAG, coding, agents, document extraction, etc., I can recommend the exact model + quantization + serving stack for your on-prem setup.

For a strict on-premise deployment where zero data leaves your network , the "best" model depends entirely on your available hardware (VRAM/RAM) and whether you need a general assistant, a coding specialist, or an agentic reasoning engine.

Because proprietary APIs are off the table, you should look at top-tier open-weights models paired with a local runtime like Ollama or vLLM.

The best-performing options are categorized by your infrastructure constraints:

1. The Single-GPU / Mid-Size Sweet Spot (24GB–48GB VRAM)
   • Model: Qwen 3.6 27B or Gemma 4 (31B)
   • Why: These dense mid-sized models strike the optimal balance for general enterprise tasks, summarization, and RAG (Retrieval-Augmented Generation). They easily fit on a single enterprise GPU or a high-end workstation, delivering fast token generation speeds without needing a clustered server rack.
   • License: Permissive (Apache 2.0 / Google terms).
2. The Heavy-Duty Coding & Long-Context Agent (Multi-GPU / Server Node)
   • Model: GLM-5.2 or DeepSeek-V4-Pro / DeepSeek-R1
   • Why: If your on-prem use case involves multi-file software engineering, repository analysis, or deep chain-of-thought reasoning, these models trade blows with closed frontier systems. GLM-5.2 features a massive context window (up to 1M tokens) under a clean MIT license, making it ideal for ingesting entire codebases or internal documentation locally.
   • Hardware note: Requires a beefier multi-GPU node (like 2x-4x A100/H100 or equivalent high-VRAM setup) or aggressive quantization (GGUF/AWQ).
3. The Low-Resource / Edge Deployment (12GB–16GB RAM/VRAM)
   • Model: Phi-4 (14B) or Gemma 4 (12B)
   • Why: Perfect if you are testing on local developer laptops or modest edge hardware while retaining strict data privacy. They punch well above their weight class for basic Q&A, drafting, and light scripting.

To narrow this down to a specific recommendation , tell me:

• What hardware do you have available on-prem (e.g., number and type of GPUs, total RAM)?
• What is the primary use case (e.g., internal document RAG search, code assistance, general chatbot)?

If your requirement is true on-prem deployment where no prompts, documents, or outputs leave your network, the OpenAI model to look at is gpt-oss rather than the hosted GPT models.

• gpt-oss-120b — my default recommendation if you have serious GPU capacity and want the strongest reasoning capability. It is designed to run on infrastructure you control.
• gpt-oss-20b — better when latency, hardware footprint, or cost is the priority.
• Both are open-weight, can be run entirely on your own infrastructure, and are available under Apache 2.0 subject to the gpt-oss usage policy. OpenAI explicitly says it does not receive or process data sent to these self-hosted models unless you explicitly share it or use a managed hosting partner.

If by "no data leaving our network" you mean air-gapped / zero external network connectivity, I'd choose gpt-oss-120b and deploy the inference stack entirely inside your environment.

By contrast, GPT-5.x via the OpenAI API is not an on-prem model. OpenAI offers regional data residency and, for eligible customers, in-region GPU inference, but that still means processing occurs on OpenAI/cloud infrastructure rather than inside your network.

If you tell me your GPU setup (e.g. 4×H100, 8×H100, A100s, MI300X, etc.) and workload (RAG, coding agent, document extraction, general chat), I can recommend the specific deployment architecture and quantization.

For a secure, 100% on-premise deployment where no data leaves your network , the right open-weight model depends heavily on your available hardware (VRAM/RAM) and your primary use case.

Top open-weight models categorized by performance and infrastructure requirements offer distinct advantages for enterprise local deployment:

• Best Overall for Enterprise / General Intelligence: GLM-5.2 or Qwen3 (e.g., Qwen3.6-27B Go to product viewer dialog for this item. or larger MoE variants) . These models offer top-tier reasoning, instruction-following, and multilingual performance that closely rivals proprietary frontier systems.
• Best for Coding & Technical Workflows: DeepSeek-V4-Pro Go to product viewer dialog for this item. (for heavy-duty repository-level reasoning and complex architecture tasks) or Kimi K2.7 Code / Qwen3-Coder-Next (for snappy developer tool integrations and IDE assistance).
• Best for Single Consumer GPU / Mid-Tier Workstations: Google Gemma 4 (31B Go to product viewer dialog for this item. or 26B-A4B MoE) or Mistral Small 3.2 (24B)Go to product viewer dialog for this item. . These strike an exceptional balance, fitting comfortably on local multi-GPU rigs or high-end single cards (like an RTX 3090/4090 or Mac Studio) while delivering near-large-model capability.
• Best Lightweight / Edge Deployment: Google Gemma 4 E4B Go to product viewer dialog for this item. or Microsoft Phi-5 (14B)Go to product viewer dialog for this item. . Perfect if you are constrained to a standard laptop or localized edge device with minimal VRAM.

Recommended Local Stacks

To serve these models safely offline without leaking telemetry, pair them with an open-source local inference runtime that exposes an OpenAI-compatible API:

• Ollama : Ideal for command-line management, Docker setups, and lightweight containerized microservices.
• LM Studio : Excellent if your team prefers a clean graphical interface to test and manage different quantizations locally.
• llama.cpp : The gold standard for raw, high-performance CPU/GPU hardware optimization.

To narrow down the exact model and sizing you should pull, let me know:

• What hardware do you have available (e.g., specific GPUs, total VRAM, or Apple Silicon unified memory)?
• What is the primary use case (e.g., internal document RAG/chat, automated coding, or general text generation)?

If you mean OpenAI models specifically, for a true on-prem deployment where inference data never leaves your network, the answer is gpt-oss rather than the hosted GPT models.

• gpt-oss-120b — my default recommendation for production. It is OpenAI's larger open-weight reasoning model, designed to run on infrastructure you control. OpenAI says it can run within 80 GB of GPU memory, making a single H100-class GPU a viable starting point.
• gpt-oss-20b — use this when latency, hardware cost, or edge deployment matters more than maximum capability. It can run in roughly 16 GB of memory.

OpenAI explicitly states that self-hosted gpt-oss models run on infrastructure you control and that OpenAI does not receive or process the data sent to them. They're also Apache 2.0 licensed, including commercial use subject to the usage policy.

What I'd choose

One important distinction: "on-prem" isn't enough by itself to guarantee no data leaves the network. You'd want the inference server, model, RAG/vector database, telemetry, and any tools/connectors behind your firewall, with outbound network access disabled or tightly controlled.

If you tell me your GPU/server specs, number of users, and what you're using it for (RAG, coding, customer support, agents, etc.), I can recommend the specific model size, quantization, inference stack, and hardware.

For a strict on-premises deployment where absolute data privacy is required (no telemetry, no external API calls, and zero data leaving your network), the best open-weight model to run depends heavily on your available hardware (VRAM/RAM) and your primary use case.

The top-performing open-weight model families and sizes categorized by hardware constraints are outlined below:

1. Best All-Rounder & General Assistant

• Qwen3 Family (Alibaba)
  • Why pick it: It features a very permissive Apache 2.0 commercial license, handles over 100 languages, and matches or beats legacy proprietary baselines in reasoning and coding. Mixture-of-Experts (MoE) variants activate very few parameters per token, making them blisteringly fast on local hardware.
  • Size to run:
    • 8B / 14B variants: Perfect for a single consumer GPU or high-end workstation.
    • Qwen3-30B-A3B (MoE): Excellent balance, fitting comfortably on a single enterprise/server GPU (like an 80GB A100 or dual-GPU rig) while offering near-frontier performance with only ~3B active parameters per token.

2. Best for Coding, Agents, & Technical Workflows

• GLM-5.2 or Qwen3-Coder-Next
  • Why pick it: GLM-5.2 leads on long-horizon context tasks, repository-level code generation, and complex terminal/agentic workflows. Qwen3-Coder-Next is exceptionally optimized for localized software development environments with massive context lengths (up to 256K+).

3. Best for Advanced Reasoning & Logic

• DeepSeek-R1 Distilled Variants
  • Why pick it: If your on-prem use case demands deep chain-of-thought, math, or complex analysis, the distilled versions of DeepSeek-R1 bring advanced reasoning behaviors directly to local hardware.
  • Size to run:
    • 7B / 14B Distills: Run easily on standard workstation hardware.
    • 32B / 70B Distills: Ideal if you have dedicated local multi-GPU servers (24GB to 48GB+ VRAM floors).

Recommended Local Serving Stacks

To keep everything entirely offline and secure, pair your chosen weights with a local inference engine:

• Ollama : Best for developer workflows, easy CLI orchestration, and serving local API endpoints.
• vLLM : Best for high-throughput production serving if you are scaling across multiple local server nodes.
• Open WebUI : Best for providing a self-hosted, ChatGPT-like interface for your internal team entirely behind your firewall.

To help narrow down the exact model and quantization level for your setup, let me know:

• What GPU/hardware specs (VRAM, RAM, and number of cards) do you have available on-prem?
• Is your primary workload general chat, document RAG/search, or coding/agents?