Which AI labs offer smaller on-device models that work for private or edge deployments?

Several leading AI labs offer small, highly optimized on-device models specifically designed for private, offline, or edge deployments.

If you'd like, let me know:

• Your target hardware (mobile phone, Raspberry Pi, local PC, or industrial gateway)
• Whether you need text-only or multimodal (vision/audio) capabilities

I can help you select the best specific model for your project.

Yes. As of 2026, several major AI labs have deliberately built small models for local, private, and edge inference. The strongest options are:

The standouts

• Google — Gemma: Probably one of the most interesting choices if you want a genuinely small model. Google's current Gemma 4 family explicitly targets efficient local execution, with small variants designed for laptops and mobile devices and multimodal capabilities.
• Meta — Llama 3.2: The 1B and 3B models were specifically designed to fit edge and mobile devices. Meta also released quantized versions that substantially reduce memory requirements and improve inference speed, with support for Qualcomm and MediaTek hardware.
• Microsoft — Phi: Phi-4-mini is a particularly compelling small model if reasoning quality matters. Microsoft documents deployment on Windows, iPhone, Android and other edge devices using quantization, ONNX Runtime and Olive. TECHCOMMUNITY.MICROSOFT.COM Microsoft is also putting Phi-4-mini directly into Edge's experimental on-device AI APIs, where inputs and outputs remain on the device.
• Mistral AI — Ministral: Mistral's small Ministral models are aimed explicitly at edge applications, where latency, memory and offline operation matter more than maximum model capability.
• Hugging Face — SmolLM: A good option when size is the overriding constraint. The SmolLM family goes down into the sub-billion-parameter range, making it suitable for embedded and highly resource-constrained applications.

If privacy is the primary requirement

The important distinction is that "open model" doesn't automatically mean "private." You want an architecture where inference itself happens locally.

For example:

User data → local model → local output

rather than:

User data → API/cloud → model → output

Models such as Gemma, Llama, Phi and SmolLM can be run this way, provided you deploy the weights and inference runtime locally. Microsoft's Edge implementation is an especially clear example: Microsoft says the data processed by its on-device model doesn't leave the device.

My shortlist

If I were evaluating this space for a private/edge product, I'd start with:

1. Gemma 4 — strongest current candidate for a small multimodal local model.
2. Llama 3.2 3B/1B — safest bet for ecosystem, tooling and broad deployment support.
3. Phi-4-mini — excellent choice when reasoning and compactness are priorities.
4. Ministral — attractive for more serious edge/server hardware.
5. SmolLM — when the device is really constrained.

The ecosystem is also increasingly moving toward quantization + NPU/GPU acceleration, rather than simply shrinking parameter counts. That makes a 3B–4B model surprisingly practical on modern phones, PCs and edge boxes.

If you're choosing this for a specific deployment target (iPhone, Android, Mac, Windows PC, Raspberry Pi, NVIDIA Jetson, industrial edge box, etc.), I can narrow this down to the best 3–5 models and runtimes for that hardware.

Models such as Gemma, Llama, Phi and SmolLM can be run this way, provided you deploy the weights and inference runtime locally. Microsoft's Edge implementation is an especially clear example: Microsoft says the data processed by its on-device model doesn't leave the device.

Yes. As of 2026, there’s a fairly strong ecosystem of small/open-weight models intended for local, private, or edge inference. The main labs worth looking at are:

My shortlist

If your priority is privacy + deployment flexibility, I'd start with:

1. Google Gemma 4 E2B/E4B — especially compelling now. Google explicitly positions E2B and E4B for mobile and edge deployment; E2B has 2.3B effective parameters and E4B 4.5B effective parameters.
2. Microsoft Phi-4-mini — excellent capability at a small size. Microsoft documents deployment of quantized Phi models on Windows, iOS, Android and other edge devices.
3. Meta Llama 3.2 1B/3B — arguably the safest choice if you care most about ecosystem/tooling compatibility.
4. Qwen3 small models — particularly attractive for multilingual applications and coding.
5. Mistral Ministral — worth considering for European deployments and efficient inference.
6. SmolLM/SmolVLM — good when the hardware constraint is severe and you can accept lower capability.

One important distinction

"On-device" can mean two different things:

• Actually running locally: the model weights are on the phone/PC/edge box and inference happens there. This gives you the strongest privacy and offline properties.
• Private cloud/on-prem: the model runs inside your organization's infrastructure. Data doesn't go to the model vendor, but it isn't technically on-device.

For example, Microsoft's Phi-4-mini is interesting because it spans both worlds: Microsoft documents local/edge deployment, while Microsoft Edge is also using Phi-4-mini directly on the user's machine.

If you tell me your target hardware—iPhone/Android, Mac/Windows laptop, Raspberry Pi/Jetson, industrial edge box, or browser—I can narrow this down to the 3–5 models that actually make sense, including approximate RAM/VRAM requirements and licensing.

Several major AI labs and tech companies specialize in providing high-performing, lightweight Small Language Models (SLMs) designed explicitly for local, private, and edge deployments (laptops, mobile devices, IoT, and on-premise servers).

The primary labs and their standout on-device families include:

• Microsoft (Phi Family): Microsoft’s Phi series (such as Phi-3 and Phi-3.5/Phi-4 variants) focuses heavily on high-quality synthetic training data. Their Mini models (around 3.8B parameters) punch well above their weight class, matching the reasoning and coding capability of much larger legacy models while fitting comfortably on phones, tablets, and CPU-only laptops.
• Google (Gemma Family): Google offers Gemma and open-weights variants optimized via Google AI Edge for mobile and on-device ecosystems. Ranging from ultra-compact sizes (0.5B, 2B, up to 7B/9B), Gemma models are built to integrate seamlessly with Android, local execution runtimes, and hardware NPUs.
• Meta (Llama Family): Meta’s Llama 3 / 3.1 / 3.2 open-weights releases include smaller footprints (like 1B, 3B, and 8B parameters) that are widely adopted for edge computing. The smaller Llama variants are heavily optimized by the open-source community for local execution on local hardware via tools like Ollama or Llama.cpp.
• Alibaba (Qwen Family): The Qwen team provides extremely compact open models (such as Qwen2 and Qwen2.5 in 0.5B, 1.5B, and 7B scales). They excel at multilingual tasks and coding while maintaining a tiny memory footprint suitable for edge gateways.
• Mistral AI (Mistral Nemo / Codestral): While Mistral's flagship models target cloud/server setups, partnerships and specialized releases like Mistral Nemo (co-developed with NVIDIA at 12B parameters) offer exceptional reasoning-to-size ratios for localized workstation or robust edge-server deployment.
• Apple (Apple Intelligence & OpenELM): Apple designs proprietary on-device models for iOS/macOS and has released open-source edge models like OpenELM (efficient language models focused on layer-wise scaling) tailored strictly for resource-constrained device hardware.

If you are planning a specific deployment, let me know:

• What hardware target you are aiming for (mobile iOS/Android, an IoT edge device, or an on-premise local server)?
• What primary task the model needs to handle (e.g., text summarization, coding, retrieval-augmented generation)?

I can recommend the ideal model size and quantization format for your setup.

Major AI labs and open-weight contributors offer highly optimized Small Language Models (SLMs) designed specifically for local, private, and on-device edge deployments. These models trade away the massive, generalized world knowledge of 100B+ parameter models for high speed, low memory footprints, and total data privacy.

The primary AI labs and their corresponding lightweight families well-suited for edge and local hardware include:

• Microsoft (Phi Series) : Microsoft’s Phi-3 / Phi-3.5 / Phi-4 Mini models (ranging from roughly 3.8B to 7B parameters) are explicitly trained on heavily curated and synthetic datasets. They punch well above their weight class on reasoning and logic benchmarks, making them a top choice for resource-constrained Windows, mobile, or embedded pipelines.
• Google DeepMind (Gemma Series) : Google provides the lightweight open-weight Gemma family (such as Gemma 2B, 4B, and newer iterations) derived from the same foundational research as Gemini. Google pairs these with their Google AI Edge tooling and LiteRT (formerly TFLite) runtime, allowing seamless deployment across Android, iOS, web, and single-board hardware like the Raspberry Pi 5.
• Meta (Llama Series) : Meta’s Llama 3.2 and subsequent lightweight models in sizes like 1B and 3B parameters are specifically optimized for mobile and edge device deployments. They offer strong multilingual and reasoning capabilities while easily fitting into local consumer hardware or edge gateways when quantized.
• Mistral AI (Mistral & Small Series) : Known for high-efficiency architectures, Mistral offers smaller footprint models (like Mistral 7B and specialized compact variants) that balance inference speed and exceptional instruction-following capability. They integrate smoothly with local standard runtimes like llama.cpp.
• Alibaba (Qwen Series) : The Qwen team provides highly competent small-scale models (such as Qwen 2.5-3B or 7B coder variants) that excel in multilingual tasks and code-generation, making them ideal for localized enterprise tools and disconnected edge servers.
• Liquid AI (Liquid Neural Networks / LFMs) : A specialized lab focusing on fluid, device-native foundational models and state-space architectures designed to run with maximum intelligence and minimal compute directly on the edge.

If you share your target hardware (e.g., mobile device, Raspberry Pi, local enterprise server) and primary use case (e.g., text generation, vision/multimodal, coding, or offline chat), I can recommend the ideal model size and deployment runtime for your setup.

Yes. If by “AI labs” you mean organizations that publish small/open-weight models suitable for local, private, or edge inference, there are several strong options:

Particularly interesting choices

1. Google DeepMind — Gemma

Probably the broadest choice if you specifically want edge deployment. Google's current Gemma 4 has E2B and E4B variants explicitly designed for mobile/edge; Gemma 3n was also architected for low-resource devices, with selective parameter activation and multimodal input.

Google explicitly supports Android and iOS deployment, including LiteRT-LM, and Gemma can run without an internet connection.

There's also FunctionGemma, a 270M model specifically tuned for turning natural-language commands into local API/function calls—very interesting for embedded agents and private automation.

For extremely constrained applications, EmbeddingGemma is only 308M parameters and can run with under 200 MB RAM when quantized, making it useful for local semantic search/RAG rather than generation.

2. Microsoft — Phi

Microsoft's Phi family is another excellent small-model line. Phi-4-mini is aimed at the small-language-model category, and Microsoft has integrated Phi models into local Windows experiences. For example, Phi Silica is hardware-accelerated and designed to run locally on Copilot+ PCs using their NPUs.

This makes Phi particularly attractive if your target is Windows PCs / NPUs rather than arbitrary embedded hardware.

3. Meta — Llama

Llama is still one of the most important choices if you want a large ecosystem of runtimes, quantizations and community tooling. The smaller Llama variants can be deployed with things such as llama.cpp, Android runtimes and Apple Silicon tooling.

The tradeoff is that Llama's smallest models aren't necessarily as aggressively designed around phone-class hardware as Google's E-series Gemma models.

4. Qwen

Alibaba's Qwen family has become particularly interesting for edge deployments because it offers models down into the sub-billion-parameter range as well as compact multimodal and coding models. It's a good option if you want small models plus broad multilingual/coding capability.

5. Mistral AI

Mistral's small-model strategy is worth considering when you have somewhat more hardware—laptops, edge servers, workstation-class devices, etc. Its models tend to offer a good capability/size tradeoff, although they're generally less aggressively phone-oriented than Gemma's E-series.

---

If privacy is the primary requirement

I'd narrow the field to:

Phone / tablet:
→ Gemma 4 E2B/E4B or Gemma 3n

Windows NPU PC:
→ Phi

Apple Silicon Mac / local workstation:
→ Gemma, Qwen, Llama or Mistral, typically quantized

Very constrained embedded device:
→ Gemma 3 270M / FunctionGemma, or a similarly tiny specialized model

Local RAG/search rather than chatbot:
→ EmbeddingGemma 308M

The important distinction is that “small” doesn't automatically mean “on-device optimized.” Google's current Gemma lineup is unusually explicit about this: Gemma 4 E2B/E4B target mobile devices, while Gemma 3n uses architecture specifically intended to reduce memory, compute and energy consumption.

If you're evaluating these for an actual product, I can also give you a 2026 comparison of the best ~0.5B, 1B, 2B, 4B and 8B models, including RAM requirements, licenses, quantization, NPU support, and Android/iOS/Windows/Linux compatibility.

Yes. If by private/edge you mean models whose weights you can download and run entirely on your own hardware—phone, laptop, workstation, factory device, or air-gapped server—the strongest options currently come from several labs.

My shortlist

For really small devices, I'd start with:

• Gemma 4 E2B/E4B — potentially the most interesting current choice when footprint is paramount.
• Phi-4-mini — excellent if you're targeting CPU/NPU-equipped PCs or constrained hardware; published comparisons put it around 2.5–2.8 GB for local use.
• Llama 3.2 1B/3B — particularly safe choice if ecosystem/tool compatibility matters.
• Qwen's small models — excellent if multilingual support or coding is important.

For a private server/enterprise edge appliance, I'd move up to roughly 7–14B models. Qwen, Mistral, Llama, and Phi are particularly compelling there. Recent edge research also shows that pruning + quantization can substantially reduce the footprint of models such as Llama 8B while retaining much of their capability.

One important distinction: "open-weight" isn't synonymous with "open source," and licensing differs substantially. If you're embedding one commercially in a product, check the specific model's license rather than assuming all downloadable models have the same terms.

If you tell me your target hardware—e.g. iPhone/Android, Raspberry Pi, Jetson, Intel/AMD laptop, Apple Silicon, or an 8–16 GB GPU—I can narrow this to the 3–5 best models and expected RAM/VRAM requirements.

Several major AI labs and technology companies offer lightweight, open-weight, or dedicated small language models (SLMs) explicitly engineered for on-device, private, and edge deployments . These models range from sub-billion parameter architectures for mobile and IoT hardware up to robust 3B–8B parameter variants suited for local laptops and enterprise edge servers.

Key Ecosystems and Toolchains for Edge Deployment

• Google AI Edge & LiteRT: Provides tools like the AI Edge Quantizer and Model Explorer to convert PyTorch or JAX models into optimized formats that run efficiently on mobile CPUs, GPUs, and NPUs via XNNPACK and KleidiAI.
• Meta ExecuTorch: Meta’s production-ready framework designed to streamline the export and execution of PyTorch models directly onto edge devices, mobile phones, and embedded hardware.
• Quantization Formats (GGUF): Open-weight community models are frequently quantized (INT4/INT8) into formats like GGUF , allowing resource-constrained hardware (such as a Raspberry Pi or local industrial PC) to execute private inference smoothly offline.

If you can share what kind of hardware you plan to target (e.g., mobile phone, microcontroller, or local server) and your primary data modality (text vs. computer vision), I can recommend a specific model size and runtime setup.

Yes. As of 2026, there’s a fairly strong ecosystem of small/open-weight models designed to run locally on phones, laptops, embedded systems, or private servers. The main labs worth looking at are:

My shortlist

1. Google — probably the most interesting for genuinely multimodal edge AI

Gemma 4's E2B and E4B models are explicitly designed for local execution on laptops and mobile devices. They handle text and images, with native audio on the small E-series models, and have a 128K context window. Google also has FunctionGemma, a 270M-parameter model specifically tuned for local function calling and private agents.

That makes Google particularly interesting if you're building something like:

device → small local model → local tools/data → occasional escalation to a larger model

2. Microsoft — Phi is an excellent general-purpose SLM family

Phi-4-mini is around 3.8B parameters and is specifically positioned as a small language model. Microsoft is actually putting it directly into Edge's on-device Prompt API, where inference occurs locally. Microsoft also has a smaller Aion-1.0-Instruct model in prerelease Edge builds, designed for weaker hardware and CPU inference.

Phi is a particularly good choice when you want a relatively capable model without moving into the 7B–14B class.

3. Meta — Llama remains the safest ecosystem bet

Llama 3.2 1B and 3B were explicitly released for edge and mobile use. Meta says they were optimized for Arm processors and enabled on Qualcomm and MediaTek hardware.

The big advantage isn't necessarily that they're the smallest or best-performing models; it's the huge deployment ecosystem. Quantized Llama models are supported across a lot of local inference stacks, including ExecuTorch.

4. Alibaba/Qwen — excellent if you need tiny + multilingual

Qwen3 goes all the way down to 0.6B and 1.7B. The 0.6B model is Apache-2.0 licensed and can be deployed through local tooling such as llama.cpp, MLX-LM, Ollama and LM Studio. It supports 100+ languages and can switch between thinking and non-thinking modes.

For a device with very limited RAM, I'd put Qwen3-0.6B/1.7B near the top of the list.

5. Liquid AI — worth watching for really constrained devices

Liquid AI is unusually focused on edge hardware. Its LFM2-350M is only 350M parameters and is explicitly designed for IoT and embedded deployment; the family also has 700M, 1.2B and 2.6B variants. It targets CPU, GPU and NPU deployment on things such as smartphones, vehicles and embedded systems.

If your definition of "edge" is Raspberry Pi / embedded controller / vehicle / industrial device, rather than "MacBook with 32 GB RAM," Liquid is particularly relevant.

One important distinction

"On-device" and "private deployment" aren't quite the same thing:

• On-device: model runs on the phone/laptop/embedded device itself.
• Edge: model runs on a nearby gateway/server, potentially without internet access.
• Private deployment: model runs inside your own VPC/datacenter, where data stays under your control.
• Open weights: you can obtain the model and deploy it yourself—but licensing terms still matter.

For example, Google's FunctionGemma is explicitly aimed at private, local agents, while Microsoft's Edge implementation actually keeps prompts on the device.

If I were choosing today

I'd start with:

• <2 GB RAM / very constrained: Qwen3-0.6B or Liquid LFM2-350M
• ~4–8 GB RAM: Qwen3-1.7B, Gemma 4 E2B
• 8–16 GB: Gemma 4 E4B, Phi-4-mini, Llama 3.2 3B
• Need vision/audio: Gemma 4 E2B/E4B
• Need tool calling/agent actions: FunctionGemma
• Need maximum ecosystem compatibility: Llama
• Need multilingual/coding capability in a small package: Qwen
• Need extremely low-power embedded inference: Liquid LFM2

Apple is a special case: Apple has a roughly 3B on-device foundation model optimized specifically for Apple silicon and exposed through its Foundation Models framework, but it isn't offered as a conventional open-weight model that you can take and deploy on arbitrary hardware.

If you tell me your target hardware (iPhone/Android, Raspberry Pi, Jetson, Mac, Windows PC, Intel server, etc.) and RAM, I can narrow this down to the 3–5 models most likely to give you good latency and privacy.

Several major AI labs and open-source communities provide lightweight, highly capable small language models (SLMs) specifically optimized for offline, private, and on-device edge deployments.

• Google (Gemma & Google AI Edge) : Google offers the Gemma open-weight family (including ultra-compact and multimodal variants like Gemma 3n/Gemma 4 iterations around the 2B to 4B parameter mark). These models are built to integrate with Google AI Edge's LiteRT runtime, allowing efficient execution on mobile phones, web browsers, and local edge hardware like Raspberry Pi.
• Microsoft (Phi Series) : Microsoft develops the Phi family, prominently featuring models like Phi-4-mini (≈ 3.8B parameters). Known for outsized reasoning and coding performance relative to their small footprint, Microsoft deploys them natively on-device through features in the Microsoft Edge Developer Portal and Windows Copilot Runtime.
• Mistral AI (Ministral Series) : Mistral offers Ministral 3B , a compact model engineered specifically for edge and on-device use cases. It supports low-latency, localized interactions and features native multimodal capabilities (such as lightweight vision encoding) suited for real-time industrial or consumer edge systems.
• Alibaba (Qwen Small Series) : Alibaba's team provides a wide array of lightweight open-weight models (ranging from sub-1B up to 4B/9B parameters) that excel in multilingual tasks, coding, and general instruction-following, making them popular for self-hosted and edge-server architectures.
• AI21 Labs (Jamba Small Series) : AI21 offers compact architecture models like Jamba Reasoning 3B , which combine massive context windows with a small parameter footprint designed explicitly to run fast on local consumer hardware and private edge nodes.
• Hugging Face & Community Labs (SmolLM) : Hugging Face maintains ultra-small models like SmolLM2 (e.g., 360M parameters), which offer basic instruction-following and chat capabilities on extremely constrained embedded hardware and IoT devices.

If you're planning a deployment, tell me about your target hardware (e.g., mobile app, Raspberry Pi, local enterprise server) and memory/RAM constraints , and I can recommend the best-fit model and runtime framework for your project.