What's the best framework for running LLMs on-device for a mobile app?

There is no single "best" framework because the right choice depends on your target platforms, your chosen model architecture, and how deeply you want to optimize hardware acceleration.

The top-tier frameworks for running LLMs on-device break down as follows, categorized by their strengths:

- **Platforms:** Cross-platform (iOS, Android, macOS, Linux, Windows).
- **Why choose it:** It is the de facto standard for running quantized GGUF models locally. It supports almost every major open-source architecture (Llama, Mistral, Gemma, Qwen) on the day they release. It runs exceptionally well on CPU and leverages Metal (iOS) and Vulkan/OpenCL (Android) for GPU offloading.
- **Trade-off:** Lower-level C/C++ implementation, meaning you will need a native wrapper or binding for your mobile app stack.[[1]](https://meetprajapati.com/blogs/running-on-device-ai-models-android-mediapipe-llamacpp-executorch/)

- **Platforms:** Android, iOS, Web, Desktop.
- **Why choose it:** Google's MediaPipe LLM Inference Guide allows you to deploy pre-optimized models (like Gemma or Phi-2) with minimal boilerplate code. It manages memory and hardware delegation (GPU/NPU) smoothly behind a clean API.
- **Trade-off:** Less flexible if you want to run arbitrary, bleeding-edge community models that haven't been converted to Google’s preferred task format.[[1]](https://meetprajapati.com/blogs/running-on-device-ai-models-android-mediapipe-llamacpp-executorch/)

- **Platforms:** iOS, iPadOS, macOS.
- **Why choose it:** If you are building strictly for Apple devices, Apple Core ML gives you uncompromised, direct access to the Apple Neural Engine (ANE) and unified memory architecture. Performance and power efficiency on iPhone and iPad are unmatched.
- **Trade-off:** Completely useless for Android; you will need a separate cross-platform strategy or native Android stack.[[1]](https://appscale.blog/ja/blog/on-device-ai-frameworks-compared-llamacpp-mlc-onnx-mediapipe-coreml-2026)

- **Platforms:** Cross-platform (iOS, Android, WebGPU).
- **Why choose it:** Built on the Apache TVM stack, MLC LLM compiles LLMs down to native machine code optimized for specific mobile chipsets (Snapdragon, Apple Silicon, MediaTek). It offers native Swift and Java/Kotlin APIs and excels at squeezing maximum tokens-per-second out of mobile hardware.
- **Trade-off:** Compilation and build pipeline can be complex to set up and debug compared to `llama.cpp`.

- **Platforms:** Cross-platform (iOS, Android).
- **Why choose it:** Meta's ExecuTorch is designed to take models straight from PyTorch training to on-device mobile execution with strong hardware acceleration support.
- **Trade-off:** Ecosystem for mobile LLMs is still maturing compared to the battle-tested `llama.cpp`.[[1]](https://meetprajapati.com/blogs/running-on-device-ai-models-android-mediapipe-llamacpp-executorch/)

Quick Decision Matrix

• Choose Core ML if you are building an iOS-only app and need absolute peak Neural Engine performance.
• Choose MediaPipe if you are on Android/cross-platform and want the fastest, lowest-friction integration with standard models like Gemma.
• Choose llama.cpp or MLC LLM if you need cross-platform support, custom quantized models, and maximum community support.

To help narrow this down, tell me:

• What is your target platform (iOS only, Android only, or Cross-platform like React Native/Flutter)?
• Which specific LLM or size (e.g., 1B, 3B, 7B parameters) are you hoping to run?

If you're building a mobile app that needs to run LLMs fully on-device, my default choice in 2026 is ExecuTorch.

My recommendation

Why ExecuTorch: it gives you a fairly unified runtime across Android and iOS while still exposing hardware-specific acceleration. On Android it supports XNNPACK, Vulkan, Qualcomm, MediaTek and other backends; on iOS it can use Core ML/Apple Neural Engine, MPS, or CPU.

It also has LLM-specific tooling for exporting models, quantization, tokenization, and running them through C++, Java/Kotlin, or Swift/Objective-C.

When I'd choose something else

Choose llama.cpp if your priority is:

• GGUF models
• supporting lots of models quickly
• maximum control over inference
• a relatively simple native C/C++ runtime
• you don't want to build around PyTorch

Choose MLC LLM if raw GPU-oriented performance and compiler optimization are especially important. It targets both iOS/iPadOS and Android and provides a unified inference engine with native mobile bindings.

Choose Core ML if you're iOS-only and want to exploit Apple's hardware as directly as possible. ExecuTorch's Core ML backend itself can dispatch computation across CPU, GPU, and Neural Engine.

What I'd build

For a new iOS + Android app, I'd probably use:

                Your App
                   │
          ┌────────┴────────┐
          │                 │
       iOS UI           Android UI
          │                 │
          └────────┬────────┘
                   │
             LLM abstraction
                   │
              ExecuTorch
             /           \
        Core ML          QNN/Vulkan
       Apple ANE       Android GPU/NPU

I'd start with a small 1–4B instruct model, quantized appropriately for the target devices, rather than trying to cram a 7B+ model onto every phone.

One important caveat: the framework isn't usually the biggest performance decision. Model architecture, quantization, KV-cache strategy, context length, and—especially—whether you're actually hitting the phone's GPU/NPU matter enormously. ExecuTorch's own documentation emphasizes backend selection as one of the biggest determinants of performance.

If you tell me iOS, Android, or both + your target model (e.g. Llama 3.2 3B, Qwen 3 4B, Gemma 3 4B), I can give you a concrete stack and architecture I'd use for production.

There isn't one universal "best" framework — the right choice depends mostly on iOS vs Android, model choice, and whether you prioritize portability, speed, or engineering simplicity. For most mobile apps in 2026, these are the serious options.

Quick recommendation

AppScale Blog RunLocalAI## My default choices

1. Best general-purpose choice: llama.cpp

llama.cpp is usually the safest starting point.

Pros

• Runs on iOS, Android, desktop, embedded devices
• Huge model compatibility
• Mature quantization ecosystem (especially GGUF)
• Easy to bundle into native apps through C/C++ bindings
• Good Metal/Vulkan support

Cons

• More low-level engineering
• Usually won't exploit vendor NPUs as deeply as native frameworks

Use it if you want:

• Llama, Mistral, Qwen, Phi, Gemma, etc.
• Offline chat
• Local assistants
• Maximum control

Cactus Compute PocketLLM

2. Best performance-oriented cross-platform option: MLC LLM

MLC LLM takes models through a compilation pipeline and generates optimized code for targets like Metal and Vulkan.

Pros

• Very good GPU utilization
• iOS + Android + WebGPU support
• Good fit for shipping a dedicated model

Cons

• More complicated build pipeline
• Model conversion/compilation adds friction

Use it when:

• You control the model
• You care about latency and throughput
• You can invest in deployment tooling

AppScale Blog

3. Best for iPhone-only apps: Core ML (sometimes MLX)

For Apple platforms, native frameworks can win because they can access Apple's optimized hardware stack.

Apple's Core ML can use CPU, GPU, and Neural Engine paths when models convert cleanly.

Pros

• Best integration with iOS
• Power-efficient
• Strong hardware acceleration

Cons

• Model conversion can be painful
• Less flexible for rapidly changing open models

A common production approach is:

• Core ML for supported/high-value models
• llama.cpp fallback for everything else

PocketLLM

4. Best Android-native route: LiteRT / MediaPipe

LiteRT (formerly TensorFlow Lite) and Google's LLM tooling are attractive if Android is your main target.

Pros

• Good Android integration
• Hardware delegate ecosystem
• Good fit for Google's own models

Cons

• Less universal than llama.cpp for arbitrary LLMs

XCKevin

What I would ship for a real mobile app

Consumer app (iOS + Android)

• Runtime: llama.cpp
• Models: 1B–8B parameter quantized models
• Format: GGUF
• Quantization: 4-bit (Q4 variants are common)
• Add cloud fallback for harder queries

Premium iPhone-only app

• Runtime: Core ML + llama.cpp fallback

AI-first app with a fixed model

• Runtime: MLC LLM

Enterprise app with multiple model types

• Runtime mix: - LLM → llama.cpp / MLC
• Vision/audio → ONNX Runtime or LiteRT

---

Practical constraints matter more than the framework

For mobile, the biggest decisions are usually:

• Model size: 1B–4B models are much easier to ship; 7B+ needs careful memory management.
• Quantization: INT4/4-bit models are usually the sweet spot.
• RAM budget: A model that loads fine on a flagship phone may crash on older devices.
• Latency target: Time-to-first-token matters more than raw tokens/sec for chat UX.

AppScale BlogIf I were starting a new cross-platform mobile app today, I'd prototype with llama.cpp first, then benchmark MLC LLM if latency becomes a problem. For an iOS-only product, I'd evaluate Core ML early because the hardware integration can be worth the extra conversion work.

The best framework for running Large Language Models (LLMs) on-device depends heavily on your target operating system and existing tech stack . There is no single "one-size-fits-all" winner, but llama.cpp stands out as the overall community favorite for cross-platform flexibility, while Meta's ExecuTorch is leading production-grade optimization for mainstream apps.

Framework Comparison Matrix

🏆 The Top Frameworks Broken Down

1. llama.cpp (The Community Gold Standard)

If you want to support a massive range of open-source models out of the box, use llama.cpp . It utilizes .gguf quantization, making it incredibly lightweight on memory.

• Integration: You will need to write native bindings or use community-maintained wrapper projects like Maid for Flutter to interface with your mobile UI.
• Why pick it: It works on almost everything and updates within hours of new model releases.

2. ExecuTorch (The Production Pick)

Backed by Meta, ExecuTorch is designed specifically to handle on-device AI deployment on mobile and edge devices. It specializes in mapping heavy math onto phone Neural Processing Units (NPUs).

• Integration: Pairs well with optimization platforms like Unsloth to fine-tune a model (like Qwen or Llama), quantize it, and deploy cleanly to iOS and Android.
• Why pick it: Essential if your app requires heavy use of specialized hardware chips rather than just draining the phone's battery via CPU.

3. MLC LLM (The Performance Pick)

MLC LLM compiles models directly to native code using the Apache TVM compiler. It cuts down boilerplate code by providing native Android (Java/Kotlin) and iOS (Swift) SDKs directly.

• Integration: It ships with drop-in wrapper examples for easy integration.
• Why pick it: It squeezes incredible token-generation speeds out of mobile phone GPUs.

4. MLX Swift (The iOS Specialist)

If your app is exclusively targeting iOS, MLX Swift (via MLX Swift LM) is the fastest option available. Created by Apple's machine learning research arm, it optimizes tensor operations exclusively for Apple Silicon.

• Integration: Drop the Swift Package right into Xcode.
• Why pick it: Yields blazingly fast text generation speeds on newer iPhones.

⚠️ Reality Check: Crucial Hardware Constraints

Keep in mind that running local models on smartphones introduces aggressive bottlenecks:

• RAM is king: A typical smartphone has 6GB–12GB of RAM. The OS will forcefully crash your app if it hogs too much memory. You will be strictly limited to small, heavily quantized models (e.g., 0.5B to 3B parameters like Qwen-0.6B or Llama-3.2-1B).
• Battery and Heat: Continuous local LLM inference can heat a phone up quickly and drain battery power fast. Offload tasks to NPUs or GPUs using frameworks like ExecuTorch to reduce this overhead.

If you would like to narrow this down, let me know:

• Is your app target iOS only, Android only, or cross-platform?
• What programming language / framework (Swift, Kotlin, Flutter, React Native) is your app built on?
• Have you already selected a specific model you want to run?

If you mean a production mobile app that runs an LLM fully on-device on both iOS and Android, my pick in 2026 is PyTorch ExecuTorch.

My ranking

Why I'd choose ExecuTorch

It has become particularly compelling for mobile because it isn't merely an inference engine—it provides an export → quantize → hardware-accelerate → deploy pipeline. Its current LLM tooling supports models such as Llama, Gemma, Qwen, Phi and others, with Android and iOS bindings.

The big advantage is hardware coverage:

• iOS: Core ML / Neural Engine, MPS, XNNPACK
• Android: XNNPACK CPU, Vulkan GPU, Qualcomm NPU, MediaTek and other accelerators
• Swift/Objective-C on iOS and Java/Kotlin on Android.

That makes it substantially more attractive than building around a CPU-oriented runtime if you're expecting to optimize for different generations of phones.

When I'd choose llama.cpp instead

llama.cpp is still arguably the easiest choice if your priority is:

• GGUF models
• Lots of community models
• Minimal dependencies
• Running the same model/runtime across many platforms
• Rapid prototyping
• You don't want a PyTorch-based export pipeline

The tradeoff is that ExecuTorch gives you a more structured path toward device-specific NPU/GPU acceleration, whereas llama.cpp is especially attractive when portability and model availability matter more.

If you're building iOS-only

I'd seriously consider Apple's Core ML / Foundation Models stack rather than adding a cross-platform runtime. You get much tighter integration with Apple's hardware and OS.

What I'd use for a new app

For something like:

React Native/Flutter app + local 1–4B model + iPhone and Android support

I'd architect it roughly as:

LLM → quantization → ExecuTorch .pte → native iOS/Android runtime → thin bridge to React Native/Flutter

rather than trying to run the model through the cross-platform UI layer itself.

One important caveat: model size and quantization matter at least as much as the framework. A well-quantized 1–3B model that fits comfortably in RAM will generally make for a much better mobile product than trying to cram a 7–14B model onto arbitrary phones. ExecuTorch's own Qualcomm example currently demonstrates a 3B Llama deployment and highlights quantization specifically to reduce memory pressure.

Bottom line:
Cross-platform production → ExecuTorch.
Maximum model/runtime portability → llama.cpp.
iOS-only → Core ML/Apple's native stack.

If you tell me iOS only vs. iOS + Android, your app framework (Swift/Kotlin/Flutter/React Native), and the model you're considering (e.g. Llama 3.2 3B, Qwen 3, Gemma 3), I can give you a concrete recommendation and architecture.

There is no single "best" framework —the right choice depends entirely on your target platforms, whether you want an out-of-the-box model or a bring-your-own-model workflow, and your team's native vs. cross-platform stack.

The top on-device LLM frameworks break down by use-case and architecture:

• llama.cpp / GGML (Best for Maximum Flexibility & Control)
  • What it is: A pure C/C++ inference engine that runs quantized .gguf models on CPU, GPU, and mobile hardware.
  • Pros: Massive ecosystem, supports virtually every open-source model (Llama, Gemma, Qwen, DeepSeek), highly optimized memory mapping (mmap ), and easily wrapped via FFI for React Native, Flutter, or native code.
  • Cons: Lower-level setup required; you have to handle threading and memory limits manually.
• MLC LLM / TVM (Best for Cross-Platform Compilation & Hardware Acceleration)
  • What it is: A machine learning compilation framework that pre-optimizes and compiles model weights specifically for target mobile accelerators (Metal on iOS, Vulkan/OpenCL on Android).
  • Pros: Excellent token-per-second performance via hardware-specific operator fusion and memory planning. Great if you also want code portability to WebGPU/WASM.
  • Cons: Ahead-of-time compile step adds build friction and pipeline complexity compared to dropping in a pre-quantized GGUF file.
• Google MediaPipe / Google AI Edge (Best for Android Ecosystem & Quick Prototyping)
  • What it is: Google’s edge AI toolkit featuring an LLM Inference API designed for streamlined deployment of models like Gemma.
  • Pros: Clean, high-level Kotlin/Swift APIs that fit nicely into standard mobile lifecycles; handles hardware delegation (NPU/GPU/CPU) out of the box.
  • Cons: Tightly aligned with Google’s recommended models (Gemma family); parts of the raw local inference pipeline have historically skewed toward research/experimental use on non-Pixel hardware.
• Apple Core ML (Best for iOS-Native / Apple Silicon Deep Integration)
  • What it is: Apple’s native framework for running machine learning models, optimized directly for the Apple Neural Engine (ANE).
  • Pros: Unmatched power efficiency and hardware acceleration on iOS devices.
  • Cons: Apple-only; completely useless if your app targets Android or cross-platform web/desktop. Requires model conversion into .mlpackage formats.

To help narrow down the ideal framework, tell me:

• Are you targeting iOS only, Android only, or Cross-Platform (e.g., Flutter, React Native)?
• What is your target model size (e.g., sub-1B parameter tiny models vs. 3B–8B models)?

The best framework for running LLMs on-device depends heavily on your target platform and performance requirements, but llama.cpp (via NDK/JNI or native wrappers) is widely considered the overall industry standard for cross-platform efficiency, while MLC-LLM (Machine Learning Compilation) and MediaPipe (Google AI Edge) lead for streamlined native integration.

For Apple-exclusive applications, MLX and Apple’s native Core ML provide the tightest hardware optimization against the Neural Engine.

Choosing Your Framework Strategy

1. Cross-Platform / Maximum Control: Use llama.cpp if you are building in C++, React Native, or custom native bridges. Pair it with heavily quantized GGUF models (like Q4_K_M ) to balance memory footprint and token generation speeds.
2. Android-First Production: Use MediaPipe for out-of-the-box mobile primitives supporting text and image modalities with minimal boilerplate.
3. iOS-First Performance: Use Apple's MLX Framework or Core ML if targeting modern iPhones and iPads to tap straight into native hardware acceleration and unified memory architectures.

If you can share your target platform (iOS, Android, or cross-platform like React Native/Flutter) and the size of the model you want to run (e.g., 1B, 3B, or 7B parameters), I can recommend a specific deployment architecture or quantization level.

If you mean shipping an LLM inside a production iOS/Android app, my default choice in 2026 is Google’s LiteRT-LM—with a couple of important exceptions.

My ranking

Why LiteRT-LM? Google has evolved its on-device stack substantially: LiteRT is now a production runtime for hardware-accelerated inference, while LiteRT-LM adds the LLM-specific orchestration layer. It supports Android and iOS, with GPU acceleration on both and NPU support on Android.

Google is also explicitly positioning LiteRT-LM for cross-platform LLM deployment, including multimodality and tool use, rather than merely basic text generation.

When I'd choose something else

Choose ExecuTorch if your models originate in PyTorch and you want a clean path from training/export → mobile deployment. It has first-class Android and iOS paths and supports backends including XNNPACK, Vulkan, Core ML and MPS. Meta is also using it in production across Instagram, WhatsApp, Messenger and Facebook.

Choose MLC LLM if you're comfortable with a compiler-oriented stack and want substantial control over model compilation and hardware optimization. It provides native iOS and Android SDKs and an OpenAI-compatible API.

Choose llama.cpp if your overriding requirement is "I want to run as many different open models as possible." Its GGUF ecosystem and portability are hard to beat, although you'll generally be doing more of the integration/hardware-optimization work yourself.

Choose MediaPipe's LLM API if you want the shortest path to something like "put a small Gemma model into my Android/iOS app and generate text." Google provides an LLM Inference API specifically for mobile, including Android and iOS.

What I'd build

For a new cross-platform app today:

App → LiteRT-LM → quantized 1–4B model
→ GPU/NPU where available
→ CPU fallback
→ optional cloud fallback for tasks that exceed the device

I'd start with Gemma, Qwen, or Phi-class small models, benchmark 1B–4B variants on your actual target phones, and avoid choosing the framework solely from desktop benchmark numbers. LiteRT currently has pre-optimized support for models including Gemma, Qwen and Phi.

One particularly important consideration is whether you're targeting iOS only, Android only, or both. If you tell me your platform (iOS/Android/both), model size (e.g. 1B/3B/7B), and whether you're using Swift/Kotlin/Flutter/React Native, I can recommend the specific stack and model I'd use.

If you mean a production mobile app that runs an LLM locally on both iOS and Android, my default choice in 2026 is LiteRT + LiteRT-LM. Google has moved beyond the old TensorFlow Lite stack, and LiteRT now has first-class GPU acceleration across Android and iOS, with NPU support on Android. LiteRT-LM adds the LLM-specific runtime/orchestration layer, including streaming, tool calling, multimodality, and support for models such as Gemma, Llama, Phi, and Qwen.

My ranking

Why LiteRT-LM: it is explicitly designed as a production LLM runtime across Android and iOS, with hardware acceleration and a growing set of model architectures. Current documentation shows GPU execution on both platforms and NPU acceleration on Android.

ExecuTorch is the one I'd seriously consider instead if your model pipeline is already PyTorch-based. It has native Android and iOS paths, with XNNPACK/Vulkan on Android and Core ML/MPS/XNNPACK on iOS, and provides dedicated LLM export/quantization tooling.

For a React Native app, I'd lean toward ExecuTorch or llama.cpp via a native bridge, rather than trying to make the high-level Google APIs fit awkwardly into RN. There are already React Native ExecuTorch integrations in the ecosystem.

The architecture I'd use

Mobile app
   │
   ├── Native UI / React Native / Flutter
   │
   └── Native LLM layer
          │
          ├── LiteRT-LM
          │      ├── GPU/NPU acceleration
          │      ├── quantized model
          │      ├── tokenizer
          │      └── KV cache
          │
          └── Model
                 └── Gemma / Llama / Qwen / Phi

I'd target roughly 1–4B parameter models for broad phone compatibility, quantized appropriately. Larger 7–8B models can work on high-RAM modern phones, but memory pressure, thermal throttling, model download size, and OS memory limits become major product concerns. Mobile LLM research increasingly emphasizes designing models specifically around latency and memory constraints rather than simply shrinking desktop models.

One important distinction: don't choose the framework before choosing your model + minimum device spec. A 2B Q4 model with a 4K context requirement has a very different runtime profile from an 8B Q4 model with 16K context.

If you tell me iOS + Android vs one platform, native vs React Native/Flutter, and what model/use case you have in mind, I can give you a concrete stack recommendation (including model, quantization, runtime, and expected RAM/latency).

There is no single "best" framework ; the right choice depends heavily on your target platform (iOS vs. Android), your model pipeline , and whether you want a turnkey solution or low-level hardware control.

The top frameworks for running on-device LLMs in mobile apps are categorized below by their strengths:

• Google MediaPipe (LLM Inference API / LiteRT) — Best for fast prototyping and Android-first deployment.
  • Why choose it: Provides high-level, ready-to-use APIs for text generation and integration with models like Gemma. It handles a lot of the heavy lifting for cross-platform support (Android and iOS) and integrates smoothly with Flutter or native Android code.
  • Watch-out: It is more opinionated and tightly bound to Google’s ecosystem. True hardware NPU acceleration can be restricted depending on the specific device chipset, sometimes falling back to CPU/GPU execution.
• Meta ExecuTorch — Best for production-grade, PyTorch-native scaling.
  • Why choose it: Meta’s production-ready edge inference framework scales seamlessly from mobile to embedded hardware. It supports a massive array of backend hardware delegates—including Apple CoreML, Qualcomm Hexagon NPUs, MediaTek, and Vulkan. If your team trains models in PyTorch, the export and quantization workflow is clean and unified.
  • Watch-out: Steeper setup curve compared to high-level turnkey tools; you manage more of the execution configuration.
• MLC LLM (Machine Learning Compilation) — Best for maximum cross-platform hardware acceleration.
  • Why choose it: Powered by Apache TVM, MLC compiles open-source models (Llama, Qwen, Phi) to natively squeeze performance out of device-specific GPUs and NPUs (like Snapdragon Hexagon or Apple Metal). It also features robust community-maintained bindings for cross-platform wrappers like React Native.
  • Watch-out: Compiling and packaging specific model weights requires specialized build steps and a larger initial configuration effort.
• Core ML (+ SwiftUI) — Best for native iOS performance.
  • Why choose it: If you are building an iOS-exclusive app (or optimizing heavily for Apple Silicon/Neural Engine), Apple's native framework offers the tightest integration, lowest latency, and most reliable power-efficiency management.
  • Watch-out: Apple-only; requires maintaining a separate or native path if you plan to support Android.
• Llama.cpp (via mobile bindings / custom wrappers) — Best for ultimate control and flexibility.
  • Why choose it: The gold standard for lightweight, highly optimized GGUF-based inference in C/C++. Perfect if you want to swap out quantized open-weight models dynamically without being locked into a rigid compiler pipeline.
  • Watch-out: Requires custom wrapper implementation for mobile state management, memory mapping, and token streaming.

To help narrow down the ideal choice, tell me:

• Are you targeting iOS, Android, or cross-platform (e.g., React Native, Flutter)?
• What is your target model size (e.g., sub-1B parameters vs. 2B–3B+ like Gemma/Llama)?