I want to build a mobile app powered by an on-device AI agent. What's the best framework for building lightweight, on-device agents?

If your goal is a mobile app with a genuinely lightweight, offline-capable AI agent, my current default choice would be pytorch.org—especially if you're targeting both iOS and Android.

The important distinction is that an agent is usually not a framework feature by itself. You typically want:

mobile app → agent runtime/orchestrator → on-device LLM → local tools/data

My recommendation

1. ExecuTorch — best overall for a cross-platform on-device agent

ExecuTorch is PyTorch's edge runtime and is explicitly designed for constrained devices. It supports LLMs, vision, speech, and other models, with CPU/GPU/NPU/DSP acceleration.

It has particularly good mobile coverage:

• Android: Kotlin/Java APIs plus XNNPACK, Vulkan, Qualcomm and MediaTek acceleration.
• iOS: Swift/Objective-C/C++, with Core ML, MPS and XNNPACK backends.
• LLM-specific runtime components and examples are included.
• Quantization and hardware-specific backends make it practical for keeping models small and efficient.

I'd build the agent layer yourself rather than trying to find a giant "agent framework":

┌─────────────────────────────┐
│       Mobile UI             │
│     Swift / Kotlin          │
├─────────────────────────────┤
│      Agent Controller       │
│  planning / tool selection  │
│  state / permissions        │
├─────────────────────────────┤
│        Local Tools          │
│  files • DB • calendar      │
│  camera • location • etc.   │
├─────────────────────────────┤
│        ExecuTorch            │
│     quantized local LLM      │
├─────────────────────────────┤
│   CPU / GPU / NPU / DSP     │
└─────────────────────────────┘

This gives you much more control over latency, memory, battery, privacy and tool permissions than putting a heavyweight agent framework inside the app.

Other strong choices

LiteRT is probably the main alternative I'd evaluate. Google's current LiteRT stack is explicitly positioned as a production on-device AI framework and now has advanced hardware acceleration.

I'd lean toward LiteRT when you're particularly invested in Google's Android/ML ecosystem.

llama.cpp is worth considering if your app is essentially "run a small GGUF LLM locally" rather than a broad multimodal ML application. It's extremely attractive for a small text-agent runtime, but I'd choose ExecuTorch when you expect the app to grow into vision/audio/multimodal models.

What I'd use for a new app

For example, if you're building an assistant that can say:

"Find my receipts from last month, summarize them, and tell me how much I spent." I'd use:

• UI: SwiftUI + Kotlin/Jetpack Compose, or Flutter/React Native with native AI bridges
• Agent: a small deterministic tool-calling loop you own
• LLM: ~1–4B quantized model initially
• Inference: ExecuTorch
• Storage: SQLite/local vector index if retrieval is needed
• Tools: strongly typed local functions rather than unrestricted code execution
• Cloud fallback: optional, only when the local model can't handle the task

The big architectural trick is not to make the LLM the whole agent. Keep orchestration deterministic and let the small local model handle language understanding, intent and tool selection. That makes a surprisingly capable agent possible on a phone without requiring a huge model.

If you tell me iOS, Android, or both, and what the agent needs to do (e.g. personal assistant, computer-use agent, voice agent, document agent), I can recommend a specific model + runtime + agent architecture and give you a minimal implementation.

I'd lean toward LiteRT when you're particularly invested in Google's Android/ML ecosystem.

llama.cpp is worth considering if your app is essentially "run a small GGUF LLM locally" rather than a broad multimodal ML application. It's extremely attractive for a small text-agent runtime, but I'd choose ExecuTorch when you expect the app to grow into vision/audio/multimodal models.

What I'd use for a new app

For example, if you're building an assistant that can say:

"Find my receipts from last month, summarize them, and tell me how much I spent." I'd use:

Building a mobile app powered by an on-device AI agent requires balancing two distinct layers: the inference engine (to run the tiny language or vision model locally) and the agent harness/orchestrator (to manage tool-use, memory, and loops).

Because heavy Python-based agent frameworks (like LangChain or AutoGen) cannot run natively and performantly inside mobile memory constraints, the ecosystem has shifted toward native, lightweight runtimes optimized for edge devices (phones with NPUs/GPUs).

The leading frameworks and runtimes for building lightweight, on-device mobile AI agents are categorized below by their strengths.

1. The Inference Engines (Where the Model Lives)

• LiteRT-LM (formerly TensorFlow Lite) / Google AI Edge

• Best for: Cross-platform mobile deployment (Android & iOS) with deep Google ecosystem support.
• Why it fits: Google consolidated its on-device generative AI tooling under LiteRT-LM , superseding older MediaPipe LLM APIs. It provides native mobile acceleration, optimized memory mapping, and direct support for compact models like the Gemma-3n family (2B/4B parameters).
• Pros: Excellent NPU/GPU hardware delegation out-of-the-box; very lightweight binaries.

• llama.cpp (and Mobile Bindings like llama.rn)

• Best for: Maximum performance and raw control over GGUF-quantized open-weights models (Llama 3, Phi-3, Mistral, Gemma).
• Why it fits: llama.cpp runs entirely in native C/C++ with zero external runtime dependencies (no Python or Node.js needed on-device). Community wrappers like llama.rn (for React Native) or native iOS/Android wrappers let you load small quantized GGUF models (e.g., 1B–3B Q4 variants) achieving fast token generation speeds on flagship mobile hardware.
• Pros: Highly optimized for low-memory footprints; supports almost all modern open-source small language models (SLMs).

• ExecuTorch (by PyTorch)

• Best for: Developers already building or training custom models inside the PyTorch ecosystem.
• Why it fits: ExecuTorch is PyTorch’s streamlined, lightweight edge runtime designed to scale from high-end mobile phones down to microcontrollers. It allows you to take custom vision, LLM, or speech models straight from PyTorch and export them with native hardware acceleration.
• Pros: Seamless pipeline if your training workflow is already PyTorch-centric.

2. The Agentic Layer (Handling Loops and Tools)

On mobile, you generally do not use a heavy external agent framework. Instead, you write a lightweight finite-state machine or use a minimalist functional loop in your app’s native language (Swift/Kotlin) or cross-platform code (Dart/JS) to handle the ReAct (Reason + Act) pattern:

• For Flutter Apps: Leverage Genkit for Dart or write custom functional state handlers.
• For React Native Apps: Combine llama.rn for inference with lightweight custom TypeScript logic to parse agent tool-calling schemas.

Recommendation on How to Choose

• Go with LiteRT-LM if you want an out-of-the-box solution optimized by Google with seamless support for models like Gemma on both Android and iOS.
• Go with llama.cpp / llama.rn if you want the flexibility to hot-swap different community GGUF models and need bare-metal C++ performance locally.

To help narrow down the best architecture, tell me:

• Are you building natively (Swift/Kotlin ) or cross-platform (Flutter/React Native)?
• What specific task does the agent need to perform on-device (e.g., text parsing, offline chat, local database searching)?

Building an on-device AI agent for mobile requires a split-layer approach: you need a lightweight inference runtime to execute the local small language model (SLM) on mobile hardware, coupled with a lean agentic loop (handling tool use, state, and planning) written in native or cross-platform code.

Heavy server-side agent frameworks (like LangChain or CrewAI) are generally too bloated and memory-hungry for mobile constraints. Instead, the best ecosystem options depend on your target stack and performance needs:

1. The Core Inference Runtimes (Where the Model Lives)

Before an agent can reason or use tools, it needs to run a quantized model (like Llama-3-8B-Instruct quantized, Phi-3, or Gemma-2B) efficiently on mobile CPUs/GPUs/NPUs.

• • Llama.cpp (via mobile bindings / GGUF): The gold standard for open-weight flexibility. It has an ultra-lightweight C/C++ core that ports smoothly to iOS and Android. It handles GGUF quantization brilliantly, allowing a small 1B–3B model to run with minimal thermal throttling and RAM usage.
• • Google MediaPipe (LLM Inference API): Perfect if you are targeting Android-heavy or cross-platform mobile apps with native Google backing. It provides high-level APIs to run models like Gemma locally with hardware acceleration out of the box.
• • Meta ExecuTorch : The evolution of PyTorch Mobile. It uses ahead-of-time (AOT) compilation to turn PyTorch/Llama models into streamlined .pte execution files. Ideal if you want strict production-grade memory safety and custom hardware backend delegation.
• • MLX (Apple Silicon / iOS specific): If you are building an iOS-first application, Apple's native MLX framework delivers phenomenal performance leveraging unified memory on modern Apple Neural Engines (ANE).

2. Implementing the Agentic Loop

Because standard Python agent frameworks don’t fit inside a compiled iOS/Android binary, you typically build a lightweight state machine or execution loop directly in your mobile architecture (Swift/Kotlin or React Native/Flutter):

• The Tool-Calling Pattern: Your local SLM must support function calling (e.g., using JSON mode or grammar constraints native to llama.cpp ). You define local device tools (like reading local calendar events, writing to SQLite, or fetching device status) as simple native functions.
• Custom State Machine: Write a minimal, deterministic loop in TypeScript, Swift, or Kotlin:
  1. User inputs prompt →
  2. Local model evaluates and outputs structured JSON tool call →
  3. Mobile app executes native API safely →
  4. Result is fed back into model context for final response.

To help narrow down the best setup, could you share:

• Are you building cross-platform (React Native / Flutter) or native (Swift / Kotlin)?
• What kind of tools/actions does the agent need to perform on the device?

For a mobile app with a lightweight, on-device AI agent, the best framework depends on what you mean by “agent”:

• A small LLM that plans, chats, calls tools, and runs offline
• A specialized AI feature (vision, speech, classification)
• A hybrid agent (local model + cloud fallback)

For a true on-device agent, I’d usually start with an LLM runtime + your own agent layer, rather than looking for a single “agent framework.” The runtimes handle inference; you build memory, tools, planning, and app integration on top.

Best choices for mobile on-device agents

1. llama.cpp — best default for flexibility

Best for: iOS + Android, offline assistants, prototypes that need many model choices.

Pros:

• Extremely portable (C/C++)
• Huge ecosystem of quantized models (especially GGUF)
• Good for 1B–8B parameter models on modern phones
• Easy experimentation with different models
• Works well with tool calling / structured output patterns

Cons:

• Lower-level integration work
• You manage more of the mobile engineering yourself

A typical stack:

Swift/Kotlin/React Native
        |
   Agent layer
(tool routing, memory, planning)
        |
    llama.cpp
        |
  Quantized LLM (GGUF)

For many startups, this is the safest starting point.

---

2. MLC LLM — best for optimized cross-platform performance

Best for: shipping a polished consumer app where inference speed matters.

Pros:

• Compiles models for device hardware
• Strong iOS/Android support
• Good GPU utilization path
• Designed around mobile deployment

Cons:

• More complex build pipeline
• Model conversion/compilation workflow adds friction

Choose MLC if you already know the model you want to ship and care about squeezing maximum performance from phones.

---

3. ExecuTorch — best if your team is already in PyTorch

Best for: production apps with custom models or multimodal pipelines.

Pros:

• Designed specifically for edge/mobile deployment
• Good hardware backend story
• Fits teams already using PyTorch

Cons:

• More ML engineering overhead than llama.cpp
• Less “download a model and run it” simplicity

Good choice if your agent uses more than text (vision, audio, custom neural networks).

---

4. Platform-native options

iOS-only

• Core ML is usually the best path for Apple hardware acceleration.

Android-focused

• LiteRT is a strong choice for specialized models and Android hardware acceleration.

These are excellent when you are targeting one ecosystem rather than building a portable agent stack.

---

My recommendation by app type

Agent architecture I’d use

A lightweight mobile agent usually looks like:

                Mobile UI
                   |
             Agent controller
                   |
       +-----------+-----------+
       |                       |
   Local LLM              Local memory
 (llama.cpp/MLC)        embeddings + DB
       |
 Tool executor
 (calendar, files, APIs,
 app actions)

Keep the “agent brain” small:

• 1B–4B parameter model for fast interactions
• Quantization (4-bit is common)
• Short context windows
• Deterministic tool calling
• Retrieval instead of huge memory

A 7B model can work on high-end phones, but a well-designed 2B–4B model often gives a better UX because latency, battery, and memory matter more than raw capability.

If I were starting a new iOS + Android AI agent app today, I’d prototype with llama.cpp + a 3B–4B instruct model, build the agent layer cleanly, then evaluate MLC LLM or ExecuTorch once I had real performance data.

A lightweight mobile agent usually looks like:

                Mobile UI
                   |
             Agent controller
                   |
       +-----------+-----------+
       |                       |
   Local LLM              Local memory
 (llama.cpp/MLC)        embeddings + DB
       |
 Tool executor
 (calendar, files, APIs,
 app actions)

Keep the “agent brain” small:

• 1B–4B parameter model for fast interactions
• Quantization (4-bit is common)
• Short context windows
• Deterministic tool calling
• Retrieval instead of huge memory

A 7B model can work on high-end phones, but a well-designed 2B–4B model often gives a better UX because latency, battery, and memory matter more than raw capability.

Building a lightweight, on-device AI agent for mobile requires balancing memory footprints, battery constraints , and inference latency . Rather than relying on heavy server-side orchestration frameworks (like LangGraph or CrewAI), the mobile ecosystem splits the "agent brain" and "execution layer" across native and cross-platform paradigms.

The top frameworks and architectural choices for building on-device mobile agents depend on your target stack:

• Kotlin Multiplatform (KMP): The Shared Brain Approach
  KMP is ideal if you are targeting both iOS and Android. It allows you to write the core agent logic, state machine, and tool-calling loop once in Kotlin, and compile it down to native binaries for both platforms. Pair this with lightweight runtimes like ExecuTorch or native model bindings to execute small language models (SLMs) without maintaining duplicate codebases.
• Apple SwiftUI + Core AI / Core ML: The iOS Native Path
  If your focus is exclusively on iOS, leveraging Apple's native hardware acceleration via the Neural Engine is unmatched. Apple's on-device intelligence frameworks provide direct pathways to run fine-tuned local foundation models with minimal latency. It handles secure, local context handling out-of-the-box without burning user data or incurring cloud API costs.
• Android Jetpack Compose + AICore (Gemini Nano): The Android Native Path
  For Android-first development, Google's system-level integration via AICore and Gemini Nano lets your app tap into a shared, system-managed on-device model. Combined with Android's AppFunctions library, your app can expose self-describing local functions that the on-device agent can securely discover and execute via natural language locally.
• React Native / Expo with ExecuTorch or Local Llama runtimes: The JavaScript Powerhouse
  If your team is already specialized in web/JS tech, React Native has matured significantly for edge AI. Utilizing lightweight native modules tied to mobile-optimized inference engines allows you to run quantized small language models directly in the JS/native bridge context.

To help narrow down the ideal choice, tell me:

• Are you targeting iOS only, Android only , or Cross-Platform (and what is your primary UI stack)?
• Will the agent use a pre-installed system model (like Gemini Nano / Apple Intelligence) or bundle a custom quantized SLM (like Llama-3-8B-Instruct quantized or Phi-3) inside the app binary?

If your goal is a mobile app with a genuinely on-device agent—not just an LLM chat UI—I’d separate the stack into agent runtime/orchestration and local model inference.

My top choice: MLC LLM + your own lightweight agent loop

For a cross-platform iOS/Android app, MLC LLM is one of the strongest foundations right now. It compiles models for mobile hardware and provides native iOS/Android APIs, with support for Apple GPUs on iOS and mobile GPUs on Android.

I'd architect it roughly like:

┌──────────────────────────────┐
│       Mobile App UI          │
│      Swift / Kotlin          │
├──────────────────────────────┤
│        Agent Runtime         │
│                              │
│  Goal → Plan → Tool →       │
│  Observe → Decide → Act     │
│                              │
│  • tool registry             │
│  • short-term memory         │
│  • state machine             │
│  • permissions / guardrails  │
├──────────────────────────────┤
│       Local LLM Runtime      │
│          MLC LLM             │
├──────────────────────────────┤
│  Apple GPU / Android GPU     │
└──────────────────────────────┘

The important bit: don't use a heavyweight agent framework on the phone. The agent itself can be a few thousand lines of deterministic application code. Let the LLM handle decisions, while your code owns state, tools, permissions, retries, and execution.

Other options

MLC is particularly attractive for a startup/product because its packaging system can compile the model and package the runtime/weights directly into an iOS or Android application. It also supports controlling things like context-window size and prefill chunk size, which matter a lot for mobile memory constraints.

What I'd actually build

For a new app in 2026, I'd start with:

Frontend

• SwiftUI for iOS
• Jetpack Compose for Android

Agent

• A custom finite-state/agent loop
• Structured tool calls
• SQLite for persistent state
• Small working-memory/context manager

Inference

• MLC LLM
• A 1–4B parameter quantized instruct model initially
• Model selected based on your actual task rather than simply choosing the largest model that fits

Tools

calendar
contacts
location
files
camera
notifications
app-specific APIs

Each tool should be a strongly typed native function rather than something the LLM can arbitrarily execute.

For example:

User: "Remind me to call Sarah when I get home."

Agent:
  1. infer intent
  2. call location tool
  3. create geofence
  4. schedule notification
  5. persist task

The LLM shouldn't be responsible for steps 2–5. It should decide what needs to happen; deterministic application code should decide how it happens.

One important design decision

I'd avoid building the agent around an LLM framework like LangChain/LangGraph initially. Those frameworks are excellent on servers, but mobile has very different constraints: binary size, memory pressure, battery, intermittent connectivity, OS permissions, lifecycle management, and latency.

Instead, make the agent runtime boring:

AgentState
    ↓
LLM inference
    ↓
Structured action
    ↓
Tool execution
    ↓
Tool result
    ↓
LLM inference
    ↓
Final response

That gives you much more control over battery, latency and failure modes.

If you tell me what the agent actually needs to do (e.g. personal assistant, camera/computer vision agent, offline coding assistant, health/fitness coach, smart-home controller, etc.), I can recommend the specific model + inference runtime + agent architecture I'd use, including approximate RAM/storage requirements for modern iPhones and Android phones.

Building a lightweight, on-device AI agent for mobile requires balancing strict constraints: limited RAM, thermal throttling, battery preservation, and tight inference latency. Heavy cloud-first agent frameworks (like standard LangChain or AutoGen) are far too bloated for edge mobile environments.

The optimal framework choice depends heavily on your target platform and whether you want to leverage native mobile OS intelligence or build a cross-platform solution.

Top Frameworks for On-Device Mobile AI Agents

• Google Agent Development Kit (ADK) for Android / Kotlin
  Google's ADK (with dedicated packages for Android and Kotlin) is purpose-built for embedding agentic workflows right into mobile apps. It supports hybrid orchestration , allowing a lightweight on-device model (like Gemini Nano via AICore/ML Kit) to act as a local sequential or retrieval agent, while gracefully offloading heavy reasoning to the cloud if needed. It natively supports local tool-use and state management tailored to mobile resource limits.
• Apple Foundation Models & App Intents Framework (Swift/SwiftUI)
  If your primary or exclusive target is iOS, building natively with Swift is unmatched for performance and battery efficiency. Apple’s unified Swift API for foundation models gives you direct access to the device's Neural Engine. By pairing on-device models with the App Intents framework, your agent gains deep system context, screen awareness, and local tool execution natively within the OS.
• PydanticAI (Cross-Platform/Python-to-Edge logic)
  While traditionally run in Python environments, PydanticAI has become a favorite for lightweight agent logic because its rigid type-validation ensures zero runtime surprises for tool calls. For mobile developers using Kotlin Multiplatform (KMP) or cross-platform architectures, mirroring Pydantic's strict structured output schema locally prevents smaller on-device models from hallucinating malformed tool arguments.
• ExecuTorch (Meta) / MediaPipe (Google)
  If you are dropping down to the metal to run custom fine-tuned tiny language models (SLMs) or quantized weights (<3B parameters) offline, Meta's ExecuTorch or Google's MediaPipe act as the underlying runtime engines. They allow React Native or native apps to execute tensor operations directly on mobile GPUs and NPUs without relying on a heavyweight orchestration framework.

Key Architecture Comparison for Mobile Agents

To help narrow down the ideal stack, tell me:

• Are you targeting iOS only, Android only, or cross-platform (e.g., Flutter, React Native, Kotlin Multiplatform)?
• Will the agent run 100% offline , or use a hybrid model (local tiny model + cloud fallback)?

If you mean an actual agent—an LLM that can reason, call tools, maintain state, and execute actions—not just “run an LLM on the phone,” I’d build the stack around ExecuTorch today.

My recommendation

Best overall for a new cross-platform mobile agent: ExecuTorch

ExecuTorch is unusually well suited to this because it is designed specifically for on-device deployment, has a very small base runtime, supports iOS and Android, and has hardware backends for Apple, Qualcomm, ARM, MediaTek, etc. It also has an LLM runner and multimodal support.

I'd structure it roughly like:

┌─────────────────────────────────────┐
│           Mobile App UI             │
│       Swift / Kotlin / RN            │
├─────────────────────────────────────┤
│          Agent Runtime               │
│  state • planning • tool selection   │
│  permissions • memory • execution    │
├─────────────────────────────────────┤
│            ExecuTorch                │
│       local model inference          │
├─────────────────────────────────────┤
│       Hardware acceleration          │
│   CPU / GPU / NPU / Neural Engine    │
└─────────────────────────────────────┘

The important distinction is that ExecuTorch is the inference/runtime layer, not your complete agent framework. I'd keep the agent loop yourself so you control latency, memory, tool permissions, and failure behavior.

The alternatives

llama.cpp remains my choice if your biggest requirement is “I want to download basically any quantized model and run it.” Its GGUF ecosystem and portability are excellent. The downside is that you're closer to the metal and will do more integration/performance work yourself.

LiteRT is particularly interesting if Android is your primary target. Google's current stack is explicitly optimized for on-device AI and its latest runtime has hardware acceleration, including NPU paths; Google reports significant gains over llama.cpp on some Android hardware.

For iOS-only, Apple's own Foundation Models framework is arguably the simplest answer when its built-in model is sufficient. You don't have to ship your own LLM at all.

What I'd actually ship

For a new app in 2026:

React Native UI + native ExecuTorch module + 1–3B quantized model + local tool runtime

For example:

React Native
     │
     ▼
AgentController
     │
     ├── conversation state
     ├── short-term memory
     ├── tool registry
     ├── permissions
     └── execution loop
              │
              ▼
         ExecuTorch
              │
              ▼
       1B–3B quantized LLM
              │
       ┌──────┼────────┐
       ▼      ▼        ▼
     Files   DB     App APIs

The agent controller is actually more important than the inference framework. Keep the model small and make the tool interface extremely structured. For a mobile agent, I'd generally prefer a 1B–3B model with constrained tool/function calling over trying to cram a 7B–14B model onto the phone.

One particularly attractive feature of ExecuTorch is that the same general stack can cover LLM + vision + speech, rather than requiring you to assemble three completely different inference systems.

Bottom line:

• Cross-platform production: ExecuTorch
• Maximum model flexibility: llama.cpp
• Android/Gemini/Gemma ecosystem: LiteRT-LM
• iOS-only: Foundation Models
• Apple-specific custom models: MLX/Core ML

If you tell me iOS, Android, or both and what you want the agent to actually do (e.g. manipulate app UI, search local files, act as a personal assistant, control IoT, etc.), I can recommend the exact model + runtime + agent architecture.

For a lightweight on-device AI agent in a mobile app, I’d separate the problem into two layers:

1. Agent runtime/orchestration — planning, memory, tool calls, workflows
2. On-device model runtime — actually running the LLM locally

There isn’t one universal “agent framework” for phones yet. The best stack depends on how much control you need.

My default recommendation: llama.cpp + a small agent layer

For most production mobile apps today, I would start with:

• LLM runtime: llama.cpp
• Model format: GGUF quantized models
• Agent layer: build a lightweight state machine yourself (or use a small workflow library outside the inference core)
• App integration: native Swift/Kotlin bindings, or wrappers for Flutter/React Native

Why:

• Very broad model compatibility
• Runs fully offline
• Excellent CPU support and Metal/Vulkan acceleration paths
• Mature quantization ecosystem
• You control memory, latency, and battery behavior

llama.cpp is widely used as the foundation for mobile local LLM deployments because it is portable and supports many quantized models.

A mobile agent usually does not need a huge autonomous framework. A simple loop is often enough:

User input
   ↓
Small local LLM
   ↓
Intent + tool selection
   ↓
Execute tool
   ↓
Update memory/state
   ↓
Generate response

For example:

• “Find my last receipt” → local retrieval tool
• “Summarize these notes” → local LLM
• “Create a reminder” → device API tool

---

Other strong options

1. MLC LLM — best when squeezing maximum performance

Good choice if:

• You target a narrow set of devices
• You want GPU-optimized inference
• You can handle a more complex build pipeline

MLC compiles models into optimized native execution paths rather than relying only on a generic runtime. That can improve performance on supported hardware, but deployment is more involved.

Use it for:

• Premium apps
• Known hardware targets
• High-throughput assistants

---

2. ExecuTorch — best for PyTorch-native teams

Good choice if:

• Your models originate in PyTorch
• You want a Meta-supported deployment stack
• You need broader edge ML, not just text generation

ExecuTorch is more of a general edge inference framework than an “agent framework.”

---

3. Core ML — best for iOS-only apps

If you are building only for Apple devices:

• Core ML + Metal can provide excellent efficiency
• Neural Engine acceleration can be valuable
• You get native OS integration

The tradeoff is model conversion/support complexity.

---

Agent frameworks: use sparingly on mobile

Cloud-oriented agent frameworks like large graph/workflow systems are usually too heavy to embed directly in a phone app.

Instead, implement:

Agent state

conversation
user preferences
task status
cached results

Tools

calendar()
contacts()
files()
camera()
local_database()
search_index()

Planner

A small model decides:

{
  "action": "calendar.create_event",
  "arguments": {
    "title": "Dentist",
    "time": "Friday 3pm"
  }
}

Structured tool calling is generally more reliable than asking a tiny model to freely reason.

---

Model sizes that are realistic on phones

Typical sweet spot:

For agents, a small model with good tool constraints often beats a larger model doing everything.

Examples worth testing:

• Gemma 3 small variants
• Phi models
• Qwen small instruct models

---

Suggested stacks by app type

Offline personal assistant

Swift/Kotlin
+
llama.cpp
+
3B–4B quantized model
+
SQLite/vector store
+
native device tools

Flutter app

Flutter UI
+
native llama.cpp bridge
+
GGUF model
+
Dart agent controller

React Native

React Native
+
native C++ inference module
+
llama.cpp wrapper
+
JS agent orchestration

iOS-only premium app

Swift
+
Core ML / Metal
+
Apple-optimized model

---

If I were starting a new lightweight mobile agent project today, I’d pick:

Cross-platform: llama.cpp + GGUF + custom agent loop
iOS-only: Core ML where supported, llama.cpp fallback
Performance-focused: MLC LLM
Enterprise edge ML: ExecuTorch

The biggest engineering challenge is usually not the LLM—it is designing the agent’s tool interface, memory model, and failure handling so a small local model can behave reliably.

Building a mobile app with an on-device AI agent requires splitting your architecture into two components: the Inference Engine (which runs the lightweight weights locally on the phone's NPU/GPU/CPU) and the Agent Orchestrator (which handles the tool-calling, state, and reasoning loops).

Because mobile devices have strict RAM, thermal, and battery limits, the "best" framework depends heavily on your target ecosystem and whether you are deploying cross-platform or native.

Phase 1: The On-Device Inference Runtimes (Where the Model Lives)

• llama.cpp / GGUF: The absolute standard for cross-platform local inference. Written in pure C/C++, it runs quantized 1B to 3B models (like Gemma 3 1B [1.15] or SmolVLM) smoothly on both iOS and Android. If you are building with React Native or Expo, you can leverage llama.rn bindings [1.23].
• MLX (MLX-Swift): If you are focusing strictly on iOS and modern Apple Silicon, Apple's open-source array framework [1.12] combined with MLX-Swift offers blazing-fast execution utilizing unified memory [1.13].
• LiteRT-LM (formerly TFLite): Google’s optimized runtime. It is exceptionally well-tuned if you plan to stick primarily to the Gemma model ecosystem on Android devices [1.13].
• ExecuTorch: Meta's production-oriented on-device inference framework (the evolution of PyTorch Mobile). It provides a clean path from PyTorch-trained models to cross-platform mobile execution with hardware-aware optimization.
• Core ML / Apple Foundation Models: Apple's native ecosystem choice [1.19]. Ideal if you want zero-dependency, deep OS-level integration and maximum battery efficiency via the Apple Neural Engine (ANE) [1.19].

Phase 2: The Agent Orchestration Layer (The Brain & Loop)

Traditional server-side agent frameworks (like LangGraph or CrewAI) are too heavy for mobile runtime environments. For lightweight, on-device loops, you want a deterministic state-machine approach:

1. Lightweight ReAct (Reasoning + Acting) Loops: Write a clean, custom asynchronous state machine in Kotlin/Swift rather than importing a massive framework. Mobile agents usually only need a 3-step loop: Perceive input -> Select local tool (AppFunction/API) -> Update UI state.
2. PydanticAI / Function Calling Runtimes: Use strict JSON/structured outputs from your local model runtime (many GGUF/MLX engines support grammar-constrained generation). Ensuring the model outputs rigid schema arguments locally prevents agent hallucinations and broken loops on-device.
3. Platform-Specific Hooks:
   • On Android, look into experimental platform features like AppFunctions (Android's local Model Context Protocol equivalent), which let your app expose its internal capabilities directly as tools for on-device agents.
   • On iOS, lean on native intents and tool-calling structures supported by Apple's local intelligence layers.

Recommendation Summary

• For Cross-Platform (React Native / Flutter): Use llama.cpp via native bindings for inference, paired with a custom, lightweight TypeScript/Kotlin state machine for the agent loop [1.23].
• For iOS Native: Build with SwiftUI + Core ML or MLX-Swift for direct hardware acceleration and minimal memory overhead.
• For Android Native: Use LiteRT-LM or llama.cpp integrated with modern Kotlin and Jetpack Compose [1.23, 1.23].

To help narrow down the ideal stack, tell me:

• Are you targeting iOS only, Android only , or cross-platform (React Native/Flutter)?
• What specific tasks should the agent perform on-device (e.g., local RAG search, UI automation, offline chat)?