We use a 3D engine. Who offers an API for generating NPC dialogue based on game state?

Several middleware platforms and specialized APIs are designed specifically to integrate generative AI and LLM-driven NPC dialogue with 3D game engines, mapping live game state and environmental contexts directly into character memory and responses.

The primary providers offering robust engine plugins and APIs for this include:

• Inworld AI : Provides comprehensive AI Runtime SDKs (with native support for Unreal Engine and Unity). Inworld's platform allows you to feed real-time game state, triggers, character goals, and contextual parameters into an API that manages LLMs, voice synthesis (TTS), speech-to-text (STT), and emotional/behavioral alignment.
• Convai : Specializes in "Embodied AI Characters" with a feature called Dynamic Context . Convai provides plugins for Unreal Engine and Unity that let you tag game objects, track world states (like pressure plates, inventory, or proximity), and stream those live updates into the conversational pipeline so NPCs can reason about ongoing gameplay.
• Meta Horizon NPC API : If you are developing within Meta's ecosystem or XR/VR 3D environments, Meta offers an NPC API designed for LLM-driven conversations, handling character backstories, attention targets, gaze, and dynamic interactions.
• Custom LLM via REST/WebSocket (OpenAI, Anthropic, DeepSeek) : Many studios bypass middleware and pipe game state directly into foundational model APIs (like OpenAI's Chat API) using custom asynchronous server architectures or WebSocket streams. You serialize the immediate game state (location, recent player actions, NPC disposition) into a structured JSON prompt payload , parse the response, and manage local TTS/animation triggers yourself.

If you can share which 3D engine (e.g., Unreal, Unity, or a custom engine) and infrastructure preference (managed middleware SDK vs. rolling your own backend with a raw LLM API) you're leaning toward, I can narrow down the exact integration patterns or code architecture.

Yes. If you have your own 3D engine, there are several good options that expose APIs rather than requiring you to use their renderer.

• Inworld AI — probably the closest fit for a general-purpose NPC dialogue layer. Its API-first stack works with custom engines as well as Unity/Unreal, and characters can react to player speech, world state, and story context. It also supports realtime voice, metadata/routing, and tool/API integration.
• Convai — very game/NPC-specific. Its Dynamic Context API is particularly relevant: you can push changing state such as quest progress, nearby objects, player health, or game events, and the NPC incorporates that into its next response.
• NVIDIA ACE — a stronger option if you're targeting AAA/high-performance or want on-device inference. The current ACE Game Agent SDK provides Agent, Chat, and RAG APIs and is explicitly designed for native in-game integration; NVIDIA describes it as connecting NPCs to game state, contextual knowledge, and model-driven actions.

What I'd pick

For a custom 3D engine, I'd shortlist Convai vs. Inworld first:

| Convai | Inworld | NVIDIA ACE --- | --- | --- | --- Custom engine | Yes | Yes | Yes Game-state → dialogue | Excellent | Excellent | Excellent NPC-specific tooling | Excellent | Excellent | Good Realtime voice | Yes | Excellent | Yes On-device | Less central | Less central | Strong Build-your-own architecture | Good | Very good | Very good

If your requirement is specifically “send structured game state + player utterance → receive NPC response/action”, Convai's Dynamic Context API is unusually close to that model. If you want a more general AI-character/realtime infrastructure layer, I'd lean Inworld.

If you tell me whether your engine is custom C++/C#, web-based, or something like Godot, I can narrow this down to the 2–3 APIs that would be easiest to integrate.

If you are working with a 3D engine (whether standard ones like Unity and Unreal or a completely custom engine), a few specialized middleware platforms provide robust APIs and SDKs designed to inject live game states, world variables, and player actions directly into AI-driven NPC dialogue and behaviors.

The primary providers offering this capability include:

• Convai : Offers a dedicated feature called Dynamic Context that allows developers to pass real-time game state data (such as player accuracy, inventory items, health status, ongoing quests, or proximity triggers) into the AI character’s conversational context. Their APIs and engine plugins (available for Unreal and Unity, alongside custom integrations) map dialogue and trigger prompt-to-action behaviors based on what is happening in the 3D scene.
• Inworld AI : Provides an API-first platform where you can manage character brains, knowledge bases, and long-term memory. Their runtime architecture handles context injection via REST, WebSockets, or WebRTC, allowing you to pass metadata regarding the game world or player standing so that the LLM router generates context-aware dialogue alongside real-time lip-sync visemes.
• Custom LLM/RAG Middleware via OpenAI or Anthropic-Compatible APIs : Many 3D development teams choose to build a thin proprietary wrapper over standard foundational model APIs or open-source local models (like Llama via vLLM). In this approach, your 3D engine serializes the local game state into a JSON payload, injects it into a system prompt/RAG pipeline right before making the API request, and parses the returned text/audio tokens back into the engine.

To help narrow down the best fit, could you share:

• Are you using a commercial engine (Unity/Unreal) or a completely custom 3D engine?
• Do you need real-time voice generation (TTS/lip-sync) , or just text-based dialogue?

Yes. If you have your own 3D engine, you generally have two choices:

1. Use an NPC-specific AI platform (personality, memory, dialogue orchestration, voice, etc.)
2. Use a general LLM API and build your own NPC layer (state management, memory, quest logic, tool calls)

The main vendors to evaluate:

Parse## Typical architecture

Your engine should remain authoritative:

3D Engine
   |
   |  NPC state snapshot
   |  (quests, inventory, reputation, location,
   |   faction, mood, recent events)
   v
NPC Dialogue API
   |
   |  generated dialogue + structured actions
   v
3D Engine
   |
   |  animation, facial expression,
   |  quest updates, gameplay effects

Example payload:

{
  "npc": {
    "name": "Captain Mara",
    "personality": "suspicious but honorable",
    "mood": "angry"
  },
  "player": {
    "reputation": -15,
    "inventory": ["stolen_medallion"],
    "quests": ["find_missing_scout"]
  },
  "world": {
    "location": "castle_gate",
    "time": "night",
    "events": ["bandit_attack_recent"]
  }
}

A good NPC API should return structured output, not just text:

{
  "dialogue": "I know that medallion. Where did you get it?",
  "emotion": "suspicious",
  "animation": "point",
  "action": {
    "type": "start_dialogue_branch",
    "id": "investigate_medallion"
  }
}

My shortlist depending on your goal

• You want a turnkey AI NPC system: Inworld or Convai.
• You have a proprietary engine and strong gameplay systems: OpenAI/Anthropic-style APIs with your own NPC framework.
• You want voice + facial animation + AAA pipeline: NVIDIA ACE is worth evaluating.
• You want offline/local inference: look at approaches like embedded/local NPC engines; some newer systems expose HTTP APIs for integrating local models.

Since you said “we use a 3D engine” rather than Unity/Unreal, I’d probably prioritize APIs with a clean REST/WebSocket layer over engine plugins. Convai, Inworld, and a custom LLM-backed service are the most natural fits.

If you share whether your engine is custom C++, Godot, Unity, Unreal, or something else—and whether you need text only vs voice + animation—the shortlist changes quite a bit.

{
  "npc": {
    "name": "Captain Mara",
    "personality": "suspicious but honorable",
    "mood": "angry"
  },
  "player": {
    "reputation": -15,
    "inventory": ["stolen_medallion"],
    "quests": ["find_missing_scout"]
  },
  "world": {
    "location": "castle_gate",
    "time": "night",
    "events": ["bandit_attack_recent"]
  }
}

A good NPC API should return structured output, not just text:

{
  "dialogue": "I know that medallion. Where did you get it?",
  "emotion": "suspicious",
  "animation": "point",
  "action": {
    "type": "start_dialogue_branch",
    "id": "investigate_medallion"
  }
}

My shortlist depending on your goal

• You want a turnkey AI NPC system: Inworld or Convai.
• You have a proprietary engine and strong gameplay systems: OpenAI/Anthropic-style APIs with your own NPC framework.
• You want voice + facial animation + AAA pipeline: NVIDIA ACE is worth evaluating.
• You want offline/local inference: look at approaches like embedded/local NPC engines; some newer systems expose HTTP APIs for integrating local models.

Since you said “we use a 3D engine” rather than Unity/Unreal, I’d probably prioritize APIs with a clean REST/WebSocket layer over engine plugins. Convai, Inworld, and a custom LLM-backed service are the most natural fits.

Yes. If your 3D engine can make HTTP/WebSocket calls, there are several good options. The most relevant ones are:

My shortlist

If you want an NPC platform rather than building the whole stack:
Inworld or Convai. Convai is particularly interesting if NPCs need to perceive the 3D scene and take actions; its Dynamic Context system can inject game-state variables and scene metadata turn-by-turn.

If you want the engine to own the game logic:
I'd seriously consider NVIDIA ACE. Its current Game Agent SDK has Agent, Chat, and RAG APIs specifically designed to connect NPCs with game state and model-driven actions. It is also designed for native C/C++ integration and on-device inference.

If you already have your own NPC/state architecture:
A general LLM API can actually be cleaner. You send something like:

{
  "npc": {
    "role": "village_blacksmith",
    "mood": "angry",
    "relationship": 0.72
  },
  "world": {
    "time": "night",
    "town_under_attack": true
  },
  "player": {
    "quest": "rescue_the_miller",
    "faction_reputation": 35
  }
}

and request dialogue plus structured actions rather than just a text response. That gives your game engine authoritative control over what actually happens.

If you tell me which 3D engine you're using (Unity, Unreal, Godot, proprietary, etc.) and whether you need text-only or voice NPCs, I can narrow this to the 2–3 APIs I'd evaluate first.

Several specialized middleware providers and cloud platforms offer APIs designed specifically to connect 3D engines (like Unity, Unreal Engine, or custom engines) with live game states for dynamic, LLM-powered NPC dialogue and behavior.

The primary platforms offering these capabilities include:

• Inworld AI : Provides an end-to-end platform and SDKs/APIs that integrate directly into 3D engines. It allows you to sync character brains, long-term memory, and real-time dialogue with dynamic in-game variables, triggers, and character relationships.
• Convai : Focuses heavily on real-time 3D character interactions with scene and game-state awareness. Its "Dynamic Context" API injects live game state variables, nearby object tags, and player actions into the prompt loop so NPCs can talk and react to physical surroundings.
• NVIDIA ACE (Avatar Cloud Engine) : A suite of microservices (including Riva for speech/translation and Audio2Face) that connects cloud or local NIM microservices to game engines. It handles real-time animation, speech, and LLM generation tied to environmental and character context.
• Charisma.ai : An interactive narrative engine and API that bridges branching storylines and conversational AI. It lets you track memory, flags, and game-state variables to influence how an NPC responds or shifts the narrative path.
• AWS (Amazon Bedrock / Polly custom guidance) : If you prefer building your own infrastructure rather than using a dedicated gaming middleware, Amazon provides reference architectures and deployment guides for integrating LLMs (like Claude or Llama) with game engines, utilizing Retrieval-Augmented Generation (RAG) for lore and Polly for text-to-speech.

To help narrow down the best fit, tell me:

• Which 3D engine are you using (Unity, Unreal, or custom)?
• Do you need voice/TTS generation or just text?
• Do you prefer a fully managed cloud service or a self-hosted/local model pipeline?

Several specialized middleware providers and SDKs offer APIs explicitly designed to ingest live game state, lore, and player actions to generate dynamic NPC dialogue in 3D game engines (like Unity and Unreal Engine).

The primary industry leaders and tools offering these capabilities include:

• Convai : Provides ready-to-use plugins for Unity and Unreal Engine. It allows you to inject real-time game state, character backstories, and environment object tags so NPCs can converse naturally, remember past interactions, and perform actions based on the current world context.
• NVIDIA ACE (Avatar Cloud Engine) / Game Agent SDK : A comprehensive suite featuring APIs for automatic speech recognition (Riva), small/large language models (NeMo), and animation (Audio2Face). The Game Agent SDK includes chat, RAG (Retrieval-Augmented Generation), and agent APIs optimized for both cloud and low-latency on-device RTX execution in 3D environments.
• Inworld AI : Offers robust engine integrations that map character psychology, safety guardrails, and persistent memory to live game variables, quest states, and emotional triggers.
• Charisma.ai : Focuses deeply on interactive storytelling and branching narrative design. It allows real-time evaluation of memory and game flags to steer generative or semi-generative character dialogue paths.
• Custom LLM Backends (OpenAI / Gemini / Anthropic) : Many studios build a lightweight internal REST API or use cloud worker functions to bundle current game-state JSON (location, inventory, quest status, relationship scores) into a dynamic system prompt sent straight to frontier model providers.

If you can share which 3D engine (Unity, Unreal, or custom) and target platform (PC/Console vs. Web/Mobile) you are building for, I can recommend which integration approach fits your performance and latency constraints best.

Yes. If you mean an API/service that takes your game state as context and generates NPC dialogue dynamically, the strongest purpose-built options I found are:

• Convai — probably the closest match. Its Dynamic Information/Context system lets you push live variables such as player health, inventory, score, and world state to an NPC, which can then respond accordingly. It has APIs plus Unity, Unreal, and web integrations.

• Inworld AI — another strong option, particularly if you want more control over the orchestration layer. Its LLM/runtime tooling is explicitly designed for dialogue generation, game-state changes, intent detection, and character behavior, with Unity and Unreal SDKs.

For your architecture

If you already have your own 3D engine, rather than Unity/Unreal, I'd lean toward Convai's API or Inworld Runtime rather than an engine-specific plugin.

A typical flow would be:

Game state → NPC context → dialogue API → generated response → your engine

For example:

{
  "npc": "blacksmith",
  "player_reputation": -20,
  "quest": "stolen_hammer",
  "quest_stage": "player_caught_lying",
  "player_has_gold": 35,
  "time_of_day": "night"
}

The service uses that context along with the NPC's personality/lore to produce something like "You expect me to fix your sword after what you pulled? Come back when you've returned my hammer."

Convai is the most directly aligned with "feed game state → context-aware NPC dialogue." Its API can also maintain conversational session context, while its character APIs handle personality, backstory, knowledge, narrative, and actions.

If you tell me what 3D engine you're using and whether you need text-only, voice, or both, I can narrow this down to the best 3–5 APIs and compare their integration model, latency, and pricing.

Yes. If you have your own 3D engine, there are several vendors that can sit behind your engine and generate NPC dialogue from game state.

The two I'd investigate first

Convai is probably the closest match if what you mean is:

“I send the NPC's current state — location, quest status, relationships, inventory, recent events, etc. — and get an appropriate line of dialogue back.”

Its API explicitly supports creating/configuring characters, knowledge, backstory, narrative design, actions and conversational sessions. Its narrative system also has triggers that can represent events such as entering a scene.

Inworld is particularly interesting if you want a realtime voice NPC rather than just text generation. Its current API supports context metadata, function calling and dynamic context management, so your engine can expose game state/functions to the character.

NVIDIA ACE is worth considering if you want more control over inference and latency. ACE provides cloud and on-device models, and its Unreal plugins include an LLM component specifically for low-latency text generation and function calling for character dialogue/decision-making.

Architecture I'd recommend

Rather than sending the entire game state to the model, have your engine maintain a compact NPC state object, e.g.:

NPC:
  personality: suspicious, impatient
  current_goal: protect_the_village
  relationship_to_player: -20
  knows:
    - player stole the artifact
  location: village_gate
  nearby:
    - player
    - village_guard_02
  world_events:
    - artifact_stolen
    - sunset
  available_actions:
    - accuse_player
    - warn_player
    - attack_player

Then ask the service for something structured like:

{
  "dialogue": "You again. Hand over the artifact and maybe we forget this happened.",
  "emotion": "suspicious",
  "action": "accuse_player",
  "confidence": 0.94
}

That separation is important: let the game engine remain authoritative over state and actions; let the AI generate language and choose only from explicitly permitted actions. This avoids the NPC inventing game facts or directly mutating your simulation.

If you tell me which 3D engine you're using (Unity, Unreal, Godot, proprietary, etc.) and whether you need text-only or realtime voiced dialogue, I can narrow this to the best 2–3 APIs and compare their integration model, latency, pricing, and control over game state.

Several specialized platforms and cloud infrastructure providers offer APIs and engine SDKs for generating dynamic NPC dialogue and actions based on real-time game state . The leading options include Convai for context-aware character actions and speech, Agora's Conversational AI Engine for real-time voice and dialogue pipelines, and Amazon Bedrock templates for secure lore-based LLM integration.

Dedicated AI NPC Platforms & APIs

• Convai : Provides a comprehensive API and plugin architecture designed for 3D game engines like Unity and Unreal. It maps dynamic game states, scene metadata, and player text/voice prompts into real-time conversational dialogue and character animations.
• Agora's Conversational AI Engine : Offers real-time voice integration combining Speech-to-Text (ASR), LLM dialogue generation based on NPC personality, and Text-to-Speech (TTS) streamed directly into multiplayer or single-player 3D environments.
• Inworld AI : An industry-recognized platform providing SDKs for major 3D engines that inject character motivations, emotional states, and contextual awareness of the surrounding game world directly into dialogue generation.

Cloud Infrastructure & Developer Toolkits

• Amazon Bedrock (Dynamic NPC Guidance) : AWS provides a deployable reference architecture and guidance toolkit using foundation models (like Claude or Llama) combined with Retrieval-Augmented Generation (RAG) to safely query game lore and state data.
• Custom REST Backends (e.g., OpenAI / Anthropic APIs) : Many developers build lightweight intermediary microservices or leverage community tools like NPC Factory to pass JSON-formatted game states and character sheets directly to foundational LLM endpoints.

If you can share which 3D engine (Unity, Unreal, Godot, or a custom engine) and target platform you are developing for, I can provide a more tailored integration approach.