We want robots to learn new warehouse tasks from natural language instructions. What platforms are best?

If your goal is specifically “give a robot a new warehouse task in plain English, without manually programming every motion”, I’d separate the stack into two layers: natural-language/robot foundation models and robot-learning/simulation infrastructure.

Best platforms to evaluate

1. Google Gemini Robotics — best fit for your actual goal

This is probably the most directly aligned with your requirement. Google's current Gemini Robotics 2 stack combines an embodied-reasoning model for planning with a vision-language-action model for physical control. Google explicitly describes it as handling everyday commands, unfamiliar tasks, long-horizon plans, and multiple robot embodiments.

For example, your warehouse worker could say:

“Pick the blue carton from the inbound pallet, place it in bin B14, and put the damaged-looking boxes aside.” The system can potentially turn that into perception → planning → manipulation actions rather than requiring a pre-programmed sequence.

Big caveat: availability is still partner/private-preview oriented for some of the most capable models, so I'd treat this as a platform to evaluate with Google rather than assuming you can deploy it immediately on arbitrary warehouse robots.

2. NVIDIA Isaac — best foundation for building the system yourselves

If you're building a serious warehouse robotics platform, Isaac is probably the infrastructure I'd choose underneath the intelligence layer.

Isaac provides simulation, robot learning, perception, CUDA-accelerated robotics libraries, ROS 2 integration, and deployment tooling. Isaac Lab specifically supports imitation learning and reinforcement learning at GPU scale.

That gives you a useful architecture:

Natural language → reasoning/VLA model → task plan → Isaac/ROS → motion planning → robot

You can train and test things like:

• bin picking
• pallet unloading
• tote sorting
• packing
• object relocation
• navigation + manipulation
• recovery from failed grasps

before putting them on expensive warehouse hardware.

3. Hugging Face LeRobot — best open-source starting point

LeRobot is particularly interesting if you want robots to learn from demonstrations rather than relying exclusively on reinforcement learning.

Its workflow is essentially:

Teleoperate → record demonstrations → train policy → deploy

It supports datasets containing synchronized video/action/state data and increasingly capable vision-language-action policies such as SmolVLA and π₀.

This is attractive for a warehouse because you could imagine an operator demonstrating:

“This is how we handle this new type of package.” Then collecting demonstrations and fine-tuning/deploying a policy.

It's less of a complete enterprise warehouse platform than Isaac, but potentially much more hackable.

4. ROS 2 + MoveIt — essential plumbing, not the intelligence layer

For production warehouse robots, I'd still expect ROS 2 and MoveIt-style motion planning somewhere underneath the AI layer.

They solve different problems from a VLA model:

• VLA/LLM: “What should I do?”
• task planner: “What sequence of actions achieves it?”
• MoveIt/motion planner: “How do I move the arm safely?”
• robot controller: “Execute these trajectories.”

So I wouldn't choose ROS vs. Gemini vs. Isaac as mutually exclusive technologies. A strong system can use all three.

What I'd build

For a warehouse startup/product, my preferred architecture today would be:

Natural language ↓ Embodied reasoning model Gemini Robotics ER or comparable model ↓ Task planner / warehouse API inventory, WMS, SKU information, bin locations, safety rules ↓ VLA policy vision + language → robot actions ↓ ROS 2 / Isaac ROS ↓ MoveIt / cuMotion / controller ↓ Robot

And use Isaac Sim + Isaac Lab to generate simulation data, test policies, and train policies before deployment. NVIDIA explicitly positions Isaac Lab for large-scale imitation/RL training and sim-to-real workflows.

One important distinction

If by “learn new tasks” you mean:

A. “The operator can describe a task the robot has never explicitly been programmed for, and the robot figures it out.”

→ Gemini Robotics is the most directly relevant technology to investigate.

If you mean:

B. “An operator demonstrates a new physical manipulation task 10–100 times and the robot learns that skill.”

→ LeRobot + a VLA/imitation-learning model + Isaac Lab is arguably the more practical research/development stack.

If you mean:

C. “We need a production warehouse platform where dozens/hundreds of robots can safely execute dynamically generated tasks.”

→ I'd build around NVIDIA Isaac/Isaac ROS + ROS 2, with a VLA/embodied model sitting above it.

The exciting part is that A + B + C are starting to converge. Google's current models explicitly target unfamiliar tasks and natural-language redirection, while NVIDIA and Hugging Face provide the infrastructure for training and deploying the underlying robot policies.

If you're evaluating this for an actual warehouse deployment, I can also give you a 2026 vendor/platform shortlist (Google vs NVIDIA vs Physical Intelligence vs Figure/Apptronik vs open-source) and score them on natural-language learning, manipulation, simulation, hardware compatibility, data requirements, latency, and commercial readiness.

The best platforms for training warehouse robots using natural language instructions rely on Vision-Language-Action (VLA) models, ROS-based agentic frameworks , and simulation-to-real AI control layers.

Key Platform Capabilities

• Vision-Language-Action (VLA) Models: These systems connect what a robot sees through its cameras with a large language model's understanding of text, allowing the robot to translate "pick up the red box" directly into joint movements.
• Tool-Calling Frameworks: Software layers that let an AI agent invoke specific robot skills (like move_to_bin or scan_barcode ) safely via plain-text prompts.

If you can share the specific type of hardware you are using (e.g., robotic arms, mobile AGVs, or humanoids) and whether you have a simulation environment set up , I can recommend a more tailored software stack.

If your goal is specifically “give a robot a natural-language warehouse instruction, and have it figure out the perception + manipulation + motion”, I’d focus on vision-language-action (VLA) / robot-foundation-model platforms, rather than conventional warehouse automation software.

My shortlist

1. NVIDIA Isaac is probably the strongest fit

NVIDIA's Isaac stack is unusually well aligned with what you're describing.

The combination I'd investigate is:

Natural language → GR00T VLA → Isaac Lab/Sim → robot

GR00T accepts video, a natural-language command, and robot state, and produces action chunks. NVIDIA also says the models can be adapted to new hardware using relatively small collections of task demonstrations.

For warehouse work, that's compelling because you could imagine instructions such as:

“Take the blue carton from the inbound pallet and put it in the empty bin on shelf three.” rather than having to hard-code a separate program for every SKU/location combination.

Isaac Lab gives you the other half of the stack: GPU-accelerated simulation, imitation learning, reinforcement learning, synthetic data, and large-scale policy training.

2. Google DeepMind's Gemini Robotics

Google DeepMind is worth evaluating if your primary research question is how far language/multimodal models can generalize to new physical tasks.

I'd view this more as a model/research direction than a complete warehouse deployment platform. If you're building the surrounding data collection, simulation, safety and robot-control infrastructure yourself, it is highly relevant.

3. Physical Intelligence

Physical Intelligence is another particularly relevant benchmark for your idea. Its work is centered on general-purpose robot policies that can learn many tasks rather than requiring a bespoke controller for every task.

This is the direction I'd study if your core hypothesis is:

“Can demonstrations + language instructions replace task-specific robot programming?”

The important distinction is that these systems still generally need grounding data and demonstrations. Natural language alone isn't magic—the robot needs to have learned enough manipulation primitives to translate “put this in that bin” into reliable physical behavior.

4. LeRobot / Hugging Face

Hugging Face's LeRobot ecosystem is attractive if you're a research team and want to own the training pipeline rather than buy a robotics platform.

I'd consider it especially for:

• collecting teleoperation demonstrations
• experimenting with imitation learning
• fine-tuning open VLA policies
• comparing different robot embodiments
• building your own warehouse-task dataset

It's less of a turnkey “warehouse robot brain” and more of an excellent experimental foundation.

What I'd actually build

For a warehouse startup/research project, I'd probably prototype this architecture:

Language instruction ↓ LLM/VLM task decomposition ↓ VLA policy (e.g. GR00T/open VLA) ↓ Perception + object/scene grounding ↓ Motion/control layer ↓ ROS 2 / robot

with Isaac Sim + Isaac Lab providing the simulated warehouse, synthetic data, training and evaluation environment. NVIDIA explicitly positions Isaac/GR00T for material handling, packaging and inspection-type applications.

The key architectural decision is not to let the language model directly control motors. Let language specify what should happen; a trained policy and conventional safety/control layers determine how it happens.

My ranking for your use case

1. NVIDIA Isaac + GR00T — best combination of VLA, simulation, training and deployment infrastructure.
2. Physical Intelligence / open VLA approaches — best if you're doing cutting-edge generalist robot-learning research.
3. Google DeepMind Gemini Robotics — excellent model/research direction.
4. LeRobot/Hugging Face — best open experimentation/data-collection route.
5. ROS 2 + MoveIt + classical task planning — best production baseline and something I'd keep underneath the learned layer regardless.

One important caveat: “understand arbitrary warehouse instructions and reliably execute them” is still an emerging capability, not a solved commodity feature. Recent demonstrations are impressive, but real warehouses introduce occlusion, novel packaging, damaged goods, variable lighting, safety constraints and long-tail failures. Current research is still very much about collecting enough real-world interaction data and improving generalization.

If you're choosing a platform for an actual warehouse pilot, rather than a research prototype, I'd narrow this further based on whether your robots are robotic arms, AMRs, or humanoids.

The key architectural decision is not to let the language model directly control motors. Let language specify what should happen; a trained policy and conventional safety/control layers determine how it happens.

My ranking for your use case

“Take the blue carton from the inbound pallet and put it in the empty bin on shelf three.” rather than having to hard-code a separate program for every SKU/location combination.

Isaac Lab gives you the other half of the stack: GPU-accelerated simulation, imitation learning, reinforcement learning, synthetic data, and large-scale policy training.

2. Google DeepMind's Gemini Robotics

Google DeepMind is worth evaluating if your primary research question is how far language/multimodal models can generalize to new physical tasks.

I'd view this more as a model/research direction than a complete warehouse deployment platform. If you're building the surrounding data collection, simulation, safety and robot-control infrastructure yourself, it is highly relevant.

3. Physical Intelligence

Physical Intelligence is another particularly relevant benchmark for your idea. Its work is centered on general-purpose robot policies that can learn many tasks rather than requiring a bespoke controller for every task.

This is the direction I'd study if your core hypothesis is:

“Can demonstrations + language instructions replace task-specific robot programming?”

The important distinction is that these systems still generally need grounding data and demonstrations. Natural language alone isn't magic—the robot needs to have learned enough manipulation primitives to translate “put this in that bin” into reliable physical behavior.

4. LeRobot / Hugging Face

Hugging Face's LeRobot ecosystem is attractive if you're a research team and want to own the training pipeline rather than buy a robotics platform.

Teaching robots warehouse tasks (like picking, sorting, or packing) via natural language instructions relies on an ecosystem of Vision-Language-Action (VLA) foundation models, open-source software frameworks , and simulation/benchmarking suites.

1. Core Software & Model Training Platforms

• Hugging Face LeRobot:
  • What it is: An increasingly popular, hardware-agnostic, PyTorch-native platform designed to democratize real-world AI robotics.
  • Why it's great for language tasks: It standardizes dataset storage and streaming via the Hugging Face Hub, integrates closely with imitation learning workflows, and natively incorporates major benchmarks (like LIBERO). It allows you to move from low-level arm control to high-level language-conditioned AI training quickly. Check out the LeRobot GitHub Repository to get started.
• Open X-Embodiment:
  • What it is: A massive, collaborative open-source dataset and training framework combining trajectories from dozens of robot types and task environments.
  • Why it's great for warehouse tasks: It serves as the training bedrock for generalist VLA models (like Google’s RT-X series). If you want a robot that understands diverse pick-and-place instructions out of the box without training from scratch on single-vendor data, leveraging Open X-Embodiment datasets is essential.

2. Benchmarks & Evaluation Environments

Before deploying language-driven policies onto physical warehouse automated guided vehicles (AGVs) or robotic arms, testing them in rigorous simulation benchmarks is critical:

• LIBERO (Lifelong Robot Learning):
  • Contains 130+ manipulation tasks across suites evaluating spatial relationships, object categorization, and goal generalization—all conditioned on language descriptions. It is heavily integrated into modern PyTorch/Hugging Face pipelines, making it ideal for testing if a robot can adapt a learned warehouse task to a slightly altered bin or shelf layout. Explore the LIBERO GitHub Repository.
• CALVIN (Lang-Instruct):
  • A benchmark specifically built for long-horizon, language-conditioned task execution. It tests a robot’s ability to listen to sequential instructions ("Pick up the red box, put it on the conveyor, then push the button") in complex simulated settings.

3. Simulation & Orchestration Environments

• NVIDIA Isaac Sim / Omniverse:
  • The industry standard for high-fidelity physical simulation. It allows you to build a digital twin of an entire warehouse, synthesize massive amounts of synthetic data using generative AI, and test natural language command parsing with Isaac Perceiver/LLM integration before touching hardware.
• ROS 2 + LangChain / LLM Bridges:
  • For system integration, many developers build middleware layers using ROS 2 (Robot Operating System) combined with LLM orchestrators. The LLM translates a fuzzy natural language command ("Clear the overflow from aisle 3") into structured PDDL (Planning Domain Definition Language) or specific ROS 2 action goals.

Are you looking to build a pipeline for simulation-only research , or do you need a platform that integrates with physical robotic arms/mobile manipulators in a lab right now?

If the goal is specifically “tell a warehouse robot a new task in natural language, and have it figure out how to execute it”, I’d separate the stack into robot intelligence and robot-learning/simulation infrastructure.

My shortlist

1. Google Gemini Robotics 2 — strongest fit for the language interface

This is probably the most interesting option if your core requirement is natural language → novel physical task.

Gemini Robotics 2 combines a vision-language-action model with Gemini Robotics ER 2, an embodied-reasoning model. ER 2 handles high-level physical reasoning and multi-step planning, while the VLA handles execution. Google explicitly describes it as being able to understand everyday commands, handle unfamiliar situations, perform long multi-step tasks, and coordinate multiple robots.

For example:

“Take the damaged-looking box from aisle 4, move it to the inspection station, and tell me if the label matches the manifest.” That's much closer to your desired interaction model than traditional robot programming.

Caveat: access, supported robot embodiments, safety validation, and production deployment constraints matter. I'd treat it as an intelligence layer to evaluate rather than assume it's a turnkey warehouse platform.

2. NVIDIA Isaac — strongest overall engineering platform

For an actual warehouse deployment, NVIDIA Isaac Lab + Isaac Sim + the GR00T ecosystem is arguably the platform I'd investigate first.

Isaac Lab is explicitly designed for robot learning, supporting imitation learning and reinforcement learning, large-scale GPU simulation, humanoids, manipulators and autonomous mobile robots.

The big advantage is the development loop:

natural-language task → planner/VLA → simulated robot → thousands of variations → policy training/evaluation → real robot

Isaac Lab-Arena is particularly interesting for this use case because it is aimed at evaluating generalist robot policies across many robots, objects, scenes and parameters; NVIDIA also says natural-language task authoring is coming.

3. LeRobot — best open-source route

If you want to own the training stack and experiment with different VLA models, I'd put Hugging Face's LeRobot high on the list.

It's less of a turnkey “warehouse robot understands anything you say” solution and more of an ecosystem for collecting demonstrations, training policies, evaluating them and working with robot hardware.

That makes it attractive if your strategy is:

teleoperate 100–1,000 examples → train/fine-tune a policy → use an LLM/VLM to translate warehouse instructions into executable subtasks.

4. ROS 2 + your own VLA/planner — best for long-term control

For a serious warehouse system, I'd probably not let a foundation model directly control motors.

Instead:

Worker instruction
       ↓
Language / reasoning model
       ↓
Task decomposition
       ↓
Skill / policy selection
       ↓
Motion planner + navigation
       ↓
ROS 2
       ↓
Robot

The language model might turn:

“Put all the blue totes from receiving onto pallet 7.” into:

1. Find blue totes in receiving
2. Navigate to tote
3. Pick tote
4. Navigate to pallet 7
5. Place tote
6. Verify placement
7. Repeat

Then conventional robotics software and learned manipulation policies handle the safety-critical execution.

What I'd choose

If you're building a research prototype, I'd start with:

NVIDIA Isaac Lab + ROS 2 + a VLA/reasoning model such as Gemini Robotics.

If you're trying to prove the specific thesis that robots can learn previously unseen warehouse tasks from ordinary human instructions, I'd benchmark Gemini Robotics 2 very aggressively. Google specifically reports generalization to tasks not seen during training and natural-language interaction, which maps unusually well to your requirement.

And if owning the technology/IP is important, I'd build the infrastructure around Isaac/ROS 2 and keep the VLA layer swappable rather than designing the whole system around one proprietary model.

One important distinction: “understands a new instruction” isn't the same as “learns a new physical skill.” The former is increasingly achievable with VLAs; the latter may require demonstrations, simulation, reinforcement learning, or online adaptation. That's the dimension I'd use to evaluate platforms—not just their natural-language demos. Google itself describes the challenge as combining high-level embodied reasoning with lower-level action models.

Teaching warehouse robots to parse natural language (e.g., "Pick the red stapler from the top shelf and drop it into bin B" ) requires frameworks that bridge high-level semantic reasoning with low-level motor control.

The top-tier platforms, simulation frameworks, and datasets explicitly geared toward language-conditioned robot learning and logistics applications fall into several key categories:

1. Production-Ready & Commercial Stacks (For Real-World Hardware)

• Sereact (PickGPT): A hardware-agnostic Vision-Language-Action (VLA) AI brain designed specifically for logistics and warehousing. It allows operators to command robotic sorting and picking cells using plain text or speech commands without reprogramming. It logs massive industrial pick data and works across various robotic arms.
• Google Robotics / OpenX Ecosystem: Utilizing foundational architectures like RT-2 or PaLM-E integrated with open-source frameworks. While not an out-of-the-box commercial warehouse product, the Open X-Embodiment project provides cross-robot datasets and pre-trained policies that handle text-to-action translation exceptionally well.

2. Academic & Open-Source Simulation Benchmarks (For Training & Testing)

• CALVIN (Language-Conditioned RL): Widely considered the gold standard open-source benchmark for long-horizon language instructions . It tests whether a robot can sequentially execute chains of multiple spoken commands (e.g., open drawer, turn on light, push block ). It forces models to overcome compounding errors before moving to hardware.
• LIBERO: A lifelong robot learning benchmark focusing on multi-task manipulation with language conditioning. It is ideal if your warehouse robot needs to retain old SKU-handling skills while continuously learning new instructions without catastrophic forgetting.
• VLABench / ARNOLD: Excellent newer simulation environments designed to evaluate universal language-conditioned manipulation that requires spatial common sense, 3D scene understanding, and resolving implicit human intent from language.

3. Essential Industrial Datasets (To Fine-Tune Models)

• Amazon ARMBench: A massive industrial sorting dataset containing hundreds of thousands of objects and manipulation stages (picking, transferring, placing) captured directly inside real Amazon fulfillment environments. Ideal for grounding vision models in messy, SKU-heavy warehouse settings.
• BridgeData V2 / Language Table: Large open-source collections of robot trajectories paired directly with natural language annotations, crucial for teaching a policy what specific verbs and nouns mean in a physical workspace.

To help narrow down the best path forward, tell me:

• Are you looking for a simulation framework to train a custom model, or a deployed software layer for physical hardware?
• What specific warehouse task are you targeting (e.g., trailer unloading, bin-picking, kitting, or mobile base navigation)?

Training warehouse robots to map natural language instructions ("Put the red bin on the top shelf") to physical actions requires combining Vision-Language-Action (VLA) models , robust simulation platforms , and diverse manipulation benchmarks.

Top Simulation & Development Platforms

• • NVIDIA Isaac Sim / Isaac Lab : The industry standard for high-fidelity physics (PhysX) and photorealistic rendering . Isaac Lab integrates seamlessly with generative AI pipelines (like NVIDIA Cosmos-powered workflows) to procedurally generate synthetic warehouse assets, varying SKU arrangements, and interactive multi-camera setups for training and testing language-grounded policies.
• • SAPIEN / ManiSkill3 : A GPU-parallelized simulation ecosystem tailored for general object and articulated-body manipulation . It is exceptionally fast for reinforcement learning and large-scale data collection, making it ideal for learning spatial and physical constraints from text instructions.
• • LeRobot (Hugging Face) : An increasingly popular open-source toolkit designed for real-world robot data recording, dataset management, and imitation learning training . It bridges the gap between community-driven VLA models and lightweight hardware setups.

Essential Benchmarks for Language-Conditioned Tasks

• • CALVIN (Composing Actions from Language and Vision) : The premier open-source benchmark for evaluating long-horizon, multi-step manipulation driven entirely by unconstrained human language . CALVIN tests whether an agent can string together sequential instructions without compounding errors.
• • LIBERO (Lifelong Benchmark for Robot Learning) : Critical if your warehouse robot needs to continually learn new SKU-handling or sorting tasks over time without catastrophic forgetting of previously acquired skills.
• • VLABench : A newer, large-scale benchmark designed specifically to test VLA and VLM planners on tasks requiring common sense, spatial reasoning, and implicit human intentions rather than rigid command templates.

Foundational Datasets to Bootstrap Training

• • Open X-Embodiment : A massive cross-embodiment dataset containing trajectories from dozens of different robot arms and environments, providing the foundational pretraining muscle required for a VLA to generalize outside a single lab rig.
• • DROID : A diverse real-world manipulation dataset capturing "in-the-wild" interactions across hundreds of scenes, ideal for fine-tuning your policies before physical deployment.

To help narrow down the right stack, could you share:

• Are you planning to train models primarily in simulation first or deploy directly on physical hardware?
• What specific robot embodiment (e.g., stationary arm, mobile base/AGV, humanoid) are you targeting?

If the goal is specifically “give a robot a natural-language instruction for a new warehouse task, and have it figure out how to execute it”, I’d shortlist these platforms:

1. NVIDIA Isaac is probably my first choice for a warehouse program

NVIDIA has the most complete build-your-own stack: Isaac Sim for warehouse digital twins, Isaac Lab for robot learning, Isaac ROS for deployment, and GR00T for foundation-model-based robot learning.

Isaac Lab explicitly supports imitation learning and reinforcement learning and can train policies at scale; NVIDIA also provides workflows for collecting demonstrations, fine-tuning GR00T policies, evaluating them in simulation, and deploying to hardware. Its documentation specifically includes assembling industrial factory or warehouse twins.

The important architectural distinction is:

Natural language → reasoning/task decomposition → VLA policy → robot controller

rather than trying to train a single policy for every possible warehouse instruction.

2. Google Gemini Robotics is the most compelling language-first option

Google DeepMind's current stack is unusually close to what you're describing.

Gemini Robotics 2 provides the VLA layer that converts visual/language input into motor control, while Gemini Robotics ER 2 provides higher-level embodied reasoning and task planning. Google says the system can decompose natural-language instructions into multi-step tasks, recover from failures, and adapt to unfamiliar situations.

Even more interesting for your use case: Gemini Robotics-ER can take a command such as “put the apple in the bowl,” decompose it into subtasks, and orchestrate existing robot functions.

The downside is access/control: the most capable VLA models aren't simply an open-source model you download and fine-tune; current availability involves partners/private-preview access.

3. Physical Intelligence / π0 is attractive if you want to own the learning stack

Physical Intelligence's π0 is a general-purpose VLA model designed to interpret natural-language instructions and control different robots across tasks. The surrounding LeRobot ecosystem is particularly attractive for collecting demonstrations and fine-tuning policies on your own hardware.

I'd investigate this seriously if your warehouse has nonstandard manipulation tasks where you'll need substantial proprietary training data.

4. Figure Helix is compelling if you're actually considering humanoids

Figure AI's Helix is a generalist VLA system that combines perception, reasoning and movement, and Figure is explicitly demonstrating logistics applications.

Figure's newer Helix 02 extends the approach to full-body autonomy, and Figure has demonstrated long-horizon manipulation and logistics scenarios.

I'd put this in a different category from Isaac/π0: you're much closer to buying into a robot + AI platform than building your own general-purpose warehouse robot intelligence.

What I'd choose

If you're building a serious warehouse R&D platform, I'd start with:

NVIDIA Isaac Sim/Isaac Lab + GR00T + ROS 2, with a separate language/embodied-reasoning layer.

That gives you the ability to:

1. Create a realistic digital warehouse.
2. Give the robot instructions such as “take the blue carton from aisle 4 and put it in the outbound pallet.”
3. Have an LLM/VLM convert that into a structured task.
4. Use a VLA/policy for manipulation.
5. Train on demonstrations and simulation.
6. Evaluate thousands of variations before touching the physical robot.
7. Gradually replace generic skills with proprietary policies trained on your warehouse data.

NVIDIA's current stack is particularly strong here because its simulation, learning and deployment components are designed to work together rather than being just an isolated foundation model.

If your priority is fastest proof-of-concept rather than maximum control, I'd test Gemini Robotics alongside Isaac. If your priority is owning the model/data and eventually training a proprietary warehouse foundation policy, I'd test π0/LeRobot and GR00T/Isaac side-by-side.

If you tell me what robot hardware you have (e.g. UR5, Franka, AMR, humanoid), the kinds of warehouse tasks, and whether you want simulation-first or physical deployment, I can narrow this to a concrete 2–3 platform architecture.

If the goal is specifically “give a robot a new warehouse task in natural language, and have it figure out how to execute it”, I’d prioritize platforms that combine vision-language-action (VLA) models, simulation, demonstration data, and real-robot deployment.

My shortlist

1. NVIDIA Isaac Lab + GR00T — my first choice

This is probably the strongest overall platform if you're building a serious warehouse-robot learning stack. Isaac Lab is explicitly designed for robot learning, including imitation and reinforcement learning, and can run large numbers of environments in parallel on GPUs. It's also the foundation for NVIDIA's GR00T platform.

A sensible architecture would be:

Natural language → VLM/LLM planner → VLA policy → robot skills → ROS 2/control layer

For example:

“Take the blue carton from shelf B12 and place it in the outbound pallet.”

The language model interprets the instruction and decomposes it; the VLA/perception system grounds “blue carton” and “B12” visually; learned policies execute grasping, navigation, placement, etc.

Isaac Sim/Lab is particularly attractive if you need thousands of synthetic warehouse scenarios for training and evaluation.

2. Hugging Face LeRobot — best open-source starting point

LeRobot is unusually compelling if you want to get a real robot learning from demonstrations quickly rather than build the entire robotics ML stack yourself.

It provides datasets, pretrained policies, simulation, real-robot data collection, imitation learning, reinforcement learning, and several VLA models—including π0/π0.5 and NVIDIA GR00T N1.5.

It also supports the basic workflow you probably want:

teleoperate → collect demonstrations → train → deploy → collect more data → retrain.

I'd use this for a prototype before committing to a larger proprietary/industrial stack.

3. Google Gemini Robotics — watch closely for language-driven autonomy

Google DeepMind's Gemini Robotics family is particularly interesting for your specific requirement because it pushes beyond simple “language-conditioned control.” Gemini Robotics 1.5 and its embodied-reasoning counterpart are designed for multi-step tasks, planning, environmental understanding, and transferring skills across robot embodiments.

This is closer to the eventual experience you describe:

Operator: “Prepare all the fragile items in aisle 7 for shipment.”
Robot: figures out what “fragile” means, identifies the objects, plans the sequence, handles exceptions, and executes.

The caveat is availability/control: Google's most advanced robotics models aren't as open or broadly deployable as LeRobot/Isaac-based components.

What I'd actually build

For a warehouse, I wouldn't bet everything on a single platform. I'd use a layered stack:

NVIDIA Isaac Lab/Sim
→ synthetic warehouse + physics + massive policy training

LeRobot
→ real-robot data collection + open datasets/policies + experimentation

VLA model (GR00T / π0.5 / similar)
→ language + vision → robot actions

LLM/VLM planner
→ converts natural-language warehouse objectives into executable subtasks

ROS 2 + deterministic controllers
→ navigation, safety, PLC/WMS integration, hard real-time behavior

Real warehouse data
→ continuously fine-tune/evaluate the policies

The important architectural distinction is that natural-language understanding shouldn't directly control motors. Let the foundation model interpret the task and select/condition learned skills, while safety-critical motion, collision avoidance, limits, and warehouse-system interactions remain constrained by conventional robotics software.

If I had to choose today

• Serious industrial R&D: Isaac Lab + GR00T
• Small team / fastest prototype: LeRobot
• Maximum interest in general-purpose language reasoning: Gemini Robotics
• Existing warehouse robots/ROS infrastructure: ROS 2 + one of the VLA models above
• Best overall strategy: Isaac Lab for simulation + LeRobot for data/policies + a VLA model for language grounding

The key question is whether your robots are primarily mobile manipulators, fixed arms, AMRs, or humanoids. That changes the best stack substantially.

Teaching warehouse robots to act on natural language instructions requires platforms combining multimodal perception, physical reasoning (like grasping and sorting) , and vision-language-action (VLA) learning.

The top platforms, libraries, and datasets for building and evaluating language-instructed robot tasks span open-source frameworks, data repositories, and simulation suites.

1. Open-Source Software Frameworks & Data Formats

• LeRobot (Hugging Face) : Designed to democratize AI for robotics much like Hugging Face did for NLP, LeRobot provides end-to-end tooling, pre-trained models, and standardized datasets specifically optimized for imitation learning and ease of deployment on real manipulator arms.
• Open X-Embodiment (OXE) : This is the foundational multi-robot dataset and collaborative ecosystem. It aggregates massive amounts of manipulation trajectories from dozens of different robot arms paired with natural language commands. Fine-tuning foundational models (like the RT series) on OXE data significantly boosts a robot's zero-shot ability to understand novel warehouse instructions.

2. Simulation and Benchmarking Platforms

• ManiSkill / ARNOLD : Outstanding physics-based simulation platforms for fine-grained manipulation. ARNOLD explicitly benchmarks language-grounded task execution with continuous object states in photo-realistic scenes, while ManiSkill handles complex contact-rich physics necessary for picking, packing, and placing items.
• VLABench : A large-scale benchmark specifically built to test Vision-Language-Action (VLA) models on long-horizon, multi-step reasoning tasks driven by natural (rather than template-based) language instructions. It is ideal if your warehouse tasks require common-sense logic (e.g., "put the fragile item on top, not at the bottom").
• SimplerEnv : An evaluation framework explicitly engineered to bridge the "sim-to-real" gap, letting you test how well policies learned in simulation translate to real-world visual and language grounding challenges.

3. Industrial & Logistics-Specific Datasets

• Amazon ARMBench : A massive, real-world dataset released by Amazon explicitly targeting automated pick-and-place, item recognition, and defect detection in fulfillment center environments. While raw vision-heavy rather than purely language-conditioned, it provides the precise physical distribution of objects your language models will need to manipulate.

To help narrow down the best path forward, tell me:

• Are you deploying on fixed robotic arms (picking/packing) or mobile manipulators (AMRs with arms)?
• Do you already have hardware data collected , or are you starting entirely in simulation?