Scout AI has demonstrated orchestration software running a fleet of autonomous air and ground systems using natural language prompts, writes Nick Flaherty.

The Fury Autonomous Vehicle Orchestrator, which uses a Vision–Language–Action (VLA) foundation model that translates a high-level objective into coordinated actions across a UGV and multiple UAS, has been executed on real hardware without scripted control or manual intervention.

VLA models are a class of multimodal foundation models that integrate computer vision and natural language to generate autonomous actions. Given an input video of the robot’s surroundings and a text instruction, the VLA directly outputs instructions for the robot to execute autonomously. The VLA models are constructed by fine-tuning a vision–language model on a large-scale dataset of labels that capture natural language instructions, synchronised with vision inputs and robot telemetry.

Scout uses a cloud-based tool from Nominal to store, review and adjust saved live audio-transcribed annotations from the field that allow engineers to review events recorded from a field test, automatically time-synchronised with the video and telemetry data collected from the vehicle.

“In autonomy, iteration speed is everything. Nominal lets us compress the development cycle of data collection, training and testing without compromising safety. It gives us the confidence to scale from tens of uncrewed systems to the world’s largest multi-domain autonomous defence fleet,” said Collin Otis, CTO and co-founder of Scout AI.

Unlike traditional autonomy stacks that rely on hand-engineered code and conditional logic, Fury functions as an interoperable layer for AI agents. The Orchestrator reads platform documentation and tool definitions, then generates structured JSON instructions native to each vehicle’s API, without modifying underlying flight controllers, mobility stacks or autonomy software.

From the language prompt, the Fury model builds the mission plan and submits it for approval before executing. It then tasks each asset in natural language and monitors mission progress, adjusting the plan as the situation changes. It coordinates the UGV and UAVs, manages timing and priorities and completes the mission with a battle damage assessment. The Orchestrator also enforces fleet-level constraints, including timing, priorities, mission phasing and operational authorities, issuing updated intent to ensure autonomous systems remain aligned with commander objectives.

The software continuously fuses telemetry, video feeds and mission data to provide a live common operational picture. When one aerial asset identifies the target vehicle, Fury redirects supporting systems in real time, adjusts tasking and autonomously sequences follow-on actions, all while keeping a human operator in the loop for supervision.