Challenge 1
GenAI-Based Natural Language Mission Generator for Autonomous Robots in Agriculture _ Intellimech
Team:
Sensifai BV, Belgium
Proposal title:
EdgeVLA-ROS: Risk-Calibrated Clarify–Validate–Execute Mission Generation for Agricultural Robots
Organisation
Sensifai BV is a Brussels-based AI company founded in 2016. It develops deep-learning recognition technology for video, image and audio, and has built ROS2 speech-to-text and human–robot interaction components for robotic platforms in EU-funded projects, including an on-robot audio pipeline for Panza Robotics’ Artaban robot.
Project info
EdgeVLA-ROS adds a natural-language mission layer to agricultural robots, converting voice or text into executable ROS1 missions using speech, camera images, vineyard maps and information about the robot’s capabilities.
Its core contribution is Risk-Calibrated Clarify–Validate–Execute (RC-CVE). Rather than executing the first interpretation of a command, the system generates three to five candidate missions through constrained decoding over a typed skill grammar, then scores each one on language uncertainty, visual-grounding uncertainty, capability mismatch, safety violations and task criticality. It executes when the lowest-risk candidate is reliable enough, asks one targeted question when that would materially reduce risk, and rejects the command when no safe, feasible interpretation exists. Hard safety violations always override the learned score.
The multimodal backbone is an open Qwen3-VL model adapted with LoRA, with the smallest model that meets accuracy and latency targets kept for on-robot inference. Only complete, whitelisted and constraint-valid missions can be dispatched. Sensifai reuses robot-audio and ROS2 interface patterns from earlier EU projects. Targets include at least 92% intent accuracy, at least 88% execution success and robustness, at least 95% rejection of invalid missions, and at least 30% faster mission definition than the challenge’s GUI baseline.
Details
Agricultural robots remain hard to use for non-specialists because defining a mission usually requires ROS knowledge, scripting or complex GUI configuration. A natural-language interface helps only if it can handle ambiguous instructions safely, which most language-to-robot approaches do not, because they act on a single interpretation.
EdgeVLA-ROS, developed by Sensifai, introduces a risk-calibrated way of deciding between executing, asking and refusing. For each instruction, the system produces several candidate missions from the robot’s whitelisted skills and scores them on how certain it is about the language and the visual scene, whether the robot can perform the task, whether safety rules are respected and how critical the task is. It then executes the lowest-risk valid mission, asks a single clarifying question, or rejects the command. Inference runs at the edge on an adapted open multimodal model, and a middleware-neutral skill schema with ROS1 and ROS2 adapters reduces lock-in to any single robot platform.
Building on Sensifai’s earlier ROS2 robot-audio and human–robot interaction work, the project will progress from TRL 3 to TRL 5, demonstrated on the JOiiNT LAB robot in a controlled outdoor vineyard. Targets are intent accuracy of at least 92%, mission success and linguistic robustness of at least 88%, mean latency of 8 seconds or less, rejection of at least 95% of invalid missions, and mission-definition time at least 30% shorter than the challenge baseline. Results will be packaged as the EdgeVLA Mission SDK, to be licensed with integration services to agricultural-robot manufacturers and integrators.
