Challenge 1
GenAI-Based Natural Language Mission Generator for Autonomous Robots in Agriculture _ Intellimech
Team:
AI4GRAPES — Athena Research Center, Greece
Proposal title:
NL2MISSION
Organisation
AI4GRAPES is a team from Athena Research Center, a leading research institution in Greece, with strong expertise in Informatics and Computational Sciences, and a focus on ICT, AI, and language technologies. The team is part of the Institute for Language and Speech Processing (ILSP), a centre of excellence in Language Technologies, and specialises in multimodal machine learning across speech, language, computer vision and foundation models, publishes at venues such as NeurIPS, ECCV and Interspeech, and trains large models on European supercomputers.
Project info
NL2MISSION lets an operator speak a task in plain Italian or English, for example “check ripeness on the next two rows and flag diseased vines”, and turns it into a safe, executable ROS mission. The system transcribes the request with Whisper adapted to field vocabulary, extracts a structured intent, grounds it against what the robot sees using open-vocabulary detection and segmentation, and compiles a human-readable mission that is checked for capability and feasibility.
The reasoning foundation is NVIDIA Cosmos 3, an open world-foundation model for physical AI. A compact edge version runs on the robot for real-time intent recognition and task decomposition, while larger versions on CINECA’s LEONARDO supercomputer generate synthetic data and distil domain skills into smaller models that fit the latency budget.
A distinctive “mission preview” step shows the operator a world-model rollout of what the mission will do before approval, turning a generative model into an explainability tool. A symbolic execution path guarantees a working baseline even if the end-to-end policy underperforms. The pipeline will be validated on the JOiiNT LAB vineyard row against the challenge KPIs and additional targets for plan safety, speech recognition and usability.
Details
Field robots could take on much of the monitoring and selective work in vineyards, but programming them still requires robotics specialists. With agricultural labour scarce and input costs rising, farms and cooperatives need robots that operators can instruct directly, quickly and safely under field conditions.
NL2MISSION, developed by the AI4GRAPES team at Athena Research Center, is a natural-language mission generator for vineyard robots. The operator gives an instruction by voice; the system transcribes it, interprets the intent, links it to grapes and vines detected in the robot’s camera view, and compiles a ROS-executable mission. Reasoning relies on NVIDIA Cosmos 3, an openly licensed world-foundation model that combines physical reasoning, prediction of future observations and action generation. Before approving, the operator can watch a predicted preview of the mission, keeping a human in the loop and making the system’s intent visible. Large models trained on European HPC infrastructure are distilled into compact models that run on the robot within the latency budget.
Starting from TRL 2–3, the project will demonstrate the integrated voice-to-mission pipeline at TRL 5 on a real outdoor vineyard row provided by JOiiNT LAB. Targets include at least 90% intent-recognition accuracy, at least 85% mission success and robustness to rephrasing, latency of 5–10 seconds, fewer than 10% of plans rejected by operators and a speech word error rate below 15%. The team will publish its synthetic dataset and evaluation harness so that others can reuse the benchmark.
