Challenge 1

GenAI-Based Natural Language Mission Generator for Autonomous Robots in Agriculture _ Intellimech

Team: 

University of Technology Nuremberg (UTN) — Fundamental AI Lab, Germany

Proposal title: 

VinePilot: Steerable Vision-Language Mission Generation for Safe Agricultural Robots

Organisation

The University of Technology Nuremberg (UTN), established in 2021, is Bavaria’s first new state university in over four decades, with an interdisciplinary, international profile and a strong AI focus. Its Fundamental AI Lab, led by Prof. Yuki M. Asano, researches visual representation learning, multimodal grounding and efficient model adaptation.

Project info

VinePilot replaces the ROS programmer with a language layer that operators can use and integrators can audit. A grower says “check the third row and tell me which clusters look ripe”; streaming speech recognition returns text with confidence scores, and an instruction-tuned open-weight LLM resolves the intent against an explicit catalogue of robot actions. Under grammar-constrained decoding it produces a typed, human-readable plan, never raw motor commands.

Perception is steered by the same language. The lab’s ECCV 2026 oral paper, “Steerable Visual Representations”, makes a frozen vision transformer respond to the mission phrase, so “ripe cluster”, “trunk” or “obstacle” redirect one backbone instead of requiring a detector per concept. A topological row map grounds spatial references such as “the third row”.

A deterministic validator checks types, preconditions, geofences, confidence and safety rules before compiling to ROS1 actions; uncertain or safety-critical plans go to a human, and a runtime monitor triggers clarification, replanning or a safe stop. Targets exceed the challenge KPIs: at least 92% intent accuracy, at least 85% success on the physical platform, median latency of 5 seconds or less, 100% validated execution, and at least 50% shorter mission-authoring time than the GUI baseline.

Details

Every new mission for a vineyard robot currently needs a ROS programmer, which limits the use of field robotics on farms and in robotics SMEs. Reliable, language-driven missions would let the people who know the vineyard direct the robot themselves, and would make targeted scouting and treatment cheap enough to replace blanket passes.

VinePilot, developed by the Fundamental AI Lab at the University of Technology Nuremberg, is a mission-generation layer that turns spoken instructions into validated ROS1 plans. An open-weight language model produces a typed mission plan restricted to the robot’s catalogue of actions, and the plan is grounded in a topological map of vineyard rows. Perception relies on the lab’s published steerable visual representations, which let a single frozen vision backbone focus on whatever the mission names, from ripe clusters to row ends, at low training cost. A deterministic validator and compiler ensure that no unchecked model output reaches the robot, and uncertain or safety-critical plans are referred to a human.

Starting from TRL 2 with its key perception component already published, the project will reach TRL 5 through trials in the JOiiNT LAB vineyard row on unseen instructions and live outdoor imagery. Targets include at least 92% intent accuracy, at least 85% mission success on the physical platform, median latency of 5 seconds, and at least 95% correct refusal of unsafe or out-of-scope instructions. The team will also measure training and inference energy, aiming for at least 40% lower adapter-training energy than full fine-tuning of end-to-end robot policies.