Challenge 1
GenAI-Based Natural Language Mission Generator for Autonomous Robots in Agriculture _ Intellimech
Team:
Eletoyia (team “VINE-TALK”), Cyprus
Proposal title:
VINE-TALK — a constrained natural-language mission generator for field robots, validated in the vineyard
Organisation
Eletoyia is a Cypriot SME, founded in 2003, active in renewable energy and smart transport. It designs, installs and operates PV systems, runs production machine learning on telemetry from over 5,000 PV installations, has developed an autonomous PV-cleaning robot (PCT patent filing), and joins European and national research projects.
Project info
VINE-TALK lets vineyard operators re-task a field robot by voice or text instead of relying on a ROS programmer. The pipeline follows the challenge objectives step by step: speech-to-text (Whisper-class open ASR); intent recognition and task decomposition by an open-weight LLM fine-tuned on a synthesised mission-dialogue corpus; visual and map grounding, including grape-cluster detection fine-tuned on the provided datasets; and mission composition strictly from the robot’s published capability list.
Safety is built into the architecture. Constrained decoding into a typed mission schema makes hallucinated actions impossible by construction, and the model asks for clarification when information is missing rather than guessing. Missions compile to behaviour trees over ROS1 actions that engineers can read and edit, and every mission is checked in Gazebo simulation for kinematics, geofence and obstacles before dispatch, with human confirmation for first-of-type missions and emergency-stop hooks.
Models run on an edge GPU, with no cloud dependency in the field. Targets are at least 90% intent accuracy, at least 85% execution success and linguistic robustness, and latency of 5 seconds or less. Portability is demonstrated by replicating the mission layer on Eletoyia’s own PV-plant inspection robot.
Details
Programming agricultural robots still requires ROS specialists, which keeps automation out of reach for most growers and field technicians. European vineyards also face chronic seasonal labour shortages, so robots that operators can re-task themselves have direct economic value.
VINE-TALK, developed by Eletoyia, is a natural-language mission generator that turns spoken or written instructions into validated, executable ROS1 missions for the JOiiNT LAB vineyard robot. The method combines open speech recognition, an open-weight language model restricted by constrained decoding to the robot’s declared capabilities, and a perception layer that grounds references such as rows, waypoints and grape clusters in maps and camera data. Missions are compiled into inspectable behaviour trees and simulated before execution, so no unvalidated command reaches the robot’s actuators. The full stack runs on an edge GPU, keeping latency low and removing dependence on field connectivity.
The project will raise the integrated pipeline to TRL 5 through simulation, controlled tests on the physical platform and final validation in an outdoor vineyard row. Target results are intent-recognition accuracy of at least 90%, task-execution success and robustness to rephrased instructions of at least 85%, and speech-to-plan latency of 5 seconds or less. A second deployment on Eletoyia’s PV-plant inspection and cleaning robots will show that the same layer transfers to industrial maintenance, giving the company an immediate route to exploitation across the PV installations it services in Cyprus and Greece.
