Challenge 4

Generative AI for Automatic Test Case Generation from Crash Databases & Standards_Siemens Industry Software NV 

Team: 

RoboSafe (team lead: Dr Cheng Wang; team member: Dr Yongqi Zhao)

Proposal title: 

EVIDENT: Generating Crash-Grounded Validation Scenarios Enriched by Naturalistic Driving Data

Organisation

RoboSafe is a team of experienced researchers, led by Dr Cheng Wang, working on the safety, verification and validation of automated driving. Team member Dr Yongqi Zhao specialises in scenario-based testing and LLMs.

Project info

EVIDENT bridges sparse crash evidence and parameterised Logical Scenarios. Crash records provide events and outcomes but usually lack trajectory-level dynamics, while naturalistic driving data offer rich kinematics but rarely capture collisions; EVIDENT links the two.

Deterministic national crosswalks map each country’s coded accident schema once to a shared ontology, an extension of the open CRUISE ontology, avoiding model calls for routine fields. Instruction-tuned LLMs with JSON-schema-constrained decoding extract manoeuvres, conflict topology and road-user types from police narratives, vision-language models read collision diagrams for road layout, trajectories and impact configuration, and an OWL reasoner checks logical consistency. An Abstract Scenario layer then defines the attributes each Logical Scenario needs. Parameters supported by evidence are extracted directly; the rest are drawn from context-conditioned joint distributions learned from matching naturalistic scenarios, constrained by physical and kinematic limits and checked for acceleration, jerk, friction and road geometry, with conformal prediction giving uncertainty intervals. Reference and accident-derived scenarios are aligned semantically and compared by parameter-space overlap to produce coverage scores and risk-weighted gaps.

The work connects two existing tools: the team’s crash-to-Functional-Scenario extraction method and ScenarioEngine, its deployed platform holding more than 500,000 extracted scenarios with OpenSCENARIO export.

Details

Simulation-based validation of automated driving depends on scenarios drawn from real safety evidence, yet European crash data is fragmented across national databases with incompatible schemas and little trajectory-level detail. Naturalistic driving recordings contain the missing dynamics but almost never capture collisions, so the two sources remain disconnected.

EVIDENT, developed by the RoboSafe team, uses a neuro-symbolic approach to join them. Crash semantics extracted from records, narratives and collision diagrams are expressed as Functional Scenarios in an extended version of the open CRUISE ontology. Where the evidence does not specify a parameter, its plausible range is inferred from naturalistic driving data recorded in similar situations, bounded by physical limits and accompanied by calibrated uncertainty. Each country’s schema is mapped once through a deterministic crosswalk, so AI is used only on unstructured evidence.

The team already runs both ends of the pipeline: a method for extracting Functional Scenarios from crash records and ScenarioEngine, a publicly deployed platform holding more than 500,000 scenarios with OpenSCENARIO export. The project builds the connecting enrichment and coverage modules and integrates them on European data to reach TRL 5. Targets include ontology-valid Functional Scenarios by construction, extraction at 90% of the human agreement ceiling, provenance for every field, parameter intervals that hold the true value 90% of the time, and onboarding a new national dataset in under a week. Accident scenarios with no matching reference test will be ranked by frequency and injury severity, giving Euro NCAP and manufacturers a quantified view of coverage gaps, including vulnerable-road-user conflicts.