In the rapidly evolving field of Automated Driving Systems (ADS), achieving reliable vehicle operation in adverse weather and low-visibility conditions represents one of the most pressing challenges. Today’s ADS technologies remain largely dependent on optimal weather for safe functioning which restricts operational availability and impacts customer acceptance. The AI-SEE Penta-Euripides² project embarked on a journey to build a novel, robust sensing system supported by Artificial Intelligence (AI) that enables automated travel in varied traffic, lighting and weather conditions.
Achievements and results of the project
Key results include the development of a high-resolution, multimodal sensor suite, AI-driven real-time sensor adaptation, and advanced sensor fusion techniques. These innovations enhance obstacle detection, even in low-visibility conditions, and enable safer, continuous operation. Additionally, a comprehensive simulation environment has been created to accelerate system validation, reducing the need for extensive real-world testing. One of the most groundbreaking aspects of the AI-SEE’s robust perception system is its ability to detect and identify lost cargo and small obstacles—issues that present considerable risk on the road. This includes recognizing both common and more severe hazards, from scattered pallets and loose tires to fallen motorcycles. AI-SEE innovative solution provides the world’s first robust approach to detect these threats at distances of up to 200 meters, even in low-visibility conditions, such as at night or in adverse weather.
Technological achievements
The beyond state-of-the-art progress and subsequent achievements are organized according to the main objectives:
- 24/365-High Resolution-Adaptive all-weather sensor suite. PolLiDAR unit integrates polarized imaging and LiDAR into a single hardware solution, combining colour, polarization, and 3D depth data for seamless sensor fusion. The system moves beyond the limitations of traditional LiDAR, which only captures distance, and expands its capabilities by utilizing the polarization state to enhance reconstruction accuracy in terms of distance, surface normal, and material property estimations. 4D MIMO Radar’s innovative waveguide-based antenna design minimizes signal loss and maximizes antenna compactness, enabling 30 transmit and 40 receive channels for a fivefold improvement in angular resolution compared to the state-of-the art. Benchmarking against key corner cases highlights its superior detection range and resolution, ensuring reliable performance beyond 350 meters, even in adverse weather conditions. Gated Camera: Leveraging advanced gating technology and a deep neural network, the GatedCam produces high resolution depth maps that significantly outperform in spatial resolution, yet with sufficient depth resolution any existing hardware solutions. By only capturing light from specific depth ranges, GatedCam significantly reduces noise from weather particles, enhancing the clarity of captured images. SWIR LiDAR based on a large 2D Ge-on-Si SPAD Array with integrated receiver read-out and a MEMS scanning architecture. AI-SEE explored the path for highly reducing the costs of LiDAR systems by establishing the fundamentals for Ge-on-Si- based SPAD array research.
- Artificial Intelligence (AI) platform for predictive detection of prevailing environmental conditions including signal enhancement and sensor adaptation. The 24/365 sensor suite has been equipped with novel signal enhancement methods based on compressive sensing employing latest neural networks and sensor adaption via semantic feedback. Using compressive sensing, the input signal can be implemented already on the individual hardware level. The main outcome is that AI-SEE has achieved improvements of up to 17% in conventional sensor technology through the use of novel deep neural networks in adverse weather conditions (rain, fog, snow). This benchmark enhances driving safety even with conventional (already installed) sensors. Furthermore, AI-SEE has made significant progress in the automatic recognition of prevailing bad weather conditions from a moving vehicle using the Gated Camera, and substantial strides in the automatic adaptation of parameters to these weather conditions.
- Smart fusion to create the 24/365 adaptive all-weather robust perception system. The AI-SEE Sensor Fusion Platform is a groundbreaking solution designed to provide seamless perception in all-weather conditions. By integrating multiple sensor modalities, the platform delivers robust, longrange object detection and precise depth estimation, specifically tackling the lost cargo problem – the challenge of detecting small, non-traversable obstacles (such as tires and pallets) over extended distances of up to 150 meters, even in low-visibility conditions such as nighttime or adverse weather. The platform integrates two innovative sensor fusion approaches that build on each other to overcome key limitations of individual sensors while improving performance across a range of scenarios.
- Demonstrator and system validation testing campaigns. AI-SEE all-weather multi-sensor perception system has been integrated in the test and demonstration vehicles. Extensive validation campaigns have been conducted throughout the project, both in controlled environments and on public roads. The campaign demonstrated the efficiency and robustness of AI-SEE perception system, proving its capability to function under all lighting and weather conditions.
Societal & Economic Impact
Societal Impact: through the development of advanced sensor and software technologies, the project aims to reduce accidents and fatalities, significantly enhancing public safety. The reliable operation of automated vehicles in various weather conditions ensures greater mobility for all, including the elderly and disabled, improving their quality of life. Additionally, the reduction in traffic congestion due to better AD systems will result in smoother commutes and less time wasted on the road, contributing to overall societal well-being.
Environmental Impact: by enhancing the reliability and performance of sensors and software under adverse weather conditions, automated vehicles can operate more efficiently and with reduced emissions. Additionally, optimized driving patterns and smoother traffic flow resulting from advanced automation will further decrease energy consumption and environmental impact.
Economic & Market Impact: The collaboration between partners in the early stages of development will foster innovation, leading to job creation and bolstering the economic health of the automotive and electronics sectors. This advancement is expected to shorten the time to market for these technologies, promoting economic growth through the creation of new market opportunities and the enhancement of existing ones.
Discover more about this project’s challanges, background, and future delelopments in the Project Impact Summary.
Penta and Euripides² are Eureka Clusters operated by AENEAS.