The CAVIAR project developed and demonstrated technologies and building blocks for CMOS image sensors, video processing hardware and algorithms, video data reduction and transmission. In all vision based professional applications the trend in image capture is towards higher spatial and temporal resolutions, wider colour gamut, higher dynamic range and improved image quality. In CAVIAR, 7 partners cooperated to develop and optimize the next generation of technologies for professional products for use in Medical, Television Broadcast, Precision Agriculture, Mobile and (in cabin) Automotive markets.
Background, objectives of the project and challenges
Image capture and transmission are generic technologies used in a multitude of business applications, including Healthcare, Precision Agriculture, Security, Automotive and Digital Entertainment. In all vision based professional applications, the trend in image capture is towards higher spatial and temporal resolutions, wider colour gamut including the use of non-visible spectral domains, higher dynamic range and improved image quality. The goal of the CAVIAR project was to collaborate on these technologies to address needs in several markets.
Technological achievements
CMOS Image Sensor achievements
The project worked on next generation CMOS image sensors. For broadcast a 4K UHDTV sensor with 2.5μm global shutter pixels was designed to provide 4K streams in full resolution at 180 frames per second to create, when played back, slow motion in 4K detail. For Precision Agriculture, an existing monochrome CMOS image sensor with a pixel pitch of 4.5μm has
been equipped with a 4×4 filter array to create 16 bands ranging from 400nm to 940 nm. For face recognition and authentication in mobile, as well as in cabin safety features in automotive, a 2M pixel global shutter CMOS image sensor was developed with improved near infrared sensitivity for better operation under bad lighting conditions.
Advances in image processing
To process multi-spectral images in Precision Agriculture, a deep learning algorithm was developed. A colour image containing a crop plant and a weed is given to the algorithm which returns two separate images: one of the crop plant and one of the weed. These images are then ready to be used in agricultural applications or by agricultural robots.
To benefit from the full 4K UHDTV specification, High Dynamic Range processing that can handle up to 15 F-stops of video, and conversion towards the smaller dynamic range in HDTV have been researched. Also an improved focus assist algorithm was developed to help the camera operator properly focus on a display with a lower spatial resolution than that of the camera.
Video data reduction and transmission
An automated ROI segmentation of Electron Microscopy images with annotation effort of less than 5 minutes was developed, saving hours in training a model to automatically segment cellular structures of interest, speeding up the digital pathology processing pipeline. For security an edge processing pipeline was developed for high resolution, high framerate imagery in aerial surveillance on a drone with limited processing power and low transmission bandwidth. A data reduction of 99% was achieved. To enable transmission of triple speed 4K video streams next to normal 4K and HDTV (2K) outputs on the camera towards the replay server, 100G Ethernet transmission using open IP standards was developed.
Hardware demonstrators
For all four applications hardware demonstrators have been developed that will result in commercial products or a demonstration kit for potential customers (automotive). compared to existing solutions. This is key for batterypowered applications such as consumer/mobile 3D authentication, AR/VR, in- cabin driver monitor. Most industrial applications make use of global shutter.
Societal & Economic Impact
To improve Health & Well Being, CAVIAR improved diagnostic capabilities in digital pathology to support our society’s increasing need for more, better and faster patient diagnostics at lower cost per diagnosis. The agricultural industry faces worldwide demands for higher yields and greater environmental protection. The CAVIAR technology for precision agriculture can enable production of more crop with less water and fewer chemicals. It can be used for image-guided operations in the field to help feed the world in a more economic and environment-friendly way. Broadcast television moves to ever higher resolutions and frame rates. The technology developed in CAVIAR makes this possible. Addition of slow motion to ultra HDTV will make it possible to place audiences in the middle of the action. Furthermore, the IP transmission
developed to enable remote productions will result in less staff and equipment traveling, thus lowering the carbon foot print of live television productions, and improving the social life of content creators who can be home more often. For a secure society, CAVIAR demonstrated technology which allows to apply high-resolution cameras in drone applications. This allows for a less operator-intensive monitoring solution, leading to a more secure society. NIR-enhanced cameras for in-cabin monitoring are essential tools to detect the status of the driver and other car occupants. NIR detection is necessary especially during night driving when indoor light is off. The car processor determines if the driver is in good driving shape. If not, alert messages are sent to the driver and ultimately the car stops. Global shutter and high quantum efficiency at NIR are required, as they allow the illuminator to be active for much shorter time (hence sparing considerable power and illuminator lifetime) compared to traditional rolling shutter sensors with low NIR QE. Apart from the benefits identified above, the project has strenghtened the position of Europe in the field of CMOS image sensors, cameras and image processing. Through the successful cooperation, important knowledge has been shared to improve the competitiveness of all. Collaboration on individual technical issues enabled partners to mature their designs faster, and find root causes for potential problems in an early stage.
Future Developments
The R&D performed in the CAVIAR project is an excellent basis for new products that will be launched by the commercial partners to fully harvest the market potential, and for follow-up research in the years to come. The good results of the project and the excellent cooperation made part of the partners decide to collaborate in a next PENTA project called Mantis Vision, which already started mid 2021.
Discover the Project Impact Summary on PENTA website.
PENTA is an EUREKA Cluster operated be AENEAS.