MANTIS Project enhanced image capture systems for broadcast, and security & safety applications with its cost-efficient solution

22 Jan 2025

Image capture system capabilities need constant improvement to help content creators support the growing need for media consumption; provide security & safety systems/operators with better information; support useability of upcoming AR/VR. Therefore, MANTIS Penta-Euripides² project goals were:

  • Develop CMOS sensor (technology) for eye-tracking in AR/VR glasses; study spectral sensitivity enhancement; and develop a performance model for electro-optical (EO) systems with human-in-the-loop.
  • Introduce single sensor broadcast camera (2/3 inch) with slow-motion capabilities to lower the market.
  • Improve detection & recognition in security/safety through sensing & embedded AI inference.

 

Technological achievements

Key technology breakthroughs realized are:
— 6-speed 4K on sensor level in broadcast allowing better slow motion replay
— Very small 3D stacked global shutter pixel and sensor with high contrast and Near-infrared (NIR) sensitivity
— First enabling results on ‘layer exchange’ process research
— Low power highly compact rugged high speed security camera on CoaXPress
— Sensor agnostics Analog Signal Processor for uncooled infrared/thermal imaging
— Proven inference accuracy on small object detection through multi-image detection model

Top-5 capabilities:
— Single 4K resolution sensor camera for live television broadcast
— 4K at 60fps rugged security/safety camera
— High performance miniaturized image sensor for eye-tracking
— Real-time multi camera stream edge inference for small object detection
— Proof of layer-exchange process feasibility

Demonstrators’ exploitation:
Broadcast

  1. 6-speed capability will be commercialized for 6-speed HDTV in combination with a 4K output in normal frame rate.
  2. Single sensor camera architecture will be commercialized in HDTV (Q1 ’25).
  3. The depth map generation requires more research.

Security&Safety

  1. 4k60 rugged camera for security and spin-off products for industrial applications are introduced in 2024, design-in & production will start in 2025
  2. Thermal camera with ASPIC waits for business case improvement
  3. Small Object Detection labelled dataset to be extended in follow-up research

Sensor Technology

  1. Spin off sensor with pixel from CAVIAR project and MANTIS project design blocks is introduced in the market (Mira016)
  2. Layer exchange process is followed up in IMAGINATION and ENTERTAIN. Exploitation expected in ENTERTAIN (2027-2028)
  3. System optical performance model will be explored with 2 leads in 2025

Market Potential

Broadcast:

  1. Competitors cannot output 6-speed HDTV slow motion in combination with a native 4K live output, as they read HDTV resolution out of their image sensors for 6-speed.
  2. With the new single sensor camera architecture Grass Valley can address lower market segments.

Security & Safety:

  1. Teledyne Adimec 4K60 will follow-up current HD and 4k30 cameras. This will be in new system designs and replacement upgrades. Spin-off cameras based on the same camera architecture to industrial applications support new cost-efficient implementations in both visible and short-wave infrared (SWIR).
  2. IDEAS Thermal imager faces competition from lowcost Chinese solutions in civil applications but may offer advantages for space and defense based on western supply-chain.

Consumer and Technology:

  1. Compact low power sensor for eye-tracking in AR/VR glasses will take-off with penetration of such devices in consumer. Spin-off sensors find their way into applications such as medical endoscopes, allowing ams OSRAM to leverage from new 3D stacked sensor technology.
  2. Cost efficient wave-length extension on CMOS sensors will drive new applications in cost-efficient environments. Potential will become clear in the ENTERTAIN project (2027-2028).

Societal & Economic Impact

Our project empowers broadcasters with advanced imaging for today’s demands—immediacy, interaction, and immersion—while IP interfacing supports remote work, cutting eventrelated travel emissions. Realised security innovations enhance detection accuracy and safety operations by system/human operators. Miniaturized sensors drive AR/VR adoption, potentially reducing leisure travel needs, while new CMOS processes target cost-effective imaging beyond visible light, unlocking new potential for our society. These advancements address societal needs, foster economic growth, and contribute indirectly to environmental sustainability.

Patents, Standardisation, Publications

Work in the project resulted in 15 patents: 10 by ams OSRAM, 5 by Grass Valley. Teledyne Adimec and Grass Valley are involved in standardisation work, including SMPTE, ANWA-NMOS, CoaXPress. Project released a white-paper and organised two ePictureThis workshops on imaging technologies. Work from partners is presented on several congress. See website https://project-mantis.eu for details.

Future Developments

  1. Research on depth map generation continues in PENTA IMAGINATION.
  2. Collaboration Teledyne Adimec & Grass Valley on broadcast 4K sensor in Machine Vision continues in next projects and will be a vehicle to research the layer exchange process development at TU Delft.
  3. In Xecs ENTERTAIN Grass Valley and Teledyne Adimec will collaborate further on CMOS image sensor development and investigate the possibility to reuse broadcast CMOS image sensor IP for the development of CMOS image sensors for Teledyne Adimec products.
  4. ams OSRAM, Grass Valley, Teledyne Adimec and TU Delft will work on prototype feasibility of CMOS sensors and cameras with Layer Exchange process technology in ENTERTAIN.

Discover more in the MANTIS Project Impact Summary here.

Penta & Euripides² are Eureka Clusters operated by AENEAS.