An incremental upgrade cycle for healthcare systems delivers faster innovations, a better response to public health challenges, and improves the quality of care. An agile paradigm to validate safety and efficiency is still lacking to accommodate the faster rate of new product updates. Vivaldy project performs research on generic components that will enable efficient verification and validation.
Project background, objectives and challenges
Artificial intelligence (AI) can improve its performance when trained with more data. This new improved product would trigger more market adoption and more users, who would provide more data, which can be used to create another update. This however assumes that an update can be put on the market in a fast and efficient way, which is more challenging in
heavily regulated domains such as healthcare where any significant update needs to prove its safety and effectiveness.
In the Vivaldy project, researched generic components have been researched that could facilitate and accelerate this crucial validation and verification step. And not only for AI-enhanced products, but also by leveraging AI and other intelligent algorithms to evaluate updates of complex healthcare systems such as image-guided therapy systems. The first challenge is that AI can be used in many different applications and forms and an efficient solution should cater to the full range of (potential) healthcare products. To cover the broad range of challenges that relate to this topic, the consortium started with 4 complementary industry use cases, both to ensure the applicability of the developed methodologies on real products and processes and also to benchmark the resulting tools. The second challenge is that this research is not constrained to the classical technical innovations, instead diverse internal and external stakeholders have an important role, e.g. regulatory officers, testers, engineers, data privacy officers, process owners, agencies responsible for certifications, etc… Any (technical) solution needs to be accepted by all stakeholders before they can create any actual impact in the product development cycle. Based on the interactions with these stakeholders the consortium developed a set of regulatory guidelines for efficient validation and verification of updates in a healthcare system, which have been disseminated to various agencies responsible for certifications.
Technological achievements
In Philips’ Azurion systems PC components must be replaced several times during the platform’s lifetime. After such a replacement, the performance of the system must remain the same. Until now this had to be tested extensively using physical test systems. In Vivaldy, an AI-based algorithm was developped to predict the performance based on the component’s characteristics, helping first-time-right design and saving months of the testing effort. There are hundreds of configurations of Azurion systems with different tables, arcs, software, etc. When updating applications, the impact of the update on all those configurations needs to be carefully assessed and tested. Vivaldy provided a solution to assess the impact using models and automatically run the relevant tests on a virtual system that dynamically switches to the right configuration. This reduces the need for real test systems and enables shorter update cycles.
The new monitoring tool has the capability to:
1. identify user groups for which the AI component underperforms,
2. do this based on real-world usage data where the pathology-proven diagnosis is not always available,
3. forecast the performance of future updates if these would be trained with more data, and
4. compare the performance of an update with the performance of an already deployed AI.
The main achievements involve different complementary methodologies, supporting tools, and accompanying practices that reduce system verification and reverification time, effort, and cost by determining and reducing verification scope and verification automation through model-based testing. The different approaches seamlessly connect to existing development and verification practices and processes.
Societal & Economic Impact
The Vivaldy project creates impact in 3 phases. Phase 1 already has proven economic impact today. The partners can now identify subpopulations where an AI could/should be improved and how much additional data is required to achieve a certain performance gain. This makes the development planning and related business case more reliable. Furthermore, due to the Vivaldy project a faster time to market can be achieved. For example, by selecting better potential subcomponents for updates of a complex healthcare system, the testing throughput time can be reduced by 3 months compared to the iterative trial and error approach, or by using the automated data quality inspection tools 87% fewer images need a manual inspection. Phase 2 required more integration into the existing development ecosystem, and therefore was implemented at the end of the project. Nevertheless, these components already resulted in the early detection of 39 bugs during the project, which has not only improved the quality of the product but also resulted in a significant cost reduction as early detection is typically 10 to 100 times cheaper than resolving these in a later stage. These software-based tests do not require scarce physical test systems, thus further reducing the time to market. The biggest gain will come in phase 3 when the proposed regulatory guidelines will be accepted by legislation, as this would reduce the time to market from several quarters to even a year.
Future Developments
The work initiated in the VIVALDY project will continue in a number of new projects e.g., VITO will create training for notified bodies in Europe in the context of the NoBoCAP (Notified Body Increased Capacity) project. This will enable notified bodies to better evaluate AI-based medical devices. Research activities will continue in the context of the Horizon Europe REALM (Real-worlddata Enabled Assessment for health regulatory decision-Making) project, while model evaluation and model performance monitoring services will be offered to the industry in the context of the European TEF-Health.
Discover the full VIVALDY Project Impact Summary.
PENTA is an EUREKA Network Cluster, operated by AENEAS.