Across various domains, the ITEA project XIVT (eXellence In Variant Testing) has developed tools and methods for the more efficient and accurate testing of products which are highly configurable or available in many variants.
Software-intensive systems are not typically designed and built from scratch per customer or order but instead appear as variants of or over an existing product. Over time, companies are faced with an enormous range of product variants to be tested – just 30 configurable features can result in a billion different products. However, no industrially viable concept for effective variant testing is currently available.
XIVT foresees a competitive advantage for end-users in the telecommunications, railway, automotive and industrial production domains by reducing their feedback cycles and optimising their development processes. 28 methods and tools have therefore been developed to test highly configurable, variant-rich embedded systems. An integrated, configurable and extendable toolchain and high-end quality assurance will also allow a shift to adaptable, flexible and autonomous applications. The result is a series of cost-effective solutions that are applicable to five markets.
Technology applied
XIVT’s key technology outcomes are methods to analyse requirements and correlations, approaches to prioritise and select requirements/test cases and certification guidelines for product families. These elements are brought together in an end-to-end testing platform and toolchain which integrates certain application-specific tools. In industrial production, for instance, XIVT has developed simulation-based methods for testing digital twins in the cloud to significantly reduce configuration efforts.
Outside of the four use-cases, Arcelik is developing software for automation scripts to test televisions using extracted user interface models. This will remain in-house, but a proof-of-concept will increase know-how among tool providers worldwide.
Regarding specific tools, a number of successes stand out. The reuse recommender solution VARA, for example, uses natural language requirements and automated similarity analysis to identify software components which could be reused in new projects, while SEAFOX is the first combinatorial test suite generation and selection tool for industrial control software. ReForm is a new algorithm and tool for the automated formalisation of textual requirements using natural language processing/ machine learning and Recorder Tester has used static analysis/horizontal scaling to increase test coverage for reusable C/C++ code with many build flags. Such solutions have formed the basis for demonstrators which prove that the project’s techniques can be applied in practice.
Making the difference
For industrial end-users, the major benefits of XIVT are effective product line engineering (enabling efficient production by reusing shared assets across different product versions) and faster development cycles to facilitate new customer requirements and provide faster feedback. VARA, for example, can reduce software reuse analysis by over 20 days with a prediction accuracy of 82%. Similar successes have been achieved
for SEAFOX (85% code coverage and an average generation time of 2.2 minutes per system under test), ReForm (90% average detection accuracy for entities) and Recorder Tester (over 80% variant coverage versus a starting point of under 5%). Overall, programmers/engineers who utilise XIVT spend 10-40% less effort on requirements analysis and the ranking, prioritisation and selection of features, while the time needed for test case generation and test execution can be reduced by up to 95%. Additionally, the project has developed a methodology for the orchestration of different tools, allowing end-users to better understand how these can serve their unique needs and forming a bridge between XIVT’s technical results and commercial uptake.
The business potential is enormous: across the four domains, a total addressable market revenue of USD 32 million is forecast for 2022, rising to USD 100 million by 2026. Direct exploitation includes integration in existing products, such as WinTrust’s InnSpect tool which was previously unable to systematically reduce the number of test cases for highly configurable, multi-variant systems, and has now reduced the number of test cases between 30 to 70%. QA Consultants will also use its channel partnerships in the automotive industry (including Ericsson, Hitachi and IBM) to promote and deliver the XIVT solution set to their clients globally. They expect to increase their market revenue from USD 7 million in 2021 to USD 37 million in 2026. Consultancy and industrial training are other important exploitation approaches, with MDU utilising XIVT in online courses for engineers and software developers. Over 300 students were admitted in the last three years, with a 60% increase in 2021. Finally, the prototypical toolchain and several plug-in modules have been made available as open source and have already been adopted by several companies, like Bombardier and ABB.
XIVT’s dissemination campaign has excelled, resulting in 31 conferences/ workshops, four journal papers and a series of podcasts with over 2000 downloads so far. This has led to recognition in multiple fields, such as an award at the 2021 Requirement Engineering for Software Quality Conference and first place in the Automated Software Testing Industrial Competition 2021. To increase the sustainability of XIVT’s results, all partners have also contributed to standardisation efforts for homologation and certification, resulting in a list of recommendations which are intended for use by assessors and safety engineers worldwide. In short, positive outcomes are expected as the project continues to build an ecosystem for the further dissemination of next-generation variant testing.