End-to-End Simulation Testing Utilizing Agile Methodologies
In the automotive industry, simulation testing is a powerful tool to verify the effectiveness of the software code behind the features of advanced driver-assistance systems (ADAS).
Simulations allow developers to run automated tests that can be performed more frequently and are faster, more cost-effective and more repeatable than on-the-road vehicle-level validation tests. Simulations also allow developers to test certain conditions before the actual hardware is available and enable rapid prototyping, which helps to reduce time to market. Results can easily be shared virtually across distributed teams, offering a scalable solution for developers to collaborate globally.
However, as with any new technological development within an organization, there can be a gap between adopting state-of-the-art technology and standardizing its use. To get the most they can out of simulation technology, organizations need to quickly standardize their approach in three major areas: global alignment, process planning and performance tracking.
A NEW WAY OF TESTING
Software is a powerful force in the automotive industry. It not only enables significant and rapid advances in vehicle capabilities but also frees developers from having to work in a specific location. They can be anywhere, writing code and uploading it to the cloud. Using simulation testing, the code can be tested in the cloud — early and often, in a multitude of simulated scenarios.
Although decentralized software development offers those strengths, it can be a weakness if the process lacks structure. Development groups dispersed around the world might bring testing approaches that are not aligned with one another, do not follow the same processes and do not have the same level of rigor when it comes to tracking performance. That can lead to inefficiencies that undermine the cost savings that simulation testing promises.
To avoid those pitfalls, the first step is to ensure that the organization is aligned at a global level.
GLOBAL ALIGNMENT
Large automotive suppliers and OEMs employ tens of thousands of engineers working in different regions around the world. The cost would be prohibitive if each regional group were to establish its own software development processes with different simulation tools. Aligning globally on which tools to employ is essential to reducing licensing costs and streamlining team responsibilities to ensure that suppliers are able to provide the maximum benefits to OEMs.
Aptiv’s experience across multiple customer programs has demonstrated that internal organizational alignment of simulation testing processes can increase the rate of successful vehicle integration from 22 percent to more than 70 percent while decreasing defects found in the vehicle by 62 percent.
There are several important factors to consider in a global approach to simulation:
Phased adoption
When a toolset is selected and introduced, implementing a phased, step-by-step adoption process — focused on specific areas or subsets of the organization — is essential to minimize disruption, ensure smooth integration and allow for adjustments based on feedback from developers.
Interoperability and compatibility
When software and hardware architecture designs are harmonized, teams can work together seamlessly. By using the same or compatible simulation platforms, it becomes easier to share files, build libraries, collaborate on projects and exchange information. This enhances interoperability, reduces compatibility issues and streamlines communication between team members and with OEMs.
Scalability and flexibility
Platform alignment makes it easier to scale up or down, adapt to changing requirements, integrate new team members or departments, quickly deploy resources, share workloads and foster cross-functional collaboration with minimal technology barriers.
Security and compliance
In the automotive industry, security is paramount. Implementing consistent security protocols, access controls and data protection measures across teams can improve security and compliance. Centralized monitoring and management of platforms allows for better compliance with regulatory requirements and faster response to security vulnerabilities or threats.
PROCESS PLANNING
While it is common to think that mandating additional development steps will slow the process down, uniform testing practices do not increase workload when implemented correctly — they decrease it.
Imagine if, during the construction of a house, the drywall were put up before the electrical wiring was installed. The drywall would have to be removed to allow the electrician to do their job, resulting in a lot of wasted time and resources.
A consistent and structured workflow is just as essential in software development. Without a plan in place, engineers at one stage of development might conduct tests without sharing the results — leading developers downstream to redo tests unnecessarily. Similarly, developers could find that certain tests that should have been conducted earlier in the process were skipped, resulting in new code having to be scrapped entirely when the product is rolled back to the previous stage.
These kinds of disruptions can add up and tremendously delay a product launch. In fact, a study with a major OEM showed that every time a vehicle is flashed with ineffective software, it costs suppliers and OEMs nearly $13,000 in lost time. To accelerate time to market and maximize cost reduction and efficiency, a standardized software development process plan must be defined globally.
Shifting left with ASPICE and the V-model
Simulation processes should be aligned with ASPICE (Automotive Software Process Improvement Capability dEtermination), the industry-standard guideline for evaluating software development processes. ASPICE helps automotive suppliers incorporate best practices to identify defects earlier in development and ensure that OEM requirements are met.
ASPICE also leverages the V-model of software development, which splits the process into two parts: The left side of the letter V represents the design and development steps, and the right side represents the testing steps. Every development step is mirrored by a testing step.
V-Model in Automotive
The V-model is the predominant style of software development and testing in the automotive industry
Performance tracking is necessary to monitor the efficiency of simulation testing programs. Simulation tools are expensive, but applying global reuse strategies allows teams to formulate ROI metrics and monitor cost savings.
Implementing an effective simulation process plans
There are two primary components to an effective simulation process plan:
- Trigger and release Each phase of the testing pipeline should be gated with a trigger-and-release protocol to ensure that software does not move from one stage to the next without first being appropriately vetted. In this scenario, it is determined in advance which steps in the development process will trigger a simulation test, and only after the code has passed the test is the product released to the next step
- Gates Establishing owners at each gate guarantees that everyone knows who is responsible for executing each test. Ensuring that all teams have access to the same testing platforms (through global alignment) allows developers to verify which tests have been completed to avoid redundant testing. It also allows the simulation teams conducting the tests to focus on the development of their execution suite rather than on debugging software and evaluating results.
PERFORMANCE TRACKING
Performance tracking is necessary to monitor the efficiency of simulation testing programs. Simulation tools are expensive, but applying global reuse strategies allows teams to formulate ROI metrics and monitor cost savings.
Process Groups
In this example, each process group employs the V-model and acts as a gate to validate software stability.
One metric is unplanned software releases. An unplanned software release is any additional software build needed to address a software defect in a sprint. For instance, if a program manager plans one internal release per two-week sprint but actually deploys three due to defects found in the initial release, the last two releases are considered unplanned. The number of unplanned software releases a program generates is a vital performance metric to determine the maturity of both the process and the build contents.
Another metric is defect mitigation. As system complexity increases, the cost of remedying software defects increases. Ensuring that the process is built around catching the defects where they occur is extremely important. The money saved from cost-avoidance techniques can be automatically collected and analyzed to ascertain where to invest further resources by determining what stages of simulation are catching the most defects.
TRANSFORMING SIMULATION TESTING
As simulation testing becomes more prevalent in automotive, employing a standardized approach that includes global alignment, process planning and performance tracking is essential to get the most benefit from these innovative tools.
As the only supplier of both the brain and the nervous system of the vehicle, Aptiv embraces intelligent integration to optimize the strengths of simulation technology. We have automated the process of building and testing software packages through our unique CI/CD pipeline that leverages Wind River Studio and virtual ECU environments — transforming simulation testing and paving the way for OEMs to accelerate automotive software development.
Measuring ROI
Here are some typical costs of software defects found at each level of testing. More perspective is available here.
--------------------------------------------------------------------------
In the automotive industry, simulation testing is a powerful tool to verify the effectiveness of the software code behind the features of advanced driver-assistance systems (ADAS).
Simulations allow developers to run automated tests that can be performed more frequently and are faster, more cost-effective and more repeatable than on-the-road vehicle-level validation tests. Simulations also allow developers to test certain conditions before the actual hardware is available and enable rapid prototyping, which helps to reduce time to market. Results can easily be shared virtually across distributed teams, offering a scalable solution for developers to collaborate globally.
However, as with any new technological development within an organization, there can be a gap between adopting state-of-the-art technology and standardizing its use. To get the most they can out of simulation technology, organizations need to quickly standardize their approach in three major areas:
Global alignment. Large automotive suppliers and OEMs employ tens of thousands of engineers working in different regions around the world. The cost would be prohibitive if each regional group were to establish its own software development processes with different simulation tools. Aligning globally on which tools to employ is essential to reducing licensing costs and streamlining team responsibilities to ensure that suppliers are able to provide the maximum benefits to OEMs.
Process planning. While it is common to think that mandating additional development steps will slow the process down, uniform testing practices do not increase workload when implemented correctly — they decrease it.
Performance tracking. Performance tracking is necessary to monitor the efficiency of simulation testing programs. Simulation tools are expensive, but applying global reuse strategies allows teams to formulate ROI metrics and monitor cost savings.
To learn more about how to implement simulation testing with each of these areas in mind, read our white paper.
