Mikhail Sudbin

Mikhail Sudbin
Chief Technology Officer at Advalange

In my previous blog post “Why DO-178C Forces Software Development To Be More Agile” I discussed the importance of verification activities for a project in safety-critical areas and the reasons why you need to conduct these activities as soon as possible. Ideally, you should test right after a portion of requirements and the code for it are developed. Such an attitude is common in the Agile world. Moreover, two specific practices support it: Test Driven Development (TDD) and Continuous Integration (CI).

In this post, I’ll show how implementing these Agile practices enhances a DO-178C project by pulling testing inside a development mini-cycle while keeping it compliant with the standard.

First, a bit of theory.

Test-driven development practice can be resumed as follows: write a test, run it and ensure that it fails, write code, run the test again and ensure that it passes (and correct the code if it does not). The idea is that you need to define your desired functionality using the test and then develop code that passes this test. TDD may seem inapplicable to DO-178C project setup at first glance. Two small adjustments, however, turn things around: write a requirements-based test and segregate code development and test development with two different persons. Thus, the spirit of the TDD can be tailored organically to heavyweight development methodology.

Continuous integration was originally a practice of frequent merging of working software copies into a mainline. The main goal of CI was to deal with integration problems. Nowadays this technique is more widely considered. Usually CI is aided with automated build process and complemented with various quality-oriented activities. Coupled with TDD, continuous integration forms an outstanding basement for project-health monitoring and tangible quality increases. My post “Is continuous integration worth the price? Yes, I’m sure” describes CI practice in more detail and sheds light on its usage in DO-178C projects.

Now let’s move from theory to practice.

Common DO-178 projects often consist of several phases. These phases are called “releases”, “builds”, “versions”, “loads”, “labels” and so on. A phase duration usually varies from a couple to a dozen months or more. In turn, the work in each phase is split into a number of change requests. When development for all change requests is finished, the build is released and transferred to the verification group for testing. The next build is then worked on, along with testing of the previous build.

V-model

Such a project setup often results in error propagation deep into the lifecycle, introducing turbulence into the project flow and completion of scrap or redundant work. Ultimately, a project gains a significant chance of encountering last-minute surprises, additional unplanned phases to clean up bugs, schedule slips, missed deadlines and a whole bunch of known but uncorrected errors in the delivered software.

The goal of project life-cycle enhancement is to drastically reduce error propagation by uncovering and fixing them right at the time as the code portion is developed.

You need to do both technological and process adjustments to make these enhancement. From the technological perspective, you need two things in place:

  • a means for automated creation and deployment of the builds
  • automated test procedures

Having a good framework for test runs and results visualization is strongly preferable, but not mandatory. These two points are essential to make TDD and CI work. No process change will do any good if your test or build sequence is cumbersomely manual. Align your technology setup before proceeding to process tuning!

From a process perspective, the big picture is left unchanged. You still may have “releases” and split the work into particular change requests. The main tweak is applied to change-request implementation. In addition, a few more actions are needed for consistency and compliance purposes.

First, implement change requests by following these 7 steps for each single change request:

  1. Develop requirements.
  2. Conduct a requirements review.
  3. Develop code and test cases concurrently, according to requirements.
  4. Add newly developed code and corresponding tests into the CI pipeline.
  5. Obtain test results.
  6. Correct code bugs and adjust tests (if needed). Repeat steps 3 – 6 until requirements coverage and code coverage are achieved and test cases pass.
  7. Conduct code reviews and test reviews. Repeat steps 3 – 6 if any deficiencies are found during the reviews.

This 7-step recipe explains the title of this blog post: you need to switch from a big, awkward
V-shaped cycle to a series of short and focused mini-v-shaped cycles.

Second, do not consider the test results from step 5 as formal verification results required by the DO-178C standard. Treat these testing activities as part of a development cycle aimed at providing high-quality output. When all change requests scheduled for the “release” are completed, proceed with the formal release procedure and conduct a formal run (aka “run-for-score”). At the time of the formal run you will have all needed code requirements, code items and tests approved and controlled. This is a bridge to assure compliance to the standard. The formal run becomes an easy and predictable process because:

  • problems were resolved during development stage
  • established CI allows the formal run to unfold automatically
multi-v-model

It might seem that we do additional work, or do the work twice. This is true, to a point. Measure the overhead for such “additional” tasks and compare it with the total cost of an unplanned clean-up phase, schedule slip or a bug found after delivery. Take into account the resulting product quality and the impacts on customer satisfaction. It is wise to spend a small additional piece of effort at a certain point in the life cycle, but save a lot more in the long run. Practical experience shows up to a 30% increase of overall project performance after this agile flip-flop is made and assimilated.

Mikhail Sudbin

Mikhail Sudbin
Chief Technology Officer at Advalange

Software development standards in safety critical areas such as DO-178C are usually associated with classical waterfall  or V-model life cycle, a common but a misleading association. Hints to the more agile process are hidden inside the standard. Let me reveal them.

To start with, DO-178C does not impose any particular life cycle or methodology. It talks about objectives and activities that can be done to satisfy those objectives. Moreover, the objectives are quite general, for example: “High-level requirements are developed.”

It may seem that DO-178C leaves a lot of space for life cycle anarchy with such general definitions of single goals. However, a complete set of several dozens interrelated  goals drastically reduce the room for maneuver in a life cycle choice. The waterfall or V-model life cycles seem to be the easiest and most straightforward way to fulfill all of the goals at once. Nevertheless, the easiest way is not always the most appropriate.

The goals are arranged into 10 groups represented by tables A1 – A10 in the annex of the standard. Each group corresponds to certain aspect of a life cycle. In addition, the table defines the rigor of the process with respect to software level:

 

DO-178C objectives per software level

Logically, the higher the level is the more goals must be satisfied. However, look at the distribution of those goals. The number of verification goals outruns the number of development goals several times. Moreover, development goals are exactly the same for level A, level B, and level C software. So DO-178C deems that the reliability of the software bases on the thoroughness of verification. This conclusion can be even strengthened: you cannot say anything about reliability of your safety-critical airborne software until verification is complete!

Unsurprisingly, waterfall and V-model can become costly if an error spreads through the whole life cycle before it  uncovered in late verification stages. IIt is not a rare for additional iterations of the whole cycle to be added at the end of a product timeline to correct problems from the beginning of the story. This all results in nasty deadlines, a demoralized and exhausted project team, angry customers, and frenzied top management.

An obvious piece of advice for project managers is to implement as much verification as early as possible. Bingo! This concept is among the outstanding Agile traits.

I do not want to add fuel to the fire of the heavy-weight vs. Agile holy war. I do want to emphasize that Agile is not the equivalent of anarchy. Treat requirements, design, and other required outputs as a valuable part of your product rather than an annoying or exhaustive documentation. With such an attitude Agile methods fit perfectly: break your life cycle into smaller iterations and pull-in verification.

Of course, you need to be very careful implementing Agile in your safety-critical project. Tailoring the spirit of Agile practices to DO-178C environment is not an easy task. I will provide a practical example of such tailoring in my next blogpost.

Mikhail Sudbin

Mikhail Sudbin
Chief Technology Officer at Advalange

Is continuous integration worth the price? Yes, I’m sure.

Software development teams from different industries recognize continuous integration as a valuable practice. Some treat it as a “must-have” piece of an effective life cycle. However, in safety-critical areas and in the aerospace domain, in particular, there is still strong resistance to implementing continuous integration. Some people may think that continuous integration cannot formally fit into DO-178B\C.  Some people may think that those agile things bring no practical value into strict and rigorous development process.

These fears are groundless and I can show you that a DO—178C project can benefit from continuous integration.

Continuous integration aims to pull testing activities inside the development cycle rather than saving them for last. This goal breaks down to the following tasks:

  • Integrate all software pieces to see how they fit each other;
  • Check for any  regression possibly introduced by recent changes;
  • Check the functionality of newly implemented changes (if applicable).

These tasks should be done frequently enough to allow removal of any bugs before moving to the next stage of the life cycle.

“Any DO—178C project can benefit
from continuous integration.”

You will need several prerequisites to implement continuous integration:

  • A version control system to store and retrieve project artifacts coupled with a build tool;
  • Automated tests and a test execution framework;
  • Means to analyze test results.

Luckily, DO-178 features of heavyweight development approach provide a solid base for all three requirements.

DO-178 mandates comprehensive configuration control and change management.
Every single change request and the resulting changes in artifacts should be segregated and tracked. Usually it’s only a matter of setting up a configuration management tool to automate the build process and see how each subsequent change fits into the previous build.

Requirements-based testing is one of the main ideas of DO-178 standard.
Usually different groups develop code and test cases concurrently. If test strategy is established and test tools and environment are set up, nothing prevents the test group from developing test procedures along with test cases. This means that when a code change is deployed, everything is ready for continuous integration and, more important, for continuous testing.

DO-178 requires establishing thorough traceability.
Usually every single function is traced to corresponding requirements. Test cases and procedures are traced to requirements as well. Trace matrices are stored in a well-structured way. Everything is in place to select an appropriate set of tests and run them against a code change. Trace matrix will also aid analysis: you can identify the broken parts almost instantly.

“DO-178 features of heavyweight
development approach provide
a solid base for continuous integration.”

Of course, this approach works well if testing is automated or at least semi-automated. Modern software testing toolsets from VectorCast, LDRA, Rational, and others provide a solid base for test automation. Model-based development environments open new horizons for continuous integration by software-in-the-loop and hardware-in-the-loop concepts. Even old school manual visual tests can be adjusted for continuous integration. A way to set inputs automatically almost always exists. Output from a screen can be recorded for further analysis. Thus, all tests, with few exceptions, can fit into the strategy: “Run automatically, when a change is ready, analyze results when needed.”

Compliance is one of the most important questions when you develop airborne software. Some people believe that every tool should be qualified for continuous integration. It is not true. The DO-178C standard states: “Tools that are used to eliminate, reduce, or automate software life cycle process activities, and whose outputs are not verified, need be qualified.” You may choose different strategies to fulfill the DO-178 requirements such as:

  • Automation and pass\fail selection functionality may be qualified’
  • Tools’ outputs may be reviewed;
  • A combination of qualification and reviews may be used.

“It is not true that
every tool should be qualified
for continuous integration”

You may also go a different way – separate continuous integration and certification tasks. You may conduct formal verification (a k a run-for-score) once at the end without using automation tools. If continuous integration is applied throughout the project, you would be sure that your code is in good shape and your tests are correct and complete. There is minimal risk of getting some sudden costly updates..

Since continuous integration is not mandated by certification authorities some may seek to avoid the additional cost. It’s your choice, but make it wisely. Time and resources spent during earlier stages of your project to introduce test automation and continuous integration will save money later. You’ll be rewarded with fewer bugs, , avoiding integration hell and decreasing the cost of changes in later stages of project (Beck’s curve). Or defer testing and pay the price closer to the project deadline as you fix all those cascaded bugs. Each single change will be a challenge under such conditions (Boehm’s curve). Look at those two curves and decide which one you would rather see in your project.

Beck’s curve

Beck's curve

Boehm’s curve

Boehm's curve