Mikhail Sudbin

Mikhail Sudbin
Chief Technology Officer at Advalange

7 Tips That Can Make Your Software Tool Qualification Easier

Qualification (aka certification) of software tools is a common requirement in safety-critical engineering. Often this aspect of development life cycle raises many questions and issues. Below I summarized my experience in aerospace area into seven tips. Though the tips are tight to RTCA DO-178C and DO-330 standards, other safety-critical domains will benefit as well. Consider these tips to increase efficiency of your project.

Tip #1. Evaluate tool qualification criteria correctly

Normally you don’t need to qualify each single software tool. Identify tool usage scenarios precisely and evaluate them against qualification criteria stated in the standards. It’s like a court trial: the standard is a judge and you are a lawyer. You need to justify that your tool does not require higher level of qualification or qualification at all. If your arguments are reasonable and sound, you will be okay with the certification authority.

Tip #2. Consider tool chain usage accurately

Usually several tools are used consecutively to obtain a result. You need to make a wise choice. One option is to qualify every tool in the chain as a development tool. Another option is to add one more tool in the chain to check the result and qualify only this tool as a verification tool. Sometimes it’s more reasonable to add an additional verification tool even if the chain consists of a single development tool.

Tip #3. Qualify only required functionality

Once again, evaluate the tool usage scenario carefully. Often you need to qualify only a portion of functionality. In that case think about partitioning this functionality, otherwise you’ll have to qualify the tool completely. A good example is splitting a tool into a GUI application and a command line utility. The command line utility implements business logic and is being qualified. GUI is developed in a less rigorous way. Review of command line log serves as a bridge between these two components. Such an approach may save you considerable effort.

Tip #4. Remember about COTS tool qualification package issue

Overreliance on COTS tool qualification package may produce a number of sleepless nights. A tool provided with a qualification package from a vendor does not mean that you can proceed without further actions. In most cases, you need to do some additional activities such as running qualification tests and evaluating the results. Such additional tasks may take significant time. Moreover, a tool qualification package may require prerequisites that your target system cannot provide. For example, it may require a file system on your target to store intermediate data and results or an ability to debug step-by-step. Consider additional qualification efforts and package constraints when planning tool usage.

Tip #5. Balance your cost

Evaluate the total cost of tool qualification. Estimate the savings that this tool will bring to your project. Sometimes it’s more efficient from a time and budget perspective to refuse tool qualification and to conduct full cycle of verification activities for outputs of the tool. This is especially useful in the cases where a unique component is being created and there is no plan of further usage of the tool.

Tip #6. Timely plan tool qualification activities

Negotiate tool usage and qualification strategy with the authorities at the start of the project. Conduct tool qualification tasks along with system development. Evaluate COTS tool qualification package before deciding on tool usage. This advice may seem obvious. Many projects, however, start thinking about tool qualification several weeks prior deadline when there are many other functional and developmental troubles.  You will not have the luxury to spend valuable time wrapping up qualification things at that time.

Tip #7. Combine previous tips

Each of these seven tips is valuable on its own. However, you can achieve better results by combining them creatively. Try different tool chains and evaluate total implementation effort for different variants. Don’t be shy about adding COTS tools into chains of homemade tools. Transfer qualification tasks from development to verification tools by adding additional activities into your product life cycle. It’s like music: you have just seven notes but you can play thousands of melodies.


Recipe on software tool qualification


In conclusion, let me illustrate these seven tips with a practical example of parameter item data generation tool. The tool uses some portion of requirements presented in formal notation to generate a binary data file. The file is then uploaded into target.

Tip #1: This tool is definitely a development tool if we use its binary outputs without further review and testing. It should be qualified according to DO-330 TQL1-TQL3 depending on the system’s design assurance level.
Tip #5: Qualification of a development tool is an expensive task that may overwhelm the budget.
Tip #1, #2 and #5: The cost may be reduced by adding additional verification tool (DO-330 TQL5) and some manual verification into the chain.
Tip #3: We can minimize qualification effort by extracting data file generation functionality from GUI, setup and other components of the tool.

As a result parameter data item tool is transformed into:

  1. GUI, setup and other components that can be developed with any rapid architecture development technique. These components produce a kind of database (DB) file for further generation, for example in xml format. No qualification is needed.
  2. Command line utility that converts DB file into a binary file for target. In addition, this utility produces a .txt file which contains DB parameters and execution log information in human-readable form. Exact correspondence between DB file, binary file and human-readable .txt file is qualified according to the verification tool criteria.
  3. The process is complemented with an additional review step. A developer adds DB and text files into configuration management system along with the binary output. A verification engineer then checks that the information in the text file matches the requirements.

Thus, this example shows that an inventive approach can transform a costly qualification of a development tool into a far easier sequence of verification and utility qualification. However, don’t forget about tip #6 – any creative plan should be approved by the certification authority.

Mikhail Sudbin

Mikhail Sudbin
Chief Technology Officer at Advalange

5 common traps can hurt your DO-178C project seriously if recognized late

DO-178C, Software Considerations in Airborne Systems and Equipment Certification by RTCA, regulates the process of software development in the aerospace domain. The document, however, does not provide any clear recipe of getting things done in a given project. Each particular team has its own way of interpreting DO-178C objectives and limitations. Below are five cases that, in my experience, have caused the most arguments and misunderstandings. I call these aspects “traps” because choosing the wrong way in implementing them will have a tangible, negative impact on your project.

#1. Waterfall lifecycle trap

The opinion that DO-178C mandates a heavyweight waterfall-like life cycle is wrong. Any methodology is good, even an agile one, if you follow transition criteria and fulfill the objectives. Choose what fits your project and organizational culture best and complement it with additional steps to get the compliance. Do not exclude any activities that bring value to your product just because their conformance to the standard is questionable. Simply do not take formal credit from such activities. And vice versa, don’t be scared to bring in extra tasks to meet formal requirements. Often a week spent on dedicated formal tasks saves a month of ineffective and useless work on artificial life cycle stages.

#2. Trap of the undefined robustness

Often people recall about robustness at late stages of verification. It may look like: “Ouch, we need to add some kind of robustness testing to pass through SOI audit.” Software made this way is nothing more than a colossus with feet of clay. You must consider robustness from the very beginning and transform it into appropriate outputs throughout project life cycle: identify abnormal situations and containment actions in the requirements, insert corresponding features into your design and do not let untraceable defensive code ooze into your implementation. Robustness testing should be just another portion of a requirements-based test set if you do it right.

#3. Structural testing trap

DO-178C stands on a requirements-based testing concept. Nevertheless, a word combination “MC\DC tests” often circulates in engineers’ conversation. Forget such word combinations. Tests should check the requirements. MC\DC decision or statement coverage is just a measure that shows overall mutual consistency and completeness between requirements, code and tests. Having a coverage gap does not necessarily mean that a test is bad. Often requirements may be inadequate and lack details or you may have untraceable additional code. The worst mistake you can make is adding a synthetic test case just to exercise certain combinations of software variables.

#4. Overzealous tool qualification trap

DO-178C pays additional attention to tool qualification. Dedicated standard’s supplement DO-330 regulates this aspect. However, there is no need to qualify every single software tool. Moreover, tool qualification is an expensive process and you need to choose your qualification strategy wisely to balance your efforts. You need to understand tool qualification criteria and evaluate costs for each tool qualification level. Sometimes it is even more efficient to introduce additional activities, such as reviews or analysis, rather than to qualify a tool. Another mistake is over-reliance on the tool qualification support package from the tool vendor. You can rarely take the package as is and provide it as qualification evidence. Tune-up to a project’s environment, qualification test runs and results analysis may consume considerable time. You need to take this effort into account when figuring out your tool qualification strategy.

#5. Reckless use of external components trap

Introducing a third party or open source component into your software often seems like an attractive idea. However, any single portion of your code must be DO-178 compliant. This means that you cannot link a library to your project without further actions. The statement that “the community uses this library for ages” will not work for a formal credit. You need to have a solid plan of adaptation of such components into your project. You may use a certification support package or go through a reengineering cycle to capture requirements for a code library and to conduct all needed verification.  Consider this extra effort when you are thinking about an external component. Sometimes developing your dedicated library is more efficient than working out a common third-party component.

The list above is not exhaustive. There are many more technological, design or process traps you can face along the way. Nonetheless, these five mistakes will be costly if revealed in the late stages of your project. My advice is: keep an eye on these aspects from the very beginning and don’t forget to negotiate your approach with your DER.