Consortium Tackles Testing of Complex Systems of Systems

The Software Engineering Institute (SEI) has announced the formation of the System-of-Systems Test and Evaluation Consortium (SoSTEC) to develop technology for identifying conclusions about the functionality, reliability, dependability, performance, and safety of systems of systems.

In short, the goal of this SEI-managed consortium is to generate new understandings and consensus about the nature of the problem, promising approaches, and additional research that would have significant payoffs.

"The vision for network-centric warfare and network-centric operations radically changes the current approach to developing and deploying systems," explains Patricia Oberndorf, director of the Dynamic Systems Program at the SEI and leader of the effort to organize the SoSTEC. "Traditional engineering techniques for both systems and software will not be able to completely meet the system-of-systems test and evaluation need."

According to the SoSTEC vision, in the service-oriented architecture (SOA) concept, services -- not systems -- will be deployed and composed dynamically into systems. These services become the components of future systems, but at the time of a service's deployment no one will know or be able to anticipate all the other services and components with which it might be combined.

Consequently, SoSTEC examines the issues related to how we develop sufficient confidence in the quality of systems of systems. Traditional engineering techniques for both systems and software -- including conventional test and evaluation approaches -- will not be able to completely meet the system-of-systems test and evaluation need for at least the following reasons:

• It is unclear what it means for a system of systems to fail.
• Even given operational scenarios defining the expected capabilities of a system of systems, the amount of testing required to develop any degree of confidence in the system (using the usual statistical approaches) would be prohibitive in time and expense.
• Systems of systems are so large that they are always undergoing changes. It is unclear what techniques should be used to ensure that changes do not have unexpected emergent effects.
• Different users expect/need different qualities of service in different situations.

The consortium will be composed of representatives from a spectrum of organizations dealing with the difficulties of testing systems of systems, including:

• Organizations responsible for evaluating the suitability of systems of systems (e.g., U.S. Department of Defense test and evaluation organizations)
• Organizations responsible for developing such systems (e.g., defense contractors, large healthcare organizations, and organizations that have established net-centric or SOAs for their own purposes)
• Academic researchers who are addressing the system of systems T&E problem (by invitation)

Posted by Jon Erickson at 12:13 PM  Permalink