Objective:
To build test development environments that produce a set of integrated testing tools, including self-adapting test suites that are dynamically created and factor in unique characteristics for defined subsets of a given specification or sets of specifications.

Background:
Many enterprises have become driven by digital data, including health care, manufacturing, e-commerce, and homeland security. Each of these enterprises defines standards for the exchange and manipulation of industry-based information. These standards are often complex, with many optional features and include references to other standards or are comprised of many base standards that must work seamlessly together. As the plethora of standards is promulgated throughout the industry, the demand for metrics to measure conformance is essential. Old methods of systematically hand crafting tests over the course of several years must give way to newer methods of generating self-adapting tests.

It has become well known that software is not adequately tested and is rushed to market full of bugs. A recent NIST study states that the annual cost to the country for inadequate testing is as high as 59 billion dollars. A primary reason for inadequate testing is the time and expense of developing comprehensive test suites. Today’s standards are often large and complex, and have gained widespread industry support through universal and all-inclusive designs with many optional features. However, this approach results in standards that cannot be sufficiently constrained to provide a single and consistent interpretation – prohibiting plug-and-play installations. As a consequence, systems are difficult to implement and debug, resulting in undue costs. Many standards organizations have addressed this problem by developing profiles. A profile is a proper subset of the standard that specifically states the optional constructs and processing rules. Profiles are key to standardization at the implementation level and promote plug and play systems. However, if profile rules are not followed then interoperability problems will still persist. Conformance testing of these profiles is essential.

Profiles provide a path to conformance and interoperability testing. However, since each installation is free to define its own set of constraints to meet individual requirements, a single profile or “standard” is not obtainable. Profile creation is essentially unbounded, which proves problematic for testing. Legacy and the dynamic nature of today’s industries limit the consolidation that is hoped for with profile creation. The state space typically increases as opposed to decreasing. In this environment, testing becomes a challenging and difficult problem. Current practice involves meticulous by hand debugging of implementations as problems appear. Test suites and tools are not available; consequently advanced testing is limited. The need for adaptable and automated test tools is apparent. Our innovative approach is centered upon the automatic and dynamic creation of tests based on profiles represented in XML. Building upon our prior experience in XML-based testing, we propose to develop a data driven approach that consists of encapsulating the structure and semantics of a profiled specification in a NIST-developed test language. A core set of reusable components will be built as the test engine, providing generic methods to iterate over test objects, generate data values, and call specific post-processing templates. Transformation templates will be developed to generate specific tests for a given profile.

This approach will be applied to Health Level 7 (HL7) V2 specifications, IEEE Medical Device Standards for ventilators, the Linux-based stdlib, and various W3C XML specifications. Each of these communities is responsible for developing large and complex specifications, thus requiring an automated and self-adapting approach. There is strong consensus that a standards-based implementation is the key for the US to reap the benefits of an information technology infrastructure. However, achieving true interoperability in major enterprises is difficult due to the complex interaction of the standards involved. Having a NIST-developed method for developing test suites that can be tailored to a particular community promises to have a positive impact on many strategic focus areas, significantly reducing the software development cost, and promoting interoperability among many critical applications.

 

Page created October 2007

Web site point of contact