COMPUTER SYSTEMS RESEARCH
Software Testing 2 - Dynamic Testing Analysis
3rd Quarter 2006
- Also see the Testing Report worksheet to attach your results

  • Resources:

    1. Background and vocabulary for dynamic analysis of your program
      • Dynamic analysis techniques:
              "While static analysis techniques do not necessitate the execution of the software, 
              dynamic analysis is what is generally considered as ``testing``, i.e. it involves 
              running the system. ``The analysis of the behaviour of a software system before, 
              during and after its execution in an artificial or real applicational environment 
              characterises dynamic analysis'' (Hausen84, 326). Dynamic analysis techniques 
              involve the running of the program formally under controlled circumstances and with 
              specific results expected (Miller84, 260). It shows whether a system is correct in 
              the system states under examination or not (Hausen84, 327)."
              (see Glass Box testing, Dynamic analysis techniques)
        
          "Verification Techniques
              There are many different verification techniques but they all basically fall into 
              2 major categories - dynamic testing and static testing.
        
            * Dynamic testing - Testing that involves the execution of a system or component. 
              Basically, a number of test cases are chosen, where each test case consists of test 
              data. These input test cases are used to determine output test results. Dynamic testing 
              can be further divided into three categories - functional testing, structural testing, 
              and random testing."
              (see Verification/Validation/Certification, CMU)
        

      • Random testing - a series of random inputs, acting as a collection of random users, to test program's general reliability
            "The selection and generation of test data in glass box tests is an important discipline. 
            The most basic approach to test data generation is random testing. For random testing 
            a number of input values are generated automatically without being based on any 
            structural or functional assumption (Sneed87, 10.3-4) and (Bukowski87, 370)."
            (Glass Box testing, Dynamic analysis techniques)
        
            "Random testing - Testing that freely chooses test cases among the set of all possible 
            test cases. The use of randomly determined inputs can detect faults that go undetected 
            by other systematic testing techniques. Exhaustive testing, where the input test 
            cases consists of every possible set of input values, is a form of random testing. 
            Although exhaustive testing performed at every stage in the life cycle results in a 
            complete verification of the system, it is realistically impossible to accomplish. [Andriole86]"
            (Verification/Validation/Certification, CMU)
        
        

      • Structural testing - Test the reliability of your program's data structures - path and branch testing
             "There are also two more sophisticated approaches to test data generation:
             structural testing and functional testing. ``Structural testing is an approach to 
             testing in which the internal control structure of a program is used to guide the 
             selection of test data. It is an attempt to take the internal functional properties 
             of a program into account during test data generation and to avoid the limitations 
             of black box functional testing'' (Howden80, 162)."
             (Generation of test data in glass box tests
        
             "Structural testing - Testing that has full knowledge of the implementation of the 
             system and is an example of white-box testing. It uses the information from the internal 
             structure of a system to devise tests to check the operation of individual components. 
             Functional and structural testing both choose test cases that investigate a particular 
             characteristic of the system."
             (Verification/Validation/Certification)
        

      • Functional testing - test your program's functions and modules; use various inputs to test the reliability of your functions/modules:
             "Functional testing as described by (Howden80) takes into account both functional 
             requirements of a system and important functional properties that are part of its 
             design or implementation and which are not described in the requirements (Howden80, 162). 
             ``In functional testing, a program is considered to be a function and is thought 
             of in terms of input values and corresponding output values.'' (Howden80, 162)."
             (Generation of test data in glass box tests)
        
             "Functional testing - Testing that involves identifying and testing all the functions 
             of the system as defined within the requirements. This form of testing is an example 
             of black-box testing since it involves no knowledge of the implementation of the system."
             (Verification Techniques
        
        
      • Path and branch testing - try to completely test all of the various logical paths the inner workings of your program can take:
             "Among the most important dynamic analysis techniques are path and branch testing. 
             During dynamic analysis path testing involves the execution of the program during 
             which as many as possible logical paths of a program are exercised (Miller84, 260) 
             and(Howden80, 163). The major quality attribute measured by path testing is program 
             complexity (Howden80, 163) and (Sneed87, 10.3-4). Branch testing requires that tests 
             be constructed in a way that every branch in a program is traversed at least once 
             (Howden80, 163). Problems when running the branches lead to the probability of later 
             program defects."
             (see Glass Box testing, Dynamic analysis techniques)
        
        

    2. Background and vocabulary for validation techniques for testing your program:
      • Validation techniques:
            "There are also numerous validation techniques, including formal methods, fault injection, 
            and dependability analysis. Validation usually takes place at the end of the development 
            cycle, and looks at the complete system as opposed to verification, which focuses on 
            smaller sub-systems.
        
            * Formal methods - Formal methods is not only a verification technique but also a 
            validation technique. Formal methods means the use of mathematical and logical techniques 
            to express, investigate, and analyze the specification, design, documentation, and 
            behavior of both hardware and software.
        
            * Fault injection - Fault injection is the intentional activation of faults by either 
            hardware or software means to observe the system operation under fault conditions.
        
            * Hardware fault injection - Can also be called physical fault injection because we 
            are actually injecting faults into the physical hardware.
            
            * Software fault injection - Errors are injected into the memory of the computer by 
            software techniques. Software fault injection is basically a simulation of hardware 
            fault injection.
        
            * Dependability analysis - Dependability analysis involves identifying hazards and then 
            proposing methods that reduces the risk of the hazard occuring.
        
            * Hazard analysis - Involves using guidelines to identify hazards, their root causes, 
            and possible countermeasures.
            
            * Risk analysis - Takes hazard analysis further by identifying the possible consequences 
            of each hazard and their probability of occuring. [Kopetz97]"
            (Validation techniques)
        

      • Tools
            "There are tools for test data generation on the market that can be used in combination 
            with specific programming languages. Particularly for embedded systems, tools for 
            test data generation are useful, since they can be used to simulate a larger system 
            environment providing input data for every possible system interface (Thaller94, 178). 
            In other words, if a system is not fully implemented or not linked to all relevant 
            data sources, not all system interfaces can be tested, because no input values are 
            given for non-implemented functions. Data Generation tools provide input values for 
            all available system interfaces as if a real module was linked to it."
            (Generation of test data in glass box tests