Talk:Unit testing and Integration testing

From DANSE

Mike, thanks for writing this. I think overall it's strong. My main objection:

  1. We don't tell people to write their component cores using the gtk GUI framework. Why tell people to write test code using one particular test framework? Put differently, shouldn't the core of the test, the part that expresses logic about how the tested code ought to work, shouldn't that be framework independent?
    • I think your presentation of test-first programming is good, and the examples of using unittest are useful. I'm only suggesting first write the tests as functions in a module or methods of a class, and then write a unittest.Testcase subclass that calls those functions. That way, the core code is unchanged if we later migrate to a different framework.

Less important stuff:

  1. Your example code is defensive. For example, it runs a bunch of conditionals to make sure that it got an integer, not a boolean (if you are doing this, why not just check n.__class__.__name__ == 'int'?), and an integer greater than 0.
    • Wouldn't a more sane approach be to say what the preconditions are for successful use, and let everything else be undefined? The long term costs of defensive programming are bound to pretty high. (I think this is distinct from the kind of validation done at the application framework level: programmers should be required to use code correctly, non-programmers should not).
  2. All exceptions are converted to XmathError, and all diagnostic information is stripped away. How is a user supposed to know that they did wrong?
  3. "profile" is better than "unittest" for profiling, and there are other modules. Why confuse things?
  4. You've carefully eliminated floating point numbers. Why do floating point problems affect an integer calculation?
  5. Why not name the functions "average" and "factorial"? It's a cheap way to make the code much more readable. Examples should be exemplary.

--Tim 10:53, 23 Feb 2005 (PST)


Tim,
Thanks for the input. I agree that we shouldn't hold people to Python's unittest framework. I have plans to add a C++ example of unit testing using the cpptest framework; however, maybe this could be better accomplished by writing a test-framework-independant tutorial for the C++ portion as a continuation to this tutorial. We could use some of the templates you have in ARCSTest...

The other points you make are also valid. True, defensive programming is not the best style -- and it tends to slow down the code. I also agree with what you say about raising a user-defined error. I did this because it made the tutorial a bit cleaner to write, and the user did not need to have any further knowledge of what kind of error to expect. When I was testing the code myself, I actually commented out the user-defined errors (XmathError) to get the standard error messages. Using the user-defined errors are probably not as good in practice, so maybe they should be changed within the tutorial. I don't want to give the impression that this is the defacto-standard DANSE way to raise errors... so maybe that should outweigh simplicity in the tutorial.
--mmckerns 9:39, 28 Feb 2005 (PST)

Comments on specification and testing

I have been working with the computational science community for almost 3 years now and have seen a real awakening to the need to treat software with the same rigor as the science. Your thinking appears to be on or near the leading edge of this trend. Below I offer a few thoughts regarding context and future directions.

The gap between natural language functional requirements and code needs to be filled with a precise specification of the response for all possible input stimuli. In your current process, the unit test driver is implicitly such a specification. Stating the specification explicitly, will further expedite writing concise, efficient code and will facilitate independent testing. For static calculations such as the examples, this specification is just a tabular definition of the response (returned value, stdout, stderr, exceptions, etc.) for all regions of the input space. For higher level compositions (classes, components, programs) the set of input stimuli is actually all possible input sequences. In this case a systematic method is needed to derive the specification from the informal requirements. One such method is Sequence-Based Specification described at [[1] (http://www.cs.utk.edu/sqrl/SpecTutorial.pdf)].

To reduce subjectivity, I suggest using a model-based approach to testing (see [[2] (http://www.cs.utk.edu/sqrl/MBSTtutorialWeb.pdf)]). A usage model defines the probability distribution for the population of expected uses of the software. Test case generation samples that population to ensure that the testing is biased toward expected usage. An important benefit of usage model based testing is support for quantitative statistical inferences about software reliability. From a productivity point of view, a usage model enables automated test case generation and reduces ongoing test suite maintenance. SQRL has develped a toolset called JUMBL ([[3] (http://www.cs.utk.edu/sqrl/esp/jumbl4/index.html)]) to support modeling, analysis, and test automation. Also my colleague Kirk Sayre is currently developing a framework for testing a C++/STL scientific library. A good overview of his approach is at [[4] (http://www.cs.utk.edu/~swain/test_framework_doc/)]. --tom 12:54, 28 Feb 2005 (PST)

Personal tools
Document Uploads/Links