Reference

This document contains some reference information about MagicTest.

The Test API

You control the behavior of MagicTest using the annotations and classes provided in the package org.magictest.client.

MagicTest provides the three annotations @Test, @Trace, and @Capture. If you annotate a method with one of these annotations, it is considered as test method by MagicTest. Additionally, you can also use the org.testng.annotations.@Test annotation provided by TestNG.

The @Trace Annotation

The @Trace annotation declares that calls to the method under test should be automatically traced and the output collected be shown in the test report.

This annotation is designed for unit tests where the functionality of single method or class must be tested.

MagicTest uses standard naming conventions and reasonable defaults to provide functionality out of the box without explicit configuration.

  • Which method calls should be traced is determined by standard naming conventions, but can be overridden using traceMethod and traceClass.

  • Parameters and return value are displayed as their toString() functions format them. It can however be that you want to see other details for evaluating your test. To support this, you can define formatters which are then called for the passed objects.

  • The auto-trace functionality is very useful for functions which actually return a result. However often the result is not returned, but stored within the called object. To support this use case, you can use the result argument of the @Trace-annotation.

If the default behavior has to be changed, this can be done by setting attributes of the @Trace annotation. The annotation has the following elements:

  • traceClass and traceMethod: specify the methods under test for tracing

  • title: specify title to use for this test

  • decription: specify description to use for all test steps

  • parameters and result: specify which arguments should be reported as parameters and result

  • parameterNames: provide names to display for parameters

  • formats: specify formatting of arguments

You can also apply this annotation on class level to set common values for all test methods easily which then can be overridden on the method level.

Have a look the Javadoc API documentation for a reference of all elements.

Elements traceClass and traceMethod

MagicTest uses the following standard naming conventions to determine the methods under test:

  • Test class: The string "Test" is removed from the start or end of the test class name, i.e. "StringToolsTest" or "TestStringTools" becomes "StringTools".

  • Test method: The string "test" at the start or "Test" at the end of the test method name is removed, i.e. "testConcat" or "concatTest" become "concat".

So for the example name used above, MagicTest will trace method calls to StringTools.concat.

You can specify the test method the following way:

  • @Trace(traceClass = "class") if the test class is in the same package

  • @Trace(traceClass = "package.class")

  • @Trace(traceMethod = "method") if the test method is in the same class

  • @Trace(traceMethod = "package.class.method")

It is also possible to use regular expressions to trace several methods or even classes at once:

  • @Trace(traceMethod = "/set.+/") will trace all setters

You can even trace methods in several classes with such settings.

Note that also a method name can be ambiguous when overloaded methods exist without using regular expressions. In this case all methods with the specified name are traced.

Elements parameters and result

The element "parameters" and "result" specify which arguments should be reported as parameters and result in the test report.

Per default, all parameters of the call to the method under test are reported as "parameters" and the return value as "result".

Where this behavior works well for pure functional methods, it can be tweaked to fit different needs.

If we have a look at a call to an instance method of a stateful object, typically the state of the object can be consired both as part of the parameters and the result. In this case, you would configure the @Trace annotations like that:

@Trace(
    parameters = Trace.ALL_PARAMS|Trace.THIS,
    result = Trace.RESULT|Trace.THIS )

In the case that a parameter value is modified by the method and has therefore to be considered as part of the result, you would use this configuration:

@Trace(result = Trace.RESULT|Trace.PARAM0 )

Element formats and annotation @Formatter

The "formats" element of the @Trace annotation and the @Formatter annotation help you to specify how parameters and result are formatted.

Normally all values are formatted using the toString() method with the following exceptions:

  • char: is formatted as Java character literal, e.g. '\t'

  • java.lang.String: is formatted as Java string literal, e.g. "line 1\nline2"

  • java.util.Date: is formatted using the format "yyyy-mm-ddThh:mm:ss.SSSZ"

Using the @Formatter annotation, you can define a formatter method for a specific type which is then used instead.

The formatter must be a static method taking one parameter of the specified object and returning a String.

@Formatter public static String formatInt(int n) { ... }


Note that the name of the method does not matter.

This allows to define a single formatter used for all values of a type for all test methods within a class. If you need to control the formatting in more detail, you can use the @Format annotation which can be used in the formats element of the @Trace annotation.

Each @Format annotation defines in the "argument" element for which parameter or result values it should be applied. The formatting which should be applied is defined by setting one of the elements "formatter", "printFormat", or "messageFormat".

You can specify the formatter the following way:

  • @Format(formatter = "method") if the formatter method is in the same class

  • @Format(formatter = "package.class.method")

Besides defining specific formatters, the @Format annotation also supports short cuts for the commonly used methods for string and message formatting.

So if you want to display an integer value in hexadecimal format, you can simply write

  • @Format(printFormat = "%x")

See the following complex example showing the power of these features:

@Trace(traceMethod = "add", formats = {
    @Format(argument = Trace.PARAM0, printFormat = "(%d)"),
    @Format(argument = Trace.PARAM1, messageFormat = "[{0,number,#.##}]"),
    @Format(argument = Trace.RESULT, formatter = "formatHex")
})
public static void testAdd() {
}

After it is clear which data will be reported, it will be determined how they should be formatted:

  • First we look for a corresponding @Format entry in the formats element on method level. If there is none, we look on class level. If there is none whether there is a marked formatter for the given type. If there is none, the type's default formatting is used.

  • Note that you can also define a @Format entry for the description or errors.

Output Type

Per default, the output created by the tests are rendered as normal flow text. It may however be that this is not appropriate for all information. Some information would better be rendered as preformatted text or even as HTML.

MagicTest supports four different output types:

Type

Description

VALUE

With this default output type, the value is rendered as flow text. Non-printable characters (including newlines) are escaped as in Java. The text is considered to be atomic, so if two values are compared and not equal, both values are shown next to each other.

TEXT

With output type text, line endings are preserved, i.e. the text is wrapped at newline characers. If texts are compared, the difference is determined line by line as known from traditional diff tools.

PRE

Output type pre is similar to text. The only difference is that a monospaced font is used for display which allows to see the exact spacing.

HTML

The collected output is interpreted as HTML code which is then included verbatim in the generated HTML report. The output is not validated for HTML conformance, but it must be well-formed XML. Diffing of HTML output is done on the HTML code level and thus showing all markup tags.

 

Per default, all output is considered to have a type value. If you want to use a different output type, there are two ways to achieve this.

First, you can use formatters and specify a different output type in the formatter definition:

@Formatter(outputType = OutputType.HTML)
public static String formatResultSet(ResultSet rs) { ... }

Second, you specify the output type for each parameter explicitly:

@Format(argument = Trace.PARAM0, outputType = OutputType.HTML)

The @Capture annotation

The @Capture annotation specifies that output is automatically captured and written to the test report. This annotation is designed for use in characterization tests.

A characterization test describes the actual behavior of an existing piece of software. It can be used to provide a safety net for extending or refactoring code that does not have adequate unit tests. Obviously the visual approach featured by MagicTest is very well suited for this.

You can use System.out or any other supported output source to create the actual test output. The following sources are supported:

Type

Description

OUT

All output written to System.out is collected.

ERR

All output written to System.err is collected.

OUT_ERR

All output written to both System.out and System.err is collected.

LOGBACK

All output written to Logback loggers is collected. For collection, the output must be arrive at the root logger which may not be the case if you use loggers with additivity set to false.

 

Different output types are also supported for capture tests. The default output type for capture tests is TEXT.

Note that exceptions are not automatically caught and traced, so it behaves more like a traditional assertion-based test.

See the examples in the package org.magictest.examples.capture in the examples ZIP file.

The @Test annotation

For completeness, MagicTest has also its own @Test annotation.

In test methods annotated with @Test you code your test traditionally using assertions.

Note that you can also use the org.testng.annotations.@Test annotation instead.

These tests are not listed in the HTML report.

See the examples in the package org.magictest.examples.assertion in the examples ZIP file.

The Report class

The trace functionality generates output automatically and provides reasonably defaults.

However, sometimes you would like to tweak some details. For such manual interventions, you can use the methods provided in the class Report. It also allows to add complete steps to the report manually.

The class Report offers static methods to create test steps.

As you have seen, it is quite easy to get a reasonable test report without any manual intervention. On the other hand, the @Trace annotation only gives you very limited functionality to control the output:

  • with the description element, you can set the description

  • you can turn automatic tracing of output off, by setting the autoTrace element to false

To write the information you like into a test report, you can use the methods by the org.magictest.client.Report class. All methods offered by this class are static, so you can make a static import to conveniently use them.

Have a look at the ReportTest class in the example source code to see possible use cases.

Directory magictest

You should already be familiar with the concept of actual and reference output used by MagicTest's visual approach.

Where as the output data is collected and compared per method, the data of all methods of a test class are stored in the same file. The data is stored in a directory hierarchy identical to the source code, i.e. if the source file is stored in src/mypackage/MyClass.java then the report files will be available under magictest/package.

For the single Java source file MyClass.java, we end up with the following four files:

Name

Description

MyClass.act.xml

This file contains the actual output of the last test run for each method. It is updated each time a test method is run. It also contains volatile information like time and duration of the last test run.

MyClass.ref.xml

This file contains the reference output for each method and is used to determine the success of a test run by comparing the actual output against this reference output. It is updated if a new reference output is saved. This file is an essential part of the test and should therefore be kept under version control like the source code of the test methods.

MyClass.act.html

This file contains a HTML report containing the result of the last test run by comparing step-by-step actual and reference output. It depends on the .act.xml and the .ref.xml file. It is updated each time a test method is run.

MyClass.ref.html

This file contains a HTML report of the reference output for each method. It depends on the .ref.xml file. It is updated if a new reference ouptut is saved.

 

If just a single test method is run in a test class, the information about the other tests methods remain unchanged.

If you run the tests of a whole class, the actual output will be discarded first, so all tests not needed any longer will disappear.

Note that you currently have to delete output files manually when they are no longer needed, because you deleted the corresponding test class.

If you work with Subversion, you can easily ignore the actual files when the svn:ignore property has the following content on all subdirectories of magictest:

*.act.html
*.act.xml

HTML Report

A separate HTML report is created for each test class which contains information about all test methods.

A test consists of a bunch of test steps, there will be as many as you need to test all your stuff.

If run a traced test, each call to the method under test will become a separate test step.

These tests steps are represented as table. A test table has the following columns:

  • step: a step number which is just an increasing integer value starting at 1

  • description: description of test, by default the name of the method under test

  • parameters: parameters passed to the method under test

  • result: result value of the method under test

  • error: message of exception thrown by message under test

  • status: indication whether this test step is considered successful or not

Internals

View of a Bird

This is the bird's view if a traced test is run:

  • Detect test methods by scanning test class for methods annotated with @Trace

  • For each test method determine which method (or methods) have to be traced

  • Instrument calls to methods under test to collect parameter and return values for each call

  • Execute test method

  • During the execution, the instrumented code makes sure that all parameter and return values for each call are collected

  • Compare collected actual output with stored reference output and update HTML report

  • Show HTML report to the user

Instrumentation

To generate the needed output without having to manually code these statements, we instrument the byte code before the test method is executed: Each call to the concat() method is instrumented so that parameters, return value, and thrown exception are automatically traced.

Looking at the byte code level, a call to the method under test will roughly look like this:

try {
   printParameters("s1, s2");
   String result = StaticMethod.concat("s1", "s2");
   printResult(result);
} catch (Throwable t) {
   printError(t);
}

Due to a technical limitation, it is not possible to simply replace the call to concat with the code shown above. Therefore an additional wrapper must be generated to embed this code:

String callConcat(String s1, String s2) {
    try { ... }
}

Note that with this approach, only the test method, but not the method under test must be changed.

Logging

It may sometimes be desirable so see more output than normal from MagicTest to understand what is happening. This can be achieved changing the logging level used internally by MagicTest.

MagicTest uses an internal copy of Logback to implement logging. To tweak the logging, you have the following options:

  • Place a file named logback-magictest.xml in the classpath or the current directory where it will be automatically picked up by MagicTest. You may use all features offered by Logback in this configuration file with the exception that the Logback classes are found with in the package org.magictest.ext. So the default console appender is referenced by org.magictest.ext.ch.qos.logback.core.ConsoleAppender.

  • Use the command line options -loglevel, -loglayout and -logfile:

  • loglevel: valid options are error, warn, info (default), debug, and trace

  • loglayout: use either option 'out' (default) or 'log' or provide a Logback pattern layout string

  • logfile: name of a configuration file to use instead of logback-magictest.xml

 
© Magicwerk.org