Integrating a custom test runner with TeamCity
JetBrains’ TeamCity build server provides a wealth of information about the tests that are run as part of your build. If you use one of the test runners that they support out of the box, you automatically get real-time individual test results, detailed timing reports, and historical performance metrics.
The StoryTeller test runner is a console application that will return an error code if any regression tests fail. It also generates an HTML report which contains the details for each test. Our initial attempt at integrating it into our build allowed us to fail the build if any test failed, but we didn’t get much feedback about why it failed. We would have to go searching through the build log messages or HTML report files for details.
I created a new StoryTeller ITestListener (TeamCityTestListener) which is used by the StoryTeller runner to log test results. It writes the test details to the console using TeamCity’s Service Message API (which I mentioned previously). TeamCity picks up these messages, and creates the nice individual test reports, complete with timing and history:
Now we know exactly which tests failed, but we don’t know which assertions in the tests failed. As I mentioned, the complete details of the test execution are recorded in an HTML report. It’d be nice if we could view individual test reports right within TeamCity. Turns out, that is pretty easy to do as well. I followed the documentation for Including Third-Party Reports in the Build Results, and added the following line to our
<report-tab title="StoryTeller" basePath="st-results" startPage="st-results.htm" />
We now have a nice new StoryTeller tab next to the other build result tabs:
We configured the build to store all of the HTML reports as artifacts, and point our custom tab to the test results index HTML file generated by StoryTeller. Clicking the StoryTeller tab displays that index file with a color coded overview of all of the tests. Clicking a test brings up the report for that test so that I can see exactly which steps failed.