Running jQuery QUnit tests under Continuous Integration
UPDATE: The code in this post is out of date. Read it for the explanation, but if you want to implement it, go grab NQUnit via nuget.
Setup
This post assumes you are already writing unit tests for your JavaScript code. If not, check out Chad’s post on Getting Started with jQuery QUnit. We use jQuery and QUnit at work, so my code examples are geared toward those frameworks. However, the approach should be very easy to adapt to your JavaScript framework of choice.
Overview
A good continuous integration server will let you run your automated tests, fail the build if a test fails, and publish a report of the test results. For all of that to work, your CI tool needs to be able to run the tests from a command-line and harvest the output in a form that it understands. One problem is that JavaScript testing frameworks like QUnit require you to open an HTML page in a browser to run and view the test results. You can certainly launch an HTML page in a browser from a command line (“start mypage.htm”), but you won’t be able to feed the results back to the CI server. We can get around this by using a tool like WatiN to control the browser from NUnit (or some other test framework which is supported by your CI server). WatiN allows you to spawn an instance of Internet Explorer, navigate to a URL, inspect the contents of the rendered DOM, and shutdown the browser, all from within an NUnit test.
A simple solution
Our first approach was to modify the QUnit test runner script so that it would create an element named TESTRESULTS that held the number of failed tests. We could then run the HTML page containing our tests and use WatiN to verify that TESTRESULTS contained a 0. An entire page of QUnit tests would be reported by NUnit as a single test. Either all of the QUnit tests passed making the single NUnit test pass, or any QUnit test failed and the NUnit test failed. You can see an example of this approach in my comment on Chad’s post referenced earlier. There are two problems with this approach: your total test count is inaccurate (you could have hundreds of QUnit tests which only show up as a single test in the CI test report), and more importantly, it is not immediately obvious which test failed, since it could have been any one of the many QUnit tests within a page.
Our current approach
A better approach would be to have a single NUnit test for each QUnit test. However, manually writing an NUnit test for each QUnit test sounds like a nightmare. What we need is a way to generate test cases dynamically. MbUnit supports this natively with its Factory attribute. We can get the same behavior in NUnit using the IterativeTest add-in (get the source and compile it against your version of NUnit). It allows you to specify a factory method that supplies a series of values to a single test method. When NUnit loads the class with this test method, it will create a separate test case for each value passed to the test method (if your factory method returns 3 values, you will have 3 test cases).
The full code to our test fixture is at the bottom of this post. The method GetQUnitTestResults takes an HTML page name as input and returns a series of QUnitTest instances. Each QUnitTest instance contains the details of an individual QUnit test run: the filename, the test name, the result, and any related failure message. The method RunQUnitTests is the actual factory method used by the IterativeTest add-in, and allows me to easily add new QUnit HTML pages without having to create new test methods. The gory details of parsing the DOM for the QUnit results are in grabTestResultsFromWebPage. Its not my proudest piece of code, but it gets the job done (for now). This is likely the only method you would need to change if you were using a JavaScript test framework other than QUnit.
Using this approach, we now get an accurate reflection of our test count, as each QUnit test is counted with every other NUnit test. Even better, when a test fails, we get detailed information about exactly which test failed and why. This is the output of a run that has a failing test:
------ Test started: Assembly: DovetailCRM.IntegrationTesting.dll ------ IterativeTest addin loaded. TestCase 'DovetailCRM.IntegrationTesting.Web.JavaScript.JavaScriptTester.QUnit([tageditortester.htm] Post tag change to server module: should post the tag Name and item Id)' failed: the item that is tagged expected: 42 actual: 0 Expected: collection containing "pass" But was: < "fail" > D:codebluesourceDovetailCRM.IntegrationTestingWebJavaScriptJavaScriptTesters.cs(128,0): at DovetailCRM.IntegrationTesting.Web.JavaScript.QUnitTestHelpers.ShouldPass(QUnitTest theTest) D:codebluesourceDovetailCRM.IntegrationTestingWebJavaScriptJavaScriptTesters.cs(29,0): at DovetailCRM.IntegrationTesting.Web.JavaScript.JavaScriptTester.QUnit(Object current) 100 passed, 1 failed, 0 skipped, took 16.48 seconds.
Notes
- NUnit add-ins must be compiled against the version of NUnit you are using. Make sure you have the same version of NUnit on your developer desktops and the CI server. We standardized on NUnit 2.4.7 simply because that was the latest version supported by TestDriven.NET.
- The IterativeTest add-in must be in the addins sub-folder of the folder that contains your nunit-console.exe. If you want to run the tests via TestDriven.NET, you’ll also want to put a copy of the add-in in
TestDriven.NET 2.0NUnit2.4addins - While there is a way to get the TeamCity custom NUnit runner to load add-ins (using /addin:), I ran into trouble getting it all to work with WatiN as well. I discovered a JetBrains add-in for NUnit that allows nunit-console.exe to report progress to TeamCity in the same way as their custom runner. While it isn’t supposed to be available until TeamCity 4.0, I found the necessary files in this patch (see the Attachments section) which enables NUnit 2.4.7 support to TeamCity 3.1.
The Code
1: [TestFixture]
2: public class JavaScriptTester
3: {
4: private IE _ie;
5:
6: [TestFixtureSetUp]
7: public void TestFixtureSetUp()
8: {
9: _ie = TestBrowser.GetInternetExplorer();
10: }
11:
12: [IterativeTest("RunQUnitTests")]
13: public void QUnit(object current)
14: {
15: ((QUnitTest)current).ShouldPass();
16: }
17:
18: public IEnumerable RunQUnitTests()
19: {
20: return new[]
21: {
22: "ContextMenuTester.htm",
23: "FilterControlTester.htm",
24: "RepeaterControlTester.htm",
25: "FinderTester.htm",
26: "ConsoleScriptTester.htm",
27: "tageditortester.htm",
28: "CrudFormTester.htm"
29: }.SelectMany(page => GetQUnitTestResults(page));
30: }
31:
32:
33: public IEnumerable<QUnitTest> GetQUnitTestResults(string testPage)
34: {
35: TestFixtureSetUp();
36: _ie.GoTo(string.Format("http://localhost/Content/scripts/tests/{0}", testPage));
37: _ie.WaitForComplete(5);
38:
39: return grabTestResultsFromWebPage(testPage);
40: }
41:
42: public IEnumerable<QUnitTest> grabTestResultsFromWebPage(string testPage)
43: {
44: // BEWARE: This logic is tightly coupled to the structure of the HTML generated by the QUnit testrunner
45: // Also, this could probably be greatly simplified with a couple well-crafted xpath expressions
46: var testOL = _ie.Elements.Filter(Find.ById("tests"))[0] as ElementsContainer;
47: if (testOL == null) yield break;
48: var documentRoot = XDocument.Load(new StringReader(makeXHtml(testOL.OuterHtml))).Root;
49: if (documentRoot == null) yield break;
50:
51: foreach (var listItem in documentRoot.Elements())
52: {
53: var testName = listItem.Elements().First( x => x.Name.Is("strong")).Value;
54: var resultClass = listItem.Attributes().First(x => x.Name.Is("class")).Value;
55: var failedAssert = String.Empty;
56: if (resultClass == "fail")
57: {
58: var specificAssertFailureListItem = listItem.Elements()
59: .First(x => x.Name.Is("ol")).Elements()
60: .First(x => x.Name.Is("li") && x.Attributes().First(a=> a.Name.Is("class")).Value == "fail");
61: if (specificAssertFailureListItem != null)
62: {
63: failedAssert = specificAssertFailureListItem.Value;
64: }
65: }
66:
67: yield return new QUnitTest
68: {
69: FileName = testPage,
70: TestName = removeAssertCounts(testName),
71: Result = resultClass, Message = failedAssert
72: };
73: }
74:
75: }
76:
77: private static string makeXHtml(string html)
78: {
79: return html.Replace("class=pass", "class="pass"")
80: .Replace("class=fail", "class="fail"")
81: .Replace("id=tests", "id="tests"");
82: }
83:
84:
85: private static string removeAssertCounts(string fullTagText)
86: {
87: if (fullTagText == null) return String.Empty;
88: int parenPosition = fullTagText.IndexOf('(');
89: if (parenPosition > 0)
90: {
91: return fullTagText.Substring(0, parenPosition - 1);
92: }
93: return fullTagText;
94: }
95: }
96:
97: public class QUnitTest
98: {
99: public string FileName { get; set; }
100: public string TestName { get; set; }
101: public string Result { get; set; }
102: public string Message { get; set; }
103:
104: public override string ToString()
105: {
106: return string.Format("[{0}] {1}", FileName, TestName);
107: }
108: }
109:
110: public static class QUnitTestHelpers
111: {
112: public static void ShouldPass(this QUnitTest theTest)
113: {
114: Assert.That(theTest.Result.Split(' '), Has.Member("pass"), theTest.Message);
115: }
116:
117: public static bool Is(this XName xname, string name)
118: {
119: return xname.ToString().Equals(name, StringComparison.OrdinalIgnoreCase);
120: }
121: }</p>
</p> </p>
It makes use of our TestBrowser class, which is just a simple wrapper around WatiN’s IE object.
1: public static class TestBrowser
2: {
3: public static readonly object LOCK_ROOT = new object();
4: public static IE _browser;
5:
6: public static IE GetInternetExplorer()
7: {
8: if( _browser == null )
9: {
10: lock(LOCK_ROOT)
11: {
12: if( _browser == null )
13: {
14: IE.Settings.AutoMoveMousePointerToTopLeft = false;
15:
16: _browser = (IE)BrowserFactory.Create(BrowserType.InternetExplorer);
17: _browser.ShowWindow(NativeMethods.WindowShowStyle.Hide);
18: }
19: }
20: }
21:
22: return _browser;
23: }
24: }</p>