Introduction

We are testing our Web portal with fully automated end-2-end tests against various combinations of Browsers and platforms. Today I had a task assigned which required to improve the quality of our Sauce Labs test failure reports on Bamboo. We are using the Robot framework to execute end-to-end tests and those tests are executed against various browsers on different platforms in Sauce Labs. Unfortunately the Robot framework doesn’t offer a direct way of manipulating the tests result output so I had to use some scripting foo to make it happen. To make things easier and platform independent I executed my scripts in a Docker container.

You can find more Docker related posts that I have written in this index.

The Goal

Bamboo by default shows a list of all failed tests on its dashboard. Each failed test is listed by name and some other attributes. The problem with our SauceLabs project was that this list did not indicate on which platform and for which browser the test was targeted. To find out more about this one had to drill down deeper in the artifacts (detail reports per platform/browser) of the test run which was painful.

The easiest solution seemed to be to change the name of the test and add a prefix which contains the platform and browser names.

The structure of a Robot generated test report (called output.xml) looks like this

Thus I had to come up with a solution to find the value of the name attribute of all <test> tags in the XML document and concatenate this value with the prefix consisting of the browser and platform names. For this purpose I used SED and a bit of Regex foo. I’m again and again baffled how powerful Regex is yet how easily I tend to forget everything about it’s syntax… The command I came up looks like this

Here the variable $prefix contains the said combination of browser and platform name and $file is the test output file who’s test names need to be changed.

Now I have the detail results of all platform/browser combinations in respective sub-folders of my reports directory. The names of the subfolders are a combination of “[Platform]-[Browser]-[Browser Version]-[Test Run ID]-[Data]-[Time]”. Thus I need to loop over all output.xml files located in those sub-folders and modify them. The $prefix I can deduce from the folder name. The logic looks like this

Once I have modified all reports I can use the Rebot tool of the Robot framework to generate a summary report. It looks like this

That’s all.

Docker enters the picture

Now why do I even want to consider using a container to run the above logic on Bamboo? First of all using a container allows me to test and run this logic on my laptop in exactly the same way as it will run on the build server Bamboo. Having a container I also do not depend on what is installed on the Bamboo build agent. The sed tool and bash are easy requirements but not so much rebot which is a Python application and needs the Robot framework library to be installed. That said, let’s define a Dockerfile

The script entrypoint.sh contains the logic specified above. I can now build the Docker image

and then run it

Please note the volume mapping I’m using to mount the directory containing the test output into the container.

Finally I can add a script tasks in Bamboo to the Sauce Labs project to build and execute this container after all tests have run and before the final parsing of the test results by Bamboo.

This is a good sample showing that containers are not only useful to run our applications and application services but they are equally well suited to run quick one-time tasks.

If you liked this post then you can find more Docker related posts I have written in this index.