Pimp my report with Docker


We are testing our Web portal with fully automated end-2-end tests against various combinations of Browsers and platforms. Today I had a task assigned which required to improve the quality of our Sauce Labs test failure reports on Bamboo. We are using the Robot framework to execute end-to-end tests and those tests are executed against various browsers on different platforms in Sauce Labs. Unfortunately the Robot framework doesn’t offer a direct way of manipulating the tests result output so I had to use some scripting foo to make it happen. To make things easier and platform independent I executed my scripts in a Docker container.

You can find more Docker related posts that I have written in this index.

The Goal

Bamboo by default shows a list of all failed tests on its dashboard. Each failed test is listed by name and some other attributes. The problem with our SauceLabs project was that this list did not indicate on which platform and for which browser the test was targeted. To find out more about this one had to drill down deeper in the artifacts (detail reports per platform/browser) of the test run which was painful.

The easiest solution seemed to be to change the name of the test and add a prefix which contains the platform and browser names.

The structure of a Robot generated test report (called output.xml) looks like this

<?xml version="1.0" encoding="UTF-8"?>
<robot generated="20160830 14:01:27.533" generator="Robot 3.0 (Python 2.7.12 on linux2)">
  <suite source="[some test directory]" id="s1" name="Tests">
    <test id="s1-t1" name="My First Test">

Thus I had to come up with a solution to find the value of the name attribute of all <test> tags in the XML document and concatenate this value with the prefix consisting of the browser and platform names. For this purpose I used SED and a bit of Regex foo. I’m again and again baffled how powerful Regex is yet how easily I tend to forget everything about it’s syntax… The command I came up looks like this

sed -r -i 's/(\<test .* name=")([^"]*)"/\1'"$prefix"': \2"/' $file

Here the variable $prefix contains the said combination of browser and platform name and $file is the test output file who’s test names need to be changed.

Now I have the detail results of all platform/browser combinations in respective sub-folders of my reports directory. The names of the subfolders are a combination of “[Platform]-[Browser]-[Browser Version]-[Test Run ID]-[Data]-[Time]“. Thus I need to loop over all output.xml files located in those sub-folders and modify them. The $prefix I can deduce from the folder name. The logic looks like this

# Where are all my reports?

# Transform the names of the tests for better platform identification
for file in $FILES
  echo "source: ${file} --> target: ${file}"
  # get the name of the parent folder
  parentname="$(basename "$(dirname "$file")")"
  # trim the trailing date/time part from the name
  prefix=$(echo ${parentname:0:${#parentname}-22} | sed s/\_/\ /)
  sed -r -i 's/(\<test .* name=")([^"]*)"/\1'"$prefix"': \2"/' $file

Once I have modified all reports I can use the Rebot tool of the Robot framework to generate a summary report. It looks like this

mkdir -p ${REPORT_DIR}/summary
rebot -N Summary-Report -d "${REPORT_DIR}/summary" -o output.xml --log log.html --report report.html --xunit xunit.xml "${REPORT_DIR}/**/output.xml"

That’s all.

Docker enters the picture

Now why do I even want to consider using a container to run the above logic on Bamboo? First of all using a container allows me to test and run this logic on my laptop in exactly the same way as it will run on the build server Bamboo. Having a container I also do not depend on what is installed on the Bamboo build agent. The sed tool and bash are easy requirements but not so much rebot which is a Python application and needs the Robot framework library to be installed. That said, let’s define a Dockerfile

FROM python:2.7
RUN pip install robotframework
RUN mkdir -p /app
COPY entrypoint.sh /app/
ENTRYPOINT /bin/bash -c "./entrypoint.sh"

The script entrypoint.sh contains the logic specified above. I can now build the Docker image

docker build -t report-transformer .

and then run it

docker run -it -v $(pwd):/app report-transformer

Please note the volume mapping I’m using to mount the directory containing the test output into the container.

Finally I can add a script tasks in Bamboo to the Sauce Labs project to build and execute this container after all tests have run and before the final parsing of the test results by Bamboo.

This is a good sample showing that containers are not only useful to run our applications and application services but they are equally well suited to run quick one-time tasks.

If you liked this post then you can find more Docker related posts I have written in this index.

About Gabriel Schenker

Gabriel N. Schenker started his career as a physicist. Following his passion and interest in stars and the universe he chose to write his Ph.D. thesis in astrophysics. Soon after this he dedicated all his time to his second passion, writing and architecting software. Gabriel has since been working for over 25 years as a consultant, software architect, trainer, and mentor mainly on the .NET platform. He is currently working as senior software architect at Alien Vault in Austin, Texas. Gabriel is passionate about software development and tries to make the life of developers easier by providing guidelines and frameworks to reduce friction in the software development process. Gabriel is married and father of four children and during his spare time likes hiking in the mountains, cooking and reading.
This entry was posted in containers, docker and tagged , , , . Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Andrew Siemer

    Love you mang! Excellent use of augmenting your delivery pipeline with more microservices. Very portable/nimble.

    • gabrielschenker

      Thanks for the encouragement!

  • I am very pleased to know about the such information !!

  • Stephanie Reinhardt

    We at Gateway TechnoLabs, have dedicated testing experts, who have a rich experience of delivering error free apps or software. You can try for http://www.gatewaytechnolabs.com/mobile-application-testing this link and use with different browser. Again I am say very thanks you to submit to such information.