What's all this Jenkins stuff anyhow? (Part 3)

In part 1 of this blog, we took a quick look at the key elements of a Continuous Integration (CI) platform and reviewed some of the software required to make this a useful tool for the embedded developer. Specifically we focussed our attention on Jenkins, but much of what we discussed is relevant to Hudson, Bamboo and many other CI platforms that are available on the market.

Part 2 then went into a little more detail, showing what the GUI of Jenkins, a popular CI, looks like and how it is configured to set up a task that regularly builds code from a repository, such as git. The result of this step was that we could be sure that our application could be built, but we were left in the dark as to whether the code built actually functioned on our target microcontroller. This is the point from which we will continue...

In BugHunter we have already covered how iSYSTEM's testIDEA software can be used to undertake unit testing on a function in code written in C. Here we have developed a simple application that is designed to allow us to focus purely on testing a single function. The project is hosted on BitBucket if you would like to examine it. The leap from manually testing source code to regular automated testing is not so great and will form the basis for this third and final part in this series of posts…


Figure 1 - testIDEA test specification for testing checkTemperature()

So, we have our project, we are using winIDEA as our development IDE together with an iTAG.2K debugger from iSYSTEM. The code has been developed and debugged and the sources, along with the winIDEA workspace, are being regularly submitted to the repository (in our case, git using the BitBucket cloud service) and our Jenkins CI can download and successfully build the latest versions. The next step is to integrate testing with our project. As we see above (Figure 1), testIDEA has been used to generate a series of tests for the code in checkTemperature(). This function expects three parameters:

Current measured temperature
The maximum allowable temperature
The minimum allowable temperature
The function then returns one of three values that depend on where the measured temperature lies in relation to the minimum and maximum values (below, between or above).

In Figure 1 (click on the image to see a larger version) we can see on the left-hand-side that there are, in total, 8 tests defined. On the right-hand-side, we can see the parameters for the first test in the series. The parameters are input in the same order as they are expected by the checkTemperature() function. In this case we have:

Current temperature = -60
Maximum = +100
Minimum = +10
In this case, we expect -60 to be considered as "below the minimum". The checkTemperature() function is designed to return the value "30" in this case. This return value is also entered under the category "Expected" in testIDEA (Figure 2)

Figure 2 - testIDEA - expected return value for test with -60/+100/+10 parameters

In total there are eight such tests defined. The tests are then stored as an "iyaml" file together with the other code in the project and entered into the repository. The "iyaml" file is referred to as the "Test Specification". This ensures that the tests live together with the code, allowing them to be executed by other embedded developers when they make changes.

An important note at this point in the proceedings is that testIDEA depends on the winIDEA workspace to work. This is because the tests are being executed on the target hardware, not some simulated environment somewhere. The method used is known as Original Binary Code (OBC) testing. Without a correctly configured winIDEA workspace, testIDEA would not know upon what target to execute the tests, nor which debugging tool is connected to the target. The nice element of this approach is that, if some time in the future an alternate microcontroller is used, the exact same Test Specification can be used to test the code functionality on a new device.

Although we now have a Test Specification, Jenkins cannot automatically work with this file type. Instead, we need to export this file as a Python script, something that is supported within testIDEA if the appropriate license is available on the debugging hardware. Within testIDEA, we simply select iTools -> Generate test script... and, via the dialogue, we can generate a Python script (Figure 3). Also important: set the Report section to generate a "JUnit" output file. Jenkins will use this later to display test result trends.

Figure 3 - A Python script for automated testing is exported from testIDEA

Before committing the script to the repository, it makes sense to execute the script once from the development PC to ensure it is working correctly. Sometimes there are some elements that require fine tuning once the repository has been downloaded onto your Jenkins platform. Simply open a command line and use Python to execute the script. A functioning version of Python is already included with the winIDEA installation, so there is no need to worry about installing it or making sure you have the correct version. In our case we executed the script as follows:

C:\Users\Stuart\sw-dev\testIDEA-demo>c:\iSYSTEM\winIDEA-9-256\Python33\python.exe testIDEA-demo.py
Figure 4 - Executing the test script on the development PC

In Figure 4 we see what the result should look like when the script is executed directly from the command line. All eight tests have been executed and passed. We can now, with confidence, commit this testing script to our repository.

On the Jenkins CI system, we can now add this testing step to the Build section of the task. Previously we used this section to "clean" and compile our application sources using the make utility. Now we can simply add the same command-line call to execute our Python test script as we entered by hand on our development machine (Figure 5).

Figure 5 - Adding the testing step using a Python script to the Jenkins task

The final step is to ensure that Jenkins can analyse and display the progress of testing after every build cycle. Provided the "JUnit Plugin" has been installed, you can tell Jenkins to review the results of testing as shown in Figure 6.

Figure 6 - Adding JUnit report evaluation to the Jenkins task

Now, every time the Jenkins task is executed, your latest code base is downloaded from your repository, the code is built, programmed into a microcontroller and then the tests are executed on that microcontroller. The result is a testing process that ensures the source code, compiler and microcontroller combination do actually work together. By installing various compiler versions and connecting various target MCUs to your Jenkins machine (Figure 8), it is possible to test the code against many varying combinations, something that Jenkins can handle via the Matrix build configuration.

Figure 7 - JUnit Plugin displaying Test Result Trends - time for a code review!
Figure 8 - Multiple test-targets running from single Jenkins platform

If you are still wondering how best to automate your testing, whether it be unit testing, integration or system testing, Jenkins is an excellent platform to use. Stable, highly configurable and able to integrate with a wide range of off-the-shelf tools, as well as self-written software, it ensures that "testing early and often" can be undertaken without too much pain. Yes, there is a learning curve and it does need the buy-in of the team and management to support the changes in process required, but overall the effort is definitely worth it.

We hope you have developed an interest for Jenkins and Continuous Integration and, if you set up your own environment, we'd love to hear about your experiences! In the mean time, Happy BugHunting!