Continuous Integration with winIDEA and Jenkins

Overview
Create python script for test execution
Install Jenkins
Run headless tests as a service
Run tests with normal user account
winIDEA coverage in Jenkins

Overview

As the name implies, continuous integration (CI) requires running of tests repeatedly during the development cycle. This way we find problems soon after they appear, which usually makes fixing easier. Since repetitive tasks are better left to machines, server applications specialized for running build/test cycles were implemented. One of the most popular CI servers is Jenkins. To simplify CI process, winIDEA SDK enables running tests and displaying tests results with Jenkins:

testIDEA can create Python scripts, which can be run directly by Jenkins.
The scripts can produce result in JUnit XML format, which is understood by Jenkins.
Coverage information can be exported in format understood by Jenkins.
Test cases can produce coverage document in HTML format, which can be linked directly from Jenkins. This document contains all coverage information recoreded by winIDEA, which can be more detailed than existing Jenkins plug-ins can show.

Create Python script for test execution

To generate Python script for running tests headless from command line or Jenkins, use menu command iTools | Generate Test Script ...). The dialog opens, which contains the following configuration items related to Jenkins:

Path to IConnectJNIx64.DLL should be set if Jenkins will run tests as a service. In this case ConnectionMgr::connectMRU() call in Python script can not find and start the last winIDEA started with normal user account.
The other solution to this problem includes setting of an environment variable to point to winIDEA installation directory, for example:
```
              SET ICONNECT_PATH="C:\winIDEA\2012\iConnect.dll"
            
```
Saving of report in JUnit format. This format is recognized by Jenkins server, which can then include testIDEA test results in its reports.
Exporting of one analyzer file in Cobertura format (download files are mapped to packages in this export). Cobertura is coverage tool for Java, and there exists plug-in for Jenkins, which displays results in graphical form.
The analyzer file is winIDEA file containing coverage, and it can be created either by test case, test group or extension script. It is recommended to use analyzer file which contains coverage of all code under test.

Install Jenkins

Jenkins home page provides download link and instructions for installation. If you're planning to show coverage results in Jenkins, install also Cobertura plug-in.

Run headless tests as a service

This section describes configuration of Jenkins to run testIDEA tests. Note, that test script can be generated with testIDEA, but it is also possible to write the script manually.

Important: Make sure you have specified path to iConnect.dll when generating script in testIDEA, or set environment variable.

Create new project Enter project name and select Freestyle project, as shown in the image below.
Configure the project

Name of Test report XMLs (JUnit file) must be the same as name of test report configured in testIDEA, but with extension .junit.xml.
Run the project Click Build Now in the menu, and wait for execution to finish. To observe progress, open console by double clicking the flashing icon next to build in the Build History box.

Click the build number to get menu with build details, including console output and test result.

Run tests with normal user account

For debugging purposes it may be useful to run tests with account of logged user, not with Jenkins's account. In this case winIDEA window is visible. The test script uses preferences of the currently logged user for starting winIDEA, and winIDEA is not started as a service.

Enable Java Web Start Open Top level menu | Manage Jenkins | Configure Global Security. Then select Enable security and Random.
Create executor, which uses Java Web Start Open Top level menu | Manage Jenkins | Manage Nodes | New Node

Click OK to get he next configuration screen.
Select agent in project configuration
Start executor agent In main dashboard, click node jwsGui in Build Executor Status box, to get the screen shown below:

Connect the agent, then run the project.

Code coverage in Jenkins

To view winIDEA/testIDEA code coverage measurements in Jenkins, you can add link to HTML report created by winIDEA, or use Cobertura plug-in in Jenkins. The first approach provides more information, while the second approach provides quick overview of trends in graphical way. It possible to use both approaches in the same project.
If analyzer file from test case is used, it is very important to set value of Measure all functions in section Analyzer | Coverage | Statistics. If this check-box is checked, all code in the project is considered as 100% coverage. If it is not checked, only functions explicitly mentioned in the statistics section are considered as coverage goal (100% coverage).

To add a link to winIDEA HTML report, open project configuration, and add post-build action Publish HTML reports.

The information shown in this report depends on coverage export configuration. See section Analyzer | Coverage, group Export configuration in testIDEA.
To see coverage trends, create test script so that it exports analyzer (trd) file in Cobertura format. Then add post-build action Publish Cobertura Coverage Report to project configuration in Jenkins.