System tests

In contrast to unit tests, which test behavior of a function, system tests verify behavior of complete system or part of the system which is greater than a single function.

The main difference to unit tests are target initialization and test execution. While unit tests define test start as function entry, and test end at function exit, system tests have no such clear definitions of test start and stop, so we have to define them explicitly either in testIDEA or script.

There are three main approaches recommended for system tests:

Scripts only (using winIDEA SDK), testIDEA is not used. This approach can be used for complex system tests, where testIDEA functionality does not match test requirements. This approach is the most flexible one, but we have to implement everything in script, including reporting. Please note, that the script must at least start and stop the target (see the minimal example below), or we get the error: Test didn't end normally. testStatus: stateInitialized`
Mixed testIDEA and script extensions. Tests are run from testIDEA, but most functionality is implemented in script. testIDEA is used for test organization and reporting, while scripts provide flexibility. Section Execute test should be empty, target should be run and stopped by script function Init test specified in section Scripts. testIDEA can be used to:
- Set options defined in section Options.
- Evaluate pre-conditions.
- Create persistent vars.
- Create user stubs. Normal stubs and test points are not used in this configuration.
- Configure analyzer (trace, coverage, profiler).
- Configure HIL.
- At this point script Init Test is called.
- Get analyzer results(trace, coverage, profiler).
- Evaluate expected expressions.
- Save test report.
testIDEA system tests. This approach can be used for simple system tests, where testIDEA functionality for initialization and execution of tests can be used. Section Execute test must be defined. In this configuration testIDEA also provides handling of stubs and test points.
To start test at a specific point, we define section System init, and to execute the test we define section Execute test.

Test scope can be specified in section Meta.

Example of minimalistic script extension function:

    def minimalSystemTest(self):

        self.testCtrl = self.__getTestCaseCtrl()
        self.testCtrl.run()  # Run the target.

        time.sleep(2)  # Test specific stuff.

        self.debug.stop()  # Stop the target

        return None  # None marks success

Example of more complex script extension function:

    
    def systemTestRun(self):
        self.testCtrl = self.__getTestCaseCtrl()

        # 1. Initializa target for system test, for example set SFRs,
        #    set breakpoint where the test should stop.
        self.testCtrl.modify('isDebugTest', '1')
      
        # 2. Use testCtrl to run target.
        self.testCtrl.run()

        # 3. Do test specific stuff, for example wait for stop condition
        while True:
            val = self.debug.evaluate(ic.IConnectDebug.fRealTime, 'iCounter')
            py_iCounter = val.getInt()
            print('iCounter =', py_iCounter) # print for debugging
            if py_iCounter > 100:
                break

        # 4. Stop the target explicitly
        self.debug.stop()

        # 5. Make value available in the report.
        self._isys_initFuncInfo = 'iCounter = ' + str(py_iCounter)
    
        # 6. Verify results
        val = self.debug.evaluate(ic.IConnectDebug.fRealTime, 'g_mode')
        py_g_mode = val.getInt()
        if py_g_mode > 200:
            return 'Test failed! g_mode = ' + str(py_g_mode)

        return None  # None marks success