Implement CKTester

[Garry Yao]

CKTester is our internal universal testing forge, it should be capable of running tests for different projects with different testing tools easily while remain simplicity, below are some basic ideas of the project based on the previous effort on 'FCKtest'.

Test Cell

A test cell is basically a plain html file reside in individual project's testing source, in which test suites/test cases are written. It'll be executed inside an iframe/popup of the runner page so full page life cycle ( from page load to page destroy ) is guaranteed when performing the test.

Each cell has only key responsibility of reporting the sucess/failure/time of it's execution to the runner, this is done by cross-frame communication with parent runner window.

The cell's testing environment, including testing target, runtime libraries, etc. will be all injected by it's belonging runner, initiatively or passively, the only required effort in the cell is to include the bootstrap file.

<script type="text/javascript" src="{path-to-runner}/bootstrap.js"></script>

It could choose to manually pull in resources provided by the runner also, e.g. A cell want to load an testing target manually, it could have:

<script type="text/javascript" src="../../bootstrap.js"></script>
<script type="text/javascript">
	CKTester.require( '${myProjectRoot}/myTarget.js' );	// inline variable ${myProjectRoot}
</script>

Note that inline variables like above used in the path are not determined by the runner but are defined in testing profile which know well about the testing stage( target ) location.

Profile

A tests profile is a js file contains information about running a bunch of test cells. It a function declaration which return a configuration object, contains:

1. Cell variables : A object contains key/value pair definition of variables which are to be used by individual test cell when communicating with runner.
2. Cells list : A list of key/values as:
   • Key : The test cell html file path, with each directory within the path is resolved as a tag;
   • Value : An array of tags.

It will basically looks like:

	CKTester.profile = function ()
	{
		return
		{
			variables : [ [ 'ckeditorRoot', 'path-to-CKEditor-root'] ],  
			cells : [
				[ 'tt/unit/htmldataprocessor/1' , [] ],	// Implicit tags declaration by file path.
				[ 'ckeditor/3210.html' , [ 'ticket', 'unit', 'htmldataprocessor'] ] // Explicit tag declaration.
			] 
		};
	}

The tag plays an important role as an identifier to denote the following aspects of a test cell:

1. Dependencies of the cell, e.g. if it has tags 'unit', 'ckeditor' and 'domiterator', it denote it's a unit test running by YUITest, the testing stage is CKEditor, and the domiterator plugin is required to run the test.
2. Tagging of the cell, whether the cell should be executed when it's been run in a runner with a specific criteria. E.g. A url criteria as '​http://t/?tt&plugins' will run only the ticket test of plugins ( with 'ticket' and 'plugins' tags).

Runner

Test runner is bridge between the testing tools/framework ( e.g. Selenium ) underneath, and the testing resources( our various test cases ), it's resided in CKTester project.

The runner is responsible for delegating the running of all the tests to the under-laying tools/framework and collect the testing result neutrally. It consist of a UI to interact with and providing report.

The runner is required to be running with a specified profile, from which it will register all the defined tests cells . Upon requesting by a specific testing criteria on running, it should filtering down the registered cells to only execute the satisfied ones.

Before runner execute each test cell, an extensible dependency resolver is running there to determinate the required/requested resources of a specific cell by analysing the tags, guarantee the cell always has all the dependencies it should receive. These resources are typically composed of :

• The testing target
• The testing library

Testing Request

A request is simply a loading of the runner page, it's the beginning of the testing life cycle. The request could optionally consist of a criteria which is a list of tags to filter down the test cells to run, could be specified via :

• A ampersand separated url params;
• A predefined JSON file denote the criteria;
• Specified through UI of the test runner page;