Opened 16 years ago
Closed 16 years ago
#4218 closed Bug (fixed)
Implement CKTester
Reported by: | Garry Yao | Owned by: | Garry Yao |
---|---|---|---|
Priority: | Normal | Milestone: | CKEditor 3.1 |
Component: | QA | Version: | |
Keywords: | Cc: |
Description (last modified by )
CKTester is our internal universal testing swarm, it should be capable of running tests for different projects
with different testing tools easily while remain simplicity, below are some basic ideas of the project
based on the previous effort on 'FCKtest'.
Test Cell
A test cell is basically a plain html file reside in individual project's testing source, in which test suites/test cases
are written. It'll be executed inside an iframe/popup of the fort so full page life cycle
( from page load to page destroy ) is guaranteed when performing the test.
Each cell has only key responsibility of reporting the sucess/failure/time of it's execution to the fort, this is
done by cross-frame communication with parent/opener window.
The cell's testing environment, including testing target, runtime libraries, etc. will be all injected by the fort, initiatively or passively, the only required effort in the cell is to include the bootstrap file.
<script type="text/javascript" src="{CKTester_ROOT}/bootstrap.js"></script>
A cell could be running in two modes:
- Managed Mode : It's the default mode when it's running in a batch of cells by the forge, where the cell is running in an iframe.
- Standalone Mode : This mode is adapted when a cell is running separately, on the opposite of 'Managed Mode', the cell will be running inside a popup window, and it doesn't require to report to the fort.
Profile
A tests profile is a js file contains information about running a bunch of test cells. It a function declaration which return a configuration object, contains:
- Cell variables : An hash of variable names and their values which are often embeded in profile paths to indicate location.
- Cells list : A list of key/values:
- Key : The test cell file path, where all the paths are relative to the runner;
- Value : An array of tags.
- Cell resolvers : An array of function which were used to manipulate each cell's attributes at run-time, e.g. Inject necessary dependencies for cells when encounter certain tags. Note that The fort already has a few default resolvers, usages including : converting the cell path to tags; calculate resource path for each cell, etc.
A profile will basically looks like:
CKTester.profile = function () { return { variables : { 'CKEDITOR_ROOT' : 'path-to-editor-root' }, cells : [ [ 'editor/tt/unit/plugins/htmldataprocessor/1' ], // Implicit tag declaration. [ '3210.html' , [ 'tt', 'unit', 'htmldataprocessor'] ] // Explicit tag declaration. ] cellResolvers : [ ... ] }; }
It's important to note that tag plays an important role as the following indicators:
- Environments( dependencies ) of a cell, e.g. if the cell has tags 'unit' and 'editor', it denote a unit test running by YUITest, along with the CKEditor testing stage is required to run the test;
- Categories of the cell, when it's been run in a runner with a specific criteria. E.g. A url criteria as 'http://t/runner?cells=tt,plugins' will run only the tests of category 'ticket' and 'plugins' ( with 'ticket' and 'plugins' tags );
- Any reasonable denotation.
Fort
The Test Fort is a bridge between the runner tools/framework ( e.g. Selenium ) underneath, and the testing resources, so it's above the level of any TC or TS. It's resided in CKTester project.
The fort is responsible for delegating the running of all the tests to the under-laying tools/framework and collecting the testing result neutrally. It consist of a UI to interact with and providing report.
The runner is required to be running with a specified profile, from which it will register all the defined tests cells in it. Upon requesting by a specific criteria on running, it filtering down the registered cells to only execute the satisfied ones.
Generally, before runner execute each test cell, an extensible cell resolver is working out the required/requested resources of a specific cell by analysing the tags, guarantee the cell always has all the dependencies it should receive. These resources are typically composed of :
- The testing target
- The testing library
- Any resources dedicated to the cell.
Testing Request
A request is simply a page load of the fort, which form the begin of the testing life cycle.
- The request must use url param to specify the location of profile file relative of the fort.
- The request url could optionally consist of a criteria which is a list of tags or path to filter the test cells to run, it could be
specified in various ways:
- A comma separated param in URL;
- Specified through UI of the test runner page;
Some example testing requests look like ( I've remove the 'profile' param for clear understanding ):
- Running only the ticket tests of unit test type: http://t/fort.html?cells=tt,unit
- Running all the stable tests : http://t/fort.html?cells=stable
- Running two specific tests : http://t/fort.html?cells=../editor/tt/unit/htmldataprocessor/1,../editor/dd/selenium/visual/1
Note that since it's not convenient to specify a cell path within URL, it's a shortcut that open the cell url directly would have it redirect to correct url back to the forge.
To-Be Implemented Features
- Auto sync URL with criteria input UI;
- Stop/Skip running tests capability;
- Separate file for customized( local ) profile variables definition to avoid frequent SVN merge of the default profile file.
- Fort APIs for communication with a test server.
Change History (16)
comment:1 Changed 16 years ago by
Owner: | set to Garry Yao |
---|---|
Status: | new → assigned |
comment:2 Changed 16 years ago by
Description: | modified (diff) |
---|
comment:3 Changed 16 years ago by
Description: | modified (diff) |
---|
Update description according to implementation:
- Rename 'runner' to 'fort' for disambiguation;
- Adding 'running mode' concepts of cell;
- Simplify 'criteria' pattern;
- Adding 'cell shortcut' concept;
comment:5 Changed 16 years ago by
Milestone: | CKEditor 3.0 → CKEditor 3.1 |
---|
comment:6 Changed 16 years ago by
Update with [4098], make it possible to run individual cell as soon as the cell starts.
comment:7 Changed 16 years ago by
Description: | modified (diff) |
---|
Adding some suggested features from Tobiasz after talking with him.
comment:8 Changed 16 years ago by
There're several changes has been decided after a dicussion with Fred:
- Fixing the following critical bugs
- Fort is loading unrelated resources even before running cells;
- Adding the following pragmatic features:
- Be able to run a specific cell directly without relaying on profile, the cell will be self-bootstrapping;
- Easily tag definition in cell html file;
- Introducing dev tool for generate testing profile file from scanning test directory.
The changes are effective with [4100].
comment:9 Changed 16 years ago by
After a dicussion with Fred, the profile file's cell path could be further simplified by making it a convention that the runner folder will reside in the root of test branch.
[ '../tt/4227/4227',[ 'editor','unit','all' ] ] // The path change to relative to the profile file( test root). [ 'tt/4227/4227',[ 'editor','unit','all' ] ]
comment:10 Changed 16 years ago by
comment:12 Changed 16 years ago by
comment:13 Changed 16 years ago by
Thanks for Tobias pointing out that the [generic testing API file http://svn.fckeditor.net/CKTester/runners/yuitest/test.js] is missing, compensate with [4132].
comment:14 Changed 16 years ago by
comment:15 Changed 16 years ago by
Fixing a bug where asking for absolute path in cell page cause error in all IEs with [4158].
comment:16 Changed 16 years ago by
Keywords: | Discussion removed |
---|---|
Resolution: | → fixed |
Status: | assigned → closed |
Little adjustments.