V3: Packager

[Frederico Caldeira Knabben]

I've worked a few days to come out with a new packager. I've started it as just a research thing, but the coded evolved into a real solution. For now, the code has been committed into the "_dev/packager/ckpackager" folder in the CKEditor prototype.

The basic idea is stop doing string manipulations to the source code with miraculous regular expressions and use a real JavaScript parser. I've chosen ​Rhino for it, because it also made it possible to code all that stuff in our preferred language: JavaScript.

Rhino is a full JavaScript parser, interpreter and compiler. I was more interested on the first part of it, the parser.

The results are pretty nice. Rhino gives us a "token tree". We then walk over this three and re-write the source, token by token. So, for example, if we find a VAR token, we just write "var" to the output. Under it, we have the NAME token, which contains the variable name, so we rename and write it to the source too. The results is "var a;", for example.

Another interesting fact is that the parser gives us the "interpreted" token three. We don't have the exact same thing we had in the original code. A simple example; if we have "var a = 1 + 2;" in the source, the final result will be "var a=3;", because Rhino automatically summed the numbers for us. Another example, "while(true){}" and "for(;;){}" give us the same token three with Rhino, so we just output it in our preferred way, as we really don't know how exactly the source was.

There are also several code enhancements that are done by this new packager. Things we would never be able to achieve with the previous packager. Check out the "test/test.js" file for a long list of examples.

Btw, this implementation contains a basic automated test system, so we can be sure we are not breaking things on changes.

In the "ckpackager/_dev" folder, you will find two batch files. Both of them are configured to act over the code available in the script.js file, present in that folder also. Just run dump.bat to have a visual presentation of the token three representation of that script. The compress.bat file will instead print the compressed result.

There are several positive things with this packager. One of them is that all pending tickets for FCKpackager have been solved with it, and no ticket regressions have been found. Also, there are no special requirements when coding. No problems with missing semicolons, or even special ways to write things, like regular expressions and division signs.

There are though a few negative things that need attention. The first is the performance. It is twice as slow as the previous packager. This is not a critical thing, and there is certainly room for enhancements here.

The other problem is that, this packager has been written based on coding patterns. I think I have handled all syntax situations in the current implementation, but, for example, I've written things like "(success ? completed : failed).push( 'thing' )" yesterday, and this pattern was not been handled. The code got broken. Fortunately I had some intuition that it may happen, so I've just added a test case for that coding line and coded the packager to make it work with it. So, to summarize, we may find that the packager is breaking our code in the future, and we'll be working to add all necessary cases to it.

For the configuration file, it is not using anymore an XML file. It uses instead a JavaScript object literal like syntax. For CKEditor, the "ckeditor.pack" file has been added to the root of the project. It exemplifies the package file syntax.

Also, the _dev/packager/packagefilegen.html file has been updated to generate both the old and the new package file contents automatically. Just run _dev/packager/package_2.bat to execute the new packager over the CKEditor code.

Regarding the deployment... Right now, the entire ckpackager folder is needed to run it. Java is required. This is another negative thing, as we just have a single file to be used with the previous packager to make it work.

I haven't investigated it well, but I'm sure we are able to compile all that stuff in a single .jar file for the deployment, and possibly even generate an exe for it. But, this is something to understand yet.

To conclude, here are the numbers I have, by running both packagers over the current CKEditor prototype (second run):

• FCKpackager:
  • ckeditor_basic.js: 6,449 bytes (18.72% of original)
  • ckeditor.js......: 45,821 bytes (21.38% of original)
  • total time.......: 1.55 seconds.

• CKPackager 2.0:
  • ckeditor_basic.js: 5,813 bytes (17% of original)
  • ckeditor.js......: 44,199 bytes (21% of original)
  • total time.......: 2.97 seconds.

If we think this new packager is the way to go for us, I'll move it to its dedicated SVN three as a separated project and put an SVN external in the prototype three pointing to it. I'll wait the ticket review for it.

[Frederico Caldeira Knabben]

I've just introduced the PACKAGER_RENAME feature with [2371], as well as the "wrap" option.

When "wrap" is set, the entire code is placed inside a (function(){...})() call, isolating it from the global scope, creating a new scope space for variable renaming.

The PACKAGER_RENAME thing instead makes it possible to declare in the code, references that can be renamed by the packager. For example, the CKEDITOR and CKEDITOR.tools references have been marked for rename, so all references to them are changed to a short variable name. No one now can blame on us because we are using namespaces.

The final results:

• ckeditor_basic.js: 5,451 bytes (16% of original)
• ckeditor.js......: 41,176 bytes (19% of original)
• total time.......: 3.187 seconds.

As we can see, the name replacement lookup impacted in the execution time a bit, but it is definitely compensated by the rest.

[Frederico Caldeira Knabben]

I've created a dedicated branch for CKPackager, including it as a SVN external now. I've also removed all the old FCKpackager stuff from the prototype, electing the new CKPackager as our packager solution. All that happened between [2378] and [2384].