id summary reporter owner description type status priority milestone component version resolution keywords cc 2477 V3: Packager Frederico Caldeira Knabben Frederico Caldeira Knabben "I've worked a few days to come out with a new packager. I've started it as just a research thing, but the coded evolved into a real solution. For now, the code has been committed into the ""_dev/packager/ckpackager"" folder in the CKEditor prototype. The basic idea is stop doing string manipulations to the source code with miraculous regular expressions and use a real JavaScript parser. I've chosen [http://www.mozilla.org/rhino/ Rhino] for it, because it also made it possible to code all that stuff in our preferred language: JavaScript. Rhino is a full JavaScript parser, interpreter and compiler. I was more interested on the first part of it, the parser. The results are pretty nice. Rhino gives us a ""token tree"". We then walk over this three and re-write the source, token by token. So, for example, if we find a VAR token, we just write ""var"" to the output. Under it, we have the NAME token, which contains the variable name, so we rename and write it to the source too. The results is ""{{{var a;}}}"", for example. Another interesting fact is that the parser gives us the ""interpreted"" token three. We don't have the exact same thing we had in the original code. A simple example; if we have ""{{{var a = 1 + 2;}}}"" in the source, the final result will be ""{{{var a=3;}}}"", because Rhino automatically summed the numbers for us. Another example, ""while(true){}"" and ""for(;;){}"" give us the same token three with Rhino, so we just output it in our preferred way, as we really don't know how exactly the source was. There are also several code enhancements that are done by this new packager. Things we would never be able to achieve with the previous packager. Check out the ""test/test.js"" file for a long list of examples. Btw, this implementation contains a basic automated test system, so we can be sure we are not breaking things on changes. In the ""ckpackager/_dev"" folder, you will find two batch files. Both of them are configured to act over the code available in the script.js file, present in that folder also. Just run dump.bat to have a visual presentation of the token three representation of that script. The compress.bat file will instead print the compressed result. There are several positive things with this packager. One of them is that '''all''' pending tickets for FCKpackager have been solved with it, and no ticket regressions have been found. Also, there are no special requirements when coding. No problems with missing semicolons, or even [ticket:1242#comment:2 special ways to write things], like regular expressions and division signs. There are though a few negative things that need attention. The first is the performance. It is twice as slow as the previous packager. This is not a critical thing, and there is certainly room for enhancements here. The other problem is that, this packager has been written based on coding patterns. I think I have handled all syntax situations in the current implementation, but, for example, I've written things like ""{{{(success ? completed : failed).push( 'thing' )}}}"" yesterday, and this pattern was not been handled. The code got broken. Fortunately I had some intuition that it may happen, so I've just added a test case for that coding line and coded the packager to make it work with it. So, to summarize, we may find that the packager is breaking our code in the future, and we'll be working to add all necessary cases to it. For the configuration file, it is not using anymore an XML file. It uses instead a JavaScript object literal like syntax. For CKEditor, the ""ckeditor.pack"" file has been added to the root of the project. It exemplifies the package file syntax. Also, the _dev/packager/packagefilegen.html file has been updated to generate both the old and the new package file contents automatically. Just run _dev/packager/package_2.bat to execute the new packager over the CKEditor code. Regarding the deployment... Right now, the entire ckpackager folder is needed to run it. Java is required. This is another negative thing, as we just have a single file to be used with the previous packager to make it work. I haven't investigated it well, but I'm sure we are able to compile all that stuff in a single .jar file for the deployment, and possibly even generate an exe for it. But, this is something to understand yet. To conclude, here are the numbers I have, by running both packagers over the current CKEditor prototype (second run): * FCKpackager: * ckeditor_basic.js: 6,449 bytes (18.72% of original) * ckeditor.js......: 45,821 bytes (21.38% of original) * total time.......: 1.55 seconds. * CKPackager 2.0: * ckeditor_basic.js: 5,813 bytes (17% of original) * ckeditor.js......: 44,199 bytes (21% of original) * total time.......: 2.97 seconds. If we think this new packager is the way to go for us, I'll move it to its dedicated SVN three as a separated project and put an SVN external in the prototype three pointing to it. I'll wait the ticket review for it. " New Feature closed Normal CKEditor 3.0 General fixed Confirmed V3ProtoOther Review+