Inputing invalid markup will certainly generate unpredictable results, not only because of the editor code, but also (and mainly) because of browsers' DOM parsers, which often are buggy (specially IE's).
This is something really hard to handle, because of the hundreds of invalid things to consider. In such cases, it's better to not give access to the source view for users that don't know how to use it properly. I know, sometimes they really want it, so they must also be aware about the consequences.
In the described case, the editor tries to fix things when switching to source the first time. Then, the fix in interpreted in a different way by the browser, and the editor generates a different output again. It continues until the code stabilize and become valid (three times in this case).
I don't see much chances for a fix for now. We could think about a custom HTML parser that fixes the input text, but this is a huge work, with impacts also in the loading performance.
