Here's the first requirement that must be met to claim Level 1 compliance. Java does not yet meet this requirement, but it could easily do so: indeed, my own regex-rewriting library implements this requirement. It takes very little code at all and is *completely* backwards compatible because it uses a syntax that was previously illegal.
This guarantees there can have been no existing valid Java regex whose behaviour would be altered by this particular implementation of the fix. I personally believe this property of not changing how existing code behaves to be critical--although I also recognize alternate viewpoints that either consider it to be merely "very important" instead of "critical", or indeed which instead hold it to be an "indispensable requirement". Details below. http://www.unicode.org/reports/tr18/#Hex_notation +-------------------------- | RL1.1 Hex Notation | | To meet this requirement, an implementation shall supply a mechanism | for specifying any Unicode code point (from U+0000 to U+10FFFF) +-------------------------- Java allows you to specify code points from U+0000 to U+FFFF, not to U+10FFFF. You can specify code points only from Plane 0, not code points from Plane 1-16. This is true of both the lexical substitution prepass in which "\uXXXX" is replaced by the corresponding code point as well as during the regex compilation itself, during which "\\uXXXX" is recognized as a code point from Plane 0. Note that the first of those two, the lexical analysis phase, is insufficient to meet the requirement even for Plane 0, and so cannot be counted. This is because one must be able to specify *ANY* Unicode code point, but you cannot do that with \uXXXX even in Plane 0 because certain code points are forbidden, such as \u000A for line feed, \u0022 for double quote, and \u005C for backslash. Those are all treated as syntactic elements of Java itself, not as the listed Plane 0 code pint. Which leaves us with the \\uXXXX notation. That indeed allows you to specify any code point *IN PLANE 0*. It does not permit you to specify any Unicode code point directly per the requirement. Specifying single Unicode code points using indirect UTF-16 notation no more suffices for this requirement than doing so using indirect UTF-8 notation. Imagine if one had to specify Unicode code points using UTF-8 instead of UTF-16; I'll use the "\xXX" notation for this. That means that U+A3, POUND SIGN, would need to be specified as "\xC2\xA3". But that is in clear violation of what Level 1 must provide: Level 1: Basic Unicode Support. At this level, the regular expression engine provides support for Unicode characters as basic logical units. (This is independent of the actual serialization of Unicode as UTF-8, UTF-16BE, UTF-16LE, UTF-32BE, or UTF-32LE.) This is a minimal level for useful Unicode support. This hypothetical example fails to meet that requirement because you are no longer dealing with Unicode characters as basic logical units. You are forced to deal with serialization issues of UTF-8. I believe that hypothetical example is a clear violation of the most basic Level 1 requirement, so I believe it necessarily therefore follows that changing UTF-8 to UTF-16 must also fail to meet that requirement. For example, consider U+1F47E, ALIEN MONSTER. Java requires you to write that "\uD83D\uDC7E" for the preprocessing step or as "\\uD83D\\uDC7E" for the regex compiler. That breaks the requirement just as surely as does the equivalent UTF-8 encoding, "\xF0\x9F\x91\xBE", because both make you consider serialization issues instead of logical code points. Please do not be distracted that this example uses an Emoji code point from Unicode 6.0. I did choose that one for its cuteness effect, but the same problems applies to all non-Plane 0 code points, not just for the silly ones or the new ones. For example, Unicode 4.0 introduced code point U+1033C, GOTHIC LETTER MANNA, which is not a "silly" or "cute" code point; there are *very* many other such, too. How to fix this to bring Java into compliance? It's actually quite easy. However, you will *not* be able to fulfill this requirement by adopting the syntax UXXXXX or \UXXXXXXX, because that syntax is already taken by the regex compiler for toupper() case translation. The standard mentions that as an alternative to \uXXXX or \UXXXXXXX, the notation \x{XXXX}, and this is what I have elected to implement in my regex-rewriting library. Even though I show four X's there, you may use any number of them, thus allowing you to write \x{A3}, \x{2019}, \x{1033C}, or even \x{10FFFF}. This now meets the requirement of being able to specify any Unicode code point between U+0000 and U+10FFFF. --tom
