Hi core-libs-dev, I have a question regarding opaque region bounds in java.util.regex, particularly when a region boundary splits a surrogate pair or an extended grapheme cluster.
The JDK often retains context from outside an opaque region, but the behavior varies between constructs. For example, when a region contains only the high surrogate of a valid pair, OpenJDK 26.0.2 rejects `.`, accepts `.*` within the region, and allows \X to match the complete pair beyond the region end. Grapheme-boundary assertions can also depend on characters outside the opaque region. I’m trying to understand which aspects of this behavior are intended. The Matcher.useTransparentBounds() Javadoc <https://docs.oracle.com/en/java/javase/26/docs/api/java.base/java/util/regex/Matcher.html#useTransparentBounds(boolean)> says that, with opaque bounds, lookaround and boundary matching constructs "cannot look past the boundaries." I can see two possible interpretations: 1. The region is independent text for character decoding and boundary analysis. A surrogate half exposed by a region boundary is unpaired, and grapheme segmentation uses only the region’s contents. For these purposes, an opaque region behaves like the corresponding substring. 2. Characters and grapheme clusters are interpreted in the context of the whole input. The region restricts matching, but opacity does not necessarily redefine surrogate pairing or segmentation at its edges. In SafeRE <https://github.com/eaftan/safere>, I’ve generally been using the first interpretation. However, reconciling that interpretation with observed JDK behavior is leading us into increasingly awkward edge cases, and I’m wondering whether we have the intended model right. We have recorded several related differences: * Ordinary character matching at split surrogate pairs: our proposed behavior treats an exposed surrogate half consistently as unpaired, whether matched by `.` or a quantified form such as `.*`. The JDK sometimes treats these differently. See SafeRE #867 <https://github.com/eaftan/safere/issues/867>. * Opaque regions beginning inside a grapheme cluster: SafeRE generally starts grapheme context at the region boundary, allowing a leading ZWJ to form a standalone cluster. The JDK sometimes retains hidden prefix context. We document this as Region-Local Grapheme Continuation Clusters <https://github.com/eaftan/safere/blob/main/INTENTIONAL_DIVERGENCES.md#region-local-grapheme-continuation-clusters>, with examples in #451 <https://github.com/eaftan/safere/issues/451>. * Opaque regions splitting CRLF: SafeRE treats a visible LF independently of a hidden preceding CR for end-anchor purposes. The JDK’s $ and \Z results can depend on that hidden CR. This is documented as Opaque Region CRLF Pair Context <https://github.com/eaftan/safere/blob/main/INTENTIONAL_DIVERGENCES.md#opaque-region-crlf-pair-context> and reported upstream as JDK-8394795 <https://bugs.openjdk.org/browse/JDK-8394795>. A recent example combines surrogate pairing and grapheme boundaries: var p = java.util.regex.Pattern.compile(".\\b{g}"); // Default opaque and anchoring bounds. System.out.println(p.matcher("\uD83D\uDE00\u0301") .region(1, 3).lookingAt()); // false System.out.println(p.matcher("x\uDE00\u0301") .region(1, 3).lookingAt()); // true System.out.println(p.matcher("\uDE00\u0301") .lookingAt()); // true These are the results on OpenJDK 26.0.2. The visible region in the first two cases is identical: a low surrogate followed by a combining acute accent. Only the hidden character before the region changes. SafeRE currently produces the same results here, which exposes an inconsistency in our own independent-region model. There are also differences between consuming constructs. For "\uD83D\uDE00" with opaque region [0,1), JDK 26.0.2 gives: * `.` -- lookingAt() returns false. * `.*` -- returns true, matching [0,1). * \p{Cs} -- returns false. * \X -- returns true, matching [0,2), beyond the region end. Thus \X can finish beyond the region end, while `.` and `.*` disagree about consuming the exposed high surrogate. For comparison, I tested the same split-surrogate example using ICU4C 74.2’s regex matcher <https://unicode-org.github.io/icu/userguide/strings/regexp.html>, with opaque region [0,1). Both . and \X succeed, matching [0,1): the exposed high surrogate is matched independently, and neither match extends beyond the region end. This is consistent with interpretation 1 in this example and differs from the JDK results above. What is the intended interpretation of opaque bounds here? Should surrogate pairing and grapheme segmentation use only the region’s contents, or may they depend on surrounding input? And is \X completing a character beyond the region end intended behavior? I’m asking primarily to clarify the contract rather than assuming that every difference above is an implementation bug. Thanks, Eddie
