Gary D. Gregory created LANG-1770:
-------------------------------------
Summary: StringUtils.abbreviate is not emoji aware, breaks
surrogate pairs
Key: LANG-1770
URL: https://issues.apache.org/jira/browse/LANG-1770
Project: Commons Lang
Issue Type: Bug
Components: lang.*
Affects Versions: 3.17.0
Reporter: Gary D. Gregory
---------- Forwarded message ---------
From: Carsten Kirschner <carsten.kirsch...@corussoft.de.invalid>
Date: Fri, Apr 11, 2025 at 10:15 AM
Subject: [lang] StringUtils.abbreviate is not emoji aware, breaks surrogate
pairs
To: d...@commons.apache.org <d...@commons.apache.org>
Hello,
The current commons lang3 StringUtils.abbreviate (3.17.0) implementation will
destroy 4 byte emoji characters and larger grapheme clusters. I know that
handling grapheme correctly before java 20 is not possible, but at least a
codepoint aware solution with String.offsetByCodPoints could be build. I wrote
a small test to show the problem.
The zero width joiners in the family emoji are questionable for the abbreviate,
but there should never be a question mark for an invalid char in the result as
there is now.
The problem is not so much the „doesn’t look nice“ aspect of the broken emoji,
but if that abbreviated string is passed to an XML Writer
(com.ctc.wstx.io.UTF8Writer in my case) it throws an exception on this broken
byte sequence. Like this: Caused by: java.io.IOException: Broken surrogate
pair: first char 0xd83c, second 0x2e; illegal combination
at
com.ctc.wstx.io.UTF8Writer._convertSurrogate(UTF8Writer.java:402)
~[woodstox-core-7.0.0.jar:7.0.0]
Thanks,
Carsten
import org.apache.commons.lang3.StringUtils;
import org.junit.Test;
import static org.junit.Assert.*;
public class AbbreviateTest {
String[] expectedResultsFox = {
"🦊...", // 4
"🦊🦊...",
"🦊🦊🦊...",
"🦊🦊🦊🦊...",
"🦊🦊🦊🦊🦊...",
"🦊🦊🦊🦊🦊🦊...",
"🦊🦊🦊🦊🦊🦊🦊...", // 10
};
String[] expectedResultsFamilyWithCodepoints = {
"👩...",
"👩🏻...",
"👩🏻...", // zero width joiner
"👩🏻👨...",
"👩🏻👨🏻...",
"👩🏻👨🏻...",
"👩🏻👨🏻👦..."
};
String[] expectedResultsFamilyWithGrapheme = {
"👩🏻👨🏻👦🏻👦🏻...", // 4
"👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼...",
"👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼👩🏽👨🏽👦🏽👦🏽...",
"👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼👩🏽👨🏽👦🏽👦🏽👩🏾👨🏾👦🏾👦🏾...",
"👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼👩🏽👨🏽👦🏽👦🏽👩🏾👨🏾👦🏾👦🏾👩🏿👨🏿👦🏿👦🏿...",
"👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼👩🏽👨🏽👦🏽👦🏽👩🏾👨🏾👦🏾👦🏾👩🏿👨🏿👦🏿👦🏿👩🏻👨🏻👦🏻👦🏻...",
"👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼👩🏽👨🏽👦🏽👦🏽👩🏾👨🏾👦🏾👦🏾👩🏿👨🏿👦🏿👦🏿👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼..."
// 10
};
@Test
public void abberviateTest() {
String abbreviateResult;
for(var i = 4; i <= 10; i++) {
abbreviateResult =
StringUtils.abbreviate("🦊🦊🦊🦊🦊🦊🦊🦊🦊🦊🦊🦊🦊🦊", i);
System.out.println(abbreviateResult);
//assertEquals(expectedResultsFox[i - 4], abbreviateResult);
}
for(var i = 4; i <= 10; i++) {
abbreviateResult =
StringUtils.abbreviate("👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼👩🏽👨🏽👦🏽👦🏽👩🏾👨🏾👦🏾👦🏾👩🏿👨🏿👦🏿👦🏿👩🏻👨🏻👦🏻👦🏻👩🏼👨🏼👦🏼👦🏼👩🏽👨🏽👦🏽👦🏽👩🏾👨🏾👦🏾👦🏾👩🏿👨🏿👦🏿👦🏿",
i);
System.out.println(abbreviateResult);
//assertEquals(expectedResultsFamilyWithCodepoints[i - 4], abbreviateResult);
}
}
}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
