singhpratech opened a new issue, #51225:
URL: https://github.com/apache/arrow/issues/51225
**Describe the bug**
`pyarrow.compute.utf8_normalize` returns decomposed text for every `form`.
With `form="NFC"` the
output is the NFD form (base letter plus combining mark), whether the input
was precomposed or
decomposed. `unicodedata.normalize` from the standard library gives the
expected result.
pyarrow 25.0.1, Python 3.13.9, macOS 26.6 (arm64), wheel from PyPI.
```python
import pyarrow as pa, pyarrow.compute as pc, unicodedata
s = "héllo" # precomposed e-acute, U+00E9
a = pa.array([s], pa.string())
for form in ("NFC", "NFD", "NFKC", "NFKD"):
out = pc.utf8_normalize(a, form=form).to_pylist()[0]
exp = unicodedata.normalize(form, s)
print(form, [hex(ord(c)) for c in out], "expected", [hex(ord(c)) for c
in exp])
```
Output:
```
NFC ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68',
'0xe9', '0x6c', '0x6c', '0x6f']
NFD ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68',
'0x65', '0x301', '0x6c', '0x6c', '0x6f']
NFKC ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68',
'0xe9', '0x6c', '0x6c', '0x6f']
NFKD ['0x68', '0x65', '0x301', '0x6c', '0x6c', '0x6f'] expected ['0x68',
'0x65', '0x301', '0x6c', '0x6c', '0x6f']
```
The same happens when the input is already decomposed (`"héllo"`): NFC and
NFKC return it
unchanged instead of composing to U+00E9.
**Expected behavior**
`form="NFC"` and `form="NFKC"` compose, as `unicodedata.normalize` does; NFD
and NFKD are correct
today.
**Component(s)**
Python, C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
