David Mollitor created SPARK-59394:
--------------------------------------
Summary: UnBase64: decode the input bytes directly instead of
converting to a String per row
Key: SPARK-59394
URL: https://issues.apache.org/jira/browse/SPARK-59394
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Summary
The {{UnBase64}} expression (SQL {{unbase64}} and {{{}try_to_binary(...,
'base64'){}}}) decodes a base64 string to bytes. Both its interpreted path and
its generated code call {{.toString()}} on the input {{UTF8String}} before
decoding, which decodes the whole input from UTF-8 into a Java {{String}} on
every row purely to hand it to the decoder. {{java.util.Base64.Decoder}}
accepts a byte array directly, so passing the raw UTF-8 bytes
({{{}UTF8String.getBytes(){}}}) removes that per-row {{String}} allocation and
UTF-8 decode.
h2. Why this is safe (no behavior change)
{{getMimeDecoder().decode(String)}} internally converts the string to bytes
with ISO-8859-1 before decoding. For valid base64 – which is always ASCII –
those bytes are identical to the raw UTF-8 bytes. For any non-ASCII or
otherwise non-base64 content, the MIME decoder ignores every byte outside the
base64 alphabet ({{{}A-Za-z0-9+/={}}}) under both paths, so the decoded output
is the same. The change therefore produces identical results for all inputs.
h2. Does this PR introduce _any_ user-facing change?
No.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
