lvyanquan commented on code in PR #4488:
URL: https://github.com/apache/flink-cdc/pull/4488#discussion_r3711028470
##########
docs/content/docs/core-concept/transform.md:
##########
@@ -178,10 +178,29 @@ Logical functions follow SQL three-valued logic for
nullable BOOLEAN values. `AN
| UPPER(string) | upper(string)
| Returns string in uppercase.
|
| LOWER(string) | lower(string)
| Returns string in lowercase.
|
| TRIM(string1) | trim('BOTH',string1)
| Returns a string that removes whitespaces at both sides.
|
+| LTRIM(string[, trimString]) | ltrim(string[,
trimString]) | Returns a string with leading characters in
trimString removed. Whitespace is removed by default.
|
+| RTRIM(string[, trimString]) | rtrim(string[,
trimString]) | Returns a string with trailing characters
in trimString removed. Whitespace is removed by default.
|
+| BTRIM(string[, trimString]) | btrim(string[,
trimString]) | Returns a string with leading and trailing
characters in trimString removed. Whitespace is removed by default.
|
| REGEXP_REPLACE(string1, string2, string3) | regexpReplace(string1,
string2, string3) | Returns a string from STRING1 with all the substrings that
match a regular expression STRING2 consecutively being replaced with STRING3.
E.g., 'foobar'.regexpReplace('oo\|ar', '') returns "fb". |
| SUBSTR(string, integer1[, integer2]) |
substr(string,integer1,integer2) | Returns a substring of STRING
starting from position integer1 with length integer2 (to the end by default).
|
| SUBSTRING(string FROM integer1 [ FOR integer2 ]) |
substring(string,integer1,integer2) | Returns a substring of STRING
starting from position integer1 with length integer2 (to the end by default).
|
+| OVERLAY(string1 PLACING string2 FROM integer1 [FOR integer2]) |
overlay(string1, string2, integer1[, integer2]) | Replaces a substring of
STRING1 with STRING2 from position integer1. The replaced length defaults to
the length of STRING2. Character and binary strings are supported.
|
+| POSITION(string1 IN string2 [ FROM integer ]) |
position(string1, string2[, integer]) | Returns the position of the
first occurrence of STRING1 in STRING2, optionally starting from integer. The
first position is 1. Returns 0 if not found. Character and binary strings are
supported. |
+| LOCATE(string1, string2[, integer]) |
locate(string1, string2[, integer]) | Returns the position of the
first occurrence of STRING1 in STRING2, optionally starting from integer. The
first position is 1. Returns 0 if not found.
|
+| INSTR(string1, string2) | instr(string1,
string2) | Returns the position of the first
occurrence of STRING2 in STRING1. The first position is 1. Returns 0 if not
found.
|
| CONCAT(string1, string2,…) | concat(string1,
string2,…) | Returns a string that concatenates string1, string2,
…. E.g., CONCAT('AA', 'BB', 'CC') returns 'AABBCC'.
|
+| CONCAT_WS(separator, string1, string2,...) |
concatWs(separator, string1, string2,...) | Returns a string that
concatenates string1, string2, ... with a separator. Null string arguments are
skipped.
|
+| LPAD(string1, integer, string2) | lpad(string1,
integer, string2) | Returns STRING1 left-padded with STRING2
to a length of integer characters. If STRING1 is longer, it is shortened to
integer characters. |
+| RPAD(string1, integer, string2) | rpad(string1,
integer, string2) | Returns STRING1 right-padded with STRING2
to a length of integer characters. If STRING1 is longer, it is shortened to
integer characters. |
+| REPLACE(string1, string2, string3) |
replace(string1, string2, string3) | Returns STRING1 with all
occurrences of STRING2 replaced by STRING3.
|
+| REPEAT(string, integer) | repeat(string,
integer) | Returns a string that repeats STRING
integer times.
|
+| LEFT(string, integer) | left(string,
integer) | Returns the leftmost integer characters
from STRING.
|
+| RIGHT(string, integer) | right(string,
integer) | Returns the rightmost integer characters
from STRING.
|
+| STARTSWITH(string1, string2) |
startswith(string1, string2) | Returns whether STRING1
starts with STRING2. Character and binary strings are supported.
|
+| ENDSWITH(string1, string2) |
endswith(string1, string2) | Returns whether STRING1 ends
with STRING2. Character and binary strings are supported.
|
+| TO_BASE64(string \| binary) |
toBase64(string \| binary) | Encodes a character or
binary string to a base64 string.
|
+| FROM_BASE64(string) |
fromBase64(string) | Decodes a base64 string to a
UTF-8 character string.
|
Review Comment:
Flink 2.2's current FROM_BASE64 runtime implementation preserves the decoded
byte sequence in BinaryStringData, including bytes that are not valid UTF-8.
For example, TO_BASE64(FROM_BASE64('wyg=')) returns 'wyg='.
This implementation converts the decoded bytes to java.lang.String, so
malformed UTF-8 sequences are replaced with U+FFFD. Consequently, the same
round trip returns '77+9KA=='.
Given that STRING values currently flow through java.lang.String in the
transform expression engine, matching Flink's byte-preserving behavior would
require broader runtime/type-conversion changes. Could we document that
FROM_BASE64 assumes valid UTF-8 input and recommend FROM_BASE64_BINARY for
arbitrary binary content? It would also be helpful to mention this intentional
divergence from Flink 2.2 in the PR description and add a test covering
non-UTF-8 decoded bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
