This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a8b95fafae8 [fix](nereids) Fold a VARBINARY to string cast to its
bytes like BE does (#69025)
a8b95fafae8 is described below
commit a8b95fafae8242104182c88e5fca7727feced5d2
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Fri Oct 16 19:15:35 2026 +0800
[fix](nereids) Fold a VARBINARY to string cast to its bytes like BE does
(#69025)
### What problem does this PR solve?
Issue Number: None
Related PR: #68968, #68930
Problem Summary:
**In short.** FE constant folding cast a VARBINARY constant to a string
type as the hex text of its bytes, while BE keeps the bytes. Every
folded expression that takes a VARBINARY constant through such a cast,
including `hex()` and `to_base64()` of a VARBINARY constant, returned a
different result from the same expression evaluated by BE. This PR folds
the cast to the bytes, leaves the cast for BE when no FE string literal
can carry its result, and makes the two FE places that could not handle
such a cast accept it; the third, in BE, is fixed by #68930. This came
up while testing #68968.
**Background.** `hex()` and `to_base64()` only have string signatures,
so a VARBINARY argument reaches them through an implicit `CAST(... AS
VARCHAR)`. BE implements `CAST(VARBINARY AS VARCHAR/STRING)` by keeping
the bytes as they are. When the argument is a constant,
`FoldConstantRuleOnFE` folds that cast with `Literal.checkedCastTo`,
which builds the string from `VarBinaryLiteral.castValueToString()`;
that method returned `toString()`, the base16 text of the bytes.
**The problem, and what it cost.**
| Query | folded on FE (master) | evaluated by BE
(`debug_skip_fold_constant=true`) / with this PR |
|---|---|---|
| `SELECT hex(cast('abc' AS varbinary(16)))` | `363136323633` | `616263`
|
| `SELECT to_base64(cast('abc' AS varbinary(16)))` | `NjE2MjYz` | `YWJj`
|
| `SELECT length(cast(cast('abc' AS varbinary(16)) AS string))` | 6 | 3
|
| `SELECT hex(X'FF')` | `4646` | `FF` |
| `SELECT k FROM t WHERE s = cast(X'61' AS string)` | no rows (compares
with `'61'`) | the row with `'a'` |
The plan shows the difference: master sends `hex('616263')` to BE, while
with folding disabled it sends `hex(CAST(X'616263' AS varchar(65533)))`.
The same wrong value appears for a column read from a constant subquery
or a view.
**How this PR fixes it.**
- `VarBinaryLiteral.castValueToString()` decodes the bytes as UTF-8
(`CodingErrorAction.REPORT`), so the folded string carries exactly the
bytes BE keeps; valid UTF-8 round-trips unchanged when the literal is
sent to BE.
- For some bytes `castValueToString()` throws an `AnalysisException`,
and `FoldConstantRuleOnFE.visitCast`, which turns only a `CastException`
into NULL, then leaves the cast for BE to evaluate:
- bytes that are not valid UTF-8, which a Java string cannot carry;
- text with a NUL byte or a supplementary character, which FE evaluates
differently from BE: `StringLikeLiteral.compareTo` takes the bytes after
a NUL as padding, so `CAST(X'610062' AS STRING) = 'a'` would fold to
TRUE, and `VARCHAR(n)` truncation and `trim_in` work on UTF-16 units, so
they would split U+1F600.
- Master always folded such a cast to the hex text, so three places had
never met it in a plan:
- BE: `ScanOperator` digests every conjunct for the condition cache (on
by default), and `VLiteral::get_digest` hashed the literal's column,
which `ColumnVarbinary` does not support. `WHERE s = CAST(X'FF' AS
STRING)` failed with `update_xxHash_with_value is not supported for
ColumnVarbinary`. #68930 has since made `VLiteral::get_digest` return 0
for a VARBINARY column, which keeps such a conjunct out of the condition
cache. This PR made the same change before it was rebased onto #68930,
and now leaves BE unchanged.
- FE, fixed here: `CheckAfterRewrite` rejected a hash or mark join
condition with any VARBINARY sub-expression, so `CAST(X'610062' AS
STRING) IN (SELECT s FROM t)` on a nullable `s` failed with `varbinary
type could not in join equal conditions`. It now rejects a condition
only when a comparison in it has a VARBINARY operand.
- FE, fixed here: with `enable_fold_constant_by_be=true`, BE folded such
a cast when it was the largest constant part of a predicate, and
`FoldConstantRuleOnBE.getResultExpression` decoded the result as UTF-8
with U+FFFD for invalid bytes, so `WHERE s = CAST(X'FF' AS STRING)`
became `s = '�'`. `shouldSkipFold` now skips a cast from a type that
contains VARBINARY, as it already skips `from_base64` for the same
reason.
**Results.** On a local cluster, with a table `t` holding `'a'`, `'a'`
NUL `'b'`, the byte 0xFF, U+FFFD, NULL and U+1F600:
| Query | master | this PR, in each of the three modes below |
|---|---|---|
| `WHERE s = CAST(X'FF' AS STRING)` | no rows | the 0xFF row |
| `WHERE s = CAST(X'610062' AS STRING)` | no rows | the `'a'` NUL `'b'`
row |
| `WHERE s = CAST(CAST(X'F09F9880' AS STRING) AS VARCHAR(1))` | no rows
| the U+1F600 row |
| `SELECT CAST(X'FF' AS STRING) IN (SELECT s FROM t WHERE k <> 4)` |
NULL | true |
The three modes are FE folding (the default), FE and BE folding
(`enable_fold_constant_by_be=true`), and no folding
(`debug_skip_fold_constant=true`); P0 runs turn
`enable_fold_constant_by_be` on at random. `test_varbinary_cast_fold`
runs every query in the three modes with the condition cache on, and
each block returns the same rows in all three. The FE UT fails when
`castValueToString()` returns the hex text again (`expected: <'abc'> but
was: <'616263'>`), and each of the two FE fixes above has a test that
fails without it.
The existing `.out` of
`query_p0/sql_functions/binary_functions/test_binary_hex_function`
recorded the folded hex text for `sql_25` (`select cast(X'61' as
string), ...` gave `61`), while its `sql_24` already expected `a` for
the same bytes read from a column. It is regenerated by the script. The
`.out` of
`external_table_p0/dialect_compatible/sql/presto/scalar/TestVarbinaryFunctions`
already expects the BE values (`YWJj` for `to_base64(CAST('abc' AS
VARBINARY))`).
---
.../nereids/rules/analysis/CheckAfterRewrite.java | 12 ++-
.../expression/rules/FoldConstantRuleOnBE.java | 8 ++
.../expressions/literal/VarBinaryLiteral.java | 39 +++++++++-
.../doris/nereids/util/TypeCoercionUtils.java | 21 ++----
.../rules/expression/FoldConstantRuleOnBETest.java | 22 ++++++
.../nereids/rules/expression/FoldConstantTest.java | 35 +++++++++
.../nereids/types/VarBinarySqlSupportTest.java | 54 +++++++++++++
.../binary_functions/test_binary_hex_function.out | 2 +-
.../binary_functions/test_varbinary_cast_fold.out | 78 +++++++++++++++++++
.../test_varbinary_cast_fold.groovy | 88 ++++++++++++++++++++++
10 files changed, 339 insertions(+), 20 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckAfterRewrite.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckAfterRewrite.java
index 8b2f19b8532..ebdbe377da1 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckAfterRewrite.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckAfterRewrite.java
@@ -23,6 +23,7 @@ import org.apache.doris.nereids.properties.OrderKey;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.Match;
import org.apache.doris.nereids.trees.expressions.Slot;
@@ -192,13 +193,13 @@ public class CheckAfterRewrite extends
OneAnalysisRuleFactory {
} else if (plan instanceof LogicalJoin) {
LogicalJoin<?, ?> join = (LogicalJoin<?, ?>) plan;
for (Expression conjunct : join.getHashJoinConjuncts()) {
- if (conjunct.anyMatch(e -> ((Expression)
e).getDataType().isVarBinaryType())) {
+ if (comparesVarBinary(conjunct)) {
throw new AnalysisException(
"varbinary type could not in join equal
conditions: " + conjunct.toSql());
}
}
for (Expression conjunct : join.getMarkJoinConjuncts()) {
- if (conjunct.anyMatch(e -> ((Expression)
e).getDataType().isVarBinaryType())) {
+ if (comparesVarBinary(conjunct)) {
throw new AnalysisException(
"varbinary type could not in join equal
conditions: " + conjunct.toSql());
}
@@ -206,6 +207,13 @@ public class CheckAfterRewrite extends
OneAnalysisRuleFactory {
}
}
+ // BE cannot hash or compare VARBINARY values, but a join condition may
compare strings cast from VARBINARY,
+ // such as CAST(X'FF' AS STRING) = s, so only the compared operands are
checked.
+ private static boolean comparesVarBinary(Expression conjunct) {
+ return conjunct.anyMatch(e -> e instanceof ComparisonPredicate &&
((ComparisonPredicate) e).children()
+ .stream().anyMatch(operand ->
operand.getDataType().isVarBinaryType()));
+ }
+
private void checkMatchIsUsedCorrectly(Plan plan) {
for (Expression expression : plan.getExpressions()) {
if (expression instanceof Match) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnBE.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnBE.java
index 2c0789e01c4..d8fd83457c8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnBE.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnBE.java
@@ -79,6 +79,7 @@ import org.apache.doris.nereids.types.MapType;
import org.apache.doris.nereids.types.StructField;
import org.apache.doris.nereids.types.StructType;
import org.apache.doris.nereids.types.TimeStampNsType;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
import org.apache.doris.proto.InternalService;
import org.apache.doris.proto.InternalService.PConstantExprResult;
import org.apache.doris.proto.Types.PScalarType;
@@ -298,6 +299,13 @@ public class FoldConstantRuleOnBE implements
ExpressionPatternRuleFactory {
cast.isStrict() || SessionVariable.enableStrictCast()) ||
cast.child().isNullLiteral()) {
return true;
}
+ // FoldConstantRuleOnFE leaves a cast of VARBINARY bytes for BE
when no FE string literal can carry
+ // its result (see VarBinaryLiteral.castValueToString). Folding
the cast here would build that literal
+ // from the BE result anyway: getResultExpression decodes invalid
UTF-8 as U+FFFD, and FE string
+ // code treats a NUL byte or a supplementary character unlike BE.
+ if
(TypeCoercionUtils.hasVarBinaryType(cast.child().getDataType())) {
+ return true;
+ }
}
// This kind of function is often used to change the attributes of
columns.
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/VarBinaryLiteral.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/VarBinaryLiteral.java
index 6813e6da409..7acfbad76da 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/VarBinaryLiteral.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/VarBinaryLiteral.java
@@ -18,14 +18,20 @@
package org.apache.doris.nereids.trees.expressions.literal;
import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
import org.apache.doris.nereids.types.DataType;
import org.apache.doris.nereids.types.VarBinaryType;
import com.google.common.io.BaseEncoding;
+import java.nio.ByteBuffer;
+import java.nio.charset.CharacterCodingException;
+import java.nio.charset.CodingErrorAction;
+import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.Locale;
+import java.util.Optional;
/**
* Represents varbinary literal
@@ -73,8 +79,39 @@ public class VarBinaryLiteral extends Literal implements
ComparableLiteral {
return byteValues;
}
+ /**
+ * BE casts VARBINARY to a string type by keeping the bytes as they are,
so CAST(X'616263' AS STRING) is
+ * 'abc', not the hex text '616263'. A Java string carries those bytes
only when they are valid UTF-8; for
+ * other bytes, such as X'FF', this throws, and
FoldConstantRuleOnFE.visitCast then leaves the cast for BE
+ * to evaluate (it turns only a CastException into NULL).
+ *
+ * <p>Valid UTF-8 with a NUL byte or a supplementary character is left for
BE too, because FE does not
+ * evaluate such string literals like BE does: StringLikeLiteral.compareTo
takes the bytes after a NUL for
+ * padding, so CAST(X'610062' AS STRING) = 'a' would fold to TRUE while BE
compares all three bytes, and
+ * VARCHAR(n) truncation and trim_in work on UTF-16 units, so
CAST(X'F09F9880' AS VARCHAR(1)) would keep
+ * half of U+1F600 where BE keeps the whole character.</p>
+ */
+ @Override
protected String castValueToString() {
- return toString();
+ String value = toUtf8String().orElseThrow(() -> new AnalysisException(
+ "Cannot cast VARBINARY " + computeToSql() + " to a string type
on FE: it is not valid UTF-8"));
+ if (value.indexOf('\0') >= 0 ||
value.codePoints().anyMatch(Character::isSupplementaryCodePoint)) {
+ throw new AnalysisException("Cannot cast VARBINARY " +
computeToSql()
+ + " to a string type on FE: it has a NUL byte or a
supplementary character");
+ }
+ return value;
+ }
+
+ /** The bytes decoded as UTF-8, or empty when they are not valid UTF-8 and
only BE can cast them. */
+ public Optional<String> toUtf8String() {
+ try {
+ return Optional.of(StandardCharsets.UTF_8.newDecoder()
+ .onMalformedInput(CodingErrorAction.REPORT)
+ .onUnmappableCharacter(CodingErrorAction.REPORT)
+ .decode(ByteBuffer.wrap(byteValues)).toString());
+ } catch (CharacterCodingException e) {
+ return Optional.empty();
+ }
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
index f8b722e40cf..1a27be89e55 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java
@@ -387,6 +387,10 @@ public class TypeCoercionUtils {
return hasSpecifiedType(dataType, TimeStampNsType.class);
}
+ public static boolean hasVarBinaryType(DataType dataType) {
+ return hasSpecifiedType(dataType, VarBinaryType.class);
+ }
+
private static boolean hasTimeStampNsCompatibleDateTimeType(DataType
dataType) {
return hasSpecifiedType(dataType, type -> type instanceof DateTimeType
|| type instanceof DateTimeV2Type || type instanceof
TimeStampTzType);
@@ -2933,27 +2937,12 @@ public class TypeCoercionUtils {
// Common binary types support value selection, but BE comparison
kernels do not.
// Check before equal-type shortcuts or casts can hide an unsupported
binary argument.
for (Expression argument : arguments) {
- if (containsVarbinary(argument.getDataType())) {
+ if (hasVarBinaryType(argument.getDataType())) {
throw new AnalysisException(operation + " does not support
VARBINARY arguments");
}
}
}
- private static boolean containsVarbinary(DataType type) {
- if (type instanceof ArrayType) {
- return containsVarbinary(((ArrayType) type).getItemType());
- }
- if (type instanceof StructType) {
- return ((StructType) type).getFields().stream()
- .anyMatch(field -> containsVarbinary(field.getDataType()));
- }
- if (type instanceof MapType) {
- return containsVarbinary(((MapType) type).getKeyType())
- || containsVarbinary(((MapType) type).getValueType());
- }
- return type.isVarBinaryType();
- }
-
private static Optional<DataType> findCommonBinaryType(DataType left,
DataType right) {
if (left.isVarBinaryType() && right.isVarBinaryType()) {
return Optional.of(VarBinaryType.createVarBinaryType(
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/FoldConstantRuleOnBETest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/FoldConstantRuleOnBETest.java
index 28037322ecc..3c88bec9250 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/FoldConstantRuleOnBETest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/FoldConstantRuleOnBETest.java
@@ -23,6 +23,7 @@ import
org.apache.doris.nereids.rules.expression.rules.FoldConstantRuleOnBE;
import org.apache.doris.nereids.rules.expression.rules.FoldConstantRuleOnFE;
import org.apache.doris.nereids.trees.expressions.Add;
import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.Cast;
import org.apache.doris.nereids.trees.expressions.EqualTo;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.IsNull;
@@ -32,8 +33,12 @@ import
org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
import org.apache.doris.nereids.trees.expressions.literal.Literal;
import org.apache.doris.nereids.trees.expressions.literal.MapLiteral;
import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.StringLiteral;
import org.apache.doris.nereids.trees.expressions.literal.StructLiteral;
import org.apache.doris.nereids.trees.expressions.literal.UuidLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.VarBinaryLiteral;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.StringType;
import org.apache.doris.nereids.types.UuidType;
import org.apache.doris.thrift.TExpr;
@@ -88,6 +93,23 @@ class FoldConstantRuleOnBETest extends
ExpressionRewriteTestHelper {
collectConstants(new And(new IsNull(uuidResult), supported)));
}
+ @Test
+ void testSkipCastFromVarBinary() {
+ // FE leaves these casts for BE because no FE string literal carries
0xFF. BE folding would build such a
+ // literal from its result anyway, decoding 0xFF as U+FFFD, so they
wait for BE to evaluate them.
+ VarBinaryLiteral notUtf8 = new VarBinaryLiteral(new byte[] {(byte)
0xFF});
+ List<Expression> casts = ImmutableList.of(new Cast(notUtf8,
StringType.INSTANCE),
+ new Cast(new ArrayLiteral(ImmutableList.of(notUtf8)),
ArrayType.of(StringType.INSTANCE)));
+ for (Expression cast : casts) {
+ Assertions.assertEquals(cast,
FoldConstantRuleOnFE.VISITOR_INSTANCE.rewrite(cast, context));
+ Assertions.assertTrue(collectConstants(cast).isEmpty(),
cast.toSql());
+ }
+ Expression filter = new EqualTo(casts.get(0), new
StringLiteral(String.valueOf((char) 0xFFFD)));
+ Expression supported = new EqualTo(new Add(new IntegerLiteral(1), new
IntegerLiteral(2)),
+ new IntegerLiteral(3));
+ Assertions.assertEquals(ImmutableList.of(supported),
collectConstants(new And(filter, supported)));
+ }
+
private List<Expression> collectConstants(Expression expression) {
Map<String, Expression> constants = new HashMap<>();
Map<String, TExpr> serialized = new HashMap<>();
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/FoldConstantTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/FoldConstantTest.java
index ea3f7ace982..652888628a4 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/FoldConstantTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/FoldConstantTest.java
@@ -123,6 +123,7 @@ import
org.apache.doris.nereids.trees.expressions.literal.StringLiteral;
import org.apache.doris.nereids.trees.expressions.literal.TimeStampNsLiteral;
import org.apache.doris.nereids.trees.expressions.literal.TimeV2Literal;
import org.apache.doris.nereids.trees.expressions.literal.TinyIntLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.VarBinaryLiteral;
import org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral;
import org.apache.doris.nereids.trees.plans.RelationId;
import org.apache.doris.nereids.types.BooleanType;
@@ -132,6 +133,7 @@ import org.apache.doris.nereids.types.DecimalV3Type;
import org.apache.doris.nereids.types.DoubleType;
import org.apache.doris.nereids.types.FloatType;
import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.types.StringType;
import org.apache.doris.nereids.types.TinyIntType;
import org.apache.doris.nereids.types.VarcharType;
import org.apache.doris.nereids.types.VariantType;
@@ -144,6 +146,7 @@ import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
import java.time.LocalDateTime;
class FoldConstantTest extends ExpressionRewriteTestHelper {
@@ -318,6 +321,38 @@ class FoldConstantTest extends ExpressionRewriteTestHelper
{
Assertions.assertEquals(new DoubleLiteral(20240229123456D), rewritten);
}
+ @Test
+ void testCastVarBinaryToStringFold() {
+ executor = new ExpressionRuleExecutor(ImmutableList.of(
+ bottomUp(FoldConstantRuleOnFE.VISITOR_INSTANCE)
+ ));
+
+ // BE keeps the bytes of a VARBINARY cast to a string type, so FE
folds to those bytes, not to
+ // their hex text '616263'.
+ VarBinaryLiteral abc = new
VarBinaryLiteral("abc".getBytes(StandardCharsets.UTF_8));
+ Assertions.assertEquals(new StringLiteral("abc"),
+ executor.rewrite(new Cast(abc, StringType.INSTANCE), context));
+ Assertions.assertEquals(new VarcharLiteral("abc", 16),
+ executor.rewrite(new Cast(abc,
VarcharType.createVarcharType(16)), context));
+ VarBinaryLiteral multiByte = new
VarBinaryLiteral("中文".getBytes(StandardCharsets.UTF_8));
+ Assertions.assertEquals(new StringLiteral("中文"),
+ executor.rewrite(new Cast(multiByte, StringType.INSTANCE),
context));
+
+ // 0xFF is not valid UTF-8, so no FE string literal carries it; the
cast is left for BE.
+ Cast notUtf8 = new Cast(new VarBinaryLiteral(new byte[] {(byte)
0xFF}), StringType.INSTANCE);
+ Assertions.assertEquals(notUtf8, executor.rewrite(notUtf8, context));
+
+ // FE does not evaluate a NUL byte or a supplementary character like
BE: a string comparison takes the
+ // bytes after a NUL for padding, and VARCHAR(n) truncation splits
U+1F600 into a lone surrogate. Those
+ // casts are left for BE as well.
+ Cast withNul = new Cast(new VarBinaryLiteral(new byte[] {'a', 0,
'b'}), StringType.INSTANCE);
+ Assertions.assertEquals(withNul, executor.rewrite(withNul, context));
+ VarBinaryLiteral grinningFace = new VarBinaryLiteral(
+ new byte[] {(byte) 0xF0, (byte) 0x9F, (byte) 0x98, (byte)
0x80});
+ Cast supplementary = new Cast(grinningFace,
VarcharType.createVarcharType(1));
+ Assertions.assertEquals(supplementary, executor.rewrite(supplementary,
context));
+ }
+
@Test
void testFoldDate() {
executor = new ExpressionRuleExecutor(ImmutableList.of(
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/types/VarBinarySqlSupportTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/types/VarBinarySqlSupportTest.java
index ca5c1475178..a737be6aa81 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/types/VarBinarySqlSupportTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/types/VarBinarySqlSupportTest.java
@@ -18,15 +18,34 @@
package org.apache.doris.nereids.types;
import org.apache.doris.catalog.Env;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.hint.DistributeHint;
+import org.apache.doris.nereids.rules.analysis.CheckAfterRewrite;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Cast;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.MarkJoinSlotReference;
+import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.literal.VarBinaryLiteral;
+import org.apache.doris.nereids.trees.plans.DistributeType;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.RelationId;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalOneRowRelation;
import org.apache.doris.nereids.util.PlanChecker;
import org.apache.doris.qe.GlobalVariable;
import org.apache.doris.thrift.TUniqueId;
import org.apache.doris.utframe.TestWithFeService;
+import com.google.common.collect.ImmutableList;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.util.List;
+import java.util.Optional;
+
public class VarBinarySqlSupportTest extends TestWithFeService {
@Test
public void testNativeVarbinaryRemainsUnsupported() {
@@ -148,6 +167,41 @@ public class VarBinarySqlSupportTest extends
TestWithFeService {
}
}
+ @Test
+ public void testJoinConditionComparingStringsCastFromBinary() {
+ // FE leaves CAST(X'FF' AS STRING) for BE because no FE string literal
carries 0xFF, so a nullable IN
+ // subquery keeps the cast in a mark join condition, which compares
two strings.
+ PlanChecker.from(connectContext)
+ .analyze("select cast(X'FF' as string) in (select encoded from
source_bytes)").rewrite();
+ PlanChecker.from(connectContext)
+ .analyze("select cast(X'610062' as string) not in (select
encoded from source_bytes)").rewrite();
+
+ // CheckAfterRewrite rejects a join condition only when it compares
VARBINARY values.
+ Plan left = new LogicalOneRowRelation(new RelationId(1),
+ ImmutableList.of(new Alias(new VarBinaryLiteral(new byte[]
{(byte) 0xFF}), "l")));
+ Plan right = new LogicalOneRowRelation(new RelationId(2),
+ ImmutableList.of(new Alias(new VarBinaryLiteral(new byte[]
{(byte) 0xFF}), "r")));
+ Slot l = left.getOutput().get(0);
+ Slot r = right.getOutput().get(0);
+ Expression strings = new EqualTo(new Cast(l, StringType.INSTANCE), new
Cast(r, StringType.INSTANCE));
+ for (boolean mark : new boolean[] {false, true}) {
+ PlanChecker.from(connectContext, join(left, right, strings,
mark)).applyBottomUp(new CheckAfterRewrite());
+ AnalysisException error =
Assertions.assertThrows(AnalysisException.class,
+ () -> PlanChecker.from(connectContext, join(left, right,
new EqualTo(l, r), mark))
+ .applyBottomUp(new CheckAfterRewrite()));
+ Assertions.assertTrue(error.getMessage().contains("varbinary type
could not in join equal conditions"),
+ error.getMessage());
+ }
+ }
+
+ private static Plan join(Plan left, Plan right, Expression condition,
boolean mark) {
+ List<Expression> conditions = ImmutableList.of(condition);
+ return new LogicalJoin<>(mark ? JoinType.LEFT_SEMI_JOIN :
JoinType.INNER_JOIN,
+ mark ? ImmutableList.of() : conditions, ImmutableList.of(),
mark ? conditions : ImmutableList.of(),
+ new DistributeHint(DistributeType.NONE),
+ mark ? Optional.of(new MarkJoinSlotReference("mark")) :
Optional.empty(), left, right, null);
+ }
+
@Test
public void testView() throws Exception {
Assertions.assertEquals("X'0080FF'", new
VarBinaryLiteral("0080FF").toSql());
diff --git
a/regression-test/data/query_p0/sql_functions/binary_functions/test_binary_hex_function.out
b/regression-test/data/query_p0/sql_functions/binary_functions/test_binary_hex_function.out
index c5c5ff465dd..ef81a03d91c 100644
---
a/regression-test/data/query_p0/sql_functions/binary_functions/test_binary_hex_function.out
+++
b/regression-test/data/query_p0/sql_functions/binary_functions/test_binary_hex_function.out
@@ -104,7 +104,7 @@
9 aaabbbcccdddeeefff aaabbbcccdddeeefff
-- !sql_25 --
-61 5A 616263647879
+a Z abcdxy
-- !sql_26 --
0x6162 0x48656C6C6F2C20215E25 0x446F726973
diff --git
a/regression-test/data/query_p0/sql_functions/binary_functions/test_varbinary_cast_fold.out
b/regression-test/data/query_p0/sql_functions/binary_functions/test_varbinary_cast_fold.out
new file mode 100644
index 00000000000..b271c270b4e
--- /dev/null
+++
b/regression-test/data/query_p0/sql_functions/binary_functions/test_varbinary_cast_fold.out
@@ -0,0 +1,78 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !data --
+1 61 1
+2 610062 3
+3 FF 1
+4 EFBFBD 3
+5 \N \N
+6 F09F9880 4
+
+-- !hex_fold --
+616263 E4B8AD FF
+
+-- !base64_fold --
+YWJj /w==
+
+-- !cast_fold --
+abc 3 中 1
+
+-- !edge_fold --
+false F09F9880 F09F9880
+
+-- !filter_fold --
+array 3
+emoji 6
+emoji_varchar1 6
+ff 3
+nul 2
+utf8 1
+
+-- !in_fold --
+true true false false
+
+-- !hex_fold_by_be --
+616263 E4B8AD FF
+
+-- !base64_fold_by_be --
+YWJj /w==
+
+-- !cast_fold_by_be --
+abc 3 中 1
+
+-- !edge_fold_by_be --
+false F09F9880 F09F9880
+
+-- !filter_fold_by_be --
+array 3
+emoji 6
+emoji_varchar1 6
+ff 3
+nul 2
+utf8 1
+
+-- !in_fold_by_be --
+true true false false
+
+-- !hex_no_fold --
+616263 E4B8AD FF
+
+-- !base64_no_fold --
+YWJj /w==
+
+-- !cast_no_fold --
+abc 3 中 1
+
+-- !edge_no_fold --
+false F09F9880 F09F9880
+
+-- !filter_no_fold --
+array 3
+emoji 6
+emoji_varchar1 6
+ff 3
+nul 2
+utf8 1
+
+-- !in_no_fold --
+true true false false
+
diff --git
a/regression-test/suites/query_p0/sql_functions/binary_functions/test_varbinary_cast_fold.groovy
b/regression-test/suites/query_p0/sql_functions/binary_functions/test_varbinary_cast_fold.groovy
new file mode 100644
index 00000000000..9aba58205d4
--- /dev/null
+++
b/regression-test/suites/query_p0/sql_functions/binary_functions/test_varbinary_cast_fold.groovy
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// A cast of a VARBINARY value to a string type keeps its bytes, so hex(),
to_base64() and length() of a
+// VARBINARY constant, which take the value through that cast, must give the
same results whether FE folds
+// the constant (fold), FE and BE fold it (fold_by_be) or BE evaluates it
(no_fold); the results of a block
+// must match in the three modes. X'FF' is not valid UTF-8, so FE leaves its
cast to BE. So does text with a
+// NUL byte or a supplementary character, which FE does not evaluate like BE
(edge): 'a', NUL, 'b' is not equal
+// to 'a', and VARCHAR(1) and trim_in keep U+1F600 (F09F9880) whole. Such a
cast also stays in a scan filter,
+// whose conjuncts the condition cache digests (filter), and in the mark join
of a nullable IN subquery (in).
+// Each mode puts its name in a column alias, so that it is not served the
cached result of another mode.
+
+suite("test_varbinary_cast_fold") {
+ sql "drop table if exists test_varbinary_cast_fold_t"
+ sql """
+ CREATE TABLE test_varbinary_cast_fold_t (k int, s string NULL)
+ DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES
('replication_num' = '1')
+ """
+ // BE evaluates the casts, so the rows hold the bytes: 'a', 'a' NUL 'b',
0xFF, U+FFFD, NULL and U+1F600.
+ sql "set enable_fold_constant_by_be = false"
+ sql "set debug_skip_fold_constant = true"
+ sql """
+ INSERT INTO test_varbinary_cast_fold_t VALUES (1, 'a'), (2,
cast(X'610062' AS string)),
+ (3, cast(X'FF' AS string)), (4, cast(X'EFBFBD' AS string)), (5,
NULL), (6, cast(X'F09F9880' AS string))
+ """
+ order_qt_data "SELECT k, hex(s), length(s) FROM test_varbinary_cast_fold_t"
+
+ sql "set enable_condition_cache = true"
+ def modes = [
+ fold : [false, false],
+ fold_by_be: [true, false],
+ no_fold : [false, true]]
+ modes.each { mode, settings ->
+ sql "set enable_fold_constant_by_be = ${settings[0]}"
+ sql "set debug_skip_fold_constant = ${settings[1]}"
+ "order_qt_hex_${mode}" """
+ SELECT hex(cast('abc' AS varbinary(16))) AS hex_${mode},
hex(X'E4B8AD'), hex(X'FF'), hex(X'')
+ """
+ "order_qt_base64_${mode}" """
+ SELECT to_base64(cast('abc' AS varbinary(16))) AS base64_${mode},
to_base64(X'FF')
+ """
+ "order_qt_cast_${mode}" """
+ SELECT cast(cast('abc' AS varbinary(16)) AS string) AS
cast_${mode},
+ length(cast(cast('abc' AS varbinary(16)) AS varchar(16))),
+ cast(X'E4B8AD' AS string),
+ length(cast(X'FF' AS string))
+ """
+ "order_qt_edge_${mode}" """
+ SELECT cast(X'610062' AS string) = 'a' AS edge_${mode},
+ hex(cast(X'F09F9880' AS varchar(1))),
+ hex(trim_in(cast(X'F09F9880' AS string), cast(X'F09F9881'
AS string)))
+ """
+ "order_qt_filter_${mode}" """
+ SELECT 'ff' AS filter_${mode}, k FROM test_varbinary_cast_fold_t
WHERE s = cast(X'FF' AS string)
+ UNION ALL
+ SELECT 'nul', k FROM test_varbinary_cast_fold_t WHERE s =
cast(X'610062' AS string)
+ UNION ALL
+ SELECT 'emoji', k FROM test_varbinary_cast_fold_t WHERE s =
cast(X'F09F9880' AS string)
+ UNION ALL
+ SELECT 'emoji_varchar1', k FROM test_varbinary_cast_fold_t
+ WHERE s = cast(cast(X'F09F9880' AS string) AS varchar(1))
+ UNION ALL
+ SELECT 'array', k FROM test_varbinary_cast_fold_t WHERE
array_contains(cast([X'FF'] AS array<string>), s)
+ UNION ALL
+ SELECT 'utf8', k FROM test_varbinary_cast_fold_t WHERE s =
cast(X'61' AS string)
+ """
+ "order_qt_in_${mode}" """
+ SELECT cast(X'610062' AS string) IN (SELECT s FROM
test_varbinary_cast_fold_t) AS in_${mode},
+ cast(X'FF' AS string) IN (SELECT s FROM
test_varbinary_cast_fold_t WHERE k <> 4),
+ cast(X'F09F9880' AS string) NOT IN (SELECT s FROM
test_varbinary_cast_fold_t WHERE s IS NOT NULL),
+ cast(X'616263' AS string) IN (SELECT s FROM
test_varbinary_cast_fold_t WHERE s IS NOT NULL)
+ """
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
