Ramin Gharib created FLINK-40840:
------------------------------------
Summary: INSERT with a column list fails when an omitted column is
of type BITMAP, RAW or STRUCTURED
Key: FLINK-40840
URL: https://issues.apache.org/jira/browse/FLINK-40840
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 2.3.0, 2.2.0, 2.1.0
Reporter: Ramin Gharib
Assignee: Ramin Gharib
An INSERT with an explicit column list fails during validation when an omitted
column has a type that only Flink knows.
{code:sql}
CREATE TABLE src (k INT) WITH ('connector' = 'datagen', 'number-of-rows' = '1');
CREATE TABLE bm_snk (k INT, b BITMAP) WITH ('connector' = 'blackhole');
CREATE TABLE st_snk (k INT, s STRUCTURED<'com.example.User', name STRING>) WITH
('connector' = 'blackhole');
INSERT INTO bm_snk (k) SELECT k FROM src;
INSERT INTO st_snk (k) SELECT k FROM src;
{code}
|| Omitted column type || Error ||
| \{{BITMAP}} | \{{Unsupported type when convertTypeToSpec: OTHER}} |
| \{{ARRAY<BITMAP>}} | \{{Unsupported type when convertTypeToSpec: OTHER}} |
| \{{RAW}}, for example \{{DataTypes.RAW(Integer.class,
IntSerializer.INSTANCE)}} | \{{Unsupported type when convertTypeToSpec: OTHER}}
|
| \{{STRUCTURED<'com.example.User', name STRING>}} | \{{Unsupported type when
convertTypeToSpec: STRUCTURED}} |
Writing the cast explicitly works, for example \{{CAST(NULL AS BITMAP)}}.
*Cause*
{\{PreValidateReWriter}} pads every omitted column with \{{CAST(NULL AS <column
type>)}}. \{{FlinkCalciteSqlValidator.maybeCast}} delegates to Calcite, which
builds the cast target with \{{SqlTypeUtil.convertTypeToSpec}}. That method
only knows Calcite types. \{{BitmapRelDataType}} and \{{RawRelDataType}} report
\{{SqlTypeName.OTHER}}, and \{{StructuredRelDataType}} reports
\{{SqlTypeName.STRUCTURED}}, so no branch matches.
FLINK-40839 fixes the same failure for VARIANT by backporting a Calcite fix.
That approach does not fit here. These types are not native to Calcite, so a
branch in Flink's copy of \{{SqlTypeUtil}} would never be removed by a Calcite
upgrade.
*Proposed fix*
Build the cast target on the Flink side, in
\{{FlinkCalciteSqlValidator.maybeCast}}, and leave the copied Calcite class
unchanged. There are two options:
# Map each Flink-only type to the type name spec the parser already has:
\{{SqlBitmapTypeNameSpec}}, \{{SqlRawTypeNameSpec}} and
\{{SqlStructuredTypeNameSpec}}. FLINK-38682 already does this for RAW in
\{{TypeInferenceOperandChecker.castTo}}, and both places could share the code.
This only covers top-level columns. \{{ARRAY<BITMAP>}} would still fail,
because \{{convertTypeToSpec}} recurses into its own code for nested types.
# Add one planner-side \{{SqlTypeNameSpec}} that wraps a Flink
\{{LogicalType}}. It derives its type through \{{FlinkTypeFactory}} and
unparses as Flink SQL, for example \{{ARRAY<BITMAP>}}. \{{maybeCast}} uses it
when the target type contains a Flink-only type. This covers nested types and
any future Flink-only type. The unparsed SQL must parse back to the same type,
because some places store SQL text, such as materialized table definition
queries.
Option 2 is preferred, because it is the only one that fixes nested types
without changing the copied Calcite class.
The workaround is to list every column and write the cast explicitly, for
example \{{CAST(NULL AS BITMAP)}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
