Owen-CH-Leung opened a new pull request, #70656:
URL: https://github.com/apache/doris/pull/70656

   ### Background
   
   Doris reads an Iceberg BINARY column as VARBINARY and a FIXED(n) column as 
VARBINARY(n) (#68786). CREATE TABLE builds the Iceberg schema in 
IcebergSchemaBuilder.buildSchema, which had no VARBINARY case and threw 
"Unsupported type for Iceberg: VARBINARY".
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #68786 (read mapping), #69285 (MODIFY COLUMN restating BINARY 
and FIXED)
   PR Type: Feature
   
   Problem Summary: Doris cannot create an Iceberg BINARY column, so tables 
created through Doris store binary payloads as STRING. Iceberg requires STRING 
to be UTF-8, and since 4.1.4 the FE decodes STRING column bounds as UTF-8 at 
commit, so a non-UTF-8 value fails the insert with "Failed to decode value as 
UTF-8". This commit maps VARBINARY and VARBINARY(n) to BINARY when CREATE TABLE 
builds the schema, at the top level and inside ARRAY, MAP and STRUCT. Users see 
no change yet, because fe-core still rejects VARBINARY before the connector is 
called; the next commit lifts that.
   
   ### Details
   
   - **Scope:** only IcebergSchemaBuilder.buildSchema. ADD COLUMN and MODIFY 
COLUMN go through buildColumnType and still reject VARBINARY, so #69285 keeps 
resolving a restated VARBINARY against the current column.
   - **Design:** a flag on the private IdAllocator, set only by buildSchema. 
toIcebergPrimitive is untouched because ALTER also calls it. VARBINARY(n) 
becomes BINARY rather than FIXED(n). Doris treats n as a maximum and does not 
enforce it (see the comment in data_type_varbinary_serde.cpp), while Iceberg 
FIXED(n) holds exactly n bytes. With FIXED(n), every shorter value would fail 
on write, and so would a CTAS from a MySQL BLOB, which arrives as 
VARBINARY(65535), or a UNION, which takes the largest length. Iceberg also 
rejects the truncate transform on FIXED and cannot promote FIXED to BINARY or 
to a longer FIXED. #69285 gives the same reason for rejecting this mapping, and 
Paimon already maps every VARBINARY to VarBinaryType(MAX_LENGTH).
   - **Usage:** none until the next commit.
   - **Default behavior:** a column declared VARBINARY(n) reads back as 
varbinary(2147483647), as any Iceberg BINARY column does. A later MODIFY COLUMN 
can restate it as VARBINARY but not as VARBINARY(n).
   - **Compatibility:** no change to existing tables or to any statement that 
works today.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to