[
https://issues.apache.org/jira/browse/FLINK-31197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17930538#comment-17930538
]
Brett Randall commented on FLINK-31197:
---------------------------------------
I think this is duplicated by FLINK-33759 - please review and close if that's
the case.
> Unable to write Parquet files containing Arrays with complex types.
> -------------------------------------------------------------------
>
> Key: FLINK-31197
> URL: https://issues.apache.org/jira/browse/FLINK-31197
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
> Affects Versions: 1.15.0, 1.15.1, 1.16.0, 1.17.0, 1.15.2, 1.15.3, 1.16.1,
> 1.15.4, 1.16.2, 1.17.1, 1.15.5
> Reporter: Krzysztof Chmielewski
> Priority: Major
> Attachments: ParquetSinkArrayOfArraysIssue.java
>
>
> After https://issues.apache.org/jira/browse/FLINK-17782 It should be possible
> to write complex types with File sink using Parquet format.
> However it turns out that still it is impossible to write types such as:
> * Array<Arrays>
> * Array<Map>
> * Array<Row>
> When trying to write a Parquet row with such types, the below exception is
> thrown:
> {code:java}
> Caused by: java.lang.RuntimeException:
> org.apache.parquet.io.ParquetEncodingException: empty fields are illegal, the
> field should be ommited completely instead
> at
> org.apache.flink.formats.parquet.row.ParquetRowDataBuilder$ParquetWriteSupport.write(ParquetRowDataBuilder.java:91)
> at
> org.apache.flink.formats.parquet.row.ParquetRowDataBuilder$ParquetWriteSupport.write(ParquetRowDataBuilder.java:71)
> at
> org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:138)
> at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:310)
> at
> org.apache.flink.formats.parquet.ParquetBulkWriter.addElement(ParquetBulkWriter.java:52)
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.BulkPartWriter.write(BulkPartWriter.java:51)
> at
> org.apache.flink.connector.file.sink.writer.FileWriterBucket.write(FileWriterBucket.java:191)
> {code}
> The exception is misleading, not showing the real problem.
> The reason why those complex types are still not working is that during
> developemnt of https://issues.apache.org/jira/browse/FLINK-17782
> code paths for those types were left without implementation, no Unsupported
> Exception no nothing, simply empty methods. In
> https://github.com/apache/flink/blob/release-1.16.1/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/row/ParquetRowDataWriter.java
> You will see
> {code:java}
> @Override
> public void write(ArrayData arrayData, int ordinal) {}
> {code}
> for MapWriter, ArrayWriter and RowWriter.
> I see two problems here:
> 1. Writing those three types is still not possible.
> 2. Flink is throwing an exception that gives no hint about the real issue
> here. It could throw "Unsupported operation" for now. Maybe this should be
> item for a different ticket?
> The code to reproduce this issue is attached to the ticket. It tries to write
> to Parquet file a single row with one column of type Array<Array<int>>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
