Quanlong Huang created IMPALA-13918:
---------------------------------------
Summary: Iceberg Table with 6M files might hit OutOfMemoryError:
Requested array size exceeds VM limit
Key: IMPALA-13918
URL: https://issues.apache.org/jira/browse/IMPALA-13918
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Quanlong Huang
TIcebergTable puts all files in TIcebergContentFileStore which might hit
OutOfMemoryError of "Requested array size exceeds VM limit" when there are more
than 6M files which requires a byte array larger than 2GB to serialize them.
For non-iceberg tables, we can divide the files into different partitions and
truncate them in partition level (IMPALA-11402). We need a similar approach for
Iceberg tables.
CC [~boroknagyz] , [~daniel.becker] , [~gaborkaszab] , [~noemi]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
