[
https://issues.apache.org/jira/browse/IMPALA-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18080769#comment-18080769
]
ASF subversion and git services commented on IMPALA-14970:
----------------------------------------------------------
Commit 5bc20e859709e44038d8c1f02c2c2b7b5705b977 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=5bc20e859 ]
IMPALA-14970: Time-travel queries can throw UnsupportedOperationException
Time-travel queries could throw UnsupportedOperationException when
they found files replicated on data nodes that were not in the host
index of the table. It's because ListMap's populate() method did not
copy the given list, but wrapped the original list via
Collections.synchronizedList(). When the given list was immutable
then we got the above error when we wanted to extend the host
index (during time-travel).
This patch fixes ListMap's populate() method to copy the given list.
Testing
* unit tests added for ListMap
* e2e tests added for Iceberg tables + time-travel
Change-Id: I3773eb7a37e9918501bfa8a22707967e79024aca
Reviewed-on: http://gerrit.cloudera.org:8080/24294
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Time-travel queries can throw UnsupportedOperationException
> -----------------------------------------------------------
>
> Key: IMPALA-14970
> URL: https://issues.apache.org/jira/browse/IMPALA-14970
> Project: IMPALA
> Issue Type: Bug
> Components: Backend, Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> Time-travel queries can throw UnsupportedOperationException, e.g.:
> {noformat}
> Caused by: java.lang.UnsupportedOperationException
> at
> com.google.common.collect.ImmutableCollection.add(ImmutableCollection.java:266)
> at
> java.base/java.util.Collections$SynchronizedCollection.add(Collections.java:2104)
> at org.apache.impala.util.ListMap.getOrAddIndex(ListMap.java:65)
> at
> org.apache.impala.catalog.FileBlock.createFbFileBlock(FileBlock.java:72)
> at
> org.apache.impala.catalog.FileDescriptor.create(FileDescriptor.java:152)
> at
> org.apache.impala.catalog.FeIcebergTable$Utils.getHdfsFileDescriptor(FeIcebergTable.java:1004)
> at
> org.apache.impala.planner.IcebergScanPlanner.getFileDescriptor(IcebergScanPlanner.java:935)
> {noformat}
> Reproduction needs an HDFS/Ozone cluster with at least R+1 data nodes where
> R=Replication factor.
> {code:java}
> drop table ice_t if exists;
> create table ice_t (i int, p int) partitioned by spec(p)
> stored as iceberg;
> insert into ice_t values (1,1);
> insert into ice_t values (2,1);
> insert into ice_t values (3,1);
> optimize table ice_t;
> show files in ice_t;
> refresh ice_t;
> describe history ice_t;
> select * from ice_t for system_version as of <earlier snapshot>;
> {code}
> The problem is that in IcebergScanPlanner.getFileDescriptor() we use
> getIceTable().getHostIndex() during time-travel which is an immutable object.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
