[
https://issues.apache.org/jira/browse/IMPALA-14437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18020492#comment-18020492
]
ASF subversion and git services commented on IMPALA-14437:
----------------------------------------------------------
Commit 68ab52f2c770c233bc1e287b6d3c40df1cdc8775 in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=68ab52f2c ]
IMPALA-14437: Fix regression in FileMetadataLoader.createFd()
IMPALA-14349 caused a regression due to change in
FileMetadataLoader.createFd(). When default FS is S3, all files is S3
should not have any FileBlock. However, after IMPALA-14349, CTAS query
that scans functional.alltypes table in S3 hit following Preconditions
in HdfsScanNode.java:
if (!fsHasBlocks) {
Preconditions.checkState(fileDesc.getNumFileBlocks() == 0);
This is because FileMetadataLoader.createFd() skip checking if the
originating FileSystem support supportsStorageIds() or not. S3
dataloading from HDFS snapshot consistently failed due this regression.
This patch fix the issue by restoring FileMetadataLoader.createFd() to
its state before IMPALA-14349. It also make
FileMetadataLoader.createFd() calls more consistent by not allowing null
parameters except for 'absPath' that is only not null for Iceberg data
files. Generalize numUnknownDiskIds parameter from Reference<Long> to
AtomicLong for parallel usage.
Testing:
Pass dataloading, FE_TEST, EE_TEST, and CLUSTER_TEST in S3.
Change-Id: Ie16c5d7b020a59b5937b52dfbf66175ac94f60cd
Reviewed-on: http://gerrit.cloudera.org:8080/23423
Reviewed-by: Zoltan Borok-Nagy <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> FileMetadataLoader.createFd() regress after IMPALA-14349
> --------------------------------------------------------
>
> Key: IMPALA-14437
> URL: https://issues.apache.org/jira/browse/IMPALA-14437
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 5.0.0
> Reporter: Riza Suminto
> Assignee: Riza Suminto
> Priority: Major
> Labels: broken-build
> Fix For: Impala 5.0.0
>
>
> IMPALA-14349 caused a regression due to change in
> FileMetadataLoader.createFd(). When default FS is S3, all files is S3 should
> not have any FileBlock. However, after IMPALA-14349, CTAS query that scans
> functional.alltypes table in S3 hit following Preconditions in
> HdfsScanNode.java:
> {code:java}
> if (!fsHasBlocks) {
> Preconditions.checkState(fileDesc.getNumFileBlocks() == 0);{code}
> Following is the snippet from the log.
> {code:java}
> I20250910 07:36:52.709887 63842 Frontend.java:2431]
> 8a47939d765631e1:4821604e00000000] Analyzing query: INSERT into TABLE
> functional_kudu.alltypes
> SELECT id, bool_col, tinyint_col, smallint_col, int_col, bigint_col,
> float_col, double_col, date_string_col, string_col,
> timestamp_col, year, month
> FROM functional.alltypes db: default...I20250910 07:36:52.956176 63842
> jni-util.cc:321] 8a47939d765631e1:4821604e00000000]
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:496)
> at
> org.apache.impala.planner.HdfsScanNode.computeScanRangeLocations(HdfsScanNode.java:1364)
> at org.apache.impala.planner.HdfsScanNode.init(HdfsScanNode.java:444)
> at
> org.apache.impala.planner.SingleNodePlanner.createHdfsScanPlan(SingleNodePlanner.java:1684)
> at
> org.apache.impala.planner.SingleNodePlanner.createScanNode(SingleNodePlanner.java:1906)
> at
> org.apache.impala.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:2235)
> at
> org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(SingleNodePlanner.java:937)
> at
> org.apache.impala.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:752)
> at
> org.apache.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:284)
> at
> org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:176)
> at
> org.apache.impala.planner.Planner.createPlanFragments(Planner.java:137)
> at org.apache.impala.planner.Planner.createPlans(Planner.java:312)
> at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2031)
> at
> org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:3356)
> at
> org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:3130)
> at
> org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:2538)
> at
> org.apache.impala.service.Frontend.getTExecRequestWithFallback(Frontend.java:2407)
> at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2096)
> at
> org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:176)
> I20250910 07:36:52.956212 63842 status.cc:129]
> 8a47939d765631e1:4821604e00000000] IllegalStateException: null {code}
> This is because, after IMPALA-14349, FileMetadataLoader.createFd() skip
> checking if the originating FileSystem support supportsStorageIds() or not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
