[
https://issues.apache.org/jira/browse/IMPALA-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115776#comment-18115776
]
ASF subversion and git services commented on IMPALA-14536:
----------------------------------------------------------
Commit f89b30507ab9ef5c85ba7ce4cf1f8d7f3f941536 in impala's branch
refs/heads/master from Noemi Pap-Takacs
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=f89b30507 ]
IMPALA-15222: Short-circuit OPTIMIZE when the table is compact
OPTIMIZE TABLE compacts the Iceberg tables by rewriting small files.
When given a file size threshold, it can automatically detect when
the table is already in a compact form. It flags the case with NO_OP
mode internally and does not create a new commit.
However, the executors still got the QueryExecRequest to scan the
table and rewrote the files, just silently left the newly written
files on the file system as orphan files.
This patch fixes this issue building on the NO_OP request handling
logic introduced by IMPALA-14536. When Impala detects that no
compaction is needed based on the given parameters, it converts the
Optimize statement to TStmtType.NO_OP and returns without creating
an execution plan with the following analysis result:
"No files selected for optimization."
This short-circuiting is enabled when
- the table is empty,
- there is a FILE_SIZE_THRESHOLD_MB set and there is at most 1
file per partition that fits the rewrite criteria. This means
compaction would not reduce file count.
Note that full-table OPTIMIZE always rewrites the entire table if
it contains any files, regardless of file size and file count per
partition to enable rewriting the table to the latest partition spec.
Testing:
- added e2e test for empty table and single file cases
- added Analysis and Planner test for table with 1 file.
Assisted-by: Claude Opus 4.8 (Claude Code)
Change-Id: I795967ca18231ef1fefdbc71a9b4505905906f95
Reviewed-on: http://gerrit.cloudera.org:8080/24633
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> ALTER TABLE CONVERT TO ICEBERG should be idempotent
> ---------------------------------------------------
>
> Key: IMPALA-14536
> URL: https://issues.apache.org/jira/browse/IMPALA-14536
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Frontend
> Affects Versions: Impala 4.3.0
> Reporter: Michael Smith
> Assignee: Dániel Gábor Vankó
> Priority: Major
> Labels: impala-iceberg, newbie, ramp-up
> Fix For: Impala 5.0.0
>
>
> Running {{ALTER TABLE <table> CONVERT TO ICEBERG}} on an Iceberg table should
> do nothing rather than produce the error
> {code}
> AnalysisException: CONVERT TO ICEBERG is not supported for
> org.apache.iceberg.mr.hive.HiveIcebergInputFormat
> {code}
> I run into this sometimes re-running functional-query dataload, because it
> creates a table and converts it to iceberg. Most of the time re-running
> dataload will just re-run, but in this case it errors because the table has
> already been converted to iceberg.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
