[
https://issues.apache.org/jira/browse/IMPALA-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17954622#comment-17954622
]
Quanlong Huang commented on IMPALA-12543:
-----------------------------------------
Saw a correctness issue due to this bug in a downstream branch without the fix.
The scenario is Impala sets a tblproperty and then Hive updates it to a new
value:
{code:sql}
Impala> alter table scale_40k_500cols_db.tbl15 set
tblproperties('modifier'='impala');
Hive> alter table scale_40k_500cols_db.tbl15 set
tblproperties('modifier'='hive');{code}
self-event detection of the ALTER_TABLE event 179097 happens before the catalog
version 771 was added to the in-flight list:
{noformat}
I0528 07:28:09.297712 3698085 JniUtil.java:166]
ff4fff3d81537b2d:b062414900000000] execDdl request: ALTER_TABLE
scale_40k_500cols_db.tbl15 issued by root
I0528 07:28:09.330451 3698085 HdfsTable.java:1283]
ff4fff3d81537b2d:b062414900000000] Reloading metadata for table definition and
all partition(s) of scale_40k_500cols_db.tbl15 (ALTER TABLE SET_TBL_PROPERTIES)
I0528 07:28:09.485790 3698085 HdfsTable.java:1323]
ff4fff3d81537b2d:b062414900000000] Incrementally loaded table metadata for:
scale_40k_500cols_db.tbl15
I0528 07:28:09.486869 3696173 CatalogServiceCatalog.java:1210] Could not find
version 771 in in-flight event list of table scale_40k_500cols_db.tbl15
I0528 07:28:09.487545 3696173 MetastoreEvents.java:822] EventId: 179097
EventType: ALTER_TABLE Skipping reloading of file metadata for table
scale_40k_500cols_db.tbl15 since SD and whitelistedTblProperties has not
changed.
I0528 07:28:09.487619 3696173 CatalogServiceCatalog.java:2670] Refreshing table
metadata: scale_40k_500cols_db.tbl15
I0528 07:28:09.492390 3698085 CatalogServiceCatalog.java:1258]
ff4fff3d81537b2d:b062414900000000] Added catalog version 771 in table's
scale_40k_500cols_db.tbl15 in-flight events
I0528 07:28:09.492594 3698085 JniUtil.java:177]
ff4fff3d81537b2d:b062414900000000] Finished execDdl request: ALTER_TABLE
scale_40k_500cols_db.tbl15 issued by root. Time spent: 195ms{noformat}
Then the next ALTER_TABLE from the Hive command is incorrectly treated as
self-event and skipped:
{noformat}
I0528 07:28:13.092660 3696173 MetastoreEvents.java:822] EventId: 179099
EventType: ALTER_TABLE Incremented events skipped counter to 11366
I0528 07:28:13.092684 3696173 MetastoreEvents.java:822] EventId: 179099
EventType: ALTER_TABLE Not processing the event as it is a self-event{noformat}
The tblproperty is unsynced with Hive until the table is reloaded in the future.
> test_iceberg_self_events failed in JDK11 build
> ----------------------------------------------
>
> Key: IMPALA-12543
> URL: https://issues.apache.org/jira/browse/IMPALA-12543
> Project: IMPALA
> Issue Type: Bug
> Reporter: Riza Suminto
> Assignee: Riza Suminto
> Priority: Major
> Labels: broken-build
> Fix For: Impala 4.4.0
>
> Attachments: catalogd.INFO, std_err.txt
>
>
> test_iceberg_self_events failed in JDK11 build with following error.
>
> {code:java}
> Error Message
> assert 0 == 1
> Stacktrace
> custom_cluster/test_events_custom_configs.py:637: in test_iceberg_self_events
> check_self_events("ALTER TABLE {0} ADD COLUMN j INT".format(tbl_name))
> custom_cluster/test_events_custom_configs.py:624: in check_self_events
> assert tbls_refreshed_before == tbls_refreshed_after
> E assert 0 == 1 {code}
> This test still pass before IMPALA-11387 merged.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
