[
https://issues.apache.org/jira/browse/IMPALA-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang resolved IMPALA-13974.
-------------------------------------
Fix Version/s: Impala 5.0.0
Resolution: Fixed
> test_hms_event_sync_with_commit_compaction is flaky
> ---------------------------------------------------
>
> Key: IMPALA-13974
> URL: https://issues.apache.org/jira/browse/IMPALA-13974
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
> Fix For: Impala 5.0.0
>
> Attachments:
> catalogd.ip-172-31-61-143.ubuntu.log.INFO.20250417-182636.2662154,
> impalad.ip-172-31-61-143.ubuntu.log.INFO.20250417-182636.2662182
>
>
> Saw this failed in several builds:
> {code}
> custom_cluster.test_events_custom_configs.TestEventSyncWaiting.test_hms_event_sync_with_commit_compaction[protocol:
> beeswax | table_format: text/none | exec_option: {'test_replan': 1,
> 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> 'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0}] (from pytest){code}
> Stacktrace
> {code:python}
> custom_cluster/test_events_custom_configs.py:1639: in
> test_hms_event_sync_with_commit_compaction
> assert len(res.data) == 1
> E assert 2 == 1
> E + where 2 =
> len(['hdfs://localhost:20500/test-warehouse/managed/test_hms_event_sync_with_commit_compaction_1ac0b73.db/foo/p=0/delta_00...aged/test_hms_event_sync_with_commit_compaction_1ac0b73.db/foo/p=0/delta_0000002_0000002_0000/000000_0\t2B\tp=0\tNONE'])
> E + where
> ['hdfs://localhost:20500/test-warehouse/managed/test_hms_event_sync_with_commit_compaction_1ac0b73.db/foo/p=0/delta_00...aged/test_hms_event_sync_with_commit_compaction_1ac0b73.db/foo/p=0/delta_0000002_0000002_0000/000000_0\t2B\tp=0\tNONE']
> = <tests.beeswax.impala_beeswax.ImpalaBeeswaxResult object at
> 0x7f9113257e50>.data{code}
> Standard Error
> {noformat}
> -- 2025-04-17 18:26:35,817 INFO MainThread: Starting cluster with
> command: /home/ubuntu/Impala/bin/start-impala-cluster.py
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3
> --log_dir=/home/ubuntu/Impala/logs/custom_cluster_tests --log_level=1
> '--catalogd_args=--debug_actions=catalogd_event_processing_delay:SLEEP@200 '
> --impalad_args=--default_query_options=
> 18:26:36 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
> 18:26:36 MainThread: Starting State Store logging to
> /home/ubuntu/Impala/logs/custom_cluster_tests/statestored.INFO
> 18:26:36 MainThread: Starting Catalog Service logging to
> /home/ubuntu/Impala/logs/custom_cluster_tests/catalogd.INFO
> 18:26:36 MainThread: Starting Impala Daemon logging to
> /home/ubuntu/Impala/logs/custom_cluster_tests/impalad.INFO
> 18:26:36 MainThread: Starting Impala Daemon logging to
> /home/ubuntu/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 18:26:36 MainThread: Starting Impala Daemon logging to
> /home/ubuntu/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 18:26:38 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 18:26:38 MainThread: wait_num_table=1
> 18:26:38 MainThread: Waiting for Impalad webserver port 25000
> 18:26:38 MainThread: Waiting for Impalad webserver port 25000
> 18:26:38 MainThread: Waiting for Impalad webserver port 25001
> 18:26:38 MainThread: Waiting for Impalad webserver port 25002
> 18:26:41 MainThread: Waiting for coordinator client services - hs2 port:
> 21050 hs2-http port: 28000 beeswax port: 21000
> 18:26:42 MainThread: Waiting for coordinator client services - hs2 port:
> 21051 hs2-http port: 28001 beeswax port: 21001
> 18:26:44 MainThread: Waiting for coordinator client services - hs2 port:
> 21052 hs2-http port: 28002 beeswax port: 21002
> 18:26:44 MainThread: Getting num_known_live_backends from
> ip-172-31-61-143:25000
> 18:26:44 MainThread: num_known_live_backends has reached value: 3
> 18:26:44 MainThread: Getting num_known_live_backends from
> ip-172-31-61-143:25001
> 18:26:44 MainThread: num_known_live_backends has reached value: 3
> 18:26:44 MainThread: Getting num_known_live_backends from
> ip-172-31-61-143:25002
> 18:26:44 MainThread: num_known_live_backends has reached value: 3
> 18:26:44 MainThread: Total wait: 6.46s
> 18:26:44 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
> executors).
> -- 2025-04-17 18:26:44,955 DEBUG MainThread: Found 3 impalad/1
> statestored/1 catalogd process(es)
> -- 2025-04-17 18:26:44,955 INFO MainThread: Getting metric:
> statestore.live-backends from ip-172-31-61-143:25010
> -- 2025-04-17 18:26:44,958 INFO MainThread: Metric
> 'statestore.live-backends' has reached desired value: 4. total_wait: 0s
> -- 2025-04-17 18:26:44,958 DEBUG MainThread: Getting
> num_known_live_backends from ip-172-31-61-143:25000
> -- 2025-04-17 18:26:44,959 INFO MainThread: num_known_live_backends has
> reached value: 3
> -- 2025-04-17 18:26:44,959 DEBUG MainThread: Getting
> num_known_live_backends from ip-172-31-61-143:25001
> -- 2025-04-17 18:26:44,961 INFO MainThread: num_known_live_backends has
> reached value: 3
> -- 2025-04-17 18:26:44,961 DEBUG MainThread: Getting
> num_known_live_backends from ip-172-31-61-143:25002
> -- 2025-04-17 18:26:44,962 INFO MainThread: num_known_live_backends has
> reached value: 3
> -- 2025-04-17 18:26:44,962 INFO MainThread: beeswax:
> set
> client_identifier=custom_cluster/test_events_custom_configs.py::TestEventSyncWaiting::()::test_hms_event_sync_with_commit_compaction[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disabl;
> -- 2025-04-17 18:26:44,963 INFO MainThread: beeswax: connected to
> localhost:21000 with beeswax
> -- 2025-04-17 18:26:44,963 INFO MainThread: hs2:
> set
> client_identifier=custom_cluster/test_events_custom_configs.py::TestEventSyncWaiting::()::test_hms_event_sync_with_commit_compaction[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disabl;
> -- 2025-04-17 18:26:44,963 INFO MainThread: hs2: connected to
> localhost:21050 with impyla hs2
> -- 2025-04-17 18:26:44,963 INFO MainThread: hs2-http:
> set
> client_identifier=custom_cluster/test_events_custom_configs.py::TestEventSyncWaiting::()::test_hms_event_sync_with_commit_compaction[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disabl;
> -- 2025-04-17 18:26:44,963 INFO MainThread: hs2-http: connected to
> localhost:28000 with impyla hs2-http
> -- 2025-04-17 18:26:44,963 INFO MainThread: hs2:
> set
> client_identifier=custom_cluster/test_events_custom_configs.py::TestEventSyncWaiting::()::test_hms_event_sync_with_commit_compaction[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disabl;
> -- 2025-04-17 18:26:44,964 INFO MainThread: hs2: connected to
> localhost:21050 with impyla hs2
> -- 2025-04-17 18:26:44,964 INFO MainThread: hs2:
> set
> client_identifier=custom_cluster/test_events_custom_configs.py::TestEventSyncWaiting::()::test_hms_event_sync_with_commit_compaction[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disabl;
> -- 2025-04-17 18:26:44,964 INFO MainThread: hs2: set_configuration:
> set sync_ddl=False;
> -- 2025-04-17 18:26:44,964 INFO MainThread: hs2: executing against Impala
> at localhost:21050. session: 264cb39583af1aab:d409990d6e5744b1 main_cursor:
> True user: None
> DROP DATABASE IF EXISTS `test_hms_event_sync_with_commit_compaction_1ac0b73`
> CASCADE;
> -- 2025-04-17 18:26:45,190 INFO MainThread:
> 77491be3975d74fb:049dc1f200000000: query started
> -- 2025-04-17 18:26:45,191 INFO MainThread:
> 77491be3975d74fb:049dc1f200000000: getting log for operation
> -- 2025-04-17 18:26:45,192 INFO MainThread:
> 77491be3975d74fb:049dc1f200000000: getting runtime profile operation
> -- 2025-04-17 18:26:45,192 INFO MainThread:
> 77491be3975d74fb:049dc1f200000000: closing query for operation
> -- 2025-04-17 18:26:45,196 INFO MainThread: hs2: executing against Impala
> at localhost:21050. session: 264cb39583af1aab:d409990d6e5744b1 main_cursor:
> True user: None
> CREATE DATABASE `test_hms_event_sync_with_commit_compaction_1ac0b73`;
> -- 2025-04-17 18:26:45,361 INFO MainThread:
> ae4576375fa730dc:52b77e0a00000000: query started
> -- 2025-04-17 18:26:45,362 INFO MainThread:
> ae4576375fa730dc:52b77e0a00000000: getting log for operation
> -- 2025-04-17 18:26:45,362 INFO MainThread:
> ae4576375fa730dc:52b77e0a00000000: getting runtime profile operation
> -- 2025-04-17 18:26:45,362 INFO MainThread:
> ae4576375fa730dc:52b77e0a00000000: closing query for operation
> -- 2025-04-17 18:26:45,362 INFO MainThread: Created database
> "test_hms_event_sync_with_commit_compaction_1ac0b73" for test ID
> "custom_cluster/test_events_custom_configs.py::TestEventSyncWaiting::()::test_hms_event_sync_with_commit_compaction[protocol:
> beeswax | table_format: text/none | exec_option: {'test_replan': 1,
> 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> 'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0}]"
> -- 2025-04-17 18:26:45,362 INFO MainThread: hs2: closing 1 sync and 0
> async hs2 connections to: localhost:21050
> -- 2025-04-17 18:26:45,363 INFO MainThread: beeswax:
> set
> client_identifier=custom_cluster/test_events_custom_configs.py::TestEventSyncWaiting::()::test_hms_event_sync_with_commit_compaction[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disabl;
> -- 2025-04-17 18:26:45,363 INFO MainThread: beeswax: set_configuration:
> set sync_hms_events_wait_time_s=10;
> -- 2025-04-17 18:26:45,363 INFO MainThread: -- executing in HiveServer2
> create transactional table
> test_hms_event_sync_with_commit_compaction_1ac0b73.foo partitioned by(p)
> as select 0 as i, 0 as p
> -- 2025-04-17 18:26:54,943 INFO MainThread: -- executing in HiveServer2
> insert into test_hms_event_sync_with_commit_compaction_1ac0b73.foo select 1,0
> -- 2025-04-17 18:27:05,047 INFO MainThread: beeswax: executing against
> localhost:21000
> show files in test_hms_event_sync_with_commit_compaction_1ac0b73.foo;
> -- 2025-04-17 18:27:05,728 INFO MainThread: Started query
> 984b75d9783805e3:33f0f1b800000000
> -- 2025-04-17 18:27:05,731 INFO MainThread: -- executing in HiveServer2
> alter table test_hms_event_sync_with_commit_compaction_1ac0b73.foo
> partition(p=0) compact 'minor' and wait
> -- 2025-04-17 18:27:36,156 INFO MainThread: beeswax: executing against
> localhost:21000
> show files in test_hms_event_sync_with_commit_compaction_1ac0b73.foo;
> -- 2025-04-17 18:27:36,776 INFO MainThread: Started query
> 2b476442f82351d8:371b333b00000000{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
