Jason Fehr created IMPALA-14012:
-----------------------------------
Summary: test_hms_event_sync_with_commit_compaction is flaky
Key: IMPALA-14012
URL: https://issues.apache.org/jira/browse/IMPALA-14012
Project: IMPALA
Issue Type: Bug
Reporter: Jason Fehr
Assignee: Quanlong Huang
Fix For: Impala 5.0.0
Saw this failed in several builds:
{code}
custom_cluster.test_events_custom_configs.TestEventSyncWaiting.test_hms_event_sync_with_commit_compaction[protocol:
beeswax | table_format: text/none | exec_option: {'test_replan': 1,
'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1,
'exec_single_node_rows_threshold': 0}] (from pytest){code}
Stacktrace
{code:python}
custom_cluster/test_events_custom_configs.py:1639: in
test_hms_event_sync_with_commit_compaction
assert len(res.data) == 1
E assert 2 == 1
E + where 2 =
len(['hdfs://localhost:20500/test-warehouse/managed/test_hms_event_sync_with_commit_compaction_1ac0b73.db/foo/p=0/delta_00...aged/test_hms_event_sync_with_commit_compaction_1ac0b73.db/foo/p=0/delta_0000002_0000002_0000/000000_0\t2B\tp=0\tNONE'])
E + where
['hdfs://localhost:20500/test-warehouse/managed/test_hms_event_sync_with_commit_compaction_1ac0b73.db/foo/p=0/delta_00...aged/test_hms_event_sync_with_commit_compaction_1ac0b73.db/foo/p=0/delta_0000002_0000002_0000/000000_0\t2B\tp=0\tNONE']
= <tests.beeswax.impala_beeswax.ImpalaBeeswaxResult object at
0x7f9113257e50>.data{code}
Standard Error
{noformat}
-- 2025-04-17 18:26:35,817 INFO MainThread: Starting cluster with command:
/home/ubuntu/Impala/bin/start-impala-cluster.py
'--state_store_args=--statestore_update_frequency_ms=50
--statestore_priority_update_frequency_ms=50
--statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3
--log_dir=/home/ubuntu/Impala/logs/custom_cluster_tests --log_level=1
'--catalogd_args=--debug_actions=catalogd_event_processing_delay:SLEEP@200 '
--impalad_args=--default_query_options=
18:26:36 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
18:26:36 MainThread: Starting State Store logging to
/home/ubuntu/Impala/logs/custom_cluster_tests/statestored.INFO
18:26:36 MainThread: Starting Catalog Service logging to
/home/ubuntu/Impala/logs/custom_cluster_tests/catalogd.INFO
18:26:36 MainThread: Starting Impala Daemon logging to
/home/ubuntu/Impala/logs/custom_cluster_tests/impalad.INFO
18:26:36 MainThread: Starting Impala Daemon logging to
/home/ubuntu/Impala/logs/custom_cluster_tests/impalad_node1.INFO
18:26:36 MainThread: Starting Impala Daemon logging to
/home/ubuntu/Impala/logs/custom_cluster_tests/impalad_node2.INFO
18:26:38 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
18:26:38 MainThread: wait_num_table=1
18:26:38 MainThread: Waiting for Impalad webserver port 25000
18:26:38 MainThread: Waiting for Impalad webserver port 25000
18:26:38 MainThread: Waiting for Impalad webserver port 25001
18:26:38 MainThread: Waiting for Impalad webserver port 25002
18:26:41 MainThread: Waiting for coordinator client services - hs2 port: 21050
hs2-http port: 28000 beeswax port: 21000
18:26:42 MainThread: Waiting for coordinator client services - hs2 port: 21051
hs2-http port: 28001 beeswax port: 21001
18:26:44 MainThread: Waiting for coordinator client services - hs2 port: 21052
hs2-http port: 28002 beeswax port: 21002
18:26:44 MainThread: Getting num_known_live_backends from ip-172-31-61-143:25000
18:26:44 MainThread: num_known_live_backends has reached value: 3
18:26:44 MainThread: Getting num_known_live_backends from ip-172-31-61-143:25001
18:26:44 MainThread: num_known_live_backends has reached value: 3
18:26:44 MainThread: Getting num_known_live_backends from ip-172-31-61-143:25002
18:26:44 MainThread: num_known_live_backends has reached value: 3
18:26:44 MainThread: Total wait: 6.46s
18:26:44 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
executors).
-- 2025-04-17 18:26:44,955 DEBUG MainThread: Found 3 impalad/1 statestored/1
catalogd process(es)
-- 2025-04-17 18:26:44,955 INFO MainThread: Getting metric:
statestore.live-backends from ip-172-31-61-143:25010
-- 2025-04-17 18:26:44,958 INFO MainThread: Metric
'statestore.live-backends' has reached desired value: 4. total_wait: 0s
-- 2025-04-17 18:26:44,958 DEBUG MainThread: Getting num_known_live_backends
from ip-172-31-61-143:25000
-- 2025-04-17 18:26:44,959 INFO MainThread: num_known_live_backends has
reached value: 3
-- 2025-04-17 18:26:44,959 DEBUG MainThread: Getting num_known_live_backends
from ip-172-31-61-143:25001
-- 2025-04-17 18:26:44,961 INFO MainThread: num_known_live_backends has
reached value: 3
-- 2025-04-17 18:26:44,961 DEBUG MainThread: Getting num_known_live_backends
from ip-172-31-61-143:25002
-- 2025-04-17 18:26:44,962 INFO MainThread: num_known_live_backends has
reached value: 3
-- 2025-04-17 18:26:44,962 INFO MainThread: beeswax:
set
client_identifier=custom_cluster/test_events_custom_configs.py::TestEventSyncWaiting::()::test_hms_event_sync_with_commit_compaction[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disabl;
-- 2025-04-17 18:26:44,963 INFO MainThread: beeswax: connected to
localhost:21000 with beeswax
-- 2025-04-17 18:26:44,963 INFO MainThread: hs2:
set
client_identifier=custom_cluster/test_events_custom_configs.py::TestEventSyncWaiting::()::test_hms_event_sync_with_commit_compaction[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disabl;
-- 2025-04-17 18:26:44,963 INFO MainThread: hs2: connected to
localhost:21050 with impyla hs2
-- 2025-04-17 18:26:44,963 INFO MainThread: hs2-http:
set
client_identifier=custom_cluster/test_events_custom_configs.py::TestEventSyncWaiting::()::test_hms_event_sync_with_commit_compaction[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disabl;
-- 2025-04-17 18:26:44,963 INFO MainThread: hs2-http: connected to
localhost:28000 with impyla hs2-http
-- 2025-04-17 18:26:44,963 INFO MainThread: hs2:
set
client_identifier=custom_cluster/test_events_custom_configs.py::TestEventSyncWaiting::()::test_hms_event_sync_with_commit_compaction[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disabl;
-- 2025-04-17 18:26:44,964 INFO MainThread: hs2: connected to
localhost:21050 with impyla hs2
-- 2025-04-17 18:26:44,964 INFO MainThread: hs2:
set
client_identifier=custom_cluster/test_events_custom_configs.py::TestEventSyncWaiting::()::test_hms_event_sync_with_commit_compaction[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disabl;
-- 2025-04-17 18:26:44,964 INFO MainThread: hs2: set_configuration:
set sync_ddl=False;
-- 2025-04-17 18:26:44,964 INFO MainThread: hs2: executing against Impala
at localhost:21050. session: 264cb39583af1aab:d409990d6e5744b1 main_cursor:
True user: None
DROP DATABASE IF EXISTS `test_hms_event_sync_with_commit_compaction_1ac0b73`
CASCADE;
-- 2025-04-17 18:26:45,190 INFO MainThread:
77491be3975d74fb:049dc1f200000000: query started
-- 2025-04-17 18:26:45,191 INFO MainThread:
77491be3975d74fb:049dc1f200000000: getting log for operation
-- 2025-04-17 18:26:45,192 INFO MainThread:
77491be3975d74fb:049dc1f200000000: getting runtime profile operation
-- 2025-04-17 18:26:45,192 INFO MainThread:
77491be3975d74fb:049dc1f200000000: closing query for operation
-- 2025-04-17 18:26:45,196 INFO MainThread: hs2: executing against Impala
at localhost:21050. session: 264cb39583af1aab:d409990d6e5744b1 main_cursor:
True user: None
CREATE DATABASE `test_hms_event_sync_with_commit_compaction_1ac0b73`;
-- 2025-04-17 18:26:45,361 INFO MainThread:
ae4576375fa730dc:52b77e0a00000000: query started
-- 2025-04-17 18:26:45,362 INFO MainThread:
ae4576375fa730dc:52b77e0a00000000: getting log for operation
-- 2025-04-17 18:26:45,362 INFO MainThread:
ae4576375fa730dc:52b77e0a00000000: getting runtime profile operation
-- 2025-04-17 18:26:45,362 INFO MainThread:
ae4576375fa730dc:52b77e0a00000000: closing query for operation
-- 2025-04-17 18:26:45,362 INFO MainThread: Created database
"test_hms_event_sync_with_commit_compaction_1ac0b73" for test ID
"custom_cluster/test_events_custom_configs.py::TestEventSyncWaiting::()::test_hms_event_sync_with_commit_compaction[protocol:
beeswax | table_format: text/none | exec_option: {'test_replan': 1,
'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1,
'exec_single_node_rows_threshold': 0}]"
-- 2025-04-17 18:26:45,362 INFO MainThread: hs2: closing 1 sync and 0 async
hs2 connections to: localhost:21050
-- 2025-04-17 18:26:45,363 INFO MainThread: beeswax:
set
client_identifier=custom_cluster/test_events_custom_configs.py::TestEventSyncWaiting::()::test_hms_event_sync_with_commit_compaction[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disabl;
-- 2025-04-17 18:26:45,363 INFO MainThread: beeswax: set_configuration:
set sync_hms_events_wait_time_s=10;
-- 2025-04-17 18:26:45,363 INFO MainThread: -- executing in HiveServer2
create transactional table
test_hms_event_sync_with_commit_compaction_1ac0b73.foo partitioned by(p)
as select 0 as i, 0 as p
-- 2025-04-17 18:26:54,943 INFO MainThread: -- executing in HiveServer2
insert into test_hms_event_sync_with_commit_compaction_1ac0b73.foo select 1,0
-- 2025-04-17 18:27:05,047 INFO MainThread: beeswax: executing against
localhost:21000
show files in test_hms_event_sync_with_commit_compaction_1ac0b73.foo;
-- 2025-04-17 18:27:05,728 INFO MainThread: Started query
984b75d9783805e3:33f0f1b800000000
-- 2025-04-17 18:27:05,731 INFO MainThread: -- executing in HiveServer2
alter table test_hms_event_sync_with_commit_compaction_1ac0b73.foo
partition(p=0) compact 'minor' and wait
-- 2025-04-17 18:27:36,156 INFO MainThread: beeswax: executing against
localhost:21000
show files in test_hms_event_sync_with_commit_compaction_1ac0b73.foo;
-- 2025-04-17 18:27:36,776 INFO MainThread: Started query
2b476442f82351d8:371b333b00000000{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
