Quanlong Huang created IMPALA-14292:
---------------------------------------
Summary: TestAdmissionController.test_timeout_reason_host_memory
is flaky
Key: IMPALA-14292
URL: https://issues.apache.org/jira/browse/IMPALA-14292
Project: IMPALA
Issue Type: Bug
Reporter: Quanlong Huang
Assignee: Quanlong Huang
h3. Stacktrace
{noformat}
custom_cluster/test_admission_controller.py:1129: in
test_timeout_reason_host_memory
assert num_reasons >= 1, \
E AssertionError: At least one query should have been timed out with topN
query details: Query (id=4347afe4d0941306:ca48ea9000000000):
E DEBUG MODE WARNING: Query profile created while running a DEBUG build
of Impala. Use RELEASE builds to measure query performance.
E Summary:
E Session ID: 5348e8a574e296af:1c5a2c0497cc7699
E Session Type: HIVESERVER2
E HiveServer2 Protocol Version: V6
E Start Time: 2025-08-05 02:50:25.109749000
E End Time: 2025-08-05 02:50:25.601402000
E Duration: 491.653ms (491653 us)
E Query Type: QUERY
E Query State: EXCEPTION
E Impala Query State: ERROR
E Query Status: Exec() rpc failed: Remote error: Service unavailable:
ExecQueryFInstances request on impala.ControlService from 127.0.0.1:49400
dropped due to backpressure. The service queue contains 0 items out of a
maximum of 2147483647; memory consumption is 0.
...
E assert 0 >= 1{noformat}
h3. Standard Error
{noformat}
-- 2025-08-05 02:50:10,671 INFO MainThread: Starting cluster with command:
/data/jenkins/workspace/impala-cdw-master-staging-core-ubsan-arm/repos/Impala/bin/start-impala-cluster.py
'--state_store_args=--statestore_update_frequency_ms=50
--statestore_priority_update_frequency_ms=50
--statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3
--log_dir=/data/jenkins/workspace/impala-cdw-master-staging-core-ubsan-arm/repos/Impala/logs/custom_cluster_tests
--log_level=1 '--impalad_args=-vmodule admission-controller=3
-default_pool_max_requests 10 -default_pool_max_queued 10
-default_pool_mem_limit 10485760 -mem_limit=2097152
-queue_wait_timeout_ms=1000 -codegen_cache_capacity=0 '
'--state_store_args=-statestore_heartbeat_frequency_ms=100
-statestore_priority_update_frequency_ms=100 '
--impalad_args=--default_query_options=
02:50:11 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
02:50:11 MainThread: Starting State Store logging to
/data/jenkins/workspace/impala-cdw-master-staging-core-ubsan-arm/repos/Impala/logs/custom_cluster_tests/statestored.INFO
02:50:11 MainThread: Starting Catalog Service logging to
/data/jenkins/workspace/impala-cdw-master-staging-core-ubsan-arm/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
02:50:11 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-cdw-master-staging-core-ubsan-arm/repos/Impala/logs/custom_cluster_tests/impalad.INFO
02:50:11 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-cdw-master-staging-core-ubsan-arm/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
02:50:11 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-cdw-master-staging-core-ubsan-arm/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
02:50:13 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
02:50:13 MainThread: Waiting for Impalad webserver port 25000
02:50:14 MainThread: Waiting for Impalad webserver port 25000
02:50:14 MainThread: Waiting for Impalad webserver port 25000
02:50:14 MainThread: Waiting for Impalad webserver port 25001
02:50:14 MainThread: Waiting for Impalad webserver port 25002
02:50:19 MainThread: Waiting for coordinator client services - hs2 port: 21050
hs2-http port: 28000 beeswax port: 21000
02:50:22 MainThread: Waiting for coordinator client services - hs2 port: 21051
hs2-http port: 28001 beeswax port: 21001
02:50:24 MainThread: Waiting for coordinator client services - hs2 port: 21052
hs2-http port: 28002 beeswax port: 21002
02:50:24 MainThread: Getting num_known_live_backends from
impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:25000
02:50:24 MainThread: num_known_live_backends has reached value: 3
02:50:24 MainThread: Getting num_known_live_backends from
impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:25001
02:50:24 MainThread: num_known_live_backends has reached value: 3
02:50:25 MainThread: Getting num_known_live_backends from
impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:25002
02:50:25 MainThread: num_known_live_backends has reached value: 3
02:50:25 MainThread: Total wait: 11.48s
02:50:25 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
executors).
-- 2025-08-05 02:50:25,089 DEBUG MainThread: Found 3 impalad/1 statestored/1
catalogd process(es)
-- 2025-08-05 02:50:25,090 INFO MainThread: Getting metric:
statestore.live-backends from
impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:25010
-- 2025-08-05 02:50:25,094 INFO MainThread: Metric
'statestore.live-backends' has reached desired value: 4. total_wait: 0s
-- 2025-08-05 02:50:25,094 DEBUG MainThread: Getting num_known_live_backends
from impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:25000
-- 2025-08-05 02:50:25,096 INFO MainThread: num_known_live_backends has
reached value: 3
-- 2025-08-05 02:50:25,096 DEBUG MainThread: Getting num_known_live_backends
from impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:25001
-- 2025-08-05 02:50:25,099 INFO MainThread: num_known_live_backends has
reached value: 3
-- 2025-08-05 02:50:25,099 DEBUG MainThread: Getting num_known_live_backends
from impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:25002
-- 2025-08-05 02:50:25,101 INFO MainThread: num_known_live_backends has
reached value: 3
-- 2025-08-05 02:50:25,102 INFO MainThread: beeswax:
set
client_identifier=custom_cluster/test_admission_controller.py::TestAdmissionController::()::test_timeout_reason_host_memory;
-- 2025-08-05 02:50:25,102 INFO MainThread: beeswax: connected to
impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:21000 with beeswax
-- 2025-08-05 02:50:25,102 INFO MainThread: hs2:
set
client_identifier=custom_cluster/test_admission_controller.py::TestAdmissionController::()::test_timeout_reason_host_memory;
-- 2025-08-05 02:50:25,102 INFO MainThread: hs2: connected to
impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:21050 with impyla
hs2
-- 2025-08-05 02:50:25,102 INFO MainThread: hs2-http:
set
client_identifier=custom_cluster/test_admission_controller.py::TestAdmissionController::()::test_timeout_reason_host_memory;
-- 2025-08-05 02:50:25,102 INFO MainThread: hs2-http: connected to
impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:28000 with impyla
hs2-http
-- 2025-08-05 02:50:25,103 INFO MainThread: hs2-feng:
set
client_identifier=custom_cluster/test_admission_controller.py::TestAdmissionController::()::test_timeout_reason_host_memory;
-- 2025-08-05 02:50:25,103 INFO MainThread: hs2-feng: connected to
impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:11050 with impyla
hs2-feng
-- 2025-08-05 02:50:25,103 INFO MainThread: beeswax:
set
client_identifier=custom_cluster/test_admission_controller.py::TestAdmissionController::()::test_timeout_reason_host_memory;
-- 2025-08-05 02:50:25,103 INFO MainThread: hs2:
set
client_identifier=custom_cluster/test_admission_controller.py::TestAdmissionController::()::test_timeout_reason_host_memory;
-- 2025-08-05 02:50:25,103 INFO MainThread: hs2-http:
set
client_identifier=custom_cluster/test_admission_controller.py::TestAdmissionController::()::test_timeout_reason_host_memory;
-- 2025-08-05 02:50:25,105 INFO MainThread: hs2:
set enable_trivial_query_for_admission=false;
-- 2025-08-05 02:50:25,105 INFO MainThread: hs2:
set
client_identifier=custom_cluster/test_admission_controller.py::TestAdmissionController::()::test_timeout_reason_host_memory;
-- 2025-08-05 02:50:25,105 INFO MainThread: hs2: connected to
impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:21050 with impyla
hs2
-- 2025-08-05 02:50:25,105 INFO MainThread: hs2:
set
client_identifier=custom_cluster/test_admission_controller.py::TestAdmissionController::()::test_timeout_reason_host_memory;
-- 2025-08-05 02:50:25,105 INFO MainThread: hs2: set_configuration:
set spool_query_results=0;
set mem_limit=2mb;
-- 2025-08-05 02:50:25,108 INFO MainThread: hs2: executing against Impala
at impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:21050. session:
5348e8a574e296af:1c5a2c0497cc7699 main_cursor: False user: None
select sleep(1000);
-- 2025-08-05 02:50:25,594 INFO MainThread: hs2: executing against Impala
at impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:21050. session:
f84214fc14d487b6:a1f5879e48352e99 main_cursor: False user: None
select sleep(1000);
-- 2025-08-05 02:50:25,601 INFO MainThread: hs2: executing against Impala
at impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:21050. session:
6a45a64f7d50440d:12e486c45726f085 main_cursor: False user: None
select sleep(1000);
-- 2025-08-05 02:50:25,608 INFO MainThread: hs2: executing against Impala
at impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:21050. session:
d845c41e70e16edb:9c6baf4d619ca3b4 main_cursor: False user: None
select sleep(1000);
-- 2025-08-05 02:50:25,615 INFO MainThread: hs2: executing against Impala
at impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:21050. session:
1e4e767b8689bb81:2bda2cfa395eb5bc main_cursor: False user: None
select sleep(1000);
-- 2025-08-05 02:50:25,620 INFO MainThread:
4347afe4d0941306:ca48ea9000000000: getting state
-- 2025-08-05 02:50:25,621 INFO MainThread:
4347afe4d0941306:ca48ea9000000000: getting runtime profile operation
-- 2025-08-05 02:50:25,622 INFO MainThread:
e047c2d275250b53:99cb9ea500000000: getting state
-- 2025-08-05 02:50:25,622 INFO MainThread:
e047c2d275250b53:99cb9ea500000000: getting runtime profile operation
-- 2025-08-05 02:50:25,623 INFO MainThread:
9145a55bb1d0548d:8fb8a4b000000000: getting state
-- 2025-08-05 02:50:25,623 INFO MainThread:
9145a55bb1d0548d:8fb8a4b000000000: getting runtime profile operation
-- 2025-08-05 02:50:25,624 INFO MainThread:
bb49b16c00c971f0:7e65f60f00000000: getting state
-- 2025-08-05 02:50:25,624 INFO MainThread:
bb49b16c00c971f0:7e65f60f00000000: getting runtime profile operation
-- 2025-08-05 02:50:25,624 INFO MainThread:
5c4eccf00e8c84fb:c37ca0b400000000: getting state
-- 2025-08-05 02:50:25,624 INFO MainThread:
5c4eccf00e8c84fb:c37ca0b400000000: getting runtime profile operation
-- 2025-08-05 02:50:25,625 INFO MainThread:
4347afe4d0941306:ca48ea9000000000: closing query for operation
-- 2025-08-05 02:50:25,625 INFO MainThread:
e047c2d275250b53:99cb9ea500000000: closing query for operation
-- 2025-08-05 02:50:25,626 INFO MainThread:
9145a55bb1d0548d:8fb8a4b000000000: closing query for operation
-- 2025-08-05 02:50:25,626 INFO MainThread:
bb49b16c00c971f0:7e65f60f00000000: closing query for operation
-- 2025-08-05 02:50:25,626 INFO MainThread:
5c4eccf00e8c84fb:c37ca0b400000000: closing query for operation
-- 2025-08-05 02:50:25,627 INFO MainThread: hs2: closing 1 sync and 5 async
hs2 connections to:
impala-ec2-rhel92-m7g-4xlarge-ondemand-0f8c.vpc.cloudera.com:21050{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
