[
https://issues.apache.org/jira/browse/IMPALA-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18013953#comment-18013953
]
Riza Suminto commented on IMPALA-14311:
---------------------------------------
Digging down the logs lead to lingering session
7c4c664971c3e70c:9f44c253c2dcd4ac, which seems to point to test that does
cancellation though WebUI.
{code:java}
I20250809 21:30:19.487442 219820 impala-server.cc:1484]
6d4def2c8f5488e4:2e8fadb900000000] Registered query
query_id=6d4def2c8f5488e4:2e8fadb900000000
session_id=7c4c664971c3e70c:9f44c253c2dcd4ac
I20250809 21:30:19.497285 219820 Frontend.java:3714]
6d4def2c8f5488e4:2e8fadb900000000] Searching for planner to use...
I20250809 21:30:19.497376 219820 Frontend.java:2370]
6d4def2c8f5488e4:2e8fadb900000000] Using Original Planner.
I20250809 21:30:19.497418 219820 Frontend.java:2398]
6d4def2c8f5488e4:2e8fadb900000000] Analyzing query: select count(*) from
functional_parquet.alltypes db: default
I20250809 21:30:19.497462 219820 Frontend.java:2434]
6d4def2c8f5488e4:2e8fadb900000000] The original executor group sets from
executor membership snapshot: [TExecutorGroupSet(curr_num_executors:3,
expected_num_executors:20, exec_group_name_prefix:)]
I20250809 21:30:19.497514 219820 Frontend.java:2454]
6d4def2c8f5488e4:2e8fadb900000000] A total of 1 executor group sets to be
considered for auto-scaling: [TExecutorGroupSet(curr_num_executors:3,
expected_num_executors:20, exec_group_name_prefix:,
max_mem_limit:9223372036854775807, num_cores_per_executor:2147483647)]
I20250809 21:30:19.497556 219820 Frontend.java:2498]
6d4def2c8f5488e4:2e8fadb900000000] Consider executor group set:
TExecutorGroupSet(curr_num_executors:3, expected_num_executors:20,
exec_group_name_prefix:, max_mem_limit:9223372036854775807,
num_cores_per_executor:2147483647) with assumption of 0 cores per node.
I20250809 21:30:19.499738 219820 DebugUtils.java:185]
6d4def2c8f5488e4:2e8fadb900000000] Sleeping for 5000 msec to execute debug
action impalad_load_tables_delay
I20250809 21:30:20.460983 148256 TAcceptQueueServer.cpp:368] New connection to
server beeswax-frontend from client <Host: 127.0.0.1 Port: 54026>
I20250809 21:30:20.546451 219818 status.cc:129] Cancelled from Impala's debug
web interface by user: 'anonymous' at 127.0.0.1:50880
@ 0x25d8016 impala::Status::Status()
@ 0x33d3fc7 impala::ImpalaHttpHandler::CancelQueryHandler()
@ 0x3a85d06 boost::function2<>::operator()()
@ 0x3a81b17 impala::Webserver::RenderUrlWithTemplate()
@ 0x3a7c1f2 impala::Webserver::BeginRequestCallback()
@ 0x3a9c1ea handle_request
@ 0x3a9bc15 process_new_connection
@ 0x3a9b627 worker_thread
@ 0x7fae6c2871ca start_thread
@ 0x7fae68eb48d3 __GI___clone
@ (nil) (unknown)
I20250809 21:30:20.588169 219818 impala-server.cc:1595] UnregisterQuery():
query_id=6d4def2c8f5488e4:2e8fadb900000000
I20250809 21:30:20.550961 219831 status.cc:129] Cancelled from Impala's debug
web interface by user: 'anonymous' at 127.0.0.1:50882
@ 0x25d8016 impala::Status::Status()
@ 0x33d3fc7 impala::ImpalaHttpHandler::CancelQueryHandler()
@ 0x3a85d06 boost::function2<>::operator()()
@ 0x3a81b17 impala::Webserver::RenderUrlWithTemplate()
@ 0x3a7c1f2 impala::Webserver::BeginRequestCallback()
@ 0x3a9c1ea handle_request
@ 0x3a9bc15 process_new_connection
@ 0x3a9b627 worker_thread
@ 0x7fae6c2871ca start_thread
@ 0x7fae68eb48d3 __GI___clone
@ (nil) (unknown)
I20250809 21:30:20.590497 219831 impala-server.cc:1595] UnregisterQuery():
query_id=6d4def2c8f5488e4:2e8fadb900000000
I20250809 21:30:20.590555 219831 impala-server.cc:1798] Cancelled from Impala's
debug web interface by user: 'anonymous' at 127.0.0.1:50880
I20250809 21:30:20.555563 219832 status.cc:129] Cancelled from Impala's debug
web interface by user: 'anonymous' at 127.0.0.1:50884
@ 0x25d8016 impala::Status::Status()
@ 0x33d3fc7 impala::ImpalaHttpHandler::CancelQueryHandler()
@ 0x3a85d06 boost::function2<>::operator()()
@ 0x3a81b17 impala::Webserver::RenderUrlWithTemplate()
@ 0x3a7c1f2 impala::Webserver::BeginRequestCallback()
@ 0x3a9c1ea handle_request
@ 0x3a9bc15 process_new_connection
@ 0x3a9b627 worker_thread
@ 0x7fae6c2871ca start_thread
@ 0x7fae68eb48d3 __GI___clone
@ (nil) (unknown)
I20250809 21:30:20.592861 219832 impala-server.cc:1595] UnregisterQuery():
query_id=6d4def2c8f5488e4:2e8fadb900000000
I20250809 21:30:20.592923 219832 impala-server.cc:1798] Cancelled from Impala's
debug web interface by user: 'anonymous' at 127.0.0.1:50880
I20250809 21:30:20.597944 219818 Canceller.java:96] Cancelling request: thread
Thread[Thread-1948,5,main] for query 6d4def2c8f5488e4:2e8fadb900000000
W20250809 21:30:20.598249 219820 DebugUtils.java:191]
6d4def2c8f5488e4:2e8fadb900000000] Sleep interrupted for the debug action
impalad_load_tables_delay
I20250809 21:30:20.599143 219820 AnalysisContext.java:512]
6d4def2c8f5488e4:2e8fadb900000000] Analysis took 1099 ms
I20250809 21:30:20.599260 219820 BaseAuthorizationChecker.java:114]
6d4def2c8f5488e4:2e8fadb900000000] Authorization check took 0 ms
I20250809 21:30:20.599306 219820 Frontend.java:2947]
6d4def2c8f5488e4:2e8fadb900000000] Analysis and authorization finished.
I20250809 21:30:20.599366 219820 RequestPoolService.java:198]
6d4def2c8f5488e4:2e8fadb900000000] Default pool only, scheduler allocation is
not specified.
I20250809 21:30:20.600855 219820 Analyzer.java:3144]
6d4def2c8f5488e4:2e8fadb900000000] null slots cache size: 0, median entry: 0.0,
99th percentile entry: 0.0, hit rate: 1.0
I20250809 21:30:20.602013 219820 jni-util.cc:321]
6d4def2c8f5488e4:2e8fadb900000000]
org.apache.impala.common.UserCancelledException: Query cancelled by user request
at
org.apache.impala.service.Canceller.throwIfCancelled(Canceller.java:106)
at
org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:2524)
at
org.apache.impala.service.Frontend.getTExecRequestWithFallback(Frontend.java:2374)
at
org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2063)
at
org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:176)
I20250809 21:30:20.602065 219820 status.cc:129]
6d4def2c8f5488e4:2e8fadb900000000] UserCancelledException: Query cancelled by
user request
@ 0x25d8016 impala::Status::Status()
@ 0x38a7e10 impala::JniUtil::GetJniExceptionMsg()
@ 0x32cf3b8 impala::JniCall::Call<>()
@ 0x32afc12 impala::JniUtil::CallJniMethod<>()
@ 0x32abfdf impala::Frontend::GetExecRequest()
@ 0x50b7775 impala::QueryDriver::DoFrontendPlanning()
@ 0x50b7ccb impala::QueryDriver::RunFrontendPlanner()
@ 0x341b384 impala::ImpalaServer::ExecuteInternal()
@ 0x341aa66 impala::ImpalaServer::Execute()
@ 0x33496b6 impala::ImpalaServer::ExecuteStatementCommon()
@ 0x334d357 impala::ImpalaServer::ExecuteStatement()
@ 0x34c68cc
apache::hive::service::cli::thrift::TCLIServiceProcessorT<>::process_ExecuteStatement()
@ 0x34e4438
apache::hive::service::cli::thrift::TCLIServiceProcessorT<>::dispatchCall()
@ 0x34c120f
impala::ImpalaHiveServer2ServiceProcessorT<>::dispatchCall()
@ 0x2299f47 apache::thrift::TDispatchProcessor::process()
@ 0x2ca0f34
apache::thrift::server::TAcceptQueueServer::Task::run()
@ 0x2c88d6e impala::ThriftThread::RunRunnable()
@ 0x2c8d45c boost::_bi::bind_t<>::operator()()
@ 0x2d3f687 boost::function0<>::operator()()
@ 0x3a5105e impala::Thread::SuperviseThread()
@ 0x3a5dcfc boost::_bi::bind_t<>::operator()()
@ 0x5307617 thread_proxy
@ 0x7fae6c2871ca start_thread
@ 0x7fae68eb48d3 __GI___clone
@ (nil) (unknown)
I20250809 21:30:20.685026 148191 impala-server.cc:1643] Query successfully
unregistered: query_id=6d4def2c8f5488e4:2e8fadb900000000 {code}
> Metric 'impala-server.num-open-hiveserver2-sessions' does not reach 0 in
> verifiers.test_verify_metrics.TestValidateMetrics.test_metrics_are_zero
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-14311
> URL: https://issues.apache.org/jira/browse/IMPALA-14311
> Project: IMPALA
> Issue Type: Bug
> Reporter: Daniel Becker
> Assignee: Riza Suminto
> Priority: Blocker
> Labels: broken-build
>
> verifiers.test_verify_metrics.TestValidateMetrics.test_metrics_are_zero fails
> in ASAN tests:
> h3. Error Message
> {code:java}
> AssertionError: Metric impala-server.num-open-hiveserver2-sessions did not
> reach value 0 in 60s. Actual value was '1'. Dumping debug webpages in JSON
> format... Dumped memz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20250812_22:51:34/json/memz.json
> Dumped metrics JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20250812_22:51:34/json/metrics.json
> Dumped queries JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20250812_22:51:34/json/queries.json
> Dumped sessions JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20250812_22:51:34/json/sessions.json
> Dumped threadz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20250812_22:51:34/json/threadz.json
> Dumped rpcz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20250812_22:51:34/json/rpcz.json
> Dumping minidumps for impalads/catalogds... Dumped minidump for Impalad PID
> 2993098 Dumped minidump for Impalad PID 2993100 Dumped minidump for Impalad
> PID 2993105 Dumped minidump for Catalogd PID 2993064{code}
> h3. Stacktrace
>
> {code:java}
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/verifiers/test_verify_metrics.py:42:
> in test_metrics_are_zero
> verifier.verify_metrics_are_zero()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/verifiers/metric_verifier.py:52:
> in verify_metrics_are_zero
> self.wait_for_metric(metric, 0, timeout)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/verifiers/metric_verifier.py:67:
> in wait_for_metric
> self.impalad_service.wait_for_metric_value(metric_name, expected_value,
> timeout)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_service.py:164:
> in wait_for_metric_value
> self.__metric_timeout_assert(metric_name, expected_value, timeout, value)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/tests/common/impala_service.py:251:
> in __metric_timeout_assert
> assert 0, assert_string
> E AssertionError: Metric impala-server.num-open-hiveserver2-sessions did
> not reach value 0 in 60s. Actual value was '1'.
> E Dumping debug webpages in JSON format...
> E Dumped memz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20250812_22:51:34/json/memz.json
> E Dumped metrics JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20250812_22:51:34/json/metrics.json
> E Dumped queries JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20250812_22:51:34/json/queries.json
> E Dumped sessions JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20250812_22:51:34/json/sessions.json
> E Dumped threadz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20250812_22:51:34/json/threadz.json
> E Dumped rpcz JSON to
> $IMPALA_HOME/logs/metric_timeout_diags_20250812_22:51:34/json/rpcz.json
> E Dumping minidumps for impalads/catalogds...
> E Dumped minidump for Impalad PID 2993098
> E Dumped minidump for Impalad PID 2993100
> E Dumped minidump for Impalad PID 2993105
> E Dumped minidump for Catalogd PID 2993064{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
