[
https://issues.apache.org/jira/browse/IMPALA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17945564#comment-17945564
]
Quanlong Huang commented on IMPALA-13897:
-----------------------------------------
The test drops database test_hms_event_sync_f2086869_2 in Hive and runs a SHOW
DATABASES in Impala. It fails since the db still exists in the results.
{noformat}
- 2025-03-24 04:05:06,780 INFO MainThread: -- executing in HiveServer2
drop database if exists test_hms_event_sync_f2086869_2 cascade
-- 2025-03-24 04:05:10,343 INFO MainThread: beeswax: executing against
localhost:21000
show databases;
-- 2025-03-24 04:05:11,553 INFO MainThread: Started query
ae45cfff4b9b0c98:7665be3a00000000{noformat}
Checking the waitForHmsEvent request in catalogd logs, it's correct that
want_db_list=true:
{code:java}
I0324 04:05:10.346038 20780 CatalogServiceCatalog.java:4338]
ae45cfff4b9b0c98:7665be3a00000000] waitForHmsEvent request:
want_minimal_response=false,
coordinator=impala-ec2-centos79-m6i-4xlarge-xldisk-0db8.vpc.cloudera.com,
timeout_s=10, want_db_list=true, want_table_list=false, objects=[]{code}
However, in the response, the removed db (test_hms_event_sync_f2086869_2) is
not found in either updated_objects or removed_objects:
{code:java}
I0324 04:05:11.548328 20780 CatalogServiceCatalog.java:4411]
ae45cfff4b9b0c98:7665be3a00000000] waitForHmsEvent succeeds.
updated_objects=[...],
removed_objects=[DATABASE:test_create_drop_function_14b68a73(9522)]
{code}
In catalogd logs, the deletion of the db is collected at 04:05:10.805657, just
before the waitForHmsEvent request collecting the results:
{noformat}
I0324 04:05:10.805648 7238 catalog-server.cc:1320] Collected deletion:
1:TABLE:test_hms_event_sync_f2086869_2.tbl_2, version=9503, original size=83,
compressed size=84
I0324 04:05:10.805657 7238 catalog-server.cc:1320] Collected deletion:
1:DATABASE:test_hms_event_sync_f2086869_2, version=9504, original size=236,
compressed size=181
I0324 04:05:10.805673 7238 catalog-server.cc:1320] Collected update:
1:CATALOG_SERVICE_ID, version=9508, original size=60, compressed
size=58{noformat}
After catalog update thread collects the catalog updates till version 9508, the
removed objects in the deleteLog are GCed. So the deletion of
test_hms_event_sync_f2086869_2 (version=9504) is GCed and can't be collected by
the waitForHmsEvent request.
This is the same cause as IMPALA-13829. The only difference is IMPALA-13829 is
for missing table deletions. This JIRA is for missing db deletions. Will track
the fix in IMPALA-13829.
> TestEventSyncWaiting.test_hms_event_sync fail in UBSAN build
> ------------------------------------------------------------
>
> Key: IMPALA-13897
> URL: https://issues.apache.org/jira/browse/IMPALA-13897
> Project: IMPALA
> Issue Type: Bug
> Reporter: Zoltán Borók-Nagy
> Assignee: Quanlong Huang
> Priority: Major
> Labels: broken-build
>
> TestEventSyncWaiting.test_hms_event_sync failed in UBSAN build:
> h3. Stacktrace
> {noformat}
> /data/jenkins/workspace/impala-cdw-master-staging-core-ubsan/repos/Impala/tests/metadata/test_event_processing.py:353:
> in test_hms_event_sync
> assert unique_database + "_2\t" not in res.data
> E assert ('test_hms_event_sync_f2086869' + '_2\t') not in
> ['_impala_builtins\tSystem database for Impala builtin functions',
> 'add_session_19f0763f_tmp\t', 'add_session_1c5e456a_tmp\t', 'default\tDefault
> Hive database', 'fetch_first_db\t', 'functional\t', ...]
> E + where ['_impala_builtins\tSystem database for Impala builtin
> functions', 'add_session_19f0763f_tmp\t', 'add_session_1c5e456a_tmp\t',
> 'default\tDefault Hive database', 'fetch_first_db\t', 'functional\t', ...] =
> <tests.beeswax.impala_beeswax.ImpalaBeeswaxResult object at
> 0x7f963dc4e310>.data
> {noformat}
> h3. Standard Error
> {noformat}
> -- 2025-03-24 04:04:38,045 INFO MainThread: beeswax:
> set
> client_identifier=metadata/test_event_processing.py::TestEventSyncWaiting::()::test_hms_event_sync[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'sync_hms_events_wait_time_s':10;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disa;
> -- 2025-03-24 04:04:38,045 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-ubsan/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-ubsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- 2025-03-24 04:04:38,046 INFO MainThread: beeswax: connected to
> localhost:21000 with beeswax
> -- 2025-03-24 04:04:38,046 INFO MainThread: hs2:
> set
> client_identifier=metadata/test_event_processing.py::TestEventSyncWaiting::()::test_hms_event_sync[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'sync_hms_events_wait_time_s':10;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disa;
> -- 2025-03-24 04:04:38,046 INFO MainThread: Could not connect to ('::1',
> 21050, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-ubsan/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-ubsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- 2025-03-24 04:04:38,060 INFO MainThread: hs2: connected to
> localhost:21050 with impyla hs2 session 3443e0f33578357d:1a303481a5f1a791
> -- 2025-03-24 04:04:38,060 INFO MainThread: hs2-http:
> set
> client_identifier=metadata/test_event_processing.py::TestEventSyncWaiting::()::test_hms_event_sync[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'sync_hms_events_wait_time_s':10;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disa;
> -- 2025-03-24 04:04:38,079 INFO MainThread: hs2-http: connected to
> localhost:28000 with impyla hs2-http session f748f9f1478fbd30:db8a48c974e91f82
> -- 2025-03-24 04:04:38,079 INFO MainThread: hs2-feng:
> set
> client_identifier=metadata/test_event_processing.py::TestEventSyncWaiting::()::test_hms_event_sync[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'sync_hms_events_wait_time_s':10;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disa;
> -- 2025-03-24 04:04:38,084 INFO MainThread: hs2-feng: connected to
> localhost:11050 with impyla hs2-feng session c947d99db4ba7cc7:dc52a967bd27f28d
> -- 2025-03-24 04:04:38,084 INFO MainThread: beeswax:
> set
> client_identifier=metadata/test_event_processing.py::TestEventSyncWaiting::()::test_hms_event_sync[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'sync_hms_events_wait_time_s':10;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disa;
> -- 2025-03-24 04:04:38,084 INFO MainThread: hs2:
> set
> client_identifier=metadata/test_event_processing.py::TestEventSyncWaiting::()::test_hms_event_sync[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'sync_hms_events_wait_time_s':10;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disa;
> -- 2025-03-24 04:04:38,084 INFO MainThread: hs2-http:
> set
> client_identifier=metadata/test_event_processing.py::TestEventSyncWaiting::()::test_hms_event_sync[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'sync_hms_events_wait_time_s':10;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disa;
> -- 2025-03-24 04:04:38,084 INFO MainThread: hs2:
> set
> client_identifier=metadata/test_event_processing.py::TestEventSyncWaiting::()::test_hms_event_sync[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'sync_hms_events_wait_time_s':10;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disa;
> -- 2025-03-24 04:04:38,084 INFO MainThread: Could not connect to ('::1',
> 21050, 0, 0)
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-ubsan/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-ubsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- 2025-03-24 04:04:38,100 INFO MainThread: hs2: connected to
> localhost:21050 with impyla hs2 session 1140ef3a123436e2:31170df825dba58c
> -- 2025-03-24 04:04:38,100 INFO MainThread: hs2:
> set
> client_identifier=metadata/test_event_processing.py::TestEventSyncWaiting::()::test_hms_event_sync[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'sync_hms_events_wait_time_s':10;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disa;
> -- 2025-03-24 04:04:38,100 INFO MainThread: hs2: set_configuration:
> set sync_ddl=False;
> -- 2025-03-24 04:04:38,100 INFO MainThread: hs2: executing against Impala
> at localhost:21050. session: 1140ef3a123436e2:31170df825dba58c main_cursor:
> True user: None
> DROP DATABASE IF EXISTS `test_hms_event_sync_f2086869` CASCADE;
> -- 2025-03-24 04:04:42,709 INFO MainThread:
> a34b01c7aa5a84ab:6cd47ea500000000: started query in session
> 1140ef3a123436e2:31170df825dba58c
> -- 2025-03-24 04:04:42,710 INFO MainThread:
> a34b01c7aa5a84ab:6cd47ea500000000: getting log for operation
> -- 2025-03-24 04:04:42,710 INFO MainThread:
> a34b01c7aa5a84ab:6cd47ea500000000: getting runtime profile operation
> -- 2025-03-24 04:04:42,710 INFO MainThread:
> a34b01c7aa5a84ab:6cd47ea500000000: closing query for operation
> -- 2025-03-24 04:04:42,715 INFO MainThread: hs2: executing against Impala
> at localhost:21050. session: 1140ef3a123436e2:31170df825dba58c main_cursor:
> True user: None
> CREATE DATABASE `test_hms_event_sync_f2086869`;
> -- 2025-03-24 04:04:42,738 INFO MainThread:
> 574329fab82029bb:dc4d412700000000: started query in session
> 1140ef3a123436e2:31170df825dba58c
> -- 2025-03-24 04:04:42,738 INFO MainThread:
> 574329fab82029bb:dc4d412700000000: getting log for operation
> -- 2025-03-24 04:04:42,739 INFO MainThread:
> 574329fab82029bb:dc4d412700000000: getting runtime profile operation
> -- 2025-03-24 04:04:42,739 INFO MainThread:
> 574329fab82029bb:dc4d412700000000: closing query for operation
> -- 2025-03-24 04:04:42,739 INFO MainThread: Created database
> "test_hms_event_sync_f2086869" for test ID
> "metadata/test_event_processing.py::TestEventSyncWaiting::()::test_hms_event_sync[protocol:
> beeswax | table_format: text/none | exec_option: {'test_replan': 1,
> 'sync_hms_events_wait_time_s': 10, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0,
> 'sync_hms_events_strict_mode': True}]"
> -- 2025-03-24 04:04:42,739 INFO MainThread: hs2: closing 1 sync and 0
> async hs2 connections to: localhost:21050
> -- 2025-03-24 04:04:42,740 INFO MainThread: beeswax:
> set
> client_identifier=metadata/test_event_processing.py::TestEventSyncWaiting::()::test_hms_event_sync[protocol:beeswax|table_format:text/none|exec_option:{'test_replan':1;'sync_hms_events_wait_time_s':10;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disa;
> -- 2025-03-24 04:04:42,740 INFO MainThread: beeswax: set_configuration:
> set test_replan=1;
> set sync_hms_events_wait_time_s=10;
> set batch_size=0;
> set num_nodes=0;
> set disable_codegen_rows_threshold=5000;
> set disable_codegen=False;
> set abort_on_error=1;
> set exec_single_node_rows_threshold=0;
> set sync_hms_events_strict_mode=True;
> -- 2025-03-24 04:04:42,740 INFO MainThread: -- executing in HiveServer2
> create table test_hms_event_sync_f2086869.tbl (i int) partitioned by (p int)
> -- 2025-03-24 04:04:46,119 INFO MainThread: beeswax: executing against
> localhost:21000
> describe test_hms_event_sync_f2086869.tbl;
> -- 2025-03-24 04:04:50,664 INFO MainThread: Started query
> f14967fcfd666e09:191bda5f00000000
> -- 2025-03-24 04:04:50,666 INFO MainThread: -- executing in HiveServer2
> create table test_hms_event_sync_f2086869.tbl_2 (i int)
> -- 2025-03-24 04:04:54,083 INFO MainThread: beeswax: executing against
> localhost:21000
> show tables in test_hms_event_sync_f2086869;
> -- 2025-03-24 04:04:54,590 INFO MainThread: Started query
> ba46da34fe527143:74fc2b8e00000000
> -- 2025-03-24 04:04:54,591 INFO MainThread: -- executing in HiveServer2
> create view test_hms_event_sync_f2086869.v as select * from
> test_hms_event_sync_f2086869.tbl
> -- 2025-03-24 04:04:58,021 INFO MainThread: beeswax: executing against
> localhost:21000
> show views in test_hms_event_sync_f2086869;
> -- 2025-03-24 04:04:58,628 INFO MainThread: Started query
> e94d3aa70c4db04e:9c27b2c500000000
> -- 2025-03-24 04:04:58,630 INFO MainThread: -- executing in HiveServer2
> create database test_hms_event_sync_f2086869_2;
> create table test_hms_event_sync_f2086869_2.tbl(i int);
> create table test_hms_event_sync_f2086869_2.tbl_2(i int);
> -- 2025-03-24 04:05:02,130 INFO MainThread: beeswax: executing against
> localhost:21000
> drop table test_hms_event_sync_f2086869_2.tbl;
> -- 2025-03-24 04:05:06,678 INFO MainThread: Started query
> a44e9f65bb10d373:01e39fa600000000
> -- 2025-03-24 04:05:06,780 INFO MainThread: -- executing in HiveServer2
> drop database if exists test_hms_event_sync_f2086869_2 cascade
> -- 2025-03-24 04:05:10,343 INFO MainThread: beeswax: executing against
> localhost:21000
> show databases;
> -- 2025-03-24 04:05:11,553 INFO MainThread: Started query
> ae45cfff4b9b0c98:7665be3a00000000{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
