Zsolt Miskolczi created HIVE-30024:
--------------------------------------
Summary: Scheduled Query auto-disable failures in HA configurations
Key: HIVE-30024
URL: https://issues.apache.org/jira/browse/HIVE-30024
Project: Hive
Issue Type: Bug
Reporter: Zsolt Miskolczi
Auto-disable fails intermittently on HA configurations, because the query is
not disabled after the given number of failures. Example beeline output from a
test run where 3 failures are expected:
{noformat}
'scheduled_executions.scheduled_execution_id'
'scheduled_executions.schedule_name'
'scheduled_executions.executor_query_id' 'scheduled_executions.state'
'scheduled_executions.start_time' 'scheduled_executions.end_time'
'scheduled_executions.elapsed' 'scheduled_executions.error_message'
'scheduled_executions.last_update_time'
'13' 'sq_ucw'
'host1/hive_20260903173105_818d5ecb-1774-4462-bd48-61938f96f0c0' 'FAILED'
'2026-09-03 17:31:05' '2026-09-03 17:31:05' '0' 'FAILED:
SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'12' 'sq_ucw'
'host1/hive_20260903172904_2d757695-f189-4eef-9bea-b205aa929a97' 'FAILED'
'2026-09-03 17:29:03' '2026-09-03 17:29:04' '1' 'FAILED:
SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'11' 'sq_ucw'
'host1/hive_20260903172702_2b06b659-d6df-491c-a13e-42dd174e5797' 'FINISHED'
'2026-09-03 17:27:01' '2026-09-03 17:27:03' '2' 'NULL' 'NULL'
'3' 'sq_ucw'
'host2/hive_20260903173004_b493b2d6-fbcc-4997-8d7e-4555173503d7' 'FAILED'
'2026-09-03 17:30:04' '2026-09-03 17:30:04' '0' 'FAILED:
SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'2' 'sq_ucw'
'host2/hive_20260903172803_d5fd643e-6902-4502-915c-5bfdc448af55' 'FAILED'
'2026-09-03 17:28:02' '2026-09-03 17:28:03'
'1''scheduled_executions.scheduled_execution_id'
'scheduled_executions.schedule_name'
'scheduled_executions.executor_query_id' 'scheduled_executions.state'
'scheduled_executions.start_time' 'scheduled_executions.end_time'
'scheduled_executions.elapsed' 'scheduled_executions.error_message'
'scheduled_executions.last_update_time'
'13' 'sq_ucw'
'host1/hive_20260903173105_818d5ecb-1774-4462-bd48-61938f96f0c0' 'FAILED'
'2026-09-03 17:31:05' '2026-09-03 17:31:05' '0' 'FAILED:
SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'3' 'sq_ucw'
'host2/hive_20260903173004_b493b2d6-fbcc-4997-8d7e-4555173503d7' 'FAILED'
'2026-09-03 17:30:04' '2026-09-03 17:30:04' '0' 'FAILED:
SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'12' 'sq_ucw'
'host1/hive_20260903172904_2d757695-f189-4eef-9bea-b205aa929a97' 'FAILED'
'2026-09-03 17:29:03' '2026-09-03 17:29:04' '1' 'FAILED:
SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'2' 'sq_ucw'
'host2/hive_20260903172803_d5fd643e-6902-4502-915c-5bfdc448af55' 'FAILED'
'2026-09-03 17:28:02' '2026-09-03 17:28:03' '1'
'11' 'sq_ucw'
'host1/hive_20260903172702_2b06b659-d6df-491c-a13e-42dd174e5797' 'FINISHED'
'2026-09-03 17:27:01' '2026-09-03 17:27:03' '2' 'NULL'
'NULL'{noformat}
Both the test and the actual implementation in ObjectStore expect the IDs to be
an incrementing sequence, but if you check the above output this is not always
the case. The above output is ordered using the ID, and start and end
timestamps should correlate to this, but they don’t.
{noformat}
Sep 03 19:26:00 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:1) thread - renaming accordingly.
Sep 03 19:27:01 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:11) thread - renaming accordingly.
Sep 03 19:28:02 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:2) thread - renaming accordingly.
Sep 03 19:29:03 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:12) thread - renaming accordingly.
Sep 03 19:30:04 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:3) thread - renaming accordingly.
Sep 03 19:31:05 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:13) thread - renaming accordingly.
Sep 03 19:32:05 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:4) thread - renaming accordingly.
Sep 03 19:33:06 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:14) thread - renaming accordingly.
Sep 03 19:37:15 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:21) thread - renaming accordingly.
Sep 03 19:38:17 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:22) thread - renaming accordingly.
Sep 03 19:39:18 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:23) thread - renaming accordingly.
Sep 03 19:40:19 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:24) thread - renaming accordingly.
Sep 03 19:41:19 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:25) thread - renaming accordingly.
Sep 03 19:42:20 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:26) thread - renaming accordingly.
Sep 03 19:43:20 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:27) thread - renaming accordingly.{noformat}
It shows that for the first scheduled query the two nodes use their own
distinct ID ranges. Gemini theorized, the evidence and the DataNucleus
documentation supports it, that this could be due to ID pre-allocation. From
the DataNucleus JDO Mapping Guide (v6.0)
:[https://www.datanucleus.org/products/accessplatform_6_0/jdo/mapping.html#value_generation]
It looks there are two possible solution:
- disable pre-allocation
- order by based on an other property that is deterministic, instead of the Id
itself (maybe end_time can be a proper candidate)
Related code:
[https://github.com/apache/hive/blob/bfe74f9b3f92188095d50cf103b86431f8aaa821/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L2956]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
