[
https://issues.apache.org/jira/browse/HIVE-30024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Rebele updated HIVE-30024:
---------------------------------
Description:
Auto-disable fails intermittently on HA configurations, because the query is
not disabled after the given number of failures. Example beeline output from a
test run where 3 failures are expected:
{noformat}
'scheduled_executions.scheduled_execution_id'
'scheduled_executions.schedule_name' 'scheduled_executions.executor_query_id'
'scheduled_executions.state'
'scheduled_executions.start_time' 'scheduled_executions.end_time'
'scheduled_executions.elapsed' 'scheduled_executions.error_message'
'scheduled_executions.last_update_time'
'13' 'sq_ucw'
'host1/hive_20260903173105_818d5ecb-1774-4462-bd48-61938f96f0c0' 'FAILED'
'2026-09-03 17:31:05' '2026-09-03 17:31:05'
'0' 'FAILED: SemanticException [Error
10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'12' 'sq_ucw'
'host1/hive_20260903172904_2d757695-f189-4eef-9bea-b205aa929a97' 'FAILED'
'2026-09-03 17:29:03' '2026-09-03 17:29:04'
'1' 'FAILED: SemanticException [Error
10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'11' 'sq_ucw'
'host1/hive_20260903172702_2b06b659-d6df-491c-a13e-42dd174e5797'
'FINISHED' '2026-09-03 17:27:01' '2026-09-03
17:27:03' '2' 'NULL'
'NULL'
'3' 'sq_ucw'
'host2/hive_20260903173004_b493b2d6-fbcc-4997-8d7e-4555173503d7' 'FAILED'
'2026-09-03 17:30:04' '2026-09-03 17:30:04'
'0' 'FAILED: SemanticException [Error
10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'2' 'sq_ucw'
'host2/hive_20260903172803_d5fd643e-6902-4502-915c-5bfdc448af55' 'FAILED'
'2026-09-03 17:28:02' '2026-09-03 17:28:03'
'1'
'scheduled_executions.scheduled_execution_id'
'scheduled_executions.schedule_name' 'scheduled_executions.executor_query_id'
'scheduled_executions.state'
'scheduled_executions.start_time' 'scheduled_executions.end_time'
'scheduled_executions.elapsed' 'scheduled_executions.error_message'
'scheduled_executions.last_update_time'
'13' 'sq_ucw'
'host1/hive_20260903173105_818d5ecb-1774-4462-bd48-61938f96f0c0' 'FAILED'
'2026-09-03 17:31:05' '2026-09-03 17:31:05'
'0' 'FAILED: SemanticException [Error
10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'3' 'sq_ucw'
'host2/hive_20260903173004_b493b2d6-fbcc-4997-8d7e-4555173503d7' 'FAILED'
'2026-09-03 17:30:04' '2026-09-03 17:30:04'
'0' 'FAILED: SemanticException [Error
10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'12' 'sq_ucw'
'host1/hive_20260903172904_2d757695-f189-4eef-9bea-b205aa929a97' 'FAILED'
'2026-09-03 17:29:03' '2026-09-03 17:29:04'
'1' 'FAILED: SemanticException [Error
10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'2' 'sq_ucw'
'host2/hive_20260903172803_d5fd643e-6902-4502-915c-5bfdc448af55' 'FAILED'
'2026-09-03 17:28:02' '2026-09-03 17:28:03'
'1'
'11' 'sq_ucw'
'host1/hive_20260903172702_2b06b659-d6df-491c-a13e-42dd174e5797'
'FINISHED' '2026-09-03 17:27:01' '2026-09-03
17:27:03' '2' 'NULL'
'NULL'{noformat}
Both the test and the actual implementation in ObjectStore expect the IDs to be
an incrementing sequence, but if you check the above output this is not always
the case. The above output is ordered using the ID, and start and end
timestamps should correlate to this, but they don’t.
{noformat}
Sep 03 19:26:00 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:1) thread - renaming accordingly.
Sep 03 19:27:01 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:11) thread - renaming accordingly.
Sep 03 19:28:02 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:2) thread - renaming accordingly.
Sep 03 19:29:03 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:12) thread - renaming accordingly.
Sep 03 19:30:04 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:3) thread - renaming accordingly.
Sep 03 19:31:05 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:13) thread - renaming accordingly.
Sep 03 19:32:05 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:4) thread - renaming accordingly.
Sep 03 19:33:06 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:14) thread - renaming accordingly.
Sep 03 19:37:15 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:21) thread - renaming accordingly.
Sep 03 19:38:17 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:22) thread - renaming accordingly.
Sep 03 19:39:18 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:23) thread - renaming accordingly.
Sep 03 19:40:19 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:24) thread - renaming accordingly.
Sep 03 19:41:19 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:25) thread - renaming accordingly.
Sep 03 19:42:20 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:26) thread - renaming accordingly.
Sep 03 19:43:20 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:27) thread - renaming accordingly.{noformat}
It shows that for the first scheduled query the two nodes use their own
distinct ID ranges. Gemini theorized, the evidence and the DataNucleus
documentation supports it, that this could be due to ID pre-allocation. From
the DataNucleus JDO Mapping Guide (v6.0)
:[https://www.datanucleus.org/products/accessplatform_6_0/jdo/mapping.html#value_generation]
It looks there are two possible solution:
- disable pre-allocation
- order by based on an other property that is deterministic, instead of the Id
itself (maybe end_time can be a proper candidate)
Related code:
[https://github.com/apache/hive/blob/bfe74f9b3f92188095d50cf103b86431f8aaa821/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L2956]
was:
Auto-disable fails intermittently on HA configurations, because the query is
not disabled after the given number of failures. Example beeline output from a
test run where 3 failures are expected:
{noformat}
'scheduled_executions.scheduled_execution_id'
'scheduled_executions.schedule_name'
'scheduled_executions.executor_query_id' 'scheduled_executions.state'
'scheduled_executions.start_time' 'scheduled_executions.end_time'
'scheduled_executions.elapsed' 'scheduled_executions.error_message'
'scheduled_executions.last_update_time'
'13' 'sq_ucw'
'host1/hive_20260903173105_818d5ecb-1774-4462-bd48-61938f96f0c0' 'FAILED'
'2026-09-03 17:31:05' '2026-09-03 17:31:05' '0' 'FAILED:
SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'12' 'sq_ucw'
'host1/hive_20260903172904_2d757695-f189-4eef-9bea-b205aa929a97' 'FAILED'
'2026-09-03 17:29:03' '2026-09-03 17:29:04' '1' 'FAILED:
SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'11' 'sq_ucw'
'host1/hive_20260903172702_2b06b659-d6df-491c-a13e-42dd174e5797' 'FINISHED'
'2026-09-03 17:27:01' '2026-09-03 17:27:03' '2' 'NULL' 'NULL'
'3' 'sq_ucw'
'host2/hive_20260903173004_b493b2d6-fbcc-4997-8d7e-4555173503d7' 'FAILED'
'2026-09-03 17:30:04' '2026-09-03 17:30:04' '0' 'FAILED:
SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'2' 'sq_ucw'
'host2/hive_20260903172803_d5fd643e-6902-4502-915c-5bfdc448af55' 'FAILED'
'2026-09-03 17:28:02' '2026-09-03 17:28:03'
'1''scheduled_executions.scheduled_execution_id'
'scheduled_executions.schedule_name'
'scheduled_executions.executor_query_id' 'scheduled_executions.state'
'scheduled_executions.start_time' 'scheduled_executions.end_time'
'scheduled_executions.elapsed' 'scheduled_executions.error_message'
'scheduled_executions.last_update_time'
'13' 'sq_ucw'
'host1/hive_20260903173105_818d5ecb-1774-4462-bd48-61938f96f0c0' 'FAILED'
'2026-09-03 17:31:05' '2026-09-03 17:31:05' '0' 'FAILED:
SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'3' 'sq_ucw'
'host2/hive_20260903173004_b493b2d6-fbcc-4997-8d7e-4555173503d7' 'FAILED'
'2026-09-03 17:30:04' '2026-09-03 17:30:04' '0' 'FAILED:
SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'12' 'sq_ucw'
'host1/hive_20260903172904_2d757695-f189-4eef-9bea-b205aa929a97' 'FAILED'
'2026-09-03 17:29:03' '2026-09-03 17:29:04' '1' 'FAILED:
SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
'2' 'sq_ucw'
'host2/hive_20260903172803_d5fd643e-6902-4502-915c-5bfdc448af55' 'FAILED'
'2026-09-03 17:28:02' '2026-09-03 17:28:03' '1'
'11' 'sq_ucw'
'host1/hive_20260903172702_2b06b659-d6df-491c-a13e-42dd174e5797' 'FINISHED'
'2026-09-03 17:27:01' '2026-09-03 17:27:03' '2' 'NULL'
'NULL'{noformat}
Both the test and the actual implementation in ObjectStore expect the IDs to be
an incrementing sequence, but if you check the above output this is not always
the case. The above output is ordered using the ID, and start and end
timestamps should correlate to this, but they don’t.
{noformat}
Sep 03 19:26:00 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:1) thread - renaming accordingly.
Sep 03 19:27:01 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:11) thread - renaming accordingly.
Sep 03 19:28:02 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:2) thread - renaming accordingly.
Sep 03 19:29:03 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:12) thread - renaming accordingly.
Sep 03 19:30:04 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:3) thread - renaming accordingly.
Sep 03 19:31:05 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:13) thread - renaming accordingly.
Sep 03 19:32:05 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:4) thread - renaming accordingly.
Sep 03 19:33:06 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
execution_id:14) thread - renaming accordingly.
Sep 03 19:37:15 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:21) thread - renaming accordingly.
Sep 03 19:38:17 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:22) thread - renaming accordingly.
Sep 03 19:39:18 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:23) thread - renaming accordingly.
Sep 03 19:40:19 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:24) thread - renaming accordingly.
Sep 03 19:41:19 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:25) thread - renaming accordingly.
Sep 03 19:42:20 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:26) thread - renaming accordingly.
Sep 03 19:43:20 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
execution_id:27) thread - renaming accordingly.{noformat}
It shows that for the first scheduled query the two nodes use their own
distinct ID ranges. Gemini theorized, the evidence and the DataNucleus
documentation supports it, that this could be due to ID pre-allocation. From
the DataNucleus JDO Mapping Guide (v6.0)
:[https://www.datanucleus.org/products/accessplatform_6_0/jdo/mapping.html#value_generation]
It looks there are two possible solution:
- disable pre-allocation
- order by based on an other property that is deterministic, instead of the Id
itself (maybe end_time can be a proper candidate)
Related code:
[https://github.com/apache/hive/blob/bfe74f9b3f92188095d50cf103b86431f8aaa821/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L2956]
> Scheduled Query auto-disable failures in HA configurations
> ----------------------------------------------------------
>
> Key: HIVE-30024
> URL: https://issues.apache.org/jira/browse/HIVE-30024
> Project: Hive
> Issue Type: Bug
> Reporter: Zsolt Miskolczi
> Priority: Major
> Labels: pull-request-available
>
> Auto-disable fails intermittently on HA configurations, because the query is
> not disabled after the given number of failures. Example beeline output from
> a test run where 3 failures are expected:
> {noformat}
> 'scheduled_executions.scheduled_execution_id'
> 'scheduled_executions.schedule_name'
> 'scheduled_executions.executor_query_id'
> 'scheduled_executions.state' 'scheduled_executions.start_time'
> 'scheduled_executions.end_time' 'scheduled_executions.elapsed'
> 'scheduled_executions.error_message'
> 'scheduled_executions.last_update_time'
> '13' 'sq_ucw'
> 'host1/hive_20260903173105_818d5ecb-1774-4462-bd48-61938f96f0c0'
> 'FAILED' '2026-09-03 17:31:05' '2026-09-03
> 17:31:05' '0' 'FAILED:
> SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
> '12' 'sq_ucw'
> 'host1/hive_20260903172904_2d757695-f189-4eef-9bea-b205aa929a97'
> 'FAILED' '2026-09-03 17:29:03' '2026-09-03
> 17:29:04' '1' 'FAILED:
> SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
> '11' 'sq_ucw'
> 'host1/hive_20260903172702_2b06b659-d6df-491c-a13e-42dd174e5797'
> 'FINISHED' '2026-09-03 17:27:01' '2026-09-03
> 17:27:03' '2' 'NULL'
> 'NULL'
> '3' 'sq_ucw'
> 'host2/hive_20260903173004_b493b2d6-fbcc-4997-8d7e-4555173503d7'
> 'FAILED' '2026-09-03 17:30:04' '2026-09-03
> 17:30:04' '0' 'FAILED:
> SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
> '2' 'sq_ucw'
> 'host2/hive_20260903172803_d5fd643e-6902-4502-915c-5bfdc448af55'
> 'FAILED' '2026-09-03 17:28:02' '2026-09-03
> 17:28:03' '1'
>
> 'scheduled_executions.scheduled_execution_id'
> 'scheduled_executions.schedule_name'
> 'scheduled_executions.executor_query_id'
> 'scheduled_executions.state' 'scheduled_executions.start_time'
> 'scheduled_executions.end_time' 'scheduled_executions.elapsed'
> 'scheduled_executions.error_message'
> 'scheduled_executions.last_update_time'
> '13' 'sq_ucw'
> 'host1/hive_20260903173105_818d5ecb-1774-4462-bd48-61938f96f0c0'
> 'FAILED' '2026-09-03 17:31:05' '2026-09-03
> 17:31:05' '0' 'FAILED:
> SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
> '3' 'sq_ucw'
> 'host2/hive_20260903173004_b493b2d6-fbcc-4997-8d7e-4555173503d7'
> 'FAILED' '2026-09-03 17:30:04' '2026-09-03
> 17:30:04' '0' 'FAILED:
> SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
> '12' 'sq_ucw'
> 'host1/hive_20260903172904_2d757695-f189-4eef-9bea-b205aa929a97'
> 'FAILED' '2026-09-03 17:29:03' '2026-09-03
> 17:29:04' '1' 'FAILED:
> SemanticException [Error 10001]: Line 1:48 Table not found 'tbl_pqa'' 'NULL'
> '2' 'sq_ucw'
> 'host2/hive_20260903172803_d5fd643e-6902-4502-915c-5bfdc448af55'
> 'FAILED' '2026-09-03 17:28:02' '2026-09-03
> 17:28:03' '1'
>
> '11' 'sq_ucw'
> 'host1/hive_20260903172702_2b06b659-d6df-491c-a13e-42dd174e5797'
> 'FINISHED' '2026-09-03 17:27:01' '2026-09-03
> 17:27:03' '2' 'NULL'
> 'NULL'{noformat}
> Both the test and the actual implementation in ObjectStore expect the IDs to
> be an incrementing sequence, but if you check the above output this is not
> always the case. The above output is ordered using the ID, and start and end
> timestamps should correlate to this, but they don’t.
>
> {noformat}
> Sep 03 19:26:00 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:1) thread - renaming accordingly.
> Sep 03 19:27:01 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:11) thread - renaming accordingly.
> Sep 03 19:28:02 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:2) thread - renaming accordingly.
> Sep 03 19:29:03 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:12) thread - renaming accordingly.
> Sep 03 19:30:04 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:3) thread - renaming accordingly.
> Sep 03 19:31:05 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:13) thread - renaming accordingly.
> Sep 03 19:32:05 host2: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:4) thread - renaming accordingly.
> Sep 03 19:33:06 host1: Starting Scheduled Query Executor(schedule:sq_ucw,
> execution_id:14) thread - renaming accordingly.
> Sep 03 19:37:15 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
> execution_id:21) thread - renaming accordingly.
> Sep 03 19:38:17 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
> execution_id:22) thread - renaming accordingly.
> Sep 03 19:39:18 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
> execution_id:23) thread - renaming accordingly.
> Sep 03 19:40:19 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
> execution_id:24) thread - renaming accordingly.
> Sep 03 19:41:19 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
> execution_id:25) thread - renaming accordingly.
> Sep 03 19:42:20 host1: Starting Scheduled Query Executor(schedule:sq_wrn,
> execution_id:26) thread - renaming accordingly.
> Sep 03 19:43:20 host2: Starting Scheduled Query Executor(schedule:sq_wrn,
> execution_id:27) thread - renaming accordingly.{noformat}
> It shows that for the first scheduled query the two nodes use their own
> distinct ID ranges. Gemini theorized, the evidence and the DataNucleus
> documentation supports it, that this could be due to ID pre-allocation. From
> the DataNucleus JDO Mapping Guide (v6.0)
> :[https://www.datanucleus.org/products/accessplatform_6_0/jdo/mapping.html#value_generation]
>
> It looks there are two possible solution:
> - disable pre-allocation
> - order by based on an other property that is deterministic, instead of the
> Id itself (maybe end_time can be a proper candidate)
> Related code:
> [https://github.com/apache/hive/blob/bfe74f9b3f92188095d50cf103b86431f8aaa821/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L2956]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
