This is an automated email from the ASF dual-hosted git repository.
michaelsmith pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
The following commit(s) were added to refs/heads/master by this push:
new 134c28d44 IMPALA-13788: [DOCS] Docs for query options
SYNC_HMS_EVENTS_WAIT_TIME_S and SYNC_HMS_EVENTS_STRICT_MODE
134c28d44 is described below
commit 134c28d4451d648232e24789ac15e5ff9113ed75
Author: m-sanjana19 <[email protected]>
AuthorDate: Fri Oct 24 15:56:24 2025 +0530
IMPALA-13788: [DOCS] Docs for query options SYNC_HMS_EVENTS_WAIT_TIME_S
and SYNC_HMS_EVENTS_STRICT_MODE
The commit documents query options SYNC_HMS_EVENTS_WAIT_TIME_S
and SYNC_HMS_EVENTS_STRICT_MODE
Url: https://impala.apache.org/docs/build/html/topics/impala_set.html
Change-Id: Ia11663c5e84794d4bca658124cde59bf97aa7158
Reviewed-on: http://gerrit.cloudera.org:8080/23592
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Quanlong Huang <[email protected]>
Reviewed-by: Jason Fehr <[email protected]>
---
docs/impala.ditamap | 2 +
docs/topics/impala_sync_hms_events_strict_mode.xml | 73 ++++++++++++++++++++
docs/topics/impala_sync_hms_events_wait_time_s.xml | 79 ++++++++++++++++++++++
3 files changed, 154 insertions(+)
diff --git a/docs/impala.ditamap b/docs/impala.ditamap
index 466bb8f86..8a0f96292 100644
--- a/docs/impala.ditamap
+++ b/docs/impala.ditamap
@@ -265,6 +265,8 @@ under the License.
<topicref href="topics/impala_spool_query_results.xml"/>
<topicref href="topics/impala_support_start_over.xml"/>
<topicref href="topics/impala_sync_ddl.xml"/>
+ <topicref href="topics/impala_sync_hms_events_strict_mode.xml"/>
+ <topicref href="topics/impala_sync_hms_events_wait_time_s.xml"/>
<topicref href="topics/impala_thread_reservation_aggregate_limit.xml"/>
<topicref href="topics/impala_thread_reservation_limit.xml"/>
<topicref href="topics/impala_timezone.xml"/>
diff --git a/docs/topics/impala_sync_hms_events_strict_mode.xml
b/docs/topics/impala_sync_hms_events_strict_mode.xml
new file mode 100644
index 000000000..45edb6da0
--- /dev/null
+++ b/docs/topics/impala_sync_hms_events_strict_mode.xml
@@ -0,0 +1,73 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="5.0.0" id="impala_sync_hms_events_strict_mode">
+ <title>SYNC_HMS_EVENTS_STRICT_MODE Query Option</title>
+ <titlealts audience="PDF"><navtitle>SYNC HMS EVENTS STRICT
MODE</navtitle></titlealts>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="Impala Query Options"/>
+ <data name="Category" value="DDL"/>
+ <data name="Category" value="SQL"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Data Analysts"/>
+ </metadata>
+ </prolog>
+ <conbody>
+ <p id="IMPALA-12152"><indexterm
audience="hidden">SYNC_HMS_EVENTS_STRICT_MODE query
+ option</indexterm>This query option controls the behavior of the query
coordinator if it
+ cannot successfully sync with the latest HMS events (e.g., if the
waiting time set by
+ <codeph>SYNC_HMS_EVENTS_WAIT_TIME_S</codeph> is reached, or the event
processor is in an
+ error state).</p>
+ <p conref="../shared/impala_common.xml#common/type_boolean"/>
+ <p conref="../shared/impala_common.xml#common/default_false_0"/>
+ <p><b>Query Range:</b>
+ <codeph>True</codeph> or <codeph>False</codeph></p>
+ <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+ <p>This option determines whether Impala favors consistency (failing the
query) or availability
+ (starting the query planning despite potential lag).<ul
id="ul_jpg_3qs_bhc">
+ <li><codeph>True</codeph> (Strict Mode): If Catalog service cannot
sync the metadata with
+ the latest HMS event ID before the timeout is reached, the
coordinator fails the query
+ with an error. This prioritizes correctness and metadata
consistency.<p>If the coordinator
+ fails the query when <codeph>SYNC_HMS_EVENTS_STRICT_MODE</codeph>
is set to
+ <codeph>TRUE</codeph> (Strict Mode), you will see an error
message starting with
+ <msgph>Failed to sync events from Metastore:</msgph> and then
the specific reason,
+ such as <msgph>HMS event processing is disabled</msgph>.</p></li>
+ <li><codeph>False</codeph> (Non-Strict Mode - Default): If the waiting
times out, the
+ coordinator starts planning the query immediately but issues a
warning message in the
+ query profile. This prioritizes availability, allowing the query to
run, but risks working
+ with slightly stale metadata.</li>
+ </ul></p>
+ <note id="note_fhz_rqs_bhc">This option is only relevant if
+ <codeph>SYNC_HMS_EVENTS_WAIT_TIME_S</codeph> is set to a value greater
than 0.</note>
+ <p><b>Example</b></p>
+ <p>The following example shows how to set
<codeph>SYNC_HMS_EVENTS_STRICT_MODE</codeph> to
+ <codeph>TRUE</codeph> for a specific query to enforce
+ consistency:<codeblock id="codeblock_fnm_hsp_fhc">SET
SYNC_HMS_EVENTS_STRICT_MODE=TRUE;
+SELECT COUNT(*) FROM functional.alltypes;</codeblock></p>
+ <p><b>Added in:</b>
+ <keyword keyref="impala50_full"/></p>
+ <p conref="../shared/impala_common.xml#common/related_info"/>
+ <p>
+ <xref href="impala_ddl.xml#ddl"/>
+ </p>
+ </conbody>
+</concept>
diff --git a/docs/topics/impala_sync_hms_events_wait_time_s.xml
b/docs/topics/impala_sync_hms_events_wait_time_s.xml
new file mode 100644
index 000000000..3a30d3e8e
--- /dev/null
+++ b/docs/topics/impala_sync_hms_events_wait_time_s.xml
@@ -0,0 +1,79 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
+<concept rev="5.0.0" id="impala_sync_hms_events_wait_time_s">
+ <title>SYNC_HMS_EVENTS_WAIT_TIME_S Query Option</title>
+ <titlealts audience="PDF"><navtitle>SYNC HMS EVENTS WAIT TIME
S</navtitle></titlealts>
+ <prolog>
+ <metadata>
+ <data name="Category" value="Impala"/>
+ <data name="Category" value="Impala Query Options"/>
+ <data name="Category" value="DDL"/>
+ <data name="Category" value="SQL"/>
+ <data name="Category" value="Developers"/>
+ <data name="Category" value="Data Analysts"/>
+ </metadata>
+ </prolog>
+ <conbody>
+ <p id="IMPALA-12152"><indexterm
audience="hidden">SYNC_HMS_EVENTS_WAIT_TIME_S query option</indexterm>This query
+ option controls the maximum time Impala will wait for the Catalog
Service to sync with the
+ latest events from the Hive Metastore (HMS) before starting query
planning.</p>
+ <p conref="../shared/impala_common.xml#common/type_integer"/>
+ <p conref="../shared/impala_common.xml#common/default_false_0"/>
+ <p><b>Query Range:</b> > = 0</p>
+ <p conref="../shared/impala_common.xml#common/usage_notes_blurb"/>
+ <p>Setting this option to a positive value (in seconds) enables a new
mechanism where only the
+ planning thread of that query will pause and wait for the Catalog
service's event processor to
+ fully apply any outstanding metadata changes from the HMS before the
query is analyzed and
+ planned.<note id="note_sxq_srs_bhc">This mechanism only guarantees
that HMS modifications
+ that occurred before query planning started will be synced by the
Catalog service. Any HMS
+ modifications that happen after the query begins its planning phase
are not guaranteed to be
+ applied.</note><ul id="ul_bh2_vrs_bhc">
+ <li>This is typically used after an external process (like Hive or
Spark) has modified a
+ dependent table, ensuring Impala's query sees the most current
metadata, such as newly
+ added partitions.</li>
+ <li>The default value of 0 disables this waiting mechanism.</li>
+ <li>The wait time could be set based on the maximum observed event
processing lag in your
+ cluster (visible via the Catalogd WebUI /events page).</li>
+ </ul></p>
+ <p><b>Example:</b></p>
+ <p>When a Hive table is updated by an <codeph>INSERT</codeph> operation on
dynamic partitions,
+ you can use the <codeph>sync_hms_events_wait_time_s</codeph> query
option in Impala to ensure
+ the metadata is synchronized before you query the table.</p>
+ <p>Impala waits up to the specified time
(<codeph>sync_hms_events_wait_time_s</codeph>) for Hive
+ Metastore Service (HMS) events to synchronize automatically.</p>
+ <p>The following example demonstrates setting the option to safely query a
Hive table immediately after it is updated. In this case, Impala waits up to
300 seconds for the synchronization to complete.</p>
+ <codeblock>hive> insert into tbl partition(p) select * from tbl2;
+impala> set sync_hms_events_wait_time_s=300;
+impala> select * from tbl;</codeblock>
+ <p>
+ <note id="note_q4b_dqz_ghc" type="note">You do not need to run a
<codeph>REFRESH</codeph>
+ command or wait explicitly on the client side.</note>
+ </p>
+ <p>
+ <b>Added in:</b>
+ <keyword keyref="impala50_full"/>
+ </p>
+ <p conref="../shared/impala_common.xml#common/related_info"/>
+ <p>
+ <xref href="impala_ddl.xml#ddl"/>
+ </p>
+ </conbody>
+</concept>
