[
https://issues.apache.org/jira/browse/HIVE-25429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage resolved HIVE-25429.
----------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
Committed to master. Thanks [~lpinter] for the +1 and [~dkuzmenko] for the
feedback!
> Delta metrics collection may cause number of tez counters to exceed
> tez.counters.max limit
> ------------------------------------------------------------------------------------------
>
> Key: HIVE-25429
> URL: https://issues.apache.org/jira/browse/HIVE-25429
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
> Affects Versions: 4.0.0
> Reporter: Karen Coppage
> Assignee: Karen Coppage
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> There's a limit to the number of tez counters allowed (tez.counters.max).
> Delta metrics collection (i.e. DeltaFileMetricsReporter) was creating 3
> counters for each partition touched by a given query, which can result in a
> huge number of counters, which is unnecessary because we're only interested
> in n number of partitions with the most deltas. This change limits the number
> of counters created to hive.txn.acid.metrics.max.cache.size*3.
> Also when tez.counters.max is reached a LimitExceededException is thrown but
> isn't caught on the Hive side and causes the query to fail. We should catch
> this and skip delta metrics collection in this case.
> Also make sure that metrics are only collected if
> hive.metastore.acidmetrics.ext.on=true
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
