Karen Coppage created HIVE-25429:
------------------------------------
Summary: Delta metrics collection may cause number of tez counters
to exceed tez.counters.max limit
Key: HIVE-25429
URL: https://issues.apache.org/jira/browse/HIVE-25429
Project: Hive
Issue Type: Sub-task
Components: Hive
Affects Versions: 4.0.0
Reporter: Karen Coppage
Assignee: Karen Coppage
There's a limit to the number of tez counters allowed (tez.counters.max). Delta
metrics collection (i.e. DeltaFileMetricsReporter) was creating 3 counters for
each partition touched by a given query, which can result in a huge number of
counters, which is unnecessary because we're only interested in n number of
partitions with the most deltas. This change limits the number of counters
created to hive.txn.acid.metrics.max.cache.size*3.
Also when tez.counters.max is reached a LimitExceededException is thrown but
isn't caught on the Hive side and causes the query to fail. We should catch
this and skip delta metrics collection in this case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
