[
https://issues.apache.org/jira/browse/HIVE-30026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marta Kuczora updated HIVE-30026:
---------------------------------
Description:
After HIVE-27444, if the number of candidates is bigger than the
compactor.fetch.size value, it can happen that the Initiator doesn't schedule
all tables for compaction.
For example: we have 4 tables, which are all eligible for compaction, but the
fetch size is 2.
In the first initiator run, it will pick two of these tables. It would be
expected that during the second run of the initiator it would pick the other
two tables, so all 4 tables have been scheduled for compaction.
Since there is no ordering when selecting the tables for compaction, this was
not always the case. It could happen that some tables never got compacted.
Also if the number of candidates is bigger than the compactor.fetch.size limit,
the aborted txn candidates will never be picked up by the initiator. Because
the "remaining budget" will always be 0 in the
FindPotentialCompactionsFunction.
{code:java}
Set<CompactionInfo> candidates = new HashSet<>(jdbcResource.execute(new
CompactionCandidateHandler(lastChecked, fetchSize)));
int remaining = fetchSize - candidates.size();
if (collectAbortedTxns && remaining > 0) {
candidates.addAll(jdbcResource.execute(new
AbortedTxnHandler(abortedTimeThreshold, abortedThreshold, remaining)));
}{code}
This affects only the cases when the aborted txn clean-up is done by the
compaction cycle and not just the cleaner.
> (ACID Compaction) Extend the query for finding potential compactions with
> ordering
> ----------------------------------------------------------------------------------
>
> Key: HIVE-30026
> URL: https://issues.apache.org/jira/browse/HIVE-30026
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Reporter: Marta Kuczora
> Assignee: Marta Kuczora
> Priority: Major
>
> After HIVE-27444, if the number of candidates is bigger than the
> compactor.fetch.size value, it can happen that the Initiator doesn't schedule
> all tables for compaction.
> For example: we have 4 tables, which are all eligible for compaction, but the
> fetch size is 2.
> In the first initiator run, it will pick two of these tables. It would be
> expected that during the second run of the initiator it would pick the other
> two tables, so all 4 tables have been scheduled for compaction.
> Since there is no ordering when selecting the tables for compaction, this was
> not always the case. It could happen that some tables never got compacted.
>
> Also if the number of candidates is bigger than the compactor.fetch.size
> limit, the aborted txn candidates will never be picked up by the initiator.
> Because the "remaining budget" will always be 0 in the
> FindPotentialCompactionsFunction.
> {code:java}
> Set<CompactionInfo> candidates = new HashSet<>(jdbcResource.execute(new
> CompactionCandidateHandler(lastChecked, fetchSize)));
> int remaining = fetchSize - candidates.size();
> if (collectAbortedTxns && remaining > 0) {
> candidates.addAll(jdbcResource.execute(new
> AbortedTxnHandler(abortedTimeThreshold, abortedThreshold, remaining)));
> }{code}
> This affects only the cases when the aborted txn clean-up is done by the
> compaction cycle and not just the cleaner.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
