[
https://issues.apache.org/jira/browse/HIVE-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15092956#comment-15092956
]
Alan Gates commented on HIVE-12353:
-----------------------------------
>From comments on CompactionTxnHandler.checkFailedCompactions:
{quote}
* 1. need config var to control retention period in COMPLETED_COMPACTIONS.
Should this
* be system wide or a table prop? Best is to have both. Some tables make
have
* frequent compactions (every 20min) others once a week. Thus a single 24h
retention
* period won't really work. Ideally, we should have a general mechanism
where every (where reasonable)
* property can be specified system wide and overridden per table.
*
* Perhaps instead retention period should be number of runs worth of history?
{quote}
+1 to having it # of runs instead of time based. This is much simpler. If you
wanted to make it per table the way to do it today would be with table
properties. So you could have a system wide configured value and then modify
it in the table properties.
In markFailed, wouldn't it be more efficient to move the row via a "insert into
COMPLETED_COMPACTIONS (x, y, z) SELECT x, y, z from COMPACTION_QUEUE rather
than using your program as a temp store and transmitting the data back and
forth twice?
I'm nervous that in the initiator and cleaner we are marking compactions as
successful (or will be once that bit is done) when they haven't been done. I
see why you're not marking them failed, since they didn't fail either. But I'm
wondering if we need a third state (never ran, or AWOL, or something).
> When Compactor fails it calls CompactionTxnHandler.markedCleaned(). it
> should not.
> -----------------------------------------------------------------------------------
>
> Key: HIVE-12353
> URL: https://issues.apache.org/jira/browse/HIVE-12353
> Project: Hive
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 1.0.0
> Reporter: Eugene Koifman
> Assignee: Eugene Koifman
> Priority: Blocker
> Attachments: HIVE-12353.2.patch, HIVE-12353.patch
>
>
> One of the things that this method does is delete entries from TXN_COMPONENTS
> for partition that it was trying to compact.
> This causes Aborted transactions in TXNS to become empty according to
> CompactionTxnHandler.cleanEmptyAbortedTxns() which means they can now be
> deleted.
> Once they are deleted, data that belongs to these txns is deemed committed...
> We should extend COMPACTION_QUEUE state with 'f' and 's' (failed, success)
> states. We should also not delete then entry from markedCleaned()
> We'll have separate process that cleans 'f' and 's' records after X minutes
> (or after > N records for a given partition exist).
> This allows SHOW COMPACTIONS to show some history info and how many times
> compaction failed on a given partition (subject to retention interval) so
> that we don't have to call markCleaned() on Compactor failures at the same
> time preventing Compactor to constantly getting stuck on the same bad
> partition/table.
> Ideally we'd want to include END_TIME field.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
