[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745713#comment-16745713
]
Eugene Koifman commented on HIVE-21052:
---------------------------------------
I left some comments on RB.
I think the way Cleaner manages concurrency is not quite right.
Currently there is 1 Cleaner per HMS. You can have several HMS instances in
the cluster for HA. Eac Cleaner run is mutexed via {{handle =
txnHandler.getMutexAPI().acquireLock(TxnStore.MUTEX_KEY.Cleaner.name());}} so
only 1 is actually running at a time.
The (new) Cleaner seems to parallelize tasks too early and then has to mutex on
the HMS access.
I would suggest resolving the paths first and then enqueue parallel tasks into
the Priority queue to just to the deletes. I would make sure that 2 clean
operations of the same partition should not be allowed, nor 2 table level
cleans. (I'm not sure if table clean could run concurrently with partition
level clean of the same table - I suspect yes if {{markCleaned()}} is such that
the table clean and partition clean remove disjoint sets of TXN_COMPONENTS
entries. For 1st pass, I'd disallow it)
You could keep a (Concurrent) Map of locks which is thrown away at the end of
Cleaner.run(). the locks are either named after Table or Partition. To
acquire Partition level lock you 1st have acquire table level lock. This way
each {{CleanWork}} work can runs separately as long as it's not violating above
rules. In other words, cleans that are guaranteed to work on entities that are
not the same/related run in parallel - otherwise in sequence.
I suspect it may be useful to see if {{findReadyToClean()}} returns a very long
list it may be useful to create several RawStore connections to do the
'resolve' operations in parallel but I'd say this is pass 2 or later. This
would actually allow these to run in parallel.
Let me know what you think.
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -------------------------------------------------------------------------------------
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 3.0.0
> Reporter: Jaume M
> Assignee: Jaume M
> Priority: Critical
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
> HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch
>
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
