sivabalan narayanan created HUDI-2774:
-----------------------------------------
Summary: Async Clustering via deltstreamer fails with
IllegalStateException: Duplicate key
[==>20211116123724586__replacecommit__INFLIGHT]
Key: HUDI-2774
URL: https://issues.apache.org/jira/browse/HUDI-2774
Project: Apache Hudi
Issue Type: Bug
Reporter: sivabalan narayanan
Attachments: Screen Shot 2021-11-16 at 12.42.20 PM.png
Setup:
Started deltastreamer with parquet dfs source. source folder did not have any
data as such. Enabled async clustering with below props
```
hoodie.clustering.async.max.commits=2
hoodie.clustering.plan.strategy.sort.columns=type,id
```
Added 1 file to the source folder. and deltastreamer failed during this. commit
went through fine. 1st replace commit got scheduled. here is my hunch : and I
guess in subsequent round of deltastreamer sync, it tries to schedule again and
fails.
clustering plan seems to be same in both requested meta files
{code:java}
grep "2b9b3f9d-f68c-4404-8352-1708089d2cca-0_13-49-202_20211116123721000"
/tmp/hudi-deltastreamer-gh-mw/.hoodie/* | grep replacecommit
grep: /tmp/hudi-deltastreamer-gh-mw/.hoodie/archived: Is a directory
Binary file
/tmp/hudi-deltastreamer-gh-mw/.hoodie/20211116123724586.replacecommit.requested
matches
Binary file
/tmp/hudi-deltastreamer-gh-mw/.hoodie/20211116123725199.replacecommit.requested
matches {code}
timeline
!Screen Shot 2021-11-16 at 12.42.20 PM.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
