tandonraghav commented on issue #2151: URL: https://github.com/apache/hudi/issues/2151#issuecomment-706400445
@bvaradar Thanks for helping here. But still issue is not solved. The Freshness of data is coming as a big concern. Probably i am missing something here. - If I run Compaction after every X min, we should get fresh data till X min. (irrespective of number of delta commits) - Putting `Inline.compact=false and delta.commits=1 and async.compact=true`, is not helping out this is generating Compaction request only for 1 delta commit. If a compaction request exists then it is not generating compaction request after every df.write(). - Also, as suggested we cannot run df.write() and `Option<String> instantTime=client.scheduleCompaction(Option.empty());` concurrently. The problem still remains same tht if I have delta.commits=5 and i have only 3 delta commits and I run compaction then i would not get fresh data and compaction request would not even kick in. How do i solve this problem? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
