bvaradar commented on issue #1830: URL: https://github.com/apache/hudi/issues/1830#issuecomment-659044105
From the highlighted section in your spark UI image, it looks like there is an increase during index lookup. Between 2 runs, there is an increase of 10 sec (around 4%). Is this the same pattern you were seeing in earlier runs. Also, I am not able to clearly see the difference between subsequent runs w.r.t total processing delay. My question is : Are you seeing more than 10 secs difference in total processing delay between subsequent runs ? If so, can you try 0.5.3 release where we fixed driver side performance issues and we have heard confirmation from users regarding speedup. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
