Jason-liujc commented on issue #11535:
URL: https://github.com/apache/hudi/issues/11535#issuecomment-2214835602
Had an offline discussion with Shiyan.
As long as the metadata table is not compacted properly, the insertion
performance will become worse and worse gradually.
Here’s some action items we are taking:
1. For future Hudi issues, we’ll try to create github issues first. I’ll
create another one for some incremental query errors (but its totally mitigable
on our end)
2. For this specific issue on metadata table not being compacted, we’ll try
the following
a. Run scripts to delete previous uncommitted instants (and any files
created if any) and see if the metadata compaction resumes
b. Run workload with synchronous cleaning to see if it can compact the
metadata table
c. After cleaning up pending commits, see if we successfully reinitialize
the metadata table
Will give an update here on how it goes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
