mikemccand commented on issue #13193: URL: https://github.com/apache/lucene/issues/13193#issuecomment-2025290139
> > The instant approach we take today gives no credit for a longish period of time when no/few bytes were written. > > My mental model is that the goal of merge throttling is to prevent merges from starving searches from IOPS. So the current approach makes sense from this perspective: if we gave merges credits that would allow them to max out IOPS from time to time, then this could temporarily starve searches from IOPS, which could be extremely disruptive to searches? Yeah that is a good point -- the instant IOPS throttle we use today might better protect concurrent searching, at the harm of slower merging. Though, with the OS typically doing readahead (on the N files currently being read for merging), and write buffering, these OS level caching operations do "soak up" the time when the merge thread is not doing IO, and so the OS can get ahead of the merge CPU threads that are consuming / producing the bytes? And so maybe in practice the OS is already "preparing" for the merge credits effectively and it wouldn't really hurt concurrent searching? And, on a box doing concurrent searching, those bytes being read for merge may already be hot since those merging segments are lit for searching? I dunno, it's complex ;) > But then, I don't recall ever seeing indexing/merging max out I/O on a modern drive, so your second suggestion of removing I/O throttling makes sense to me, and I wouldn't mind removing all the complexity around I/O throttling. Presumably, we would still want to have some mechanisms to prevent merges from using all CPU for use-cases when indexing/merging and search are co-located on the same host? I suspect merging is mostly CPU bound these days? Except probably the "bulk copy merge" that we can sometimes do for merging stored fields? And if that's the case, can't the application limit CPU consumption for merging by limiting number of merges / number of merge threads? Hmm I forgot that CMS's "auto IO throttle" is enabled by default ... we could also explore disabling it by default (maybe for 10.0) and noting that in `MIGRATE.md`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
