mikemccand commented on PR #13190:
URL: https://github.com/apache/lucene/pull/13190#issuecomment-2009413290
Phew, it took me two hours of strongly caffeinated time to catch up on this!
Thank you @benwtrent.
What an awesome change, modernizing Lucene's merge concurrency so that
intra-merge (just HNSW, but maybe other index parts soon) concurrency is
enabled by default.
(Separately, the nightly benchy had a silly off-by-one bug preventing it
from resuming after you reverted the previous PR ... I think I've fixed that
bug now, and kicked off a one-off nightly benchy run that will hopefully finish
late today).
Some high level questions (sorry if these were already asked/answered):
* How do we control the risk that a massive merge with KNN vectors soaks
up all available concurrency from the shared Executor for intra-merge
concurrency (all threads doing HNSW merging) and then starves smaller merges
that would finish quickly?
* Maybe we don't need the IO write rate limiter anymore? It's a tricky
setting because the `mbPerSec` you set is then multiplied by the number of
concurrent merges that are running. It is a per-merge setting, not a global
setting, so it's kinda trappy today.
Finally, I worry that the rate-limiter's simplistic "instantaneous" measure
is making it effectively super buggy (there is a TODO about this), because
merging that does a lot of CPU work and then writes a lot of bytes will be
effectively throttled to far below the target `mbPerSec` in aggregate. A
better solution might be something like the "burst IOPs bucket" that AWS (and
likely other cloud providers) offer ("Burst IOPS is a feature of Amazon Web
Services (AWS) EBS volume types that allows applications to store unused IOPS
in a burst bucket and then drain them when needed" -- thank you Gemini for the
summary). This is clearly not a blocker for this issue, but we really should
(separately) fix it. I'll open a follow-on issue about this ... but to me it's
another reason to maybe remove this feature entirely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
