errose28 commented on PR #8243:
URL: https://github.com/apache/ozone/pull/8243#issuecomment-2787815473
The bulk of this change looks related to parallel iteration, not parallel
deserialization and consumption. I would recommend:
1. Initial PR to implement general purpose serial iteration with
deserialization and consumption pushed off to a thread pool.
2. If benchmarks show significant improvements, add support for
parallelizing the iteration step using a different thread pool from the
consumers.
- This would probably be good to expose as a parallel stream, for
example by implementing it as a `Spliterator` and returning a stream from
[StreamSupport#stream](https://docs.oracle.com/javase/8/docs/api/java/util/stream/StreamSupport.html#stream-java.util.Spliterator-boolean-).
3. Migrate each use case to the new implementation in individual PRs
- In HDDS-11777 we would probably create a wrapper around this utility
to allow processing all locations in an FSO tree for debug or repair
efficiently cc @dombizita
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
