FrankChen021 commented on issue #18439: URL: https://github.com/apache/druid/issues/18439#issuecomment-4294451459
> Thanks for checking, actually having a separate index, task runner and supervisor make it easier to implement since in Share group (kafka queue), the broker handles distribution of records, no partition-task assignment or offset management at client end required those are done in Kafka broker side. > > Also these relaxation come up with cost of less control over record lifecycle management, replay. I know it's mainly for easier implementation. But from product side or user's view, it's not a good design. I think maybe we need to some refactoring on current `KafkaIndexTask` to be facade class, and: 1. create a new NonSharedQueueKafkaIndexTask, which is exactly current implementation 2. create a new SharedQueueKafkaIndexTask, which is the one your want to implement 3. the KafkaIndexTask holds either instance of NonSharedQueueKafkaIndexTask or SharedQueueKafkaIndexTask based on the "type": "kafka_share_group" defined in the `ioConfig`, and delegate all public methods to these instance By this way, enabling or disabling this feature is only gated by this type property inside ioConfig. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
