Hi Jeremy, Thanks for creating the FLIP! I think this is a pretty useful feature for users to save operation costs. It would be great to have this feature in Flink.
I have a couple of questions below: 1) Could you update the FLIP to specify the concrete signature of the setRackId(..) method proposed in this FLIP? For example, it will be useful to know the signature of the callback expected by this method. 2) Could you explain how are users expected to implement this callback to provide the right rackId at runtime? For example, it will be useful to know how to get the right rack id of the subtask/TM at runtime. 3) Any chance we can provide a default callback implementation that users can use out-of-the-box, instead of asking users to write a non-trivial callback? 4) The Java API proposed in this FLIP allows DataStream users to enjoy this feature. It could be great to also allow Flink SQL users to use this feature. We can probably add an option to the Kafka connector <https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/kafka/#connector-options> so that users can specify it in SQL. This can be done in a separate FLIP if needed. BTW, the Kafka config used by the proposed FLIP, i.e. client.rack, is added in KIP-268 <https://cwiki.apache.org/confluence/display/FLINK/FLIP-268%3A+Kafka+Rack+Awareness>. It might be useful to also reference this KIP in addition to KIP-36. Cheers, Dong On Fri, Nov 11, 2022 at 5:36 AM Jeremy DeGroot <jeremy.degr...@gmail.com> wrote: > Kafak has a rack awareness feature that allows brokers and consumers to > communicate about the rack (or AWS Availability Zone) they're located in. > Reading from a local broker can save money in bandwidth and improve latency > for your consumers. > > This improvement proposes that a Kafka Consumer could be configured with a > callback that could be run when it's being configured on the task manager, > that will set the appropriate value at runtime if a value is provided. > > More detail about this proposal can be found at > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-268%3A+Kafka+Rack+Awareness > > > More information about the Kafka rack awareness feature is at > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-708%3A+Rack+awareness+for+Kafka+Streams > > > Best, > > Jeremy >
