Thanks a lot Guozhang. Will give it a try. Thanks, Tony
On Tue, Oct 20, 2020 at 10:42 PM Guozhang Wang <wangg...@gmail.com> wrote: > Hello Tony, > > I think you already know the consumer-client side fetch-from-follower > feature: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica > > Say your Kafka deployment is cross-AZs, and your Streams is also deployed > cross-AZ, you can, by setting the "client.rack" through streams configs, to > let the embedded consumers to read from close replicas within the same AZ > as well. > > Guozhang > > > On Mon, Oct 19, 2020 at 9:19 PM Tony John <tonyjohnant...@gmail.com> > wrote: > > > Thanks Matthias.. I don't think I will be able to take it up.. Will wait > > for it to be available in future. :) > > > > On Mon, Oct 19, 2020 at 11:56 PM Matthias J. Sax <mj...@apache.org> > wrote: > > > > > No suggestions... Sorry. > > > > > > The task assignment algorithm cannot be customized. > > > > > > The only thing you _could_ do, is to pickup the ticket yourself (to get > > > the feature maybe in 2.8 release). Not sure if you would be interested > > > to contribute :) > > > > > > > > > -Matthias > > > > > > On 10/19/20 11:08 AM, Tony John wrote: > > > > Thanks for the quick response Matthias. We were planning to move to > AWS > > > > MSK, and I think we will have to defer it until rack awareness is in > > > place > > > > for Kafka Streams. Reason being consumers consuming from brokers in > > > > different availability zones would result in high data transfer > > charges. > > > Do > > > > you have any suggestions? > > > > > > > > Thanks, > > > > Tony > > > > > > > > On Mon, Oct 19, 2020 at 11:08 PM Matthias J. Sax <mj...@apache.org> > > > wrote: > > > > > > > >> There is no such feature for Kafka Streams yet. > > > >> > > > >> We have one ticket for rack aware standby task placement > > > >> (https://issues.apache.org/jira/browse/KAFKA-6718) but nobody is > > > working > > > >> on it. > > > >> > > > >> > > > >> -Matthias > > > >> > > > >> On 10/19/20 9:20 AM, Tony John wrote: > > > >>> Hi All, > > > >>> > > > >>> I have been trying to figure out some documentation on how to > enable > > > rack > > > >>> awareness on a Kafka Streams app. I do see the broker.rack > > > configuration > > > >>> which needs to be done at the broker side and client.rack > > configuration > > > >> for > > > >>> the consumers. Is there any specific configuration which is > required > > to > > > >>> enable rack awareness in a streams app? Please throw some light on > > > this. > > > >>> > > > >>> PS: I am using Kafka / Kafka Streams 2.2.1 > > > >>> > > > >>> Thanks, > > > >>> Tony > > > >>> > > > >> > > > > > > > > > > > > -- > -- Guozhang >
