Re: Can we use transform to exchange data between two streams

Hi, So in my example what I notice is that records order is: [1, (KA, AA1)] [2, (KA, AB1)] [3, (KB, B1)] Stream is branched as: input .branch( (k, v) -> //filter records of type AA, (k, v) -> //filter records of type AB, (k, v) -> //filter records of type B, ) Topolo

Understanding concept of grace in JoinWindows

Hi, Reading the kafka docs I see that grace period is defined as: the time to admit late-arriving events after the end of the window I however have not understood as when to use it? If I see that some records are arriving after the end of the window and hence not included in the join, should I no

Central cluster with remote publishers - any downsides?

Hello, Per our deployment topology , we have a central cluster (5 brokers) and remote producers in various geographic locations(#100) (~30 hosts set up in each of geographic location). We will be publishing records at the rate of 1 events/ sec(across all sites) to the cluster. Each event is

Re: [External] Throttling replication when a node is bootstrapping

Okay. Thanks for your help, I'll do some tests. On Fri, Feb 21, 2020, at 23:40, Brian Sang wrote: > I think so? We've always set both leader and follower, like the example on > the page you linked earlier. But I don't see why just setting follower > wouldn't work either. > > On Fri, Feb 21, 2020

Re: [External] Throttling replication when a node is bootstrapping

I think so? We've always set both leader and follower, like the example on the page you linked earlier. But I don't see why just setting follower wouldn't work either. On Fri, Feb 21, 2020 at 2:08 PM Vincent Rischmann wrote: > yeah that's not a bad idea. > > So to recap, it's enough to set follo

Re: [External] Throttling replication when a node is bootstrapping

yeah that's not a bad idea. So to recap, it's enough to set follower.replication.throttled.replicas for every partition assigned to that broker ? I'm assuming that during bootstrap the broker is a follower for everything. On Fri, Feb 21, 2020, at 22:09, Brian Sang wrote: > You can get the list

Re: [External] Throttling replication when a node is bootstrapping

You can get the list of partitions that was on the broker that you replaced. That way you don't need to apply throttle to every topic partition in the cluster. On Fri, Feb 21, 2020 at 4:48 AM Vincent Rischmann wrote: > Hi Brian, > > thanks for the recommendation. I do in fact plan to use autothr

Re: Can we use transform to exchange data between two streams

>From the description it seems there's no repartition topics throughout your topology, in that case the ordering should be guaranteed. I.e. a record from the input topic would only be processed after all previous records from that same topics have been processed entirely; if there's repartition top

Re: kafka-consumer-perf-test error when printing metrics

I forgot to mention I'm using Kafka 2.4.0. Regards, *João Paulo Leonidas Fernandes Dias da Silva aka JP**Lead Consultant - Software Developer* emailjsi...@thoughtworks.com Telephone+1 513 628 7609 [image: ThoughtWorks] O

Some questions about kafka/kafka-connect

Hello all, We’ve been testing out kafka-connect as a possible way of streaming data and have run into some issues. I am a kafka/kafka connect newbie so hopefully these aren’t too basic. I have listed a couple questions below. Thank you 1. Currently our connected-distributed.properties for

Re: [External] Throttling replication when a node is bootstrapping

Hi Brian, thanks for the recommendation. I do in fact plan to use autothrottle (although I need to make it work with our prometheus metrics first). However if I understand correctly, I would need to apply a throttle to essentially every partition of every topic, since in the use case I describe

How to set the log level of Kafka clients?

Hi, How can the log level of the Kafka clients (Producers/Consumers) be set? Is it by using a log4j.properties file? Or, is there a way it can be programmatically set, as well? What are the possible options that are presently available to configure the log level of the clients? Thanks, Pradhan.

Re: Port already in use

I have used this port to emable jmx while starting kafka. Kafka started properly. I can see metrics on kafka manager console. Uptill now its fine. However when I started using utilities under /bin for some testing, those are not running. This has happened because i enable jmx port 9099. When you

Re: Port already in use

Choose a different port or check what's already listening on 9099 using something like: `ss -tunapl | grep 9099` On Fri, Feb 21, 2020 at 1:08 PM sunil chaudhari wrote: > Hi, > I just enlabled the Jmx port in kafka broker. > Since then I am not able to run utilities under /bin > Example when I ru

Listing consumer groups from old versions of consumers

Hello all, I recently had the experience of using the script kafka-consumer-groups.sh from version 2.x of Kafka on a cluster of the same version which was serving consumers using version 0.8 or earlier libraries, hence stored the groups and offsets in Zookeeper. On --list of the consumer groups,

Port already in use

Hi, I just enlabled the Jmx port in kafka broker. Since then I am not able to run utilities under /bin Example when I run ./kafka-topics.sh —create Then it throws bindException port already in use 9099 Before it was running. Same thing happening for perf test utilities under /bin. Please

Re: [External] Re: Urgent helep please! How to measure producer and consumer throughput for single partition?

Hi Sunil, You'll have to write the object name in a manner that Metricbeats understand. I'd suggest using JConsole to explore a running app on the JVM that has JMX enabled and a producer in it, to get an understanding of the object names you need to use in your Metricbeat conf. Cheers, Liam Clar