Re: callback function

Hello, when you say "something is new in the queue" you mean a new message/record is available in a partition within a topic? Cheers! On Tue, Jan 7, 2020, 8:46 PM Tavares Forby wrote: > Hi, > > Is there a method in which a function can wake when something is new in > the queue? This has to be n

Re: High CPU Usage on Brokers

Hi Navneeth like the bug you said above, do you set sun.security.jgss.native = true? if not, there are some items need to be check. 1. GC, but you say gc is not problem 2. if you suspect network thread, how many thread did you set? 3. if you enable compression 4. did you change the value of bat

Re: callback function

Tavares, if you're asking about the consumer then I think you might have a misconception about how it works: The application calls poll() to fetch the latest records from the broker(s). The broker is not pushing records into some queue in the consumer. It might be worth reading https://kafka.apache

MirrorMaker 2 - Does it write anything to source cluster?

Hi, so I am planning to use MM2 and was thinking if it has any impact on the source cluster when mirroring. Obviously it impacts the performance of the source cluster, so I plan to use quotas to solve that, but other than that, Does MM2 write anything back to the source cluster? As I understand

Re: callback function

Hey Tavares, On Wed, 8 Jan 2020 at 09:38, Tom Bentley wrote: > Tavares, if you're asking about the consumer then I think you might have a > misconception about how it works: The application calls poll() to fetch the > latest records from the broker(s). The broker is not pushing records into > s

Sporadic kafka API timeouts during consumers lag fetching

Hello, All, We are observing strange behavior on one of our kafka clusters (version 2.2.0). Sporadically, kafka admin API returns "*Consumer group 'consumer_name' has no active members.*" when we try to fetch consumer offset via ./kafka-consumer-groups.sh. Issue comes back to normal in a second a

Re: MM2 startup delay

Hey, I did some further debugging on this. I say that the following happened just before the real huge mirroring traffic starts: [2020-01-08 10:48:51,926] INFO Starting with 2294 previously uncommitted partitions. (org.apache.kafka.connect.mirror.MirrorSourceTask:94) [2020-01-08 10:48:51,928] IN

Re: MirrorMaker 2 - Does it write anything to source cluster?

Peter, MM2 writes offset syncs upstream to the source cluster, which are then used to emit checkpoints to the target cluster. There is no particular reason why offset syncs are stored on the source cluster instead of the target, and it's been suggested that we swap that around. Ryanne On Wed, Jan

MirrorMaker 2 throttling

Hi, I'd like to throttle the mirroring process when I start Mirror Maker 2 at the first time, so it starts to pull all the messages that exists on the source cluster. I'd like to do this only to avoid putting too much traffic on the source cluster that may slow down existing production client on i

Secured communication between Kafka and Zookeeper

Kafka version: 2.3.0 Zookeeper version: 3.5.5 Hi! I'm trying to keep all communication secure in my test cluster, but somehow I'm unable to get Kafka->Zookeeper connection using SSL. If I don't open the "clientPort" next to "secureClientPort" I get: Zookeeper: WARN > [epollEventLoopGroup-7-4:

Re: MirrorMaker 2 throttling

Peter, have you tried overriding the client ID used by MM2's consumers? Otherwise, the client IDs are dynamic, which would make it difficult to throttle using quotas. Ryanne On Wed, Jan 8, 2020, 10:12 AM Péter Sinóros-Szabó wrote: > Hi, > > I'd like to throttle the mirroring process when I star

maximum lag between current offset, and last commit

I liked this article: https://blog.newrelic.com/engineering/kafka-consumer-config-auto-commit-data-loss/ I particularly like the illustration in the article: http://newrelic-wpengine.netdna-ssl.com/wp-content/uploads/offset_explained2.jpg What is the maximum difference between the current offset i

Need Help For Kafka Android Clients

Dear Concerned,​ Hope you are doing well. We wanted to create Apache Kafka Producer and Consumer Client for our Android app. But we are getting this error that the Management Factory is not present in Android. Is there any alternative way we can use for that? If we can, will you suggest any a

Consuming from N topics "co-partitioned"

Hi everybody, I wish to implement a KafkaConsumer that consumes messages from N co-partitioned topics, in a similar way to KafkaStreams for join semantic. e.g. for two consumers and two topics with two partitions: C0 = {T0P0, T1P0} C1 = {T0P1, T1P1} ... Any ideas/tips? I assume most of the work

RE: callback function

Sorry, not queue. I guess this is different than rabbitmq. Kafka consumer pulls data from the broker. There’s probably no event triggers to call a function. [Kafka Architecture - Kafka Zookeeper Coordination Diagram] From: Tom Bentley Sent: Wednesday, January 8, 2020 1:38 AM To: Users Cc: T

Re: maximum lag between current offset, and last commit

Perhaps my understanding is off. https://docs.confluent.io/current/installation/configuration/broker-configs.html Perhaps this is the parameter I am looking for? offsets.retention.check.interval.ms Perhaps the re-send behavior is controlled by time, and not the maximum distance between the current

Re: High CPU Usage on Brokers

Has the behavior changed after an upgrade or has it been consistent since the start? Ismael On Thu, Jan 2, 2020 at 4:18 PM Navneeth Krishnan wrote: > Hi All, > > We have a kafka cluster with 12 nodes and we are pretty much seeing 90% > cpu usage on all the nodes. Here is all the information. Ne

Re: complicated logic for tombstone records

Hi Boyang, Hi Alex, thank you for your reply. I can't use windowing so currently I'm managing removals by wrapping messages in a delete-aware wrapper whenever I have to do aggregation but this has a big impact on all the logic. For me the ideal situation would be to get a handle on the state stor

Free Kafka Stream Data

Hi , I am trying to learn Kafka. Is there any free API which I can use like twitter? I created twitter account but looks like ti will take days before I can use their streaming data . Thanks Priyanka

Re: Free Kafka Stream Data

Priyanka, On Wed, 8 Jan 2020 at 20:42, cool girl wrote: > Hi , > > I am trying to learn Kafka. Is there any free API which I can use like > twitter? I created twitter account but looks like ti will take days before > I can use their streaming data . > Welcome to Kafka. If you are seeking a RE

Re: Free Kafka Stream Data

Hi, This blog shows using Twitter as a source for Kafka: https://www.confluent.io/blog/stream-processing-twitter-data-with-ksqldb/ The credentials necessary for that Twitter API are created instantly AFAIK. If you just want messages to be generated in Kafka have a look at https://www.confluent.io

Re: Free Kafka Stream Data

Thanks. On Wed, Jan 8, 2020, 1:16 PM Robin Moffatt wrote: > Hi, > > This blog shows using Twitter as a source for Kafka: > https://www.confluent.io/blog/stream-processing-twitter-data-with-ksqldb/ > The credentials necessary for that Twitter API are created instantly AFAIK. > > If you just want

Re: Consuming from N topics "co-partitioned"

> I assume most of the work must be done at PartitionAssignor level, Correct. > there is >> any implementation of it that is able to assign partitions like this? Well, you can look into `StreamsPartitionAssignor` (but it does much more than just co-partitioning, hence, it might be hard to extra

Re: about default value of struct of connect api

update my findings, below is code of method 'defaultValue': public SchemaBuilder defaultValue(Object value) { checkCanSet(DEFAULT_FIELD, defaultValue, value); checkNotNull(TYPE_FIELD, type, DEFAULT_FIELD); try { ConnectSchema.validateValue(this, value);

Does Kafka Support IPV6

hello：     I want to make sure that Kafka supports IPV6, and that IPV6 is likely to be used in the near future. Thank you bowu lu vastdata.com.cn

java.net.BindException: Address already in use (Bind failed) with kafka-topics command

Could you please let me know why sometimes I am getting, below exception? Cluster is up and running. Kafka and ZK is running in same server. We also have jmx exporter configured for monitoring. kafka-topics --zookeeper localhost:2181 --list Error: Exception thrown by the agent : java.rmi.server.

Re: java.net.BindException: Address already in use (Bind failed) with kafka-topics command

Hello, Looks like your server is running out of ephemeral ports. The below command confirms it is running out of ephemeral ports. We can restart the process to solve the issue. Must probably root cause of the issue will be wrong configuration of mirror maker process / it is a bug. netstat -an |

RE: java.net.BindException: Address already in use (Bind failed) with kafka-topics command

we don’t have mirrormaker configured in this cluster. Please find output of netstat command. netstat -an | wc -l 570322 -Original Message- From: kalai selvan Sent: Wednesday, January 8, 2020 11:41 PM To: users@kafka.apache.org Subject: Re: java.net.BindException: Address already in us

Re: java.net.BindException: Address already in use (Bind failed) with kafka-topics command

Hello, Please refer the documentation for increasing the ephemeral ports. Mostly it will solve the issue. https://www.google.com/amp/s/www.cyberciti.biz/tips/linux-increase-outgoing-network-sockets-range.html/amp Regards, Kalai On Thu, 9 Jan, 2020, 11:14 AM JOHN, BIBIN, wrote: > we don’t hav