Re: Custom Kafka Connect REST Extension for Token Authentication

Hi All,  Has anyone managed to get a REST extension working with multiple connect instances / workers? Many Thanks,  Jamie On Wednesday 16 April 2025 at 10:57:01 BST, Jamie wrote: Hi All,  I'm trying to implement a custom REST extension for Kafka Connect that allows a token

Custom Kafka Connect REST Extension for Token Authentication

nyone been able to get either the basic authentication or another custom authentication extension working with Kafka connect when using more than one instance / worker?  I'm using version 2.7.1 of Kafka and Kafka connect.  Many Thanks,  Jamie.

Re: Kafka consumer filtering

Hi abdelali, If you can’t get your producers to send the different types of events to different topics (or you don’t want to) you could use Kafka streams to filter the data in the topic to new topics that are subsets of the data.  I have also seen apache spark used to do similar. Thanks, Jamie

Re: Help on MM2 -- only copy new data

offset that is valid. Since when you stated MM2 with below config it will now have an offset stored in the offset storage topic for each topic partition being consumed and therefore the auto.offset.reset config won’t be applied. Thanks, Jamie Sent from the all-new AOL app for iOS On Tuesday

Re: MM2 - Overriding MirrorSourceConnector Consumer & Producer values

Hi All,  Has anyone been able to override the producer and consumer values of the MM2 MirrorSourceConnector as described below? Many Thanks,  Jamie -Original Message- From: Jamie To: users@kafka.apache.org Sent: Thu, 16 Sep 2021 15:52 Subject: MM2 - Overriding MirrorSourceConnector

MM2 - Overriding MirrorSourceConnector Consumer & Producer values

. Am I trying to override the values of the consumer incorrectly or do I need to set these in a different place? Many Thanks,  Jamie 

Mirror Maker 2 - High Throughput Identity Mirroring

essing the data only for the MM2 producer to then re-compress it again. Has this feature been implemented yet in MM2? Many Thanks, Jamie 

Unable to produce output from any non-trivial examples

I'm trying to run some very simple examples with kafka and running into a strange problem. I've documented this in detail in https://issues.apache.org/jira/browse/KAFKA-12594#. The linked repo - https://github.com/jamii/streaming-consistency/tree/main/kafka-streams - should be runnable on any

Re: MM2 Message Handlers

Hi Ryanne, Great, thanks  Jamie Sent from AOL Mobile Mail Get the new AOL app: mail.mobile.aol.com On Monday, 18 May 2020, Ryanne Dolan  wrote: Jamie, MM2 uses Connect, so you can use Connect's SMTs for the same effect. Ryanne On Mon, May 18, 2020, 6:17 AM Jamie wrote: > Hi All, &g

MM2 Message Handlers

Hi All, Does MM2 currently support message handlers, the same way MM1 did? If not, are there any plans to support this in future? Many Thanks,  Jamie

Re: Problems when Consuming from multiple Partitions

, Jamie Sent from AOL Mobile Mail Get the new AOL app: mail.mobile.aol.com On Sunday, 8 March 2020, James Olsen  wrote: Using 2.3.1 Brokers makes things worse.  There are now 2 fetch.max.wait.ms delays before messages are delivered even though they were available at the beginning. 2020-03-09 11:40

Running MM2 In Connect Cluster

.topic.replication.factor" "1",                "offset-syncs.topic.replication.factor: "1",                "target.security.protocol": "SASL_SSL",                "target.sasl.kerberos.service.name": "kafka",                "target.sasl.jaas.config": "com.sun.security.auth.module.lrb5LoginModule required useTicketCache=true"        }} At the moment I'm just trying to replicate the "testing1" topic from the source to the target cluster. The connector starts successfully but I can't replicate from source to target.  Firstly, if someone could confirm this is the correct way in terms of configuration to run MM in a connect cluster that would be great then I can describe the issues I'm seeing.  Many Thanks,  Jamie

Re: Config producer for kafka connect

Hi Patrick, You need to use the producer prefix in your connect.distributed properties file. The same file you have your Kafka connect config in which applies it to all producers. Thanks,   Jamje Sent from AOL Mobile Mail Get the new AOL app: mail.mobile.aol.com On Thursday, 6 February 2020, Pat

Re: Kafka consumers freeze

Hi Tim, Can you use the “bytes-consumed-rate” jinx metric which reports the bytes consumed per topic?  What errors are you seeing in the consumer when it freezes? Thanks, Jamie  Sent from AOL Mobile Mail Get the new AOL app: mail.mobile.aol.com On Friday, 31 January 2020, Sullivan, Tim  wrote

Re: High CPU in 2.2.0 kafka cluster

hanks,  Jamie -Original Message- From: Brandon Barron To: users@kafka.apache.org Sent: Thu, 30 Jan 2020 16:11 Subject: High CPU in 2.2.0 kafka cluster Hi, We had a small cluster (4 brokers) dealing with very low throughput - a couple hundred messages per minute at the very most. In that cl

Kafka 2.4.0 & Mirror Maker 2.0 Error

isolation.DelegatingClassLoader@187eb9a8 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) I've checked the jar file containing these class file is in the class path.  Is there anything I need to add to plugin.path for the connect properties when running mirror maker? Many Thanks,  Jamie

Re: Kafka consumer group keeps moving to PreparingRebalance and stops consuming

sumer (max.poll.records) could take longer than the time configured for max.poll.interval.ms? Maybe you could log when the records are returned to the streams task and then when the records have finished processing to determine how long this normally takes?   Thanks,  Jamie -Original Me

Calculating Required Number of Fetch sessions

consumers than fetch session cache slots. Is there a formula for working out the number of fetch sessions required? Maybe: num_replica_fetchers_per_broker + (num_consumers / num_brokers)  Many thanks, Jamie 

Re: Producer send blocking when destination partition does not exist

Hi Sean, Out of interest, is auto topic creation enabled on the brokers? Thanks, Jamie -Original Message- From: Sean Glover To: users Sent: Mon, Nov 4, 2019 04:21 PM Subject: Producer send blocking when destination partition does not exist Hi, I accidentally created a scenario where

Re: Monitor Kafka connect jobs

is failed on any (or define threshold of an acceptable number of failures ) Hope this helps,  Jamie -Original Message- From: KhajaAsmath Mohammed To: users Sent: Thu, Oct 24, 2019 03:52 PM Subject: Monitor Kafka connect jobs Hi, We are using kafka connect in production and

Re: Re: Keep getting expiring X record(s) for changelog topic:326554 ms has passed since batch creation

Have you considered increasing max.block.ms or buffer.memory? Thanks,  Jamie  -Original Message- From: Matthias J. Sax To: users Sent: Thu, Oct 24, 2019 06:35 AM Subject: Re: Keep getting expiring X record(s) for changelog topic:326554 ms has passed since batch creation I would

Re: poor producing performance with very low CPU utilization?

Hi Eric,  I found increasing the linger.ms to between 50-100 ms significantly increases performance (fewer larger requests instead of many small ones), I'd also increase the batch size and the buffer.memory.  Thanks,  Jamie -Original Message- From: Eric Owhadi To:

Re: Kafka Broker - High CPU

Have you got any producers using an older version of Kafka? Does the broker with high CPU usage contain the leader of any topics which don't have leaders on other brokers? Is the server.properties file the same across all brokers (minus broker.id etc)?  Many Thanks,  Jamie -Ori

Re: Truncation

Hi Guozhang,  Thanks very much, this was really useful. Jamie -Original Message- From: Guozhang Wang To: users Sent: Tue, 23 Jul 2019 2:40 Subject: Re: Truncation Hi Jamie, The most relevant materials I can think of would be in KIP-101: https://cwiki.apache.org/confluence/display

Truncation

Hi All, Could someone confirm what truncation happens when a partition changes from a follower to a leader and why?  Any help would be greatly appreciated. Many Thanks, Jamie

Re: Kafka logs are getting deleted too soon

factor of the topic? If you have less than 3 brokers, have you set offsets.topic.replication.factor to the number of brokers?  Thanks,  Jamie -Original Message- From: Sachin Nikumbh To: users Sent: Wed, 17 Jul 2019 20:21 Subject: Re: Kafka logs are getting deleted too soon Broker

Re: Broker Startup Time (Prolonged URPs)

Hi All,  Does anyone have any thoughts on this? Many Thanks,  Jamie -Original Message- From: Jamie To: dev ; users Sent: Mon, 8 Jul 2019 18:38 Subject: Broker Startup Time (Prolonged URPs) Hi All, I'm currently doing some work into investigating why our Kafka deployment

Broker Startup Time (Prolonged URPs)

you have more replica fetcher threads then each thread is responsible for setting up less partitions?  Any help with the above would be greatly appreciated.  Many Thanks,  Jamie 

Disable anonymous access to zookeeper

ndering if I disabled this, would it impact any Kafka operations. I am only using a single node Kafka (no cluster). Would appreciate any information or pointers on how to proceed with this or any particular documentation I should read. Thanks I advance for your help. Jamie

Broker failure causes some producers to hang for about 15 minutes

We are using Kafka 0.8.2.1 and the old producer. When there is an issue with a broker machine (we are not completely sure what the issue is, though it generally looks like host down), sometimes it takes some of the producers about 15 minutes to time out. Our producer settings are request.required.a

RE: Support https or ssl

. In this manner, just modify my current Kafka produce client to issue REST API calls instead writing to the socket direct. Am I getting this right? Thanks!! Jamie From: Sriharsha Chintalapani [mailto:harsh...@fastmail.fm] Sent: Friday, May 08, 2015 2:55 PM To: Jamie Wang; users@kafka.apache.org

Re: Compression and batching

at 9:57 AM, Jamie X wrote: > (sorry if this messes up the mailing list, I didn't seem to get replies in > my inbox) > > Jiangjie, I am indeed using the old producer, and on sync mode. > > > Notice that the old producer uses number of messages as batch limitation >

Re: Compression and batching

regardless of whether it would exceed a size limit? If so that may explain things. Thanks, Jamie On Tue, May 12, 2015 at 4:40 PM, Jamie X wrote: > Hi, > > I'm wondering when you call kafka.javaapi.Producer.send() with a list of > messages, and also have compression on (snappy in

Compression and batching

:98) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) Thanks, Jamie

RE: Support https or ssl

available for GA? Thanks again. Jamie -Original Message- From: Sriharsha Chintalapani [mailto:ka...@harsha.io] Sent: Thursday, May 07, 2015 7:22 PM To: Jamie Wang; users@kafka.apache.org Subject: Re: Support https or ssl Hi Jamie,         I am currently working on providing ssl support for

Support https or ssl

would supported and if there is any suggestion on what I can do in the interim to provide a similar functionality using Kakfa. Thank you in advance for your time and help. Jamie

Re: multiple consumers on a topic

sorry, I believe I answered my own question. Yes it supports this based on the group.id. On Fri, May 24, 2013 at 4:28 PM, Jamie Johnson wrote: > I am very new to kafka, so I'll apologize in advance for any stupid > questions... > > > That being said is it possible

multiple consumers on a topic

I am very new to kafka, so I'll apologize in advance for any stupid questions... That being said is it possible within kafka to have multiple consumers on a single topic? I had assumed the answer was yes, but I am running into some issues setting this up. Any information would be greatly apprec

Process for accepting code contribution

sking. Thanks for any information you may have. Also just want to say we are very happy with Kafka. You guys on the development team did a great job. Kudos to you all! Jamie

RE: How do you handle server down in producer

consumer as well for that matter) to provide ability for handling timeout. In a distributed environment, the ability to handle timeout is common and useful. Thanks. Jamie -Original Message- From: Philip O'Toole [mailto:phi...@loggly.com] Sent: Thursday, April 04, 2013 2:38 PM To:

How do you handle server down in producer

hang in memory and not do anything indefinitly is not desirable in our scenario. Thanks for your help. Jamie

RE: Loading kafka server within a Java process

I am able to start kafka broker within the main Java thread. But when I tried to start the broker within a Java thread, the broker just terminates. I tried join() on the thread and has no effect. Any ideas? Thanks. Jamie -Original Message- From: Eric Tschetter [mailto:ched

RE: Loading kafka server within a Java process

Thank you both for the info. I'll keep my eye open since I do have a need to embed a consumer in the same process. Jamie -Original Message- From: Eric Tschetter [mailto:ched...@metamarkets.com] Sent: Thursday, February 14, 2013 2:44 PM To: users@kafka.apache.org Subject: Re: Lo

Loading kafka server within a Java process

will work? What are some of the things I need to watch out? And is there another better way to do this? Thank you for your time. Jamie

Jconsole not showing all messages on the broker for a partition

t the consumer which reads from the beginning of the partition, it read all 50 messages. Any thoughts? Thanks Jamie

RE: How do you keep track of offset in a partition

Thanks. This makes sense. Jamie -Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: Monday, January 28, 2013 5:27 PM To: users@kafka.apache.org Subject: Re: How do you keep track of offset in a partition Jamie, You need to use the getOffsetsBefore() API to

RE: Subscription Issue.

The email below from Jun will should help you on the subscription. On Wed, Dec 5, 2012 at 1:18 PM, Jun Rao wrote: > We recently graduated to a top-level Apache project and have new > mailing lists. You need to subscribe to > dev-subscr...@kafka.apache.org > users-subscr...@kafka.apache.org

RE: Design questions and your opinion and suggestions are appreciated

like and what version will have this new behavior? Thanks again. Jamie -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Wednesday, December 12, 2012 10:10 PM To: users@kafka.apache.org Subject: Re: Design questions and your opinion and suggestions are appreciated 1) In

RE: Running Kafka on Windows?

bat file to include the path to the run-class .bat. Jamie -Original Message- From: Jamie Wang [mailto:jamie.w...@actuate.com] Sent: Wednesday, December 05, 2012 12:33 PM To: users@kafka.apache.org Subject: RE: Running Kafka on Windows? Thanks for all the replies. I downloaded th

RE: Running Kafka on Windows?

Thanks for all the replies. I downloaded the 0.72 source as you mentioned below. My question is what does the patch do? Can I download the patch and apply to 0.72 source? Jamie -Original Message- From: Antoine Vianey [mailto:antoine.via...@gmail.com] Sent: Tuesday, December 04, 2012

Running Kafka on Windows?

c8434 for " provides windows batch script for starting Kafka/Zookeeper; patched b...". Is this the one I should download and use to run Kafka on Windows? Can someone please point me to the right direction? Thanks for your help. Jamie

Security measure

information (especially within a financial company). Anyone has any suggestions on how to handle this? Thanks Jamie

RE: producer queue size

log.flush.interval is set to 1. Should I increase or decrease? thanks Jamie -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Wednesday, November 28, 2012 10:00 PM To: users@kafka.apache.org Subject: Re: producer queue size Is the server busy on I/O? What&#

RE: producer queue size

As I read the doc page on the site, there is none... Thanks Jamie -Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: Monday, November 26, 2012 4:40 PM To: kafka-us...@incubator.apache.org Subject: Re: producer queue size This will happen if you push d