Re: node.js client library?

Hi Joe, I've started a node.js implementation for 0.8. (https://github.com/wurstmeister/node-kafka-0.8-plus) I'd welcome any feedback or help. Regards Thomas On 24/12/2013 15:24, Joe Stein wrote: Hi, I wanted to reach out if folks are using https://github.com/cainus/Prozess for

Re: node.js client library?

a for 0.8.0 also to the client list and my company's Scala DSL too thnx =) Joestein On Fri, Dec 27, 2013 at 1:15 AM, 小宇 wrote: Hi, here is a Node.js client for latest Kafka: https://github.com/SOHU-Co/kafka-node.git 2013/12/25 Thomas Hi Joe, I've started a node.js implementation

Re: Kafka with Docker - producer disconnecting

Hi, you can have a look at https://github.com/wurstmeister/kafka-docker maybe that helps On 10/01/2014 19:30, Alex Artigues wrote: Hi everyone, I am attempting to run Zookeeper and Kafka in dockers. Both startup normally and Kafka connects ok. I think my containers are linked fine because I a

what to do if replicas are not in sync

e partitions are out of sync. What do I do to get them in sync again? thanks thomas

Source Code for FetchRequestBuilder

Hello, I was trying to find the source code for FetchRequestBuilder on github, but I cannot find it anywhere. Can anyone help me on this? Thanks. T.E.

Re: Source Code for FetchRequestBuilder

aat > the bottom > > /*** > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > ****/ >

Why is there no ZookeeperProducerConnector ?

Hi all, I'm looking for a simple way to create *a producer using zookeeper has source for metadata.broker.list.* I found the consumer equivalent : ZookeeperConsumerConnector but no sign of ZookeeperProducerConnector. Thanks for your help. -- *Thomas Pocreau* Recherche & Déve

Re: Kafka 0.11.0.0

0.11.0.0 is a brand new release, with a very large number of changes compared to the previous stable release (0.10.2.1). As a thing that stores data, I would not recommend you switch to it without a very large amount of testing and validation, probably involving running a shadow setup of your produ

Re: [VOTE] 1.0.0 RC1

Hi Guozhang, This says the due date on the testing is October 13th, which was the day this email was sent. Is that accurate, or is it meant to read October 17th, which is next Tuesday? I feel like this short a testing window for a 1.0 RC is a little low, as 1.0 is clearly a big announcement of st

Re: [VOTE] 1.0.0 RC1

Hi Ghouzang, We have indeed started our performance testing at Heroku for RC1. However, we are more than happy to retest once RC2 is available, especially given larger amounts of time to do so. Thanks Tom Crayford Heroku Kafka On Tue, Oct 17, 2017 at 2:50 AM, Ismael Juma wrote: > If you don't

Client addressable listener

I can't seem to get a listeners and advertised.listeners configuration for server properties figured out so I can connect remotely with my producer and consumers. If I set it like this... listeners=CLIENT://:9090,PLAINTEXT://:9092 advertised.listeners=CLIENT://:9090,PLAINTEXT://:9092 >From my ex

Re: Kafka JVM heap limit

Hi there, There are some cases you may want to consider bigger heaps. Heroku runs some clusters with 20GB heaps now, because said clusters: 1. Use SSL exclusively for connectivity. SSL means message bytes traverse the JVM heap during encryption/etc, which uses more memory 2. Have a high number of

Re: Client addressable listener

2 advertised.listeners=CLIENT://my.public.dns.name:9090,PLAINTEXT://:9092 This will tell the listener to listen on your local ip addresses but to advertise the DNS name. Jakub On Tue, Nov 7, 2017 at 11:04 PM, Thomas Stringer wrote: I can't seem to get a listeners and advertised.listeners con

Re: Client addressable listener

me:9090,PLAINTEXT://:9092 Also as I mentioned before the DNS should be only in the "advertised.listeners". Jakub On Wed, Nov 8, 2017 at 10:06 PM, Thomas Stringer wrote: Thank you for the reply! I think I'm doing something wrong. I tried using exactly verbatim what you had for li

Documentation or reference on listener names/types?

I've been working with Kafka broker listeners and I'm curious is there any documentation that explains what all of them apply to? Such as CLIENT, PLAINTEXT, SASL/SSL, etc. I see the encryption part of the documentation, but is it just inferred what these listeners apply to? Thank you in advanc

Listeners and reference/docs

I've been working with Kafka broker listeners and I'm curious is there any documentation that explains what all of them apply to? Such as CLIENT, PLAINTEXT, SASL/SSL, etc. I see the encryption part of the documentation, but is it just inferred what these listeners apply to? Thank you in advance!

Re: Listeners and reference/docs

l+and+External+traffic > > On Fri, Nov 10, 2017 at 10:10 AM, Thomas Stringer > wrote: > > > I've been working with Kafka broker listeners and I'm curious is there > > any documentation that explains what all of them apply to? Such as > > CLIENT, PLAINTEXT, SA

Re: Listeners and reference/docs

s=CLIENT://192.1.1.8:9092,REPLICATION://10.1.1.5:9093 ,INTERNAL_PLAINTEXT://10.1.1.5:9094,INTERNAL_SASL://10.1.1.5:9095 On Fri, Nov 10, 2017 at 7:10 PM, Thomas Stringer wrote: Yep I'm familiar with that. Just curious where it's documented that, for instance, the CLIENT listener is for cl

Re: Kafka in virtualized environments

We run many thousands of clusters on EC2 without notable issues, and achieve great performance there. The real thing that matters is how good your virtualization layer is and how much of a performance impact it has. E.g. in modern EC2, the performance overhead of using virtualized IO is around 1-5%

Re: Kafka streams on Kubernetes

How did you create Kafka in the k8s cluster? Can you share your config? On Tue, Dec 5, 2017, 7:49 AM Artur Mrozowski wrote: > Hi, has anyone experience with running Kafka streams application on > Kuberentes? > > Do I have to define any in/outbound ports? > Application fails because it cannot co

Class Files\OpenLink not found

charger la classe principale Files\OpenLink I downloaded kafka binary file from apache kafka website and then unzip it in my C:\user\username folder. I modified my log.dirs property in server.properties but that's all. How can I do ? Thank you very much. Thomas.

Re: broker properties explanations

Hi Adrien, log.dirs exists to facilitate multiple data directories which allows more than one disk to be used without the need for RAID. This increases throughput but beware of naive load balancing that may fill up one disk way before another. When log.flush.interval.ms is null the log.flush.i

Re: When a broker down, Producer LOST messages!

Hi, The log is not attached. I'm assuming your topic has a replication factor greater than 1 so that it is available from another Broker if the partition leader fails. Try adding props.put("acks", "all"); to your producer and run your experiment again. If you configured your topic to have --

Re: Hardware Guidance

Hi Adrien, Without asking the author directly I can't give the exact answer but I would interpret that as per broker. Kafka will make use of as much hardware as you give it so it's not uncommon to see many CPU cores and lots or RAM per broker. That being said it's completely down to your use c

Re: Running kafka in containers

We (heroku) have run databases in containers since 2012, and kafka works just as well as everything else. So: yes

OffsetsLoadInProgressException and Consumer Rewind

Hi, I am running kafka 0.8.2.1 and we have some consumers that is using kafka to store its offsets (offsets.storage=kafka). About once a week the consumer will see the exception OffsetsLoadInProgressException and when the exception goes away the consumer is reset the the earliest offset. It does

kafka in undefined state after zookeeper network issues

the "sick" kafka node. Any help would be very appreciated! Thanks Thomas

Re: Kafka behind NAT

Try setting: advertised.listeners=EXTERNAL://:9093,INTERNAL://:9092 inter.broker.listener.name=INTERNAL listener.security.protocol.map=EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT Then you should be able to use :9093 as your bootstrap.servers from outside the network or :9092 from inside. Obviously th

retention.ms not honored for topic

A single topic does not appear to be honoring the retention.ms setting. Three other topics (plus __consumer_offsets) on the Kafka instance are deleting segments normally. Kafka version: 2.12-0.10.2.1 OS: CentOS 7 Java: openjdk version "1.8.0_161" Zookeeper: 3.4.6 Retention settings (from kafka-to

Re: retention.ms not honored for topic

just rolling many segments. So keep it some reasonable value. > > On Tue, May 29, 2018 at 9:31 PM Thomas Hays wrote: > >> A single topic does not appear to be honoring the retention.ms >> setting. Three other topics (plus __consumer_offsets) on the Kafka >> instance are deletin

Re: [VOTE] 2.0.0 RC0

+1 (non-binding) Heroku has run our usual set of upgrade and performance tests, and we haven't found any notable issues through that. On Sat, Jun 23, 2018 at 12:30 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > +1 (non-binding) > > Built from source and ran quickstart successfully on

Streams SessionStore - bug or documentation error?

I've been experimenting with the streams SessionStore and found some behavior that contradicts the javadoc. Specifically: I have a SessionStore, and put() a session with key K1 and session of time T0-T5. I then call findSessions("K1", T2, T4) and it comes back empty. I would expect the session t

Question about Kafka TLS

there a possibility that we get truststore.jks from a service and store in memory so we don’t have to maintain a file in client side. Thanks, Thomas

Re: Kafka logs are getting deleted too soon

Hi Sachin, Try adding --from-beginning to your console consumer to view the historically produced data. By default the console consumer starts from the last offset. Tom Aley thomas.a...@ibm.com From: Sachin Nikumbh To: Kafka Users Date: 17/07/2019 16:01 Subject:[EXTERNAL] K

Broker Interceptors

Hi Kafka community, I am hoping to get some feedback and thoughts about broker interceptors. KIP-42 Added Producer and Consumer interceptors which have provided Kafka users the ability to collect client side metrics and trace the path of individual messages end-to-end. This KIP also mentioned

RE: Broker Interceptors

intercept for their use case. Tom Aley thomas.a...@ibm.com From: "M. Manna" To: Kafka Users Cc: d...@kafka.apache.org Date: 02/12/2019 11:31 Subject:[EXTERNAL] Re: Broker Interceptors Hi Tom, On Mon, 2 Dec 2019 at 09:41, Thomas Aley wrote: > Hi Kafka commun

RE: Broker Interceptors

7:21 AM Sönke Liebau wrote: > Hi Thomas, > > I think that idea is worth looking at. As you say, if no interceptor is > configured then the performance overhead should be negligible. Basically it > is then up to the user to decide if he wants tomtake the performance hit. >

RecordTooLargeException with old (0.10.0.0) consumer

We have some legacy applications using an old (0.10.0.0) version of the consumer that are hitting RecordTooLargeExceptions with the following message: org.apache.kafka.common.errors.RecordTooLargeException: There are some messages at [Partition=Offset]: {mytopic-0=13920987} whose size is larger

Re: RecordTooLargeException with old (0.10.0.0) consumer

rties: max.request.size=15728640 consumer max.partition.fetch.bytes On 7/28/20, 9:51 AM, "Thomas Becker" mailto:thomas.bec...@xperi.com>> wrote: [External] We have some legacy applications using an old (0.10.0.0) version of the consumer that are hitting RecordTooLar

Consumer cannot move past missing offset

We are having issues with some of our older consumers getting stuck reading a topic. The issue seems to occur at specific offsets. Here's an excerpt from kafka-dump-log on the topic partition around the offset in question: baseOffset: 13920966 lastOffset: 13920987 count: 6 baseSequence: -1 last

Re: [ANNOUNCE] New committer: A. Sophie Blee-Goldman

Congrats Sophie! On Mon, Oct 19, 2020 at 11:41 AM Matthias J. Sax wrote: > Hi all, > > I am excited to announce that A. Sophie Blee-Goldman has accepted her > invitation to become an Apache Kafka committer. > > Sophie is actively contributing to Kafka since Feb 2019 and has > accumulated 140 com

Re: Sub Tasks being processed only after restart

Hi Nitay, What version of Kafka are you running? If you could also give the topology you're using that would be great. Do you have a sense of if the lag is happening on all partitions or just a few? Also if you're using rocksDB there are some rocksDB metrics in newer versions of Kafka that could b

Re: Sub Tasks being processed only after restart

e topic and partitions are changing, and reset usually fixes the > problem almost immediately - i find it hard to believe it has anything to > do with the topology or business logic but I might be missing something > (since, after restart, the lag disappear with no real effort). > > Tha

Unable to run unit tests inside a docker container

Hi, I have cloned the apache-Kafka repo inside a docker container that has java 11 installed on it and on running the ./gradlew uniTest command I'm getting build failure with failing network tests. The logs from the container are uploaded here https://gist.github.com/STEPHINRACHEL/9fbce12db303eb02

Unable to run unit tests inside docker container

ACHEL/9fbce12db303eb0285d14ff322dbd15d#file-gistfile1-txt-L1235 Could someone guide me on how to run these tests inside a docker container? -- *Thanks and Regards,* *Stephin Rachel Thomas* She/Her Quality Engineer, Managed Application Services <https://www.redhat.com/>

Re: Unable to run unit tests inside a docker container

no Compression? > > (I am admittedly more of a redhat guy than docker expert) > > > From: Stephin Thomas > Sent: Tuesday, February 2, 2021 9:11 AM > To: users@kafka.apache.org > Subject: Unable to run unit tests inside a docker container >

Re: Unable to run unit tests inside a docker container

> gradle daemons wont run in CI containers please add > > ./gradlew build --no-daemon > > Please let me know your results > martin > > From: Stephin Thomas > Sent: Wednesday, February 3, 2021 2:58 AM > To: users@kafka.apache.org > S

Re: Unable to run unit tests inside a docker container

k.SslTransportLayerTest.testTlsDefaults(SslTransportLayerTest.java:587) > > m- > > > From: Stephin Thomas > Sent: Wednesday, February 3, 2021 9:30 AM > To: users@kafka.apache.org > Subject: Re: Unable to run unit tests inside a docker containe

Re: Unable to run unit tests inside a docker container

} > maven { > url "https://plugins.gradle.org/m2/"; > } > } > apply from: file('gradle/buildscript.gradle'), to: buildscript > apply from: "$rootDir/gradle/dependencies.gradle" > > dependencies { > scalaCompiler

Re: Unable to run unit tests inside a docker container

is the scala compiler > > i asked the gradle people which repository is supposed to contain > scala-compiler-plugin (any version) > Gradle support response is "we dont know scala" > > any idea? > > > > From: Stephin Thomas >

Kafka 3.0.0 KRaft: Node was unable to process the fetch request

Hi, I am trying to setup a 3-node cluster using Kafka 3.0.0 and KRaft. However, I always see the following error every millisecond in at least one of the bokers: 2021-10-06 08:24:49,309] INFO [ReplicaFetcher replicaId=3, leaderId=2, fetcherId=0] Node 2 was unable to process the fetch request w

State Store might migrated to another instance

ld need to create a Kubernetes stateful set instead of a Kubernetes deployment. It would be really great if someone could help us here. Because of this, we start to think to get rid of Kafka again as it brings us no value of what we have expected. Regards Thomas

State Store might migrated to another instance

at we would need to create a Kubernetes stateful set instead of a Kubernetes deployment. It would be really great if someone could help us here. Because of this, we start to think to get rid of Kafka again as it brings us no value of what we have expected. Regards Thomas ___

Re: Few partitions stuck in under replication

Hi Dhirenda, Firstly, I am interested in why are you restarting the ZK and Kafka cluster every night? Secondly, how are you doing the restarts. For example, in [Strimzi](https://strimzi.io/), when we roll the Kafka cluster we leave the designated controller broker until last. For each of the o

Re: Few partitions stuck in under replication

tation using stateful set. restart is triggered by updating a > annotation in the pod. > Issue is not triggered by kafka cluster restart but the zookeeper servers > restart. > Thanks, > Dhirendra. > > On Thu, Mar 3, 2022 at 7:19 PM Thomas Cooper wrote: > >> Hi Dhire

Re: Few partitions stuck in under replication

ced in the version after 2.5.0. > > Thanks, > Dhirendra. > > On Thu, Mar 3, 2022 at 11:09 PM Thomas Cooper wrote: > >> I suspect this nightly rolling will have something to do with your issues. >> If you are just rolling the stateful set in order, with no depe

Re: Switchin from Zookepper to Kafka KRaft mode / Using ACLs with Kafka KRaft mode

Hi Florian, Switching from a Zookeeper based cluster to a KRaft based one is not currently supported. AFAIK that functionality should be coming in Kafka 3.4 (or possibly later). Cheers, Tom On 16/05/2022 12:42, Florian Blumenstein wrote: > Hi guys, > > I currently try to switch from Kafka 3.1.0

Data Loss from Kafka Connect when Schema Registry requests fail

Hi we notice data loss i.e. dropped records when running Debezium on Kafka Connect with Apicurio Schema Registry. Specifically, multiple times we have observed that a single record is dropped when we get this exception (full stack trace

Data Corruption in Netty 4.1.111.Final (Kafka 3.9.0)

With the completion of KAFKA-17046 , the netty version has been upgraded to 4.1.111.Final in Kafka 3.9.0. This netty version has a known issue with data corruption, see netty issue and grpc-java issue <

Re: Disadvantages of Upgrading Kafka server without upgrading client libraries?

The only obvious downside I'm aware of is not being able to benefit from the bugfixes in the client. We are essentially doing the same thing; we upgraded the broker side to 0.10.0.0 but have yet to upgrade our clients from 0.8.1.x. On Tue, 2016-11-29 at 09:30 -0500, Tim Visher wrote: > Hi Everyone

One Kafka Broker Went Rogue

Hi, I encountered a strange issue in our kafka cluster, where randomly a single broker entered a state where it seemed to think it was the only broker in the cluster (it shrank all of its ISRs to just existing on itself). Some details about the kafka cluster: - running in an EC2 VPC on AWS - 3 no

Looking for guidance on setting ZK session timeouts in AWS

I know several folks are running Kafka in AWS, can someone give me an idea of what sort of values you're using for ZK session timeouts? -- Tommy Becker Senior Software Engineer O +1 919.460.4747 tivo.com This email and any attachments may co

Re: Looking for guidance on setting ZK session timeouts in AWS

pon. -Tommy On Mon, 2016-12-05 at 11:00 -0500, Radek Gruchalski wrote: > Hi Thomas, > > Defaults are good for sure. Never had a problem with default timeouts > in AWS. > – > Best regards, > Radek Gruchalski > ra...@gruchalski.com > > > On December 5, 2016 at 4:58:

Re: One Kafka Broker Went Rogue

coming-mainstream> | SIA <http://newsmanager.commpartners.com/sianews2/issues/2016-08-19/11.html> On Tue, Nov 29, 2016 at 1:29 PM, Thomas DeVoe wrote: > Hi, > > I encountered a strange issue in our kafka cluster, where randomly a > single broker entered a state where it seemed to think it

kafka CN domain and keyword

associated with your company or not? Best Regards, Thomas Liu | Service & Operations Manager China Registry (Head Office) | 6012, Xingdi Building, No. 1698 Yishan Road, Shanghai 201103, China Tel: +86-2161918696 | Fax: +86-2161918697 | Mob: +86-13816428671 Email: tho...@china

Old producer slow/no recovery on broker failure

We ran into an incident a while back where one of our broker machines abruptly went down (AWS is fun). While the leadership transitions and so forth seemed to work correctly with the remaining brokers, our producers hung shortly thereafter. I should point out that we are using the old Scala produce

Request for JIRA access

. Regards, Thomas Dutta

Request for JIRA access

. Regards, Thomas Dutta

Lost ISR when upgrading kafka from 0.10.0.1 to any newer version like 0.10.1.0 or 0.10.2.0

-- <https://fr.linkedin.com/in/thomas-kieffer-28517324> Thomas Kieffer Senior Linux Systems Administrator Skype: thomas.kieffer.corporate | Phone: (+352) 691444263 | www.olamobile.com <http://www.olamobile.com> -- The information transmitted is intended only for the

Re: Lost ISR when upgrading kafka from 0.10.0.1 to any newer version like 0.10.1.0 or 0.10.2.0

rsion=CURRENT_KAFKA_VERSION (See potential performance impact following the upgrade for the details on what this configuration does.) On 03/14/2017 11:26 AM, Ismael Juma wrote: Hi Thomas, Did you follow the instructions: https://kafka.apache.org/documentation/#upgrade Ismael On Mon, Mar 13, 20

Re: Lost ISR when upgrading kafka from 0.10.0.1 to any newer version like 0.10.1.0 or 0.10.2.0

n Tue, Mar 14, 2017 at 11:22 AM, Thomas KIEFFER < thomas.kief...@olamobile.com.invalid> wrote: Hello Ismael, Thank you for your feedback. Yes I've done this changes on a previous upgrade and set them accordingly with the new version when trying to do the upgrade. inter.broker.

Re: kafka streams in-memory Keyvalue store iterator remove broken on upgrade to 0.10.2.0 from 0.10.1.1

Couldn't this have been solved by returning a ReadOnlyKeyValueIterator that throws an exception from remove() from the ReadOnlyKeyValueStore.iterator()? That preserves the ability to call remove() when it's appropriate and moves the refused bequest to when you shouldn't. On Thu, 2017-03-23 at 11:0

Re: Custom stream processor not triggering #punctuate()

Does this fix the problem though? The docs indicate that new data is required for each *partition*, not topic. Overall I think the "stream time" notion is a good thing for a lot of use-cases, but some others definitely require wall-clock based windowing. Is something planned for this? -Tommy O

handling failed offsets

kafka itself and retry that queue at preset intervals. Thanks, Roshni Thomas

RE: Why do I need to specify replication factor when creating a topic?

Yes, this has been an issue for some time. The problem is that the AdminUtils requires this info to be known client side, but there is no API to get it. I think things will be better in 0.11.0 where we have the AdminClient that includes support for both topic CRUD APIs (not just ZK modifications

RE: Why do I need to specify replication factor when creating a topic?

sounds like the CRUD API's still require explicitly including the replication factor param in the CreateTopic call. That's essentially the crux of my question... why does the client ever need to know the default param if the broker is already aware of it? Was this an explicit design decis

RE: Order of punctuate() and process() in a stream processor

I'm a bit troubled by the fact that it fires 3 times despite the stream time being advanced all at once; is there a scenario when this is beneficial? From: Matthias J. Sax [matth...@confluent.io] Sent: Friday, May 12, 2017 12:38 PM To: users@kafka.apache.o

RE: Order of punctuate() and process() in a stream processor

multiple times. It's again due to the coarse grained advance of "stream time". @Thomas: I think, the way we handle it just simplifies the implementation of punctuations. I don't see any other "advantage". I will create a JIRA to track this -- we are currently working

Re: Data Corruption in Netty 4.1.111.Final (Kafka 3.9.0)

This was fixed with this PR https://github.com/apache/kafka/pull/17860 On Wed, Nov 20, 2024 at 9:49 AM Thomas Thornton wrote: > With the completion of KAFKA-17046 > <https://issues.apache.org/jira/browse/KAFKA-17046>, the netty version > has been upgraded to 4.1.111.Final in Ka

Re: Data Loss from Kafka Connect when Schema Registry requests fail

This was fixed in https://github.com/apache/kafka/pull/18146 On Tue, Nov 12, 2024 at 4:41 PM Thomas Thornton wrote: > Hi we notice data loss i.e. dropped records when running Debezium on Kafka > Connect with Apicurio Schema Registry. Specifically, multiple times we have > observed that