Re: Async Non Blocking Kafka Producer

Try adding props.put("max.block.ms", "0"); -hans > On Jun 7, 2017, at 12:24 PM, Ankit Jain wrote: > > Hi, > > We want to use the non blocking Kafka producer. The producer thread should > not block if the Kafka is cluster is down or not reachable. > > Currently, we are setting following prop

Re: Async Non Blocking Kafka Producer

even the Cluster is > available. > > Thanks > Ankit Jain > >> On Wed, Jun 7, 2017 at 12:56 PM, Hans Jespersen wrote: >> >> Try adding props.put("max.block.ms", "0"); >> >> -hans >> >> >> >>> On Jun 7, 2017, at

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

Offset commit is something that is done in the act of consuming (or reading) Kafka messages. Yes technically it is a write to the Kafka consumer offset topic but it's much easier for administers to think of ACLs in terms of whether the user is allowed to write (Produce) or read (Consume) mes

Re: help!Kafka failover do not work as expected in Kafka quick start tutorial

Do you list all three brokers on your consumers bootstrap-server list? -hans > On Jun 22, 2017, at 5:15 AM, 夏昀 wrote: > > hello: > I am trying the quickstart of kafka documentation,link is, > https://kafka.apache.org/quickstart. when I moved to Step 6: Setting up a > multi-broker cluster,I ha

Re: question about document

Correct. The use of the word "server" in that sentence is meant as broker (or KafkaServer as it shows up in the 'jps' command) not as a physical or virtual machine. -hans > On Jun 27, 2017, at 1:22 AM, James <896066...@qq.com> wrote: > > Hello, >At https://kafka.apache.org/intro, I found a

Re: Requires suggestions for Producer request throttling

Request quotas was just added to 0.11. Does that help in your use case? https://cwiki.apache.org/confluence/display/KAFKA/KIP-124+-+Request+rate+quotas -hans > On Jun 29, 2017, at 12:55 AM, sukumar.np wrote: > > Hi Team, > > > > We are having a Kafka cluster with multiple Topics in it and s

Re: about Exactly-once Semantics

When you write the msg results to MySQL, you include the offset of the message with the results. This can be done in one atomic write transaction. Then if your application crashes, when it starts back up, it should read the offset stored with the last message results in the database, then seek()

Re: kafka connect

If you are looking for connectors there is a good curated list of connectors here https://www.confluent.io/product/connectors/ I don't see tcp on the list but the general naming scheme for open source connectors on github is to call them “kafka-con

Re: Adding partitons | Unaffected producers

See the producer param called metadata.max.age.ms which is "The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions." -hans > On Aug 4, 2017, at 5:17 AM, Sameer

Re: How to clear a particular partition?

This is an area that is being worked on. See KIP-107 for details. https://cwiki.apache.org/confluence/display/KAFKA/KIP-107%3A+Add+purgeDataBefore%28%29+API+in+AdminClient -hans > On Aug 10,

Re: How to clear a particular partition?

ytest --from-beginning 91 92 93 94 95 96 97 98 99 100 -hans > On Aug 18, 2017, at 10:32 AM, Manikumar wrote: > > This feature got released in Kafka 0.11.0.0. You can > use kafka-delete-records.sh script to

Re: Pinning clients to specific brokers

Doing that doesn't really make sense in a Kafka cluster because the topic partitions and their replicas are spread out across many brokers in the cluster. That's what enables the parallel processing and fault tolerance features of Kafka. -hans > On Aug 22, 2017, at 3:14 AM, Mohit Chawla > wr

Re: Pinning clients to specific brokers

We (Confluent) run Kafka as a SaaS-based cloud offering and we do not see any reason for this feature so I just don’t understand the motivation for it. Please explain. -hans -- /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ >

Re: Flush Kafka topic

in 0.11 and above see the CLI command bin//kafka-delete-records.sh -hans > On Aug 23, 2017, at 7:28 PM, Rahul Singh wrote: > > Hello all, > > I am unable to purge the topic data from Kafka. Is there any class to flush > all topic data. > > Thank you

Re: Offsets across brokers

Yes the offsets are the same. -hans > On Aug 28, 2017, at 8:32 PM, Vignesh wrote: > > Hi, > > If a topic partition is replicated and leader switches from broker 1 to > broker 2 , are the offsets for messages in broker 2 same as broker1 ? If > not, how can applications that store offsets out

Re: KSQL with Apache Kafka

Those prerequisites are just for the Confluent CLI used in the quickstart. The Apache Kafka and Zookeeper versions included in the Confluent distribution are the latest and the same as the Apache Kafka download so it will work. You will just need to start Zookeeper and Kafka with the shell scrip

Re: Can't reset consumer group offsets in Kafka 0.11.0.1

Did you add the --execute flag? -hans > On Sep 21, 2017, at 11:37 AM, shargan wrote: > > Testing kafka-consumer-groups.sh in my dev environment, I'm unable to reset > offsets even when CURRENT-OFFSET is inbounds. Again, it returns as if the > change took effect but describe still shows the orig

Re: Debugging invalid_request response from a .10.2 server for list offset api using librdkafka client

tps://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI(AKAListOffset) -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Wed, Sep 27, 2017 at 10:20 AM, Vignesh wrote: > Correct

Re: Failure to reset consumer offsets for specific topics

I think you are just missing the —execute flag. -hans > On Oct 25, 2017, at 1:24 PM, Ted Yu wrote: > > I wonder if you have hit KAFKA-5600. > > Is it possible that you try out 0.11.0.1 ? > > Thanks > >> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin wrote: >> >> I am using 0.11.0.0. >> >>

Re: Kafka streams for golang

You can call the REST endpoints in KSQL from any programming language. I wrote some stuff in node.js to call KSQL this way and it works great. The results don't even have to go to a Kafka topic as the results of and POST to /query all stream using HTTP. -hans /** * Hans Jespersen, Prin

Re: Seeking advice on Kafka Streams and Kafka Connect

It might be possible to do all the transformations in #2 inside Kafka Connect. Connect has a simple one message at a time transformation capability called Single Message Transforms (SMT). There are built in Transformation functions that you can declaratively add to any existing connector via con

Re: Consumer client not able to receive messages when one of broker is pushed down in the cluster

Check that your __consumer_offsets topic is also setup with replication factor of 3 and has In Sync Replicas. Often it gets setup first as a one node cluster with RF=1 and then when the cluster is expanded to 3 nodes the step to increase the replication factor of this topic gets missed. -hans

Re: Capturing and storing these Kafka events for query.

Another approach would be to create the query first (in something like KSQL) and then send the Kafka data through the pre-existing streaming query. In this case the results would be going into various result topics. Tools like KSQL also let you query historical data but you need to be sure that

Re: Kafka Consumer Offsets unavailable during rebalancing

I believe this is expected behavior. If there are no subscriptions to a new topic, and therefor no partition assignments, and definitely no committed offsets, then lag is an undefined concept. When the consumers subscribe to this new topic they may chose to start at the beginning or end of the

Re: Kafka Consumer Offsets unavailable during rebalancing

; 3. A second later: I have a partition assignment for consumer group 'X' for > topics 'A' and 'B' again and the consumer group is still "Stable". > > I expected the state of consumer group 'X' during step 2b to be > "PreparingRebalanc

Re: replica.fetch.max.bytes split message or not ?

This is the KIP-74 write up if you want to learn more about the motivation and implementation of the fetch.max.bytes feature. For example, the 5th message in your example can now be fetched even if it is larger than fetch.max.bytes which was a great improvement in large message handling. Previo

Re: Is Kafka Streams right for me ?

"If your system is stateless and the transformations are not interdependent" then I would just look at using Kafka Connect's Single Message Transform (SMT) feature. -hans /** * Hans Jespersen, Director Systems Engineering, Confluent Inc. * h...@confluent.io (650)924-2670 */

Re: Is Restart needed after change in trust store for Kafka 1.1 ?

Kafka 1.1.0 https://issues.apache.org/jira/browse/KAFKA-6240 <https://issues.apache.org/jira/browse/KAFKA-6240> which seems in include dynamic reconfiguration of SSL keystores https://issues.apache.org/jira/browse/KAFKA-6241 <https://issues.apache.org/jira/browse/KAFKA-6241> -- /**

Re: kafka new consumer is ready for production setup?

why the 0.10.1 docs are hard to find. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Tue, Oct 4, 2016 at 11:42 PM, Gaurav Shaha wrote: > Hi, > > I want to use kafka new consumer. But in the documentation of 0.10.0 >

Re: Deleting a message after all consumer have consumed it

Kafka doesn’t work that way. Kafka is “Publish-subscribe messaging rethought as a distributed commit log”. The messages in the log do not get deleted just because “all" clients have consumed the messages. Besides you could always have a late joining consumer come along and if you mistakenly dele

Re: Regarding Kafka

Why do you have 10 topics? It seems like you are treating topics like partitions and it's unclear why you don't just have 1 topic with 10, 20, or even 30 partitions. Ordering is only guaranteed at a partition level. In general if you want to capacity plan for partitions you benchmark a single

Re: Regarding Kafka

Abhi On Sun, Oct 9, 2016 at 10:13 AM, Hans Jespersen wrote: > Why do you have 10 topics?  It seems like you are treating topics like > partitions and it's unclear why you don't just have 1 topic with 10, 20, or > even 30 partitions. Ordering is only guaranteed at a partitio

Re: Regarding Kafka

sumed the way I sent, then my analytics will go haywire. Abhi On Sun, Oct 9, 2016 at 12:50 PM, Hans Jespersen wrote: > You don't even have to do that because the default partitioner will spread > the data you publish to the topic over the available partitions for you. > Just try it

Re: Regarding Kafka

7:07.500]AxThreadId 23516 ->ID:4495 offset: 81][ID date: 2016-09-28 20:07:39.000 ] On Sun, Oct 9, 2016 at 1:31 PM, Hans Jespersen wrote: > Then publish with the user ID as the key and all messages for the same key > will be guaranteed to go to the same partition and ther

Re: I found kafka lsot message

Watch this talk. Kafka will not lose messages it configured correctly. http://www.confluent.io/kafka-summit-2016-ops-when-it-absolutely-positively-has-to-be-there -hans > On Oct 13, 2016, at 2:48

RE: A question about kafka

Because the producer-property option is used to set other properties that are not compression type. //h...@confluent.io Original message From: ZHU Hua B Date: 10/16/16 11:20 PM (GMT-08:00) To: Radoslaw Gruchalski , users@kafka.apache.org Subject: RE: A question about kafka

Re: Problem in Deleting Kafka Logs

-1489 <https://issues.apache.org/jira/browse/KAFKA-1489> and KIP-61 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-61%3A+Add+a+log+retention+parameter+for+maximum+disk+space+usage+percentage> for more detail. -hans /** * Hans Jespersen, Principal Systems Engineer, Co

Re: Kafka consumer rate

Yes. See the description of quotas. https://kafka.apache.org/documentation#design_quotas -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Thu, Oct 20, 2016 at 3:20 PM, Adrienne Kole wrote: > Hi, > > Is there a way to

Re: Kafka unable to find topic details after sudden EC2 server restart

You are going to lose everything you store in /tmp. In a production system you never configure Kafka or zookeeper to store critical data in /tmp. This has nothing to do with AWS or EBS it is just standard Linux than everything under /tmp is deleted when Linux reboots. -hans /** * Hans Jespersen

RE: Mirror multi-embedded consumer's configuration

Yes. //h...@confluent.io Original message From: ZHU Hua B Date: 10/24/16 12:09 AM (GMT-08:00) To: users@kafka.apache.org Subject: RE: Mirror multi-embedded consumer's configuration Hi, Many thanks for your confirm! I have another question, if I deleted a mirrored topic on

Re: Removing kafka topic contents from Java

-hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Mon, Oct 24, 2016 at 7:01 AM, Demian Calcaprina wrote: > Hi Guys, > > Is there a way to remove a kafka topic from the java api? > > I have the following scenar

Re: Kafka unable to find topic details after sudden EC2 server restart

linux-linuxfoundationx-lfs101x-0 -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Mon, Oct 24, 2016 at 1:50 AM, Gourab Chowdhury wrote: > Thanks for the reply, I tried changing the data directory as follows:- > dataDir=/data/zook

Re: Kafka Multi DataCenter HA/Failover

What is the latency between the two datacenters? I ask because unless they are very close, you probably don’t want to do any form of synchronous replication. The Confluent Replicator (coming very soon in Confluent Enterprise 3.1) will do async replication of both messages and configuration metad

Re: Kafka Multi DataCenter HA/Failover

Are you willing to have a maximum throughput of 6.67 messages per second? -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Fri, Oct 28, 2016 at 9:07 AM, Mudit Agarwal wrote: > Hi Hans, > > The latency between my

Re: multiple consumers for a same topic in multiple copies

Just make sure they are not in the same consumer group by creating a unique value for group.id for each independent consumer. /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Mon, Oct 31, 2016 at 9:42 AM, Patrick Viet wrote: >

Re: Deadlock using latest 0.10.1 Kafka release

The 0.10.1 broker will use more file descriptor than previous releases because of the new timestamp indexes. You should expect and plan for ~33% more file descriptors to be open. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On

Re: is there a way to make sure two consumers receive the same message from the broker?

There is no built in mechanism to do this in Apache Kafka but if you can write consumer 1 and consumer 2 to share a common external offset storage then you may be able to build the functionality you seek. -hans > On Nov 5, 2016, at 3:55 PM, kant kodali wrote: > > Sorry there is a typo. here

Re: is there a way to make sure two consumers receive the same message from the broker?

are a common >> external offset storage" ? can you please elaborate a bit more. >> >> Thanks! >> >> On Sat, Nov 5, 2016 at 4:00 PM, Hans Jespersen wrote: >> >>> There is no built in mechanism to do this in Apache Kafka but if you can >>> w

Re: is there a way to make sure two consumers receive the same message from the broker?

ort of 2PC >> but >>> that would degrade the write performance. Morever we dont really have a >>> control over how fast each write/store can happen at each storage layer >>> (because these two storages are completely different). >>> >>> so I starte

Re: is there a way to make sure two consumers receive the same message from the broker?

case this is no longer a Kafka question and has become more of a a distributed database design question. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Sun, Nov 6, 2016 at 7:08 PM, kant kodali wrote: > Hi Hans, > > Th

Re: Kafka mirror maker help

The latest Confluent packages now ship with systemd scripts. That is since Confluent Version 4.1 - which included Apache Kafka 1.1 -hans /** * Hans Jespersen, Director Systems Engineering, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Fri, Apr 27, 2018 at 11:15 AM, Andrew Otto

Re: Kafka mirror maker help

Sorry I hit send a bit too soon. I was so focused on the systemd part of the email and not the Mirror Maker part. Confluent packages include Mirror Maker but the systemd scripts are setup to use Confluent Replicator rather than Mirror Maker. My apologies. -hans /** * Hans Jespersen, Director

Re: Can anyone help me to send messages in their original order?

If you create a topic with one partition they will be in order. Alternatively if you publish with the same key for every message they will be in the same order even if your topic has more than 1 partition. Either way above will work for Kafka. -hans > On May 25, 2018, at 8:56 PM, Raymond Xie

Re: Can anyone help me to send messages in their original order?

e are only 9 lines from 1 to 9, but on consumer side their > original order becomes messed up. > > ~~~sent from my cell phone, sorry if there is any typo > > Hans Jespersen 于 2018年5月26日周六 上午12:16写道： >> If you create a topic with one partition they will be in order. >> &

Re: Facing Duplication Issue in kakfa

Are you seeing 1) duplicate messages stored in a Kafka topic partition or 2) duplicate consumption and processing of a single message stored in a Kafka topic? If it’s #1 then you can turn on the idempotent producer feature to get Exactly Once Semantics (EOS) while publishing. If it’s #2 then y

Re: Round-Robin assignment when non-nullable record key

Why don’t to just put the metadata in the header and leave the key null so it defaults to round robin? -hans > On May 31, 2018, at 6:54 AM, M. Manna wrote: > > Hello, > > I can see the this has been set as "KIP required". > > https://issues.apache.org/jira/browse/KAFKA- > > I have a use

Re: Is there expiration for committed Offset in the partition

You should just recommit the same offsets sooner than every 24 hours (or whatever your commit topic retention period is set to). The expiry of offsets is based on the timestamp of the commits. -hans > On Jun 1, 2018, at 1:03 AM, Dinesh Subramanian > wrote: > > Hi, > > Facing duplication in

Re: Security for individual partitions

Kafka ACLs are at the topic level, not partition level. Probably better to make 10 topics of 1 partition each and use topic ACLs to control access. -hans > On Jun 25, 2018, at 9:50 PM, Yash Ganthe wrote: > > Hi, > > If I have a topic with 10 partitions, I would like each partition to be > ac

Re: The asynchronous sending of a message returns no error if the Kafka server is not started

That is expected behavior. Typically there are multiple kafka brokers and so if one is down the client retries to send to a newly elected leader. A send should not be considered successful until an ACK is received in the client from the kafka cluster. By default the ACK is async for perform

Re: How to acknowledge after consuming the message from Kafka topic?

Are you using kafka-node or node-rdkafka? In either case you should call Consumer.commit(cb) or something similar to manually commit offsets (aka acknowledge messages). Alternatively so can set a config parameter on the consumer to autoCommit. https://github.com/SOHU-Co/kafka-node/blob/master/

Re: How to acknowledge after consuming the message from Kafka topic?

this one. -hans > On Jan 21, 2019, at 10:02 AM, Rahul Singh > wrote: > > I am using node-kafka, I have used consumer.commit to commit offsets but > don't know why when I restart the consumer it consume the committed offsets. > > Thanks > >> On Mon, Jan 21, 2019

Re: How to balance messages in kafka topics with newly added partitions?

Yes but I find this even easier to do with KSQL. CREATE STREAM OUTPUTTOPIC AS SELECT * FROM INPUTTOPIC; There are similar examples like this that also filter messages while copying, or change the message format while copying on the KSQL Recipe page here https://www.confluent.io/stream-processin

Re: Kafka - Connect for logs processing

Take a look at kafka-connect-spooldir and see if it meets your needs. https://www.confluent.io/connector/kafka-connect-spooldir/ This connector can monitor a directory and pick up any new files that are created. Great for picking up batch files, parsing them, and publishing each line as if it w

Re: Proxying the Kafka protocol

You might want to take a look at kafka-proxy ( see https://github.com/grepplabs/kafka-proxy ). It’s a true kafka protocol proxy and modified the metadata like advertized listeners so it works when there is no ip routing between the client and the brok

Re: Question on performance data for Kafka vs NATS

Thats a 4.5 year old benchmark and it was run with a single broker node and only 1 producer and 1 consumer all running on a single MacBookPro. Definitely not the target production environment for Kafka. -hans > On Mar 21, 2019, at 11:43 AM, M. Manna wrote: > > HI All, > > https://nats.io/ab

Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka

Doesn’t every one of the 20,000 POS terminals want to get the same price list messages? If so then there is no need for 20,000 partitions. -hans > On Mar 31, 2019, at 7:24 PM, wrote: > > Hello! > > > > I ask for your help in connection with the my recent task: > > - Price lists are delive

Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka

; We use location specific SKU pricing and send specific price lists to the > specific POS terminal. > > пн, 1 апр. 2019 г., 3:01 Hans Jespersen : > >> Doesn’t every one of the 20,000 POS terminals want to get the same price >> list messages? If so then there is no need for 2

Re: Something like a unique key to prevent same record from being inserted twice?

yes. Idempotent publish uses a unique messageID to discard potential duplicate messages caused by failure conditions when publishing. -hans > On Apr 1, 2019, at 9:49 PM, jim.me...@concept-solutions.com > wrote: > > Does Kafka have something that behaves like a unique key so a producer can’

Re: Something like a unique key to prevent same record from being inserted twice?

.me...@concept-solutions.com > wrote: > > > >> On 2019/04/02 22:43:31, jim.me...@concept-solutions.com >> wrote: >> >> >>> On 2019/04/02 22:25:16, jim.me...@concept-solutions.com >>> wrote: >>> >>> >>>> On

Re: Source Connector Task in a distributed env

Your connector sounds a lot like this one https://github.com/jcustenborder/kafka-connect-spooldir I do not think you can run such a connector in distributed mode though. Typically something like this runs in standalone mode to avoid conflicts. -hans On Wed, Apr 24, 2019 at 1:08 AM Venkata S A

Re: Kafka Connect - HDFS or FileStream

Can you just use kafka-console-consumer and just redirect the output into a file? -hans On Mon, May 13, 2019 at 1:55 PM Vinay Jain wrote: > Hi > > The data needs to be transferred to some other system in other network, and > due to some security reasons, the other systems cannot be exposed . S

Re: Performance Testing Using Consumer-Perf-Test

1) Are all 10 publishers producing to the same topic? What level of ACKs do you have set? How many partitions are in your topic? Are all 10 consumers in the same consumer group or are they supposed to be independent consumers that each get the full set of messages published? 2) Depends what you are

Re: Customers are getting same emails for roughly 30-40 times

Its not just the config, you need to change your code. kafka.auto.commit.interval.ms=3000 means that consumers only commit offsets every 3 seconds so if there is any failure or rebalance they will reconsume up to 3 seconds of data per partition. That could be many hundreds or thousands of mes

Re: Update Replication Factor

Take a look at the Admin Client API here https://kafka.apache.org/22/javadoc/index.html?org/apache/kafka/clients/admin/AdminClient.html -hans On Mon, Jun 17, 2019 at 4:27 PM shubhmeet kaur wrote: > hi, > > I wish to updater the replciation factor of already created topic through > Java code us

Re: Requirements

Gwen Shapira published a great whitepaper with Reference Architectures for all Kafka and Confluent components in big and small environements and for bare metal, VMs, and all 3 major public clouds. https://www.confluent.io/resources/apache-kafka-confluent-enterprise-reference-architecture/ On Fri

Re: kafka connection from docker

This is a great blog post that explains how kafka works with advertised listeners and docker https://rmoff.net/2018/08/02/kafka-listeners-explained/ -hans > On Oct 18, 2019, at 5:36 AM, Mich Talebzadeh > wrote: > > I do not understand this. > > You have on a physical host running zookeeper

Re: Scaling Apache Kafka Producers & Consumers

> As per my understanding, in Apache Kafka a single consumer from a consumer > group can consume messages from one partition only. Not correct. A single consumer from a consumer group can consume from many partitions. For example if you had a topic with 12 partitions and 4 consumers in a consume

Re: Reg : Slowness in Kafka

Yes it should be going much faster than that. Something is wrong in your setup. -hans > On Mar 26, 2020, at 5:58 PM, Vidhya Sakar wrote: > > Hi Team, > > The Kafka consumer is reading only 8 records per second.We have implemented > apache Kafka and confluent connect S3. The confluent connect

Re: Scaling Apache Kafka Producers & Consumers

nsumer group. > > If you have 12 partitions and you run 4 consumers with 4 consumer threads > then 4 threads will be idle at any time T1. > > I hope this is clear. > > Thanks, > Sunil. > > On Thu, 26 Mar 2020 at 7:52 PM, Hans Jespersen wrote: > >>> As

Re: Kafka with RAID 5 on. busy cluster.

RAID 5 typically is slower because Kafka is very write heavy load and that creates a bottleneck because writes to any disk require parity writes on the other disks. -hans > On Mar 28, 2020, at 2:55 PM, Vishal Santoshi > wrote: > > Ny one ? We doing a series of tests to be confident, but if

Re: Newbie Question

I can tell from the terminology you use that you are familiar with traditional message queue products. Kafka is very different. Thats what makes it so interesting and revolutionary in my opinion. Clients do not connect to topics because kafka is a distributed and clustered system where topics a

Re: Is it possible to resubcribe KafkaStreams in runtime to different set of topics?

I believe that the new topics are picked up at the next metadata refresh which is controlled by the metadata.max.age.ms parameter. The default value is 30 (which is 5 minutes). -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On

Re: Massive SSL performance degredation

What is the difference using the bin/kafka-console-producer and kafka-console-consumer as pub/sub clients? see http://docs.confluent.io/3.1.0/kafka/ssl.html -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Thu, Nov 17, 2016 at 11

Re: Massive SSL performance degredation

pher suites in either the broker or the client > config which I gather leaves it up to the broker/client to decide during > TLS handshaking. I'm not sure if there is an easy way to figure out which > one they ended up with... I'll work on specifying which cipher suite I > want

Re: Disadvantages of Upgrading Kafka server without upgrading client libraries?

The performance impact of upgrading and some settings you can use to mitigate this impact when the majority of your clients are still 0.8.x are documented on the Apache Kafka website https://kafka.apache.org/documentation#upgrade_10_performance_impact -hans /** * Hans Jespersen, Principal

Re: Storing Kafka Message JSON to deep storage like S3

I know several people that use the qubole Kafka Sink Connector for S3 ( see https://github.com/qubole/streamx ) to store Kafka messages in S3 for long term archiving. You can also do this with the Confluent HDFS Kafka Connector if you have access to a Hadoop c

Re: Kafka Queue

are you setting the group.id property to be the same on both consumers? https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Wed, Dec 7, 2016 at 12:36 PM

Re: Kafka as a database/repository question

a newer message of the same key. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Thu, Dec 15, 2016 at 10:16 AM, Kenny Gorman wrote: > A couple thoughts.. > > - If you plan on fetching old messages in a non-contiguous ma

Re: Regarding Connection Problem

I would recommend you use either the Blizzard node-rdkafka module ( see https://github.com/Blizzard/node-rdkafka ) or the Confluent kafka-rest-node module ( see https://github.com/confluentinc/kafka-rest-node

Re: How does Kafka emulate exactly once processing?

Exactly once Kafka Sink Connectors typically store the offset externally in the same atomic write as they store the messages. That way after a crash, they can check the external store (HSFS, JSDC, etc) retrieve the last committed offset and seek the the next message and continue processing with

Re: How does Kafka emulate exactly once processing?

> > On Wed, Dec 21, 2016 at 5:11 PM, Hans Jespersen wrote: > >> Exactly once Kafka Sink Connectors typically store the offset externally >> in the same atomic write as they store the messages. That way after a >> crash, they can check the external store (HSFS, JSDC, etc)

Re: Heavy traffic between Kafka nodes

How much more data is flowing between the two nodes versus the total ingress rate of the cluster? I would expect inter-broker bandwidth use to be about the same as the ingress rate for the entire cluster since all the data coming into node 1 needs to get replicated to node 2 (for redundancy) and

Re: can kafka 10 stream API read the topic from a Kafka 9 cluster?

No. All Java clients (including Streams) need to be the same version (or lower) as the brokers they connect to. -hans > On Dec 23, 2016, at 1:03 AM, Sachin Mittal wrote: > > Is Kafka streams 0.10.2.0-SNAPSHOT compatible with 0.10.0.1 broker. > I was facing broker connect issue and it just d

Re: Kafka delay

latency guaranteed messaging -- Jiangjie (Becket) Qin (LinkedIn) <https://www.youtube.com/watch?v=oQe7PpDDdzA> which might give you much better context and understanding of what these parameters mean and how they work. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc

Re: Kafka Protocol : about "Offset" in the MessageSet

Producer will not know the offset of the message(s) at the time they send to the broker but they can receive that information back as confirmation of successful publish. -hans > On Jan 18, 2017, at 1:25 AM, Paolo Patierno wrote: > > Hi, > > > reading about the Kafka protocol specificati

Re: Kafka 0.10 quickstart: consumer fails when “primary” broker is brought down

but when it’s down, there is not a replica on any of the other brokers. Try creating a new topic with replication-factor 3 and you should get better availability in the event of one or even two broker failures. -hans -- /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h

Re: Confluent platform for Kafka 0.10.1.1

Today! Confluent 3.1.2 supports Kafka 0.10.1.1 https://www.confluent.io/blog/confluent-delivers-upgrades-clients-kafka-streams-brokers-apache-kafka-0-10-1-1/ -hans > On Jan 25, 201

Re: Kafka Connect in different nodes than Kafka.

If you are asking if technically does Kafka Connect need the entire Apache Kafka distribution to run then, then the answer is no, it does not because Connectors just remotely connect to Kafka Brokers on separate machines. If you are asking if there is a separate distribution for a “connect node”

Re: Question about messages in __consumer_offsets topic

The __consumer_offsets topic should also get a tombstone message as soon as a topic is deleted. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Wed, Feb 22, 2017 at 5:59 PM, Jun MA wrote: > Hi Todd, > > Thank you so

Re: creating partitions programmatically

The current Java AdminUtils are older code that talks to zookeeper and does not support a secured Kafka cluster. There will be a new Java Admin API in the future that talks to the Kafka brokers directly using the admin extensions to the Kafka protocol which are already in the 0.10.2 brokers. I w

  1   2   >