Lookup table common to all threads in a Kafka Streams app

I'm experimenting with the Kafka Streams preview and understand that joins can only happen between KStreams and/or KTables that are co-partitioned. This is a reasonable limitation necessary to support large streams. What if I have a small topic, though, that I'd like to be able to join based on va

subscribe

subscribe

Re: New client commitAsync SendFailedException

Hey Alexey, looks like we didn't have a JIRA before, but I created one just now: https://issues.apache.org/jira/browse/KAFKA-3412. As soon as I get a little time, I'll polish up my old patch and submit it. -Jason On Wed, Mar 16, 2016 at 4:10 AM, Alexey Romanchuk < alexey.romanc...@gmail.com> wrot

Kafka Streams Parallelism

Hello everyone, I've read the Kafka Streams docs available at http://docs.confluent.io/2.1.0-alpha1/streams/index.html. Since I'm coming from the world of Spark, Dataflow and friends, I couldn't avoid having some mind-breaking questions with how Kafka Streams handles its parallelism. In spark, whe

Kafka Streams scaling questions

Hi All, I read through the doc on KStreams here: http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple

Re: Question regarding compression of topics in Kafka

Thanks for the response Ben. I am wondering why is "du" command not showing reduced size when compression is used. I ran an experiment with compression enabled on a topic and without compression enabled sending same amount of data in both cases. I used single node 1 replication factor Kafka inst

Re: Not seeing disk writes on all replica brokers

> > what exactly are you monitoring in terms of the "disk writes"? Looking at both bytes written and number of IOPs. I do see the batched writes activity from the OS but they seem to be missing on some replicas. I looked into the kafka log and found there is frequent change in cluster/ISR for th

Using lz4 compression

More like getting a feel from the community about using lz4 for compression? Has anyone used in the kafka setup. I am aware that gzip and snappy are more older implementation and regressed. Given that lz4 has better compression/decompression cycles (though slightly less compression ratio), was t

Re: Question regarding compression of topics in Kafka

Yes it will compress the data stored on the file system if you specify compression in the producer. You can check whether the data is compressed on disk by running the following command in the data directory. kafka-run-class kafka.tools.DumpLogSegments --print-data-log --files latest-log-file.l

Re: Zookeeper DNS TTL

Ah I have been down this path. It is the zookeeper client. It resolves and caches the ip addresses: https://github.com/apache/zookeeper/blob/bd9a1448f9b29859092e6bdca93da121ec166b7a/src/java/main/org/apache/zookeeper/client/StaticHostProvider.java#L108 I believe they are cached forever. We have h

Re: Kafka 0.8.2.2 is using high CPU

Consider upgrading to 0.9.0.1 - our cluster is much more stable since upgrade. Clients can be upgraded later. On Fri, Mar 18, 2016, 23:22 li jinyu wrote: > I want to add more information. > > we've already setup a completely new Kafka cluster with 0.8.2.2, and the > old cluster is still there, b

Editing Kafka wiki pages (Confluence)?

Hi All, I want to publish C++ Kafka client. I have my git repository ready. Now I want add entry on Kafka “Clients” page (Confluence wiki page) for this new client. I did create my login for the Confluence and login with that. But I am not able to edit the page. Do I require to do some other st

Re: Zookeeper DNS TTL

https://issues.apache.org/jira/browse/ZOOKEEPER-1356 which was closed as a dupe of https://issues.apache.org/jira/browse/ZOOKEEPER-338 are relevant to this... It's a zk client issue, and there are things you can do to avoid having to reconfigure the clients while you're bouncing them (CNAMEs and t

Re: Kafka offset moved back on rebalance

Hi, The offsets could belong to different partitions of the same topic. Each partition has its own offset. Try logging *record.partition()*. Regards, Daniel. On Thu, Mar 17, 2016 at 1:58 PM, Erik Pettersson wrote: > Hi, > > I have the 3 consumers subscribing from a single topic with the foll

Re: Not seeing disk writes on all replica brokers

what exactly are you monitoring in terms of the "disk writes"? kafka writes to disk but does not fsync. so you may or may not see activity at the disk when writing; it'll be up to the OS to write back to the disk. On Wed, Mar 16, 2016 at 5:17 PM, Kiran Nagaraja wrote: > Hi, > > I'm running versi

Re: How to assign all the partitions of a topic to consumer ?

Have you looked at partitionsFor()? -Jason On Wed, Mar 16, 2016 at 4:58 AM, Kamal C wrote: > Hi, > > I'm using the new consumer in assign mode. I would like to assign all > the partitions of a topic to the consumer. > > For that, I need to know the number of partitions available in the topi

How to assign all the partitions of a topic to consumer ?

Hi, I'm using the new consumer in assign mode. I would like to assign all the partitions of a topic to the consumer. For that, I need to know the number of partitions available in the topic. *consumer.assign(List partitions);* How to programmatically get the number of partitions in a topic?

Re: Kafka topics with infinite retention?

If you use camus to backup you could use something like this? https://groups.google.com/forum/#!searchin/camus_etl/sumac/camus_etl/hIi1cbOCFvU/grqd7UHLoFUJ Please note I haven't used this, but remember seeing it ages ago. On Wed, Mar 16, 2016 at 2:24 PM, Giidox wrote: > > I'm not sure what HDF

0.9.0.0 and 0.9.0.1 - Metadata fetching from remote server continuously fails - "Failed to find leader"

Hi, I tried 0.9.0.0 as well as 0.9.0.1 client and server combinations. Broker + Zookeeper runs on VM (Ubuntu 14.04) and consumer (console-consumer) as well as producer (Java client) runs on host (windows) Changed advertised.host.name to reflect the VM's hostname - kafka-zook. Due to firewall i

Kafka list

Hi, please add me to the Kafka list. Andreas

Re: Kafka 0.8.2.2 is using high CPU

thanks, the new kafka cluster is stable now, I want to know the root cause. On Sat, Mar 19, 2016 at 3:34 PM, Stevo Slavić wrote: > Consider upgrading to 0.9.0.1 - our cluster is much more stable since > upgrade. Clients can be upgraded later. > > On Fri, Mar 18, 2016, 23:22 li jinyu wrote: > >

Re: Kafka list

You need to send a mail to users-subscr...@kafka.apache.org http://kafka.apache.org/contact.html On Sat, Mar 19, 2016 at 4:14 AM, Andreas Thoelke wrote: > Hi, > > please add me to the Kafka list. > > Andreas > -- *Christian Posta* twitter: @christianposta http://www.christianposta.com/blog h

Re: Larger Size Error Message

DumpLogSegments tool is used to dump partition data logs (not application logs). Usage: ./bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /tmp/kafka-logs/TEST-TOPIC-0/.log Use --key-decoder-class , --value-decoder-class options to pass deserializers. On Fri, Mar 1

Kafka FetchRequest

Hi, I am trying to create a kafka consumer from scratch and I am seeing some that is weird. Note: I am not creating a client group, I want to get the fetch request working first. What I am seeing is that every time I make the request the total size of the length of the response grows. This h

Re: Kafka topics with infinite retention?

> I'm not sure what HDFS backups would bring I’m not sure how realistic the threat is, but I was thinking a case in which bug in Kafka corrupts the log files. I would personally sleep better knowing there is a Kafka-independent backup of all data. > how would you recover from e.g. all Kafka n

Kafka offset moved back on rebalance

Hi, I have the 3 consumers subscribing from a single topic with the following settings. auto.commit.interval.ms = 1000 enable.auto.commit = true request.timeout.ms = 6 session.timeout.ms = 59000 heartbeat.interval.ms = 4000 Version 0.9.0. The consumers looks like this: whi

Kafka Mirroring Issue: Leader Not Available Exception

I am trying to mirror a three node kafka cluster to a single node cluster. When I starts the mirror maker I got the exception LeaderNotAvailable, but at the same time topics are created and messages are commited. Please help me out. Best Regards Anshul Bhatnagar

Zookeeper DNS TTL

Is there any way to make Kafka 0.8.2.2 honor the DNS TTLs of its Zookeeper ensemble? I'm specifying the Zookeepers by hostname, but Kafka doesn't seem to ever re-resolve those hostnames after startup. I've tried setting the relevant JVM system properties via "-Dnetworkaddress.cache.ttl=0 -Dnetw

KafkaStreams for counting/analytics use case

Hi everyone, Is KafkaStream the way to go to allow a light real time counting/analytics that need to do things like ? : 1) Continuously update the total count of currently open purchase orders for each user as purchase orders updates are passing thought the broker. 2) Continuously update the

Re: How to assign all the partitions of a topic to consumer ?

Thanks Jason! It worked. I've missed it. On 17-Mar-2016 2:00 AM, "Jason Gustafson" wrote: > Have you looked at partitionsFor()? > > -Jason > > On Wed, Mar 16, 2016 at 4:58 AM, Kamal C wrote: > > > Hi, > > > > I'm using the new consumer in assign mode. I would like to assign all > > the par

Help: Producer with SSL did not work after upgrading the kafka 0.8.2 to Kafka 0.9

Hi folks, We just finished the upgrade from 0.8.2 to 0.9 with the instructions in Kafka web site (that set the protocol version as 0.8.2.x in Kafka server 0.9). After the upgrade, we want to try the producer with SSL endpoint, but never worked. Here's the error: ~/kafka_2.11-0.9.0.0$ ./bin/kafka-c

Re: Question regarding compression of topics in Kafka

Assuming you’re using the new producer (org.apache.kafka.clients.producer) the property is called compression.type I believe. Double check it’s being passed correctly to the process. The producer logs the properties it uses if you set the logging level to info. B > On 18 Mar 2016, at 16:10, R

Re: Kafka topics with infinite retention?

Since Kafka itself has replication, I'm not sure what HDFS backups would bring – how would you recover from e.g. all Kafka nodes blowing up if you only have an HDFS backup? Why not use MirrorMaker to replicate the cluster to a remote DC, with a process of reversing the direction in case you need to

Re: Larger Size Error Message

DumpLogSegments tool is used to dump partition data logs (not application logs). Usage: ./bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /tmp/kafka-logs/TEST-TOPIC-0/.log Use --key-decoder-class , --key-decoder-class options to pass deserializers. On Fri, Mar 18,

Re: Larger Size Error Message

Before 0.9 and before anyone knows your server host / port can produce request to you unless you have a hardware LB or firewall. In the recent release of 0.9, there is a Security feature added to Kafka, including encryption / authentication / authorization. For your case, I would suggest you upgra

can kafka consumer auto restart on failure

Hi , Lets say I am using a kafka consumer with 4 consumer instances are part of a consumer group. I want to make sure that , if a consumer fails , because of any reason then it should be restarted on its own and should start from the offset it left behind. I know using storm-kafka architecture ,

Re: Question regarding compression of topics in Kafka

Hello Kafka Users, Hoping for best to hear any thoughts from you guys on following questions. Has anybody enabled and tried compression of logs? What are the steps to configure compression of logs on disk? Is there any option to compress existing logs already persisted on disk? Why is compress

Re: Larger Size Error Message

Fang, You can use the kafka.tools.DumpLogSegments to scan and view the logs, but you need the right deserializers to illustrate the content. Guozhang On Wed, Mar 16, 2016 at 4:03 PM, Fang Wong wrote: > Thanks Guozhang! > We are in the process of upgrading to 0.9.0.0. We will look into using >

Re: Kafka Streams Parallelism

Hello Luis, Thank you for your detailed question! The short answer to your question: yes, Kafka Streams does apply re-partitioning for aggregations and joins, etc so that the streams can be (co-)partitioned by key. In your specific example, after the split operator, Kafka Streams will write the w

Not seeing disk writes on all replica brokers

Hi, I'm running version kafka_2.11-0.9.0.1 in a 2 broker configuration with 10 partitions that are replicated - replication.factor=2 Here's the output from 'kafkat partitions': TopicPartitionLeaderReplicasISRs bookmarks00[0, 1]

Re: Kafka Streams question

Hi Mike I'm don't have much experience with Kafka Streams yet, but from the common sense point of view, maybe it would be easier to model as Kafka Stream for actual data processing with output to another topic, that would be consumed by Kafka Connect's sinks? I see one in discussion for HBase

Re: Help understanding what happened

If you hard kill the broker when it restarts it doesn't know the status of it's on-disk files, it will need to run though the last log segment to validate the checksums of messages and rebuild the index off this to ensure consistency. (Why does it need to do this validation? Because in the event of

Reg : Unable to produce message

All, I am getting this error for few topics WARN Error while fetching metadata with correlation id 4802 : {next=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) WARN Error while fetching metadata with correlation id 4803 : {next=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.Netwo

Kafka offset moved back on rebalance

Hi, I have the 3 consumers subscribing from a single topic with the following settings. auto.commit.interval.ms = 1000 enable.auto.commit = true request.timeout.ms = 6 session.timeout.ms = 59000 heartbeat.interval.ms = 4000 Version 0.9.0. The consumers looks like this: whi

Re: Reg : Unable to produce message

We may get few warning exceptions, on first produce to unknown topic , with default server config property auto.create.topics.enable = true. If this is the case, then it is harmless exception. On Sun, Mar 20, 2016 at 11:19 AM, Mohamed Ashiq wrote: > All, > > I am getting this error for few topic