Re: Reg: SSL setup

Hi Bharat, As a disclaimer, I haven't tried it, but it seems like you would have to import: - Each server's certificate into the truststore of each client - (if ssl.client.auth=required) Each client's certificate into the truststore of each server - (if security.inter.broker.protocol=SSL) Each se

kakfa-console-consumer multiple topics

Is there a way to have the kafka-console-consumer read from multiple topics? I'm using this for some testing and that would be helpful. -Dave This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual

[VOTE] 0.10.0.1 RC2

Hello Kafka users, developers and client-developers, This is the third candidate for the release of Apache Kafka 0.10.0.1. This is a bug fix release and it includes fixes and improvements from 53 JIRAs (including a few critical bugs). See the release notes for more details: http://home.apache.org

Re: kakfa-console-consumer multiple topics

You can pass a pattern string using whitelist config option. ex: sh kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --whitelist ".*" On Thu, Aug 4, 2016 at 7:14 PM, Tauzell, Dave wrote: > Is there a way to have the kafka-console-consumer read from multiple > topics? I

Kafka REST proxy performance vs Akka HTTP

Hi all, I am familiar with Akka HTTP and have built a prototype of an application that ingest data to kafka topic with Akka HTTP. My app is using a new Kafka Producer shared among multiple Akka actors behind a router. Apps are dedicated per customer and launch in Mesos/Marathon containers. I have

Kafka Streams on Windows?

Hey all, Is it anyone developing Kafka Streams applications on Windows? It seems like the RocksDB Java library doesn't include a native JNI library for Windows, which prevents a Kafka Streams app from running on Windows. I was just wondering if others have run into this, and if so, what approach

Re: Unable to write, leader not available

Using the --producer option in kafka-acls.sh it looks to have allowed create operations on the cluster. Turning on trace logging for authorization shows repeated mentions of my user and that its allowed to create on the cluster and describe the topic. Looks like I might not be the only one with th

Kafka Streams input

Hi community, Can Kafka Streams use input sources other than Kafka? For example can it read data from HDFS? You can assume that for each Streams instance initiating, I will have separate partition (in this case file path containing data) and each instance will read from one file path. Cheers Jey

Changing producer or consumer configuration during runtime.

Hello. Let's say that I have a producer (or consumer) running. Does Kafka allow me to add, remove or change its configuration? As an example, could I change compression.type or buffer.memory without first stopping the producer? Cheers, Rodrigo

Re: Kafka Streams input

Hi Jeyhun, You can use Kafka Connect to bring data into Kafka from HDFS then use Kafka Streams as usual. I believe there is a desire to make the Connect + Streams more integrated so it doesn't feel like you're running two different things, and hopefully that integration happens soon. Thanks E

Re: Unable to write, leader not available

I managed to replicate this issue using the default provided config for kafka/zookeeper in the binary artifact and using the console producer to write a message On Thu, Aug 4, 2016 at 10:23 AM Bryan Baugher wrote: > Using the --producer option in kafka-acls.sh it looks to have allowed > create o

Re: Kafka Streams on Windows?

Hi Mathieu, Have you had a chance to look at http://rocksdb.org/blog/2033/rocksdb-is-now-available-in-windows-platform/? Curious to hear your and other's comments on whether that worked. It is possible to have Kafka

Re: Unable to write, leader not available

Figured this out. This had to do with me being on a VPN and running everything locally On Thu, Aug 4, 2016 at 11:26 AM Bryan Baugher wrote: > I managed to replicate this issue using the default provided config for > kafka/zookeeper in the binary artifact and using the console producer to > write

Re: Kafka Streams on Windows?

Hi Eno, Yes, I've looked at that. RocksDB can be built and run in Windows, but, the JNI wrapper does not include Windows binarie ( https://github.com/facebook/rocksdb/issues/703). rocksdbjni-4.4.1.jar includes librocksdbjni-linux32.so, librocksdbjni-linux64.so, and librocksdbjni-osx.jnilib, so o

No removal of old replicas following re-balance in 0.8.0?

Today following a re-balance operation in our 0.8.0 cluster , we had a bunch of disks fill up, even though our retention property values are designed to prevent that level of disk usage. When we investigated, it appeared the following had occurred: 1. Broker A was in the replica list for partition

Re: Connection to broker which lost Zk connection

Hi, I am not an expert, but as I can say as an ops guy is that when a broker loses its connection with Zk, it triggers a reelection and A will be out of ISR. So no data will go to broker A anymore and during this timeframe (from connection lost to reelection) the producer should receive an Ex

Re: SSD or not for Kafka brokers?

Hi, You might consider SSD specially for reads - if your consumers are the bottleneck (as kafka usually sends sequential bulk messages from memory, it won't add much value to writes). Latency can be also a factor to move to SSD (as seek time can vary from 0.2 to 0.8ms on magnetic discs compa

Re: Kafka consumers in cluster

Hi Kamal, Thanks for your prompt response. Does our custom partition assignor gets called during every rebalancing. Thanks and Regards A.SathishKumar >Implement your own custom >`org.apache.kafka.clients.consumer.internals.PartitionAssignor` >and assign all the subscribed partitions to the fir

Re: Re : A specific use case

Yeah, if you can buffer yourself in the process() function and then rely on punctuate() for generating the outputs that would resolve your issue. Remember that punctuate() function itself is event-time driven so if you do not have any data coming in then it may not be triggered. Details: https://

Re: Issue with KTable State Store

I remember observing this issue, that whenever you received a new value and tries to get the old aggregate value from store, it always returns null and hence you never aggregate the old value with new record. I remember I fixed that issue but cannot remember how.. So I would recommend: 1) Try comp

Runtime JARs for 0.10 Producer and Consumer

Hi, This is slightly related to the "Jars in Kafka 0.10" thread. I am trying to figure out exactly which JARs I need at runtime for the 0.10 Kafka Producer and Kafka Consumer clients. Are the compile time dependencies and runtime dependencies documented anywhere I simply can't find? I know for su

Re: Kafka REST proxy performance vs Akka HTTP

On Thu, Aug 4, 2016 at 2:11 AM, vincent gromakowski < vincent.gromakow...@gmail.com> wrote: > Hi all, > I am familiar with Akka HTTP and have built a prototype of an application > that ingest data to kafka topic with Akka HTTP. > My app is using a new Kafka Producer shared among multiple Akka acto

Re: Changing producer or consumer configuration during runtime.

No, these configs are static. On the Kafka side you can change things like topic configurations dynamically, but on the clients we assume that the config you pass in at initialization is valid for the lifetime of the app. If you want dynamic configs you'd have to provide your own mechanism for this

Re: Runtime JARs for 0.10 Producer and Consumer

Alex, Normally you'd just depend on kafka-clients 0.10.0.0 and let any transitive dependencies get pulled in automatically. This snippet from the Kafka build files shows the libraries that clients depend on: dependencies { compile libs.lz4 compile libs.snappy compile libs.slf4jApi So basi

Re: Kafka consumers in cluster

Yes, it gets called on every re-balance. -- Kamal On Thu, Aug 4, 2016 at 11:24 PM, sat wrote: > Hi Kamal, > > Thanks for your prompt response. Does our custom partition assignor gets > called during every rebalancing. > > Thanks and Regards > A.SathishKumar > > > > >Implement your own custom >

Re: [VOTE] 0.10.0.1 RC2

+1 (non-binding) On Thu, Aug 4, 2016 at 6:54 AM, Ismael Juma wrote: > Hello Kafka users, developers and client-developers, > > This is the third candidate for the release of Apache Kafka 0.10.0.1. This > is a bug fix release and it includes fixes and improvements from 53 JIRAs > (including a few

Compress before or after serialization?

We are using Avro as our format when writing to a Kafka topic. After we enabled snappy compression on the producer, we don't see a change in the compression ratio (still 1). I was wondering if we should compress the message before serialization. Thanks, David