RE: Http Kafka producer

Ewen, Thanks for the valuable information. I will surely try this and come up with my comments. Thanks again Hemanth -Original Message- From: Ewen Cheslack-Postava [mailto:e...@confluent.io] Sent: Thursday, August 27, 2015 10:21 AM To: users@kafka.apache.org Subject: Re: Http Kafka pro

Re: Http Kafka producer

Apologies if this is somewhat redundant, I'm quite new to both Kafka and the Confluent Platform. Ewen, when you say "Under the hood, the new producer will automatically batch requests." Do you mean that this is a current or planned behavior of the REST proxy? Are there any durability guarantee

Re: Http Kafka producer

Hemanth, The Confluent Platform 1.0 version of have JSON embedded format support (i.e. direct embedding of JSON messages), but you can serialize, base64 encode, and use the binary mode, paying a bit of overhead. However, since then we merged a patch to add JSON support: https://github.com/confluen

RE: Http Kafka producer

Ewen, Thanks for the explanation. We have control over the logs format coming to HAProxy. Right now, these are plain JSON logs (just like syslog messages with few additional meta information) sent to HAProxy from remote clients using HTTPs. No serialization is used. Currently, we have one log

Re: Http Kafka producer

Hemanth, Can you be a bit more specific about your setup? Do you have control over the format of the request bodies that reach HAProxy or not? If you do, Confluent's REST proxy should work fine and does not require the Schema Registry. It supports both binary (encoded as base64 so it can be passed

RE: Http Kafka producer

Marc, Thanks for your response. Let's have more details on the problem. As I already mentioned in the previous post, here is our expected data flow: logs -> HAProxy -> {new layer } -> Kafka Cluster The 'new layer' should receive logs as HTTP requests from HAproxy and produce the same logs to

Re: Http Kafka producer

I'm actually also really interested in this...I had a chat about this on the distributed systems slack's Kafka channel a few days ago, but we're not much further than griping about the problem. We're basically migrating an existing event system, one which packed messages

Http Kafka producer

Hi, Our application receives events through a HAProxy server on HTTPs, which should be forwarded and stored to Kafka cluster. What should be the best option for this ? This layer should receive events from HAProxy & produce them to Kafka cluster, in a reliable and efficient way (and should scal

Reg Leader re-election when leader crashes

Hi, I have configured 3 brokers and 3 zookeepers in different unix boxes. All are receiving message successfully. 1. Now leader broker got crashed and when I try to publish 10 message. some of them are getting failed with org.apache.kafka.common.errors.TimeoutException: Failed to update metadata

Re: Mirror a partition of a topic

Thank you everyone. On Tue, Aug 25, 2015 at 11:49 PM, Kishore Senji wrote: > Flume could be an option with an Interceptor although the throughput could > be less compared to Mirror Maker with compression and shallow iterator > enabled. > On Tue, Aug 25, 2015 at 10:28 PM tao xiao wrote: > > > In

Re: Storing offsets in ZooKeeper for a SimpleConsumer

Hi Buck, So, if you are on 0.8.2, you can use the SimpleConsumer¹s commitOffsets function which allows you to commit offsets to either zookeeper or a kafka topic. I would avoid rolling your own solution that edits data within existing zookeeper paths because it might make upgrading to future versi

Re: 0.8.x

That’s a fair few connections indeed! You may be able to take the route of pinning producers to specific partitions. KIP-22 made this easier by exposing a partitioner interface to pr

Re: Painfully slow kafka recovery / cluster breaking

Thanks for updates Jörg. It's very useful. Thanks, Raja. On Wed, Aug 26, 2015 at 8:58 AM, Jörg Wagner wrote: > Just a little feedback on our issue(s) as FYI to whoever is interested. > > It basically all boiled down to the configuration of topics. We noticed > while performance testing (or tryi

Re: Painfully slow kafka recovery / cluster breaking

Just a little feedback on our issue(s) as FYI to whoever is interested. It basically all boiled down to the configuration of topics. We noticed while performance testing (or trying to ;) ) that the partitioning was most critical to us. We originally followed the linkedin recommendation and us

Re: 0.8.x

Hi Ben, Yes we have a VIP fronting our kafka 0.7 clusters. The producers connect with a broker list that just points to the VIP. In 0.8, with the new producer at least, the producer doesn't go through Zookeeper - which is good. However, all producers will connect to all brokers to send messages.

Re: 0.8.x

Hi Damian Just clarifying - you’re saying you currently have Kafka 0.7.x running with a dedicated broker addresses (bypassing ZK) and hitting a VIP which you use for load balancing writes. Is that correct? Are you worried about something specific in the 0.8.x way of doing things (ZK under that