Re: How to use kafka for log aggregation

Hi Snehalata, I did a presentation on a this topic a few months back. Here are the slides: https://docs.google.com/a/joekiller.com/presentation/d/1qNkFdm3fMOEc2p7qfp3g1I1riDS-7IBTT5CYnFW-QQI/edit?usp=docslist_api Hope this gives you a start! Let me know if you have questions. Sincerely, Joe

Re: abstracting ZooKeeper

Netflix contributed Curator (http://curator.apache.org/) to Apache which implements some generic zk recipes. On Aug 9, 2015 11:39 AM, "Julio Castillo" wrote: > Had there been any thought at abstracting the interface to ZooKeeper? > > The reason I'm asking is because I'm looking at Consul for serv

Re: abstracting ZooKeeper

Inline responses below. Sincerely, Joe Lawson On Aug 9, 2015 1:52 PM, "Julio Castillo" wrote: > > Thank for the lead. > Does that mean that Kafka is/will be using Curator? I don't think so. > > Also, this appears to simplify the interaction with ZooKeeper, but

Re: how to get single record from kafka topic+partition @ specified offset

Ewen, Do you have an example or link for the changes/plans that will bring the benefits you describe? Cheers, Joe Lawson On Aug 10, 2015 3:27 PM, "Ewen Cheslack-Postava" wrote: > You can do this using the SimpleConsumer. See > > https://cwiki.apache.org/confluence/

Re: how to get single record from kafka topic+partition @ specified offset

gt;ConsumerRecords records = consumer.poll(); >if (!records.isEmpty()) { > // records[0] will be the message you wanted > break; >} > } > > > > On Mon, Aug 10, 2015 at 3:52 PM, Joe Lawson < > jlaw...@opensourceconnections.com> wrote: > &g

Re: Kafka on AWS - issue publishing from remote producer

Make sure you have host.name in you server properties setup right. I usually give it the ec2 DNS name. Another case where that helped: https://discuss.elastic.co/t/logstash-kafka-output- plugins

Re: Release of 0.9.0

I think RC3 is getting voted on so it's close. Kafka I don't think the Kafka community ever has "release dates". It's more that RC's get voted on and then if they are good, release happens. So just watch for RC votes. On Thu, Nov 19, 2015 at 1:16 PM, Mohit Anchlia wrote: > Is there a tentative d

jruby-kafka v2.0 and v1.5 released!

ssues to the project here: https://github.com/joekiller/jruby-kafka The release notes are below and at https://github.com/joekiller/jruby-kafka/blob/master/CHANGELOG. Thank you for the support! Sincerely, Joe Lawson 2.0 (December 31, 2015) # general - Updated integration tests to suppor

Re: kafka “stops working” after a large message is enqueued

Make sure the topic is created after message Max bytes is set. On Feb 2, 2016 9:04 PM, "Tech Bolek" wrote: > I'm running kafka_2.11-0.9.0.0 and a java-based producer/consumer. With > messages ~70 KB everything works fine. However, after the producer enqueues > a larger, 70 MB message, kafka appe

Re: kafka “stops working” after a large message is enqueued

ed to consume the message. > Wondering why the consumer remained silent, i.e. no out of heap memory > error or anything. > > On Tuesday, February 2, 2016 8:35 PM, Joe Lawson < > jlaw...@opensourceconnections.com> wrote: > > > Make sure the topic is created after me

JRuby Kafka v2.1 (0.8.2.2) and v3.1 (0.9.0.1) released.

that when jruby-kafka is installed no unnecessary downloads are performed. Enjoy! -Joe Lawson

Re: Kafka DNS Caching in AWS

In the past on classic EC2 with an autoscaling group of zookeeper instances, I've used elastic IPs for my list. There we subscribed an SQS queue to the autoscaling SNS topic and when a new instances was brought online one of the spare IPs was allocated to the instance. It has to try over and over s

Re: kafka 0.9 running on AWS environment caching zookeeper IP instead of using the DNS

This recently came up on the list, check out: http://search-hadoop.com/m/uyzND12ZzwunNJzS&subj=Kafka+DNS+Caching+in+AWS On Mon, Aug 15, 2016 at 10:46 AM, Digumarthi, Prabhakar Venkata Surya < prabhakarvenkatasurya.digumar...@capitalone.com> wrote: > Hi All, > > > We are using kafka version 0.9 an

Re: kafka 0.9 running on AWS environment caching zookeeper IP instead of using the DNS

11 AM, Joe Lawson < jlaw...@opensourceconnections.com> wrote: > This recently came up on the list, check out: http://search-hadoop.com/ > m/uyzND12ZzwunNJzS&subj=Kafka+DNS+Caching+in+AWS > > On Mon, Aug 15, 2016 at 10:46 AM, Digumarthi, Prabhakar Venkata Surya < &

Re: Getting maximum message size

The maximum message size is all dependent on the message max bytes you set on the topic (or global broker). You want to make sure the consumer max is in sync with the broker max. Remember that pumping huge messages though can saturate your network quickly. Try to trim down message structure to onl