Re: What is the best way to write Kafka data into HDFS?

ractices and options to write data from Kafka to HDFS? > > Thanks, > R P > > > > -- *Steve Morin | Managing Partner - CTO* *Nvent* O 800-407-1156 ext 803 <800-407-1156;803> | M 347-453-5579 smo...@nventdata.com *Enabling the Data Driven Enterprise* *(Ask us how w

Re: Kakfa question about starting kafka with same broker id

Why would you want to ever do that? > On Feb 18, 2015, at 15:16, Deepak Dhakal wrote: > > Hi, > > My name is Deepak and I work for salesforce. We are using kafka 8.11 and > have a question about starting kafka with same broker id. > > Steps: > > Start a kakfa broker with broker id =1 -> it s

Re: New Producer - ONLY sync mode?

didn't make use of it, but you definitely can make use of it if you > want to. > > -Jay > > On Wed, Feb 4, 2015 at 9:58 AM, Steve Morin wrote: > > > Looking at this thread I would ideally want something at least the right > > recipe to mimic sync behavior like Oti

Re: [ANNOUNCEMENT] Apache Kafka 0.8.2.0 Released

Congratz team it's a big accomplishment > On Feb 5, 2015, at 14:22, Otis Gospodnetic wrote: > > Big thanks to Jun and everyone else involved! We're on 0.8.2 as of today. > :) > > Otis > -- > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > Solr & Elasticsearch Support *

Re: New Producer - ONLY sync mode?

Looking at this thread I would ideally want something at least the right recipe to mimic sync behavior like Otis is talking about. In the second case, would like to be able to individually know if messages have failed even regardless if they are in separate batches, sort of like what Kinesis does

Re: Client Offset Storage

Suren, Like out of the box storage or roll your own? -Steve On Fri, Dec 12, 2014 at 6:33 AM, Surendranauth Hiraman < suren.hira...@velos.io> wrote: > My team is using Kafka 0.8.1 and we may not be able to upgrade to 0.8.2 to > take advantage of the broker-side commit of client offsets. > > Is a

Re: Programmatic Kafka version detection/extraction?

That would be great! On Mon, Nov 10, 2014 at 9:45 PM, Jun Rao wrote: > Otis, > > We don't have an api for that now. We can probably expose this as a JMX as > part of kafka-1481. > > Thanks, > > Jun > > On Mon, Nov 10, 2014 at 7:17 PM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > > >

Re: Announcing Confluent

Jay, Neha and Jun congratz!! On Thu, Nov 6, 2014 at 11:09 AM, Bhavesh Mistry wrote: > HI Guys, > > Thanks for your awesome support. I wish you good luck !! Thanks for open > sources Kafka !! > > Thanks, > > Bhavesh > > On Thu, Nov 6, 2014 at 10:52 AM, Rajasekar Elango > wrote: > > > Congrats

Re: migrating log data to new locations

Neha, I log volume or can it be volumes plural? -Steve On Tue, Oct 7, 2014 at 6:41 AM, Neha Narkhede wrote: > Is it possible to perform this migration without losing the data currently > stored in the kafka cluster? > > Though I haven't tested this, the way this is designed should allow you to

Re: Right Tool

hat format I would write it in yet, I > guess json > > On Fri, Sep 12, 2014 at 7:00 PM, Steve Morin > wrote: > > > What record format are you writing to Kafka with? > > > > > On Sep 12, 2014, at 17:45, Patrick Barker > > wrote: > > > > > &g

Re: Right Tool

What record format are you writing to Kafka with? > On Sep 12, 2014, at 17:45, Patrick Barker wrote: > > O, I'm not trying to use it for persistence, I'm wanting to sync 3 > databases: sql, mongo, graph. I want to publish to kafka and then have it > update the db's. I'm wanting to keep this as e

Re: Embedded Kafka/Zookeeper for unit testing

We have built that into NeverwinterDP for testing if you want to see an open-source example of it. We actually have embedded elastic search and elements of Hadoop to for testing and developer mode. https://github.com/DemandCube/NeverwinterDP > On Aug 16, 2014, at 7:13, Vjeran Marcinko > wrot

Re: Consume more than produce

You have to remember statsd uses udp and possibly lossy which might account for the errors. -Steve On Fri, Aug 1, 2014 at 1:28 AM, Guy Doulberg wrote: > Hey, > > > After a year or so I have Kafka as my streaming layer in my production, I > decided it is time to audit, and to test how many event

Re: kafka support in collectd and syslog-ng

Cool On Fri, Jul 25, 2014 at 9:25 AM, Joe Stein wrote: > Awesome! > > /*** > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop

Re: Kafka on yarn

Kam, Give it some time and think it's getting better as a real possibility for Kafka on Yarn. There are new capabilities coming out in Yarn/HDFS to allow for node groups/label that can work with locality and secondarily new functionality in HDFS that depending on the use-case can be very interes

Re: Performance/Stress tools

andy for observing various Kafka metrics while you run your > performance tests. > > Otis > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > > > > On Wed, Jul 16, 2014 at 1:07 PM, Steve Morin wrot

Re: Performance/Stress tools

We are working on this Yarn based tool , but it's still in alpha https://github.com/DemandCube/DemandSpike On Wed, Jul 16, 2014 at 11:30 AM, Dayo Oliyide wrote: > Hi, > > I'm setting up a Kafka Cluster and would like to carry out some > performance/stress tests on different configurations. > >

Re: kafka producer pulling from custom restAPI

The answer is no, it doesn't work that way. You would have to write a process to consume from the API back end and have that back end write to Kafka > On Jun 27, 2014, at 19:35, wrote: > > Hi, > > I have a quick question. Say I have a custom REST API with data in JSON > format. Can the Kafka

Re: Building Kafka on Mac OS X

Have seen if you have a write with zero data it will hang > On Jun 16, 2014, at 21:02, Timothy Chen wrote: > > Can you try running it in debug mode? (./gradlew jar -d) > > Tim > >> On Mon, Jun 16, 2014 at 8:44 PM, Jorge Marizan >> wrote: >> It just hangs there without any output at all. >>

Re: Use Kafka To Send Files

You also wouldn't have any meta data about the file so I would avoid doing this. > On Jun 15, 2014, at 20:51, Mark Roberts wrote: > > You would ship the contents of the file across as a message. In general this > would mean that your maximum file size must be smaller than your maximum > messag

Re: Make kafka storage engine pluggable and provide a HDFS plugin?

Hangjun, Does having Kafka in Yarn would be a big architectural change from where it is now? From what I have seen on most typical setup you want machines optimized for Kafka, not just it on top of hdfs. -Steve On Tue, May 20, 2014 at 8:37 PM, Hangjun Ye wrote: > Thanks Jun and Francois. > >

Re: Java API to monitor Consumer Offset and Lag

Otis, What's SPM that your referencing? -Steve On Fri, Mar 28, 2014 at 4:45 PM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hi Harsh, > > Are you trying to monitor lag? > SPM for Kafka has this in one of the graphs. We get this data from JMX. > There are other tools that were men

Re: Anouncing Kafka Offset Monitor 0.1

Very nice > On Mar 7, 2014, at 11:55, Pierre Andrews wrote: > > Claude, we should join forces ;) > > >> On Fri, Mar 7, 2014 at 4:45 PM, Claude Mamo wrote: >> >> Awesome!!! ;-) >> >> Claude >> >> >> On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews >> wrote: >> >>> Great! Thanks! >>> >>> >

Re: Logging in new clients

My vote would be with log4j, I don't have that much experience with log4j2 or a good feel for how much the industry is moving towards it. On Mon, Feb 3, 2014 at 11:17 AM, Joel Koshy wrote: > We are already using other libraries in various parts of our code > (e.g., metrics, zkclient, joptsimpl

Re: There is insufficient memory for the Java Runtime Environment to continue.

Do you have anything like Graphite or Ganglia monitoring the box to see exactly what's going on? On Fri, Jan 31, 2014 at 1:45 PM, David Montgomery wrote: > Welll...I did get kafka to run on a digiocean box with 4 gigs or ram. All > great but now i am paying 40 USD a month for dev servers when

Re: New Producer Public API

Is the new producer API going to maintain protocol compatibility with old version if the API under the hood? > On Jan 29, 2014, at 10:15, Jay Kreps wrote: > > The challenge of directly exposing ProduceRequestResult is that the offset > provided is just the base offset and there is no way to kno

Re: Kafka @ apachecon Denver?

Are you using mesos? > On Jan 27, 2014, at 8:39, Joe Stein wrote: > > I was going to submit a talk on Kafka and Mesos. I still am trying to nail > down the dates in my schedule though. Anyone else going? Maybe we could do a > meetup or bof or something? > > > /*

Re: Anyone working on a Kafka book?

Would you mind sharing your connection setup? > On Dec 13, 2013, at 10:36, Shafaq wrote: > > Thats already done. > >> On Dec 13, 2013 9:28 AM, "Steve Morin" wrote: >> Shafaq, >> That's pretty cool, have you already connected Kafka to spark RRD

Re: Anyone working on a Kafka book?

I/O. > > Next, we would be using featurizing the stream to be able to develop > machine learning models using SVMs (Support Vector Machines) etc to provide > rich insights. > > Would be soon giving talk on this one, so keep tuned. > > Regards, > S.Abdullah > > > On Tue,

Re: Anyone working on a Kafka book?

Dec 10, 2013 at 9:15 AM, chetan conikee wrote: > >> Hey Guys >> >> Yes, Ben Lorica (Oreilly) and I are planning to pen a "Beginning Kafka" >> book. >> We only finalized this late October are hoping to start this mid-month >> >> Chetan &g

Re: Anyone working on a Kafka book?

I'll let chetan comment if he's up for it. -Steve On Tue, Dec 10, 2013 at 8:40 AM, David Arthur wrote: > There was some talk a few months ago, not sure what the current status is. > > > On 12/10/13 10:01 AM, S Ahmed wrote: > >> Is there a book or this was just an idea? >> >> >> On Mon, Mar 25,

Re: Anyone working on a Kafka book?

I forget but think Chetan was with oreilly > On Dec 10, 2013, at 7:01, S Ahmed wrote: > > Is there a book or this was just an idea? > > > On Mon, Mar 25, 2013 at 12:42 PM, Chris Curtin wrote: > >> Thanks Jun, >> >> I've updated the example with this information. >> >> I've also removed some

Re: Using Kafka 0.8 from Scala and Akka

Chetan, Are you also releasing a Scala RxJava producer as well? -Steve On Tue, Dec 3, 2013 at 10:42 PM, Richard Rodseth wrote: > Any update on this, Chetan? Thanks. > > > On Thu, Oct 31, 2013 at 4:11 PM, chetan conikee wrote: > > > I am in the process of releasing out Scala and RxJava consum

Re: Loggly's use of Kafka on AWS

Philip this is definitely useful > On Dec 2, 2013, at 14:55, Surendranauth Hiraman > wrote: > > S Ahmed, > > This combination of Kafka and Storm to process streaming data is becoming > pretty common. Definitely worth looking at. > > The throughput will vary depending on your workload (cpu usa

Re: kafka producer - retry messages

Philip, How would do you mirror this to a main Kafka instance? -Steve > On Nov 28, 2013, at 16:14, Philip O'Toole wrote: > > I should add in our custom producers we buffer in RAM if required, so Kafka > can be restarted etc. But I would never code streaming to disk now. I would > just run a

Re: kafka producer - retry messages

Philip, Do you do that at loggly? Otis, How was your retry code structured? Have you open sourced it? > On Nov 28, 2013, at 16:08, Philip O'Toole wrote: > > By FS I guess you mean file system. > > In that case, if one is that concerned, why not run a single Kafka broker on > the same ma

Re: kafka producer - retry messages

a cluster to > go down. If that the case, what we do with the producers? Hopefully it will > be a time window of a couple of hours. If your concern are the queued > messages, we have only a few thousands per day. > > Thanks, > > > On Thu, Nov 28, 2013 at 1:12 PM, Steve Morin

Re: kafka producer - retry messages

What I mean by that is that your looking to have the Kafka cluster able to be down for like 5 minutes or upto a day. The problem is estimating how long it will take to recover. Is this work your doing for a consulting project? Or are you doing something on behalf of an employer. Basically would

Re: kafka producer - retry messages

Demian, I have been looking at building that into Sparkngin ( https://github.com/DemandCube/Sparkngin) What kind of window are you looking for? -Steve On Thu, Nov 28, 2013 at 7:23 AM, Demian Berjman wrote: > Joe, i meant that all the kafka cluster is down, even the replicas of that > topic.

Re: producer exceptions when broker dies

Kane and Aniket, I am interested in knowing what the pattern/solution that people usually use to implement exactly once as well. -Steve On Fri, Oct 25, 2013 at 11:39 AM, Kane Kane wrote: > Guozhang, but i've posted a piece from kafka documentation above: > So effectively Kafka guarantees at-l

Re: Flush configuration per topic

Is there only time delay or can it be set to flush for every message with the obvious performance hit? On Wed, Oct 16, 2013 at 9:49 AM, Jay Kreps wrote: > Yes, the change in trunk is that all log configurations are automatically > available at both the log level and the global default level and

Re: Kafka and Zookeeper node removal from two nodes Kafka cluster

If you have a double broker failure with replication factor of 2 and only have 2 brokers in the cluster. Wouldn't every partition be not available? On Tue, Oct 15, 2013 at 8:48 AM, Jun Rao wrote: > If you have double broker failures with a replication factor of 2, some > partitions will not be