Hi,
Right, broker on localhost and localhost connection don't help if the
broker is actually down. It's not only about network reachability and
such. We write to FS (yes, file system) as "well, what is the simplest
thing that we can do and where we are least likely to hit some other
issues? Writ
Not exactly. But we do pay an enormous amount of attention to our
producer-Kafka-consumer subsystems. They are certainly mission-critical for us.
Philip
> On Nov 28, 2013, at 4:12 PM, Steve Morin wrote:
>
> Philip,
> Do you do that at loggly?
>
> Otis,
> How was your retry code structured?
Sure, and the disk could go bad, the machine itself could fail.
My point is that my experience of Kafka 0.72 has been that it is very reliable.
The only time I have seen it go down is when the disk underneath fills up. So
if one is going to write all the code to stream to disk *efficiently* fro
There are many options. Another simple consumer could read from it to a second
broker.
Philip
> On Nov 28, 2013, at 4:18 PM, Steve Morin wrote:
>
> Philip,
> How would do you mirror this to a main Kafka instance?
> -Steve
>
>> On Nov 28, 2013, at 16:14, Philip O'Toole wrote:
>>
>> I shoul
Philip, what about if the broker goes down?
I may be missing something.
Diego.
El 28/11/2013 21:09, "Philip O'Toole" escribió:
> By FS I guess you mean file system.
>
> In that case, if one is that concerned, why not run a single Kafka broker
> on the same machine, and connect to it over localh
Philip,
How would do you mirror this to a main Kafka instance?
-Steve
> On Nov 28, 2013, at 16:14, Philip O'Toole wrote:
>
> I should add in our custom producers we buffer in RAM if required, so Kafka
> can be restarted etc. But I would never code streaming to disk now. I would
> just run a
I should add in our custom producers we buffer in RAM if required, so Kafka can
be restarted etc. But I would never code streaming to disk now. I would just
run a Kafka instance on the same node.
Philip
On Nov 28, 2013, at 4:08 PM, Philip O'Toole wrote:
> By FS I guess you mean file system.
Philip,
Do you do that at loggly?
Otis,
How was your retry code structured? Have you open sourced it?
> On Nov 28, 2013, at 16:08, Philip O'Toole wrote:
>
> By FS I guess you mean file system.
>
> In that case, if one is that concerned, why not run a single Kafka broker on
> the same ma
By FS I guess you mean file system.
In that case, if one is that concerned, why not run a single Kafka broker on
the same machine, and connect to it over localhost? And disable ZK mode too,
perhaps.
I may be missing something, but I never fully understand why people try really
hard to build
Hi,
We've done this at Sematext, where we use Kafka in all 3 products/services
you see in my signature. When we fail to push a message into Kafka we
store it in the FS and from there we can process it later.
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch S
I would say follow Sparkngin and you'll be able to use the "Log
persistence if the log producer connection is down". functionality when
it's complete.
On Thu, Nov 28, 2013 at 8:36 AM, Demian Berjman wrote:
> Steve, our use case is very simple. There are many reasons for a cluster to
> go down.
Steve, our use case is very simple. There are many reasons for a cluster to
go down. If that the case, what we do with the producers? Hopefully it will
be a time window of a couple of hours. If your concern are the queued
messages, we have only a few thousands per day.
Thanks,
On Thu, Nov 28, 20
What I mean by that is that your looking to have the Kafka cluster able to
be down for like 5 minutes or upto a day. The problem is estimating how
long it will take to recover.
Is this work your doing for a consulting project? Or are you doing
something on behalf of an employer. Basically would
Steve, thanks for the response! I don't understand what you mean by "what
kind of window?"
I am looking for something like, i think you did it in Sparkngin: "Log
persistence if the log producer connection is down".
Demian,
I have been looking at building that into Sparkngin (
https://github.com/DemandCube/Sparkngin) What kind of window are you
looking for?
-Steve
On Thu, Nov 28, 2013 at 7:23 AM, Demian Berjman wrote:
> Joe, i meant that all the kafka cluster is down, even the replicas of that
> topic.
Joe, i meant that all the kafka cluster is down, even the replicas of that
topic.
Thanks,
On Thu, Nov 28, 2013 at 12:10 PM, Joe Stein wrote:
> Have you tried the replication feature added to 0.8
> http://kafka.apache.org/documentation.html#replication
>
>
> /***
Have you tried the replication feature added to 0.8
http://kafka.apache.org/documentation.html#replication
/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
*
17 matches
Mail list logo
