I think it'd be possible to avoid special-casing replacements, but it might
be a bad idea network-traffic-wise, especially for rolling upgrades.
My experience running Kafka on AWS is that rebalancing with multi-day
retention periods can take a really long time, and can torch the cluster if
you reb
It would be cool to know if Netflix runs kafka in ASGs ... I can't find any
mention of it online. (https://github.com/Netflix/suro/wiki/FAQ sorta
implies that maybe they do, but it's not clear, and also old.)
I've seen other people talking about running kafka in ASGs, e.g.
http://blog.jimjh.com/b
Great talks, but not relevant to either of my problems -- the golang client
not rebalancing the consumer offset topic, or autoscaling group behavior
(which is I think is probably just a consequence of the first).
Thanks though, there's good stuff in here.
On Sun, Jul 3, 2016 at 10:23 AM, James Ch
Charity,
I'm not sure about the specific problem you are having, but about Kafka on AWS,
Netflix did a talk at a meetup about their Kafka installation on AWS. There
might be some useful information in there. There is a video stream as well as
slides, and maybe you can get in touch with the spea
Gwen, thanks for the response.
1.1 Your life may be a bit simpler if you have a way of starting a new
> broker with the same ID as the old one - this means it will
> automatically pick up the old replicas and you won't need to
> rebalance. Makes life slightly easier in some cases.
>
Yeah, this i
Charity,
1. Nothing you do seems crazy to me. Kafka should be able to work with
auto-scaling and we should be able to fix the issues you are running
into.
There are few things you should be careful about when using the method
you described though:
1.1 Your life may be a bit simpler if you have a
Reasons.
Investigated it thoroughly, believe me. Some of the limitations that
Kinesis uses to protect itself are non starters for us.
Forgot to mention, we are using 0.9.0.1-0.
On Tue, Jun 28, 2016 at 3:56 PM, Pradeep Gollakota
wrote:
> Just out of curiosity, if you guys are in AWS for ever
Just out of curiosity, if you guys are in AWS for everything, why not use
Kinesis?
On Tue, Jun 28, 2016 at 3:49 PM, Charity Majors wrote:
> Hi there,
>
> I just finished implementing kafka + autoscaling groups in a way that made
> sense to me. I have a _lot_ of experience with ASGs and various
