Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

Should not matter. We're running 12.04. Wes On Jun 16, 2015 12:18 PM, "Henry Cai" wrote: > Does it still matter whether we are using Ubuntu 14 or 12? > > On Tue, Jun 16, 2015 at 8:44 AM, Wesley Chow wrote: > > > > > A call with Amazon confirmed instability for d2 and c4 instances > triggered >

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

Does it still matter whether we are using Ubuntu 14 or 12? On Tue, Jun 16, 2015 at 8:44 AM, Wesley Chow wrote: > > A call with Amazon confirmed instability for d2 and c4 instances triggered > by lots of network activity. They fixed the problem and have since rolled > it out. We've been running K

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

A call with Amazon confirmed instability for d2 and c4 instances triggered by lots of network activity. They fixed the problem and have since rolled it out. We've been running Kafka with d2's for a little while now and so far so good. Wes On Tue, Jun 2, 2015 at 1:39 PM, Wes Chow wrote: > > We

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

1369725251. It's a fix in the underline virtualization platform. Here is the quote from the ticket. "The issue is a bug in a performance improvement (10% improved PPS when using Xen PV "netback/netfront" networking) in the latest build of the virtualization platform, which has only been released t

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

On Wed, Jun 3, 2015 at 12:01 PM, Dan wrote: > A general point we've found with higher network load on some of the newer > ec2 instances. > > The MTU defaults to be larger than usual for jumbo packet support on the > faster network interfaces, but with a bug in the kernel / xen this causes > kerne

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

A general point we've found with higher network load on some of the newer ec2 instances. The MTU defaults to be larger than usual for jumbo packet support on the faster network interfaces, but with a bug in the kernel / xen this causes kernel panic's. So we've set the MTU to be 1500 which fixed th

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

Henry: We run Kafka on the old and trusty m1.xlarge. We avoid EBS completely, it's network storage that pretends to be local and when the network, which is AWS' weak spot, acts up EBS is a big liability. It's also slow and expensive. Others: Thanks for sharing your experience with the d2's. We hav

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

Steven, Do you have the AWS case # (or the Ubuntu bug/case #) when you hit that kernel panic issue? Our company will still be running on AMI image 12.04 for a while, I will see whether the fix was also ported onto Ubuntu 12.04 On Tue, Jun 2, 2015 at 2:53 PM, Steven Wu wrote: > now I remember w

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

now I remember we had same kernel panic issue in the first week of D2 rolling-out. then AWS fixed it and we haven't seen any issue since. try Ubuntu 14.04 and see if it resolves your remaining kernel/instability issue. On Tue, Jun 2, 2015 at 2:30 PM, Wes Chow wrote: > > Daniel Nelson > June

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

Daniel Nelson June 2, 2015 at 4:39 PM On Jun 2, 2015, at 1:22 PM, Steven Wu wrote: can you elaborate what kind of instability you have encountered? We have seen the nodes become completely non-responsive. Usually they get rebooted automatically after 10-20 m

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

On Jun 2, 2015, at 1:22 PM, Steven Wu wrote: > > can you elaborate what kind of instability you have encountered? We have seen the nodes become completely non-responsive. Usually they get rebooted automatically after 10-20 minutes, but occasionally they get stuck for days in a state where they

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

Wes/Daniel, can you elaborate what kind of instability you have encountered? we are on Ubuntu 14.04.2 and haven't encountered any issues so far. in the announcement, they did mention using Ubuntu 14.04 for better disk throughput. not sure whether 14.04 also addresses any instability issue you enc

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

Our workaround is to switch to i2's. Amazon didn't mention anything, though we're getting on a call with them soon so I'll be sure to ask. Fwiw, we're also on 12.04. Wes Daniel Nelson June 2, 2015 at 2:42 PM Do you have any workarounds for the d2 issues?

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

> On Jun 2, 2015, at 10:39 AM, Wes Chow wrote: > > > We have run d2 instances with Kafka. They're currently unstable -- Amazon > confirmed a host issue with d2 instances that gets tickled by a Kafka > workload yesterday. Otherwise, it seems the d2 instance type is ideal as it > gets an enormo

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

We have run d2 instances with Kafka. They're currently unstable -- Amazon confirmed a host issue with d2 instances that gets tickled by a Kafka workload yesterday. Otherwise, it seems the d2 instance type is ideal as it gets an enormous amount of disk throughput and you'll likely be network b

Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

EBS (network attached storage) has got a lot better over the last a few years. we don't quite trust it for kafka workload. At Netflix, we were going with the new d2 instance type (HDD). our perf/load testing shows it satisfy our workload. SSD is better in latency curve but pretty comparable in ter

HDD or SSD or EBS for kafka brokers in Amazon EC2

We have been hosting kafka brokers in Amazon EC2 and we are using EBS disk. But periodically we were hit by long I/O wait time on EBS in some Availability Zones. We are thinking to change the instance types to a local HDD or local SSD. HDD is cheaper and bigger and seems quite fit for the Kafka u