Tom, Thanks, that's very good to know. What kind of instances EC2 instances are you using for your brokers?
-barry On Fri, Jun 10, 2016 at 4:17 PM, Tom Crayford <tcrayf...@heroku.com> wrote: > Barry, > > No, because Kafka also relies heavily on the OS page cache, which uses > memory. You'd roughly want to allocate enough page cache to hold all the > messages for your consumers for say, 30s. > > Kafka also (in our experience on EC2) tends to run out of network far > before it runs out of memory or disk bandwidth, so colocating brokers makes > that much more likely. > > Thanks > > Tom Crayford, Heroku Kafka > > On Fri, Jun 10, 2016 at 7:02 AM, Barry Kaplan <bkap...@memelet.com> wrote: > > > If too much heap cause problems, would it make sense to run multiple > > brokers on a box with lots memory? For example, an EC2 D2 instance types > > has way way more ram than kafka could ever use - -but it has fast > connected > > disks. > > > > Would running a broker per disk make sense in this case? > > > > -barry > > >
