Great point by Girish--its the delays of syncing with Zookeeper that are particularly problematic. Moreover, Zookeeper sync delays and session timeouts impact other systems as well such as Storm.
--John On Thu, Nov 30, 2017 at 10:14 AM, Girish Aher <girisha...@gmail.com> wrote: > We did not face any problems with kafka application per se but we have > faced problems with zookeeper in virtualized environments due to slowness > in fsyncs. We were using a shared SAN storage with shared pools with other > VMs. So every time, there was some kind of considerable storage activity > like DB backup or something, our zookeeper fsyncs used to take tens of > seconds causing kafka-zookeeper sessions to timeout. > > On Nov 30, 2017 2:22 AM, "Viktor Somogyi" <viktorsomo...@gmail.com> wrote: > > > Hi folks, > > > > Recently I bumped into an interesting question: using kafka in > virtualized > > environments, such as vmware. I'm not really familiar with virtualization > > in-depth (how disk virtualization works, what are the OS level supports > > etc.), therefore I think this is an interesting discussion from Kafka's > > point. As far as I know Kafka is designed for a non-virtualized > environment > > mainly (although I haven't seen it explicitly anywhere) but thinking of > > it's hard reliance on disk optimization I always assumed this. > > > > Anyone has experiences with virtualized Kafka? Are you aware of any pain > > points that people should consider (or performance issues)? > > Are there any publications on this topic? > > > > Regards, > > Viktor > > >
