Giresh, I'm curious what your solution was. Did you use locally attached storage for your ZK ensemble? Did you move it to static machines?
On Thu, Nov 30, 2017 at 4:50 PM, John Yost <hokiege...@gmail.com> wrote: > Great point by Girish--its the delays of syncing with Zookeeper that are > particularly problematic. Moreover, Zookeeper sync delays and session > timeouts impact other systems as well such as Storm. > > --John > > On Thu, Nov 30, 2017 at 10:14 AM, Girish Aher <girisha...@gmail.com> > wrote: > > > We did not face any problems with kafka application per se but we have > > faced problems with zookeeper in virtualized environments due to slowness > > in fsyncs. We were using a shared SAN storage with shared pools with > other > > VMs. So every time, there was some kind of considerable storage activity > > like DB backup or something, our zookeeper fsyncs used to take tens of > > seconds causing kafka-zookeeper sessions to timeout. > > > > On Nov 30, 2017 2:22 AM, "Viktor Somogyi" <viktorsomo...@gmail.com> > wrote: > > > > > Hi folks, > > > > > > Recently I bumped into an interesting question: using kafka in > > virtualized > > > environments, such as vmware. I'm not really familiar with > virtualization > > > in-depth (how disk virtualization works, what are the OS level supports > > > etc.), therefore I think this is an interesting discussion from Kafka's > > > point. As far as I know Kafka is designed for a non-virtualized > > environment > > > mainly (although I haven't seen it explicitly anywhere) but thinking of > > > it's hard reliance on disk optimization I always assumed this. > > > > > > Anyone has experiences with virtualized Kafka? Are you aware of any > pain > > > points that people should consider (or performance issues)? > > > Are there any publications on this topic? > > > > > > Regards, > > > Viktor > > > > > > -- Senior Software Engineer, Lightbend, Inc. <http://lightbend.com> @seg1o <https://twitter.com/seg1o>
