We did not face any problems with kafka application per se but we have faced problems with zookeeper in virtualized environments due to slowness in fsyncs. We were using a shared SAN storage with shared pools with other VMs. So every time, there was some kind of considerable storage activity like DB backup or something, our zookeeper fsyncs used to take tens of seconds causing kafka-zookeeper sessions to timeout.
On Nov 30, 2017 2:22 AM, "Viktor Somogyi" <viktorsomo...@gmail.com> wrote: > Hi folks, > > Recently I bumped into an interesting question: using kafka in virtualized > environments, such as vmware. I'm not really familiar with virtualization > in-depth (how disk virtualization works, what are the OS level supports > etc.), therefore I think this is an interesting discussion from Kafka's > point. As far as I know Kafka is designed for a non-virtualized environment > mainly (although I haven't seen it explicitly anywhere) but thinking of > it's hard reliance on disk optimization I always assumed this. > > Anyone has experiences with virtualized Kafka? Are you aware of any pain > points that people should consider (or performance issues)? > Are there any publications on this topic? > > Regards, > Viktor >
