Re: Cross-site Kafka installation

2013-01-22 Thread Pablo Barrera González
Hi Joel Thanks for the hints. Apparently it was a configuration error at operating system level. We are using Debian Linux. Kafka uses setsockopt call with SO_SNDBUF to setup the buffer size (socket.send.buffer). The operating system then set the real buffer size to min(socket.send.buffer, net.co

Re: Cross-site Kafka installation

2013-01-21 Thread Joel Koshy
We do mirroring across data-centers (but in the same continent). You should basically set a high fetch size and socket buffer size in such scenarios. In general, you should set a high value for the socket buffer size on the consumer configuration (socket.buffersize) and the source cluster's broker

Cross-site Kafka installation

2013-01-21 Thread Pablo Barrera González
Hello In my enterprise we are deploying an cross-site installation of Kakfa. One of the Kafka cluster is located in USA and one consumer is in Europe. Does anybody have experience in such an environment? Any comments on the configuration and best practices? Thanks in advance Pablo