Re: FlinkKafkaConsumer bootstrap.servers vs. broker hosts

2015-10-15 Thread Juho Autio
Hi, Don't worry – this is quite a low priority question. Definitely not a production issue and as a work around it can be fixed rather easily with suitable network setup. Probably quite rare, too, that this kind of network scenario happens with anyone. But I think that it might be possible to imp

Re: FlinkKafkaConsumer bootstrap.servers vs. broker hosts

2015-10-14 Thread Robert Metzger
Hi Juho, sorry for the late reply, I was busy with Flink Forward :) The Flink Kafka Consumer needs both addresses. Kafka uses the bootstrap servers to connect to the brokers to consume messages. The Zookeeper connection is used to commit the offsets of the consumer group once a state snapshot in

FlinkKafkaConsumer bootstrap.servers vs. broker hosts

2015-10-13 Thread Juho Autio
The FlinkKafkaConsumer takes the two arguments --bootstrap.servers (kafka servers) and --zookeeper.connect (zookeeper servers). Then it seems to resolve Kafka brokers from zookeeper, and uses those host names to consume kafka. But it also connects to the given bootstrap servers to fetch some metada