Hi Juho, sorry for the late reply, I was busy with Flink Forward :)
The Flink Kafka Consumer needs both addresses. Kafka uses the bootstrap servers to connect to the brokers to consume messages. The Zookeeper connection is used to commit the offsets of the consumer group once a state snapshot in Flink has been completed. Our consumers are pretty minimalistic because we are waiting for the Kafka project to finish their new consumer API [2]. It seems that its a matter of weeks for the Kafka project to release the new consumer [1]. With the new consumer API from kafka, we can actually give Flink users a lot more features, for example subscribing to multiple topics with one source instance, committing offsets to brokers, etc. I think the new consumer will not need both bootstrap servers and zookeeper. Is the problem you're reporting preventing you from using the KafkaConsumer in production? If so, can you tell me when exactly the data consumption is failing? (maybe with some logs / stacktrace?) If this contains confidential information, you can also send me a private mail. Regards, Robert [1] https://twitter.com/gwenshap/status/653718350648897536 [2] https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design On Tue, Oct 13, 2015 at 3:29 PM, Juho Autio <juho.au...@rovio.com> wrote: > The FlinkKafkaConsumer takes the two arguments --bootstrap.servers (kafka > servers) and --zookeeper.connect (zookeeper servers). Then it seems to > resolve Kafka brokers from zookeeper, and uses those host names to consume > kafka. But it also connects to the given bootstrap servers to fetch some > metadata, it seems. > > This is problematic when zookeeper uses internal kafka hostnames – > consuming > won't work when those resolved kafka hosts can't be reached. > > Could the consumer be changed to > - respect the provided kafka hosts / ips and not use host names resolved > from zookeeper > and optionally > - not require bootstrap.servers argument at all, just resolve broker host > names from zookeeper and use those as "bootstrap servers", too? > > Is the concept of bootstrap server something entirely else, or what am I > missing here? > > Thanks! > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/FlinkKafkaConsumer-bootstrap-servers-vs-broker-hosts-tp3109.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >
