Hi Daniel, Yes I have specified the zookeeper host in server.properties file , so the broker is connected to zookeeper.
https://kafka.apache.org/documentation#brokerconfigs -> according to this link, I guess all these configs are done in server.prop , so from where did you get kafka09 as brokerPath ? this is my entry in server.prop file -> zookeeper.connect=localhost:2181 Have you set this as zkhost:2181/kafka09 ? Regards, Vinay Patil On Tue, Nov 8, 2016 at 4:27 PM, Daniel Santos [via Apache Flink User Mailing List archive.] <ml-node+s2336050n997...@n4.nabble.com> wrote: > Hi, > > Your kafka broker is connected to zookeeper I believe. > > I am using kafka 0.9.0.1 my self too. > > On kafka broker 0.9.0.1 I have configured the zookeeper connect to a path, > for instances : > > zk1:2181,zk2:2181,zk3:2181/kafka09 > > https://kafka.apache.org/documentation#brokerconfigs > > Now on the flink side I would configure > "props.setProperty("zookeeper.connect", > zkHosts)" the same resulting in : > > > props.setProperty("zookeeperconnect", "zk1:2181,zk2:2181,zk3:2181/ > kafka09") > > > That is what I mean by broker's path. > > Best Regards, > > Daniel Santos > > On 11/08/2016 10:49 AM, vinay patil wrote: > > Hi Daniel, > > I have the same properties set for the consumer and the same code > > *brokerspath only needed if you have set it on kafka config* -> I did not > get this, do you mean to check the brokerspath in conf/server.properties > file ? I have even tried by setting offset.storage property to zookeeeper, > but still not getting the consumers listed > > I am using Kafka 0.9.0.1 > > > Regards, > Vinay Patil > > On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink User > Mailing List archive.] <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=9978&i=0>> wrote: > >> Hello, >> >> This is my config. >> >> On kafka props : >> >> val props = new Properties() >> >> props.setProperty("zookeeper.connect", zkHosts) >> props.setProperty("bootstrap.servers", kafHosts) >> props.setProperty("group.id", "prod") >> props.setProperty("auto.offset.reset", "earliest") >> >> Now for zkHosts beware that all your hosts quorum has to be included. >> >> For instances you have zk1 and zk2 and zk3 to form a quorum. >> >> Then it will result in zkHosts being -> >> zk1:2181,zk2:2181,zk3:2181/[brokerspath] >> . >> >> brokerspath only needed if you have set it on kafka config. Ignore it >> otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" . >> >> After that -> val source = env.addSource(new >> FlinkKafkaConsumer09[String](KAFKA_TOPIC, new SimpleStringSchema(), >> props)) >> >> Then on kafkamanager -> consumers I have the groupID prod. >> >> Hope it helps. >> >> Best Regards, >> >> Daniel Santos >> On 11/08/2016 08:45 AM, vinay patil wrote: >> >> Hi Limbo, >> >> I am using 0.9, I am not able to see updated results even after >> refreshing. >> There is some property that we have to set in order to make this work >> >> Regards, >> Vinay Patil >> >> On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing >> List archive.] <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=9969&i=0>> wrote: >> >>> I am using kafka 0.8, just refresh the page and you will see the updated >>> results, it’s not auto update. >>> >>> There is our logstash consumer: >>> >>> >>> >>> and this is our flink consumer: >>> >>> >>> I find the flink consumer just write the offset of the kafka partition >>> to zookeeper without owner and ids, >>> so we can’t find the consumer in the manager page, we can only find the >>> offset info. >>> >>> 在 2016年11月8日，下午12:02，vinay patil <[hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>> 写道： >>> >>> Hi Limbo, >>> >>> I can see the lag by using that URL, but the Lag there is not showing >>> updated results, it does not change, also if you try to to change the >>> consumer group value it will still show you the same value instead of >>> saying consumer group does not exist or similar kind of error :) >>> >>> According to documentation of 0.9.x the offsets are stored in Kafka, but >>> we can set offset.storage property to zookeeper. >>> Even by setting this I am not getting the consumer listed. >>> >>> Kafka cli command also does not show this consumer >>> >>> Regards, >>> Vinay Patil >>> >>> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User Mailing >>> List archive.] <<a href="x-msg://4/user/SendEmail >>> .jtp?type=node&node=9964&i=0" target="_top" rel="nofollow" >>> link="external" class="">[hidden email]> wrote: >>> >>>> Hi, >>>> >>>> I have the same problem, I think the reason is that the consumer of >>>> flink use the low level API, >>>> and when I type the group name in manager url I can get the lag of the >>>> flink consumer, like this: >>>> >>>> http://your_manager_url/clusters/<kafka_name>/consumers/<con >>>> sumer_name>/topic/<topic_name>/type/ZK >>>> >>>> 在 2016年11月8日，上午5:12，Daniel Santos <[hidden email] >>>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>> 写道： >>>> >>>> Hello, >>>> >>>> I have been using that setup. >>>> From my understanding, if one desires to see the offset being consumed >>>> by Flink on KafkaManger, one has to set it up with zookeeper. On 0.9 it >>>> will only serve as a view of progress. >>>> >>>> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing >>>> purposes only. >>>> >>>> Best Regards, >>>> Daniel Santos >>>> >>>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden email] >>>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> I am monitoring Kafka using KafkaManager for checking offset lag and >>>>> other Kafka metrics, however I am not able to see the consumers when I >>>>> use >>>>> FlinkKafkaConsumer , for console-consumer it shows them in the Consumers >>>>> list. >>>>> >>>>> I have set the required parameters for the kafka consumer while >>>>> running the application. >>>>> >>>>> Has anyone faced this issue ? >>>>> I am using Kafka 0.9.0.1 >>>>> >>>>> Regards, >>>>> Vinay Patil >>>>> >>>> >>>> >>>> >>>> ------------------------------ >>>> If you reply to this email, your message will be added to the >>>> discussion below: >>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab >>>> ble.com/Kafka-Monitoring-tp9957p9962.html >>>> To start a new topic under Apache Flink User Mailing List archive., >>>> email <a >>>> href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1" >>>> target="_top" rel="nofollow" link="external" class="">[hidden email] >>>> To unsubscribe from Apache Flink User Mailing List archive., click here >>>> . >>>> NAML >>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>> >>> >>> >>> ------------------------------ >>> View this message in context: Re: Kafka Monitoring >>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html> >>> Sent from the Apache Flink User Mailing List archive. mailing list >>> archive >>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> >>> at Nabble.com <http://nabble.com/>. >>> >>> >>> >>> >>> ------------------------------ >>> If you reply to this email, your message will be added to the discussion >>> below: >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab >>> ble.com/Kafka-Monitoring-tp9957p9965.html >>> To start a new topic under Apache Flink User Mailing List archive., >>> email [hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=9969&i=1> >>> To unsubscribe from Apache Flink User Mailing List archive., click here. >>> NAML >>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>> >> >> >> ------------------------------ >> View this message in context: Re: Kafka Monitoring >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html> >> Sent from the Apache Flink User Mailing List archive. mailing list >> archive >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> >> at Nabble.com. >> >> >> >> >> ------------------------------ >> If you reply to this email, your message will be added to the discussion >> below: >> http://apache-flink-user-mailing-list-archive.2336050.n4. >> nabble.com/Kafka-Monitoring-tp9957p9976.html >> To start a new topic under Apache Flink User Mailing List archive., email >> [hidden >> email] <http:///user/SendEmail.jtp?type=node&node=9978&i=1> >> To unsubscribe from Apache Flink User Mailing List archive., click here. >> NAML >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > > ------------------------------ > View this message in context: Re: Kafka Monitoring > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html> > Sent from the Apache Flink User Mailing List archive. mailing list archive > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at > Nabble.com. > > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka- > Monitoring-tp9957p9979.html > To start a new topic under Apache Flink User Mailing List archive., email > ml-node+s2336050n1...@n4.nabble.com > To unsubscribe from Apache Flink User Mailing List archive., click here > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx> > . > NAML > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9980.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
