Hi Daniel,
Yes I have specified the zookeeper host in server.properties file
, so the broker is connected to zookeeper.
https://kafka.apache.org/documentation#brokerconfigs
<https://kafka.apache.org/documentation#brokerconfigs> ->
according to this link, I guess all these configs are done in
server.prop , so from where did you get kafka09 as brokerPath ?
this is my entry in server.prop file ->
zookeeper.connect=localhost:2181
Have you set this as zkhost:2181/kafka09 ?
Regards,
Vinay Patil
On Tue, Nov 8, 2016 at 4:27 PM, Daniel Santos [via Apache Flink
User Mailing List archive.] <[hidden email]
<http:///user/SendEmail.jtp?type=node&node=9980&i=0>> wrote:
Hi,
Your kafka broker is connected to zookeeper I believe.
I am using kafka 0.9.0.1 my self too.
On kafka broker 0.9.0.1 I have configured the zookeeper
connect to a path, for instances :
zk1:2181,zk2:2181,zk3:2181/kafka09
https://kafka.apache.org/documentation#brokerconfigs
<https://kafka.apache.org/documentation#brokerconfigs>
Now on the flink side I would configure
"props.setProperty("zookeeper.connect", zkHosts)" the same
resulting in :
props.setProperty("zookeeperconnect",
"zk1:2181,zk2:2181,zk3:2181/kafka09")
That is what I mean by broker's path.
Best Regards,
Daniel Santos
On 11/08/2016 10:49 AM, vinay patil wrote:
Hi Daniel,
I have the same properties set for the consumer and the same
code
/brokerspath only needed if you have set it on kafka config/
-> I did not get this, do you mean to check the brokerspath
in conf/server.properties file ? I have even tried by
setting offset.storage property to zookeeeper, but still not
getting the consumers listed
I am using Kafka 0.9.0.1
Regards,
Vinay Patil
On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache
Flink User Mailing List archive.] <[hidden email]
<http:///user/SendEmail.jtp?type=node&node=9978&i=0>> wrote:
Hello,
This is my config.
On kafka props :
val props = new Properties()
props.setProperty("zookeeper.connect", zkHosts)
props.setProperty("bootstrap.servers", kafHosts)
props.setProperty("group.id <http://group.id>", "prod")
props.setProperty("auto.offset.reset", "earliest")
Now for zkHosts beware that all your hosts quorum has to
be included.
For instances you have zk1 and zk2 and zk3 to form a quorum.
Then it will result in zkHosts being ->
zk1:2181,zk2:2181,zk3:2181/[brokerspath] .
brokerspath only needed if you have set it on kafka
config. Ignore it otherwise, resulting in
"zk1:2181,zk2:2181,zk3:2181" .
After that -> val source = env.addSource(new
FlinkKafkaConsumer09[String](KAFKA_TOPIC, new
SimpleStringSchema(), props))
Then on kafkamanager -> consumers I have the groupID prod.
Hope it helps.
Best Regards,
Daniel Santos
On 11/08/2016 08:45 AM, vinay patil wrote:
Hi Limbo,
I am using 0.9, I am not able to see updated results
even after refreshing.
There is some property that we have to set in order to
make this work
Regards,
Vinay Patil
On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache
Flink User Mailing List archive.] <[hidden email]
<http:///user/SendEmail.jtp?type=node&node=9969&i=0>>
wrote:
I am using kafka 0.8, just refresh the page and you
will see the updated results, it’s not auto update.
There is our logstash consumer:
and this is our flink consumer:
I find the flink consumer just write the offset of
the kafka partition to zookeeper without owner and
ids,
so we can’t find the consumer in the manager page,
we can only find the offset info.
在 2016年11月8日，下午12:02，vinay patil <[hidden
email]
<http:///user/SendEmail.jtp?type=node&node=9965&i=0>>
写道：
Hi Limbo,
I can see the lag by using that URL, but the Lag
there is not showing updated results, it does not
change, also if you try to to change the consumer
group value it will still show you the same value
instead of saying consumer group does not exist or
similar kind of error :)
According to documentation of 0.9.x the offsets
are stored in Kafka, but we can set offset.storage
property to zookeeper.
Even by setting this I am not getting the consumer
listed.
Kafka cli command also does not show this consumer
Regards,
Vinay Patil
On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache
Flink User Mailing List archive.]<<a
href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=0"
target="_top" rel="nofollow" link="external"
class="">[hidden email]>wrote:
Hi,
I have the same problem, I think the reason is
that the consumer of flink use the low level API,
and when I type the group name in manager url
I can get the lag of the flink consumer, like
this:
http://your_manager_url/clusters/<kafka_name>/consumers/<consumer_name>/topic/<topic_name>/type/ZK
<http://your_manager_url/clusters/%3Ckafka_name%3E/consumers/%3Cconsumer_name%3E/topic/%3Ctopic_name%3E/type/ZK>
在 2016年11月8日，上午5:12，Daniel Santos
<[hidden email]
<http://user/SendEmail.jtp?type=node&node=9962&i=0>>
写道：
Hello,
I have been using that setup.
From my understanding, if one desires to see
the offset being consumed by Flink on
KafkaManger, one has to set it up with
zookeeper. On 0.9 it will only serve as a
view of progress.
Basically what's mandatory on 0.8 is optional
on 0.9, and for viewing purposes only.
Best Regards,
Daniel Santos
On November 7, 2016 7:13:54 PM GMT+00:00,
Vinay Patil <[hidden email]
<http://user/SendEmail.jtp?type=node&node=9962&i=1>>
wrote:
Hi,
I am monitoring Kafka using KafkaManager
for checking offset lag and other Kafka
metrics, however I am not able to see the
consumers when I use FlinkKafkaConsumer
, for console-consumer it shows them in
the Consumers list.
I have set the required parameters for
the kafka consumer while running the
application.
Has anyone faced this issue ?
I am using Kafka 0.9.0.1
Regards,
Vinay Patil
------------------------------------------------------------------------
If you reply to this email, your message will
be added to the discussion below:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html>
To start a new topic under Apache Flink User
Mailing List archive., email<a
href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1"
target="_top" rel="nofollow" link="external"
class="">[hidden email]
To unsubscribe from Apache Flink User Mailing
List archive.,click here.
NAML
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
------------------------------------------------------------------------
View this message in context:Re: Kafka Monitoring
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
Sent from theApache Flink User Mailing List
archive. mailing list archive
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>atNabble.com
<http://nabble.com/>.
------------------------------------------------------------------------
If you reply to this email, your message will be
added to the discussion below:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html>
To start a new topic under Apache Flink User
Mailing List archive., email [hidden email]
<http:///user/SendEmail.jtp?type=node&node=9969&i=1>
To unsubscribe from Apache Flink User Mailing List
archive., click here.
NAML
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
------------------------------------------------------------------------
View this message in context: Re: Kafka Monitoring
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
Sent from the Apache Flink User Mailing List archive.
mailing list archive
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
at Nabble.com.
------------------------------------------------------------------------
If you reply to this email, your message will be added
to the discussion below:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9976.html
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9976.html>
To start a new topic under Apache Flink User Mailing
List archive., email [hidden email]
<http:///user/SendEmail.jtp?type=node&node=9978&i=1>
To unsubscribe from Apache Flink User Mailing List
archive., click here.
NAML
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
------------------------------------------------------------------------
View this message in context: Re: Kafka Monitoring
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html>
Sent from the Apache Flink User Mailing List archive.
mailing list archive
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
at Nabble.com.
------------------------------------------------------------------------
If you reply to this email, your message will be added to the
discussion below:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9979.html
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9979.html>
To start a new topic under Apache Flink User Mailing List
archive., email [hidden email]
<http:///user/SendEmail.jtp?type=node&node=9980&i=1>
To unsubscribe from Apache Flink User Mailing List archive.,
click here.
NAML
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
------------------------------------------------------------------------
View this message in context: Re: Kafka Monitoring
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9980.html>
Sent from the Apache Flink User Mailing List archive. mailing
list archive
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
at Nabble.com.
