Hi Daniel,
I have the same properties set for the consumer and the same code
*brokerspath only needed if you have set it on kafka config* -> I did not
get this, do you mean to check the brokerspath in conf/server.properties
file ? I have even tried by setting offset.storage property to zookeeeper,
but still not getting the consumers listed
I am using Kafka 0.9.0.1
Regards,
Vinay Patil
On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink User
Mailing List archive.] <ml-node+s2336050n997...@n4.nabble.com> wrote:
> Hello,
>
> This is my config.
>
> On kafka props :
>
> val props = new Properties()
>
> props.setProperty("zookeeper.connect", zkHosts)
> props.setProperty("bootstrap.servers", kafHosts)
> props.setProperty("group.id", "prod")
> props.setProperty("auto.offset.reset", "earliest")
>
> Now for zkHosts beware that all your hosts quorum has to be included.
>
> For instances you have zk1 and zk2 and zk3 to form a quorum.
>
> Then it will result in zkHosts being ->
> zk1:2181,zk2:2181,zk3:2181/[brokerspath]
> .
>
> brokerspath only needed if you have set it on kafka config. Ignore it
> otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" .
>
> After that -> val source = env.addSource(new
> FlinkKafkaConsumer09[String](KAFKA_TOPIC,
> new SimpleStringSchema(), props))
>
> Then on kafkamanager -> consumers I have the groupID prod.
>
> Hope it helps.
>
> Best Regards,
>
> Daniel Santos
> On 11/08/2016 08:45 AM, vinay patil wrote:
>
> Hi Limbo,
>
> I am using 0.9, I am not able to see updated results even after refreshing.
> There is some property that we have to set in order to make this work
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing List
> archive.] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9969&i=0>> wrote:
>
>> I am using kafka 0.8, just refresh the page and you will see the updated
>> results, it’s not auto update.
>>
>> There is our logstash consumer:
>>
>>
>>
>> and this is our flink consumer:
>>
>>
>> I find the flink consumer just write the offset of the kafka partition to
>> zookeeper without owner and ids,
>> so we can’t find the consumer in the manager page, we can only find the
>> offset info.
>>
>> 在 2016年11月8日，下午12:02，vinay patil <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>> 写道：
>>
>> Hi Limbo,
>>
>> I can see the lag by using that URL, but the Lag there is not showing
>> updated results, it does not change, also if you try to to change the
>> consumer group value it will still show you the same value instead of
>> saying consumer group does not exist or similar kind of error :)
>>
>> According to documentation of 0.9.x the offsets are stored in Kafka, but
>> we can set offset.storage property to zookeeper.
>> Even by setting this I am not getting the consumer listed.
>>
>> Kafka cli command also does not show this consumer
>>
>> Regards,
>> Vinay Patil
>>
>> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User Mailing List
>> archive.] <<a
>> href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=0"
>> target="_top" rel="nofollow" link="external" class="">[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> I have the same problem, I think the reason is that the consumer of
>>> flink use the low level API,
>>> and when I type the group name in manager url I can get the lag of the
>>> flink consumer, like this:
>>>
>>> http://your_manager_url/clusters/<kafka_name>/consumers/<con
>>> sumer_name>/topic/<topic_name>/type/ZK
>>>
>>> 在 2016年11月8日，上午5:12，Daniel Santos <[hidden email]
>>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>> 写道：
>>>
>>> Hello,
>>>
>>> I have been using that setup.
>>> From my understanding, if one desires to see the offset being consumed
>>> by Flink on KafkaManger, one has to set it up with zookeeper. On 0.9 it
>>> will only serve as a view of progress.
>>>
>>> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing
>>> purposes only.
>>>
>>> Best Regards,
>>> Daniel Santos
>>>
>>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden email]
>>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am monitoring Kafka using KafkaManager for checking offset lag and
>>>> other Kafka metrics, however I am not able to see the consumers when I use
>>>> FlinkKafkaConsumer , for console-consumer it shows them in the Consumers
>>>> list.
>>>>
>>>> I have set the required parameters for the kafka consumer while running
>>>> the application.
>>>>
>>>> Has anyone faced this issue ?
>>>> I am using Kafka 0.9.0.1
>>>>
>>>> Regards,
>>>> Vinay Patil
>>>>
>>>
>>>
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>> ble.com/Kafka-Monitoring-tp9957p9962.html
>>> To start a new topic under Apache Flink User Mailing List archive., email
>>> <a href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1"
>>> target="_top" rel="nofollow" link="external" class="">[hidden email]
>>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>>> NAML
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>
>>
>> ------------------------------
>> View this message in context: Re: Kafka Monitoring
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
>> Nabble.com <http://nabble.com/>.
>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Kafka-Monitoring-tp9957p9965.html
>> To start a new topic under Apache Flink User Mailing List archive., email
>> [hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=9969&i=1>
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Kafka Monitoring
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-
> Monitoring-tp9957p9976.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1...@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
