Hi all,
I set the log level to INFO and here are the logs.
I’m not sure why those logs say that the Group coordinator is unavailable
or invalid because the brokers are all up (including the ones with the ip
specified in the log) and just changing the consumer group makes it work.
If anybody had an idea, that would be very helpful. (also committing the
offsets to Zookeeper works, so I'm wondering if there is a bug with
__Consummer_Offset)
Thank you for your help
kafka-console-consumer --bootstrap-server ip1:9092,ip2:9092 --topic
ASSETS --group ASSETS_PROCESSOR_TOPOLOGY
[2018-12-26 15:47:15,243] INFO Registered
kafka:type=kafka.Log4jController MBean
(kafka.utils.Log4jControllerRegistration$)
[2018-12-26 15:47:15,387] INFO ConsumerConfig values:
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [ip1:9092, ip2:9092]
check.crcs = true
client.id =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = ASSETS_PROCESSOR_TOPOLOGY
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
(org.apache.kafka.clients.consumer.ConsumerConfig)
[2018-12-26 15:47:15,579] INFO Kafka version : 2.0.1-cp1
(org.apache.kafka.common.utils.AppInfoParser)
[2018-12-26 15:47:15,579] INFO Kafka commitId : 3d167ab3fdad2e73
(org.apache.kafka.common.utils.AppInfoParser)
[2018-12-26 15:47:16,711] INFO Cluster ID: AkXF_gdZS5eVEEYbvUiyRg
(org.apache.kafka.clients.Metadata)
[2018-12-26 15:47:16,712] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address1:9092 (id: 2147483643 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:16,713] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Revoking previously assigned
partitions [] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2018-12-26 15:47:16,714] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:17,351] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address1:9092
(id: 2147483643 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:18,189] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:18,190] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:19,501] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:19,723] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:19,723] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,079] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,304] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,305] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,661] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,885] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:20,885] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,240] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,465] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,465] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:21,820] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,046] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,046] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,403] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,622] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,623] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:22,976] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:23,198] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Discovered group coordinator
ip_address2:9092 (id: 2147483645 rack: null)
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:23,199] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2018-12-26 15:47:23,551] INFO [Consumer clientId=consumer-1,
groupId=ASSETS_PROCESSOR_TOPOLOGY] Group coordinator ip_address2:9092
(id: 2147483645 rack: null) is unavailable or invalid, will attempt
rediscovery (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
On Thu, Dec 20, 2018 at 1:23 PM Karim Lamouri <ka...@lamouri.me> wrote:
> Hi Ryanne,
>
> Here is the output of the command:
>
> bin/kafka-consumer-groups --bootstrap-server server --describe --group
> group_name
> Error: Executing consumer group command failed due to The consumer group
> command timed out while waiting for group to initialize
>
> The group_name consumer group is not running in the cluster. What is
> noteworthy is that the consumer group group_name cannot be reset by the
> CLI (to either latest or earliest) and is still not working way past the
> retention period of all the topics (from the consumers with latest or
> earliest auto.offset.reset).
>
> Here is the output:
>
> bin/kafka-consumer-groups --bootstrap-server server --group group_name
> --reset-offsets --to-latest --execute
> Error: Executing consumer group command failed due to The consumer group
> command timed out while waiting for group to initialize
>
>
> Karim
>
> On Tue, Dec 18, 2018 at 8:27 PM Ryanne Dolan <ryannedo...@gmail.com>
> wrote:
>
>> Karim, can you inspect the offsets using kafka-consumer-groups.sh and let
>> us know what you see?
>>
>> Also, is it possible that "group_name" is being used by a consumer group
>> that is still running? For example, say you have a topic with one
>> partition
>> and a consumer with "group_name" already. In that case, a second console
>> consumer with "group_name" wouldn't find any partitions to consume.
>>
>> Ryanne
>>
>> On Tue, Dec 18, 2018 at 11:18 AM Karim Lamouri <ka...@lamouri.me> wrote:
>>
>> > Hi,
>> >
>> > One of our brokers went down and when it came back up the consumer of
>> one
>> > topic couldn’t read using the original consumer group.
>> >
>> > This doesn’t output anything:
>> >
>> > bin/kafka-console-consumer --bootstrap-server server --topic ASSETS
>> > --group group_name
>> >
>> > However, if I change the name of the group it does output data:
>> >
>> > bin/kafka-console-consumer --bootstrap-server server --topic ASSETS
>> > --group group_name_renamed
>> >
>> > Can someone help debug the issue? We are running Kafka 1.1.1
>> >
>> > Thank you
>> >
>>
>
