[
https://issues.apache.org/jira/browse/KAFKA-9568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17051503#comment-17051503
]
Oleg Khoruzhenko commented on KAFKA-9568:
-----------------------------------------
We run Kafka Streams application with _Static Membership_ and uses _Interactive
Queries_ functionality. Say we have two instances A and B running on HOST_A and
HOST_B correspondingly. Then A migrated to another HOST_C. The
APPLICATION_SERVER_CONFIG changes to the IP of HOST_C. Issue appears when you
need to query remote store on the instance A. In that case {{StreamsMetadata}}
doesn't get updated and returns ip address of HOST_A, even though A is already
migrated to HOST_C.
Let me try to explain the case we are dealing with on example from
kafka-streams-example project.
# I added static group membership to WordCountInteractiveQueriesExample.java
see
[https://github.com/o0oxid/kafka-streams-examples/commit/54cb74f40ef3c659588dbedc496bc456123db215]
# Started confluent-platform: {noformat}
48a1ede4f588 confluentinc/cp-enterprise-kafka:5.4.0
"/etc/confluent/dock…" 2 hours ago Up 2 hours
0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp
kafka-streams-examples_kafka_1
bd6cf95c56b7 confluentinc/cp-zookeeper:5.4.0
"/etc/confluent/dock…" 2 hours ago Up 2 hours 2181/tcp,
2888/tcp, 3888/tcp, 0.0.0.0:32181->32181/tcp
kafka-streams-examples_zookeeper_1
{noformat}
# Run first application in static group membership mode - group.instance.id
'client-1' and port '7070'
{noformat}
> java -cp target/kafka-streams-examples-5.4.0-standalone.jar
> io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesExample
> 7070 localhost:9092 client-1
[2020-03-04 10:38:31,071] WARN [main] The configuration 'admin.retries' was
supplied but isn't a known config.
(org.apache.kafka.clients.consumer.ConsumerConfig)
[2020-03-04 10:38:31,072] WARN [main] The configuration
'admin.retry.backoff.ms' was supplied but isn't a known config.
(org.apache.kafka.clients.consumer.ConsumerConfig)
Mar 04, 2020 10:38:31 AM org.glassfish.jersey.internal.inject.Providers
checkProviderRuntime
WARNING: A provider
io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService
registered in SERVER runtime does not implement any provider interfaces
applicable in the SERVER runtime. Due to constraint configuration problems the
provider
io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService
will be ignored.
[2020-03-04 10:38:49,717] WARN
[interactive-queries-example-client-StreamThread-1] [Consumer
instanceId=client-1-1,
clientId=interactive-queries-example-client-StreamThread-1-consumer,
groupId=interactive-queries-example] The following subscribed topics are not
assigned to any members: [interactive-queries-example-word-count-repartition]
(org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
{noformat}
# Run second application in different terminal - group.instance.id 'client-2'
and port '7071'
{noformat}
> java -cp target/kafka-streams-examples-5.4.0-standalone.jar
> io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesExample
> 7071 localhost:9092 client-2
[2020-03-04 10:39:05,188] WARN [main] The configuration 'admin.retries' was
supplied but isn't a known config.
(org.apache.kafka.clients.consumer.ConsumerConfig)
[2020-03-04 10:39:05,189] WARN [main] The configuration
'admin.retry.backoff.ms' was supplied but isn't a known config.
(org.apache.kafka.clients.consumer.ConsumerConfig)
Mar 04, 2020 10:39:05 AM org.glassfish.jersey.internal.inject.Providers
checkProviderRuntime
WARNING: A provider
io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService
registered in SERVER runtime does not implement any provider interfaces
applicable in the SERVER runtime. Due to constraint configuration problems the
provider
io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService
will be ignored.
{noformat}
Querying StreamsMetadata returns two instances 'localhost:7070' and
'localhost:7071':
{noformat}
curl -s localhost:7070/state/instances | jq
[
{
"host": "localhost",
"port": 7070,
"storeNames": [
"windowed-word-count"
]
},
{
"host": "localhost",
"port": 7071,
"storeNames": [
"word-count"
]
}
]
{noformat}
Now let's re-run the second application with same group.instance.id 'client-2',
but different port '7072'
{noformat}
>java -cp target/kafka-streams-examples-5.4.0-standalone.jar
>io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesExample
> 7072 localhost:9092 client-2
[2020-03-04 10:39:05,188] WARN [main] The configuration 'admin.retries' was
supplied but isn't a known config.
(org.apache.kafka.clients.consumer.ConsumerConfig)
[2020-03-04 10:39:05,189] WARN [main] The configuration
'admin.retry.backoff.ms' was supplied but isn't a known config.
(org.apache.kafka.clients.consumer.ConsumerConfig)
Mar 04, 2020 10:39:05 AM org.glassfish.jersey.internal.inject.Providers
checkProviderRuntime
WARNING: A provider
io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService
registered in SERVER runtime does not implement any provider interfaces
applicable in the SERVER runtime. Due to constraint configuration problems the
provider
io.confluent.examples.streams.interactivequeries.WordCountInteractiveQueriesRestService
will be ignored.
{noformat}
Querying instance metadata returns stale data. Note the port is the same
'7071', while application is running on '7072':
{noformat}
> curl -s localhost:7070/state/instances | jq
[
{
"host": "localhost",
"port": 7070,
"storeNames": [
"windowed-word-count"
]
},
{
"host": "localhost",
"port": 7071,
"storeNames": [
"word-count"
]
}
]
{noformat}
So now it's impossible to perform interactive queries against remote machine
because metadata points to the old port '7071'.
Issue is not reproducible with dynamic membership.
> Kstreams APPLICATION_SERVER_CONFIG is not updated with static membership
> ------------------------------------------------------------------------
>
> Key: KAFKA-9568
> URL: https://issues.apache.org/jira/browse/KAFKA-9568
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.4.0
> Reporter: David J. Garcia
> Priority: Major
>
> A kstreams application with static membership, and
> StreamsConfg.APPLICATION_SERVER_CONFIG set, will NOT update old server config
> upon restart of application on new host.
> Steps to reproduce:
>
> # start two kstreams applications (with same consumer group) and enable
> static membership (and set application server config to <ip-addr>:<port>)
> # kill one of the applications and restart it on a new host(with new ip)
> before timeout ends (so that rebalancing doesn't occur).
> # the other kstreams application will now have an invalid
> application_server_config
> Possible fix:
> If an application restarts with a new host/identity..etc, it could trigger a
> "light-rebalance" where the other applications in the consumer group don't
> change partition assignments ,but instead just get their configuration
> updated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
