[
https://issues.apache.org/jira/browse/KAFKA-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731353#comment-13731353
]
Guozhang Wang edited comment on KAFKA-992 at 8/6/13 10:26 PM:
--------------------------------------------------------------
The zookeeper bug can be reproduced as follows:
1. Checkout a clean 0.8 branch, revert back the KAFKA-992 fix:
2. Build and create a server connecting to a Zookeeper instance (make sure
maxClientCnxns = 0 in ZK config so that one IP address can create as many
connections as wanted)
3. Load the Zookeeper with dummy sessions, each creates and maintains a
thousand ephemeral nodes.
4. Write a script that pause and resume the Zookeeper process continuously, for
example:
---
while true
do
kill -STOP $1
sleep 8
kill -CONT $1
sleep 60
done
---
5. Then when the Zookeeper process resumes, it will mark all sessions as
timeout, but since the ephemeral nodes to delete are too many, the server's
registration node may not be deleted yet when the servers tries to re-register
itself and the server think himself as registered successfully.
6. And later Zookeeper will delete the server's registration node without the
server's awareness.
7. If we re-apply KAFKA-992's patch, and redo the same testing setup. Under
similar conditions the server will wait and retry.
-----
Since we can now re-produce the bug and verify the fix, the same fix will be
applied to Controller and Consumer.
was (Author: guozhang):
The zookeeper bug can be reproduced as follows:
1. Checkout a clean 0.8 branch, revert back the KAFKA-992 fix:
2. Build and create a server connecting to a Zookeeper instance (make sure
maxClientCnxns = 0 in ZK config so that one IP address can create as many
connections as wanted)
3. Load the Zookeeper with dummy sessions, each creates and maintains a
thousand ephemeral nodes.
4. Write a script that pause and resume the Zookeeper process continuously, for
example:
---
while true
do
kill -STOP $1
sleep 8
kill -CONT $1
sleep 60
done
---
5. Then when the Zookeeper process resumes, it will mark all sessions as
timeout, but since the ephemeral nodes to delete are too many, the server's
registration node may not be deleted yet when the servers tries to re-register
itself and the server think himself as registered successfully.
6. And later Zookeeper will delete the server's registration node without the
server's awareness.
7. If we re-apply KAFKA-992's patch, and redo the same testing setup. Under
similar conditions the server will wait and retry.
> Double Check on Broker Registration to Avoid False NodeExist Exception
> ----------------------------------------------------------------------
>
> Key: KAFKA-992
> URL: https://issues.apache.org/jira/browse/KAFKA-992
> Project: Kafka
> Issue Type: Bug
> Reporter: Neha Narkhede
> Assignee: Guozhang Wang
> Attachments: KAFKA-992.v1.patch, KAFKA-992.v2.patch,
> KAFKA-992.v3.patch, KAFKA-992.v4.patch
>
>
> The current behavior of zookeeper for ephemeral nodes is that session
> expiration and ephemeral node deletion is not an atomic operation.
> The side-effect of the above zookeeper behavior in Kafka, for certain corner
> cases, is that ephemeral nodes can be lost even if the session is not
> expired. The sequence of events that can lead to lossy ephemeral nodes is as
> follows -
> 1. The session expires on the client, it assumes the ephemeral nodes are
> deleted, so it establishes a new session with zookeeper and tries to
> re-create the ephemeral nodes.
> 2. However, when it tries to re-create the ephemeral node,zookeeper throws
> back a NodeExists error code. Now this is legitimate during a session
> disconnect event (since zkclient automatically retries the
> operation and raises a NodeExists error). Also by design, Kafka server
> doesn't have multiple zookeeper clients create the same ephemeral node, so
> Kafka server assumes the NodeExists is normal.
> 3. However, after a few seconds zookeeper deletes that ephemeral node. So
> from the client's perspective, even though the client has a new valid
> session, its ephemeral node is gone.
> This behavior is triggered due to very long fsync operations on the zookeeper
> leader. When the leader wakes up from such a long fsync operation, it has
> several sessions to expire. And the time between the session expiration and
> the ephemeral node deletion is magnified. Between these 2 operations, a
> zookeeper client can issue a ephemeral node creation operation, that could've
> appeared to have succeeded, but the leader later deletes the ephemeral node
> leading to permanent ephemeral node loss from the client's perspective.
> Thread from zookeeper mailing list:
> http://zookeeper.markmail.org/search/?q=Zookeeper+3.3.4#query:Zookeeper%203.3.4%20date%3A201307%20+page:1+mid:zma242a2qgp6gxvx+state:results
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
