Manikumar created ZOOKEEPER-3644:
------------------------------------
Summary: Data loss after upgrading 3.4.14 to 3.5.6 with
snapshot.trust.empty=true
Key: ZOOKEEPER-3644
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3644
Project: ZooKeeper
Issue Type: Bug
Components: server
Affects Versions: 3.5.6
Reporter: Manikumar
We have tried to upgrade single node ZK server from 3.4.14 to 3.5.6. There
were no snapshot files, so as suggested in ZOOKEEPER-3056,
I have set snapshot.trust.empty to true. After server startup, when I tried to
list the znodes, I found that znodes are missing.
Steps to reproduce:
# Start a single node ZK 3.4.14 server and create few znodes
# Upgrade the server to 3.5.6 with snapshot.trust.empty=true config
# try to list the znodes using zkShell
Looking into the [source
code|https://github.com/apache/zookeeper/blob/release-3.5.6/zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/FileTxnSnapLog.java#L224],
looks like we are not reading transaction log if there are no snapshot files
and snapshot.trust.empty is set to true.
ZK 3.5.6 logs:
[2019-12-07 12:13:35,007] INFO Created server with tickTime 3000
minSessionTimeout 6000 maxSessionTimeout 60000 datadir
/var/lib/zookeeper/version-2 snapdir /var/lib/zookeeper/version-2
(org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-07 12:13:35,012] INFO Using
org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
(org.apache.zookeeper.server.ServerCnxnFactory)
[2019-12-07 12:13:35,014] INFO Configuring NIO connection handler with 10s
sessionless connection timeout, 1 selector thread(s), 12 worker threads, and 64
kB direct buffers. (org.apache
.zookeeper.server.NIOServerCnxnFactory)
[2019-12-07 12:13:35,017] INFO binding to port
[0.0.0.0/0.0.0.0:2181|http://0.0.0.0/0.0.0.0:2181]
(org.apache.zookeeper.server.NIOServerCnxnFactory)
[2019-12-07 12:13:35,027] INFO zookeeper.snapshotSizeFactor = 0.33
(org.apache.zookeeper.server.ZKDatabase)
[2019-12-07 12:13:35,029] DEBUG Created new input stream
/var/lib/zookeeper/version-2/log.1
(org.apache.zookeeper.server.persistence.FileTxnLog)
[2019-12-07 12:13:35,031] DEBUG Created new input archive
/var/lib/zookeeper/version-2/log.1
(org.apache.zookeeper.server.persistence.FileTxnLog)
[2019-12-07 12:13:35,035] DEBUG EOF exception java.io.EOFException: Failed to
read /var/lib/zookeeper/version-2/log.1
(org.apache.zookeeper.server.persistence.FileTxnLog)
[2019-12-07 12:13:35,035] WARN No snapshot found, but there are log entries.
This should only be allowed during upgrading.
(org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2019-12-07 12:13:35,035] INFO Snapshotting: 0x0 to
/var/lib/zookeeper/version-2/snapshot.0
(org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2019-12-07 12:13:35,036] INFO Snapshotting: 0x0 to
/var/lib/zookeeper/version-2/snapshot.0
(org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2019-12-07 12:13:35,050] INFO Using checkIntervalMs=60000 maxPerMinute=10000
(org.apache.zookeeper.server.ContainerManager)
[2019-12-07 12:15:07,137] DEBUG Accepted socket connection from
/[127.0.0.1:38888|http://127.0.0.1:38888/]
(org.apache.zookeeper.server.NIOServerCnxnFactory)
[2019-12-07 12:15:07,144] DEBUG Session establishment request from client
/[127.0.0.1:38888|http://127.0.0.1:38888/] client's lastZxid is 0x0
(org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-07 12:15:07,145] DEBUG Adding session 0x100006e15fb0000
(org.apache.zookeeper.server.SessionTrackerImpl)
[2019-12-07 12:15:07,148] TRACE SessionTrackerImpl — Adding session
0x100006e15fb0000 30000 (org.apache.zookeeper.server.SessionTrackerImpl)
[2019-12-07 12:15:07,149] DEBUG Client attempting to establish new session:
session = 0x100006e15fb0000, zxid = 0x0, timeout = 30000, address =
/[127.0.0.1:38888|http://127.0.0.1:38888/]
(org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-07 12:15:07,155] TRACE :Psessionid:0x100006e15fb0000
type:createSession cxid:0x0 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a
(org.apache.zookeeper.server.PrepRequestProcessor)
[2019-12-07 12:15:07,155] TRACE SessionTrackerImpl — Existing session
0x100006e15fb0000 30000 (org.apache.zookeeper.server.SessionTrackerImpl)
[2019-12-07 12:15:07,155] INFO Creating new log file: log.1
(org.apache.zookeeper.server.persistence.FileTxnLog)
[2019-12-07 12:15:07,170] DEBUG Processing request::
sessionid:0x100006e15fb0000 type:createSession cxid:0x0 zxid:0x1 txntype:-10
reqpath:n/a (org.apache.zookeeper.server.FinalRequestProcessor)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
