Hi,
We are in the process of attempting to upgrade from Kafka Streams 1.1.0 to
2.1.0(-cp1) however we get some wildly different behaviour with regards to disk
usage between these two versions.
An update that uses existing state data exhibits the same behaviour as starting
with a completely clean state data directories.
With 1.1.0 the same topologies restoring the same state stores will at most use
275MB.
With 2.1.0 they will quickly take up to 60GB – I extended the volumes to see
just how far it would go, already magnitudes larger than before.
Only remotely interesting information in the logs are the following exceptions
which other than the explicit disk space exception I presume is related to out
of disk space:
org.apache.kafka.streams.errors.ProcessorStateException: task [0_7] Exception
caught while trying to restore state from
foo-v4-KSTREAM-JOINTHIS-0000000012-store-changelog-7
at
org.apache.kafka.streams.processor.internals.ProcessorStateManager.updateStandbyStates(ProcessorStateManager.java:185)
at
org.apache.kafka.streams.processor.internals.StandbyTask.update(StandbyTask.java:188)
at
org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:1114)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:895)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747)
Caused by: java.lang.NullPointerException: null
java.lang.NullPointerException: null
at
org.apache.kafka.streams.state.internals.RocksDBStore.toggleDbForBulkLoading(RocksDBStore.java:229)
at
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.getWriteBatches(RocksDBSegmentedBytesStore.java:230)
at
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.restoreAllInternal(RocksDBSegmentedBytesStore.java:208)
at
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$RocksDBSegmentsBatchingRestoreCallback.restoreAll(RocksDBSegmentedBytesStore.java:262)
at
org.apache.kafka.streams.processor.internals.StateRestoreCallbackAdapter.lambda$adapt$0(StateRestoreCallbackAdapter.java:42)
at
org.apache.kafka.streams.processor.internals.CompositeRestoreListener.restoreBatch(CompositeRestoreListener.java:89)
at
org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:83)
at
org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:310)
at
org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:92)
at
org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:321)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:839)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747)
org.apache.kafka.streams.errors.ProcessorStateException: Error opening store
KSTREAM-JOINTHIS-0000000018-store.1551343860000 at location
/data/foo/0_1/KSTREAM-JOINTHIS-0000000018-store/KSTREAM-JOINTHIS-0000000018-store.1551343860000
at
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:158)
at
org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:45)
at
org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:101)
at
org.apache.kafka.streams.state.internals.Segments.getOrCreateSegmentIfLive(Segments.java:81)
at
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.getWriteBatches(RocksDBSegmentedBytesStore.java:224)
at
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.restoreAllInternal(RocksDBSegmentedBytesStore.java:208)
at
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$RocksDBSegmentsBatchingRestoreCallback.restoreAll(RocksDBSegmentedBytesStore.java:262)
at
org.apache.kafka.streams.processor.internals.StateRestoreCallbackAdapter.lambda$adapt$0(StateRestoreCallbackAdapter.java:42)
at
org.apache.kafka.streams.processor.internals.CompositeRestoreListener.restoreBatch(CompositeRestoreListener.java:89)
at
org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:83)
at
org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:310)
at
org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:92)
at
org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:321)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:839)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747)
Caused by: org.rocksdb.RocksDBException: While appending to file:
/data/foo/0_1/KSTREAM-JOINTHIS-0000000018-store/KSTREAM-JOINTHIS-0000000018-store.1551343860000/MANIFEST-000001:
No space left on device
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:231)
at
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:156)
... 15 common frames omitted
It feels likely that this is some serious amount of preallocation that is
happening that didn’t use to?
I have eliminated
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8202261 being an issue
– we see this behaviour with 8u141, 8u181 and 8u202
Any help or avenues of investigation are welcome.
Thanks
Adrian
Disclaimer
This e-mail has been sent to you by Zopa.
Zopa Limited is authorised and regulated by the Financial Conduct Authority,
and entered on the Financial Services Register under firm registration number
718925. Zopa Bank Limited is authorised by the Prudential Regulation Authority
and regulated by the Financial Conduct Authority and the Prudential Regulation
Authority, and entered on the Financial Services Register (800542).
Registered Office: Zopa Limited (05197592) and Zopa Bank Limited
(10627575) are both incorporated in England & Wales and have their
registered office at: 1st Floor, Cottons Centre, Tooley Street, London, SE1 2QG.
