Hi,

We are in the process of attempting to upgrade from Kafka Streams 1.1.0 to 
2.1.0(-cp1) however we get some wildly different behaviour with regards to disk 
usage between these two versions.

An update that uses existing state data exhibits the same behaviour as starting 
with a completely clean state data directories.

With 1.1.0 the same topologies restoring the same state stores will at most use 
275MB.
With 2.1.0 they will quickly take up to 60GB – I extended the volumes to see 
just how far it would go, already magnitudes larger than before.

Only remotely interesting information in the logs are the following exceptions 
which other than the explicit disk space exception I presume is related to out 
of disk space:

org.apache.kafka.streams.errors.ProcessorStateException: task [0_7] Exception 
caught while trying to restore state from 
foo-v4-KSTREAM-JOINTHIS-0000000012-store-changelog-7
                at 
org.apache.kafka.streams.processor.internals.ProcessorStateManager.updateStandbyStates(ProcessorStateManager.java:185)
                at 
org.apache.kafka.streams.processor.internals.StandbyTask.update(StandbyTask.java:188)
                at 
org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:1114)
                at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:895)
                at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777)
                at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747)
Caused by: java.lang.NullPointerException: null
java.lang.NullPointerException: null
                at 
org.apache.kafka.streams.state.internals.RocksDBStore.toggleDbForBulkLoading(RocksDBStore.java:229)
                at 
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.getWriteBatches(RocksDBSegmentedBytesStore.java:230)
                at 
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.restoreAllInternal(RocksDBSegmentedBytesStore.java:208)
                at 
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$RocksDBSegmentsBatchingRestoreCallback.restoreAll(RocksDBSegmentedBytesStore.java:262)
                at 
org.apache.kafka.streams.processor.internals.StateRestoreCallbackAdapter.lambda$adapt$0(StateRestoreCallbackAdapter.java:42)
                at 
org.apache.kafka.streams.processor.internals.CompositeRestoreListener.restoreBatch(CompositeRestoreListener.java:89)
                at 
org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:83)
                at 
org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:310)
                at 
org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:92)
                at 
org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:321)
                at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:839)
                at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777)
                at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747)


org.apache.kafka.streams.errors.ProcessorStateException: Error opening store 
KSTREAM-JOINTHIS-0000000018-store.1551343860000 at location 
/data/foo/0_1/KSTREAM-JOINTHIS-0000000018-store/KSTREAM-JOINTHIS-0000000018-store.1551343860000
                at 
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:158)
                at 
org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:45)
                at 
org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:101)
                at 
org.apache.kafka.streams.state.internals.Segments.getOrCreateSegmentIfLive(Segments.java:81)
                at 
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.getWriteBatches(RocksDBSegmentedBytesStore.java:224)
                at 
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.restoreAllInternal(RocksDBSegmentedBytesStore.java:208)
                at 
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$RocksDBSegmentsBatchingRestoreCallback.restoreAll(RocksDBSegmentedBytesStore.java:262)
                at 
org.apache.kafka.streams.processor.internals.StateRestoreCallbackAdapter.lambda$adapt$0(StateRestoreCallbackAdapter.java:42)
                at 
org.apache.kafka.streams.processor.internals.CompositeRestoreListener.restoreBatch(CompositeRestoreListener.java:89)
                at 
org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:83)
                at 
org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:310)
                at 
org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:92)
                at 
org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:321)
                at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:839)
                at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777)
                at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747)
Caused by: org.rocksdb.RocksDBException: While appending to file: 
/data/foo/0_1/KSTREAM-JOINTHIS-0000000018-store/KSTREAM-JOINTHIS-0000000018-store.1551343860000/MANIFEST-000001:
 No space left on device
                at org.rocksdb.RocksDB.open(Native Method)
                at org.rocksdb.RocksDB.open(RocksDB.java:231)
                at 
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:156)
                ... 15 common frames omitted

It feels likely that this is some serious amount of preallocation that is 
happening that didn’t use to?
I have eliminated 
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8202261 being an issue 
– we see this behaviour with 8u141, 8u181 and 8u202

Any help or avenues of investigation are welcome.

Thanks
Adrian

Disclaimer

This e-mail has been sent to you by Zopa.
 
Zopa Limited is authorised and regulated by the Financial Conduct Authority,
and entered on the Financial Services Register under firm registration number 
718925. Zopa Bank Limited is authorised by the Prudential Regulation Authority
and regulated by the Financial Conduct Authority and the Prudential Regulation
Authority, and entered on the Financial Services Register (800542).
 
Registered Office: Zopa Limited (05197592) and Zopa Bank Limited
(10627575) are both incorporated in England & Wales and have their
registered office at: 1st Floor, Cottons Centre, Tooley Street, London, SE1 2QG.

Reply via email to