Hi, We are in the process of attempting to upgrade from Kafka Streams 1.1.0 to 2.1.0(-cp1) however we get some wildly different behaviour with regards to disk usage between these two versions.
An update that uses existing state data exhibits the same behaviour as starting with a completely clean state data directories. With 1.1.0 the same topologies restoring the same state stores will at most use 275MB. With 2.1.0 they will quickly take up to 60GB – I extended the volumes to see just how far it would go, already magnitudes larger than before. Only remotely interesting information in the logs are the following exceptions which other than the explicit disk space exception I presume is related to out of disk space: org.apache.kafka.streams.errors.ProcessorStateException: task [0_7] Exception caught while trying to restore state from foo-v4-KSTREAM-JOINTHIS-0000000012-store-changelog-7 at org.apache.kafka.streams.processor.internals.ProcessorStateManager.updateStandbyStates(ProcessorStateManager.java:185) at org.apache.kafka.streams.processor.internals.StandbyTask.update(StandbyTask.java:188) at org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:1114) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:895) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747) Caused by: java.lang.NullPointerException: null java.lang.NullPointerException: null at org.apache.kafka.streams.state.internals.RocksDBStore.toggleDbForBulkLoading(RocksDBStore.java:229) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.getWriteBatches(RocksDBSegmentedBytesStore.java:230) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.restoreAllInternal(RocksDBSegmentedBytesStore.java:208) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$RocksDBSegmentsBatchingRestoreCallback.restoreAll(RocksDBSegmentedBytesStore.java:262) at org.apache.kafka.streams.processor.internals.StateRestoreCallbackAdapter.lambda$adapt$0(StateRestoreCallbackAdapter.java:42) at org.apache.kafka.streams.processor.internals.CompositeRestoreListener.restoreBatch(CompositeRestoreListener.java:89) at org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:83) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:310) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:92) at org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:321) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:839) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747) org.apache.kafka.streams.errors.ProcessorStateException: Error opening store KSTREAM-JOINTHIS-0000000018-store.1551343860000 at location /data/foo/0_1/KSTREAM-JOINTHIS-0000000018-store/KSTREAM-JOINTHIS-0000000018-store.1551343860000 at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:158) at org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:45) at org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:101) at org.apache.kafka.streams.state.internals.Segments.getOrCreateSegmentIfLive(Segments.java:81) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.getWriteBatches(RocksDBSegmentedBytesStore.java:224) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.restoreAllInternal(RocksDBSegmentedBytesStore.java:208) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$RocksDBSegmentsBatchingRestoreCallback.restoreAll(RocksDBSegmentedBytesStore.java:262) at org.apache.kafka.streams.processor.internals.StateRestoreCallbackAdapter.lambda$adapt$0(StateRestoreCallbackAdapter.java:42) at org.apache.kafka.streams.processor.internals.CompositeRestoreListener.restoreBatch(CompositeRestoreListener.java:89) at org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:83) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:310) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:92) at org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:321) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:839) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747) Caused by: org.rocksdb.RocksDBException: While appending to file: /data/foo/0_1/KSTREAM-JOINTHIS-0000000018-store/KSTREAM-JOINTHIS-0000000018-store.1551343860000/MANIFEST-000001: No space left on device at org.rocksdb.RocksDB.open(Native Method) at org.rocksdb.RocksDB.open(RocksDB.java:231) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:156) ... 15 common frames omitted It feels likely that this is some serious amount of preallocation that is happening that didn’t use to? I have eliminated https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8202261 being an issue – we see this behaviour with 8u141, 8u181 and 8u202 Any help or avenues of investigation are welcome. Thanks Adrian Disclaimer This e-mail has been sent to you by Zopa. Zopa Limited is authorised and regulated by the Financial Conduct Authority, and entered on the Financial Services Register under firm registration number 718925. Zopa Bank Limited is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority, and entered on the Financial Services Register (800542). Registered Office: Zopa Limited (05197592) and Zopa Bank Limited (10627575) are both incorporated in England & Wales and have their registered office at: 1st Floor, Cottons Centre, Tooley Street, London, SE1 2QG.