Hello Adrian, What you described did sounds wired to me. I'm not aware of any regressions on rocksDB disk usage from 1.1 to 2.1.
Could you file a JIRA ticket with more details like state dir snapshots, your code snippet and configs etc so we can find a way to reproduce it? Guozhang On Fri, Mar 1, 2019 at 5:54 AM Adrian McCague <adrian.mcca...@zopa.com> wrote: > Hi, > > We are in the process of attempting to upgrade from Kafka Streams 1.1.0 to > 2.1.0(-cp1) however we get some wildly different behaviour with regards to > disk usage between these two versions. > > An update that uses existing state data exhibits the same behaviour as > starting with a completely clean state data directories. > > With 1.1.0 the same topologies restoring the same state stores will at > most use 275MB. > With 2.1.0 they will quickly take up to 60GB – I extended the volumes to > see just how far it would go, already magnitudes larger than before. > > Only remotely interesting information in the logs are the following > exceptions which other than the explicit disk space exception I presume is > related to out of disk space: > > org.apache.kafka.streams.errors.ProcessorStateException: task [0_7] > Exception caught while trying to restore state from > foo-v4-KSTREAM-JOINTHIS-0000000012-store-changelog-7 > at > org.apache.kafka.streams.processor.internals.ProcessorStateManager.updateStandbyStates(ProcessorStateManager.java:185) > at > org.apache.kafka.streams.processor.internals.StandbyTask.update(StandbyTask.java:188) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:1114) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:895) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747) > Caused by: java.lang.NullPointerException: null > java.lang.NullPointerException: null > at > org.apache.kafka.streams.state.internals.RocksDBStore.toggleDbForBulkLoading(RocksDBStore.java:229) > at > org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.getWriteBatches(RocksDBSegmentedBytesStore.java:230) > at > org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.restoreAllInternal(RocksDBSegmentedBytesStore.java:208) > at > org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$RocksDBSegmentsBatchingRestoreCallback.restoreAll(RocksDBSegmentedBytesStore.java:262) > at > org.apache.kafka.streams.processor.internals.StateRestoreCallbackAdapter.lambda$adapt$0(StateRestoreCallbackAdapter.java:42) > at > org.apache.kafka.streams.processor.internals.CompositeRestoreListener.restoreBatch(CompositeRestoreListener.java:89) > at > org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:83) > at > org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:310) > at > org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:92) > at > org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:321) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:839) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747) > > > org.apache.kafka.streams.errors.ProcessorStateException: Error opening > store KSTREAM-JOINTHIS-0000000018-store.1551343860000 at location > /data/foo/0_1/KSTREAM-JOINTHIS-0000000018-store/KSTREAM-JOINTHIS-0000000018-store.1551343860000 > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:158) > at > org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:45) > at > org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:101) > at > org.apache.kafka.streams.state.internals.Segments.getOrCreateSegmentIfLive(Segments.java:81) > at > org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.getWriteBatches(RocksDBSegmentedBytesStore.java:224) > at > org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.restoreAllInternal(RocksDBSegmentedBytesStore.java:208) > at > org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$RocksDBSegmentsBatchingRestoreCallback.restoreAll(RocksDBSegmentedBytesStore.java:262) > at > org.apache.kafka.streams.processor.internals.StateRestoreCallbackAdapter.lambda$adapt$0(StateRestoreCallbackAdapter.java:42) > at > org.apache.kafka.streams.processor.internals.CompositeRestoreListener.restoreBatch(CompositeRestoreListener.java:89) > at > org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:83) > at > org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:310) > at > org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:92) > at > org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:321) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:839) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747) > Caused by: org.rocksdb.RocksDBException: While appending to file: > /data/foo/0_1/KSTREAM-JOINTHIS-0000000018-store/KSTREAM-JOINTHIS-0000000018-store.1551343860000/MANIFEST-000001: > No space left on device > at org.rocksdb.RocksDB.open(Native Method) > at org.rocksdb.RocksDB.open(RocksDB.java:231) > at > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:156) > ... 15 common frames omitted > > It feels likely that this is some serious amount of preallocation that is > happening that didn’t use to? > I have eliminated > https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8202261 being an > issue – we see this behaviour with 8u141, 8u181 and 8u202 > > Any help or avenues of investigation are welcome. > > Thanks > Adrian > > Disclaimer > > This e-mail has been sent to you by Zopa. > > Zopa Limited is authorised and regulated by the Financial Conduct > Authority, > and entered on the Financial Services Register under firm registration > number > 718925. Zopa Bank Limited is authorised by the Prudential Regulation > Authority > and regulated by the Financial Conduct Authority and the Prudential > Regulation > Authority, and entered on the Financial Services Register (800542). > > Registered Office: Zopa Limited (05197592) and Zopa Bank Limited > (10627575) are both incorporated in England & Wales and have their > registered office at: 1st Floor, Cottons Centre, Tooley Street, London, > SE1 2QG. > > -- -- Guozhang
