Hi Frank, Very hard to say from the stack trace. The exception isn't the most informative! Is there anything obvious in the RocksDB LOG?
Thanks, Damian On Wed, 15 Feb 2017 at 06:38 Frank Lyaruu <flya...@gmail.com> wrote: > Hi Kafka crew, > > I'm rolling out a Kafka Streams application on a pretty large dataset, and > I see some exceptions that worry me, which I haven't seen with smaller data > sets > (Using Kafka streams trunk of a few days ago) > > 2017-02-15 15:25:14,431 | INFO | StreamThread-42 | StreamTask > | 86 - com.dexels.kafka.streams - 0.0.53.201702141103 | task > [9_4] Initializing state stores > 2017-02-15 15:25:14,433 | ERROR | StreamThread-42 | ConsumerCoordinator > | 86 - com.dexels.kafka.streams - 0.0.53.201702141103 | User > provided listener > org.apache.kafka.streams.processor.internals.StreamThread$1 for group > KNVB-test-generation-175-person failed on partition assignment > org.apache.kafka.streams.errors.ProcessorStateException: Error opening > store KNVB-test-generation-175-person-sportlinkkernel-PERSONCOMMUNICATION > at location > > /output/env.managed.repository-master/storage/KNVB-test-generation-175-person/9_4/rocksdb/KNVB-test-generation-175-person-sportlinkkernel-PERSONCOMMUNICATION > at > > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:175) > at > > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:145) > at > > org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:150) > at > > org.apache.kafka.streams.state.internals.MeteredKeyValueStore$7.run(MeteredKeyValueStore.java:100) > at > > org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188) > at > > org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:131) > at > > org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:62) > at > > org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:85) > at > > org.apache.kafka.streams.processor.internals.StreamTask.<init>(StreamTask.java:143) > at > > org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834) > at > > org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207) > at > > org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180) > at > > org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937) > at > > org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69) > at > > org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236) > at > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255) > at > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339) > at > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303) > at > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1030) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995) > at > > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:582) > at > > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368) > Caused by: org.rocksdb.RocksDBException: > at org.rocksdb.RocksDB.open(Native Method) > at org.rocksdb.RocksDB.open(RocksDB.java:231) > at > > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:168) > ... 22 more > > I've checked the folder it points to: > > [root@842376-demeter1 KNVB-test-generation-175-personcore-PERSONCORE]# ls > -l > total 223348 > -rw-r--r--. 1 431 433 68153335 Feb 15 11:09 000353.sst > -rw-r--r--. 1 431 433 68207125 Feb 15 11:09 000355.sst > -rw-r--r--. 1 431 433 7241027 Feb 15 11:09 000356.sst > -rw-r--r--. 1 431 433 59070938 Feb 15 11:12 000459.sst > -rw-r--r--. 1 431 433 18783280 Feb 15 11:13 000489.sst > -rw-r--r--. 1 431 433 0 Feb 15 11:38 000502.log > -rw-r--r--. 1 431 433 7183476 Feb 15 11:38 000505.sst > -rw-r--r--. 1 431 433 16 Feb 15 11:38 CURRENT > -rw-r--r--. 1 431 433 37 Feb 15 10:57 IDENTITY > -rw-r--r--. 1 431 433 0 Feb 15 10:57 LOCK > -rw-r--r--. 1 431 433 13446 Feb 15 11:38 LOG > -rw-r--r--. 1 431 433 13130 Feb 15 10:57 LOG.old.1487158687061100 > -rw-r--r--. 1 431 433 456 Feb 15 11:38 MANIFEST-000501 > -rw-r--r--. 1 431 433 4275 Feb 15 10:57 OPTIONS-000005 > -rw-r--r--. 1 431 433 4275 Feb 15 11:38 OPTIONS-000504 > [root@842376-demeter1 KNVB-test-generation-175-personcore-PERSONCORE]# > > What strikes me as odd is that none of the files have been touched in the > last few hours (the stacktrace is from 15:25), so there doesn't appear to > have been any logging of that error (on the rocksdb side) > > Any ideas? Or should I post this to RocksDb? > > regards, Frank >
