At this moment I have 5 instances each running 2 threads. Single instance / machine.
Define 'full logs' ? On Mon, Dec 12, 2016 at 3:54 AM, Damian Guy <damian....@gmail.com> wrote: > Jon, > > How many StreamThreads do you have running? > How many application instances? > Do you have more than one instance per machine? If yes, are they sharing > the same State Directory? > Do you have full logs that can be provided so we can try and see how/what > is happening? > > Thanks, > Damian > > On Mon, 12 Dec 2016 at 10:17 Jon Yeargers <jon.yearg...@cedexis.com> > wrote: > > > No luck here. Moved all state storage to a non-tmp folder and restarted. > > Still hitting the 'No locks available' error quite frequently. > > > > On Sun, Dec 11, 2016 at 3:45 PM, Jon Yeargers <jon.yearg...@cedexis.com> > > wrote: > > > > > I moved the state folder to a separate drive and linked out to it. > > > > > > I'll try your suggestion and point directly. > > > > > > On Sun, Dec 11, 2016 at 2:20 PM, Matthias J. Sax < > matth...@confluent.io> > > > wrote: > > > > > >> I am not sure, but this might be related with your state directory. > > >> > > >> You use default directory that is located in /tmp -- could it be, that > > >> /tmp gets clean up and thus you loose files/directories? > > >> > > >> Try to reconfigure your state directory via StreamsConfig: > > >> http://docs.confluent.io/current/streams/developer-guide. > > >> html#optional-configuration-parameters > > >> > > >> > > >> -Matthias > > >> > > >> On 12/11/16 1:28 AM, Jon Yeargers wrote: > > >> > Seeing this appearing somewhat frequently - > > >> > > > >> > org.apache.kafka.streams.errors.ProcessorStateException: Error > opening > > >> > store minute_agg_stream-201612100812 at location > > >> > /tmp/kafka-streams/MinuteAgg/1_9/minute_agg_stream/minute_ag > > >> g_stream-201612100812 > > >> > > > >> > at > > >> > org.apache.kafka.streams.state.internals.RocksDBStore.openDB > > >> (RocksDBStore.java:196) > > >> > > > >> > at > > >> > org.apache.kafka.streams.state.internals.RocksDBStore.openDB > > >> (RocksDBStore.java:158) > > >> > > > >> > at > > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore$ > > >> Segment.openDB(RocksDBWindowStore.java:72) > > >> > > > >> > at > > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore. > > >> getOrCreateSegment(RocksDBWindowStore.java:402) > > >> > > > >> > at > > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore. > > >> putInternal(RocksDBWindowStore.java:333) > > >> > > > >> > at > > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore. > > >> access$100(RocksDBWindowStore.java:51) > > >> > > > >> > at > > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore$ > > >> 2.restore(RocksDBWindowStore.java:212) > > >> > > > >> > at > > >> > org.apache.kafka.streams.processor.internals.ProcessorStateM > > >> anager.restoreActiveState(ProcessorStateManager.java:235) > > >> > > > >> > at > > >> > org.apache.kafka.streams.processor.internals.ProcessorStateM > > >> anager.register(ProcessorStateManager.java:198) > > >> > > > >> > at > > >> > org.apache.kafka.streams.processor.internals.ProcessorContex > > >> tImpl.register(ProcessorContextImpl.java:123) > > >> > > > >> > at > > >> > org.apache.kafka.streams.state.internals.RocksDBWindowStore. > > >> init(RocksDBWindowStore.java:206) > > >> > > > >> > at > > >> > org.apache.kafka.streams.state.internals.MeteredWindowStore. > > >> init(MeteredWindowStore.java:66) > > >> > > > >> > at > > >> > org.apache.kafka.streams.state.internals.CachingWindowStore. > > >> init(CachingWindowStore.java:64) > > >> > > > >> > at > > >> > org.apache.kafka.streams.processor.internals.AbstractTask.in > > >> itializeStateStores(AbstractTask.java:81) > > >> > > > >> > at > > >> > org.apache.kafka.streams.processor.internals.StreamTask.< > > >> init>(StreamTask.java:120) > > >> > > > >> > at > > >> > org.apache.kafka.streams.processor.internals.StreamThread. > > >> createStreamTask(StreamThread.java:633) > > >> > > > >> > at > > >> > org.apache.kafka.streams.processor.internals.StreamThread. > > >> addStreamTasks(StreamThread.java:660) > > >> > > > >> > at > > >> > org.apache.kafka.streams.processor.internals.StreamThread. > > >> access$100(StreamThread.java:69) > > >> > > > >> > at > > >> > org.apache.kafka.streams.processor.internals.StreamThread$1. > > >> onPartitionsAssigned(StreamThread.java:124) > > >> > > > >> > at > > >> > org.apache.kafka.clients.consumer.internals.ConsumerCoordina > > >> tor.onJoinComplete(ConsumerCoordinator.java:228) > > >> > > > >> > at > > >> > org.apache.kafka.clients.consumer.internals.AbstractCoordina > > >> tor.joinGroupIfNeeded(AbstractCoordinator.java:313) > > >> > > > >> > at > > >> > org.apache.kafka.clients.consumer.internals.AbstractCoordina > > >> tor.ensureActiveGroup(AbstractCoordinator.java:277) > > >> > > > >> > at > > >> > org.apache.kafka.clients.consumer.internals.ConsumerCoordina > > >> tor.poll(ConsumerCoordinator.java:259) > > >> > > > >> > at > > >> > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce( > > >> KafkaConsumer.java:1013) > > >> > > > >> > at > > >> > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaCo > > >> nsumer.java:979) > > >> > > > >> > at > > >> > org.apache.kafka.streams.processor.internals.StreamThread. > > >> runLoop(StreamThread.java:407) > > >> > > > >> > at > > >> > org.apache.kafka.streams.processor.internals.StreamThread. > > >> run(StreamThread.java:242) > > >> > > > >> > Caused by: org.rocksdb.RocksDBException: IO error: lock > > >> > /tmp/kafka-streams/MinuteAgg/1_9/minute_agg_stream/minute_ag > > >> g_stream-201612100812/LOCK: > > >> > No locks available > > >> > > > >> > at org.rocksdb.RocksDB.open(Native Method) > > >> > > > >> > at org.rocksdb.RocksDB.open(RocksDB.java:184) > > >> > > > >> > at > > >> > org.apache.kafka.streams.state.internals.RocksDBStore.openDB > > >> (RocksDBStore.java:189) > > >> > > > >> > ... 26 common frames omitted > > >> > > > >> > > >> > > > > > >
