Re: rocksdb error(s)

2016-12-13 Thread Damian Guy
Hi Jon, But i don't see any attached logs? Thanks, Damian On Mon, 12 Dec 2016 at 19:35 Jon Yeargers wrote: > The attached error log shows several of the problems I've run into. After > hitting this list (and typically much less) the app dies. > > On Mon, Dec 12, 2016 at 4:48 AM, Damian Guy wr

Re: rocksdb error(s)

2016-12-12 Thread Jon Yeargers
The attached error log shows several of the problems I've run into. After hitting this list (and typically much less) the app dies. On Mon, Dec 12, 2016 at 4:48 AM, Damian Guy wrote: > Just set the log level to debug and then run your app until you start > seeing the problem. > Thanks > > On Mon

Re: rocksdb error(s)

2016-12-12 Thread Damian Guy
Just set the log level to debug and then run your app until you start seeing the problem. Thanks On Mon, 12 Dec 2016 at 12:47 Jon Yeargers wrote: > I can log whatever you need. Tell me what is useful. > > On Mon, Dec 12, 2016 at 4:43 AM, Damian Guy wrote: > > > If you provide the logs from your

Re: rocksdb error(s)

2016-12-12 Thread Jon Yeargers
I can log whatever you need. Tell me what is useful. On Mon, Dec 12, 2016 at 4:43 AM, Damian Guy wrote: > If you provide the logs from your streams application then we might have > some chance of working out what is going on. Without logs then we really > don't have much hope of diagnosing the p

Re: rocksdb error(s)

2016-12-12 Thread Damian Guy
If you provide the logs from your streams application then we might have some chance of working out what is going on. Without logs then we really don't have much hope of diagnosing the problem. On Mon, 12 Dec 2016 at 12:18 Jon Yeargers wrote: > Im running as many threads as I have partitions on

Re: rocksdb error(s)

2016-12-12 Thread Jon Yeargers
Im running as many threads as I have partitions on this topic. Just curious if it would make any difference to the seemingly endless rebalancing woes. So far no change. In fact, I'll often see all 10 partitions (plus the 2 x 10 for the two aggregations) assigned to a single thread. On Mon, Dec 12

Re: rocksdb error(s)

2016-12-12 Thread Jon Yeargers
At this moment I have 5 instances each running 2 threads. Single instance / machine. Define 'full logs' ? On Mon, Dec 12, 2016 at 3:54 AM, Damian Guy wrote: > Jon, > > How many StreamThreads do you have running? > How many application instances? > Do you have more than one instance per machine?

Re: rocksdb error(s)

2016-12-12 Thread Damian Guy
Jon, How many StreamThreads do you have running? How many application instances? Do you have more than one instance per machine? If yes, are they sharing the same State Directory? Do you have full logs that can be provided so we can try and see how/what is happening? Thanks, Damian On Mon, 12 De

Re: rocksdb error(s)

2016-12-12 Thread Jon Yeargers
No luck here. Moved all state storage to a non-tmp folder and restarted. Still hitting the 'No locks available' error quite frequently. On Sun, Dec 11, 2016 at 3:45 PM, Jon Yeargers wrote: > I moved the state folder to a separate drive and linked out to it. > > I'll try your suggestion and point

Re: rocksdb error(s)

2016-12-11 Thread Jon Yeargers
I moved the state folder to a separate drive and linked out to it. I'll try your suggestion and point directly. On Sun, Dec 11, 2016 at 2:20 PM, Matthias J. Sax wrote: > I am not sure, but this might be related with your state directory. > > You use default directory that is located in /tmp --

Re: rocksdb error(s)

2016-12-11 Thread Matthias J. Sax
I am not sure, but this might be related with your state directory. You use default directory that is located in /tmp -- could it be, that /tmp gets clean up and thus you loose files/directories? Try to reconfigure your state directory via StreamsConfig: http://docs.confluent.io/current/streams/d

rocksdb error(s)

2016-12-11 Thread Jon Yeargers
Seeing this appearing somewhat frequently - org.apache.kafka.streams.errors.ProcessorStateException: Error opening store minute_agg_stream-201612100812 at location /tmp/kafka-streams/MinuteAgg/1_9/minute_agg_stream/minute_agg_stream-201612100812 at org.apache.kafka.streams.state.internals