Interesting. $ ulimit -n 2560
I remember seeing something in the documentation about this…I also see Riak returning HTTP 405's every now and then when it's under load. Perhaps related? On Aug 29, 2012, at 8:43 PM, Alexander Sicular <sicul...@gmail.com> wrote: > What's your "ulimit -n" ? > > I think you ran out of fd's. I cite the "io error: lock" mumbo jumbo. > > -Alexander > > @siculars > http://siculars.posterous.com > > Sent from my iRotaryPhone > > On Aug 29, 2012, at 23:07, Brad Heller <b...@cloudability.com> wrote: > >> Hello Riak world, >> >> I've been experimenting with migrating some of our OLAP data in to Riak >> recently. I'm still learning about the…particulars…of Riak, so apologies is >> the solution to this is obvious or this is an overt n00b question. >> >> I'm developing on a three-ring Riak cluster on my machine (OSX 10.8.1). I'm >> primarily using Ruby + Ripple but I've done a lot exploration with Curl too. >> I'm also using Rekon as a way to peek in to data I'm storing. >> >> The issue I'm facing: I tried to run an improperly-formatted MapReduce job >> against a bucket with about 45k keys in it and it seemed to crash Riak. >> Here's the job itself: >> >> 1.9.3p194 :065 > puts job.to_json >> {"inputs":{"bucket":"raw_statistics","key_filters":[["starts_with","some_string"],["and",[[["tokenize",":",4]],[["between",1345197700,1345697700,true]]]]]},"query":[{"map":{"language":"javascript","keep":true,"name":"Riak.mapValuesJson"}}]} >> >> I would expect about 2.5k matches to the map. Here's the output from one of >> the vnodes' error.log >> >> 2012-08-29 19:27:52.908 [error] <0.420.0>@riak_pipe_vnode:new_worker:766 >> Pipe worker startup failed:fitting was gone before startup >> 2012-08-29 19:45:41.739 [error] <0.959.0> gen_fsm <0.959.0> in state active >> terminated with reason: no match of right hand value >> {error,{bad_filter,[<<"tokenize">>,<<":">>,4]}} in >> riak_kv_mapred_filters:'-logical_and/1-fun-0-'/1 line 176 >> 2012-08-29 19:45:41.773 [error] <0.594.0> gen_fsm <0.594.0> in state active >> terminated with reason: no match of right hand value >> {error,{bad_filter,[<<"tokenize">>,<<":">>,4]}} in >> riak_kv_mapred_filters:'-logical_and/1-fun-0-'/1 line 176 >> 2012-08-29 19:45:41.778 [error] <0.594.0> CRASH REPORT Process <0.594.0> >> with 1 neighbours exited with reason: no match of right hand value >> {error,{bad_filter,[<<"tokenize">>,<<":">>,4]}} in >> riak_kv_mapred_filters:'-logical_and/1-fun-0-'/1 line 176 in >> gen_fsm:terminate/7 line 611 >> 2012-08-29 19:45:41.785 [error] <0.23924.70>@riak_kv_vnode:init:265 Failed >> to start riak_kv_multi_backend Reason: >> [{riak_kv_eleveldb_backend,{db_open,"IO error: lock >> ../../tmp/riak/instance1/leveldb/0/LOCK: Resource temporarily unavailable"}}] >> 2012-08-29 19:45:41.814 [error] <0.141.0> Supervisor riak_core_vnode_sup had >> child undefined started with {riak_core_vnode,start_link,undefined} at >> <0.594.0> exit with reason no match of right hand value >> {error,{bad_filter,[<<"tokenize">>,<<":">>,4]}} in >> riak_kv_mapred_filters:'-logical_and/1-fun-0-'/1 line 176 in context >> child_terminated >> 2012-08-29 19:45:41.818 [error] <0.959.0> CRASH REPORT Process <0.959.0> >> with 1 neighbours exited with reason: no match of right hand value >> {error,{bad_filter,[<<"tokenize">>,<<":">>,4]}} in >> riak_kv_mapred_filters:'-logical_and/1-fun-0-'/1 line 176 in >> gen_fsm:terminate/7 line 611 >> 2012-08-29 19:45:41.822 [error] <0.141.0> Supervisor riak_core_vnode_sup had >> child undefined started with {riak_core_vnode,start_link,undefined} at >> <0.959.0> exit with reason no match of right hand value >> {error,{bad_filter,[<<"tokenize">>,<<":">>,4]}} in >> riak_kv_mapred_filters:'-logical_and/1-fun-0-'/1 line 176 in context >> child_terminated >> 2012-08-29 19:45:41.943 [error] <0.962.0> gen_fsm <0.962.0> in state ready >> terminated with reason: no match of right hand value >> {error,{bad_filter,[<<"tokenize">>,<<":">>,4]}} in >> riak_kv_mapred_filters:'-logical_and/1-fun-0-'/1 line 176 >> >> For what it's worth the format of my key is as follows (if anyone has any >> suggestions on a smarter way to format these, I'm all ears). >> >> <some piece of user data>:<user ID>:<some other piece of data>:<timestamp in >> seconds> >> >> So my question is: Why did this completely kill Riak? This makes me pretty >> nervous--a bug in our app has the potential to bring down the ring! Is there >> anything we can do to protect against this? >> >> And a bonus question: What is a reasonable way to query this? I can't >> maintain links as there will potentially be hundreds of thousands of these >> objects to query over (each one is pretty small). Is this a good candidate >> for a compound secondary index? >> >> Thanks for any help. >> >> Cheers, >> >> Brad Heller | Engineering Lead | Cloudability.com | 541-231-1514 | Skype: >> brad.heller | @bradhe | @cloudability >> >> We're hiring! http://cloudability.com/jobs >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
