With three instances running you should definitely bump it up. Try "ulimit -n 16384" (or more) in a terminal and then start your three instances from that terminal. Rerun your tests.
@siculars http://siculars.posterous.com Sent from my iRotaryPhone On Aug 30, 2012, at 2:10, Brad Heller <b...@cloudability.com> wrote: > Interesting. > > $ ulimit -n > 2560 > > I remember seeing something in the documentation about this…I also see Riak > returning HTTP 405's every now and then when it's under load. Perhaps related? > > On Aug 29, 2012, at 8:43 PM, Alexander Sicular <sicul...@gmail.com> wrote: > >> What's your "ulimit -n" ? >> >> I think you ran out of fd's. I cite the "io error: lock" mumbo jumbo. >> >> -Alexander >> >> @siculars >> http://siculars.posterous.com >> >> Sent from my iRotaryPhone >> >> On Aug 29, 2012, at 23:07, Brad Heller <b...@cloudability.com> wrote: >> >>> Hello Riak world, >>> >>> I've been experimenting with migrating some of our OLAP data in to Riak >>> recently. I'm still learning about the…particulars…of Riak, so apologies is >>> the solution to this is obvious or this is an overt n00b question. >>> >>> I'm developing on a three-ring Riak cluster on my machine (OSX 10.8.1). I'm >>> primarily using Ruby + Ripple but I've done a lot exploration with Curl >>> too. I'm also using Rekon as a way to peek in to data I'm storing. >>> >>> The issue I'm facing: I tried to run an improperly-formatted MapReduce job >>> against a bucket with about 45k keys in it and it seemed to crash Riak. >>> Here's the job itself: >>> >>> 1.9.3p194 :065 > puts job.to_json >>> {"inputs":{"bucket":"raw_statistics","key_filters":[["starts_with","some_string"],["and",[[["tokenize",":",4]],[["between",1345197700,1345697700,true]]]]]},"query":[{"map":{"language":"javascript","keep":true,"name":"Riak.mapValuesJson"}}]} >>> >>> I would expect about 2.5k matches to the map. Here's the output from one of >>> the vnodes' error.log >>> >>> 2012-08-29 19:27:52.908 [error] <0.420.0>@riak_pipe_vnode:new_worker:766 >>> Pipe worker startup failed:fitting was gone before startup >>> 2012-08-29 19:45:41.739 [error] <0.959.0> gen_fsm <0.959.0> in state active >>> terminated with reason: no match of right hand value >>> {error,{bad_filter,[<<"tokenize">>,<<":">>,4]}} in >>> riak_kv_mapred_filters:'-logical_and/1-fun-0-'/1 line 176 >>> 2012-08-29 19:45:41.773 [error] <0.594.0> gen_fsm <0.594.0> in state active >>> terminated with reason: no match of right hand value >>> {error,{bad_filter,[<<"tokenize">>,<<":">>,4]}} in >>> riak_kv_mapred_filters:'-logical_and/1-fun-0-'/1 line 176 >>> 2012-08-29 19:45:41.778 [error] <0.594.0> CRASH REPORT Process <0.594.0> >>> with 1 neighbours exited with reason: no match of right hand value >>> {error,{bad_filter,[<<"tokenize">>,<<":">>,4]}} in >>> riak_kv_mapred_filters:'-logical_and/1-fun-0-'/1 line 176 in >>> gen_fsm:terminate/7 line 611 >>> 2012-08-29 19:45:41.785 [error] <0.23924.70>@riak_kv_vnode:init:265 Failed >>> to start riak_kv_multi_backend Reason: >>> [{riak_kv_eleveldb_backend,{db_open,"IO error: lock >>> ../../tmp/riak/instance1/leveldb/0/LOCK: Resource temporarily >>> unavailable"}}] >>> 2012-08-29 19:45:41.814 [error] <0.141.0> Supervisor riak_core_vnode_sup >>> had child undefined started with {riak_core_vnode,start_link,undefined} at >>> <0.594.0> exit with reason no match of right hand value >>> {error,{bad_filter,[<<"tokenize">>,<<":">>,4]}} in >>> riak_kv_mapred_filters:'-logical_and/1-fun-0-'/1 line 176 in context >>> child_terminated >>> 2012-08-29 19:45:41.818 [error] <0.959.0> CRASH REPORT Process <0.959.0> >>> with 1 neighbours exited with reason: no match of right hand value >>> {error,{bad_filter,[<<"tokenize">>,<<":">>,4]}} in >>> riak_kv_mapred_filters:'-logical_and/1-fun-0-'/1 line 176 in >>> gen_fsm:terminate/7 line 611 >>> 2012-08-29 19:45:41.822 [error] <0.141.0> Supervisor riak_core_vnode_sup >>> had child undefined started with {riak_core_vnode,start_link,undefined} at >>> <0.959.0> exit with reason no match of right hand value >>> {error,{bad_filter,[<<"tokenize">>,<<":">>,4]}} in >>> riak_kv_mapred_filters:'-logical_and/1-fun-0-'/1 line 176 in context >>> child_terminated >>> 2012-08-29 19:45:41.943 [error] <0.962.0> gen_fsm <0.962.0> in state ready >>> terminated with reason: no match of right hand value >>> {error,{bad_filter,[<<"tokenize">>,<<":">>,4]}} in >>> riak_kv_mapred_filters:'-logical_and/1-fun-0-'/1 line 176 >>> >>> For what it's worth the format of my key is as follows (if anyone has any >>> suggestions on a smarter way to format these, I'm all ears). >>> >>> <some piece of user data>:<user ID>:<some other piece of data>:<timestamp >>> in seconds> >>> >>> So my question is: Why did this completely kill Riak? This makes me pretty >>> nervous--a bug in our app has the potential to bring down the ring! Is >>> there anything we can do to protect against this? >>> >>> And a bonus question: What is a reasonable way to query this? I can't >>> maintain links as there will potentially be hundreds of thousands of these >>> objects to query over (each one is pretty small). Is this a good candidate >>> for a compound secondary index? >>> >>> Thanks for any help. >>> >>> Cheers, >>> >>> Brad Heller | Engineering Lead | Cloudability.com | 541-231-1514 | Skype: >>> brad.heller | @bradhe | @cloudability >>> >>> We're hiring! http://cloudability.com/jobs >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
