For those interested, I identified my performance problem. I was creating a lot of buckets, and the properties did not match the default bucket properties of the node. So getting the bucket was taking between 300-400 milliseconds instead of 3-4. Apparently creating buckets with non default bucket properties is not just a bad idea, but a REALLY bad idea.
I changed the default to be what I wanted for most of my buckets (allow_mult=true) and tweaked the code to handle siblings where I wasn't before, and now I'm getting lots of operations per second. Thanks the for help every one! Paul Paul Ingalls Founder & CEO Fanzo p...@fanzo.me @paulingalls http://www.linkedin.com/in/paulingalls On Aug 2, 2013, at 12:00 AM, Paul Ingalls <p...@fanzo.me> wrote: > One thing I am doing different than the benchmark is creating a lot of > buckets… > > > Paul Ingalls > Founder & CEO Fanzo > p...@fanzo.me > @paulingalls > http://www.linkedin.com/in/paulingalls > > > > On Aug 1, 2013, at 11:52 PM, Paul Ingalls <p...@fanzo.me> wrote: > >> The objects are either tweet jsons, so between 1-2k, or simple strings with >> some links. >> >> I am using secondary indexes with timestamps so I can do range queries. >> Would this significantly impact performance? I am not indexing for search... >> >> Here are some more bench results. Obviously using the azure load balancer >> is a bad idea. Fortunately I'm not. >> >> This was run from my laptop using 8 concurrent connections and the pb >> client, using the load balancer in azure. Lots of errors. >> >> <summary.png> >> >> >> >> this was run on a vm in the same cloud service using 8 concurrent workers >> and the pb client, but clustered directly (i input all the IP addresses to >> riakc_pb_ips). Much better ops. So it must be something with my usage that >> isn't getting captured by the benchmark. >> >> <summary.png> >> >> Paul Ingalls >> Founder & CEO Fanzo >> p...@fanzo.me >> @paulingalls >> http://www.linkedin.com/in/paulingalls >> >> >> >> On Aug 1, 2013, at 10:46 PM, Jeremy Ong <jer...@quarkgames.com> wrote: >> >>> What is the average size of one of your objects? Are you indexing? >>> >>> >>> On Thu, Aug 1, 2013 at 10:23 PM, Paul Ingalls <p...@fanzo.me> wrote: >>> Running them on separate machines. >>> >>> I just ran basho bench against the service from my laptop. here is the >>> result. in this case I am just hitting one node. will try it again >>> shortly using a load balancer to hit all the nodes. >>> >>> <summary.png> >>> >>> Paul Ingalls >>> Founder & CEO Fanzo >>> p...@fanzo.me >>> @paulingalls >>> http://www.linkedin.com/in/paulingalls >>> >>> >>> >>> On Aug 1, 2013, at 8:57 PM, Jared Morrow <ja...@basho.com> wrote: >>> >>>> Also, are you generating the load from the same VMs that Riak is running >>>> on or do you have separate machines generating load? >>>> >>>> >>>> >>>> On Thursday, August 1, 2013, Jeremy Ong wrote: >>>> What erlang version did you build with? How are you load balancing >>>> between the nodes? What kind of disks are you using? >>>> >>>> On Thu, Aug 1, 2013 at 7:53 PM, Paul Ingalls <p...@fanzo.me> wrote: >>>> > FYI, 2 more nodes died with the end of the last test. Storm, which I'm >>>> > using to put data in, kills the topology a bit abruptly, perhaps the >>>> > nodes >>>> > don't like a client going away like that? >>>> > >>>> > log from one of the nodes: >>>> > >>>> > 2013-08-02 02:27:23 =ERROR REPORT==== >>>> > Error in process <0.4959.0> on node 'riak@riak004' with exit value: >>>> > {badarg,[{riak_core_stat,vnodeq_len,1,[{file,"src/riak_core_stat.erl"},{line,181}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[{file,"src/riak_core_stat.erl"},{line,172}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[... >>>> > >>>> > 2013-08-02 02:27:33 =ERROR REPORT==== >>>> > Error in process <0.5055.0> on node 'riak@riak004' with exit value: >>>> > {badarg,[{riak_core_stat,vnodeq_len,1,[{file,"src/riak_core_stat.erl"},{line,181}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[{file,"src/riak_core_stat.erl"},{line,172}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[... >>>> > >>>> > 2013-08-02 02:27:51 =ERROR REPORT==== >>>> > Error in process <0.5228.0> on node 'riak@riak004' with exit value: >>>> > {badarg,[{riak_core_stat,vnodeq_len,1,[{file,"src/riak_core_stat.erl"},{line,181}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[{file,"src/riak_core_stat.erl"},{line,172}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[... >>>> > >>>> > and the log from the other node: >>>> > >>>> > 2013-08-02 00:09:39 =ERROR REPORT==== >>>> > Error in process <0.4952.0> on node 'riak@riak007' with exit value: >>>> > {badarg,[{riak_core_stat,vnodeq_len,1,[{file,"src/riak_core_stat.erl"},{line,181}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[{file,"src/riak_core_stat.erl"},{line,172}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[... >>>> > >>>> > 2013-08-02 00:09:44 =ERROR REPORT==== >>>> > ** State machine <0.2368.0> terminating >>>> > ** Last event in was unregistered >>>> > ** When State == active >>>> > ** Data == >>>> > {state,114179815416476790484662877555959610910619729920,riak_kv_vnode,{deleted,{state,114179815416476790484662877555959610910619729920,riak_kv_eleveldb_backend,{state,<<>>,"/mnt/datadrive/riak/data/leveldb/114179815416476790484662877555959610910619729920",[{create_if_missing,true},{max_open_files,128},{use_bloomfilter,true},{write_buffer_size,58858594}],[{add_paths,[]},{allow_strfun,false},{anti_entropy,{on,[]}},{anti_entropy_build_limit,{1,3600000}},{anti_entropy_concurrency,2},{anti_entropy_data_dir,"/mnt/datadrive/riak/data/anti_entropy"},{anti_entropy_expire,604800000},{anti_entropy_leveldb_opts,[{write_buffer_size,4194304},{max_open_files,20}]},{anti_entropy_tick,15000},{create_if_missing,true},{data_root,"/mnt/datadrive/riak/data/leveldb"},{fsm_limit,50000},{hook_js_vm_count,2},{http_url_encoding,on},{included_applications,[]},{js_max_vm_mem,8},{js_thread_stack,16},{legacy_stats,true},{listkeys_backpressure,true},{map_js_vm_count,8},{mapred_2i_pipe,true},{mapred_name,"mapred"},{max_open_files,128},{object_format,v1},{reduce_js_vm_count,6},{stats_urlpath,"stats"},{storage_backend,riak_kv_eleveldb_backend},{use_bloomfilter,true},{vnode_vclocks,true},{write_buffer_size,58858594}],[],[],[{fill_cache,false}],true,false},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},undefined,3000,1000,100,100,true,true,undefined}},riak@riak003,none,undefined,undefined,undefined,{pool,riak_kv_worker,10,[]},undefined,107615} >>>> > ** Reason for termination = >>>> > ** >>>> > {badarg,[{eleveldb,close,[<<>>],[]},{riak_kv_eleveldb_backend,stop,1,[{file,"src/riak_kv_eleveldb_backend.erl"},{line,149}]},{riak_kv_vnode,terminate,2,[{file,"src/riak_kv_vnode.erl"},{line,836}]},{riak_core_vnode,terminate,3,[{file,"src/riak_core_vnode.erl"},{line,847}]},{gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,586}]},{proc_lib,init_p_do_apply,3,[{file," >>> >>> >> >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
