Excellent news. How many buckets with custom settings did you create? Sent from my iPhone
On Aug 2, 2013, at 6:51 PM, Paul Ingalls <p...@fanzo.me> wrote: For those interested, I identified my performance problem. I was creating a lot of buckets, and the properties did not match the default bucket properties of the node. So getting the bucket was taking between 300-400 milliseconds instead of 3-4. Apparently creating buckets with non default bucket properties is not just a bad idea, but a REALLY bad idea. I changed the default to be what I wanted for most of my buckets (allow_mult=true) and tweaked the code to handle siblings where I wasn't before, and now I'm getting lots of operations per second. Thanks the for help every one! Paul Paul Ingalls Founder & CEO Fanzo p...@fanzo.me @paulingalls http://www.linkedin.com/in/paulingalls On Aug 2, 2013, at 12:00 AM, Paul Ingalls <p...@fanzo.me> wrote: One thing I am doing different than the benchmark is creating a lot of buckets… Paul Ingalls Founder & CEO Fanzo p...@fanzo.me @paulingalls http://www.linkedin.com/in/paulingalls On Aug 1, 2013, at 11:52 PM, Paul Ingalls <p...@fanzo.me> wrote: The objects are either tweet jsons, so between 1-2k, or simple strings with some links. I am using secondary indexes with timestamps so I can do range queries. Would this significantly impact performance? I am not indexing for search... Here are some more bench results. Obviously using the azure load balancer is a bad idea. Fortunately I'm not. This was run from my laptop using 8 concurrent connections and the pb client, using the load balancer in azure. Lots of errors. <summary.png> this was run on a vm in the same cloud service using 8 concurrent workers and the pb client, but clustered directly (i input all the IP addresses to riakc_pb_ips). Much better ops. So it must be something with my usage that isn't getting captured by the benchmark. <summary.png> Paul Ingalls Founder & CEO Fanzo p...@fanzo.me @paulingalls http://www.linkedin.com/in/paulingalls On Aug 1, 2013, at 10:46 PM, Jeremy Ong <jer...@quarkgames.com> wrote: What is the average size of one of your objects? Are you indexing? On Thu, Aug 1, 2013 at 10:23 PM, Paul Ingalls <p...@fanzo.me> wrote: > Running them on separate machines. > > I just ran basho bench against the service from my laptop. here is the > result. in this case I am just hitting one node. will try it again > shortly using a load balancer to hit all the nodes. > > <summary.png> > > Paul Ingalls > Founder & CEO Fanzo > p...@fanzo.me > @paulingalls > http://www.linkedin.com/in/paulingalls > > > > On Aug 1, 2013, at 8:57 PM, Jared Morrow <ja...@basho.com> wrote: > > Also, are you generating the load from the same VMs that Riak is running > on or do you have separate machines generating load? > > > > On Thursday, August 1, 2013, Jeremy Ong wrote: > >> What erlang version did you build with? How are you load balancing >> between the nodes? What kind of disks are you using? >> >> On Thu, Aug 1, 2013 at 7:53 PM, Paul Ingalls <p...@fanzo.me> wrote: >> > FYI, 2 more nodes died with the end of the last test. Storm, which I'm >> > using to put data in, kills the topology a bit abruptly, perhaps the >> nodes >> > don't like a client going away like that? >> > >> > log from one of the nodes: >> > >> > 2013-08-02 02:27:23 =ERROR REPORT==== >> > Error in process <0.4959.0> on node 'riak@riak004' with exit value: >> > >> {badarg,[{riak_core_stat,vnodeq_len,1,[{file,"src/riak_core_stat.erl"},{line,181}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[{file,"src/riak_core_stat.erl"},{line,172}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[... >> > >> > 2013-08-02 02:27:33 =ERROR REPORT==== >> > Error in process <0.5055.0> on node 'riak@riak004' with exit value: >> > >> {badarg,[{riak_core_stat,vnodeq_len,1,[{file,"src/riak_core_stat.erl"},{line,181}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[{file,"src/riak_core_stat.erl"},{line,172}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[... >> > >> > 2013-08-02 02:27:51 =ERROR REPORT==== >> > Error in process <0.5228.0> on node 'riak@riak004' with exit value: >> > >> {badarg,[{riak_core_stat,vnodeq_len,1,[{file,"src/riak_core_stat.erl"},{line,181}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[{file,"src/riak_core_stat.erl"},{line,172}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[... >> > >> > and the log from the other node: >> > >> > 2013-08-02 00:09:39 =ERROR REPORT==== >> > Error in process <0.4952.0> on node 'riak@riak007' with exit value: >> > >> {badarg,[{riak_core_stat,vnodeq_len,1,[{file,"src/riak_core_stat.erl"},{line,181}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[{file,"src/riak_core_stat.erl"},{line,172}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[... >> > >> > 2013-08-02 00:09:44 =ERROR REPORT==== >> > ** State machine <0.2368.0> terminating >> > ** Last event in was unregistered >> > ** When State == active >> > ** Data == >> > >> {state,114179815416476790484662877555959610910619729920,riak_kv_vnode,{deleted,{state,114179815416476790484662877555959610910619729920,riak_kv_eleveldb_backend,{state,<<>>,"/mnt/datadrive/riak/data/leveldb/114179815416476790484662877555959610910619729920",[{create_if_missing,true},{max_open_files,128},{use_bloomfilter,true},{write_buffer_size,58858594}],[{add_paths,[]},{allow_strfun,false},{anti_entropy,{on,[]}},{anti_entropy_build_limit,{1,3600000}},{anti_entropy_concurrency,2},{anti_entropy_data_dir,"/mnt/datadrive/riak/data/anti_entropy"},{anti_entropy_expire,604800000},{anti_entropy_leveldb_opts,[{write_buffer_size,4194304},{max_open_files,20}]},{anti_entropy_tick,15000},{create_if_missing,true},{data_root,"/mnt/datadrive/riak/data/leveldb"},{fsm_limit,50000},{hook_js_vm_count,2},{http_url_encoding,on},{included_applications,[]},{js_max_vm_mem,8},{js_thread_stack,16},{legacy_stats,true},{listkeys_backpressure,true},{map_js_vm_count,8},{mapred_2i_pipe,true},{mapred_name,"mapred"},{max_open_files,128},{object_format,v1},{reduce_js_vm_count,6},{stats_urlpath,"stats"},{storage_backend,riak_kv_eleveldb_backend},{use_bloomfilter,true},{vnode_vclocks,true},{write_buffer_size,58858594}],[],[],[{fill_cache,false}],true,false},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},undefined,3000,1000,100,100,true,true,undefined}},riak@riak003 >> ,none,undefined,undefined,undefined,{pool,riak_kv_worker,10,[]},undefined,107615} >> > ** Reason for termination = >> > ** >> > >> {badarg,[{eleveldb,close,[<<>>],[]},{riak_kv_eleveldb_backend,stop,1,[{file,"src/riak_kv_eleveldb_backend.erl"},{line,149}]},{riak_kv_vnode,terminate,2,[{file,"src/riak_kv_vnode.erl"},{line,836}]},{riak_core_vnode,terminate,3,[{file,"src/riak_core_vnode.erl"},{line,847}]},{gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,586}]},{proc_lib,init_p_do_apply,3,[{file," > > > _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
