We are creating a lot of buckets because I needed the extra namespace. My alternative was constantly updating a value, which in some cases would grow the value very large. I figured keeping the key/value groups small would be quite a bit faster, and since the docs said you could have as many buckets as you like I thought I was good. I just didn't play close enough attention to the "except when" clause…:)
Thanks for the help everyone. I'll have some more questions soon. I ran into some challenges over the weekend that I want to learn more about… Paul Paul Ingalls Founder & CEO Fanzo p...@fanzo.me @paulingalls http://www.linkedin.com/in/paulingalls On Aug 2, 2013, at 5:08 PM, John Daily <jda...@basho.com> wrote: > 50k definitely is a fair few. What's the objective for that many, if I may > ask? Always interesting to hear about different data models. > > Sent from my iPhone > > On Aug 2, 2013, at 7:56 PM, Paul Ingalls <p...@fanzo.me> wrote: > >> Not that many, I didn't let it run that long since the performance was so >> poor. Maybe 50k or so... >> >> Paul Ingalls >> Founder & CEO Fanzo >> p...@fanzo.me >> @paulingalls >> http://www.linkedin.com/in/paulingalls >> >> >> >> On Aug 2, 2013, at 3:54 PM, John Daily <jda...@basho.com> wrote: >> >>> Excellent news. How many buckets with custom settings did you create? >>> >>> Sent from my iPhone >>> >>> On Aug 2, 2013, at 6:51 PM, Paul Ingalls <p...@fanzo.me> wrote: >>> >>>> For those interested, I identified my performance problem. >>>> >>>> I was creating a lot of buckets, and the properties did not match the >>>> default bucket properties of the node. So getting the bucket was taking >>>> between 300-400 milliseconds instead of 3-4. Apparently creating buckets >>>> with non default bucket properties is not just a bad idea, but a REALLY >>>> bad idea. >>>> >>>> I changed the default to be what I wanted for most of my buckets >>>> (allow_mult=true) and tweaked the code to handle siblings where I wasn't >>>> before, and now I'm getting lots of operations per second. >>>> >>>> Thanks the for help every one! >>>> >>>> Paul >>>> >>>> Paul Ingalls >>>> Founder & CEO Fanzo >>>> p...@fanzo.me >>>> @paulingalls >>>> http://www.linkedin.com/in/paulingalls >>>> >>>> >>>> >>>> On Aug 2, 2013, at 12:00 AM, Paul Ingalls <p...@fanzo.me> wrote: >>>> >>>>> One thing I am doing different than the benchmark is creating a lot of >>>>> buckets… >>>>> >>>>> >>>>> Paul Ingalls >>>>> Founder & CEO Fanzo >>>>> p...@fanzo.me >>>>> @paulingalls >>>>> http://www.linkedin.com/in/paulingalls >>>>> >>>>> >>>>> >>>>> On Aug 1, 2013, at 11:52 PM, Paul Ingalls <p...@fanzo.me> wrote: >>>>> >>>>>> The objects are either tweet jsons, so between 1-2k, or simple strings >>>>>> with some links. >>>>>> >>>>>> I am using secondary indexes with timestamps so I can do range queries. >>>>>> Would this significantly impact performance? I am not indexing for >>>>>> search... >>>>>> >>>>>> Here are some more bench results. Obviously using the azure load >>>>>> balancer is a bad idea. Fortunately I'm not. >>>>>> >>>>>> This was run from my laptop using 8 concurrent connections and the pb >>>>>> client, using the load balancer in azure. Lots of errors. >>>>>> >>>>>> <summary.png> >>>>>> >>>>>> >>>>>> >>>>>> this was run on a vm in the same cloud service using 8 concurrent >>>>>> workers and the pb client, but clustered directly (i input all the IP >>>>>> addresses to riakc_pb_ips). Much better ops. So it must be something >>>>>> with my usage that isn't getting captured by the benchmark. >>>>>> >>>>>> <summary.png> >>>>>> >>>>>> Paul Ingalls >>>>>> Founder & CEO Fanzo >>>>>> p...@fanzo.me >>>>>> @paulingalls >>>>>> http://www.linkedin.com/in/paulingalls >>>>>> >>>>>> >>>>>> >>>>>> On Aug 1, 2013, at 10:46 PM, Jeremy Ong <jer...@quarkgames.com> wrote: >>>>>> >>>>>>> What is the average size of one of your objects? Are you indexing? >>>>>>> >>>>>>> >>>>>>> On Thu, Aug 1, 2013 at 10:23 PM, Paul Ingalls <p...@fanzo.me> wrote: >>>>>>> Running them on separate machines. >>>>>>> >>>>>>> I just ran basho bench against the service from my laptop. here is the >>>>>>> result. in this case I am just hitting one node. will try it again >>>>>>> shortly using a load balancer to hit all the nodes. >>>>>>> >>>>>>> <summary.png> >>>>>>> >>>>>>> Paul Ingalls >>>>>>> Founder & CEO Fanzo >>>>>>> p...@fanzo.me >>>>>>> @paulingalls >>>>>>> http://www.linkedin.com/in/paulingalls >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Aug 1, 2013, at 8:57 PM, Jared Morrow <ja...@basho.com> wrote: >>>>>>> >>>>>>>> Also, are you generating the load from the same VMs that Riak is >>>>>>>> running on or do you have separate machines generating load? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thursday, August 1, 2013, Jeremy Ong wrote: >>>>>>>> What erlang version did you build with? How are you load balancing >>>>>>>> between the nodes? What kind of disks are you using? >>>>>>>> >>>>>>>> On Thu, Aug 1, 2013 at 7:53 PM, Paul Ingalls <p...@fanzo.me> wrote: >>>>>>>> > FYI, 2 more nodes died with the end of the last test. Storm, which >>>>>>>> > I'm >>>>>>>> > using to put data in, kills the topology a bit abruptly, perhaps the >>>>>>>> > nodes >>>>>>>> > don't like a client going away like that? >>>>>>>> > >>>>>>>> > log from one of the nodes: >>>>>>>> > >>>>>>>> > 2013-08-02 02:27:23 =ERROR REPORT==== >>>>>>>> > Error in process <0.4959.0> on node 'riak@riak004' with exit value: >>>>>>>> > {badarg,[{riak_core_stat,vnodeq_len,1,[{file,"src/riak_core_stat.erl"},{line,181}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[{file,"src/riak_core_stat.erl"},{line,172}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[... >>>>>>>> > >>>>>>>> > 2013-08-02 02:27:33 =ERROR REPORT==== >>>>>>>> > Error in process <0.5055.0> on node 'riak@riak004' with exit value: >>>>>>>> > {badarg,[{riak_core_stat,vnodeq_len,1,[{file,"src/riak_core_stat.erl"},{line,181}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[{file,"src/riak_core_stat.erl"},{line,172}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[... >>>>>>>> > >>>>>>>> > 2013-08-02 02:27:51 =ERROR REPORT==== >>>>>>>> > Error in process <0.5228.0> on node 'riak@riak004' with exit value: >>>>>>>> > {badarg,[{riak_core_stat,vnodeq_len,1,[{file,"src/riak_core_stat.erl"},{line,181}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[{file,"src/riak_core_stat.erl"},{line,172}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[... >>>>>>>> > >>>>>>>> > and the log from the other node: >>>>>>>> > >>>>>>>> > 2013-08-02 00:09:39 =ERROR REPORT==== >>>>>>>> > Error in process <0.4952.0> on node 'riak@riak007' with exit value: >>>>>>>> > {badarg,[{riak_core_stat,vnodeq_len,1,[{file,"src/riak_core_stat.erl"},{line,181}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[{file,"src/riak_core_stat.erl"},{line,172}]},{riak_core_stat,'-vnodeq_stats/0-lc$^0/1-0-',1,[... >>>>>>>> > >>>>>>>> > 2013-08-02 00:09:44 =ERROR REPORT==== >>>>>>>> > ** State machine <0.2368.0> terminating >>>>>>>> > ** Last event in was unregistered >>>>>>>> > ** When State == active >>>>>>>> > ** Data == >>>>>>>> > {state,114179815416476790484662877555959610910619729920,riak_kv_vnode,{deleted,{state,114179815416476790484662877555959610910619729920,riak_kv_eleveldb_backend,{state,<<>>,"/mnt/datadrive/riak/data/leveldb/114179815416476790484662877555959610910619729920",[{create_if_missing,true},{max_open_files,128},{use_bloomfilter,true},{write_buffer_size,58858594}],[{add_paths,[]},{allow_strfun,false},{anti_entropy,{on,[]}},{anti_entropy_build_limit,{1,3600000}},{anti_entropy_concurrency,2},{anti_entropy_data_dir,"/mnt/datadrive/riak/data/anti_entropy"},{anti_entropy_expire,604800000},{anti_entropy_leveldb_opts,[{write_buffer_size,4194304},{max_open_files,20}]},{anti_entropy_tick,15000},{create_if_missing,true},{data_root,"/mnt/datadrive/riak/data/leveldb"},{fsm_limit,50000},{hook_js_vm_count,2},{http_url_encoding,on},{included_applications,[]},{js_max_vm_mem,8},{js_thread_stack,16},{legacy_stats,true},{listkeys_backpressure,true},{map_js_vm_count,8},{mapred_2i_pipe,true},{mapred_name,"mapred"},{max_open_files,128},{object_format,v1},{reduce_js_vm_count,6},{stats_urlpath,"stats"},{storage_backend,riak_kv_eleveldb_backend},{use_bloomfilter,true},{vnode_vclocks,true},{write_buffer_size,58858594}],[],[],[{fill_cache,false}],true,false},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},undefined,3000,1000,100,100,true,true,undefined}},riak@riak003,none,undefined,undefined,undefined,{pool,riak_kv_worker,10,[]},undefined,107615} >>>>>>>> > ** Reason for termination = >>>>>>>> > ** >>>>>>>> > {badarg,[{eleveldb,close,[<<>>],[]},{riak_kv_eleveldb_backend,stop,1,[{file,"src/riak_kv_eleveldb_backend.erl"},{line,149}]},{riak_kv_vnode,terminate,2,[{file,"src/riak_kv_vnode.erl"},{line,836}]},{riak_core_vnode,terminate,3,[{file,"src/riak_core_vnode.erl"},{line,847}]},{gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,586}]},{proc_lib,init_p_do_apply,3,[{file," >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
