The real Riak developers have suggested this might be your problem with stats being stuck:
https://github.com/basho/riak_core/pull/467 The fix is included in the upcoming 1.4.4 maintenance release (which is overdue so I am not going to bother guessing when it will actually arrive). Matthew On Dec 11, 2013, at 2:47 PM, Simon Effenberg <seffenb...@team.mobile.de> wrote: > I will do.. > > but one other thing: > > watch Every 10.0s: sudo riak-admin status | grep put_fsm > node_put_fsm_time_mean : 2208050 > node_put_fsm_time_median : 39231 > node_put_fsm_time_95 : 17400382 > node_put_fsm_time_99 : 50965752 > node_put_fsm_time_100 : 59537762 > node_put_fsm_active : 5 > node_put_fsm_active_60s : 364 > node_put_fsm_in_rate : 5 > node_put_fsm_out_rate : 3 > node_put_fsm_rejected : 0 > node_put_fsm_rejected_60s : 0 > node_put_fsm_rejected_total : 0 > > this is not changing at all.. so maybe my expectations are _wrong_?! So > I will start searching around if there is a "status" bug or I'm > looking in the wrong place... maybe there is no problem while searching > for one?! But I see that at least the app has some issues on GET and > PUT (more on PUT).. so I would like to know how fast the things are.. > but "status" isn't working.. aaaaargh... > > Cheers > Simon > > > On Wed, 11 Dec 2013 14:32:07 -0500 > Matthew Von-Maszewski <matth...@basho.com> wrote: > >> An additional thought: if increasing max_open_files does NOT help, try >> removing +S 4:4 from the vm.args. Typically +S setting helps leveldb, but >> one other user mentioned that the new sorted 2i queries needed more CPU in >> the Erlang layer. >> >> Summary: >> - try increasing max_open_files to 170 >> - helps: try setting sst_block_size to 32768 in app.config >> - does not help: try removing +S from vm.args >> >> Matthew >> >> On Dec 11, 2013, at 1:58 PM, Simon Effenberg <seffenb...@team.mobile.de> >> wrote: >> >>> Hi Matthew, >>> >>> On Wed, 11 Dec 2013 18:38:49 +0100 >>> Matthew Von-Maszewski <matth...@basho.com> wrote: >>> >>>> Simon, >>>> >>>> I have plugged your various values into the attached spreadsheet. I >>>> assumed a vnode count to allow for one of your twelve servers to die (256 >>>> ring size / 11 servers). >>> >>> Great, thanks! >>> >>>> >>>> The spreadsheet suggests you can safely raise your max_open_files from 100 >>>> to 170. I would suggest doing this for the next server you upgrade. If >>>> part of your problem is file cache thrashing, you should see an >>>> improvement. >>> >>> I will try this out.. starting the next server in 3-4 hours. >>> >>>> >>>> Only if max_open_files helps, you should then consider adding >>>> {sst_block_size, 32767} to the eleveldb portion of app.config. This >>>> setting, given your value sizes, would likely half the size of the >>>> metadata held in the file cache. It only impacts the files newly >>>> compacted in the upgrade, and would gradually increase space in the file >>>> cache while slowing down the file cache thrashing. >>> >>> So I'll do this at the over-next server if the next server is fine. >>> >>>> >>>> What build/packaging of Riak do you use, or do you build from source? >>> >>> Using the debian packages from the basho site.. >>> >>> I'm really wondering why the "put" performance is that bad. >>> Here are the changes which were introduced/changed only on the new >>> upgraded servers: >>> >>> >>> + fsm_limit => 50000, >>> --- our '+P' is set to 262144 so more than 3x fsm_limit which was >>> --- stated somewhere >>> + # after finishing the upgrade this should be switched to v1 !!! >>> + object_format => '__atom_v0', >>> >>> - '-env ERL_MAX_ETS_TABLES' => 8192, >>> + '-env ERL_MAX_ETS_TABLES' => 256000, # old package used 8192 >>> but 1.4.2 raised it to this high number >>> + '-env ERL_MAX_PORTS' => 64000, >>> + # Treat error_logger warnings as warnings >>> + '+W' => 'w', >>> + # Tweak GC to run more often >>> + '-env ERL_FULLSWEEP_AFTER' => 0, >>> + # Force the erlang VM to use SMP >>> + '-smp' => 'enable', >>> + ################################# >>> >>> Cheers >>> Simon >>> >>> >>>> >>>> Matthew >>>> >>>> >>>> >>>> On Dec 11, 2013, at 9:48 AM, Simon Effenberg <seffenb...@team.mobile.de> >>>> wrote: >>>> >>>>> Hi Matthew, >>>>> >>>>> thanks for all your time and work.. see inline for answers.. >>>>> >>>>> On Wed, 11 Dec 2013 09:17:32 -0500 >>>>> Matthew Von-Maszewski <matth...@basho.com> wrote: >>>>> >>>>>> The real Riak developers have arrived on-line for the day. They are >>>>>> telling me that all of your problems are likely due to the extended >>>>>> upgrade times, and yes there is a known issue with handoff between 1.3 >>>>>> and 1.4. They also say everything should calm down after all nodes are >>>>>> upgraded. >>>>>> >>>>>> I will review your system settings now and see if there is something >>>>>> that might make the other machines upgrade quicker. So three more >>>>>> questions: >>>>>> >>>>>> - what is the average size of your keys >>>>> >>>>> bucket names are between 5 and 15 characters (only ~ 10 buckets).. >>>>> key names are normally something like 26iesj:hovh7egz >>>>> >>>>>> >>>>>> - what is the average size of your value (data stored) >>>>> >>>>> I have to guess.. but mean is (from Riak) 12kb but 95th percentile is >>>>> at 75kb and in theory we have a limit of 1MB (then it will be split up) >>>>> but sometimes thanks to sibblings (we have to buckets with allow_mult) >>>>> we have also some 7MB in MAX but this will be reduced again (it's a new >>>>> feature in our app which has to many parallel wrights below of 15ms). >>>>> >>>>>> >>>>>> - in regular use, are your keys accessed randomly across their entire >>>>>> range, or do they contain a date component which clusters older, less >>>>>> used keys >>>>> >>>>> normally we don't search but retrieve keys by key name.. and we have >>>>> data which is up to 6 months old and normally we access mostly >>>>> new/active/hot data and not all the old ones.. besides this we have a >>>>> job doing a 2i query every 5mins and another one doing this maybe once >>>>> an hour.. both don't work while the upgrade is ongoing (2i isn't >>>>> working). >>>>> >>>>> Cheers >>>>> Simon >>>>> >>>>>> >>>>>> Matthew >>>>>> >>>>>> >>>>>> On Dec 11, 2013, at 8:43 AM, Simon Effenberg <seffenb...@team.mobile.de> >>>>>> wrote: >>>>>> >>>>>>> Oh and at the moment they are waiting for some handoffs and I see >>>>>>> errors in logfiles: >>>>>>> >>>>>>> >>>>>>> 2013-12-11 13:41:47.948 UTC [error] >>>>>>> <0.7157.24>@riak_core_handoff_sender:start_fold:269 hinted_handoff >>>>>>> transfer of riak_kv_vnode from 'riak@10.46.109.202' >>>>>>> 468137243207554840987117797979434404733540892672 >>>>>>> >>>>>>> but I remember that somebody else had this as well and if I recall >>>>>>> correctly it disappeared after the full upgrade was done.. but at the >>>>>>> moment it's hard to think about upgrading everything at once.. >>>>>>> (~12hours 100% disk utilization on all 12 nodes will lead to real slow >>>>>>> puts/gets) >>>>>>> >>>>>>> What can I do? >>>>>>> >>>>>>> Cheers >>>>>>> Simon >>>>>>> >>>>>>> PS: transfers output: >>>>>>> >>>>>>> 'riak@10.46.109.202' waiting to handoff 17 partitions >>>>>>> 'riak@10.46.109.201' waiting to handoff 19 partitions >>>>>>> >>>>>>> (these are the 1.4.2 nodes) >>>>>>> >>>>>>> >>>>>>> On Wed, 11 Dec 2013 14:39:58 +0100 >>>>>>> Simon Effenberg <seffenb...@team.mobile.de> wrote: >>>>>>> >>>>>>>> Also some side notes: >>>>>>>> >>>>>>>> "top" is even better on new 1.4.2 than on 1.3.1 machines.. IO >>>>>>>> utilization of disk is mostly the same (round about 33%).. >>>>>>>> >>>>>>>> but >>>>>>>> >>>>>>>> 95th percentile of response time for get (avg over all nodes): >>>>>>>> before upgrade: 29ms >>>>>>>> after upgrade: almost the same >>>>>>>> >>>>>>>> 95th percentile of response time for put (avg over all nodes): >>>>>>>> before upgrade: 60ms >>>>>>>> after upgrade: 1548ms >>>>>>>> but this is only because of 2 of 12 nodes are >>>>>>>> on 1.4.2 and are really slow (17000ms) >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Simon >>>>>>>> >>>>>>>> On Wed, 11 Dec 2013 13:45:56 +0100 >>>>>>>> Simon Effenberg <seffenb...@team.mobile.de> wrote: >>>>>>>> >>>>>>>>> Sorry I forgot the half of it.. >>>>>>>>> >>>>>>>>> seffenberg@kriak46-1:~$ free -m >>>>>>>>> total used free shared buffers cached >>>>>>>>> Mem: 23999 23759 239 0 184 >>>>>>>>> 16183 >>>>>>>>> -/+ buffers/cache: 7391 16607 >>>>>>>>> Swap: 0 0 0 >>>>>>>>> >>>>>>>>> We have 12 servers.. >>>>>>>>> datadir on the compacted servers (1.4.2) ~ 765 GB >>>>>>>>> >>>>>>>>> AAE is enabled. >>>>>>>>> >>>>>>>>> I attached app.config and vm.args. >>>>>>>>> >>>>>>>>> Cheers >>>>>>>>> Simon >>>>>>>>> >>>>>>>>> On Wed, 11 Dec 2013 07:33:31 -0500 >>>>>>>>> Matthew Von-Maszewski <matth...@basho.com> wrote: >>>>>>>>> >>>>>>>>>> Ok, I am now suspecting that your servers are either using swap >>>>>>>>>> space (which is slow) or your leveldb file cache is thrashing >>>>>>>>>> (opening and closing multiple files per request). >>>>>>>>>> >>>>>>>>>> How many servers do you have and do you use Riak's active >>>>>>>>>> anti-entropy feature? I am going to plug all of this into a >>>>>>>>>> spreadsheet. >>>>>>>>>> >>>>>>>>>> Matthew Von-Maszewski >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Dec 11, 2013, at 7:09, Simon Effenberg >>>>>>>>>> <seffenb...@team.mobile.de> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Matthew >>>>>>>>>>> >>>>>>>>>>> Memory: 23999 MB >>>>>>>>>>> >>>>>>>>>>> ring_creation_size, 256 >>>>>>>>>>> max_open_files, 100 >>>>>>>>>>> >>>>>>>>>>> riak-admin status: >>>>>>>>>>> >>>>>>>>>>> memory_total : 276001360 >>>>>>>>>>> memory_processes : 191506322 >>>>>>>>>>> memory_processes_used : 191439568 >>>>>>>>>>> memory_system : 84495038 >>>>>>>>>>> memory_atom : 686993 >>>>>>>>>>> memory_atom_used : 686560 >>>>>>>>>>> memory_binary : 21965352 >>>>>>>>>>> memory_code : 11332732 >>>>>>>>>>> memory_ets : 10823528 >>>>>>>>>>> >>>>>>>>>>> Thanks for looking! >>>>>>>>>>> >>>>>>>>>>> Cheers >>>>>>>>>>> Simon >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, 11 Dec 2013 06:44:42 -0500 >>>>>>>>>>> Matthew Von-Maszewski <matth...@basho.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> I need to ask other developers as they arrive for the new day. >>>>>>>>>>>> Does not make sense to me. >>>>>>>>>>>> >>>>>>>>>>>> How many nodes do you have? How much RAM do you have in each >>>>>>>>>>>> node? What are your settings for max_open_files and cache_size in >>>>>>>>>>>> the app.config file? Maybe this is as simple as leveldb using too >>>>>>>>>>>> much RAM in 1.4. The memory accounting for maz_open_files changed >>>>>>>>>>>> in 1.4. >>>>>>>>>>>> >>>>>>>>>>>> Matthew Von-Maszewski >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Dec 11, 2013, at 6:28, Simon Effenberg >>>>>>>>>>>> <seffenb...@team.mobile.de> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Matthew, >>>>>>>>>>>>> >>>>>>>>>>>>> it took around 11hours for the first node to finish the >>>>>>>>>>>>> compaction. The >>>>>>>>>>>>> second node is running already 12 hours and is still doing >>>>>>>>>>>>> compaction. >>>>>>>>>>>>> >>>>>>>>>>>>> Besides that I wonder because the fsm_put time on the new 1.4.2 >>>>>>>>>>>>> host is >>>>>>>>>>>>> much higher (after the compaction) than on an old 1.3.1 (both are >>>>>>>>>>>>> running in the cluster right now and another one is doing the >>>>>>>>>>>>> compaction/upgrade while it is in the cluster but not directly >>>>>>>>>>>>> accessible because it is out of the Loadbalancer): >>>>>>>>>>>>> >>>>>>>>>>>>> 1.4.2: >>>>>>>>>>>>> >>>>>>>>>>>>> node_put_fsm_time_mean : 2208050 >>>>>>>>>>>>> node_put_fsm_time_median : 39231 >>>>>>>>>>>>> node_put_fsm_time_95 : 17400382 >>>>>>>>>>>>> node_put_fsm_time_99 : 50965752 >>>>>>>>>>>>> node_put_fsm_time_100 : 59537762 >>>>>>>>>>>>> node_put_fsm_active : 5 >>>>>>>>>>>>> node_put_fsm_active_60s : 364 >>>>>>>>>>>>> node_put_fsm_in_rate : 5 >>>>>>>>>>>>> node_put_fsm_out_rate : 3 >>>>>>>>>>>>> node_put_fsm_rejected : 0 >>>>>>>>>>>>> node_put_fsm_rejected_60s : 0 >>>>>>>>>>>>> node_put_fsm_rejected_total : 0 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 1.3.1: >>>>>>>>>>>>> >>>>>>>>>>>>> node_put_fsm_time_mean : 5036 >>>>>>>>>>>>> node_put_fsm_time_median : 1614 >>>>>>>>>>>>> node_put_fsm_time_95 : 8789 >>>>>>>>>>>>> node_put_fsm_time_99 : 38258 >>>>>>>>>>>>> node_put_fsm_time_100 : 384372 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> any clue why this could/should be? >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers >>>>>>>>>>>>> Simon >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, 10 Dec 2013 17:21:07 +0100 >>>>>>>>>>>>> Simon Effenberg <seffenb...@team.mobile.de> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Matthew, >>>>>>>>>>>>>> >>>>>>>>>>>>>> thanks!.. that answers my questions! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers >>>>>>>>>>>>>> Simon >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, 10 Dec 2013 11:08:32 -0500 >>>>>>>>>>>>>> Matthew Von-Maszewski <matth...@basho.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2i is not my expertise, so I had to discuss you concerns with >>>>>>>>>>>>>>> another Basho developer. He says: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Between 1.3 and 1.4, the 2i query did change but not the 2i >>>>>>>>>>>>>>> on-disk format. You must wait for all nodes to update if you >>>>>>>>>>>>>>> desire to use the new 2i query. The 2i data will properly >>>>>>>>>>>>>>> write/update on both 1.3 and 1.4 machines during the migration. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Does that answer your question? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> And yes, you might see available disk space increase during the >>>>>>>>>>>>>>> upgrade compactions if your dataset contains numerous delete >>>>>>>>>>>>>>> "tombstones". The Riak 2.0 code includes a new feature called >>>>>>>>>>>>>>> "aggressive delete" for leveldb. This feature is more >>>>>>>>>>>>>>> proactive in pushing delete tombstones through the levels to >>>>>>>>>>>>>>> free up disk space much more quickly (especially if you perform >>>>>>>>>>>>>>> block deletes every now and then). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Matthew >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Dec 10, 2013, at 10:44 AM, Simon Effenberg >>>>>>>>>>>>>>> <seffenb...@team.mobile.de> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Matthew, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> see inline.. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, 10 Dec 2013 10:38:03 -0500 >>>>>>>>>>>>>>>> Matthew Von-Maszewski <matth...@basho.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The sad truth is that you are not the first to see this >>>>>>>>>>>>>>>>> problem. And yes, it has to do with your 950GB per node >>>>>>>>>>>>>>>>> dataset. And no, nothing to do but sit through it at this >>>>>>>>>>>>>>>>> time. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> While I did extensive testing around upgrade times before >>>>>>>>>>>>>>>>> shipping 1.4, apparently there are data configurations I did >>>>>>>>>>>>>>>>> not anticipate. You are likely seeing a cascade where a >>>>>>>>>>>>>>>>> shift of one file from level-1 to level-2 is causing a shift >>>>>>>>>>>>>>>>> of another file from level-2 to level-3, which causes a >>>>>>>>>>>>>>>>> level-3 file to shift to level-4, etc … then the next file >>>>>>>>>>>>>>>>> shifts from level-1. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The bright side of this pain is that you will end up with >>>>>>>>>>>>>>>>> better write throughput once all the compaction ends. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I have to deal with that.. but my problem is now, if I'm doing >>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>> node by node it looks like 2i searches aren't possible while >>>>>>>>>>>>>>>> 1.3 and >>>>>>>>>>>>>>>> 1.4 nodes exists in the cluster. Is there any problem which >>>>>>>>>>>>>>>> leads me to >>>>>>>>>>>>>>>> an 2i repair marathon or could I easily wait for some hours >>>>>>>>>>>>>>>> for each >>>>>>>>>>>>>>>> node until all merges are done before I upgrade the next one? >>>>>>>>>>>>>>>> (2i >>>>>>>>>>>>>>>> searches can fail for some time.. the APP isn't having >>>>>>>>>>>>>>>> problems with >>>>>>>>>>>>>>>> that but are new inserts with 2i indices processed >>>>>>>>>>>>>>>> successfully or do >>>>>>>>>>>>>>>> I have to do the 2i repair?) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> /s >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> one other good think: saving disk space is one advantage ;).. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Riak 2.0's leveldb has code to prevent/reduce compaction >>>>>>>>>>>>>>>>> cascades, but that is not going to help you today. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Matthew >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Dec 10, 2013, at 10:26 AM, Simon Effenberg >>>>>>>>>>>>>>>>> <seffenb...@team.mobile.de> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi @list, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I'm trying to upgrade our Riak cluster from 1.3.1 to 1.4.2 >>>>>>>>>>>>>>>>>> .. after >>>>>>>>>>>>>>>>>> upgrading the first node (out of 12) this node seems to do >>>>>>>>>>>>>>>>>> many merges. >>>>>>>>>>>>>>>>>> the sst_* directories changes in size "rapidly" and the node >>>>>>>>>>>>>>>>>> is having >>>>>>>>>>>>>>>>>> a disk utilization of 100% all the time. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I know that there is something like that: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "The first execution of 1.4.0 leveldb using a 1.3.x or 1.2.x >>>>>>>>>>>>>>>>>> dataset >>>>>>>>>>>>>>>>>> will initiate an automatic conversion that could pause the >>>>>>>>>>>>>>>>>> startup of >>>>>>>>>>>>>>>>>> each node by 3 to 7 minutes. The leveldb data in "level #1" >>>>>>>>>>>>>>>>>> is being >>>>>>>>>>>>>>>>>> adjusted such that "level #1" can operate as an overlapped >>>>>>>>>>>>>>>>>> data level >>>>>>>>>>>>>>>>>> instead of as a sorted data level. The conversion is simply >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> reduction of the number of files in "level #1" to being less >>>>>>>>>>>>>>>>>> than eight >>>>>>>>>>>>>>>>>> via normal compaction of data from "level #1" into "level >>>>>>>>>>>>>>>>>> #2". This is >>>>>>>>>>>>>>>>>> a one time conversion." >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> but it looks much more invasive than explained here or >>>>>>>>>>>>>>>>>> doesn't have to >>>>>>>>>>>>>>>>>> do anything with the (probably seen) merges. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Is this "normal" behavior or could I do anything about it? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> At the moment I'm stucked with the upgrade procedure because >>>>>>>>>>>>>>>>>> this high >>>>>>>>>>>>>>>>>> IO load would probably lead to high response times. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Also we have a lot of data (per node ~950 GB). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Cheers >>>>>>>>>>>>>>>>>> Simon >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> riak-users mailing list >>>>>>>>>>>>>>>>>> riak-users@lists.basho.com >>>>>>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Simon Effenberg | Site Ops Engineer | mobile.international GmbH >>>>>>>>>>>>>>>> Fon: + 49-(0)30-8109 - 7173 >>>>>>>>>>>>>>>> Fax: + 49-(0)30-8109 - 7131 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Mail: seffenb...@team.mobile.de >>>>>>>>>>>>>>>> Web: www.mobile.de >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Geschäftsführer: Malte Krüger >>>>>>>>>>>>>>>> HRB Nr.: 18517 P, Amtsgericht Potsdam >>>>>>>>>>>>>>>> Sitz der Gesellschaft: Kleinmachnow >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Simon Effenberg | Site Ops Engineer | mobile.international GmbH >>>>>>>>>>>>>> Fon: + 49-(0)30-8109 - 7173 >>>>>>>>>>>>>> Fax: + 49-(0)30-8109 - 7131 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Mail: seffenb...@team.mobile.de >>>>>>>>>>>>>> Web: www.mobile.de >>>>>>>>>>>>>> >>>>>>>>>>>>>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Geschäftsführer: Malte Krüger >>>>>>>>>>>>>> HRB Nr.: 18517 P, Amtsgericht Potsdam >>>>>>>>>>>>>> Sitz der Gesellschaft: Kleinmachnow >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> riak-users mailing list >>>>>>>>>>>>>> riak-users@lists.basho.com >>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Simon Effenberg | Site Ops Engineer | mobile.international GmbH >>>>>>>>>>>>> Fon: + 49-(0)30-8109 - 7173 >>>>>>>>>>>>> Fax: + 49-(0)30-8109 - 7131 >>>>>>>>>>>>> >>>>>>>>>>>>> Mail: seffenb...@team.mobile.de >>>>>>>>>>>>> Web: www.mobile.de >>>>>>>>>>>>> >>>>>>>>>>>>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Geschäftsführer: Malte Krüger >>>>>>>>>>>>> HRB Nr.: 18517 P, Amtsgericht Potsdam >>>>>>>>>>>>> Sitz der Gesellschaft: Kleinmachnow >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Simon Effenberg | Site Ops Engineer | mobile.international GmbH >>>>>>>>>>> Fon: + 49-(0)30-8109 - 7173 >>>>>>>>>>> Fax: + 49-(0)30-8109 - 7131 >>>>>>>>>>> >>>>>>>>>>> Mail: seffenb...@team.mobile.de >>>>>>>>>>> Web: www.mobile.de >>>>>>>>>>> >>>>>>>>>>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Geschäftsführer: Malte Krüger >>>>>>>>>>> HRB Nr.: 18517 P, Amtsgericht Potsdam >>>>>>>>>>> Sitz der Gesellschaft: Kleinmachnow >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Simon Effenberg | Site Ops Engineer | mobile.international GmbH >>>>>>>>> Fon: + 49-(0)30-8109 - 7173 >>>>>>>>> Fax: + 49-(0)30-8109 - 7131 >>>>>>>>> >>>>>>>>> Mail: seffenb...@team.mobile.de >>>>>>>>> Web: www.mobile.de >>>>>>>>> >>>>>>>>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany >>>>>>>>> >>>>>>>>> >>>>>>>>> Geschäftsführer: Malte Krüger >>>>>>>>> HRB Nr.: 18517 P, Amtsgericht Potsdam >>>>>>>>> Sitz der Gesellschaft: Kleinmachnow >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Simon Effenberg | Site Ops Engineer | mobile.international GmbH >>>>>>>> Fon: + 49-(0)30-8109 - 7173 >>>>>>>> Fax: + 49-(0)30-8109 - 7131 >>>>>>>> >>>>>>>> Mail: seffenb...@team.mobile.de >>>>>>>> Web: www.mobile.de >>>>>>>> >>>>>>>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany >>>>>>>> >>>>>>>> >>>>>>>> Geschäftsführer: Malte Krüger >>>>>>>> HRB Nr.: 18517 P, Amtsgericht Potsdam >>>>>>>> Sitz der Gesellschaft: Kleinmachnow >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Simon Effenberg | Site Ops Engineer | mobile.international GmbH >>>>>>> Fon: + 49-(0)30-8109 - 7173 >>>>>>> Fax: + 49-(0)30-8109 - 7131 >>>>>>> >>>>>>> Mail: seffenb...@team.mobile.de >>>>>>> Web: www.mobile.de >>>>>>> >>>>>>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany >>>>>>> >>>>>>> >>>>>>> Geschäftsführer: Malte Krüger >>>>>>> HRB Nr.: 18517 P, Amtsgericht Potsdam >>>>>>> Sitz der Gesellschaft: Kleinmachnow >>>>>> >>>>> >>>>> >>>>> -- >>>>> Simon Effenberg | Site Ops Engineer | mobile.international GmbH >>>>> Fon: + 49-(0)30-8109 - 7173 >>>>> Fax: + 49-(0)30-8109 - 7131 >>>>> >>>>> Mail: seffenb...@team.mobile.de >>>>> Web: www.mobile.de >>>>> >>>>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany >>>>> >>>>> >>>>> Geschäftsführer: Malte Krüger >>>>> HRB Nr.: 18517 P, Amtsgericht Potsdam >>>>> Sitz der Gesellschaft: Kleinmachnow >>>> >>> >>> >>> -- >>> Simon Effenberg | Site Ops Engineer | mobile.international GmbH >>> Fon: + 49-(0)30-8109 - 7173 >>> Fax: + 49-(0)30-8109 - 7131 >>> >>> Mail: seffenb...@team.mobile.de >>> Web: www.mobile.de >>> >>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany >>> >>> >>> Geschäftsführer: Malte Krüger >>> HRB Nr.: 18517 P, Amtsgericht Potsdam >>> Sitz der Gesellschaft: Kleinmachnow >> > > > -- > Simon Effenberg | Site Ops Engineer | mobile.international GmbH > Fon: + 49-(0)30-8109 - 7173 > Fax: + 49-(0)30-8109 - 7131 > > Mail: seffenb...@team.mobile.de > Web: www.mobile.de > > Marktplatz 1 | 14532 Europarc Dreilinden | Germany > > > Geschäftsführer: Malte Krüger > HRB Nr.: 18517 P, Amtsgericht Potsdam > Sitz der Gesellschaft: Kleinmachnow _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
