Hey David.
Does the below log output look like it could be caused by the issue you
fixed?
Alex
==== Fri Jul 30 14:22:34 CDT 2010
=ERROR REPORT==== 30-Jul-2010::14:22:34 ===
** State machine <0.176.0> terminating
** Last event in was {riak_vnode_req_v1,
593735040165679310520246963290989976735222595584,
{fsm,undefined,<0.12466.0>},
{riak_kv_put_req_v1,
{<<"test.groups">>,<<"EghzXywWrGGtp2fCcSLoatIdjML">>},
{r_object,<<"test.groups">>,
<<"EghzXywWrGGtp2fCcSLoatIdjML">>,
[{r_content,
{dict,5,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],
[[<<"Links">>]],
[],[],[],[],[],[],[],
[[<<"content-type">>,97,112,112,108,105,99,97,
116,105,111,110,47,106,115,111,110],
[<<"X-Riak-VTag">>,89,69,78,55,55,111,66,121,73,
69,78,53,122,101,85,105,117,68,89,80,52]],
[],[],
[[<<"X-Riak-Last-Modified">>|
{1280,517754,951062}]],
[],
[[<<"X-Riak-Meta">>]]}}},
<<"{\"name\":\"foo\",\"created_at\":\"2010-07-30T19:22:34.947Z\",\"type\":\"group\",\"version\":1}">>}],
[{<<0,55,119,231>>,{1,63447736954}}],
{dict,1,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[[clean|true]],
[]}}},
undefined},
33218311,63447736954,
[{returnbody,true}]}}
** When State == active
** Data == {state,593735040165679310520246963290989976735222595584,
riak_kv_vnode,
{state,593735040165679310520246963290989976735222595584,
riak_kv_bitcask_backend,
{#Ref<0.0.0.611>,
"data/bitcask/593735040165679310520246963290989976735222595584"},
[],false},
undefined,none}
** Reason for termination =
** {{badmatch,{error,emfile}},
[{bitcask_fileops,create_file_loop,3},
{bitcask,put,3},
{riak_kv_bitcask_backend,put,3},
{riak_kv_vnode,perform_put,3},
{riak_kv_vnode,do_put,7},
{riak_kv_vnode,handle_command,3},
{riak_core_vnode,vnode_command,3},
{gen_fsm,handle_msg,7}]}
=ERROR REPORT==== 30-Jul-2010::14:22:35 ===
Failed to open lock file
data/bitcask/593735040165679310520246963290989976735222595584/bitcask.write.lock:
emfile
=ERROR REPORT==== 30-Jul-2010::14:22:35 ===
** State machine <0.12471.0> terminating
** Last event in was {riak_vnode_req_v1,
593735040165679310520246963290989976735222595584,
{fsm,undefined,<0.12470.0>},
{riak_kv_put_req_v1,
{<<"test.users">>,<<"ZrAxzFghd51VG902GuCcJ2gYOMJ">>},
{r_object,<<"test.users">>,
<<"ZrAxzFghd51VG902GuCcJ2gYOMJ">>,
[{r_content,
{dict,5,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],
[[<<"Links">>,
{{<<"test.groups">>,
<<"EghzXywWrGGtp2fCcSLoatIdjML">>},
<<"groups">>}]],
[],[],[],[],[],[],[],
[[<<"content-type">>,97,112,112,108,105,99,97,
116,105,111,110,47,106,115,111,110],
[<<"X-Riak-VTag">>,50,88,97,105,85,49,70,49,55,
54,48,71,113,115,75,103,54,102,84,56,118,84]],
[],[],
[[<<"X-Riak-Last-Modified">>|{1280,517755,638}]],
[],
[[<<"X-Riak-Meta">>]]}}},
<<"{\"name\":\"bar\",\"created_at\":\"2010-07-30T19:22:34.998Z\",\"type\":\"user\",\"version\":1}">>}],
[{<<3,30,180,15>>,{1,63447736955}}],
{dict,1,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[[clean|true]],
[]}}},
undefined},
122945524,63447736955,
[{returnbody,true}]}}
** When State == active
** Data == {state,593735040165679310520246963290989976735222595584,
riak_kv_vnode,
{state,593735040165679310520246963290989976735222595584,
riak_kv_bitcask_backend,
{#Ref<0.0.0.16357>,
"data/bitcask/593735040165679310520246963290989976735222595584"},
[],false},
undefined,none}
** Reason for termination =
** {bad_return_value,{error,{write_locked,emfile}}}
==> sasl-error.log <==
=CRASH REPORT==== 30-Jul-2010::14:22:35 ===
crasher:
initial call: riak_core_vnode:init/1
pid: <0.176.0>
registered_name: []
exception exit: {{badmatch,{error,emfile}},
[{bitcask_fileops,create_file_loop,3},
{bitcask,put,3},
{riak_kv_bitcask_backend,put,3},
{riak_kv_vnode,perform_put,3},
{riak_kv_vnode,do_put,7},
{riak_kv_vnode,handle_command,3},
{riak_core_vnode,vnode_command,3},
{gen_fsm,handle_msg,7}]}
in function gen_fsm:terminate/7
ancestors: [riak_core_vnode_sup,riak_core_sup,<0.98.0>]
messages: []
links: [#Port<0.3507>,<0.100.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 1597
stack_size: 24
reductions: 11185
neighbours:
=SUPERVISOR REPORT==== 30-Jul-2010::14:22:35 ===
Supervisor: {local,riak_core_vnode_sup}
Context: child_terminated
Reason: {{badmatch,{error,emfile}},
[{bitcask_fileops,create_file_loop,3},
{bitcask,put,3},
{riak_kv_bitcask_backend,put,3},
{riak_kv_vnode,perform_put,3},
{riak_kv_vnode,do_put,7},
{riak_kv_vnode,handle_command,3},
{riak_core_vnode,vnode_command,3},
{gen_fsm,handle_msg,7}]}
Offender: [{pid,<0.176.0>},
{name,undefined},
{mfa,
{riak_core_vnode,start_link,
[riak_kv_vnode,
593735040165679310520246963290989976735222595584]}},
{restart_type,temporary},
{shutdown,brutal_kill},
{child_type,worker}]
=CRASH REPORT==== 30-Jul-2010::14:22:35 ===
crasher:
initial call: riak_core_vnode:init/1
pid: <0.12471.0>
registered_name: []
exception exit: {bad_return_value,{error,{write_locked,emfile}}}
in function gen_fsm:terminate/7
ancestors: [riak_core_vnode_sup,riak_core_sup,<0.98.0>]
messages: []
links: [<0.100.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 2584
stack_size: 24
reductions: 1955
neighbours:
=SUPERVISOR REPORT==== 30-Jul-2010::14:22:35 ===
Supervisor: {local,riak_core_vnode_sup}
Context: child_terminated
Reason: {bad_return_value,{error,{write_locked,emfile}}}
Offender: [{pid,<0.12471.0>},
{name,undefined},
{mfa,
{riak_core_vnode,start_link,
[riak_kv_vnode,
593735040165679310520246963290989976735222595584]}},
{restart_type,temporary},
{shutdown,brutal_kill},
{child_type,worker}]
==> erlang.log.4 <==
=ERROR REPORT==== 30-Jul-2010::14:23:27 ===
** Generic server memsup terminating
** Last message in was {'EXIT',<0.21020.78>,
{emfile,
[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}}
** When Server state == {state,{unix,darwin},
false,
{1897500000,7776508000},
{<0.81.0>,972008},
false,60000,30000,0.8,0.05,<0.21020.78>,
#Ref<0.0.2.58535>,undefined,
[reg],
[]}
** Reason for termination ==
** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
2>&1"},[stream]]},
{os,start_port_srv_loop,2}]}
=ERROR REPORT==== 30-Jul-2010::14:23:27 ===
** Generic server memsup terminating
** Last message in was {'EXIT',<0.21186.78>,
{emfile,
[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}}
** When Server state == {state,{unix,darwin},
false,undefined,undefined,false,60000,30000,
0.8,0.05,<0.21186.78>,#Ref<0.0.2.58551>,
undefined,
[reg],
[]}
** Reason for termination ==
** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
2>&1"},[stream]]},
{os,start_port_srv_loop,2}]}
=ERROR REPORT==== 30-Jul-2010::14:23:27 ===
** Generic server memsup terminating
** Last message in was {'EXIT',<0.21252.78>,
{emfile,
[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}}
** When Server state == {state,{unix,darwin},
false,undefined,undefined,false,60000,30000,
0.8,0.05,<0.21252.78>,#Ref<0.0.2.58559>,
undefined,
[reg],
[]}
** Reason for termination ==
** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
2>&1"},[stream]]},
{os,start_port_srv_loop,2}]}
=ERROR REPORT==== 30-Jul-2010::14:23:27 ===
** Generic server memsup terminating
** Last message in was {'EXIT',<0.21374.78>,
{emfile,
[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}}
** When Server state == {state,{unix,darwin},
false,undefined,undefined,false,60000,30000,
0.8,0.05,<0.21374.78>,#Ref<0.0.2.58569>,
undefined,
[reg],
[]}
** Reason for termination ==
** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
2>&1"},[stream]]},
{os,start_port_srv_loop,2}]}
=ERROR REPORT==== 30-Jul-2010::14:23:27 ===
** Generic server memsup terminating
** Last message in was {'EXIT',<0.21396.78>,
{emfile,
[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}}
** When Server state == {state,{unix,darwin},
false,undefined,undefined,false,60000,30000,
0.8,0.05,<0.21396.78>,#Ref<0.0.2.58575>,
undefined,
[reg],
[]}
** Reason for termination ==
** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
2>&1"},[stream]]},
{os,start_port_srv_loop,2}]}
=ERROR REPORT==== 30-Jul-2010::14:23:27 ===
** State machine <0.175.0> terminating
** Last event in was {riak_vnode_req_v1,
570899077082383952423314387779798054553098649600,
{fsm,undefined,<0.12470.0>},
{riak_kv_put_req_v1,
{<<"test.users">>,<<"ZrAxzFghd51VG902GuCcJ2gYOMJ">>},
{r_object,<<"test.users">>,
<<"ZrAxzFghd51VG902GuCcJ2gYOMJ">>,
[{r_content,
{dict,5,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],
[[<<"Links">>,
{{<<"test.groups">>,
<<"EghzXywWrGGtp2fCcSLoatIdjML">>},
<<"groups">>}]],
[],[],[],[],[],[],[],
[[<<"content-type">>,97,112,112,108,105,99,97,
116,105,111,110,47,106,115,111,110],
[<<"X-Riak-VTag">>,50,88,97,105,85,49,70,49,55,
54,48,71,113,115,75,103,54,102,84,56,118,84]],
[],[],
[[<<"X-Riak-Last-Modified">>|{1280,517755,638}]],
[],
[[<<"X-Riak-Meta">>]]}}},
<<"{\"name\":\"bar\",\"created_at\":\"2010-07-30T19:22:34.998Z\",\"type\":\"user\",\"version\":1}">>}],
[{<<3,30,180,15>>,{1,63447736955}}],
{dict,1,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[[clean|true]],
[]}}},
undefined},
122945524,63447736955,
[{returnbody,true}]}}
** When State == active
** Data == {state,570899077082383952423314387779798054553098649600,
riak_kv_vnode,
{state,570899077082383952423314387779798054553098649600,
riak_kv_bitcask_backend,
{#Ref<0.0.0.598>,
"data/bitcask/570899077082383952423314387779798054553098649600"},
[],false},
undefined,none}
** Reason for termination =
** {{badmatch,{error,emfile}},
[{bitcask_fileops,create_file_loop,3},
{bitcask,put,3},
{riak_kv_bitcask_backend,put,3},
{riak_kv_vnode,perform_put,3},
{riak_kv_vnode,do_put,7},
{riak_kv_vnode,handle_command,3},
{riak_core_vnode,vnode_command,3},
{gen_fsm,handle_msg,7}]}
=INFO REPORT==== 30-Jul-2010::14:23:27 ===
application: os_mon
exited: shutdown
type: permanent
=INFO REPORT==== 30-Jul-2010::14:23:27 ===
Spidermonkey VM host stopping (<0.139.0>)
=INFO REPORT==== 30-Jul-2010::14:23:27 ===
Spidermonkey VM host stopping (<0.137.0>)
=INFO REPORT==== 30-Jul-2010::14:23:27 ===
Spidermonkey VM host stopping (<0.144.0>)
=INFO REPORT==== 30-Jul-2010::14:23:27 ===
Spidermonkey VM host stopping (<0.140.0>)
=INFO REPORT==== 30-Jul-2010::14:23:27 ===
Spidermonkey VM host stopping (<0.138.0>)
=INFO REPORT==== 30-Jul-2010::14:23:27 ===
Spidermonkey VM host stopping (<0.142.0>)
==> sasl-error.log <==
=CRASH REPORT==== 30-Jul-2010::14:23:27 ===
crasher:
initial call: memsup:init/1
pid: <0.52.0>
registered_name: memsup
exception exit: {emfile,[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}
in function gen_server:terminate/6
ancestors: [os_mon_sup,<0.49.0>]
messages: []
links: [<0.50.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 2584
stack_size: 24
reductions: 53031
neighbours:
=SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===
Supervisor: {local,os_mon_sup}
Context: child_terminated
Reason: {emfile,[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}
Offender: [{pid,<0.52.0>},
{name,memsup},
{mfa,{memsup,start_link,[]}},
{restart_type,permanent},
{shutdown,2000},
{child_type,worker}]
=CRASH REPORT==== 30-Jul-2010::14:23:27 ===
crasher:
initial call: memsup:init/1
pid: <0.21163.78>
registered_name: memsup
exception exit: {emfile,[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}
in function gen_server:terminate/6
ancestors: [os_mon_sup,<0.49.0>]
messages: []
links: [<0.50.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 377
stack_size: 24
reductions: 181
neighbours:
=SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===
Supervisor: {local,os_mon_sup}
Context: child_terminated
Reason: {emfile,[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}
Offender: [{pid,<0.21163.78>},
{name,memsup},
{mfa,{memsup,start_link,[]}},
{restart_type,permanent},
{shutdown,2000},
{child_type,worker}]
=CRASH REPORT==== 30-Jul-2010::14:23:27 ===
crasher:
initial call: memsup:init/1
pid: <0.21226.78>
registered_name: memsup
exception exit: {emfile,[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}
in function gen_server:terminate/6
ancestors: [os_mon_sup,<0.49.0>]
messages: []
links: [<0.50.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 377
stack_size: 24
reductions: 181
neighbours:
=SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===
Supervisor: {local,os_mon_sup}
Context: child_terminated
Reason: {emfile,[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}
Offender: [{pid,<0.21226.78>},
{name,memsup},
{mfa,{memsup,start_link,[]}},
{restart_type,permanent},
{shutdown,2000},
{child_type,worker}]
=CRASH REPORT==== 30-Jul-2010::14:23:27 ===
crasher:
initial call: memsup:init/1
pid: <0.21343.78>
registered_name: memsup
exception exit: {emfile,[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}
in function gen_server:terminate/6
ancestors: [os_mon_sup,<0.49.0>]
messages: []
links: [<0.50.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 377
stack_size: 24
reductions: 193
neighbours:
=SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===
Supervisor: {local,os_mon_sup}
Context: child_terminated
Reason: {emfile,[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}
Offender: [{pid,<0.21343.78>},
{name,memsup},
{mfa,{memsup,start_link,[]}},
{restart_type,permanent},
{shutdown,2000},
{child_type,worker}]
=CRASH REPORT==== 30-Jul-2010::14:23:27 ===
crasher:
initial call: memsup:init/1
pid: <0.21393.78>
registered_name: memsup
exception exit: {emfile,[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}
in function gen_server:terminate/6
ancestors: [os_mon_sup,<0.49.0>]
messages: []
links: [<0.50.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 377
stack_size: 24
reductions: 189
neighbours:
=SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===
Supervisor: {local,os_mon_sup}
Context: child_terminated
Reason: {emfile,[{erlang,open_port,
[{spawn,"/bin/sh -s unix:cmd 2>&1"},
[stream]]},
{os,start_port_srv_loop,2}]}
Offender: [{pid,<0.21393.78>},
{name,memsup},
{mfa,{memsup,start_link,[]}},
{restart_type,permanent},
{shutdown,2000},
{child_type,worker}]
=SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===
Supervisor: {local,os_mon_sup}
Context: shutdown
Reason: reached_max_restart_intensity
Offender: [{pid,<0.21393.78>},
{name,memsup},
{mfa,{memsup,start_link,[]}},
{restart_type,permanent},
{shutdown,2000},
{child_type,worker}]
=CRASH REPORT==== 30-Jul-2010::14:23:27 ===
crasher:
initial call: riak_core_vnode:init/1
pid: <0.175.0>
registered_name: []
exception exit: {{badmatch,{error,emfile}},
[{bitcask_fileops,create_file_loop,3},
{bitcask,put,3},
{riak_kv_bitcask_backend,put,3},
{riak_kv_vnode,perform_put,3},
{riak_kv_vnode,do_put,7},
{riak_kv_vnode,handle_command,3},
{riak_core_vnode,vnode_command,3},
{gen_fsm,handle_msg,7}]}
in function gen_fsm:terminate/7
ancestors: [riak_core_vnode_sup,riak_core_sup,<0.98.0>]
messages: []
links: [#Port<0.3521>,<0.100.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 4181
stack_size: 24
reductions: 221476677
neighbours:
=SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===
Supervisor: {local,riak_core_vnode_sup}
Context: child_terminated
Reason: {{badmatch,{error,emfile}},
[{bitcask_fileops,create_file_loop,3},
{bitcask,put,3},
{riak_kv_bitcask_backend,put,3},
{riak_kv_vnode,perform_put,3},
{riak_kv_vnode,do_put,7},
{riak_kv_vnode,handle_command,3},
{riak_core_vnode,vnode_command,3},
{gen_fsm,handle_msg,7}]}
Offender: [{pid,<0.175.0>},
{name,undefined},
{mfa,
{riak_core_vnode,start_link,
[riak_kv_vnode,
570899077082383952423314387779798054553098649600]}},
{restart_type,temporary},
{shutdown,brutal_kill},
{child_type,worker}]
==> erlang.log.4 <==
=INFO REPORT==== 30-Jul-2010::14:23:27 ===
Spidermonkey VM host stopping (<0.143.0>)
=INFO REPORT==== 30-Jul-2010::14:23:27 ===
Spidermonkey VM host stopping (<0.141.0>)
{"Kernel pid
terminated",application_controller,"{application_terminated,os_mon,shutdown}"}
Crash dump was written to: erl_crash.dump
Kernel pid terminated (application_controller)
({application_terminated,os_mon,shutdown})
On Jul 30, 2010, at 4:10 PM, David Smith wrote:
Given the amount of data Ken has inserted, I'd say there's a 90%
chance this problem is related to a bug I fixed earlier this week:
http://bitbucket.org/basho/bitcask/changeset/6a74d3aac4fb
But without more information, it's hard to say. I presume, Ken, you
are also seeing a vnode crash error before this happens and I'm
betting it's emfile related.
D.
On Fri, Jul 30, 2010 at 3:06 PM, Alexander Sicular <sicul...@gmail.com
<mailto:sicul...@gmail.com>> wrote:
This may be another max file handler type of error. Or maybe even
an oom thing if the key length is large.
On Jul 30, 2010, at 4:59 PM, Grant Schofield wrote:
> I am not sure if you hit an already fixed bug in Bitcask or not.
What version of Riak are you running on currently?
>
> Grant Schofield
> Developer Advocate
> Basho Technologies
>
> On Jul 30, 2010, at 1:28 PM, Ken Matsumoto wrote:
>
>> Hi all,
>>
>> I just tried to insert 1Billion data records.
>> But I got the "write_lock" error after 12Million data.
>> What is the reason and how should I avoid this?
>> I use bitcask (default) backend and no parameters changed in
config file.
>> 1 record is just 70B text data.
>>
>> Regards,
>>
>> Ken.
>>
>> --
>> Ken Matsumoto
>> VP / Research & Development
>> Nomura Research Institute America, Inc.
>> NRI Pacific
>> 1400 Fashion Island Blvd., Suite 1010
>> San Mateo, CA 94404, U.S.A.
>>
>> PLEASE READ：This e-mail is confidential and intended for the
named recipient only. If you are not an intended recipient, please
notify the sender and delete this e-mail.
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com <mailto:riak-users@lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
