Hi all,
I managed to insert 100 million records.
I tried MapReduce to fetch 1 million records, but it failed with timeout
error.
I can fetch 1 record by using GET method, of course.
But I can't do after this timeout error comes out until rebooting Riak.
Anyone know how I get rid of this?
Data format is like this:
key(UUID as String):
69e5f286-c511-4c8e-bce5-4173674478f4
Data(JSON Object):
{"gettime":"000419","getdate":"20080101","meshcode":"503012","longitude":"783005
4","latitude":"1829547"}
MapReduce code:
{"inputs":"lpb_probes",
"query":[
{"map": {
"language":"javascript",
"source":
"function(value, keyData, arg) {
var data = Riak.mapValuesJson(value)[0];
if (data.getdate && data.getdate >= '20080101') {
if (data.getdate && data.getdate < '20080102') {
return [value.key];
} else {
return [];
}
} else {
return [];
}
}",
"keep":true}
}
],
"timeout":6000000
}
Error log is as follows:
=INFO REPORT==== 2-Aug-2010::20:31:19 ===
Starting handoff of partition riak_kv_vnode
707914855582156101004909840846949587645842325504 to 'r...@192.168.202.2'
=ERROR REPORT==== 2-Aug-2010::20:31:24 ===
** gen_event handler riak_core_ring_handler crashed.
** Was installed in riak_core_ring_events
** Last event was: {ring_update,
{chstate,'r...@192.168.202.1',
[{'r...@192.168.202.1',{1,63447751303}},
{'r...@192.168.202.2',{43,63447751211}},
{'r...@192.168.202.3',{32,63447751205}}],
{64,
[{0,'r...@192.168.202.1'},
{22835963083295358096932575511191922182123945984,
'r...@192.168.202.2'},
{45671926166590716193865151022383844364247891968,
'r...@192.168.202.3'},
{68507889249886074290797726533575766546371837952,
'r...@192.168.202.1'},
{91343852333181432387730302044767688728495783936,
'r...@192.168.202.2'},
{114179815416476790484662877555959610910619729920,
'r...@192.168.202.3'},
{137015778499772148581595453067151533092743675904,
'r...@192.168.202.1'},
{159851741583067506678528028578343455274867621888,
'r...@192.168.202.2'},
{182687704666362864775460604089535377456991567872,
'r...@192.168.202.3'},
{205523667749658222872393179600727299639115513856,
'r...@192.168.202.1'},
{228359630832953580969325755111919221821239459840,
'r...@192.168.202.2'},
{251195593916248939066258330623111144003363405824,
'r...@192.168.202.3'},
{274031556999544297163190906134303066185487351808,
'r...@192.168.202.1'},
{296867520082839655260123481645494988367611297792,
'r...@192.168.202.2'},
{319703483166135013357056057156686910549735243776,
'r...@192.168.202.3'},
{342539446249430371453988632667878832731859189760,
'r...@192.168.202.1'},
{365375409332725729550921208179070754913983135744,
'r...@192.168.202.2'},
{388211372416021087647853783690262677096107081728,
'r...@192.168.202.3'},
{411047335499316445744786359201454599278231027712,
'r...@192.168.202.1'},
{433883298582611803841718934712646521460354973696,
'r...@192.168.202.2'},
{456719261665907161938651510223838443642478919680,
'r...@192.168.202.3'},
{479555224749202520035584085735030365824602865664,
'r...@192.168.202.1'},
{502391187832497878132516661246222288006726811648,
'r...@192.168.202.2'},
{525227150915793236229449236757414210188850757632,
'r...@192.168.202.3'},
{548063113999088594326381812268606132370974703616,
'r...@192.168.202.1'},
{570899077082383952423314387779798054553098649600,
'r...@192.168.202.2'},
{593735040165679310520246963290989976735222595584,
'r...@192.168.202.3'},
{616571003248974668617179538802181898917346541568,
'r...@192.168.202.1'},
{639406966332270026714112114313373821099470487552,
'r...@192.168.202.2'},
{662242929415565384811044689824565743281594433536,
'r...@192.168.202.3'},
{685078892498860742907977265335757665463718379520,
'r...@192.168.202.1'},
{707914855582156101004909840846949587645842325504,
'r...@192.168.202.2'},
{730750818665451459101842416358141509827966271488,
'r...@192.168.202.3'},
{753586781748746817198774991869333432010090217472,
'r...@192.168.202.1'},
{776422744832042175295707567380525354192214163456,
'r...@192.168.202.2'},
{799258707915337533392640142891717276374338109440,
'r...@192.168.202.3'},
{822094670998632891489572718402909198556462055424,
'r...@192.168.202.1'},
{844930634081928249586505293914101120738586001408,
'r...@192.168.202.2'},
{867766597165223607683437869425293042920709947392,
'r...@192.168.202.3'},
{890602560248518965780370444936484965102833893376,
'r...@192.168.202.1'},
{913438523331814323877303020447676887284957839360,
'r...@192.168.202.2'},
{936274486415109681974235595958868809467081785344,
'r...@192.168.202.3'},
{959110449498405040071168171470060731649205731328,
'r...@192.168.202.1'},
{981946412581700398168100746981252653831329677312,
'r...@192.168.202.2'},
{1004782375664995756265033322492444576013453623296,
'r...@192.168.202.3'},
{1027618338748291114361965898003636498195577569280,
'r...@192.168.202.1'},
{1050454301831586472458898473514828420377701515264,
'r...@192.168.202.2'},
{1073290264914881830555831049026020342559825461248,
'r...@192.168.202.3'},
{1096126227998177188652763624537212264741949407232,
'r...@192.168.202.1'},
{1118962191081472546749696200048404186924073353216,
'r...@192.168.202.2'},
{1141798154164767904846628775559596109106197299200,
'r...@192.168.202.3'},
{1164634117248063262943561351070788031288321245184,
'r...@192.168.202.1'},
{1187470080331358621040493926581979953470445191168,
'r...@192.168.202.2'},
{1210306043414653979137426502093171875652569137152,
'r...@192.168.202.3'},
{1233142006497949337234359077604363797834693083136,
'r...@192.168.202.1'},
{1255977969581244695331291653115555720016817029120,
'r...@192.168.202.2'},
{1278813932664540053428224228626747642198940975104,
'r...@192.168.202.3'},
{1301649895747835411525156804137939564381064921088,
'r...@192.168.202.1'},
{1324485858831130769622089379649131486563188867072,
'r...@192.168.202.2'},
{1347321821914426127719021955160323408745312813056,
'r...@192.168.202.3'},
{1370157784997721485815954530671515330927436759040,
'r...@192.168.202.1'},
{1392993748081016843912887106182707253109560705024,
'r...@192.168.202.2'},
{1415829711164312202009819681693899175291684651008,
'r...@192.168.202.3'},
{1438665674247607560106752257205091097473808596992,
'r...@192.168.202.1'}]},
{dict,1,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[[{bucket,<<"lpb_probes">>}|
{meta_entry,
[{n_val,2},
{name,<<"lpb_probes">>},
{allow_mult,false},
{last_write_wins,false},
{precommit,[]},
{postcommit,[]},
{chash_keyfun,{riak_core_util,chash_std_keyfun}},
{linkfun,
{modfun,riak_kv_wm_link_walker,
mapreduce_linkfun}},
{old_vclock,86400},
{young_vclock,20},
{big_vclock,50},
{small_vclock,10},
{r,quorum},
{w,quorum},
{dw,quorum},
{rw,quorum}],
63447751303}]],
[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}}}
** When handler state == {state}
** Reason ==
{timeout,{gen_server,call,[riak_kv_vnode_master,{0,get_vnode}]}}
=INFO REPORT==== 2-Aug-2010::20:31:38 ===
Receiving handoff data for partition
riak_kv_vnode:479555224749202520035584085735030365824602865664
=ERROR REPORT==== 2-Aug-2010::20:31:43 ===
** Generic server <0.325.0> terminating
** Last message in was {tcp,#Port<0.2658>,
[0|<<84,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0>>]}
** When Server state == {state,#Port<0.2658>,undefined,riak_kv_vnode,
undefined,0}
** Reason for termination ==
** {timeout,{gen_server,call,
[riak_kv_vnode_master,
{479555224749202520035584085735030365824602865664,
get_vnode}]}}
=ERROR REPORT==== 2-Aug-2010::20:32:03 ===
** State machine <0.166.0> terminating
** Last event in was {riak_vnode_req_v1,
707914855582156101004909840846949587645842325504,
{server,undefined,{<0.173.0>,#Ref<0.0.0.398>}},
{riak_core_fold_req_v1,
#Fun<riak_core_handoff_sender.1.105285116>,
{#Port<0.2652>,<0.166.0>,riak_kv_vnode,0,0}}}
** When State == active
** Data == {state,707914855582156101004909840846949587645842325504,
riak_kv_vnode,
{state,707914855582156101004909840846949587645842325504,
riak_kv_bitcask_backend,
{#Ref<0.0.0.369>,
"/var/lib/riak/bitcask/707914855582156101004909840846949587645842325504"},
[],true},
4,'r...@192.168.202.2'}
** Reason for termination =
** {{badmatch,{error,closed}},
[{riak_core_handoff_sender,visit_item,3},
{bitcask_fileops,fold_loop,5},
{bitcask,subfold,3},
{riak_kv_vnode,handle_command,3},
{riak_core_vnode,vnode_handoff_command,4},
{gen_fsm,handle_msg,7},
{proc_lib,init_p_do_apply,3}]}
=ERROR REPORT==== 2-Aug-2010::20:33:14 ===
** State machine <0.571.0> terminating
** Last message in was flow_timeout
** When State == executing
** Data == {state,102679823,
[<0.572.0>],
<0.554.0>,66000,
{1280806394047730,#Ref<0.0.0.850>},
#Fun<riak_kv_mapred_json.jsonify_not_found.1>,[]}
** Reason for termination =
** flow_timeout
--
Ken Matsumoto
VP / Research & Development
Nomura Research Institute America, Inc.
NRI Pacific
1400 Fashion Island Blvd., Suite 1010
San Mateo, CA 94404, U.S.A.
PLEASE READ：This e-mail is confidential and intended for the named
recipient only. If you are not an intended recipient, please notify the
sender and delete this e-mail.
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
