errors after rolling upgrade to 1.3
Hi I'm upgrading nodes from 1.2.1 to 1.3 and also changing the backend to eleveldb to use secondary indexes. I'm following the steps as; 1. on a node i run riak-admin cluster leave 2. after that node shuts down, I upgrade to 1.3 and update config files to eleveldb 3. after cleaning up the directories, I start riak again. 4. I run riak-admin cluster join After joining, the cluster contains 3 nodes on 1.2 with bitcask and 1 with 1.3 eleveldb. Riak search is enabled in all. I'm seeing errors on the new nodes like below and riak search and mapreduce sometimes fails to return results. 2013-03-15 09:45:35.104 [error] <0.4065.0> gen_fsm <0.4065.0> in state wait_for_input terminated with reason: bad function #Fun in riak_kv_w_reduce:reduce/3 line 206 2013-03-15 09:45:35.110 [error] <0.4065.0> CRASH REPORT Process <0.4065.0> with 0 neighbours exited with reason: bad function #Fun in riak_kv_w_reduce:reduce/3 line 206 in gen_fsm:terminate/7 line 611 2013-03-15 09:58:17.883 [error] <0.8505.0> gen_fsm <0.8505.0> in state wait_for_input terminated with reason: bad function #Fun in riak_kv_w_reduce:reduce/3 line 206 2013-03-15 09:58:17.886 [error] <0.361.0>@riak_pipe_vnode:new_worker:768 Pipe worker startup failed:fitting was gone before startup 2013-03-15 09:58:17.890 [error] <0.8505.0> CRASH REPORT Process <0.8505.0> with 0 neighbours exited with reason: bad function #Fun in riak_kv_w_reduce:reduce/3 line 206 in gen_fsm:terminate/7 line 611 Thanks -- Mridul ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
riak cluster suddenly became unresponsive
Hi,
we have a 12 nodes cluster running riak 1.2.1 which went live a week
ago. Yesterday, suddenly from one minute to another the put_fsm_time_95
and the get_fsm_time_95 raised from something below 100ms up to several
seconds. This went on for about 25 min and than went away.
Checking the riak-logs of the nodes, I find a lot of these:
2013-03-14 17:48:06.388 [info]
<0.62.0>@riak_core_sysmon_handler:handle_event:89 Monitor got
{suppressed,port_events,1}
2013-03-14 17:48:06.889 [info]
<0.62.0>@riak_core_sysmon_handler:handle_event:85 monitor busy_dist_port
<0.7156.1>
[{initial_call,{riak_core_vnode,init,1}},{almost_current_function,{erlang,bif_return_trap,1}},{message_queue_len,1}]
{#Port<0.9083226>,'riak@172.22.3.22'}
This messages are logged all day, but only once every few minutes but in
the problematic time frame between 17:45 and 18:17 it gets logged
several times every second. The node ip differs though, but it seems
only three nodes were involved.
Except of these three nodes the cpu utilisation drops by half during
this on all other nodes. On the three nodes there's only a slight drop.
We are using leveldb as storage backend. I also checked some of the LOG
files of leveldb and there are compactions logged, but these are logged
all the day every few hours.
In this time our software was quite unresponsive too so I would like to
know what was causing this and what I might do to stop. Any ideas, hints?
I found this:
https://groups.google.com/forum/?fromgroups=#!topic/nosql-databases/GqbaeiKCSYE
where Jon Meredith suggests to raise the buffer size to get rid of the
busy buffers by adding +zdbbl 16384 to the vm.args file. Might this help?
Regards,
Ingo
--
Software Architect
Blue Lion mobile GmbH
Tel. +49 (0) 221 788 797 14
Fax. +49 (0) 221 788 797 19
Mob. +49 (0) 176 24 87 30 89
ingo.roc...@bluelionmobile.com
>>> qeep: Hefferwolf
www.bluelionmobile.com
www.qeep.net
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: errors after rolling upgrade to 1.3
Looks like you're running a JS map-reduce job. Scanning the code of riak_kv_w_reduce, it looks like the culprit could be line 270, … something in the middle of running a JS map/reduce job with an Erlang Fun object which is incompatible across releases. It looks like with the present 1.3.0 codebase, JS M/R will likely not work until all the nodes have been upgraded.Some background here: http://www.javalimit.com/2010/05/passing-funs-to-other-erlang-nodes.html The fix going forward should be to have js_runner/1 return a symbolic (external) fun object. But that may not be the answer you're looking for. Kresten On Mar 15, 2013, at 11:03 AM, Mridul Kashatria mailto:mri...@readwhere.com>> wrote: 2013-03-15 09:45:35.104 [error] <0.4065.0> gen_fsm <0.4065.0> in state wait_for_input terminated with reason: bad function #Fun in riak_kv_w_reduce:reduce/3 line 206 ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Unable to delete using Java Client
Hello,
I am trying to delete items using the Java client, but for some reason, the
data is still there when I try to get it out later.
I have posted the relevant parts of the Java Class performing the deletion
here: https://gist.github.com/joachimhs/5171629
The following unit test fails on the last assertion:
@Test
public void testTreeMenu() throws InterruptedException {
newEnv.getTreeMenuDao().persistTreeMenu(new
BasicStatistics("EurekaJAgent:Memory:Heap:Used %", "Account Name", "Y"));
Statistics statOne =
newEnv.getTreeMenuDao().getTreeMenu("EurekaJAgent:Memory:Heap:Used %", "Account
Name");
Assert.assertNotNull(statOne);
Assert.assertEquals("EurekaJAgent:Memory:Heap:Used %",
statOne.getGuiPath());
Assert.assertEquals("Account Name", statOne.getAccountName());
Assert.assertEquals("Y", statOne.getNodeLive());
newEnv.getTreeMenuDao().deleteTreeMenu("EurekaJAgent:Memory:Heap:Used
%", "Account Name");
Thread.sleep(550);
Statistics deletedStatOne =
newEnv.getTreeMenuDao().getTreeMenu("EurekaJAgent:Memory:Heap:Used %", "Account
Name");
Assert.assertNull(deletedStatOne);
}
Med Vennlig Hilsen | Very Best Regards,
Joachim Haagen Skeie
joac...@haagen-software.no
http://haagen-software.no
+47 4141 5805
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Unable to delete using Java Client
Hi Joachim,
The problem with your code is here:
myBucket.delete(guiPath);
The way the API flow works is just like doing a store and fetch; the
Bucket.delete() method returns a DeleteObject on which you then call
.execute() to actually perform the operation:
myBucket.delete(guiPath).execute();
Thanks,
- Roach
On Fri, Mar 15, 2013 at 11:45 AM, Joachim Haagen Skeie
wrote:
> Hello,
>
> I am trying to delete items using the Java client, but for some reason, the
> data is still there when I try to get it out later.
>
> I have posted the relevant parts of the Java Class performing the deletion
> here: https://gist.github.com/joachimhs/5171629
>
> The following unit test fails on the last assertion:
>
> @Test
> public void testTreeMenu() throws InterruptedException {
> newEnv.getTreeMenuDao().persistTreeMenu(new
> BasicStatistics("EurekaJAgent:Memory:Heap:Used %", "Account Name", "Y"));
>
>
>
> Statistics statOne =
> newEnv.getTreeMenuDao().getTreeMenu("EurekaJAgent:Memory:Heap:Used %",
> "Account Name");
>
>
>
> Assert.assertNotNull(statOne);
> Assert.assertEquals("EurekaJAgent:Memory:Heap:Used %",
> statOne.getGuiPath());
> Assert.assertEquals("Account Name", statOne.getAccountName());
> Assert.assertEquals("Y", statOne.getNodeLive());
>
>
>
> newEnv.getTreeMenuDao().deleteTreeMenu("EurekaJAgent:Memory:Heap:Used
> %", "Account Name");
>
>
>
> Thread.sleep(550);
> Statistics deletedStatOne =
> newEnv.getTreeMenuDao().getTreeMenu("EurekaJAgent:Memory:Heap:Used %",
> "Account Name");
>
>
>
> Assert.assertNull(deletedStatOne);
> }
>
> Med Vennlig Hilsen | Very Best Regards,
>
> Joachim Haagen Skeie
> joac...@haagen-software.no
> http://haagen-software.no
> +47 4141 5805
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Recap for March 6 - 12
Hi all, I just wanted to follow up point #10 with a small update. There seems to be a bigger interest among .NET people in Brisbane, so the Azure meetup group is now pairing up with the Queensland MSDN UG to host the event on the 10th of April instead of the 20th March. Updated details here for those interested: http://www.meetup.com/Brisbane-Azure-User-Group/events/100594152/ Thanks John for the mention. OJ On Wed, Mar 13, 2013 at 11:43 AM, John Daily wrote: > Reminder: early bird pricing for RICON East ends Friday, as does the CFP. > > * ---> http://ricon.io/east.html > > And don't forget: hoodies! > > John > twitter.com/macintux > > Riak Recap for March 6 - 12 > === > > 1) A new release of the Perl library Net::Riak has been released. It > now supports SSL and secondary indexes > > * ---> http://search.cpan.org/~rge/Net-Riak-0.1701/ > > 2) RICON update on the Basho Blog > > * Hoodies! ---> > http://basho.com/update-on-ricon-east-cfp-early-bird-and-talk-announcements/ > > 3) Will Laurance just released version 0.0.4 of Fuji CLI > > * Package ---> https://npmjs.org/package/fuji > * Github ---> https://github.com/wlaurance/fuji > > 4) Mark Phillips gave the inaugural presentation at the Twin Cities Riak > Meetup > > * Slides ---> > https://speakerdeck.com/pharkmillups/riak-your-next-or-current-favorite-database > > 5) Cloud Talk in Bellevue, WA got a Riak Basics talk with Adron Hall + > surprise impromptu Windows Azure talk from Glenn Block + combo how it > all works together w/ VM Depot > > * Adron's slides ---> > http://compositecode.com/coding-community/speaking-presentations-workshops/riak-windows-azure/ > > 6) Gideon de Kok wrote a post about how he's using Riak, what hurdles > he had to overcome in development, and why he thinks it's worth > it. Also, a breakdown of his Akka based Riak client, Raiku > > * ---> > http://lab050.com/blog/2013/3/7/how-we-use-riak-to-model-complexer-data-types > * ---> > http://lab050.com/blog/2013/3/6/the-anatomy-of-raiku-a-akka-based-riak-client > > 7) Andy Gross will be talking about building cloud services on Riak at > the next Silicon Valley Cloud Computing meetup > > * ---> http://www.meetup.com/cloudcomputing/events/101130082/ > > 8) Simon Vans Collina stood in for Chris Molozian at TakeOff 2013 to > talk about how to scale with Riak > > * Video ---> http://www.youtube.com/watch?v=I2im1KWklGs > > 9) A Riak CS provider was merged into Fog thanks to Basho engineer > Christopher Meiklejohn > > * ---> https://github.com/fog/fog/pull/1644 > > 10) CorrugatedIron developer (and friend of Basho) OJ Reeves will be > talking about Riak at Microsoft in Brisbane on March 20 > > * Meetup details ---> > http://www.meetup.com/Brisbane-Azure-User-Group/events/100594152/ > > 11) Software Craftsmanship Kraków will be talking CRDTs in Riak on > March 27 (and if anyone involved is on this list, please feel free to > contact us to let us know how it goes) > > * Meetup details ---> http://www.meetup.com/sc-krk/events/107610172/ > > 12) Ian Lewis wrote a blog post about Riak and the Python Client > > * ---> http://www.ianlewis.org/jp/riak-python > > 13) Miscellaneous Q & A > > * ---> http://www.quora.com/l/ma2KosxU6F > * ---> > http://www.quora.com/Riak/Are-there-any-riak-CLI-tools-for-executing-GET-POST-PUT-and-DELETE-operations-on-keys > * ---> > http://stackoverflow.com/questions/15261400/using-term-boost-in-riak-search-for-boosting-specific-conditions > * ---> > http://stackoverflow.com/questions/15211143/do-we-need-to-disable-swap-for-riak > * ---> > http://stackoverflow.com/questions/15166756/error-while-installing-riak-from-source > * ---> > http://stackoverflow.com/questions/15274070/how-to-append-data-to-a-riak-key-under-a-heavily-distributed-environment > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > -- OJ Reeves +61 431 952 586 http://buffered.io/ ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
