Re: Ownership handoff timed out

2015-10-30 Thread Vladyslav Zakhozhai
Hi, Matthew thank you for information. On Thu, Oct 29, 2015 at 8:19 PM Matthew Von-Maszewski wrote: > I queried Basho’s Client Services team. They tell me the upgrade / > coexist should be no problem. > > Matthew > > On Oct 29, 2015, at 1:38 PM, Vladyslav Zakhozhai < > v.zakhoz...@smartweb.co

Re: Ownership handoff timed out

2015-10-29 Thread Matthew Von-Maszewski
I queried Basho’s Client Services team. They tell me the upgrade / coexist should be no problem. Matthew > On Oct 29, 2015, at 1:38 PM, Vladyslav Zakhozhai > wrote: > > Matthew can you describe the bug more detail? > > My plan was to migrate to eleveldb and only then to migrate to Riak 2.0.

Re: Ownership handoff timed out

2015-10-29 Thread Vladyslav Zakhozhai
Matthew can you describe the bug more detail? My plan was to migrate to eleveldb and only then to migrate to Riak 2.0. It seems that I need to change my plans to migrate to Riak 2.0 first. It is sad. Is it safe to migrate Riak 1.4.12/Riak CS 1.5.0 to Riak 2.0 on production environment? According

Re: Ownership handoff timed out

2015-10-29 Thread Matthew Von-Maszewski
Sad to say your LOG files suggest the same bug as seen elsewhere and fixed by recent changes in the leveldb code. The tougher issue is that the fixes are currently only available for our 2.0 product series. A backport would be non-trivial due to the number of places changed between 1.4 and 2.0

Re: Ownership handoff timed out

2015-10-29 Thread Matthew Von-Maszewski
Hi, There was a known eleveldb bug with handoff receiving that could cause a timeout. But it does not sound like bug fits your symptoms. However, I am willing to verify my diagnosis. I would need you to gather the LOG files from all vnodes on the RECEIVING side (or at least from the vnode th

Re: Ownership handoff timed out

2015-10-29 Thread Vladyslav Zakhozhai
Hi, Average size of objects in Riak - 300 Kb. This objects are images. This data updates very very rearly (there almost no updates). I have GC turned on and works: root@python:~# riak-cs-gc status There is no garbage collection in progress The current garbage collection interval is: 900 The c

Re: Ownership handoff timed out

2015-10-29 Thread Vladyslav Zakhozhai
Hi, I want to make small update. Jon your hint about problems on sender side is correct. As I've already told there problems with available resources on sender nodes. There are no enough available RAM which is a cause of swapiness and load on disks. Restarting of sender nodes helps me (at least te

Re: Ownership handoff timed out

2015-10-27 Thread Vladyslav Zakhozhai
Hi, Jon thank you for the answer. During approval of my mail to this list I've troubleshoot my issue more deep. And yes, your are right. Neither {error, enotconn} nor max_concurrency is my problem. I'm going to migrate my cluster entierly to eleveldb only, i.e. I need to refuse using bitcask. I h

Re: Ownership handoff timed out

2015-10-27 Thread Jon Meredith
Hi, Handoff problems without obvious disk issues can be due to the database containing large objects. Do you frequently update objects in CS, and if so have you had garbage collection running? The timeout is happening on the receiver side after not receiving any tcp data for handoff_receive_time

Re: Ownership handoff timed out

2015-10-26 Thread Jon Meredith
Hi, I suspect your {error,enotconn} messages are unrelated - that's likely to be caused by an HTTP client closing the connection while Riak looks up some networking information about the requestor. The max_concurrency message you are seeing is related to the handoff transfer limit - it should be

Ownership handoff timed out

2015-10-26 Thread Vladyslav Zakhozhai
Hello, I have a problem with persistent timeouts during ownership handoffs. I've tried to surf over Internet and current mail list but no success. I have Riak 1.4.12 cluster with 17 nodes. Almost all nodes use multibackend with bitcask and eleveldb as storage backends (we need multiple backend fo