Hi,
Matthew thank you for information.
On Thu, Oct 29, 2015 at 8:19 PM Matthew Von-Maszewski <matth...@basho.com>
wrote:
> I queried Basho’s Client Services team. They tell me the upgrade /
> coexist should be no problem.
>
> Matthew
>
> On Oct 29, 2015, at 1:38 PM, Vladyslav Zakhozhai <
> v.zakhoz...@smartweb.com.ua> wrote:
>
> Matthew can you describe the bug more detail?
>
> My plan was to migrate to eleveldb and only then to migrate to Riak 2.0.
> It seems that I need to change my plans to migrate to Riak 2.0 first. It is
> sad.
>
> Is it safe to migrate Riak 1.4.12/Riak CS 1.5.0 to Riak 2.0 on production
> environment? According to official upgrade guides I can upgrade nodes one
> by one in the same cluster. So Riak 2.0 and Riak 1.4.12 nodes can coexist
> in one cluster. Am I right?
>
> Thank you.
>
> On Thu, Oct 29, 2015 at 7:04 PM Matthew Von-Maszewski <matth...@basho.com>
> wrote:
>
>> Sad to say your LOG files suggest the same bug as seen elsewhere and
>> fixed by recent changes in the leveldb code.
>>
>> The tougher issue is that the fixes are currently only available for our
>> 2.0 product series. A backport would be non-trivial due to the number of
>> places changed between 1.4 and 2.0 and the number of places the fix
>> overlaps those changes. The corrected code is tagged “2.0.9” in eleveldb
>> and leveldb.
>>
>> The only path readily available to you is to have your receiving cluster
>> upgraded to 2.0 Riak CS and manually build/patch eleveldb to the 2.0.9
>> version. Then start your handoffs. (eleveldb version 2.0.9 is not present
>> in any shipping version of Riak … yet).
>>
>> I will write again if I can think of an easier solution. But nothing is
>> occurring to me or the team members I have queried.
>>
>> Matthew
>>
>> On Oct 29, 2015, at 12:14 PM, Vladyslav Zakhozhai <
>> v.zakhoz...@smartweb.com.ua> wrote:
>>
>> Hi,
>>
>> Matthew thank for you answer. eleveldb LOGs are attached.
>> Here is LOGs from 2 eleveldb nodes (eggeater was not restarted; what
>> about rattlesnake I'm not sure).
>>
>> On Thu, Oct 29, 2015 at 5:24 PM Matthew Von-Maszewski <matth...@basho.com>
>> wrote:
>>
>>> Hi,
>>>
>>> There was a known eleveldb bug with handoff receiving that could cause a
>>> timeout. But it does not sound like bug fits your symptoms. However, I am
>>> willing to verify my diagnosis. I would need you to gather the LOG files
>>> from all vnodes on the RECEIVING side (or at least from the vnode that you
>>> are attempting and is failing).
>>>
>>> I will check it for the symptoms of the known bug.
>>>
>>> Note: the LOG files reset on each restart of Riak. So you must gather
>>> the LOG files right after the failure without restarting Riak.
>>>
>>> Matthew
>>>
>>>
>>> On Oct 29, 2015, at 11:11 AM, Vladyslav Zakhozhai <
>>> v.zakhoz...@smartweb.com.ua> wrote:
>>>
>>> Hi,
>>>
>>> I want to make small update. Jon your hint about problems on sender side
>>> is correct. As I've already told there problems with available resources on
>>> sender nodes. There are no enough available RAM which is a cause of
>>> swapiness and load on disks. Restarting of sender nodes helps me (at least
>>> temoprarily).
>>>
>>>
>>> On Thu, Oct 29, 2015 at 12:19 PM Vladyslav Zakhozhai <
>>> v.zakhoz...@smartweb.com.ua> wrote:
>>>
>>>> Hi,
>>>>
>>>> Average size of objects in Riak - 300 Kb. This objects are images. This
>>>> data updates very very rearly (there almost no updates).
>>>>
>>>> I have GC turned on and works:
>>>> root@python:~# riak-cs-gc status
>>>> There is no garbage collection in progress
>>>> The current garbage collection interval is: 900
>>>> The current garbage collection leeway time is: 86400
>>>> Last run started at: 20151029T100600Z
>>>> Next run scheduled for: 20151029T102100Z
>>>>
>>>> Network misconfigurations were not detected. The result of your script
>>>> shows correct info.
>>>>
>>>> But I see that almost all nodes with bitcask suffers from low free
>>>> memory and they swapped. I think that it can be an issue. But my question
>>>> is, what workaround is for this problem.
>>>>
>>>> I've wrote in my first post that I tuned handoff_timeout and
>>>> handoff_receive_timeout (now this vaules are 300000 and 600000). But
>>>> situation is the same.
>>>>
>>>>
>>>> On Tue, Oct 27, 2015 at 4:06 PM Jon Meredith <jmered...@basho.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Handoff problems without obvious disk issues can be due to the
>>>>> database containing large objects. Do you frequently update objects in
>>>>> CS,
>>>>> and if so have you had garbage collection running?
>>>>>
>>>>> The timeout is happening on the receiver side after not receiving any
>>>>> tcp data for handoff_receive_timeout *milli*seconds. I know you said you
>>>>> increased it, but not how high. I would bump that up to 300000 to give
>>>>> the
>>>>> sender a chance to read larger objects off disk.
>>>>>
>>>>> To check if the sender is transmitting, on the source node you could
>>>>> run
>>>>> redbug:start("riak_core_handoff_sender:visit_item", [{arity,
>>>>> true},{print_file,"/tmp/visit_item.log"},{time, 3600000},{msgs,
>>>>> 1000000}]).
>>>>>
>>>>> That file should fill fairly fast with an entry for every object the
>>>>> sender tries to transmit.
>>>>>
>>>>> There's a long shot it could be network misconfiguration. Run this
>>>>> from the source node having problems
>>>>>
>>>>> rpc:multicall(erlang, apply, [fun() -> TargetNode = node(),
>>>>> [_Name,Host] = string:tokens(atom_to_list(TargetNode), "@"), {ok, Port} =
>>>>> riak_core_gen_server:call({riak_core_handoff_listener, TargetNode},
>>>>> handoff_port), HandoffIP = riak_core_handoff_listener:get_handoff_ip(),
>>>>> TNHandoffIP = case HandoffIP of error -> Host; {ok, "0.0.0.0"} -> Host;
>>>>> {ok, Other} -> Other end, {node(), HandoffIP, TNHandoffIP,
>>>>> inet:gethostbyname(TNHandoffIP), Port} end, []]).
>>>>>
>>>>> and it will print out a a list of remote nodes and IP addresses (and
>>>>> hopefully an empty list of failed nodes)
>>>>>
>>>>> {[{'dev1@127.0.0.1', <---- node name
>>>>> {ok,"0.0.0.0"}, <---- handoff ip address configured in
>>>>> app.config
>>>>> "127.0.0.1", <---- hostname passed to socket open
>>>>> {ok,{hostent,"127.0.0.1",[],inet,4,[{127,0,0,1}]}}, <--- DNS entry
>>>>> for hostname
>>>>> 10019}], <---- handoff port
>>>>> []} <--- empty list of errors
>>>>>
>>>>> Good luck, Jon.
>>>>>
>>>>> On Tue, Oct 27, 2015 at 3:55 AM Vladyslav Zakhozhai <
>>>>> v.zakhoz...@smartweb.com.ua> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Jon thank you for the answer. During approval of my mail to this list
>>>>>> I've troubleshoot my issue more deep. And yes, your are right. Neither
>>>>>> {error, enotconn} nor max_concurrency is my problem.
>>>>>>
>>>>>> I'm going to migrate my cluster entierly to eleveldb only, i.e. I
>>>>>> need to refuse using bitcask. I have a talk with basho support and they
>>>>>> said that it is tricky to tune bitcask on servers with 32 GB RAM (and I
>>>>>> guess that it is not tricky, but it is impossible, because bitcask loads
>>>>>> all keys in memory regardless of free available RAM). With LevelDB I have
>>>>>> opportunity to tune using RAM on servers.
>>>>>>
>>>>>> So I have 15 nodes with multibackend (bitcask for data and leveldb
>>>>>> for metadata). 2 additional servers are without multibackend - only with
>>>>>> leveldb. Now I'm not sure do I need still use mutibackend with
>>>>>> levedb-only
>>>>>> backend.
>>>>>>
>>>>>> And my problem is (as I mentioned earlier) the following. On
>>>>>> leveldb-only nodes I see handoffs timedout and no further progress.
>>>>>>
>>>>>> On multibackend hosts I have configuration:
>>>>>>
>>>>>> {riak_kv, [
>>>>>> {add_paths, ["/usr/lib/riak-cs/lib/riak_cs-1.5.0/ebin"]},
>>>>>> {storage_backend, riak_cs_kv_multi_backend},
>>>>>> {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
>>>>>> {multi_backend_default, be_default},
>>>>>> {multi_backend, [
>>>>>> {be_default, riak_kv_eleveldb_backend, [
>>>>>> {max_open_files, 50},
>>>>>> {data_root, "/var/lib/riak/leveldb"}
>>>>>> ]},
>>>>>> {be_blocks, riak_kv_bitcask_backend, [
>>>>>> {data_root, "/var/lib/riak/bitcask"}
>>>>>> ]}
>>>>>> ]},
>>>>>>
>>>>>> And for hosts with leveldb-only backend:
>>>>>>
>>>>>> {riak_kv, [
>>>>>> {storage_backend, riak_kv_eleveldb_backend},
>>>>>> ...
>>>>>> {eleveldb, [
>>>>>> {data_root, "/var/lib/riak/leveldb"}
>>>>>> (default values for leveldb)
>>>>>>
>>>>>> In leveldb logs I see nothing that could help me (no errors in logs).
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 26, 2015 at 3:57 PM Jon Meredith <jmered...@basho.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I suspect your {error,enotconn} messages are unrelated - that's
>>>>>>> likely to be caused by an HTTP client closing the connection while Riak
>>>>>>> looks up some networking information about the requestor.
>>>>>>>
>>>>>>> The max_concurrency message you are seeing is related to the handoff
>>>>>>> transfer limit - it should be labelled as informational. When a node has
>>>>>>> data to handoff it starts the handoff sender process and if there are
>>>>>>> either too many local handoff processes or too many on the remote side
>>>>>>> it
>>>>>>> exits with max_concurrency. You could increase with riak-admin
>>>>>>> transfer-limit but that probably won't help if you're timing out.
>>>>>>>
>>>>>>> As you're using the multi-backend you're transferring data from
>>>>>>> bitcask and leveldb. The next place I would look is in the leveldb LOG
>>>>>>> files to see if there are any leveldb vnodes that are having problems
>>>>>>> that's preventing repair.
>>>>>>>
>>>>>>> Jon
>>>>>>>
>>>>>>> On Mon, Oct 26, 2015 at 7:15 AM Vladyslav Zakhozhai <
>>>>>>> v.zakhoz...@smartweb.com.ua> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I have a problem with persistent timeouts during ownership
>>>>>>>> handoffs. I've tried to surf over Internet and current mail list but no
>>>>>>>> success.
>>>>>>>>
>>>>>>>> I have Riak 1.4.12 cluster with 17 nodes. Almost all nodes use
>>>>>>>> multibackend with bitcask and eleveldb as storage backends (we need
>>>>>>>> multiple backend for Riak CS 1.5.0 integration).
>>>>>>>>
>>>>>>>> Now I'm working to migrate Riak cluster to eleveldb as primary and
>>>>>>>> only backend. For now I have 2 nodes with eleveldb backend in the same
>>>>>>>> cluster.
>>>>>>>>
>>>>>>>> During ownership handoff process I permanently see errors of timed
>>>>>>>> out handoff receivers and sender.
>>>>>>>>
>>>>>>>> Here is partial output of riak-admin transfers:
>>>>>>>> ...
>>>>>>>> transfer type: ownership_transfer
>>>>>>>> vnode type: riak_kv_vnode
>>>>>>>> partition: 331121464707782692405522344912282871640797216768
>>>>>>>> started: 2015-10-21 08:32:55 [46.66 min ago]
>>>>>>>> last update: no updates seen
>>>>>>>> total size: unknown
>>>>>>>> objects transferred: unknown
>>>>>>>>
>>>>>>>> unknown
>>>>>>>> riak@taipan.pleiad.uaprom =======> r...@eggeater.pleiad.uapr
>>>>>>>> om
>>>>>>>> | | 0%
>>>>>>>> unknown
>>>>>>>>
>>>>>>>> transfer type: ownership_transfer
>>>>>>>> vnode type: riak_kv_vnode
>>>>>>>> partition: 336830455478606531929755488790080852186328203264
>>>>>>>> started: 2015-10-21 08:32:54 [46.68 min ago]
>>>>>>>> last update: no updates seen
>>>>>>>> total size: unknown
>>>>>>>> objects transferred: unknown
>>>>>>>> ...
>>>>>>>>
>>>>>>>> Some of partition handoffs state never updates, some of them
>>>>>>>> terminates after partial handoff objects and never starts again.
>>>>>>>>
>>>>>>>> I see nothing in logs but following:
>>>>>>>>
>>>>>>>> On receiver side:
>>>>>>>>
>>>>>>>> 2015-10-21 11:33:55.131 [error]
>>>>>>>> <0.25390.1266>@riak_core_handoff_receiver:handle_info:105 Handoff
>>>>>>>> receiver
>>>>>>>> for partition 331121464707782692405522344912282871640797216768 timed
>>>>>>>> out
>>>>>>>> after processing 0 objects.
>>>>>>>>
>>>>>>>> On sender side:
>>>>>>>>
>>>>>>>> 2015-10-21 11:01:58.879 [error] <0.13177.1401> CRASH REPORT Process
>>>>>>>> <0.13177.1401> with 0 neighbours crashed with reason: no function
>>>>>>>> clause
>>>>>>>> matching webmachine_request:peer_from_peername({error,enotconn},
>>>>>>>> {webmachine_request,{wm_reqstate,#Port<0.50978116>,[],undefined,undefined,undefined,{wm_reqdata,...},...}})
>>>>>>>> line 150
>>>>>>>> 2015-10-21 11:32:50.055 [error] <0.207.0> Supervisor
>>>>>>>> riak_core_handoff_sender_sup had child riak_core_handoff_sender started
>>>>>>>> with {riak_core_handoff_sender,start_link,undefined} at <0.22312.1090>
>>>>>>>> exit
>>>>>>>> with reason max_concurrency in context child_terminated
>>>>>>>>
>>>>>>>> {error, enotconn} - seems to be network issue. But I have no any
>>>>>>>> problems with network. All hosts resolve their neighbors correctly and
>>>>>>>> /etc/hosts on each node are correct.
>>>>>>>>
>>>>>>>> I've tried to increase handoff_timeout and handoff_receive_timeout.
>>>>>>>> But no success.
>>>>>>>>
>>>>>>>> Forcing handoff helped me but for short period of time:
>>>>>>>>
>>>>>>>> rpc:multicall([node() | nodes()], riak_core_vnode_manager,
>>>>>>>> force_handoffs, []).
>>>>>>>>
>>>>>>>>
>>>>>>>> I see progress of handoffs (riak-admin transfers) but then I see
>>>>>>>> handoff timed out again.
>>>>>>>>
>>>>>>>>
>>>>>>>> A week ago I've joined 4 nodes with bitcask. And there was no such
>>>>>>>> problems.
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm confused a little bit and need to understand my next steps in
>>>>>>>> troubleshooting this issue.
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> riak-users mailing list
>>>>>>>> riak-users@lists.basho.com
>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>>
>>>>>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>> <eggeater-leveldb-logs-old.tar.gz><rattlesnake-leveldb-logs-old.tar.gz>
>> <rattlesnake-leveldb-logs.tar.gz><eggeater-leveldb-logs.tar.gz>
>>
>>
>>
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
