Hi,
inspecting the logs further, I get this in /etc/riak/console.log even
before running riak-admin repair-2i :
2017-02-15 23:41:12.441 [warning] <0.714.0> Hintfile
'/var/lib/riak/bitcask/205523667749658222872393179600727299639115513856/2.bitcask.hint'
invalid
2017-02-15 23:41:12.441 [warning] <0.702.0> Hintfile
'/var/lib/riak/bitcask/22835963083295358096932575511191922182123945984/2.bitcask.hint'
invalid
2017-02-15 23:41:12.441 [warning] <0.716.0> Hintfile
'/var/lib/riak/bitcask/251195593916248939066258330623111144003363405824/2.bitcask.hint'
invalid
2017-02-15 23:41:12.441 [warning] <0.717.0> Hintfile
'/var/lib/riak/bitcask/296867520082839655260123481645494988367611297792/2.bitcask.hint'
invalid
2017-02-15 23:41:12.441 [warning] <0.700.0> Hintfile
'/var/lib/riak/bitcask/91343852333181432387730302044767688728495783936/2.bitcask.hint'
invalid
2017-02-15 23:41:12.441 [warning] <0.715.0> Hintfile
'/var/lib/riak/bitcask/228359630832953580969325755111919221821239459840/2.bitcask.hint'
invalid
2017-02-15 23:41:12.442 [warning] <0.697.0> Hintfile
'/var/lib/riak/bitcask/68507889249886074290797726533575766546371837952/2.bitcask.hint'
invalid
2017-02-15 23:41:12.442 [warning] <0.712.0> Hintfile
'/var/lib/riak/bitcask/159851741583067506678528028578343455274867621888/2.bitcask.hint'
invalid
2017-02-15 23:41:12.442 [warning] <0.719.0> Hintfile
'/var/lib/riak/bitcask/342539446249430371453988632667878832731859189760/2.bitcask.hint'
invalid
All of this is very surprising since I started riak-cs and riak properly.
Then at the end of console.log :
2017-02-15 23:41:13.651 [info] <0.481.0>@riak_core:wait_for_service:498
Wait complete for service riak_kv (10 seconds)
2017-02-15 23:41:13.652 [info] <0.678.0>@riak_core:wait_for_service:498
Wait complete for service riak_kv (10 seconds)
2017-02-15 23:41:13.668 [info] <0.7.0> Application yokozuna started on node
'riak@127.0.0.1'
2017-02-15 23:41:13.672 [info] <0.7.0> Application cluster_info started on
node 'riak@127.0.0.1'
2017-02-15 23:41:13.678 [info]
<0.201.0>@riak_core_capability:process_capability_changes:555 New
capability: {riak_control,member_info_version} = v1
2017-02-15 23:41:13.680 [info] <0.7.0> Application riak_control started on
node 'riak@127.0.0.1'
2017-02-15 23:41:13.680 [info] <0.7.0> Application erlydtl started on node '
riak@127.0.0.1'
2017-02-15 23:41:13.687 [info] <0.7.0> Application riak_auth_mods started
on node 'riak@127.0.0.1'
2017-02-15 23:41:17.714 [info]
<0.474.0>@riak_core_throttle:maybe_log_throttle_change:372 Changing
throttle for riak_kv/aae_throttle from undefined to 0 based on load factor 0
2017-02-15 23:41:32.719 [info]
<0.2388.0>@riak_kv_index_hashtree:build_or_rehash:1055 Starting AAE tree
build: 159851741583067506678528028578343455274867621888
2017-02-15 23:42:02.186 [info]
<0.2388.0>@riak_kv_index_hashtree:handle_fold_keys_result:629 Finished AAE
tree build: 159851741583067506678528028578343455274867621888
I assume it means riak is properly started.
So I start stanchion, then riak-cs. But I still have the exact same error...
Regards,
2017-02-15 22:16 GMT+01:00 Jean-Marc Le Roux <jeanmarc.ler...@aerys.in>:
> Forgot to mention ACLs are alright AFAIK :
>
> root@b4394bf1de78:/var/lib/riak# ls -la
> total 52
> drwxr-xr-x. 10 riak riak 179 Feb 9 23:43 .
> drwxr-xr-x. 1 root root 95 Feb 15 20:48 ..
> -r--------. 1 riak riak 20 Feb 9 01:00 .erlang.cookie
> drwxrwxr-x. 67 riak riak 8192 Feb 15 21:31 anti_entropy
> drwxrwxr-x. 66 riak riak 8192 Feb 9 23:42 bitcask
> drwxrwxr-x. 3 riak riak 40 Feb 9 23:42 cluster_meta
> drwxrwxr-x. 2 riak riak 225 Feb 15 22:09 generated.configs
> drwxrwxr-x. 2 riak riak 8192 Feb 15 22:09 kv_vnode
> drwxrwxr-x. 66 riak riak 8192 Feb 9 23:42 leveldb
> drwxrwxr-x. 2 riak riak 6 Feb 15 22:14 riak_kv_exchange_fsm
> drwxr-xr-x. 2 riak riak 186 Feb 15 22:09 ring
>
> 2017-02-15 22:13 GMT+01:00 Jean-Marc Le Roux <jeanmarc.ler...@aerys.in>:
>
>> Hi,
>>
>> I'll try to send the log archive ASAP.
>> Here is what I get in /var/log/riak/error.log after running riak-admin
>> repair-2i :
>>
>> 2017-02-15 22:09:06.535 [error]
>> <0.3287.0>@riak_kv_2i_aae:repair_partition:297
>> Failed to acquire hashtree lock on partition 125597796958124469533129165311
>> 5555720016817029120
>> 2017-02-15 22:09:06.535 [error]
>> <0.3288.0>@riak_kv_2i_aae:repair_partition:297
>> Failed to acquire hashtree lock on partition 127881393266454005342822422862
>> 6747642198940975104
>> 2017-02-15 22:09:06.535 [error]
>> <0.3289.0>@riak_kv_2i_aae:repair_partition:297
>> Failed to acquire hashtree lock on partition 479555224749202520035584085735
>> 030365824602865664
>> 2017-02-15 22:09:06.535 [error]
>> <0.3290.0>@riak_kv_2i_aae:repair_partition:297
>> Failed to acquire hashtree lock on partition 502391187832497878132516661246
>> 222288006726811648
>> 2017-02-15 22:09:06.535 [error]
>> <0.3291.0>@riak_kv_2i_aae:repair_partition:297
>> Failed to acquire hashtree lock on partition 111896219108147254674969620004
>> 8404186924073353216
>>
>> I tried to remove all "LOCK" files in /var/lib/riak but to no avail...
>> I'm guessing there is something here...
>>
>> Any idea ?
>>
>> 2017-02-09 17:37 GMT+01:00 Luke Bakken <lbak...@basho.com>:
>>
>>> Hi Jean-Marc -
>>>
>>> Can you provide a complete archive of the log directory? I wonder if
>>> another file might have more information.
>>>
>>> --
>>> Luke Bakken
>>> Engineer
>>> lbak...@basho.com
>>>
>>> On Thu, Feb 9, 2017 at 1:58 AM, Jean-Marc Le Roux
>>> <jeanmarc.ler...@aerys.in> wrote:
>>> >
>>> > Hello,
>>> >
>>> > here is the original github issue :
>>> >
>>> > https://github.com/basho/riak_cs/issues/1329
>>> >
>>> > I'm using riak-cs 2.1.1-1.el6 with stanchion 1.5.0-1.el6 on CentOS 6.8
>>> in a Docker container.
>>> > To make the data persistent, the following directories are mounted
>>> from outside the container :
>>> >
>>> > /var/log
>>> > /var/lib/riak/
>>> >
>>> > Everything works fine except when I remove/reimport the container.
>>> > Even when it's the same container.
>>> > The riak data is here in /var/lib/riak (bitcask and leveldb stuff).
>>> ACLs look fine on those files.
>>> >
>>> > Riak starts. Stanchion starts. But riak-cs won't start.
>>> > With a riak-cs concole, it looks like the problem is here :
>>> >>
>>> >> (riak-cs@127.0.0.1)1> [os_mon] memory supervisor port (memsup):
>>> Erlang has closed
>>> >>
>>> >> =INFO REPORT==== 18-Jan-2017::09:38:31 ===
>>> >> alarm_handler: {clear,system_memory_high_watermark}
>>> >> [os_mon] cpu supervisor port (cpu_sup): Erlang has closed
>>> >> {"Kernel pid terminated",application_contro
>>> ller,"{application_start_failure,riak_cs,{notfound,{riak_cs_
>>> app,start,[normal,[]]}}}"}
>>> >
>>> > var/log/riak-cs/access.log.2017_01_18_09 is empty.
>>> > Here is what /var/log/riak-cs/crash.log says:
>>> >>
>>> >> 2017-01-18 09:38:31 =CRASH REPORT====
>>> >> crasher:
>>> >> initial call: application_master:init/4
>>> >> pid: <0.148.0>
>>> >> registered_name: []
>>> >> exception exit: {{notfound,{riak_cs_app,start,
>>> [normal,[]]}},[{application_master,init,4,[{file,"applicatio
>>> n_master.erl"},{line,133}]},{proc_lib,init_p_do_apply,3,[{fi
>>> le,"proc_lib.erl"},{line,239}]}]}
>>> >> ancestors: [<0.147.0>]
>>> >> messages: [{'EXIT',<0.149.0>,normal}]
>>> >> links: [<0.147.0>,<0.7.0>]
>>> >> dictionary: []
>>> >> trap_exit: true
>>> >> status: running
>>> >> heap_size: 376
>>> >> stack_size: 27
>>> >> reductions: 119
>>> >> neighbours:
>>>
>>
>>
>>
>> --
>> *Jean-Marc Le Roux*
>>
>>
>> Founder and CEO of Aerys (http://aerys.in)
>>
>> Blog: http://blogs.aerys.in/jeanmarc-leroux
>> Cell: (+33)6 20 56 45 78 <+33%206%2020%2056%2045%2078>
>> Phone: (+33)9 72 40 17 58 <+33%209%2072%2040%2017%2058>
>>
>
>
>
> --
> *Jean-Marc Le Roux*
>
>
> Founder and CEO of Aerys (http://aerys.in)
>
> Blog: http://blogs.aerys.in/jeanmarc-leroux
> Cell: (+33)6 20 56 45 78 <+33%206%2020%2056%2045%2078>
> Phone: (+33)9 72 40 17 58 <+33%209%2072%2040%2017%2058>
>
--
*Jean-Marc Le Roux*
Founder and CEO of Aerys (http://aerys.in)
Blog: http://blogs.aerys.in/jeanmarc-leroux
Cell: (+33)6 20 56 45 78
Phone: (+33)9 72 40 17 58
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
