Another thing that I have seen cause riak_kv to stall like this is LevelDB
corruption preventing vnodes from starting properly. I use this command to
look for LOG files with the word error in them. It is also worth checking the
anti_entropy levelDB LOG files as well.
find . -name "LOG" -exec grep -l -i 'error' {} \;
-cv
> On Nov 11, 2015, at 7:05 PM, Steven Sylvester <[email protected]> wrote:
>
> Hi Markus,
>
> One thing that stands out in what you pasted is the following line:
>
> 2015-11-09 11:31:45.263 [info] <0.73.0> alarm_handler:
> {set,{{disk_almost_full,"/"},[]}}
>
> Check the output of `df` and add nodes/disk as necessary to keep your disk
> usage at a manageable level. Our docs recommend keeping disk use below 60%:
> http://docs.basho.com/riak/latest/ops/building/planning/cluster/#Disk
> <http://docs.basho.com/riak/latest/ops/building/planning/cluster/#Disk>
>
> Disk capacity issues can sneak up on you due to some backend operations that
> require a lot of space all at once, such as Bitcask merging or AAE. *Bad
> things* tend to happen to a Riak cluster with full disks, including data
> loss/corruption.
>
> Other than that, are there any changes that have been made to the environment
> lately? Have you performed any upgrades recently?
>
> Check the output of `ps aux` and kill any processes related to Riak, such as
> any instances of `beam.smp` before attempting to start the node again.
>
> If the node won't start after these troubleshooting steps, run `riak console`
> which will attempt to start the node and put you into an Erlang attach
> shell. Monitor the messages that are printed to stdout and if/when it seems
> to hang, paste the output of the whole thing here and we can take a look.
>
> --
> Steven Sylvester, CISSP
> Client Services Engineer
> Basho Technologies
> _______________________________________________
> riak-users mailing list
> [email protected]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
