Re: Segmentation fault in eleveldb.so on Riak startup

2013-06-12 Thread Matthew Von-Maszewski
Vladimir, You have one of two problems, either there is something in one of the existing .sst files causing compaction to crash, or there is some type of threading problem that crash during the hash/bloom creation. I am going to assume an .sst file problem for now. The last time you crashed,

Re: Segmentation fault in eleveldb.so on Riak startup

2013-06-12 Thread Matthew Von-Maszewski
Vladimir, Also, my colleague sent this to me after my first email. This is roughly the plan I intend to follow … if the LOG file gives us a direct pointer to a file corruption: https://gist.github.com/gburd/b88aee6da7fee81dc036 However, your crash point suggests we might have to do a bit more

Re: Segmentation fault in eleveldb.so on Riak startup

2013-06-12 Thread Matthew Von-Maszewski
Vladimir, I asked around the Basho chat room and you have a crash that has never been seen. This should be interesting. The crash is happening during a compaction, specifically during the creation of the bloom filter for a new .sst file. Maybe we can isolate the old file that feeding this co

Segmentation fault in eleveldb.so on Riak startup

2013-06-12 Thread Vladimir Shabanov
Hello, I have a cluster of 8 Riak-1.3.1 nodes. Recently one of my nodes silently crashed. Nothing unusual was reported in logs. When I've tried to start my node again it worked for few seconds and silently crashed again. I've run 'riak console' and seen "Segmentation fault". gdb with dumped core