Just an update: I've also nuked /var/lib/riak/ on all nodes, uninstalled riak and riak-cs and started completely fresh to no avail. There are still two nodes which don't respond to riak-admin test only once riak-cs is running. Before I start it up, they respond fine. Which leads me to believe it may be related.
Please let me know if there's anything I can do to troubleshoot this further. Thanks, Andrew On Thu, Nov 28, 2013 at 1:16 AM, Andrew Tynefield <atynefi...@gmail.com>wrote: > ==== > Brian was sent the attachments directly. I've added them to my Google > drive aswell: > > https://drive.google.com/folderview?id=0B5My_8GYp_UjenJzcmlHLXp0cUk&usp=sharing > Due to mailman's 40kb limit. > ==== > > Hey Brian, > > Thanks for responding. Apparently, the newer nodes had RiakCS 1.4.3 > installed, invalidating the line of my app.config: > > {add_paths, ["/usr/lib64/riak-cs/lib/riak_cs-1.4.2/ebin"]}, > > I've updated all versions of riak-cs and the config file template to > include the correct version, still encountering this issue on 2/5 nodes. > > > CLI: > (12:55:34) [riak3] ~ $ service riak start > Starting riak: [ OK ] > (12:55:45) [riak3] ~ $ service riak-cs start > Starting riak-cs: [ OK ] > (12:56:47) [riak3] ~ $ riak-admin test > Failed to write test value: {error,timeout} > (12:58:34) [riak3] ~ $ > > > I've also included a riak-debug of all of the nodes. riak1 and riak3 are > the only ones with this issue. They all appear to have a horrific amount of > this: > > 2013-11-28 00:43:58.031 [error] <0.1996.0> Hintfile > '/var/lib/riak/bitcask/1295940904977011572000923660260141583835533934592/1.bitcask.hint' > invalid > 2013-11-28 00:44:10.073 [error] <0.2121.0> Hintfile > '/var/lib/riak/bitcask/1341612831143602288194788811282525428199781826560/1.bitcask.hint' > invalid > 2013-11-28 00:44:12.061 [error] <0.2205.0> Hintfile > '/var/lib/riak/bitcask/1455792646560079078679451688838485039110401556480/1.bitcask.hint' > invalid > 2013-11-28 00:44:12.061 [error] <0.2173.0> Hintfile > '/var/lib/riak/bitcask/1410120720393488362485586537816101194746153664512/1.bitcask.hint' > invalid > 2013-11-28 00:44:12.250 [error] <0.2137.0> Hintfile > '/var/lib/riak/bitcask/1364448794226897646291721386793717350381905772544/1.bitcask.hint' > invalid > 2013-11-28 00:44:12.259 [error] <0.2157.0> Hintfile > '/var/lib/riak/bitcask/1387284757310193004388653962304909272564029718528/1.bitcask.hint' > invalid > 2013-11-28 00:44:12.362 [error] <0.2189.0> Hintfile > '/var/lib/riak/bitcask/1432956683476783720582519113327293116928277610496/1.bitcask.hint' > invalid > > Which seems like something bad if it's at the "error" log level. > > Since this issue has arisen, newly created users cannot successfully > authenticate, even when queried directly. However, the admin user and first > couple of users added are able to successfully authenticate. > > I really appreciate any help with this. I'm trying more experience with > riak/riak-cs until I feel comfortable approaching issues before I proceed > with development of an application built with riak in mind. > > Thanks, > Andrew > > > On Thu, Nov 28, 2013 at 1:07 AM, Andrew Tynefield <atynefi...@gmail.com>wrote: > >> Hey Brian, >> >> Thanks for responding. Apparently, the newer nodes had RiakCS 1.4.3 >> installed, invalidating the line of my app.config: >> >> {add_paths, ["/usr/lib64/riak-cs/lib/riak_cs-1.4.2/ebin"]}, >> >> I've updated all versions of riak-cs and the config file template to >> include the correct version, still encountering this issue on 2/5 nodes. >> >> >> CLI: >> (12:55:34) [riak3] ~ $ service riak start >> Starting riak: [ OK ] >> (12:55:45) [riak3] ~ $ service riak-cs start >> Starting riak-cs: [ OK ] >> (12:56:47) [riak3] ~ $ riak-admin test >> Failed to write test value: {error,timeout} >> (12:58:34) [riak3] ~ $ >> >> I've included a tail -f /var/log/riak*/*log of the above commands, >> attached as file 1385621726.log. >> >> I've also included a riak-debug of all of the nodes. riak1 and riak3 are >> the only ones with this issue. They all appear to have a horrific amount of >> this: >> >> 2013-11-28 00:43:58.031 [error] <0.1996.0> Hintfile >> '/var/lib/riak/bitcask/1295940904977011572000923660260141583835533934592/1.bitcask.hint' >> invalid >> 2013-11-28 00:44:10.073 [error] <0.2121.0> Hintfile >> '/var/lib/riak/bitcask/1341612831143602288194788811282525428199781826560/1.bitcask.hint' >> invalid >> 2013-11-28 00:44:12.061 [error] <0.2205.0> Hintfile >> '/var/lib/riak/bitcask/1455792646560079078679451688838485039110401556480/1.bitcask.hint' >> invalid >> 2013-11-28 00:44:12.061 [error] <0.2173.0> Hintfile >> '/var/lib/riak/bitcask/1410120720393488362485586537816101194746153664512/1.bitcask.hint' >> invalid >> 2013-11-28 00:44:12.250 [error] <0.2137.0> Hintfile >> '/var/lib/riak/bitcask/1364448794226897646291721386793717350381905772544/1.bitcask.hint' >> invalid >> 2013-11-28 00:44:12.259 [error] <0.2157.0> Hintfile >> '/var/lib/riak/bitcask/1387284757310193004388653962304909272564029718528/1.bitcask.hint' >> invalid >> 2013-11-28 00:44:12.362 [error] <0.2189.0> Hintfile >> '/var/lib/riak/bitcask/1432956683476783720582519113327293116928277610496/1.bitcask.hint' >> invalid >> >> Which seems like something bad if it's at the "error" log level. >> >> Since this issue has arisen, newly created users cannot successfully >> authenticate, even when queried directly. However, the admin user and first >> couple of users added are able to successfully authenticate. >> >> I really appreciate any help with this. I'm trying more experience with >> riak/riak-cs until I feel comfortable approaching issues before I proceed >> with development of an application built with riak in mind. >> >> Thanks, >> Andrew >> >> >> On Wed, Nov 27, 2013 at 8:18 AM, Brian Sparrow <bspar...@basho.com>wrote: >> >>> Hey Andrew, >>> >>> Would you mind attaching a `riak-debug` from that node? That will give >>> us a full picture of the node as well as all the logs. >>> >>> Thanks, >>> Brian >>> >>> -- >>> Brian Sparrow >>> Developer Advocate >>> Basho Technologies >>> >>> Sent with Sparrow <http://www.sparrowmailapp.com/?sig> >>> >>> >>> > > > -- > [Andrew Tynefield] > > > On Thu, Nov 28, 2013 at 1:07 AM, Andrew Tynefield <atynefi...@gmail.com>wrote: > >> Hey Brian, >> >> Thanks for responding. Apparently, the newer nodes had RiakCS 1.4.3 >> installed, invalidating the line of my app.config: >> >> {add_paths, ["/usr/lib64/riak-cs/lib/riak_cs-1.4.2/ebin"]}, >> >> I've updated all versions of riak-cs and the config file template to >> include the correct version, still encountering this issue on 2/5 nodes. >> >> >> CLI: >> (12:55:34) [riak3] ~ $ service riak start >> Starting riak: [ OK ] >> (12:55:45) [riak3] ~ $ service riak-cs start >> Starting riak-cs: [ OK ] >> (12:56:47) [riak3] ~ $ riak-admin test >> Failed to write test value: {error,timeout} >> (12:58:34) [riak3] ~ $ >> >> I've included a tail -f /var/log/riak*/*log of the above commands, >> attached as file 1385621726.log. >> >> I've also included a riak-debug of all of the nodes. riak1 and riak3 are >> the only ones with this issue. They all appear to have a horrific amount of >> this: >> >> 2013-11-28 00:43:58.031 [error] <0.1996.0> Hintfile >> '/var/lib/riak/bitcask/1295940904977011572000923660260141583835533934592/1.bitcask.hint' >> invalid >> 2013-11-28 00:44:10.073 [error] <0.2121.0> Hintfile >> '/var/lib/riak/bitcask/1341612831143602288194788811282525428199781826560/1.bitcask.hint' >> invalid >> 2013-11-28 00:44:12.061 [error] <0.2205.0> Hintfile >> '/var/lib/riak/bitcask/1455792646560079078679451688838485039110401556480/1.bitcask.hint' >> invalid >> 2013-11-28 00:44:12.061 [error] <0.2173.0> Hintfile >> '/var/lib/riak/bitcask/1410120720393488362485586537816101194746153664512/1.bitcask.hint' >> invalid >> 2013-11-28 00:44:12.250 [error] <0.2137.0> Hintfile >> '/var/lib/riak/bitcask/1364448794226897646291721386793717350381905772544/1.bitcask.hint' >> invalid >> 2013-11-28 00:44:12.259 [error] <0.2157.0> Hintfile >> '/var/lib/riak/bitcask/1387284757310193004388653962304909272564029718528/1.bitcask.hint' >> invalid >> 2013-11-28 00:44:12.362 [error] <0.2189.0> Hintfile >> '/var/lib/riak/bitcask/1432956683476783720582519113327293116928277610496/1.bitcask.hint' >> invalid >> >> Which seems like something bad if it's at the "error" log level. >> >> Since this issue has arisen, newly created users cannot successfully >> authenticate, even when queried directly. However, the admin user and first >> couple of users added are able to successfully authenticate. >> >> I really appreciate any help with this. I'm trying more experience with >> riak/riak-cs until I feel comfortable approaching issues before I proceed >> with development of an application built with riak in mind. >> >> Thanks, >> Andrew >> >> >> On Wed, Nov 27, 2013 at 8:18 AM, Brian Sparrow <bspar...@basho.com>wrote: >> >>> Hey Andrew, >>> >>> Would you mind attaching a `riak-debug` from that node? That will give >>> us a full picture of the node as well as all the logs. >>> >>> Thanks, >>> Brian >>> >>> -- >>> Brian Sparrow >>> Developer Advocate >>> Basho Technologies >>> >>> Sent with Sparrow <http://www.sparrowmailapp.com/?sig> >>> >>> >>> > > > -- > [Andrew Tynefield] > -- [Andrew Tynefield]
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
