====
Brian was sent the attachments directly. I've added them to my Google drive
aswell:
https://drive.google.com/folderview?id=0B5My_8GYp_UjenJzcmlHLXp0cUk&usp=sharing
Due to mailman's 40kb limit.
====
Hey Brian,
Thanks for responding. Apparently, the newer nodes had RiakCS 1.4.3
installed, invalidating the line of my app.config:
{add_paths, ["/usr/lib64/riak-cs/lib/riak_cs-1.4.2/ebin"]},
I've updated all versions of riak-cs and the config file template to
include the correct version, still encountering this issue on 2/5 nodes.
CLI:
(12:55:34) [riak3] ~ $ service riak start
Starting riak: [ OK ]
(12:55:45) [riak3] ~ $ service riak-cs start
Starting riak-cs: [ OK ]
(12:56:47) [riak3] ~ $ riak-admin test
Failed to write test value: {error,timeout}
(12:58:34) [riak3] ~ $
I've also included a riak-debug of all of the nodes. riak1 and riak3 are
the only ones with this issue. They all appear to have a horrific amount of
this:
2013-11-28 00:43:58.031 [error] <0.1996.0> Hintfile
'/var/lib/riak/bitcask/1295940904977011572000923660260141583835533934592/1.bitcask.hint'
invalid
2013-11-28 00:44:10.073 [error] <0.2121.0> Hintfile
'/var/lib/riak/bitcask/1341612831143602288194788811282525428199781826560/1.bitcask.hint'
invalid
2013-11-28 00:44:12.061 [error] <0.2205.0> Hintfile
'/var/lib/riak/bitcask/1455792646560079078679451688838485039110401556480/1.bitcask.hint'
invalid
2013-11-28 00:44:12.061 [error] <0.2173.0> Hintfile
'/var/lib/riak/bitcask/1410120720393488362485586537816101194746153664512/1.bitcask.hint'
invalid
2013-11-28 00:44:12.250 [error] <0.2137.0> Hintfile
'/var/lib/riak/bitcask/1364448794226897646291721386793717350381905772544/1.bitcask.hint'
invalid
2013-11-28 00:44:12.259 [error] <0.2157.0> Hintfile
'/var/lib/riak/bitcask/1387284757310193004388653962304909272564029718528/1.bitcask.hint'
invalid
2013-11-28 00:44:12.362 [error] <0.2189.0> Hintfile
'/var/lib/riak/bitcask/1432956683476783720582519113327293116928277610496/1.bitcask.hint'
invalid
Which seems like something bad if it's at the "error" log level.
Since this issue has arisen, newly created users cannot successfully
authenticate, even when queried directly. However, the admin user and first
couple of users added are able to successfully authenticate.
I really appreciate any help with this. I'm trying more experience with
riak/riak-cs until I feel comfortable approaching issues before I proceed
with development of an application built with riak in mind.
Thanks,
Andrew
On Thu, Nov 28, 2013 at 1:07 AM, Andrew Tynefield <atynefi...@gmail.com>wrote:
> Hey Brian,
>
> Thanks for responding. Apparently, the newer nodes had RiakCS 1.4.3
> installed, invalidating the line of my app.config:
>
> {add_paths, ["/usr/lib64/riak-cs/lib/riak_cs-1.4.2/ebin"]},
>
> I've updated all versions of riak-cs and the config file template to
> include the correct version, still encountering this issue on 2/5 nodes.
>
>
> CLI:
> (12:55:34) [riak3] ~ $ service riak start
> Starting riak: [ OK ]
> (12:55:45) [riak3] ~ $ service riak-cs start
> Starting riak-cs: [ OK ]
> (12:56:47) [riak3] ~ $ riak-admin test
> Failed to write test value: {error,timeout}
> (12:58:34) [riak3] ~ $
>
> I've included a tail -f /var/log/riak*/*log of the above commands,
> attached as file 1385621726.log.
>
> I've also included a riak-debug of all of the nodes. riak1 and riak3 are
> the only ones with this issue. They all appear to have a horrific amount of
> this:
>
> 2013-11-28 00:43:58.031 [error] <0.1996.0> Hintfile
> '/var/lib/riak/bitcask/1295940904977011572000923660260141583835533934592/1.bitcask.hint'
> invalid
> 2013-11-28 00:44:10.073 [error] <0.2121.0> Hintfile
> '/var/lib/riak/bitcask/1341612831143602288194788811282525428199781826560/1.bitcask.hint'
> invalid
> 2013-11-28 00:44:12.061 [error] <0.2205.0> Hintfile
> '/var/lib/riak/bitcask/1455792646560079078679451688838485039110401556480/1.bitcask.hint'
> invalid
> 2013-11-28 00:44:12.061 [error] <0.2173.0> Hintfile
> '/var/lib/riak/bitcask/1410120720393488362485586537816101194746153664512/1.bitcask.hint'
> invalid
> 2013-11-28 00:44:12.250 [error] <0.2137.0> Hintfile
> '/var/lib/riak/bitcask/1364448794226897646291721386793717350381905772544/1.bitcask.hint'
> invalid
> 2013-11-28 00:44:12.259 [error] <0.2157.0> Hintfile
> '/var/lib/riak/bitcask/1387284757310193004388653962304909272564029718528/1.bitcask.hint'
> invalid
> 2013-11-28 00:44:12.362 [error] <0.2189.0> Hintfile
> '/var/lib/riak/bitcask/1432956683476783720582519113327293116928277610496/1.bitcask.hint'
> invalid
>
> Which seems like something bad if it's at the "error" log level.
>
> Since this issue has arisen, newly created users cannot successfully
> authenticate, even when queried directly. However, the admin user and first
> couple of users added are able to successfully authenticate.
>
> I really appreciate any help with this. I'm trying more experience with
> riak/riak-cs until I feel comfortable approaching issues before I proceed
> with development of an application built with riak in mind.
>
> Thanks,
> Andrew
>
>
> On Wed, Nov 27, 2013 at 8:18 AM, Brian Sparrow <bspar...@basho.com> wrote:
>
>> Hey Andrew,
>>
>> Would you mind attaching a `riak-debug` from that node? That will give us
>> a full picture of the node as well as all the logs.
>>
>> Thanks,
>> Brian
>>
>> --
>> Brian Sparrow
>> Developer Advocate
>> Basho Technologies
>>
>> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>>
>>
>>
--
[Andrew Tynefield]
On Thu, Nov 28, 2013 at 1:07 AM, Andrew Tynefield <atynefi...@gmail.com>wrote:
> Hey Brian,
>
> Thanks for responding. Apparently, the newer nodes had RiakCS 1.4.3
> installed, invalidating the line of my app.config:
>
> {add_paths, ["/usr/lib64/riak-cs/lib/riak_cs-1.4.2/ebin"]},
>
> I've updated all versions of riak-cs and the config file template to
> include the correct version, still encountering this issue on 2/5 nodes.
>
>
> CLI:
> (12:55:34) [riak3] ~ $ service riak start
> Starting riak: [ OK ]
> (12:55:45) [riak3] ~ $ service riak-cs start
> Starting riak-cs: [ OK ]
> (12:56:47) [riak3] ~ $ riak-admin test
> Failed to write test value: {error,timeout}
> (12:58:34) [riak3] ~ $
>
> I've included a tail -f /var/log/riak*/*log of the above commands,
> attached as file 1385621726.log.
>
> I've also included a riak-debug of all of the nodes. riak1 and riak3 are
> the only ones with this issue. They all appear to have a horrific amount of
> this:
>
> 2013-11-28 00:43:58.031 [error] <0.1996.0> Hintfile
> '/var/lib/riak/bitcask/1295940904977011572000923660260141583835533934592/1.bitcask.hint'
> invalid
> 2013-11-28 00:44:10.073 [error] <0.2121.0> Hintfile
> '/var/lib/riak/bitcask/1341612831143602288194788811282525428199781826560/1.bitcask.hint'
> invalid
> 2013-11-28 00:44:12.061 [error] <0.2205.0> Hintfile
> '/var/lib/riak/bitcask/1455792646560079078679451688838485039110401556480/1.bitcask.hint'
> invalid
> 2013-11-28 00:44:12.061 [error] <0.2173.0> Hintfile
> '/var/lib/riak/bitcask/1410120720393488362485586537816101194746153664512/1.bitcask.hint'
> invalid
> 2013-11-28 00:44:12.250 [error] <0.2137.0> Hintfile
> '/var/lib/riak/bitcask/1364448794226897646291721386793717350381905772544/1.bitcask.hint'
> invalid
> 2013-11-28 00:44:12.259 [error] <0.2157.0> Hintfile
> '/var/lib/riak/bitcask/1387284757310193004388653962304909272564029718528/1.bitcask.hint'
> invalid
> 2013-11-28 00:44:12.362 [error] <0.2189.0> Hintfile
> '/var/lib/riak/bitcask/1432956683476783720582519113327293116928277610496/1.bitcask.hint'
> invalid
>
> Which seems like something bad if it's at the "error" log level.
>
> Since this issue has arisen, newly created users cannot successfully
> authenticate, even when queried directly. However, the admin user and first
> couple of users added are able to successfully authenticate.
>
> I really appreciate any help with this. I'm trying more experience with
> riak/riak-cs until I feel comfortable approaching issues before I proceed
> with development of an application built with riak in mind.
>
> Thanks,
> Andrew
>
>
> On Wed, Nov 27, 2013 at 8:18 AM, Brian Sparrow <bspar...@basho.com> wrote:
>
>> Hey Andrew,
>>
>> Would you mind attaching a `riak-debug` from that node? That will give us
>> a full picture of the node as well as all the logs.
>>
>> Thanks,
>> Brian
>>
>> --
>> Brian Sparrow
>> Developer Advocate
>> Basho Technologies
>>
>> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>>
>>
>>
--
[Andrew Tynefield]
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
