Well, maybe you had deleted your objects when you set leeway_second with 1 day, and later you changed leeway_second to 5min? Changing leeway_second won't change deletion timing of already deleted objects.
FYI, maybe it's not what you want, but there's a detailed description [1] on internal design and behaviour of garbage collection. Of course Riak CS and Riak is not a black box, how Riak CS uses Riak is open-sourced and not hidden. For developers and hard-core operators, there's just very primitive tool to diagnose Riak CS by hitting Riak directly. It is riak_cs_inspector, installed in /usr/lib/riak-cs/lib/riak_cs-1.4.5/priv/tools/internal/riak_cs_inspector.erl . See [2] for detailed usage. It prints Riak CS internal data in structured manner but it is still hard to find the answer "how many objects to be deleted today?" . I think your assumption is very natural, because there should be some operational visibility tools like you said. [1] https://github.com/basho/riak_cs/wiki/Object-Chunking-and-Garbage-Collection#garbage-collection [2] https://gist.github.com/kuenishi/bf286cf28f16ebbf1256 On Tue, Jun 10, 2014 at 1:46 AM, Tyler Flint <tylerfl...@gmail.com> wrote: > Unfortunately, it doesn’t seem to have made a difference. > > I did restart the riak-cs process after updating the app.config. Triggering a > riak-cs-gc batch did trigger a job, but did not seem to yield any results. > > I’m a bit concerned that I’m unable to root cause the issue here. Is there > some sort of visibility tools that I’m overlooking? I would assume that I’d > be able to query into riak-cs and find “x entries are ready to be garbage > collected” or something to provide some visibility. I’m not even sure if the > problem is within riak-cs or riak/bitcask. > > Are riak-cs and riak just black boxes? I can’t imagine this to be the case if > people are using these solutions in production. Please tell me I’m just an > idiot and not aware of some utility/configuration to make this work. > > Thanks > > On Jun 8, 2014, at 7:22 PM, Tyler Flint <tylerfl...@gmail.com> wrote: > >> Yes it is riak 1.4.5 >> >> Let me see if that helps. >> >> Thanks >> >> >> >>> On Jun 8, 2014, at 7:20 PM, Kota Uenishi <k...@basho.com> wrote: >>> >>> Spelling mistake, sorry: >>> >>> {gc_paginated_indexes, true}, >>> >>>> On Mon, Jun 9, 2014 at 10:10 AM, Kota Uenishi <k...@basho.com> wrote: >>>> Hi, >>>> >>>> Are you using 1.4.5? I'm not sure but you may have a log message like this: >>>> >>>> "Error occurred trying to query from time 0 to ~p in gc key index. Reason: >>>> ~p" >>>> >>>> in console.log. If so, it's even failing not only GC but in listing >>>> objects to be collected. Then setting a line >>>> >>>> {gc_pagenated_indexes, true}, >>>> >>>> into riak_cs section of your app.config . This is implicit option and >>>> its default is false. >>>> >>>> >>>>> On Fri, Jun 6, 2014 at 11:52 PM, Tyler Flint <tylerfl...@gmail.com> wrote: >>>>> Thank you in advance for any assistance. >>>>> >>>>> To explore riak-cs, I setup a 2 node cluster with riak and riak-cs on >>>>> each node, and put the riak nodes on the same cluster. I then proceeded >>>>> to use risk-cs by uploading a 3G file using the multi-part upload >>>>> protocol. I was unaware, however, that I needed to finalize the multipart >>>>> upload and thus had many in progress multi-part uploads. Through the >>>>> duration of my troubleshooting, the storage on the filesystem grew to >>>>> 224G. Once I discovered my error, I proceeded to cancel all of the >>>>> previous uploads. At this point there are no uploads in progress, and >>>>> only a single 3G file available in the bucket. However, it has been 48 >>>>> hours since the exploration and the storage use is still 224G on both >>>>> machines. >>>>> >>>>> I have attempted multiple strategies in attempt to reclaim space, all >>>>> without any effect. >>>>> >>>>> Within riak-cs: >>>>> >>>>> - I have shortened the gc leeway duration to 5 minutes and set all of the >>>>> gc intervals as low as 5 minutes. >>>>> - I have manually triggered the riak-cs-gc batch multiple times. (After >>>>> running batch, status shows a gc run and eventually complete) >>>>> >>>>> Within riak: >>>>> >>>>> - I have adjusted the bitcask settings in attempt to trigger a merge. >>>>> - I have force merged all of the bitcask partitions manually in the >>>>> erlang console. >>>>> >>>>> None of these strategies seem to have any effect whatsoever. >>>>> >>>>> At this point I’m not even sure where to start, nor do I know if the >>>>> issue is within riak-cs or riak (via bitcask). Any suggestions would be >>>>> appreciated. >>>>> >>>>> Thanks >>>>> >>>>> Tyler >>>>> _______________________________________________ >>>>> riak-users mailing list >>>>> riak-users@lists.basho.com >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>>> >>>> -- >>>> Kota UENISHI / @kuenishi >>>> Basho Japan KK >>> >>> >>> >>> -- >>> Kota UENISHI / @kuenishi >>> Basho Japan KK > -- Kota UENISHI / @kuenishi Basho Japan KK _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
