Hello Matthew,
Space is reducing slooooowly , now we are faced with another problem :
Alot of applications store data on this node and we don't have the bucket
keys (they are uuid4) so :
We are using listkeys ( I know its bad ) on the erlang client and we also
tried with curl using both blocking and stream methods.
And they are all returning {error, timeout}.
We are 95 % sure that not all data has been migrated so , is there any way
to get the keys of the bucket, even if we have to shutdown the node (
Uptime/Availability not important for us ).
We are currently looking at mapreduce ?!
Thanks for all the patience
Antonio
2015-07-14 22:28 GMT+01:00 Matthew Von-Maszewski <matth...@basho.com>:
> Antonio,
>
> Someone reminded me that you could make temporary space on your servers by
> deactivating active_anti_entropy, then deleting its data. Of course, this
> assumes you are running “anti_entropy = active” in your riak.conf file.
>
> I will send you some better notes if you think this is worth researching.
> Let me know your thoughts.
>
> Matthew
>
>
>
> On Jul 14, 2015, at 4:21 PM, Antonio Teixeira <eagle.anto...@gmail.com>
> wrote:
>
> Ok Matthew,
>
> We will proceed with the deletion , will monitor the disk space and will
> come back with further reports to the list.
>
> Thanks for your time,
> Antonio
>
> 2015-07-14 18:32 GMT+01:00 Matthew Von-Maszewski <matth...@basho.com>:
>
>> Antonio,
>>
>> A Riak delete operation happens in these steps:
>>
>> - Riak writes a “tombstone” value for the key to the N vnodes that
>> contain it (this is a new record)
>>
>> - Riak by default, waits 3 seconds to verify all vnodes agree to the
>> tombstone/delete
>>
>> - Riak issues an actual delete operation against the key to leveldb
>>
>> - leveldb creates its own tombstone
>>
>> - the leveldb tombstone “floats” through level-0 and level-1 as part of
>> normal compactions
>>
>> - upon reaching level-2, leveldb will initiate immediate compaction and
>> propagation of tombstones in .sst table files containing 1000 or more
>> tombstones. This is when disk space recovery begins.
>>
>>
>> Yes, this means that initially the leveldb vnodes will grow in size until
>> enough stuff (new data and/or tombstones) forces the tombstones to level-2
>> via normal compaction operations. “Enough stuff” to fill levels 0 and 1 is
>> about 4.2Gbytes of compressed Riak objects.
>>
>> The “get” operation you mentioned is something that happens internally.
>> A manual “get” by your code will not influence the operation.
>>
>> Matthew
>>
>>
>> On Jul 14, 2015, at 1:01 PM, Antonio Teixeira <eagle.anto...@gmail.com>
>> wrote:
>>
>> Hi Matthew,
>>
>> We will be removing close to 1 TB of data from the node , and since we
>> are short on "disk space" when we saw that the disk space was actually
>> rising we halted the data removal.
>>
>> Now according to some docs I have read, if after a deletion ( a few
>> seconds ) we make a .get() it force the release of the diskspace , is this
>> true ?
>>
>> For us it's not possible to move data to another node, is there any way
>> even manually to release the space ? or at least to force :
>> " The actual release occurs significantly later (days, weeks, or even
>> months later) when the tombstone record merges with the actual data in a
>> background compaction."
>>
>> Thanks for all the help,
>> Antonio
>>
>>
>> 2015-07-14 17:43 GMT+01:00 Matthew Von-Maszewski <matth...@basho.com>:
>>
>>> Antonio,
>>>
>>> Here is a detailed discussion of the Riak / leveldb delete scenario:
>>>
>>> https://github.com/basho/leveldb/wiki/mv-aggressive-delete
>>>
>>> Pay close attention to the section titled “Update April 6, 2014”. This
>>> explains why as much as 4.2G bytes per vnode might remain within leveldb
>>> after deleting all keys. There is no mechanism to override the logic that
>>> causes the disk space retention.
>>>
>>> One workaround is to use Riak’s handoff mechanism to transfer vnodes
>>> from one physical server to another. The vnode transfer will remove all
>>> deletion tombstones on the destination. The last step of the transfer then
>>> deletes all leveldb files on the original server, recovering the space.
>>>
>>> Matthew
>>>
>>>
>>>
>>> > On Jul 14, 2015, at 12:32 PM, Antonio Teixeira <
>>> eagle.anto...@gmail.com> wrote:
>>> >
>>> > Hello,
>>> >
>>> > We have been migrating our Riak Database to another infrastructure
>>> through a "streaming" process and right now we should have somewhere around
>>> 2Gb of free space the Hard Disk, however those 2Gb are still being used by
>>> Riak. After some research I believe the problem is the Objects are only
>>> being marked for deletion and not actually deleted at runtime. What we need
>>> is a way to aggressively deleted those keys or some way to force Riak to
>>> delete those marked keys and subsequently release the Disk Space. The Riak
>>> version we are using is v2.0.2
>>> > _______________________________________________
>>> > riak-users mailing list
>>> > riak-users@lists.basho.com
>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>>
>
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
