Antonio, Someone reminded me that you could make temporary space on your servers by deactivating active_anti_entropy, then deleting its data. Of course, this assumes you are running “anti_entropy = active” in your riak.conf file.
I will send you some better notes if you think this is worth researching. Let me know your thoughts. Matthew > On Jul 14, 2015, at 4:21 PM, Antonio Teixeira <eagle.anto...@gmail.com> wrote: > > Ok Matthew, > > We will proceed with the deletion , will monitor the disk space and will come > back with further reports to the list. > > Thanks for your time, > Antonio > > 2015-07-14 18:32 GMT+01:00 Matthew Von-Maszewski <matth...@basho.com > <mailto:matth...@basho.com>>: > Antonio, > > A Riak delete operation happens in these steps: > > - Riak writes a “tombstone” value for the key to the N vnodes that contain it > (this is a new record) > > - Riak by default, waits 3 seconds to verify all vnodes agree to the > tombstone/delete > > - Riak issues an actual delete operation against the key to leveldb > > - leveldb creates its own tombstone > > - the leveldb tombstone “floats” through level-0 and level-1 as part of > normal compactions > > - upon reaching level-2, leveldb will initiate immediate compaction and > propagation of tombstones in .sst table files containing 1000 or more > tombstones. This is when disk space recovery begins. > > > Yes, this means that initially the leveldb vnodes will grow in size until > enough stuff (new data and/or tombstones) forces the tombstones to level-2 > via normal compaction operations. “Enough stuff” to fill levels 0 and 1 is > about 4.2Gbytes of compressed Riak objects. > > The “get” operation you mentioned is something that happens internally. A > manual “get” by your code will not influence the operation. > > Matthew > > >> On Jul 14, 2015, at 1:01 PM, Antonio Teixeira <eagle.anto...@gmail.com >> <mailto:eagle.anto...@gmail.com>> wrote: >> >> Hi Matthew, >> >> We will be removing close to 1 TB of data from the node , and since we are >> short on "disk space" when we saw that the disk space was actually rising we >> halted the data removal. >> >> Now according to some docs I have read, if after a deletion ( a few seconds >> ) we make a .get() it force the release of the diskspace , is this true ? >> >> For us it's not possible to move data to another node, is there any way even >> manually to release the space ? or at least to force : >> " The actual release occurs significantly later (days, weeks, or even months >> later) when the tombstone record merges with the actual data in a background >> compaction." >> >> Thanks for all the help, >> Antonio >> >> >> 2015-07-14 17:43 GMT+01:00 Matthew Von-Maszewski <matth...@basho.com >> <mailto:matth...@basho.com>>: >> Antonio, >> >> Here is a detailed discussion of the Riak / leveldb delete scenario: >> >> https://github.com/basho/leveldb/wiki/mv-aggressive-delete >> <https://github.com/basho/leveldb/wiki/mv-aggressive-delete> >> >> Pay close attention to the section titled “Update April 6, 2014”. This >> explains why as much as 4.2G bytes per vnode might remain within leveldb >> after deleting all keys. There is no mechanism to override the logic that >> causes the disk space retention. >> >> One workaround is to use Riak’s handoff mechanism to transfer vnodes from >> one physical server to another. The vnode transfer will remove all deletion >> tombstones on the destination. The last step of the transfer then deletes >> all leveldb files on the original server, recovering the space. >> >> Matthew >> >> >> >> > On Jul 14, 2015, at 12:32 PM, Antonio Teixeira <eagle.anto...@gmail.com >> > <mailto:eagle.anto...@gmail.com>> wrote: >> > >> > Hello, >> > >> > We have been migrating our Riak Database to another infrastructure through >> > a "streaming" process and right now we should have somewhere around 2Gb of >> > free space the Hard Disk, however those 2Gb are still being used by Riak. >> > After some research I believe the problem is the Objects are only being >> > marked for deletion and not actually deleted at runtime. What we need is a >> > way to aggressively deleted those keys or some way to force Riak to delete >> > those marked keys and subsequently release the Disk Space. The Riak >> > version we are using is v2.0.2 >> > _______________________________________________ >> > riak-users mailing list >> > riak-users@lists.basho.com <mailto:riak-users@lists.basho.com> >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com> >> >> > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
