One update. I cleared the snapshot using nodetool clearsnapshot command.
Disk space is recovered now.

Because of this issue, I have mounted one more drive to the server and
there are some data files there. How can I migrate the data so that I can
decommission the drive?
Will it work if I just copy all the contents in the table directory to one
of the drives?

Thanks for all the help.

Regards,
Rahul

On Thursday 14 January 2016, Rahul Ramesh <rr.ii...@gmail.com> wrote:

> Hi Jan,
> I checked it. There are no old Key Spaces or tables.
> Thanks for your pointer, I started looking inside the directories. I see
> lot of snapshots directory inside the table directory. These directories
> are consuming space.
>
> However these snapshots are not shown  when I issue listsnapshots
> ./bin/nodetool listsnapshots
> Snapshot Details:
> There are no snapshots
>
> Can I safely delete those snapshots? why listsnapshots is not showing the
> snapshots? Also in future, how can we find out if there are snapshots?
>
> Thanks,
> Rahul
>
>
>
> On Thu, Jan 14, 2016 at 12:50 PM, Jan Kesten <j.kes...@enercast.de
> <javascript:_e(%7B%7D,'cvml','j.kes...@enercast.de');>> wrote:
>
>> Hi Rahul,
>>
>> just an idea, did you have a look at the data directorys on disk
>> (/var/lib/cassandra/data)? It could be that there are some from old
>> keyspaces that have been deleted and snapshoted before. Try something like
>> "du -sh /var/lib/cassandra/data/*" to verify which keyspace is consuming
>> your space.
>>
>> Jan
>>
>> Von meinem iPhone gesendet
>>
>> Am 14.01.2016 um 07:25 schrieb Rahul Ramesh <rr.ii...@gmail.com
>> <javascript:_e(%7B%7D,'cvml','rr.ii...@gmail.com');>>:
>>
>> Thanks for your suggestion.
>>
>> Compaction was happening on one of the large tables. The disk space did
>> not decrease much after the compaction. So I ran an external compaction.
>> The disk space decreased by around 10%. However it is still consuming close
>> to 750Gb for load of 250Gb.
>>
>> I even restarted cassandra thinking there may be some open files. However
>> it didnt help much.
>>
>> Is there any way to find out why so much of data is being consumed?
>>
>> I checked if there are any open files using lsof. There are not any open
>> files.
>>
>> *Recovery:*
>> Just a wild thought
>> I am using replication factor of 2 and I have two nodes. If I delete
>> complete data on one of the node, will I be able to recover all the data
>> from the active node?
>> I don't want to pursue this path as I want to find out the root cause of
>> the issue!
>>
>>
>> Any help will be greatly appreciated
>>
>> Thank you,
>>
>> Rahul
>>
>>
>>
>>
>>
>>
>> On Wed, Jan 13, 2016 at 3:37 PM, Carlos Rolo <r...@pythian.com
>> <javascript:_e(%7B%7D,'cvml','r...@pythian.com');>> wrote:
>>
>>> You can check if the snapshot exists in the snapshot folder.
>>> Repairs stream sstables over, than can temporary increase disk space.
>>> But I think Carlos Alonso might be correct. Running compactions might be
>>> the issue.
>>>
>>> Regards,
>>>
>>> Carlos Juzarte Rolo
>>> Cassandra Consultant
>>>
>>> Pythian - Love your data
>>>
>>> rolo@pythian | Twitter: @cjrolo | Linkedin: 
>>> *linkedin.com/in/carlosjuzarterolo
>>> <http://linkedin.com/in/carlosjuzarterolo>*
>>> Mobile: +351 91 891 81 00 | Tel: +1 613 565 8696 x1649
>>> www.pythian.com
>>>
>>> On Wed, Jan 13, 2016 at 9:24 AM, Carlos Alonso <i...@mrcalonso.com
>>> <javascript:_e(%7B%7D,'cvml','i...@mrcalonso.com');>> wrote:
>>>
>>>> I'd have a look also at possible running compactions.
>>>>
>>>> If you have big column families with STCS then large compactions may be
>>>> happening.
>>>>
>>>> Check it with nodetool compactionstats
>>>>
>>>> Carlos Alonso | Software Engineer | @calonso
>>>> <https://twitter.com/calonso>
>>>>
>>>> On 13 January 2016 at 05:22, Kevin O'Connor <ke...@reddit.com
>>>> <javascript:_e(%7B%7D,'cvml','ke...@reddit.com');>> wrote:
>>>>
>>>>> Have you tried restarting? It's possible there's open file handles to
>>>>> sstables that have been compacted away. You can verify by doing lsof and
>>>>> grepping for DEL or deleted.
>>>>>
>>>>> If it's not that, you can run nodetool cleanup on each node to scan
>>>>> all of the sstables on disk and remove anything that it's not responsible
>>>>> for. Generally this would only work if you added nodes recently.
>>>>>
>>>>>
>>>>> On Tuesday, January 12, 2016, Rahul Ramesh <rr.ii...@gmail.com
>>>>> <javascript:_e(%7B%7D,'cvml','rr.ii...@gmail.com');>> wrote:
>>>>>
>>>>>> We have a 2 node Cassandra cluster with a replication factor of 2.
>>>>>>
>>>>>> The load factor on the nodes is around 350Gb
>>>>>>
>>>>>> Datacenter: Cassandra
>>>>>> ==========
>>>>>> Address      Rack        Status State   Load            Owns
>>>>>>        Token
>>>>>>
>>>>>>       -5072018636360415943
>>>>>> 172.31.7.91  rack1       Up     Normal  328.5 GB        100.00%
>>>>>>       -7068746880841807701
>>>>>> 172.31.7.92  rack1       Up     Normal  351.7 GB        100.00%
>>>>>>       -5072018636360415943
>>>>>>
>>>>>> However,if I use df -h,
>>>>>>
>>>>>> /dev/xvdf       252G  223G   17G  94% /HDD1
>>>>>> /dev/xvdg       493G  456G   12G  98% /HDD2
>>>>>> /dev/xvdh       197G  167G   21G  90% /HDD3
>>>>>>
>>>>>>
>>>>>> HDD1,2,3 contains only cassandra data. It amounts to close to 1Tb in
>>>>>> one of the machine and in another machine it is close to 650Gb.
>>>>>>
>>>>>> I started repair 2 days ago, after running repair, the amount of disk
>>>>>> space consumption has actually increased.
>>>>>> I also checked if this is because of snapshots. nodetool listsnapshot
>>>>>> intermittently lists a snapshot but it goes away after sometime.
>>>>>>
>>>>>> Can somebody please help me understand,
>>>>>> 1. why so much disk space is consumed?
>>>>>> 2. Why did it increase after repair?
>>>>>> 3. Is there any way to recover from this state.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Rahul
>>>>>>
>>>>>>
>>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>
>

Reply via email to