The table is too large to be copied fast/effectively , so I'd like to leverage
immutableness property of SSTables.
My idea is to:
1) create new empty table (NewTable) with the same structure as existing one
(OldTable)
2) at some time run simultaneous 'nodetool snapshot -t ttt <keyspace> OldTable'
on all nodes -- this will create point in time state of OldTable
3) on each node run:
for each file in OldTable ttt snapshot directory:
ln
..../<keyspace>/OldTable-<uuid>/snapshots/ttt/<keyspace>_OldTable_xxxxxx
...../<keyspace>/Newtable/<keyspace>_NewTable_xxxxx
then:
nodetool refresh <keyspace> NewTable
4) nodetool repair NewTable
5) Use OldTable and NewTable independently (Read/Write)
Are there any issues with using hardlinks (ln) instead of copying (cp) in this
case?
Thanks,
Kyrill
________________________________
From: Rahul Singh <[email protected]>
Sent: Wednesday, April 18, 2018 2:07:17 AM
To: [email protected]
Subject: Re: copy from one table to another
1. Make a new table with the same schema.
For each node
2. Shutdown node
3. Copy data from Source sstable dir to new sstable dir.
This will do what you want.
--
Rahul Singh
[email protected]
Anant Corporation
On Apr 16, 2018, 4:21 PM -0500, Kyrylo Lebediev <[email protected]>,
wrote:
Thanks, Ali.
I just need to copy a large table in production without actual copying by using
hardlinks. After this both tables should be used independently (RW). Is this a
supported way or not?
Regards,
Kyrill
________________________________
From: Ali Hubail <[email protected]>
Sent: Monday, April 16, 2018 6:51:51 PM
To: [email protected]
Subject: Re: copy from one table to another
If you want to copy a portion of the data to another table, you can also use
sstable cql writer. It is more of an advanced feature and can be tricky, but
doable.
once you write the new sstables, you can then use the sstableloader to stream
the new data into the new table.
check this out:
https://www.datastax.com/dev/blog/using-the-cassandra-bulk-loader-updated
I have recently used this to clean up 500 GB worth of sstable data in order to
purge tombstones that were mistakenly generated by the client.
obviously this is not as fast as hardlinks + refresh, but it's much faster and
more efficient than using cql to copy data accross the tables.
take advantage of CQLSSTableWriter.builder.sorted() if you can, and utilize
writetime if you have to.
Ali Hubail
Confidentiality warning: This message and any attachments are intended only for
the persons to whom this message is addressed, are confidential, and may be
privileged. If you are not the intended recipient, you are hereby notified that
any review, retransmission, conversion to hard copy, copying, modification,
circulation or other use of this message and any attachments is strictly
prohibited. If you receive this message in error, please notify the sender
immediately by return email, and delete this message and any attachments from
your system. Petrolink International Limited its subsidiaries, holding
companies and affiliates disclaims all responsibility from and accepts no
liability whatsoever for the consequences of any unauthorized person acting, or
refraining from acting, on any information contained in this message. For
security purposes, staff training, to assist in resolving complaints and to
improve our customer service, email communications may be monitored and
telephone calls may be recorded.
Kyrylo Lebediev <[email protected]>
04/16/2018 10:37 AM
Please respond to
[email protected]
To
"[email protected]" <[email protected]>,
cc
Subject
Re: copy from one table to another
Any issues if we:
1) create an new empty table with the same structure as the old one
2) create hardlinks ("ln without -s"):
.../<newtable>-<newuuid>/<newkeyspacename>-<newtable>-* --->
.../<oldtable>-<olduuid>/<oldkeyspacename>-<oldtable>-*
3) run nodetool refresh -- newkeyspacename newtable
and then query/modify both tables independently/simultaneously?
In theory, as SSTables are immutable, this should work, but could there be some
hidden issues?
Regards,
Kyrill
________________________________
From: Dmitry Saprykin <[email protected]>
Sent: Sunday, April 8, 2018 7:33:03 PM
To: [email protected]
Subject: Re: copy from one table to another
You can copy hardlinks to ALL SSTables from old to new table and then delete
part of data you do not need in a new one.
On Sun, Apr 8, 2018 at 10:20 AM, Nitan Kainth
<[email protected]<mailto:[email protected]>> wrote:
If it for testing and you don’t need any specific data, just copy a set of
sstables with all files of that sequence and move to target tables directory
and rename it.
Restart target node or run nodetool refresh
Sent from my iPhone
On Apr 8, 2018, at 4:15 AM, onmstester onmstester
<[email protected]<mailto:[email protected]>> wrote:
Is there any way to copy some part of a table to another table in cassandra? A
large amount of data should be copied so i don't want to fetch data to client
and stream it back to cassandra using cql.
Sent using Zoho Mail<https://www.zoho.com/mail/>
