Lots of simultaneous connections?

[Max Campos]

Hi -

We’re finally putting our new application under load, and we’re starting to get 
this error message from the Python driver when under heavy load:

('Unable to connect to any servers', {‘x.y.z.205': 
OperationTimedOut('errors=None, last_host=None',), ‘x.y.z.204': 
OperationTimedOut('errors=None, last_host=None',), ‘x.y.z.206': 
OperationTimedOut('errors=None, last_host=None',)})' (22.7s)

Our cluster is running 3.0.6, has 3 nodes and we use RF=3, CL=QUORUM 
reads/writes.  We have a few thousand machines which are each making 1-10 
connections to C* at once, but each of these connections only reads/writes a 
few records, waits several minutes, and then writes a few records — so while 
netstat reports ~5K connections per node, they’re generally idle.  Peak 
read/sec today was ~1500 per node, peak writes/sec was ~300 per node.  
Read/write latencies peaked at 2.5ms.

Some questions:
1) Is anyone else out there making this many simultaneous connections?  Any 
idea what a reasonable number of connections is, what is too many, etc?

2) Any thoughts on which JMX metrics I should look at to better understand what 
exactly is exploding?  Is there a “number of active connections” metric?  We 
currently look at:
- client reads/writes per sec
- read/write latency
- compaction tasks
- repair tasks
- disk used by node
- disk used by table
- avg partition size per table

3) Any other advice?  

I think I’ll try doing an explicit disconnect during the waiting period of our 
application’s execution; so as to get the C* connection count down.  Hopefully 
that will solve the timeout problem.

Thanks for your help.

- Max
-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Deleted data comes back on node decommission

[Python_Max]

Thank you for reply.

No, I did not execute 'nodetool cleanup'. Documentation 
https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRemoveNode.html 
does not mention that cleanup is required.


Do yo think that extra data which node is not responsible for can lead 
to zombie data?



On 13.12.17 18:43, Jeff Jirsa wrote:

Did you run cleanup before you shrank the cluster?



--

Best Regards,
Python_Max.


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: nodetool rebuild data size

[Anshu Vajpayee]

You will require to rebuild each node  with nodetool rebuild command.  it
would be 60TB.

On Thu, Dec 14, 2017 at 11:35 AM, Peng Xiao <2535...@qq.com> wrote:

> Hi there,
>
> if we have a Cassandra DC1 with data size 60T,RF=3,then we rebuild a new
> DC2(RF=3),how much data will stream to DC2?20T or 60T?
>
> Thanks,
> Peng Xiao
>



-- 
*C*heers,*
*Anshu V*

Upgrade using rebuild

[Anshu Vajpayee]

Hi -

Is it possible to upgrade a  cluster ( DC wise) using nodetool rebuild ?



-- 
*C*heers,*
*Anshu V*

Re: Upgrade using rebuild

[Hannu Kröger]

If you want to do a version upgrade, you need to basically do follow node
by node:

0) stop repairs
1) make sure your sstables are at the latest version (nodetool
upgradesstables can do it)
2) stop cassandra
3) update cassandra software and update cassandra.yaml and cassandra-env.sh
files
4) start cassandra

After all nodes are up, run “nodetool upgradesstables” on each node to
update your sstables to the latest version.

Also please note that when you upgrade, you need to upgrade only between
compatible versions.

E.g. 2.2.x -> 3.0.x  but not 1.2 to 3.11

Cheers,
Hannu

On 14 December 2017 at 12:33:49, Anshu Vajpayee (anshu.vajpa...@gmail.com)
wrote:

Hi -

Is it possible to upgrade a  cluster ( DC wise) using nodetool rebuild ?



--
*C*heers,*
*Anshu V*

Re: Deleted data comes back on node decommission

[kurt greaves]

Are you positive your repairs are completing successfully? Can you send
through an example of the data in the wrong order? What you're saying
certainly shouldn't happen, but there's a lot of room for mistakes.

On 14 Dec. 2017 20:13, "Python_Max"  wrote:

> Thank you for reply.
>
> No, I did not execute 'nodetool cleanup'. Documentation
> https://docs.datastax.com/en/cassandra/3.0/cassandra/operati
> ons/opsRemoveNode.html does not mention that cleanup is required.
>
> Do yo think that extra data which node is not responsible for can lead to
> zombie data?
>
>
> On 13.12.17 18:43, Jeff Jirsa wrote:
>
>> Did you run cleanup before you shrank the cluster?
>>
>>
> --
>
> Best Regards,
> Python_Max.
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> happen

Re: Lots of simultaneous connections?

[kurt greaves]

I see time outs and I immediately blame firewalls. Have you triple checked
then?
Is this only occurring to a subset of clients?

Also, 3.0.6 is pretty dated and has many bugs, you should definitely
upgrade to the latest 3.0 (don't forget to read news.txt)
On 14 Dec. 2017 19:18, "Max Campos"  wrote:

Hi -

We’re finally putting our new application under load, and we’re starting to
get this error message from the Python driver when under heavy load:

('Unable to connect to any servers', {‘x.y.z.205':
OperationTimedOut('errors=None, last_host=None',), ‘x.y.z.204':
OperationTimedOut('errors=None, last_host=None',), ‘x.y.z.206':
OperationTimedOut('errors=None, last_host=None',)})' (22.7s)

Our cluster is running 3.0.6, has 3 nodes and we use RF=3, CL=QUORUM
reads/writes.  We have a few thousand machines which are each making 1-10
connections to C* at once, but each of these connections only reads/writes
a few records, waits several minutes, and then writes a few records — so
while netstat reports ~5K connections per node, they’re generally idle.
Peak read/sec today was ~1500 per node, peak writes/sec was ~300 per node.
Read/write latencies peaked at 2.5ms.

Some questions:
1) Is anyone else out there making this many simultaneous connections?  Any
idea what a reasonable number of connections is, what is too many, etc?

2) Any thoughts on which JMX metrics I should look at to better understand
what exactly is exploding?  Is there a “number of active connections”
metric?  We currently look at:
- client reads/writes per sec
- read/write latency
- compaction tasks
- repair tasks
- disk used by node
- disk used by table
- avg partition size per table

3) Any other advice?

I think I’ll try doing an explicit disconnect during the waiting period of
our application’s execution; so as to get the C* connection count down.
Hopefully that will solve the timeout problem.

Thanks for your help.

- Max
-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

RE: When Replacing a Node, How to Force a Consistent Bootstrap

[Fd Habash]

“ … but it's better to repair before and after if possible …”

After, I simply run ‘nodetool repair –full’ on the replaced node. But before 
bootstrapping, if my cluster is distributed over 3 AZ’s, what do I repair? The 
entire other AZ’s? As one pointed out earlier, I can use ‘nodetool repair 
-hosts”, how do you identify what specific hosts to repair?

Thanks 


Thank you

From: Fd Habash
Sent: Thursday, December 7, 2017 12:09 PM
To: user@cassandra.apache.org
Subject: RE: When Replacing a Node, How to Force a Consistent Bootstrap

Thank you.

How do I identify what other 2 nodes the former downed node replicated with? A 
replica set of 3 nodes A,B,C. Now, C has been terminated by AWS and is gone. 
Using the getendpoints assumes knowing a partition key value, but how do you 
even know what key to use?

If there is a way to identify A and B, I, then, can simply run ‘nodetool 
repair’ to repair ALL the ranges on either.

Thanks 


Thank you

From: kurt greaves
Sent: Wednesday, December 6, 2017 6:45 PM
To: User
Subject: Re: When Replacing a Node, How to Force a Consistent Bootstrap

That's also an option but it's better to repair before and after if possible, 
if you don't repair beforehand you could end up missing some replicas until you 
repair after replacement, which could cause queries to return old/no data. 
Alternatively you could use ALL after replacing until the repair completes.

For example, A and C have replica a, A dies, on replace A streams the partition 
owning a from B, and thus is still inconsistent. QUORUM query hits A and B, and 
no results are returned for a.

On 5 December 2017 at 23:04, Fred Habash  wrote:
Or, do a full repair after bootstrapping completes?



On Dec 5, 2017 4:43 PM, "Jeff Jirsa"  wrote:
You cant ask cassandra to stream from the node with the "most recent data", 
because for some rows B may be most recent, and for others C may be most recent 
- you'd have to stream from both (which we don't support).

You'll need to repair (and you can repair before you do the replace to avoid 
the window of time where you violate consistency - use the -hosts option to 
allow repair with a down host, you'll repair A+C, so when B starts it'll 
definitely have all of the data).


On Tue, Dec 5, 2017 at 1:38 PM, Fd Habash  wrote:
Assume I have cluster of 3 nodes (A,B,C). Row x was written with CL=LQ to node 
A and B. Before it was written to C, node B crashes. I replaced B and it 
bootstrapped data from node C.
 
Now, row x is missing from C and B.  If node A crashes, it will be replaced and 
it will bootstrap from either C or B. As such, row x is now completely gone 
from the entire ring. 
 
Is this scenario possible at all (at least in C* < 3.0). 
 
How can a newly replaced node be forced to bootstrap from the node in the 
replica set that has the most recent data? 
 
Otherwise, we have to repair a node immediately after bootstrapping it for a 
node replacement.
 
Thank you
 

Re: Upgrade using rebuild

[Anshu Vajpayee]

Thanks! I am aware with these steps.

I m just thinking , is it possible to do the upgrade using nodetool rebuild
like  we rebuld new dc ?

Has anyone tried -  upgrade with nodetool rebuild ?



On Thu, 14 Dec 2017 at 7:08 PM, Hannu Kröger  wrote:

> If you want to do a version upgrade, you need to basically do follow node
> by node:
>
> 0) stop repairs
> 1) make sure your sstables are at the latest version (nodetool
> upgradesstables can do it)
> 2) stop cassandra
> 3) update cassandra software and update cassandra.yaml and
> cassandra-env.sh files
> 4) start cassandra
>
> After all nodes are up, run “nodetool upgradesstables” on each node to
> update your sstables to the latest version.
>
> Also please note that when you upgrade, you need to upgrade only between
> compatible versions.
>
> E.g. 2.2.x -> 3.0.x  but not 1.2 to 3.11
>
> Cheers,
> Hannu
>
> On 14 December 2017 at 12:33:49, Anshu Vajpayee (anshu.vajpa...@gmail.com)
> wrote:
>
> Hi -
>
> Is it possible to upgrade a  cluster ( DC wise) using nodetool rebuild ?
>
>
>
> --
> *C*heers,*
> *Anshu V*
>
>
> --
*C*heers,*
*Anshu V*

Re: Upgrade using rebuild

[Jon Haddad]

no

> On Dec 14, 2017, at 10:59 AM, Anshu Vajpayee  wrote:
> 
> Thanks! I am aware with these steps.
> 
> I m just thinking , is it possible to do the upgrade using nodetool rebuild 
> like  we rebuld new dc ?
> 
> Has anyone tried -  upgrade with nodetool rebuild ?
> 
> 
> 
> On Thu, 14 Dec 2017 at 7:08 PM, Hannu Kröger  > wrote:
> If you want to do a version upgrade, you need to basically do follow node by 
> node:
> 
> 0) stop repairs
> 1) make sure your sstables are at the latest version (nodetool 
> upgradesstables can do it)
> 2) stop cassandra
> 3) update cassandra software and update cassandra.yaml and cassandra-env.sh 
> files
> 4) start cassandra
> 
> After all nodes are up, run “nodetool upgradesstables” on each node to update 
> your sstables to the latest version.
> 
> Also please note that when you upgrade, you need to upgrade only between 
> compatible versions.
> 
> E.g. 2.2.x -> 3.0.x  but not 1.2 to 3.11
> 
> Cheers,
> Hannu
> On 14 December 2017 at 12:33:49, Anshu Vajpayee (anshu.vajpa...@gmail.com 
> ) wrote:
> 
>> Hi - 
>> 
>> Is it possible to upgrade a  cluster ( DC wise) using nodetool rebuild ? 
>> 
>> 
>> 
>> --
>> C*heers,
>> Anshu V
>> 
>> 
> 
> -- 
> C*heers,
> Anshu V
> 
> 

Re: Upgrade using rebuild

[Jon Haddad]

Heh, hit send accidentally.

You generally can’t run rebuild to upgrade, because it’s a streaming operation. 
 Streaming isn’t supported between versions, although on 3.x it might work.

> On Dec 14, 2017, at 11:01 AM, Jon Haddad  wrote:
> 
> no
> 
>> On Dec 14, 2017, at 10:59 AM, Anshu Vajpayee > > wrote:
>> 
>> Thanks! I am aware with these steps.
>> 
>> I m just thinking , is it possible to do the upgrade using nodetool rebuild 
>> like  we rebuld new dc ?
>> 
>> Has anyone tried -  upgrade with nodetool rebuild ?
>> 
>> 
>> 
>> On Thu, 14 Dec 2017 at 7:08 PM, Hannu Kröger > > wrote:
>> If you want to do a version upgrade, you need to basically do follow node by 
>> node:
>> 
>> 0) stop repairs
>> 1) make sure your sstables are at the latest version (nodetool 
>> upgradesstables can do it)
>> 2) stop cassandra
>> 3) update cassandra software and update cassandra.yaml and cassandra-env.sh 
>> files
>> 4) start cassandra
>> 
>> After all nodes are up, run “nodetool upgradesstables” on each node to 
>> update your sstables to the latest version.
>> 
>> Also please note that when you upgrade, you need to upgrade only between 
>> compatible versions.
>> 
>> E.g. 2.2.x -> 3.0.x  but not 1.2 to 3.11
>> 
>> Cheers,
>> Hannu
>> On 14 December 2017 at 12:33:49, Anshu Vajpayee (anshu.vajpa...@gmail.com 
>> ) wrote:
>> 
>>> Hi - 
>>> 
>>> Is it possible to upgrade a  cluster ( DC wise) using nodetool rebuild ? 
>>> 
>>> 
>>> 
>>> --
>>> C*heers,
>>> Anshu V
>>> 
>>> 
>> 
>> -- 
>> C*heers,
>> Anshu V
>> 
>> 
> 

TWCS on partitions spanning multiple time windows

[Hannu Kröger]

Hi,

I have been reading a bit about TWCS to understand how it functions.

Current assumption: TWCS uses same tombstone checks as any other compaction
strategy to make sure that it doesn’t remove tombstones unless it is safe
to do so.

Scenario 1:

So let’s assume that I have a tables like this:

CREATE TABLE twcs.twcs (
user_id int,
id int,
value int,
text_value text,
PRIMARY KEY (user_id, id, value)
)

I insert data for multiple users but also multiple events per user with TTL
of 10 days and have time window set to 1 day.

Basically when 10 days are up the first sstable contains just TTL'd data.
However TWCS cannot just drop sstables because same partition exists in
sstables of other windows. Otherwise it wouldn’t be safe, right?

Scenario 2:

Table is as follows

CREATE TABLE twcs.twcs2 (
user_id int,
day int,
id int,
value int,
text_value text,
PRIMARY KEY ((user_id, day), id, value)
)

I insert data with TTL of 10 days and have time window set to 2 days.

Basically when 10 days are up the first sstable contains just TTL:d data.
In this case TWCS can drop the whole sstable because the whole partition is
in the same time window and same sstable. Correct?

If we for some reason have time window and partition time buckets
misaligned, e.g. time window is 25 hours and time bucket is 24 hours, then
we end up in situation where we will actually never get rid of all
tombstones because same partition data will be across multiple time windows
which won’t be compacted together. So we would be in trouble, right?

Did I get it right?

Hannu

What is the maximum size/number of lines of cassandra cqlsh_history and cli.history?

[Laxmikant Upadhyay]

I am using cassandra version 2.1.16. I have verified that the cli.history
files contains max 501 lines. However I don't see there is a limit for
cqlsh_history file. Any idea on that ?

-- 

regards,
Laxmikant Upadhyay

Re: TWCS on partitions spanning multiple time windows

[Jeff Jirsa]

Inline

-- 
Jeff Jirsa


> On Dec 14, 2017, at 8:37 AM, Hannu Kröger  wrote:
> 
> Hi,
> 
> I have been reading a bit about TWCS to understand how it functions.
> 
> Current assumption: TWCS uses same tombstone checks as any other compaction 
> strategy to make sure that it doesn’t remove tombstones unless it is safe to 
> do so.
> 

Correct, though there’s a flag in 3.11.2/trunk to let you tell Cassandra you’re 
smart enough to ask it to do unsafe things

> Scenario 1:
> 
> So let’s assume that I have a tables like this:
> 
> CREATE TABLE twcs.twcs (
> user_id int,
> id int,
> value int,
> text_value text,
> PRIMARY KEY (user_id, id, value)
> )
> 
> I insert data for multiple users but also multiple events per user with TTL 
> of 10 days and have time window set to 1 day.
> 
> Basically when 10 days are up the first sstable contains just TTL'd data. 
> However TWCS cannot just drop sstables because same partition exists in 
> sstables of other windows. Otherwise it wouldn’t be safe, right?
> 

It won’t drop an sstable that overlaps with a live sstable, that doesn’t mean 
it can’t purge out most of the expired data.

Overlaps are almost always caused by read repair, not by partitions spanning 
windows, unless you’re writing some data with ttls and some without, writing 
with USING TIMESTAMP to write data that doesn’t match “now”


> Scenario 2:
> 
> Table is as follows
> 
> CREATE TABLE twcs.twcs2 (
> user_id int,
> day int,
> id int,
> value int,
> text_value text,
> PRIMARY KEY ((user_id, day), id, value)
> )
> 
> I insert data with TTL of 10 days and have time window set to 2 days.
> 
> Basically when 10 days are up the first sstable contains just TTL:d data. In 
> this case TWCS can drop the whole sstable because the whole partition is in 
> the same time window and same sstable. Correct?

This model is probably better not because of the drop behavior, but because 
reads won’t have to merge literally every sstable on the disk

Again, the full drop behavior looks for sstables with live data and overlapping 
time stamp at a full sstable level, it’s not per partition. But limiting 
partitions to one window will make the partitions easier to drop with the 
tombstone compaction.

> 
> If we for some reason have time window and partition time buckets misaligned, 
> e.g. time window is 25 hours and time bucket is 24 hours, then we end up in 
> situation where we will actually never get rid of all tombstones because same 
> partition data will be across multiple time windows which won’t be compacted 
> together. So we would be in trouble, right?
> 

You can still turn on tombstone compaction and purge most of the data, I think 
(without reading the code again) that it’ll  drop fully after 2 windows 
assuming there’s no read repair polluting the new window 

> Did I get it right?
> 
> Hannu

Bring 2 nodes down

[Alaa Zubaidi (PDF)]

Hi,
I have a cluster of 8 Nodes, 4 physical machines 2 VMs each physical
machine.
RF=3, and we have read/write with QUORUM consistency requirement.

One of the machines needs to be down for an hour or two to fix local disk.
What is the best way to do that with out losing data?

Regards
-- Alaa

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Re: Lots of simultaneous connections?

[Max Campos]

Hi Kurt, thanks for your reply — really appreciate your (and everyone else’s!) 
continual assistance of people in the C* user community.

All of these clients & servers are on the same (internal) network, so there is 
no firewall between the clients & servers.  

Our C* application is a QA test results system.  We have thousands of machines 
in-house which we use to test the software (not C* related) which we sell, and 
we’re using C* to capture the results of those tests.

So the flow is:
On each machine (~2500):
… we run tests (~5-20 per machine)
… each test has ~8 steps
… each step makes a connection to the DB, logs to C* the start time, sub 
invokes the test script which runs the step (2 mins to 20 hours — no C* usage 
during this part), and then captures the result to C* (end time, exit status, 
etc).

Today we’re not disconnecting C* during the “run the step” part - and we’re 
getting OperationTimedOut errors as we scale up the number of tests executing 
using our C* application.  My theory is that we’re overwhelming C* with the 
sheer number of (mostly idle) connections to our 3-node cluster.

I’m hoping someone has seen this sort of problem and can say “Yeah, that’s too 
many connections — I’m sure that’s your problem.”  or “We regularly make 12M 
connections per C* node — you’re screwed up in some other way — have you 
checked file descriptor limits?  What’s your Java __whatever__ setting?”  etc.

thanks Kurt.  :-)

- Max

> On Dec 14, 2017, at 6:19 am, kurt greaves  > wrote:
> 
> I see time outs and I immediately blame firewalls. Have you triple checked 
> then?
> Is this only occurring to a subset of clients?
> 
> Also, 3.0.6 is pretty dated and has many bugs, you should definitely upgrade 
> to the latest 3.0 (don't forget to read news.txt)
> On 14 Dec. 2017 19:18, "Max Campos"  > wrote:
> Hi -
> 
> We’re finally putting our new application under load, and we’re starting to 
> get this error message from the Python driver when under heavy load:
> 
> ('Unable to connect to any servers', {‘x.y.z.205': 
> OperationTimedOut('errors=None, last_host=None',), ‘x.y.z.204': 
> OperationTimedOut('errors=None, last_host=None',), ‘x.y.z.206': 
> OperationTimedOut('errors=None, last_host=None',)})' (22.7s)
> 
> Our cluster is running 3.0.6, has 3 nodes and we use RF=3, CL=QUORUM 
> reads/writes.  We have a few thousand machines which are each making 1-10 
> connections to C* at once, but each of these connections only reads/writes a 
> few records, waits several minutes, and then writes a few records — so while 
> netstat reports ~5K connections per node, they’re generally idle.  Peak 
> read/sec today was ~1500 per node, peak writes/sec was ~300 per node.  
> Read/write latencies peaked at 2.5ms.
> 
> Some questions:
> 1) Is anyone else out there making this many simultaneous connections?  Any 
> idea what a reasonable number of connections is, what is too many, etc?
> 
> 2) Any thoughts on which JMX metrics I should look at to better understand 
> what exactly is exploding?  Is there a “number of active connections” metric? 
>  We currently look at:
> - client reads/writes per sec
> - read/write latency
> - compaction tasks
> - repair tasks
> - disk used by node
> - disk used by table
> - avg partition size per table
> 
> 3) Any other advice?
> 
> I think I’ll try doing an explicit disconnect during the waiting period of 
> our application’s execution; so as to get the C* connection count down.  
> Hopefully that will solve the timeout problem.
> 
> Thanks for your help.
> 
> - Max
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> 
> For additional commands, e-mail: user-h...@cassandra.apache.org 
> 
> 
> 

Re: Bring 2 nodes down

[Jeff Jirsa]

Options, in order of desirability:

The "right" way to configure such a setup when creating the cluster would
be to define each physical machine as a "rack", and then use the right
replication/snitch configurations to give you rack awareness, so you
wouldn't have 2 replicas on the same physical machine.

If you haven't done that, your next option is probably to decommission one
of the VMs on the machine (or migrate it to another machine), so the
physical machine only has 1 replica of the data. Then you can turn the
machine off temporarily and only 1 replica will be down.

If neither of those are options, running repair will guarantee that at
least one other replica has the data (at least at the time you run repair),
but you'll lose quorum while the machine is down.




On Thu, Dec 14, 2017 at 11:00 AM, Alaa Zubaidi (PDF) 
wrote:

> Hi,
> I have a cluster of 8 Nodes, 4 physical machines 2 VMs each physical
> machine.
> RF=3, and we have read/write with QUORUM consistency requirement.
>
> One of the machines needs to be down for an hour or two to fix local disk.
> What is the best way to do that with out losing data?
>
> Regards
> -- Alaa
>
> *This message may contain confidential and privileged information. If it
> has been sent to you in error, please reply to advise the sender of the
> error and then immediately permanently delete it and all attachments to it
> from your systems. If you are not the intended recipient, do not read,
> copy, disclose or otherwise use this message or any attachments to it. The
> sender disclaims any liability for such unauthorized use. PLEASE NOTE that
> all incoming e-mails sent to PDF e-mail accounts will be archived and may
> be scanned by us and/or by external service providers to detect and prevent
> threats to our systems, investigate illegal or inappropriate behavior,
> and/or eliminate unsolicited promotional e-mails (“spam”). If you have any
> concerns about this process, please contact us at *
> *legal.departm...@pdf.com* *.*

Any Cassandra Backup and Restore tool like Cassandra Reaper?

[Harika Vangapelli -T (hvangape - AKRAYA INC at Cisco)]

Any Cassandra Backup and Restore tool like Cassandra Reaper for Repairs?

[http://wwwin.cisco.com/c/dam/cec/organizations/gmcc/services-tools/signaturetool/images/logo/logo_gradient.png]



Harika Vangapelli
Engineer - IT
hvang...@cisco.com
Tel:

Cisco Systems, Inc.



United States
cisco.com


[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif]Think before you 
print.

This email may contain confidential and privileged material for the sole use of 
the intended recipient. Any review, use, distribution or disclosure by others 
is strictly prohibited. If you are not the intended recipient (or authorized to 
receive for the recipient), please contact the sender by reply email and delete 
all copies of this message.
Please click 
here for 
Company Registration Information.

Re: Any Cassandra Backup and Restore tool like Cassandra Reaper?

[Roger Brown]

I've found nothing affordable that works with vnodes. If you have money,
you could use DataStax OpsCenter or Datos.io Recoverx.

I ended up creating a cron job to make snapshots along with
incremental_backups: true in the cassandra.yaml. And I'm thinking of
setting up a replication strategy so that one rack contains 1  replica of
each keyspace and then using r1soft to image each of those servers to tape
for offsite backup.


On Thu, Dec 14, 2017 at 1:30 PM Harika Vangapelli -T (hvangape - AKRAYA INC
at Cisco)  wrote:

> Any Cassandra Backup and Restore tool like Cassandra Reaper for Repairs?
>
>
>
> [image:
> http://wwwin.cisco.com/c/dam/cec/organizations/gmcc/services-tools/signaturetool/images/logo/logo_gradient.png]
>
>
>
> *Harika Vangapelli*
>
> Engineer - IT
>
> hvang...@cisco.com
>
> Tel:
>
> *Cisco Systems, Inc.*
>
>
>
>
> United States
> cisco.com
>
>
>
> [image: http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif]Think
> before you print.
>
> This email may contain confidential and privileged material for the sole
> use of the intended recipient. Any review, use, distribution or disclosure
> by others is strictly prohibited. If you are not the intended recipient (or
> authorized to receive for the recipient), please contact the sender by
> reply email and delete all copies of this message.
>
> Please click here
>  for
> Company Registration Information.
>
>
>

Re: Any Cassandra Backup and Restore tool like Cassandra Reaper?

[Rutvij Bhatt]

There is tablesnap/tablechop/tableslurp -
https://github.com/JeremyGrosser/tablesnap.


On Thu, Dec 14, 2017 at 3:49 PM Roger Brown <
roger.br...@perfectsearchcorp.com> wrote:

> I've found nothing affordable that works with vnodes. If you have money,
> you could use DataStax OpsCenter or Datos.io Recoverx.
>
> I ended up creating a cron job to make snapshots along with
> incremental_backups: true in the cassandra.yaml. And I'm thinking of
> setting up a replication strategy so that one rack contains 1  replica of
> each keyspace and then using r1soft to image each of those servers to tape
> for offsite backup.
>
>
> On Thu, Dec 14, 2017 at 1:30 PM Harika Vangapelli -T (hvangape - AKRAYA
> INC at Cisco)  wrote:
>
>> Any Cassandra Backup and Restore tool like Cassandra Reaper for Repairs?
>>
>>
>>
>> [image:
>> http://wwwin.cisco.com/c/dam/cec/organizations/gmcc/services-tools/signaturetool/images/logo/logo_gradient.png]
>>
>>
>>
>> *Harika Vangapelli*
>>
>> Engineer - IT
>>
>> hvang...@cisco.com
>>
>> Tel:
>>
>> *Cisco Systems, Inc.*
>>
>>
>>
>>
>> United States
>> cisco.com
>>
>>
>>
>> [image: http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif]Think
>> before you print.
>>
>> This email may contain confidential and privileged material for the sole
>> use of the intended recipient. Any review, use, distribution or disclosure
>> by others is strictly prohibited. If you are not the intended recipient (or
>> authorized to receive for the recipient), please contact the sender by
>> reply email and delete all copies of this message.
>>
>> Please click here
>>  for
>> Company Registration Information.
>>
>>
>>
>

Tablesnap with custom endpoint?

[Roger Brown]

I wanted to use tablesnap for backups. Instead of s3.amazonaws.com, I
wanted to use our own s3-compatible endpoint. I could never figure it out.
Do you know how to override the S3 endpoint tablesnap uses?

Roger

Re: Tablesnap with custom endpoint?

[Lerh Chuan Low]

Out of the box it assumes AWS S3 and is tailored to that using boto, so I
think you would have to checkout the repository and make changes to specify
a different endpoint (and submit a PR back to it if you feel it's useful :)
)

On 15 December 2017 at 08:03, Roger Brown  wrote:

> I wanted to use tablesnap for backups. Instead of s3.amazonaws.com, I
> wanted to use our own s3-compatible endpoint. I could never figure it out.
> Do you know how to override the S3 endpoint tablesnap uses?
>
> Roger
>
>

Re: Any Cassandra Backup and Restore tool like Cassandra Reaper?

[Lerh Chuan Low]

Tablesnap assumes S3, and tableslurp can set up the stage for restoring by
downloading the relevant SSTables (but then it's up to the operator to
complete the restore from there). Restoring (especially point-in-time
restore) isn't easy to handle so there aren't a lot available out there.

There's also Netflix's Priam https://github.com/Netflix/Priam but I think
it's a little bit old and is meant to run alongside C* as an agent and be
the agent for repairs, monitoring, backups and restores, configuring
Cassandra YAMLs...

One other one I've heard of is
https://github.com/tbarbugli/cassandra_snapshotter but there's no restore
yet.

On 15 December 2017 at 07:52, Rutvij Bhatt  wrote:

> There is tablesnap/tablechop/tableslurp - https://github.com/
> JeremyGrosser/tablesnap.
>
>
> On Thu, Dec 14, 2017 at 3:49 PM Roger Brown  perfectsearchcorp.com> wrote:
>
>> I've found nothing affordable that works with vnodes. If you have money,
>> you could use DataStax OpsCenter or Datos.io Recoverx.
>>
>> I ended up creating a cron job to make snapshots along with
>> incremental_backups: true in the cassandra.yaml. And I'm thinking of
>> setting up a replication strategy so that one rack contains 1  replica of
>> each keyspace and then using r1soft to image each of those servers to tape
>> for offsite backup.
>>
>>
>> On Thu, Dec 14, 2017 at 1:30 PM Harika Vangapelli -T (hvangape - AKRAYA
>> INC at Cisco)  wrote:
>>
>>> Any Cassandra Backup and Restore tool like Cassandra Reaper for Repairs?
>>>
>>>
>>>
>>> [image:
>>> http://wwwin.cisco.com/c/dam/cec/organizations/gmcc/services-tools/signaturetool/images/logo/logo_gradient.png]
>>>
>>>
>>>
>>> *Harika Vangapelli*
>>>
>>> Engineer - IT
>>>
>>> hvang...@cisco.com
>>>
>>> Tel:
>>>
>>> *Cisco Systems, Inc.*
>>>
>>>
>>>
>>>
>>> United States
>>> cisco.com
>>>
>>>
>>>
>>> [image: http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif]Think
>>> before you print.
>>>
>>> This email may contain confidential and privileged material for the sole
>>> use of the intended recipient. Any review, use, distribution or disclosure
>>> by others is strictly prohibited. If you are not the intended recipient (or
>>> authorized to receive for the recipient), please contact the sender by
>>> reply email and delete all copies of this message.
>>>
>>> Please click here
>>> 
>>> for Company Registration Information.
>>>
>>>
>>>
>>

Re: Tablesnap with custom endpoint?

[Jon Haddad]

Tablesnap uses boto, you may be able to override the S3 endpoint.  This Stack 
Overflow answer suggests it’s possible, but you might have to modify the 
tablesnap script a little: 
https://stackoverflow.com/questions/32618216/overwrite-s3-endpoint-using-boto3-configuration-file
 


> On Dec 14, 2017, at 4:03 PM, Roger Brown  
> wrote:
> 
> I wanted to use tablesnap for backups. Instead of s3.amazonaws.com 
> , I wanted to use our own s3-compatible endpoint. I 
> could never figure it out. Do you know how to override the S3 endpoint 
> tablesnap uses?
> 
> Roger
> 

DataStax Java driver QueryBuilder: CREATE table?

[Oliver Ruebenacker]

 Hello,

  I'm using the DataStax Java Driver, which has a QueryBuilder class to
construct CQL statements. I can see how to build SELECT, INSERT, TRUNCATE
etc statements, but I can't find how to build a CREATE statement. Am I
missing something?

  Thanks!

 Best, Oliver

-- 
Oliver Ruebenacker
Senior Software Engineer, Diabetes Portal
, Broad Institute

Re: DataStax Java driver QueryBuilder: CREATE table?

[Andy Tolbert]

Hi Oliver,

SchemaBuilder

enables building schema DDL statements like CREATE TABLE, KEYSPACE and so
on.  You can find some examples in the tests

.

Thanks,
Andy

On Thu, Dec 14, 2017 at 5:16 PM Oliver Ruebenacker  wrote:

>
>  Hello,
>
>   I'm using the DataStax Java Driver, which has a QueryBuilder class to
> construct CQL statements. I can see how to build SELECT, INSERT, TRUNCATE
> etc statements, but I can't find how to build a CREATE statement. Am I
> missing something?
>
>   Thanks!
>
>  Best, Oliver
>
>
> --
> Oliver Ruebenacker
> Senior Software Engineer, Diabetes Portal
> , Broad Institute
> 
>
>

Re: Batch : Isolation and Atomicity for same partition on multiple table

[Mickael Delanoë]

Thanks Jeff,
I am a little disappointed when you said the guarantee are even weeker.But
I will take a look on this and try to understand what is really done.



Le 13 déc. 2017 18:18, "Jeff Jirsa"  a écrit :

Entry point is here: https://github.com/apache/cassandra/blob/trunk/
src/java/org/apache/cassandra/cql3/statements/BatchStatement.java#L346 ,
which will call through to https://github.com/apache/
cassandra/blob/trunk/src/java/org/apache/cassandra/service/
StorageProxy.java#L938-L953

I believe the guarantees are weaker than the blog suggests, but it's
nuanced, and a lot of these types of questions come down to data model (you
can model it in a way that you can avoid problems with weaknesses in
isolation, but that requires a detailed explanation of your use case, etc).




On Wed, Dec 13, 2017 at 8:56 AM, Mickael Delanoë 
wrote:

> Hi Nicolas,
> Thanks for you answer.
> Is your assumption 100% sure ?
> Because the few test I did - using nodetools getendpoints - shown that the
> data for the two tables when I used the same partition key went to the same
> "nodes" . So I would have expected cassandra to be smart enough to apply
> them in the memtable in a single operation to achieve the isolation as the
> whole batch will be executed on a single node.
> Does anybody know where I can find, where the batch operations are
> processed in the Cassandra source code, so I could check how all this is
> processed ?
>
> Regards,
> Mickaël
>
>
>
> 2017-12-13 11:18 GMT+01:00 Nicolas Guyomar :
>
>> Hi Mickael,
>>
>> Partition are related to the table they exist in, so in your case, you
>> are targeting 2 partitions in 2 different tables.
>> Therefore, IMHO, you will only get atomicity using your batch statement
>>
>> On 11 December 2017 at 15:59, Mickael Delanoë 
>> wrote:
>>
>>> Hello,
>>>
>>> I have a question regarding batch isolation and atomicity with query
>>> using a same partition key.
>>>
>>> The Datastax documentation says about the batches :
>>> "Combines multiple DML statements to achieve atomicity and isolation
>>> when targeting a single partition or only atomicity when targeting multiple
>>> partitions. A batch applies all DMLs within a single partition before the
>>> data is available, ensuring atomicity and isolation.""
>>>
>>> But I try to find exactly what can be considered as a "single partition"
>>> and I cannot find a clear response yet. The examples and explanations
>>> always speak about partition with only one table used inside the batch. My
>>> concern is about partition when we use different table in a batch. So I
>>> would like some clarification.
>>>
>>> Here is my use case, I have 2 tables with the same partition-key which
>>> is "user_id" :
>>>
>>> CREATE TABLE tableA (
>>>user_id text,
>>>clustering text,
>>>value text,
>>>PRIMARY KEY (user_id, clustering));
>>>
>>> CREATE TABLE tableB (
>>>user_id text,
>>>clustering1 text,
>>>clustering2 text,
>>>value text,
>>>PRIMARY KEY (user_id, clustering1, clustering2));
>>>
>>> If I do a batch query like this :
>>>
>>> BEGIN BATCH
>>> INSERT INTO tableA (user_id, clustering, value) VALUES ('1234', 'c1',
>>> 'val1');
>>> INSERT INTO tableB (user_id, clustering1, clustering1, value) VALUES
>>> ('1234', 'cl1', 'cl2', 'avalue');
>>> APPLY BATCH;
>>>
>>> the DML statements uses the same partition-key, can we say they are
>>> targetting the same partition or, as the partition key are for different
>>> table, should we consider this is different partition? And so does this
>>> batch ensure atomicity and isolation (in the sense described in Datastax
>>> doc)? Or only atomicity?
>>>
>>> Thanks for you help,
>>> Mickaël Delanoë
>>>
>>
>>
>
>
> --
> Mickaël Delanoë
>

Re: Batch : Isolation and Atomicity for same partition on multiple table

[Jeff Jirsa]

Again, a lot of potential problems can be solved with data modeling - in 
particular consider things like conditional batches where the condition is on a 
static cell/column and writes go to different CQL rows. 

-- 
Jeff Jirsa


> On Dec 14, 2017, at 9:57 PM, Mickael Delanoë  wrote:
> 
> Thanks Jeff, 
> I am a little disappointed when you said the guarantee are even weeker.But I 
> will take a look on this and try to understand what is really done.
> 
> 
> 
> Le 13 déc. 2017 18:18, "Jeff Jirsa"  a écrit :
> Entry point is here: 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java#L346
>  , which will call through to 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageProxy.java#L938-L953
> 
> I believe the guarantees are weaker than the blog suggests, but it's nuanced, 
> and a lot of these types of questions come down to data model (you can model 
> it in a way that you can avoid problems with weaknesses in isolation, but 
> that requires a detailed explanation of your use case, etc).
> 
> 
> 
> 
>> On Wed, Dec 13, 2017 at 8:56 AM, Mickael Delanoë  
>> wrote:
>> Hi Nicolas, 
>> Thanks for you answer. 
>> Is your assumption 100% sure ?
>> Because the few test I did - using nodetools getendpoints - shown that the 
>> data for the two tables when I used the same partition key went to the same 
>> "nodes" . So I would have expected cassandra to be smart enough to apply 
>> them in the memtable in a single operation to achieve the isolation as the 
>> whole batch will be executed on a single node.
>> Does anybody know where I can find, where the batch operations are processed 
>> in the Cassandra source code, so I could check how all this is processed ?
>> 
>> Regards,
>> Mickaël
>> 
>> 
>> 
>> 2017-12-13 11:18 GMT+01:00 Nicolas Guyomar :
>>> Hi Mickael,
>>> 
>>> Partition are related to the table they exist in, so in your case, you are 
>>> targeting 2 partitions in 2 different tables.
>>> Therefore, IMHO, you will only get atomicity using your batch statement
>>> 
 On 11 December 2017 at 15:59, Mickael Delanoë  wrote:
 Hello,
 
 I have a question regarding batch isolation and atomicity with query using 
 a same partition key.
 
 The Datastax documentation says about the batches :
 "Combines multiple DML statements to achieve atomicity and isolation when 
 targeting a single partition or only atomicity when targeting multiple 
 partitions. A batch applies all DMLs within a single partition before the 
 data is available, ensuring atomicity and isolation.""
 
 But I try to find exactly what can be considered as a "single partition" 
 and I cannot find a clear response yet. The examples and explanations 
 always speak about partition with only one table used inside the batch. My 
 concern is about partition when we use different table in a batch. So I 
 would like some clarification.
 
 Here is my use case, I have 2 tables with the same partition-key which is 
 "user_id" :
 
 CREATE TABLE tableA (
user_id text, 
clustering text, 
value text, 
PRIMARY KEY (user_id, clustering));
 
 CREATE TABLE tableB (
user_id text, 
clustering1 text, 
clustering2 text, 
value text, 
PRIMARY KEY (user_id, clustering1, clustering2));
 
 If I do a batch query like this : 
 
 BEGIN BATCH 
 INSERT INTO tableA (user_id, clustering, value) VALUES ('1234', 'c1', 
 'val1');
 INSERT INTO tableB (user_id, clustering1, clustering1, value) VALUES 
 ('1234', 'cl1', 'cl2', 'avalue');
 APPLY BATCH;
 
 the DML statements uses the same partition-key, can we say they are 
 targetting the same partition or, as the partition key are for different 
 table, should we consider this is different partition? And so does this 
 batch ensure atomicity and isolation (in the sense described in Datastax 
 doc)? Or only atomicity?
 
 Thanks for you help, 
 Mickaël Delanoë
>>> 
>> 
>> 
>> 
>> -- 
>> Mickaël Delanoë
> 
>