Re: How to prevent the removed DC comes back automactically?

[Mark Reddy]

>
> How can we prevent a disconnected DC from coming back automatically?


You could use firewall rules to prevent the disconnected DC from contacting
your live DCs when it becomes live again


Mark


On Thu, Aug 14, 2014 at 6:48 AM, Lu, Boying  wrote:

> Hi, All,
>
>
>
> We are using Cassandra 2.0.7 in a multi DCs environments.  If a connected
> DC is powered off, we use the ‘nodetool removenode’ command to remove it
> from the connected DCs.
>
> But we found that once the disconnected DC is powered on, it will connect
> to other DCs automatically.
>
>
>
> How can we prevent a disconnected DC from coming back automatically?
>
>
>
> Thanks a lot
>
>
>
> Boying
>

Re: How to prevent the removed DC comes back automactically?

[Artur Kronenberg]

Hey,

not sure if that's what you're looking for but you can use 
auto_bootstrap=false in your yaml file to prevent nodes from 
bootstrapping themselves on startup. This option has been removed and 
the default is true. You can add it to your configuration though. 
There's a bit of documentation here:


http://www.datastax.com/documentation/cassandra/1.2/cassandra/configuration/configCassandra_yaml_r.html

-- artur

On 14/08/14 06:48, Lu, Boying wrote:


Hi, All,

We are using Cassandra 2.0.7 in a multi DCs environments.  If a 
connected DC is powered off, we use the ‘nodetool removenode’ command 
to remove it from the connected DCs.


But we found that once the disconnected DC is powered on, it will 
connect to other DCs automatically.


How can we prevent a disconnected DC from coming back automatically?

Thanks a lot

Boying

RE: How to prevent the removed DC comes back automactically?

[Lu, Boying]

Thanks a lot.

But we want to block all the communications to/from the disconnected VDC 
without reboot it.

From: Artur Kronenberg [mailto:artur.kronenb...@openmarket.com]
Sent: 2014年8月14日 17:00
To: user@cassandra.apache.org
Subject: Re: How to prevent the removed DC comes back automactically?

Hey,

not sure if that's what you're looking for but you can use auto_bootstrap=false 
in your yaml file to prevent nodes from bootstrapping themselves on startup. 
This option has been removed and the default is true. You can add it to your 
configuration though. There's a bit of documentation here:

http://www.datastax.com/documentation/cassandra/1.2/cassandra/configuration/configCassandra_yaml_r.html

-- artur

On 14/08/14 06:48, Lu, Boying wrote:
Hi, All,

We are using Cassandra 2.0.7 in a multi DCs environments.  If a connected DC is 
powered off, we use the ‘nodetool removenode’ command to remove it from the 
connected DCs.
But we found that once the disconnected DC is powered on, it will connect to 
other DCs automatically.

How can we prevent a disconnected DC from coming back automatically?

Thanks a lot

Boying

Re: Secondary indexes not working properly since 2.1.0-rc2 ?

[Fabrice Larcher]

Hello, I created https://issues.apache.org/jira/browse/CASSANDRA-7766 about
that

Fabrice LARCHER


2014-08-13 14:58 GMT+02:00 DuyHai Doan :

> Hello Fabrice.
>
>  A quick hint, try to create your secondary index WITHOUT the "IF NOT
> EXISTS" clause to see if you still have the bug.
>
> Another idea is to activate query tracing on client side to see what's
> going on underneath.
>
>
> On Wed, Aug 13, 2014 at 2:48 PM, Fabrice Larcher <
> fabrice.larc...@level5.fr> wrote:
>
>> Hello,
>>
>> I have used C* 2.1.0-rc1, 2.1.0-rc2, 2.1.0-rc3 and I currently use
>> 2.1.0-rc5. Since 2.1.0-rc2, it appears that the secondary indexes are not
>> always working. Just after the INSERT of a row, the index seems to be
>> there. But after a while (I do not know when or why), SELECT statements
>> based on any secondary index do not return the corresponding row(s)
>> anymore. I noticed that a restart of C* may have an impact (the data
>> inserted before the restart may be seen through the index, even if it was
>> not returned before the restart).
>>
>> Here is a use-case example (in order to clarify my request) :
>>
>> CREATE TABLE IF NOT EXISTS ks.cf ( k int PRIMARY KEY, ind ascii, value
>> text);
>> CREATE INDEX IF NOT EXISTS ks_cf_index ON ks.cf(ind);
>> INSERT INTO ks.cf (k, ind, value) VALUES (1, 'toto', 'Hello');
>> SELECT * FROM ks.cf WHERE ind = 'toto'; // Returns no result after a
>> while
>>
>> The last SELECT statement may or may not return a row depending on the
>> instant of the request. I experienced that with 2.1.0-rc5 through CQLSH
>> with clusters of one and two nodes. Since it depends on the instant of the
>> request, I am not able to deliver any way to reproduce that systematically
>> (It appears to be linked with some scheduled job inside C*).
>>
>> Is anyone working on this issue ?
>> Am I possibly missing some configuration that prevent secondary indexes
>> to return empty result ?
>> Should I rather create a new table for each secondary index and remove
>> secondary indexes ?
>>
>> Many thanks for your support.
>>
>> ​Fabrice​
>>
>>
>

A question to nodetool removenode command

[Lu, Boying]

Hi, All,

We have a Cassandra 2.0.7 running in three connected DCs, say DC1, DC2 and DC3.

DC3 is powered off, so we run  'nodetool removenode' command on DC1 to remove 
all nodes of DC3.
Do we need to run the same command on DC2?

Thanks

Boying

Re: A question to nodetool removenode command

[Mark Reddy]

Hi,

Gossip will propagate to all nodes in a cluster. So if you have a cluster
spanning DC1, DC2 and DC3 and you then remove all nodes in DC3 via nodetool
removenode from a node in DC1, all nodes in both DC1 and DC2 will be
informed of the nodes removal so no need to run it from a node in DC2.


Mark


On Thu, Aug 14, 2014 at 10:20 AM, Lu, Boying  wrote:

> Hi, All,
>
>
>
> We have a Cassandra 2.0.7 running in three connected DCs, say DC1, DC2 and
> DC3.
>
>
>
> DC3 is powered off, so we run  ‘nodetool removenode’ command on DC1 to
> remove all nodes of DC3.
>
> Do we need to run the same command on DC2?
>
>
>
> Thanks
>
>
>
> Boying
>

RE: A question to nodetool removenode command

[Lu, Boying]

Thanks a lot ☺

From: Mark Reddy [mailto:mark.re...@boxever.com]
Sent: 2014年8月14日 18:02
To: user@cassandra.apache.org
Subject: Re: A question to nodetool removenode command

Hi,

Gossip will propagate to all nodes in a cluster. So if you have a cluster 
spanning DC1, DC2 and DC3 and you then remove all nodes in DC3 via nodetool 
removenode from a node in DC1, all nodes in both DC1 and DC2 will be informed 
of the nodes removal so no need to run it from a node in DC2.


Mark

On Thu, Aug 14, 2014 at 10:20 AM, Lu, Boying 
mailto:boying...@emc.com>> wrote:
Hi, All,

We have a Cassandra 2.0.7 running in three connected DCs, say DC1, DC2 and DC3.

DC3 is powered off, so we run  ‘nodetool removenode’ command on DC1 to remove 
all nodes of DC3.
Do we need to run the same command on DC2?

Thanks

Boying

Communication between data-centers

[Rene Kochen]

Hi all,

I have a question about communication between two data-centers, both with
replication-factor three.

If I read data using local_quorum from datacenter1, I see that digest
requests are sent to datacenter2. This is for read-repair I guess. How can
I prevent this from happening? Setting read_repair_chance does not help,
since it is ignored when using local_quorum.

In general, is read-repair between data-centers a performance issue when
doing a lot of local reads with local_quorum?

Thanks!

Rene

Re: Communication between data-centers

[DuyHai Doan]

dclocal_read_repair_chance option on the table is your friend

http://www.datastax.com/documentation/cassandra/2.0/cassandra/reference/referenceTableAttributes.html?scroll=reference_ds_zyq_zmz_1k__dclocal_read_repair_chance


On Thu, Aug 14, 2014 at 4:53 PM, Rene Kochen 
wrote:

> Hi all,
>
> I have a question about communication between two data-centers, both with
> replication-factor three.
>
> If I read data using local_quorum from datacenter1, I see that digest
> requests are sent to datacenter2. This is for read-repair I guess. How can
> I prevent this from happening? Setting read_repair_chance does not help,
> since it is ignored when using local_quorum.
>
> In general, is read-repair between data-centers a performance issue when
> doing a lot of local reads with local_quorum?
>
> Thanks!
>
> Rene
>

Re: Communication between data-centers

[Rene Kochen]

I am using 1.0.11, so I only have read_repair_chance.

However, after testing I see that read_repair_chance does work for
local_quorum.

Based on this site:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureClientRequestsRead_c.html
I got the impression that read_repair_chance only applies to consistency
level ONE.

However, when I test with local_quorum, I see that the read_repair_chance
is honored. In my test, there are two data-centers of three nodes each and
RF = 3 for both data-centers. Read-repair chance is set to 0.1

>From the logging I see that read-repair is most of the times not triggered.

Blockfor/repair is 2/false; setting up requests to /10.80.88.143,/
10.81.129.87 -> most cases
Blockfor/repair is 2/true; setting up requests to /10.80.88.143,/
10.81.129.87,/10.80.59.90,/10.80.30.12,/10.50.85.77,/10.80.19.244

Thanks!

Rene


2014-08-14 17:31 GMT+02:00 DuyHai Doan :

> dclocal_read_repair_chance option on the table is your friend
>
>
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/reference/referenceTableAttributes.html?scroll=reference_ds_zyq_zmz_1k__dclocal_read_repair_chance
>
>
> On Thu, Aug 14, 2014 at 4:53 PM, Rene Kochen 
> wrote:
>
>> Hi all,
>>
>> I have a question about communication between two data-centers, both with
>> replication-factor three.
>>
>> If I read data using local_quorum from datacenter1, I see that digest
>> requests are sent to datacenter2. This is for read-repair I guess. How can
>> I prevent this from happening? Setting read_repair_chance does not help,
>> since it is ignored when using local_quorum.
>>
>> In general, is read-repair between data-centers a performance issue when
>> doing a lot of local reads with local_quorum?
>>
>> Thanks!
>>
>> Rene
>>
>
>

Re: Communication between data-centers

[Robert Coli]

On Thu, Aug 14, 2014 at 9:24 AM, Rene Kochen 
wrote:

> I am using 1.0.11, so I only have read_repair_chance.
>

I'm sure this goes without saying, but you should upgrade to the head of
1.2.x (probably via 1.1.x) ASAP. I would not want to operate 1.0.11 in
production in 2014.

=Rob

Re: How to prevent the removed DC comes back automactically?

[Robert Coli]

On Thu, Aug 14, 2014 at 1:59 AM, Artur Kronenberg <
artur.kronenb...@openmarket.com> wrote:

>  not sure if that's what you're looking for but you can use
> auto_bootstrap=false in your yaml file to prevent nodes from bootstrapping
> themselves on startup. This option has been removed and the default is
> true. You can add it to your configuration though. There's a bit of
> documentation here:
>
>
> http://www.datastax.com/documentation/cassandra/1.2/cassandra/configuration/configCassandra_yaml_r.html
>

That's not what auto_bootstrap:false is for, the formerly-dead nodes will
contact the currently-live nodes, which is what OP does not want.

Also if those nodes had been removed, they would take over ranges that OP
does not want them to. OP should definitely not use auto_bootstrap:false
here.

https://engineering.eventbrite.com/changing-the-ip-address-of-a-cassandra-node-with-auto_bootstrapfalse/

=Rob

question about OpsCenter agent

[Clint Kelly]

Hi all,

I just installed DataStax Enterprise 4.5.  I installed OpsCenter
Server on one of my four machines.  The port that OpsCenter usually
uses () was used by something else, so I modified
/usr/share/opscenter/conf/opscenterd.conf to set the port to 8889.

When I log into OpsCenter, it says "0 of 4 agents connected."

I have started datastax-agent on all four of the nodes in my Cassandra
cluster.  Is there another OpsCenter agent that I have to start?  I'm
not sure how to fix this.

I clicked on the "Fix" link and it prompted me "You have nodes that
need the OpsCenter agent installed."  How do I install the OpsCenter
agent without using this GUI?  I don't recall getting prompted for
that when I installed DSE.

Best regards,
Clint

Could table partitioning be implemented using a customer compaction strategy?

[Kevin Burton]

We use log structured tables to hold logs for analysis.

It's basically append only, and immutable.  Every record has a timestamp
for each record inserted.

Having this in ONE big monolithic table can be problematic.

1.  compactions have to compact old data that might not even be used often.

2.  it might be nice to not have the old data touched on disk so that your
can just use it for map reduce.  Being able to fadvise away old data so
that it's not in cache can be valuable.

3.  the ability to drop large chunks of old data is also useful .  For
example, if you run out of disk space, you can just drop the oldest day's
worth of data without having to use tombstones.

MySQL has a somewhat decent partition engine:

http://dev.mysql.com/doc/refman/5.1/en/partitioning.html

It seems like this come be easily implemented using a custom compaction
strategy.

Essentially, you would take each SSTable and first group them into
partitions.  So if you were using day partitions, you could take all
SSTables for that day, and then use another , nested compaction strategy,
like leveled, on just those SSTables.

The older days would yield one SSTable per day, once all the individual
SSTables are compacted.   For a month, you would need a minimum of 30
SSTables.

You would need to implement some custom ways to prune the older partitions.
  And you'd also need some way to define the partitions.

… but maybe an initial prototype could just read from a configuration file,
or another system table which defines them…

(just thinking out loud)

Kevin

-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile

Bootstrap failures: unable to find sufficient sources for streaming range

[Peter Haggerty]

When adding nodes via bootstrap to a 27 node 2.0.9 cluster with a
cluster-wide phi_convict_threshold of 12 the nodes fail to bootstrap.
This worked a half dozen times in the past few weeks as we've scaled
this cluster from 21 to 24 and then to 27 nodes. There have been no
configuration or Cassandra version changes since the cluster was
booted several weeks ago. This is adding a single node at a time just
as we've done before.

When using a phi of 16 on the node that is bootstrapping we've had one
failure and one success. Adjusting RING_DELAY from 30 seconds to 45
doesn't appear to help.

We have DEBUG level logs for a successful attempt and a failed attempt
but they seem quite similar in behavior, even bad behavior like ERRORS
reported by MigrationTask.java and FailureDetector.java.


In the failed attempts the bootstrap immediately falls over with
"unable to find ...":
java.lang.IllegalStateException: unable to find sufficient sources for
streaming range (1445230693272554509,1449885986247309687]

The logs clearly show that it knows where those ranges can be found:
DEBUG [main] 2014-08-14 18:56:45,019 RangeStreamer.java (line 122)
Bootstrap: range (1445230693272554509,1449885986247309687] exists on
/10.11.12.13
DEBUG [main] 2014-08-14 18:56:45,019 RangeStreamer.java (line 122)
Bootstrap: range (1445230693272554509,1449885986247309687] exists on
/10.11.12.14

If you look for the various non-debug log lines for these addresses
you'll see some odd behavior, but you see similar odd behavior in the
logs of the node that is able to successfully bootstrap.
$ grep '10.11.12.13' system.log | grep -v ^DEBUG
 INFO [GossipStage:1] 2014-08-14 18:56:09,870 Gossiper.java (line 910)
Node /10.11.12.13 is now part of the cluster
 INFO [GossipStage:1] 2014-08-14 18:56:15,267 Gossiper.java (line 910)
Node /10.11.12.13 is now part of the cluster
 INFO [HANDSHAKE-/10.11.12.13] 2014-08-14 18:56:45,571
OutboundTcpConnection.java (line 386) Handshaking version with
/10.11.12.13
 INFO [RequestResponseStage:5] 2014-08-14 18:56:45,579 Gossiper.java
(line 876) InetAddress /10.11.12.13 is now UP
 INFO [RequestResponseStage:2] 2014-08-14 18:56:45,579 Gossiper.java
(line 876) InetAddress /10.11.12.13 is now UP
 INFO [RequestResponseStage:7] 2014-08-14 18:56:45,580 Gossiper.java
(line 876) InetAddress /10.11.12.13 is now UP
 INFO [RequestResponseStage:1] 2014-08-14 18:56:45,579 Gossiper.java
(line 876) InetAddress /10.11.12.13 is now UP
 INFO [RequestResponseStage:8] 2014-08-14 18:56:45,579 Gossiper.java
(line 876) InetAddress /10.11.12.13 is now UP

$ grep '10.11.12.14' system.log | grep -v ^DEBUG
 INFO [GossipStage:1] 2014-08-14 18:56:09,865 Gossiper.java (line 910)
Node /10.11.12.14 is now part of the cluster
 INFO [GossipStage:1] 2014-08-14 18:56:15,106 Gossiper.java (line 910)
Node /10.11.12.14 is now part of the cluster

The general order of events, as told by JOINING lines:
$ grep 'JOINING' system.log
 INFO [main] 2014-08-14 18:56:10,638 StorageService.java (line 1009)
JOINING: waiting for ring information
 INFO [main] 2014-08-14 18:56:14,640 StorageService.java (line 1009)
JOINING: schema complete, ready to bootstrap
 INFO [main] 2014-08-14 18:56:14,640 StorageService.java (line 1009)
JOINING: waiting for pending range calculation
 INFO [main] 2014-08-14 18:56:14,641 StorageService.java (line 1009)
JOINING: calculation complete, ready to bootstrap
 INFO [main] 2014-08-14 18:56:14,641 StorageService.java (line 1009)
JOINING: getting bootstrap token
 INFO [main] 2014-08-14 18:56:14,712 StorageService.java (line 1009)
JOINING: sleeping 3 ms for pending range setup
 INFO [main] 2014-08-14 18:56:44,713 StorageService.java (line 1009)
JOINING: Starting to bootstrap...


Thanks in advance for any suggestions.


Peter

Re: question about OpsCenter agent

[Mark Reddy]

Hi Clint,

You need to configure the datastax-agents so they know what machine
OpsCenter is running on. To do this you will need to edit the address.yaml
of the datastax-agent, located in /var/lib/datastax-agent/conf/. In this
file you need to add the following line:

stomp_interface: 


This will allow your agents to connect to the OpsCenter server (opscenterd)
over the stomp protocol (stomp.github.io/). For a more detailed look at the
configuration options available to the datastax-agent see this page:
datastax.com/documentation/opscenter/5.0/opsc/configure/agentAddressConfiguration.html


Mark



On Fri, Aug 15, 2014 at 3:32 AM, Clint Kelly  wrote:

> Hi all,
>
> I just installed DataStax Enterprise 4.5.  I installed OpsCenter
> Server on one of my four machines.  The port that OpsCenter usually
> uses () was used by something else, so I modified
> /usr/share/opscenter/conf/opscenterd.conf to set the port to 8889.
>
> When I log into OpsCenter, it says "0 of 4 agents connected."
>
> I have started datastax-agent on all four of the nodes in my Cassandra
> cluster.  Is there another OpsCenter agent that I have to start?  I'm
> not sure how to fix this.
>
> I clicked on the "Fix" link and it prompted me "You have nodes that
> need the OpsCenter agent installed."  How do I install the OpsCenter
> agent without using this GUI?  I don't recall getting prompted for
> that when I installed DSE.
>
> Best regards,
> Clint
>