Re: upgrading 2.1.x cluster with ec2multiregionsnitch system.peers "corruption"

2019-03-28 Thread Oleksandr Shulgin
On Wed, Mar 27, 2019 at 6:36 PM Carl Mueller wrote: > > EIPs per the aws experts cost money, > >From what I know they only cost you when you're not using them. This page shows that you are also charged if you remap them too often (more then 100 times a month), this I didn't realize: https://aws

Re: upgrading 2.1.x cluster with ec2multiregionsnitch system.peers "corruption"

2019-03-27 Thread Carl Mueller
We are probably going to just have a VM startup script for now that automatically updates the yaml on instance restart. It seems to be the least-sucky approach at this point. On Wed, Mar 27, 2019 at 12:36 PM Carl Mueller wrote: > I filed https://issues.apache.org/jira/browse/CASSANDRA-15068 > >

Re: upgrading 2.1.x cluster with ec2multiregionsnitch system.peers "corruption"

2019-03-27 Thread Carl Mueller
I filed https://issues.apache.org/jira/browse/CASSANDRA-15068 EIPs per the aws experts cost money, are limited in resources (we have a lot of VMs) and cause a lot of headaches in our autoscaling / infrastructure as code systems. On Wed, Mar 27, 2019 at 12:35 PM Carl Mueller wrote: > I'll try to

Re: upgrading 2.1.x cluster with ec2multiregionsnitch system.peers "corruption"

2019-03-27 Thread Carl Mueller
I'll try to get a replicated error message, but it was along the lines of what is in the gossip strategy agnostic description in cassandra.yaml comments of what happens when you set rpc_address to 0.0.0.0: you must then set broadcast_rpc_address. On Wed, Mar 27, 2019 at 3:21 AM Oleksandr Shulgin <

Re: upgrading 2.1.x cluster with ec2multiregionsnitch system.peers "corruption"

2019-03-27 Thread Oleksandr Shulgin
On Tue, Mar 26, 2019 at 10:28 PM Carl Mueller wrote: > - the AWS people say EIPs are a PITA. > Why? > - if we hardcode the global IPs in the yaml, then yaml editing is required > for the occaisional hard instance reboot in aws and its attendant global ip > reassignment > - if we try leaving br

Re: upgrading 2.1.x cluster with ec2multiregionsnitch system.peers "corruption"

2019-03-26 Thread Carl Mueller
- the AWS people say EIPs are a PITA. - if we hardcode the global IPs in the yaml, then yaml editing is required for the occaisional hard instance reboot in aws and its attendant global ip reassignment - if we try leaving broadcast_rpc_address blank, null , or commented out with rpc_address set to

Re: upgrading 2.1.x cluster with ec2multiregionsnitch system.peers "corruption"

2019-03-26 Thread Oleksandr Shulgin
On Tue, Mar 26, 2019 at 5:49 PM Carl Mueller wrote: > Looking at the code it appears it shouldn't matter what we set the yaml > params to. The Ec2MultiRegionSnitch should be using the aws metadata > 169.254.169.254 to pick up the internal/external ips as needed. > This is somehow my expectation

Re: upgrading 2.1.x cluster with ec2multiregionsnitch system.peers "corruption"

2019-03-26 Thread Carl Mueller
Looking at the code it appears it shouldn't matter what we set the yaml params to. The Ec2MultiRegionSnitch should be using the aws metadata 169.254.169.254 to pick up the internal/external ips as needed. I think I'll just have to dig in to the code differences between 2.1 and 2.2. We don't want t

Re: upgrading 2.1.x cluster with ec2multiregionsnitch system.peers "corruption"

2019-03-26 Thread Oleksandr Shulgin
On Mon, Mar 25, 2019 at 11:13 PM Carl Mueller wrote: > > Since the internal IPs are given when the client app connects to the > cluster, the client app cannot communicate with other nodes in other > datacenters. > Why should it? The client should only connect to its local data center and leave

upgrading 2.1.x cluster with ec2multiregionsnitch system.peers "corruption"

2019-03-25 Thread Carl Mueller
This is a multi-dc cluster with public IPs for the nodes and also addressed with private IPs as well in AWS. The apps connect via java-driver to a public IP. When we built the 2.1.X cluster with ec2multiregionsnitch, the system.peers table had public ips for the nodes in the rpc_address column. A