It would be handy to work out what the corruption is. Could you snapshot the system keyspace and store it somewhere, just incase we can look at it later ?
> Is there a way I can confirm this Errors in the client and/or the server log is the the traditional way. > go about cleaning up/restoring the proper schema? If you need to get it back, and can handle the down time, the simple thing is drop the KS and re-create it. Remember to take a snapshot first. Drop keyspace takes one but it's the sort of thing I would do myself. Or you can try _try_ nodetool resetlocalschema. Without knowing what the error is it's hard to say if it would work. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 29/08/2012, at 9:10 AM, Bryce Godfrey <bryce.godf...@azaleos.com> wrote: > I believe what may be really going on is that my schema is in a bad or > corrupt state. I also have one keyspace that I just cannot drop an existing > column family from even though it shows no errors. > > So right now I was able to get 4 of my 6 keyspaces over to Network Topology > strategy. > > I think I got into this bad state after pointing Opscenter at this cluster > for the first time, as it started throwing errors after that and crashed a > couple of my nodes until I stopped it and its agents. > > Is there a way I can confirm this or go about cleaning up/restoring the > proper schema? > > From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com] > Sent: Tuesday, August 28, 2012 11:09 AM > To: user@cassandra.apache.org > Subject: RE: Expanding cluster to include a new DR datacenter > > So in an interesting turn of events, this works on my other 4 keyspaces but > just not this ‘EBonding’ one which will not recognize the changes. I can > probably get around this by dropping and re-creating this keyspace since its > uptime is not too important for us. > > [default@AlertStats] describe AlertStats; > Keyspace: AlertStats: > Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy > Durable Writes: true > Options: [Fisher:3] > > From: Mohit Anchlia [mailto:mohitanch...@gmail.com] > Sent: Monday, August 27, 2012 3:50 PM > To: user@cassandra.apache.org > Subject: Re: Expanding cluster to include a new DR datacenter > > Can you describe your schema again with TierPoint in it? > > On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <bryce.godf...@azaleos.com> > wrote: > Same results. I restarted the node also to see if it just wasn’t picking up > the changes and it still shows Simple. > > When I specify the DC for strategy_options I should be using the DC name from > properfy file snitch right? Ours is “Fisher” and “TierPoint” so that’s what > I used. > > From: Mohit Anchlia [mailto:mohitanch...@gmail.com] > Sent: Monday, August 27, 2012 1:21 PM > > To: user@cassandra.apache.org > Subject: Re: Expanding cluster to include a new DR datacenter > > In your update command is it possible to specify RF for both DC? You could > just do DC1:2, DC2:0. > > On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <bryce.godf...@azaleos.com> > wrote: > Show schema output show the simple strategy still > [default@unknown] show schema EBonding; > create keyspace EBonding > with placement_strategy = 'SimpleStrategy' > and strategy_options = {replication_factor : 2} > and durable_writes = true; > > This is the only thing I see in the system log at the time on all the nodes: > > INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java (line > 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228 > serialized/live bytes, 4 ops) > INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) > Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 > ops) > INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) > Completed flushing > /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db > (241 bytes) for commitlog p$ > > > Should I turn the logging level up on something to see some more info maybe? > > From: aaron morton [mailto:aa...@thelastpickle.com] > Sent: Monday, August 27, 2012 1:35 AM > > To: user@cassandra.apache.org > Subject: Re: Expanding cluster to include a new DR datacenter > > I did a quick test on a clean 1.1.4 and it worked > > Can you check the logs for errors ? Can you see your schema change in there ? > > Also what is the output from show schema; in the cli ? > > Cheers > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 25/08/2012, at 6:53 PM, Bryce Godfrey <bryce.godf...@azaleos.com> wrote: > > > Yes > > [default@unknown] describe cluster; > Cluster Information: > Snitch: org.apache.cassandra.locator.PropertyFileSnitch > Partitioner: org.apache.cassandra.dht.RandomPartitioner > Schema versions: > 9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, > 10.20.8.1, 10.20.8.2, 10.20.8.3] > > From: Mohit Anchlia [mailto:mohitanch...@gmail.com] > Sent: Friday, August 24, 2012 1:55 PM > To: user@cassandra.apache.org > Subject: Re: Expanding cluster to include a new DR datacenter > > That's interesting can you do describe cluster? > > On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <bryce.godf...@azaleos.com> > wrote: > So I’m at the point of updating the keyspaces from Simple to NetworkTopology > and I’m not sure if the changes are being accepted using Cassandra-cli. > > I issue the change: > > [default@EBonding] update keyspace EBonding > ... with placement_strategy = > 'org.apache.cassandra.locator.NetworkTopologyStrategy' > ... and strategy_options={Fisher:2}; > 9511e292-f1b6-3f78-b781-4c90aeb6b0f6 > Waiting for schema agreement... > ... schemas agree across the cluster > > Then I do a describe and it still shows the old strategy. Is there something > else that I need to do? I’ve exited and restarted Cassandra-cli and it still > shows the SimpleStrategy for that keyspace. Other nodes show the same > information. > > [default@EBonding] describe EBonding; > Keyspace: EBonding: > Replication Strategy: org.apache.cassandra.locator.SimpleStrategy > Durable Writes: true > Options: [replication_factor:2] > > > From: Bryce Godfrey [mailto:bryce.godf...@azaleos.com] > Sent: Thursday, August 23, 2012 11:06 AM > To: user@cassandra.apache.org > Subject: RE: Expanding cluster to include a new DR datacenter > > Thanks for the information! Answers my questions. > > From: Tyler Hobbs [mailto:ty...@datastax.com] > Sent: Wednesday, August 22, 2012 7:10 PM > To: user@cassandra.apache.org > Subject: Re: Expanding cluster to include a new DR datacenter > > If you didn't see this particular section, you may find it > useful:http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster > > Some comments inline: > > On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <bryce.godf...@azaleos.com> > wrote: > We are in the process of building out a new DR system in another Data Center, > and we want to mirror our Cassandra environment to that DR. I have a couple > questions on the best way to do this after reading the documentation on the > Datastax website. We didn’t initially plan for this to be a DR setup when > first deployed a while ago due to budgeting, but now we need to. So I’m just > trying to nail down the order of doing this as well as any potential issues. > > For the nodes, we don’t plan on querying the servers in this DR until we fail > over to this data center. We are going to have 5 similar nodes in the DR, > should I join them into the ring at token+1? > > Join them at token+10 just to leave a little space. Make sure you're using > LOCAL_QUORUM for your queries instead of regular QUORUM. > > > All keyspaces are set to the replication strategy of SimpleStrategy. Can I > change the replication strategy after joining the new nodes in the DR to > NetworkTopologyStategy with the updated replication factor for each dr? > > Switch your keyspaces over to NetworkTopologyStrategy before adding the new > nodes. For the strategy options, just list the first dc until the second is > up (e.g. {main_dc: 3}). > > > Lastly, is changing snitch from default of SimpleSnitch to > RackInferringSnitch going to cause any issues? Since its in the > Cassandra.yaml file I assume a rolling restart to pick up the value would be > ok? > > This is the first thing you'll want to do. Unless your node IPs would > naturally put all nodes in a DC in the same rack, I recommend using > PropertyFileSnitch, explicitly using the same rack. (I tend to prefer > PFSnitch regardless; it's harder to accidentally mess up.) A rolling restart > is required to pick up the change. Make sure to fill out > cassandra-topology.properties first if using PFSnitch. > > > This is all on Cassandra 1.1.4, Thanks for any help! > > > > > > -- > Tyler Hobbs > DataStax > > >
