No such thing as a stupid question☺ I know they exist in some nodes, but if they replicated correctly is a different story. I’m checking this one now,
Ok, hooked up OpsCenter to see what it was saying, Out of the 100 keyspaces creted, 9 are missing one CF 2 are missing two CF’s 1 is missing three CF’s It looks like the replication of the tables did not complete to all nodes? Looking at each of the 4 nodes at the keyspace with 3 missing CF’s (via CQLSH_HOST=x.x.x.x cqlsh & “Describe keyspace XXX;”) Node 1 : has all CF’s Node 2 : has all CF’s Node 3 : has all CF’s Node 4 : has all CF’s This is indeed very strange…. From: Carlos Alonso [mailto:i...@mrcalonso.com] Sent: 01 October 2015 12:05 To: user@cassandra.apache.org Subject: Re: Consistency Issues And that's a stupid one, I know, but does the column you're trying to access actually exist? Carlos Alonso | Software Engineer | @calonso<https://twitter.com/calonso> On 1 October 2015 at 11:09, Walsh, Stephen <stephen.wa...@aspect.com<mailto:stephen.wa...@aspect.com>> wrote: I did think of that and they are all the same version ☺ From: Carlos Alonso [mailto:i...@mrcalonso.com<mailto:i...@mrcalonso.com>] Sent: 01 October 2015 10:11 To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: Consistency Issues Hi Stephen. The UnknownColumnFamilyException made me thought of a possible schema disagreement in which any of your nodes has a different version and therefore you cannot reach quorum? Can you run nodetool describecluster and see if all nodes have the same schema versions? Cheers! Carlos Alonso | Software Engineer | @calonso<https://twitter.com/calonso> On 1 October 2015 at 09:49, Walsh, Stephen <stephen.wa...@aspect.com<mailto:stephen.wa...@aspect.com>> wrote: If you’re looking for the clean-up of the old gen in the jvm heap, it doesn’t happen. We have a new gen turning 15 times before its pushed to old gen. Seems all our data only has a TTL of 10 seconds – very little data is sent to the old gen. Add in heap size of 8GB with a new gen size of 2GB, I don’t think gc is our issue. I’m more worried about error messages in the Cassandra log file that state. UnknownColumnFamilyException reading from socket; closing org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=cf411b50-6785-11e5-a435-e7be20c92086 and cassandra OutboundTcpConnection.java:313 - error writing to Connection. But I really need to understand this best practice that was mentioned (on number of CF’s) by Jack Krupansky. Anyone more information on this? Many thanks for all your help guys keep it coming ☺ Steve From: Ricardo Sancho [mailto:sancho.rica...@gmail.com<mailto:sancho.rica...@gmail.com>] Sent: 01 October 2015 09:39 To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: RE: Consistency Issues Can you tell us how much time your gcs are taking? Do you see any especially long ones? On 1 Oct 2015 09:37, "Walsh, Stephen" <stephen.wa...@aspect.com<mailto:stephen.wa...@aspect.com>> wrote: There is no load balancer in front of Cassandra, it’s in front of our application. Everyone seems hung up on this point? But it’s not the root causing of the inconsistency issue. Can anyone verify the best practice for number of CF’s? From: Robert Coli [mailto:rc...@eventbrite.com<mailto:rc...@eventbrite.com>] Sent: 30 September 2015 18:45 To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: Consistency Issues On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen <stephen.wa...@aspect.com<mailto:stephen.wa...@aspect.com>> wrote: We never had these issue with our first run. Its only when we added another 25% of writes. As Jack said, you are probably pushing your GC over a threshold, leading to long pause times and inability to meet quorum. As Sebastian said, you probably shouldn't need a load balancer in front of Cassandra. =Rob This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain information that is confidential. If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immediately, delete it from your system and destroy any copies. You may not further disclose or distribute this email or its attachments. This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain information that is confidential. If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immediately, delete it from your system and destroy any copies. You may not further disclose or distribute this email or its attachments. This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain information that is confidential. If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immediately, delete it from your system and destroy any copies. You may not further disclose or distribute this email or its attachments. This email (including any attachments) is proprietary to Aspect Software, Inc. and may contain information that is confidential. If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immediately, delete it from your system and destroy any copies. You may not further disclose or distribute this email or its attachments.
