You're running describe with CL quorum aren't you? To see the inconsistency you'd have to check the system.schema_column family tables on each node. On Oct 1, 2015 8:07 AM, "Walsh, Stephen" <stephen.wa...@aspect.com> wrote:
> No such thing as a stupid questionJ > > I know they exist in some nodes, but if they replicated correctly is a > different story. > > I’m checking this one now, > > > > Ok, hooked up OpsCenter to see what it was saying, > > Out of the 100 keyspaces creted, > > 9 are missing one CF > > 2 are missing two CF’s > > 1 is missing three CF’s > > > > It looks like the replication of the tables did not complete to all nodes? > > > > Looking at each of the 4 nodes at the keyspace with 3 missing CF’s > > (via CQLSH_HOST=x.x.x.x cqlsh & “Describe keyspace XXX;”) > > > > Node 1 : has all CF’s > > Node 2 : has all CF’s > > Node 3 : has all CF’s > > Node 4 : has all CF’s > > > > > > This is indeed very strange…. > > > > > > *From:* Carlos Alonso [mailto:i...@mrcalonso.com] > *Sent:* 01 October 2015 12:05 > *To:* user@cassandra.apache.org > *Subject:* Re: Consistency Issues > > > > And that's a stupid one, I know, but does the column you're trying to > access actually exist? > > > Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso> > > > > On 1 October 2015 at 11:09, Walsh, Stephen <stephen.wa...@aspect.com> > wrote: > > I did think of that and they are all the same version J > > > > > > *From:* Carlos Alonso [mailto:i...@mrcalonso.com] > *Sent:* 01 October 2015 10:11 > > > *To:* user@cassandra.apache.org > *Subject:* Re: Consistency Issues > > > > Hi Stephen. > > > > The UnknownColumnFamilyException made me thought of a possible schema > disagreement in which any of your nodes has a different version and > therefore you cannot reach quorum? > > > > Can you run nodetool describecluster and see if all nodes have the same > schema versions? > > > > Cheers! > > > Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso> > > > > On 1 October 2015 at 09:49, Walsh, Stephen <stephen.wa...@aspect.com> > wrote: > > If you’re looking for the clean-up of the old gen in the jvm heap, it > doesn’t happen. > > We have a new gen turning 15 times before its pushed to old gen. > > Seems all our data only has a TTL of 10 seconds – very little data is sent > to the old gen. > > > > Add in heap size of 8GB with a new gen size of 2GB, I don’t think gc is > our issue. > > > > > > I’m more worried about error messages in the Cassandra log file that state. > > > > > > UnknownColumnFamilyException reading from socket; closing > > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find > cfId=cf411b50-6785-11e5-a435-e7be20c92086 > > > > and > > > > cassandra OutboundTcpConnection.java:313 - error writing to Connection. > > > > > > > > But I really need to understand this best practice that was mentioned (on > number of CF’s) by Jack Krupansky. > > Anyone more information on this? > > > > > > Many thanks for all your help guys keep it coming J > > Steve > > > > *From:* Ricardo Sancho [mailto:sancho.rica...@gmail.com] > *Sent:* 01 October 2015 09:39 > *To:* user@cassandra.apache.org > *Subject:* RE: Consistency Issues > > > > Can you tell us how much time your gcs are taking? > Do you see any especially long ones? > > On 1 Oct 2015 09:37, "Walsh, Stephen" <stephen.wa...@aspect.com> wrote: > > There is no load balancer in front of Cassandra, it’s in front of our > application. > > Everyone seems hung up on this point? But it’s not the root causing of the > inconsistency issue. > > > > Can anyone verify the best practice for number of CF’s? > > > > > > *From:* Robert Coli [mailto:rc...@eventbrite.com] > *Sent:* 30 September 2015 18:45 > *To:* user@cassandra.apache.org > *Subject:* Re: Consistency Issues > > > > On Wed, Sep 30, 2015 at 9:06 AM, Walsh, Stephen <stephen.wa...@aspect.com> > wrote: > > > > We never had these issue with our first run. Its only when we added > another 25% of writes. > > > > As Jack said, you are probably pushing your GC over a threshold, leading > to long pause times and inability to meet quorum. > > > > As Sebastian said, you probably shouldn't need a load balancer in front of > Cassandra. > > > > =Rob > > > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have received > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have received > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > > > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have received > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have received > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. >
