NullPointerException causing repair to hang
Using Cassandra 2.0.3, I'm seeing repairs hanging on all nodes on our cluster. On the node running the repair, the error is: Caused by: org.apache.cassandra.exceptions.RepairException: [repair #8d77b2a0-670b-11e3-949c-6176e8469a51 on OpsCenter/rollups300, (251968055115794,2549561992225510945]] Sync failed between / 172.31.26.229 and /172.31.2.25 at org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:200) And on the node it's streaming from: ERROR [STREAM-IN-/172.31.26.228] 2013-12-17 10:38:06,199 StreamSession.java (line 410) [Stream #505c11d0-6707-11e3-a961-c10c67829566] Streaming error occurred java.lang.NullPointerException at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:174) at org.apache.cassandra.streaming.StreamSession.prepare(StreamSession.java:436) at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:358) at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:293) at java.lang.Thread.run(Thread.java:724) This seems to be the same error as CASSANDRA-6210, however there are no nodes bootstrapping (I haven't added any nodes since starting the cluster up). Does anyone have any ideas about how to get past this error? -- Russ Garrett r...@garrett.co.uk
Why was Thrift defined obsolete?
Hi, I've read some articles about Cassandra and I noticed an opinion that Thrift protocol has some flaws. Thrift should go away in the nearest futures. But I cannot find any reference answering the question why is it so bad? -- Daneel S. Yaitskov
Re: Why was Thrift defined obsolete?
That's just mis-information by people that don't understand thrift. The thrift drivers are still much more mature than the java drivers right now. DataStax has stated on multiple occasions thrift isn't going any where. CQL is fine if people only want to use SQL-like language. Search the cassandra mailing list and you'll see this topic comes up regularly. My bias perspective is "use thrift" to use 100% of the features that Cassandra provides, or use pure CQL and limit yourself to 90%. The main downside of using pure CQL (ie no thrift) is that once you've defined a default type for column values, you can't stick arbitrary data into dynamic columns. Cassandra will tell you when you try to insert a Date when the default value type is int. Basically, it will throw an exception. In contrast, if you use thrift, you can insert what ever type you want and have total control over what goes into dynamic columns. For me, the biggest value of dynamic columns + data types is that I can insert what ever I want into dynamic columns and still have the safety net of knowing the type. A lot of people are afraid of thrift and don't want to dive deep, which is fine. I prefer to understand things at a deep level, and use a tool to the fullest extent. peter On Tue, Dec 17, 2013 at 8:40 AM, Daneel Yaitskov wrote: > Hi, > > I've read some articles about Cassandra and I noticed an opinion that > Thrift protocol > has some flaws. Thrift should go away in the nearest futures. > > But I cannot find any reference answering the question why is it so bad? > > > -- > Daneel S. Yaitskov >
Re: Why was Thrift defined obsolete?
Hopefully this is a valid clarification, rather than a hijack of your thread! How does the binary protocol fit into this? I have not used it but was told you can implement CQL calls via thrift or via the binary protocol. Is the binary protocol superior to thrift? If you use the binary protocol do you have to use CQL? Cheers, Stuart On 17 Dec 2013 14:01, "Peter Lin" wrote: > > That's just mis-information by people that don't understand thrift. > > The thrift drivers are still much more mature than the java drivers right > now. DataStax has stated on multiple occasions thrift isn't going any > where. CQL is fine if people only want to use SQL-like language. Search the > cassandra mailing list and you'll see this topic comes up regularly. > > My bias perspective is "use thrift" to use 100% of the features that > Cassandra provides, or use pure CQL and limit yourself to 90%. > > The main downside of using pure CQL (ie no thrift) is that once you've > defined a default type for column values, you can't stick arbitrary data > into dynamic columns. Cassandra will tell you when you try to insert a Date > when the default value type is int. Basically, it will throw an exception. > In contrast, if you use thrift, you can insert what ever type you want and > have total control over what goes into dynamic columns. > > For me, the biggest value of dynamic columns + data types is that I can > insert what ever I want into dynamic columns and still have the safety net > of knowing the type. A lot of people are afraid of thrift and don't want to > dive deep, which is fine. I prefer to understand things at a deep level, > and use a tool to the fullest extent. > > peter > > > On Tue, Dec 17, 2013 at 8:40 AM, Daneel Yaitskov > wrote: > >> Hi, >> >> I've read some articles about Cassandra and I noticed an opinion that >> Thrift protocol >> has some flaws. Thrift should go away in the nearest futures. >> >> But I cannot find any reference answering the question why is it so bad? >> >> >> -- >> Daneel S. Yaitskov >> > >
Re: Why was Thrift defined obsolete?
thrift is binary protocol, it just happens to support multiple platforms. DataStax Java and C# drivers are also binary, but they are optimized for CQL. For example, the DataStax C# drivers use google protocol buffers, which is a different binary protocol. the documentation on this stuff hasn't been very good, so people keep asking the same questions again and again. I encourage people to take time and learn about thrift. In the long run, it will make you a better cassandra user/developer. On Tue, Dec 17, 2013 at 9:08 AM, Stuart Broad wrote: > Hopefully this is a valid clarification, rather than a hijack of your > thread! > > How does the binary protocol fit into this? I have not used it but was > told you can implement CQL calls via thrift or via the binary protocol. Is > the binary protocol superior to thrift? > > If you use the binary protocol do you have to use CQL? > > Cheers, > > Stuart > On 17 Dec 2013 14:01, "Peter Lin" wrote: > >> >> That's just mis-information by people that don't understand thrift. >> >> The thrift drivers are still much more mature than the java drivers right >> now. DataStax has stated on multiple occasions thrift isn't going any >> where. CQL is fine if people only want to use SQL-like language. Search the >> cassandra mailing list and you'll see this topic comes up regularly. >> >> My bias perspective is "use thrift" to use 100% of the features that >> Cassandra provides, or use pure CQL and limit yourself to 90%. >> >> The main downside of using pure CQL (ie no thrift) is that once you've >> defined a default type for column values, you can't stick arbitrary data >> into dynamic columns. Cassandra will tell you when you try to insert a Date >> when the default value type is int. Basically, it will throw an exception. >> In contrast, if you use thrift, you can insert what ever type you want and >> have total control over what goes into dynamic columns. >> >> For me, the biggest value of dynamic columns + data types is that I can >> insert what ever I want into dynamic columns and still have the safety net >> of knowing the type. A lot of people are afraid of thrift and don't want to >> dive deep, which is fine. I prefer to understand things at a deep level, >> and use a tool to the fullest extent. >> >> peter >> >> >> On Tue, Dec 17, 2013 at 8:40 AM, Daneel Yaitskov < >> rtfm.rtfm.r...@gmail.com> wrote: >> >>> Hi, >>> >>> I've read some articles about Cassandra and I noticed an opinion that >>> Thrift protocol >>> has some flaws. Thrift should go away in the nearest futures. >>> >>> But I cannot find any reference answering the question why is it so bad? >>> >>> >>> -- >>> Daneel S. Yaitskov >>> >> >>
various Cassandra performance problems when CQL3 is really used
Hi all,
we are reimplementing a legacy interface of an inventory-like service
(currently built on top of mysql) on Cassandra and I thought I would share
some findings with the list. The interface semantics is given and cannot be
changed. We chose Cassandra due to its multiple datacenter capabilities and
no-spof qualities. The dataset is small (6 tables having 150.000 records, a
bunch of tables with up to thousands of records), but the model is not
trivial - the mysql model has some 20+ tables, joins are frequent, m:n
relationships are frequent and the like. The interface is read heavy. We
thought the size of the dataset should allow the whole dataset to fit into
memory of each node (3 node cluster in each DC, 3 replicas, local quorum
operations) and that even though some operations (like secondary index
lookup) are not superfast, due to the size of the dataset it should perform
ok. We were wrong.
We use CQL3 exclusively and we use all of its capabilities (collections,
secondary indexes, filtering), because they make the data model
maintainable. We denormalised what had to be denormalised in order to avoid
client side joins. Usual query to the storage means one CQL query on
a denormalised table. We need to support integer offset/limit paging,
filter-by-example kind of queries, M:N relationship queries and all the
usual suspects of old SQL-backed interface.
This is the list of operations that perform really poorly we identified so
far. Row id is called id in the following:
* select id from table where token(id) > token(some_value) and
secondary_index = other_val limit 2 allow filtering;
Filtering absolutely kills the performance. On a table populated with
130.000 records, single node Cassandra server (on my i7 notebook, 2GB of
JVM heap) and secondary index built on column with low cardinality of its
value set this query takes 156 seconds to finish.
By the way, the performance is order of magnitude better if this patch is
applied:
diff --git
a/src/java/org/apache/cassandra/db/index/composites/CompositesSearcher.java
b/src/java/org/apache/cassandra/db/index/composites/CompositesSearcher.java
index 5ab1df6..13af671 100644
---
a/src/java/org/apache/cassandra/db/index/composites/CompositesSearcher.java
+++
b/src/java/org/apache/cassandra/db/index/composites/CompositesSearcher.java
@@ -190,7 +190,8 @@ public class CompositesSearcher extends
SecondaryIndexSearcher
private int meanColumns =
Math.max(index.getIndexCfs().getMeanColumns(), 1);
// We shouldn't fetch only 1 row as this provides buggy paging
in case the first row doesn't satisfy all clauses
-private final int rowsPerQuery =
Math.max(Math.min(filter.maxRows(), filter.maxColumns() / meanColumns), 2);
+//private final int rowsPerQuery =
Math.max(Math.min(filter.maxRows(), filter.maxColumns() / meanColumns), 2);
+private final int rowsPerQuery = 10;
public boolean needsFiltering()
{
* select id from table;
As we saw in the trace log, the query - although it queries just row ids -
scans all columns of all the rows and (probably) compares TTL with current
time (?) (we saw hundreds of thousands of gettimeofday(2)). This means that
if the table somehow mixes wide and narrow rows, the performance suffers
horribly.
* CQL collections
See the point above with mixing wide rows and narrow rows. As Cassandra
checks all the columns in selects, performance suffers badly if the
collection is of any interesting size.
Additionally, we saw various random irreproducible freezes, high CPU
consumption when nothing happens (even with trace log level set no activity
was reported) and highly inpredictable performance characteristics after
nodetool flush and/or major compaction.
Conclusions:
- do not use collections
- do not use secondary indexes
- do not use filtering
- have your rows as narrow as possible if you need any kind of all row keys
traversal
With these conclusions in mind, CQL seems redundant, plain old thrift may
be used, joins should be done client side and/or all indexes need to be
handled manually. Correct?
Thanks for reading,
ondrej cernos
Re: Why was Thrift defined obsolete?
Based on an question I posted a while back I got the following answer (for something unrelated to this): When we speak of "binary protocol", we talk about the protocol introduced > in Cassandra 1.2 that is an alternative to thrift for CQL3. It's a custom, > binary, protocol, that has not link to thrift whatsoever. > That protocol is defined by the document here: > > https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=doc/native_protocol_v1.spec;hb=HEAD So I guess my question still stands (although from this previous answer it looks like it is only for CQL). On Tue, Dec 17, 2013 at 2:15 PM, Peter Lin wrote: > > thrift is binary protocol, it just happens to support multiple platforms. > DataStax Java and C# drivers are also binary, but they are optimized for > CQL. For example, the DataStax C# drivers use google protocol buffers, > which is a different binary protocol. > > the documentation on this stuff hasn't been very good, so people keep > asking the same questions again and again. > > I encourage people to take time and learn about thrift. In the long run, > it will make you a better cassandra user/developer. > > > On Tue, Dec 17, 2013 at 9:08 AM, Stuart Broad wrote: > >> Hopefully this is a valid clarification, rather than a hijack of your >> thread! >> >> How does the binary protocol fit into this? I have not used it but was >> told you can implement CQL calls via thrift or via the binary protocol. Is >> the binary protocol superior to thrift? >> >> If you use the binary protocol do you have to use CQL? >> >> Cheers, >> >> Stuart >> On 17 Dec 2013 14:01, "Peter Lin" wrote: >> >>> >>> That's just mis-information by people that don't understand thrift. >>> >>> The thrift drivers are still much more mature than the java drivers >>> right now. DataStax has stated on multiple occasions thrift isn't going any >>> where. CQL is fine if people only want to use SQL-like language. Search the >>> cassandra mailing list and you'll see this topic comes up regularly. >>> >>> My bias perspective is "use thrift" to use 100% of the features that >>> Cassandra provides, or use pure CQL and limit yourself to 90%. >>> >>> The main downside of using pure CQL (ie no thrift) is that once you've >>> defined a default type for column values, you can't stick arbitrary data >>> into dynamic columns. Cassandra will tell you when you try to insert a Date >>> when the default value type is int. Basically, it will throw an exception. >>> In contrast, if you use thrift, you can insert what ever type you want and >>> have total control over what goes into dynamic columns. >>> >>> For me, the biggest value of dynamic columns + data types is that I can >>> insert what ever I want into dynamic columns and still have the safety net >>> of knowing the type. A lot of people are afraid of thrift and don't want to >>> dive deep, which is fine. I prefer to understand things at a deep level, >>> and use a tool to the fullest extent. >>> >>> peter >>> >>> >>> On Tue, Dec 17, 2013 at 8:40 AM, Daneel Yaitskov < >>> rtfm.rtfm.r...@gmail.com> wrote: >>> Hi, I've read some articles about Cassandra and I noticed an opinion that Thrift protocol has some flaws. Thrift should go away in the nearest futures. But I cannot find any reference answering the question why is it so bad? -- Daneel S. Yaitskov >>> >>> >
Re: Why was Thrift defined obsolete?
Peter - I missed your comment regarding optimised for CQL (I was distracted by the statement of thift is binary protocol - As I got 'corrected' for a similar statement to that in one of my previous posts). So comparing thrift to the 'newer' ninary protocols it sounds like the only real benefit is optimised for CQL (which has limitations as you posted in your first email). On Tue, Dec 17, 2013 at 3:15 PM, Stuart Broad wrote: > Based on an question I posted a while back I got the following answer (for > something unrelated to this): > > When we speak of "binary protocol", we talk about the protocol introduced >> in Cassandra 1.2 that is an alternative to thrift for CQL3. It's a custom, >> binary, protocol, that has not link to thrift whatsoever. >> That protocol is defined by the document here: >> >> https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=doc/native_protocol_v1.spec;hb=HEAD > > > So I guess my question still stands (although from this previous answer it > looks like it is only for CQL). > > > > > On Tue, Dec 17, 2013 at 2:15 PM, Peter Lin wrote: > >> >> thrift is binary protocol, it just happens to support multiple platforms. >> DataStax Java and C# drivers are also binary, but they are optimized for >> CQL. For example, the DataStax C# drivers use google protocol buffers, >> which is a different binary protocol. >> >> the documentation on this stuff hasn't been very good, so people keep >> asking the same questions again and again. >> >> I encourage people to take time and learn about thrift. In the long run, >> it will make you a better cassandra user/developer. >> >> >> On Tue, Dec 17, 2013 at 9:08 AM, Stuart Broad wrote: >> >>> Hopefully this is a valid clarification, rather than a hijack of your >>> thread! >>> >>> How does the binary protocol fit into this? I have not used it but was >>> told you can implement CQL calls via thrift or via the binary protocol. Is >>> the binary protocol superior to thrift? >>> >>> If you use the binary protocol do you have to use CQL? >>> >>> Cheers, >>> >>> Stuart >>> On 17 Dec 2013 14:01, "Peter Lin" wrote: >>> That's just mis-information by people that don't understand thrift. The thrift drivers are still much more mature than the java drivers right now. DataStax has stated on multiple occasions thrift isn't going any where. CQL is fine if people only want to use SQL-like language. Search the cassandra mailing list and you'll see this topic comes up regularly. My bias perspective is "use thrift" to use 100% of the features that Cassandra provides, or use pure CQL and limit yourself to 90%. The main downside of using pure CQL (ie no thrift) is that once you've defined a default type for column values, you can't stick arbitrary data into dynamic columns. Cassandra will tell you when you try to insert a Date when the default value type is int. Basically, it will throw an exception. In contrast, if you use thrift, you can insert what ever type you want and have total control over what goes into dynamic columns. For me, the biggest value of dynamic columns + data types is that I can insert what ever I want into dynamic columns and still have the safety net of knowing the type. A lot of people are afraid of thrift and don't want to dive deep, which is fine. I prefer to understand things at a deep level, and use a tool to the fullest extent. peter On Tue, Dec 17, 2013 at 8:40 AM, Daneel Yaitskov < rtfm.rtfm.r...@gmail.com> wrote: > Hi, > > I've read some articles about Cassandra and I noticed an opinion that > Thrift protocol > has some flaws. Thrift should go away in the nearest futures. > > But I cannot find any reference answering the question why is it so > bad? > > > -- > Daneel S. Yaitskov > >> >
Re: Why was Thrift defined obsolete?
that is correct from studying DataStax java and C# driver. On Tue, Dec 17, 2013 at 10:22 AM, Stuart Broad wrote: > Peter - I missed your comment regarding optimised for CQL (I was > distracted by the statement of thift is binary protocol - As I got > 'corrected' for a similar statement to that in one of my previous posts). > So comparing thrift to the 'newer' ninary protocols it sounds like the > only real benefit is optimised for CQL (which has limitations as you posted > in your first email). > > > On Tue, Dec 17, 2013 at 3:15 PM, Stuart Broad wrote: > >> Based on an question I posted a while back I got the following answer >> (for something unrelated to this): >> >> When we speak of "binary protocol", we talk about the protocol introduced >>> in Cassandra 1.2 that is an alternative to thrift for CQL3. It's a >>> custom, >>> binary, protocol, that has not link to thrift whatsoever. >>> That protocol is defined by the document here: >>> >>> https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=doc/native_protocol_v1.spec;hb=HEAD >> >> >> So I guess my question still stands (although from this previous answer >> it looks like it is only for CQL). >> >> >> >> >> On Tue, Dec 17, 2013 at 2:15 PM, Peter Lin wrote: >> >>> >>> thrift is binary protocol, it just happens to support multiple >>> platforms. DataStax Java and C# drivers are also binary, but they are >>> optimized for CQL. For example, the DataStax C# drivers use google protocol >>> buffers, which is a different binary protocol. >>> >>> the documentation on this stuff hasn't been very good, so people keep >>> asking the same questions again and again. >>> >>> I encourage people to take time and learn about thrift. In the long run, >>> it will make you a better cassandra user/developer. >>> >>> >>> On Tue, Dec 17, 2013 at 9:08 AM, Stuart Broad wrote: >>> Hopefully this is a valid clarification, rather than a hijack of your thread! How does the binary protocol fit into this? I have not used it but was told you can implement CQL calls via thrift or via the binary protocol. Is the binary protocol superior to thrift? If you use the binary protocol do you have to use CQL? Cheers, Stuart On 17 Dec 2013 14:01, "Peter Lin" wrote: > > That's just mis-information by people that don't understand thrift. > > The thrift drivers are still much more mature than the java drivers > right now. DataStax has stated on multiple occasions thrift isn't going > any > where. CQL is fine if people only want to use SQL-like language. Search > the > cassandra mailing list and you'll see this topic comes up regularly. > > My bias perspective is "use thrift" to use 100% of the features that > Cassandra provides, or use pure CQL and limit yourself to 90%. > > The main downside of using pure CQL (ie no thrift) is that once you've > defined a default type for column values, you can't stick arbitrary data > into dynamic columns. Cassandra will tell you when you try to insert a > Date > when the default value type is int. Basically, it will throw an exception. > In contrast, if you use thrift, you can insert what ever type you want and > have total control over what goes into dynamic columns. > > For me, the biggest value of dynamic columns + data types is that I > can insert what ever I want into dynamic columns and still have the safety > net of knowing the type. A lot of people are afraid of thrift and don't > want to dive deep, which is fine. I prefer to understand things at a deep > level, and use a tool to the fullest extent. > > peter > > > On Tue, Dec 17, 2013 at 8:40 AM, Daneel Yaitskov < > rtfm.rtfm.r...@gmail.com> wrote: > >> Hi, >> >> I've read some articles about Cassandra and I noticed an opinion that >> Thrift protocol >> has some flaws. Thrift should go away in the nearest futures. >> >> But I cannot find any reference answering the question why is it so >> bad? >> >> >> -- >> Daneel S. Yaitskov >> > > >>> >> >
Slow pre-decommission repair
Hi, I'm trying to decommission a node from a six node Cassandra 2.0.3 cluster, following the instructions at http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/operations/ops_remove_node_t.html. The node has just short of 11GB data in one keyspace (RF=3), and the repair is taking forever (around 25 hours if it's keeping the current pace). Re-reading the instructions I realized that I ran nodetool repair, instead of nodetool repair *keyspace_name*, but this can't have this kind of impact right? I ran nodetool repair -pr on the same node yesterday, and it took around 35 min. The cluster is not under significant load, neither CPU, RAM, disk nor network IO. Nothing has ever been deleted from this keyspace and no entries expire. Does anyone have any idea why a repair would take this long? When decommissioning nodes, would it be sufficient to run nodetool repair -pr on the node to be decommissioned?
Re: Issues while fetching data with pycassa get for super columns
On Fri, Dec 13, 2013 at 2:25 PM, Kumar Ranjan wrote:
> Results: {u'narrativebuddieswin': ['609548930995445799_752368319',
> '609549303525138481_752368319', '610162034020180814_752368319',
> '610162805856002905_752368319', '610163571417146213_752368319',
> '610165900312830861_752368319']}
>
> none of the subcolumns are returned for above super column ??? Please
> help..
>
>
> CODE: ->>>
>
> if start:
>
> res_rows = col_fam.get(key, column_count=count,
> column_start=start, include_timestamp=True, include_ttl=True, )
>
Is "narrativebuddieswin" the name of a supercolumn? What are the
'609548930995445799_752368319' values supposed to be? It shouldn't even
return a list in the results, so I'm really not sure what's happening.
--
Tyler Hobbs
DataStax
Re: Slow pre-decommission repair
On Tue, Dec 17, 2013 at 8:00 AM, Joel Segerlind wrote: > Hi, > > I'm trying to decommission a node from a six node Cassandra 2.0.3 cluster, > following the instructions at > http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/operations/ops_remove_node_t.html > ... > When decommissioning nodes, would it be sufficient to run nodetool repair > -pr on the node to be decommissioned? > Not sure why those instructions would tell you to repair *before* removing a node from the cluster? Decommission streams a nodes key to the new node responsible for the range, why would a repair be required beforehand? > The node has just short of 11GB data in one keyspace (RF=3), and the > repair is taking forever (around 25 hours if it's keeping the current pace). > Do you see it actually doing anything? > Re-reading the instructions I realized that I ran nodetool repair, > instead of nodetool repair *keyspace_name*, but this can't have this kind > of impact right? I ran nodetool repair -pr on the same node yesterday, > and it took around 35 min. > Do you use vnodes? =Rob
Re: Restore with archive commitlog
On Mon, Dec 16, 2013 at 2:55 AM, Bonnet Jonathan. < jonathan.bon...@externe.bnpparibas.com> wrote: > The link is not allowed by my company where i Work, so nobody never manage > to make work a restore with archive commit log ? Strange to me to let this > option in all new releases of cassandra if it doesn't work. > There is a fair amount of code in a lot of open source software which was added for or because of a single (usually large/prominent) end user shop. It's a bit strange, but is part of the essential nature of patches : they are produced by people who need them and have the resources to produce them. For Cassandra, this list probably includes : - streaming commit log (netflix) - replace_token/node (netflix) - counters (digg/twitter) etc.. =Rob
Re: NullPointerException causing repair to hang
On Tue, Dec 17, 2013 at 4:32 AM, Russ Garrett wrote: > This seems to be the same error as CASSANDRA-6210, however there are > no nodes bootstrapping (I haven't added any nodes since starting the > cluster up). > I would comment to that effect on CASSANDRA-6210, were I you. Are you using vnodes? Have you tried a rolling restart of all nodes? =Rob
Re: Slow pre-decommission repair
On 17 December 2013 20:40, Robert Coli wrote: > On Tue, Dec 17, 2013 at 8:00 AM, Joel Segerlind wrote: > >> Hi, >> >> I'm trying to decommission a node from a six node Cassandra 2.0.3 >> cluster, following the instructions at >> http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/operations/ops_remove_node_t.html >> > ... > >> When decommissioning nodes, would it be sufficient to run nodetool >> repair -pr on the node to be decommissioned? >> > > Not sure why those instructions would tell you to repair *before* removing > a node from the cluster? > > Decommission streams a nodes key to the new node responsible for the > range, why would a repair be required beforehand? > I wondered the same thing, but I thought that the one writing that ought to know way better than me. > > >> The node has just short of 11GB data in one keyspace (RF=3), and the >> repair is taking forever (around 25 hours if it's keeping the current pace). >> > > Do you see it actually doing anything? > Sure, I'm tailing the system.log and it's about a third through atm. > > >> Re-reading the instructions I realized that I ran nodetool repair, >> instead of nodetool repair *keyspace_name*, but this can't have this >> kind of impact right? I ran nodetool repair -pr on the same node >> yesterday, and it took around 35 min. >> > > Do you use vnodes? > Yes. > > =Rob > >
Re: Slow pre-decommission repair
On Tue, Dec 17, 2013 at 11:46 AM, Joel Segerlind wrote: > Decommission streams a nodes key to the new node responsible for the >> range, why would a repair be required beforehand? >> > > I wondered the same thing, but I thought that the one writing that ought > to know way better than me. > Not always the case... Do you use vnodes? >> > > Yes. > https://issues.apache.org/jira/browse/CASSANDRA-5220 =Rob
WriteTimeoutException instead of UnavailableException
Question. I have a 5 node cluster (local with ccm). A keyspace with rf: 3. Three nodes are down. I run "nodetool ring" in the two living nodes and both see the other three nodes down. Then i do an insert with cs quorum and get an UnavailableException. It's ok. I am using Datastax java driver v 2.0.0-rc2. But in some cases, from one certain node, I get an WriteTimeoutException for a few minutes until an UnavailableException. It's like the coordinator don't know the status of the cluster. Any clue why is this happening? Thanks,
Re: Why was Thrift defined obsolete?
Also you are going to encounter code that uses thirft/hector thrift/asyanax and if you work on a codebase that was designed before CQL you still need to support it. There are some concepts people have employed in those tools like VirtualKeyspaces etc that have not made their way into CQL. On Tue, Dec 17, 2013 at 9:00 AM, Peter Lin wrote: > > That's just mis-information by people that don't understand thrift. > > The thrift drivers are still much more mature than the java drivers right > now. DataStax has stated on multiple occasions thrift isn't going any > where. CQL is fine if people only want to use SQL-like language. Search the > cassandra mailing list and you'll see this topic comes up regularly. > > My bias perspective is "use thrift" to use 100% of the features that > Cassandra provides, or use pure CQL and limit yourself to 90%. > > The main downside of using pure CQL (ie no thrift) is that once you've > defined a default type for column values, you can't stick arbitrary data > into dynamic columns. Cassandra will tell you when you try to insert a Date > when the default value type is int. Basically, it will throw an exception. > In contrast, if you use thrift, you can insert what ever type you want and > have total control over what goes into dynamic columns. > > For me, the biggest value of dynamic columns + data types is that I can > insert what ever I want into dynamic columns and still have the safety net > of knowing the type. A lot of people are afraid of thrift and don't want to > dive deep, which is fine. I prefer to understand things at a deep level, > and use a tool to the fullest extent. > > peter > > > On Tue, Dec 17, 2013 at 8:40 AM, Daneel Yaitskov > wrote: > >> Hi, >> >> I've read some articles about Cassandra and I noticed an opinion that >> Thrift protocol >> has some flaws. Thrift should go away in the nearest futures. >> >> But I cannot find any reference answering the question why is it so bad? >> >> >> -- >> Daneel S. Yaitskov >> > >
Re: Why was Thrift defined obsolete?
For me, it's best to now both and use both where each is strong. that way you get the most out of cassandra. I am bias in favor of thrift, since I've been contributing to hector and ported hector to C# over the summer. On Tue, Dec 17, 2013 at 4:20 PM, Edward Capriolo wrote: > Also you are going to encounter code that uses thirft/hector > thrift/asyanax and if you work on a codebase that was designed before CQL > you still need to support it. There are some concepts people have employed > in those tools like VirtualKeyspaces etc that have not made their way into > CQL. > > > On Tue, Dec 17, 2013 at 9:00 AM, Peter Lin wrote: > >> >> That's just mis-information by people that don't understand thrift. >> >> The thrift drivers are still much more mature than the java drivers right >> now. DataStax has stated on multiple occasions thrift isn't going any >> where. CQL is fine if people only want to use SQL-like language. Search the >> cassandra mailing list and you'll see this topic comes up regularly. >> >> My bias perspective is "use thrift" to use 100% of the features that >> Cassandra provides, or use pure CQL and limit yourself to 90%. >> >> The main downside of using pure CQL (ie no thrift) is that once you've >> defined a default type for column values, you can't stick arbitrary data >> into dynamic columns. Cassandra will tell you when you try to insert a Date >> when the default value type is int. Basically, it will throw an exception. >> In contrast, if you use thrift, you can insert what ever type you want and >> have total control over what goes into dynamic columns. >> >> For me, the biggest value of dynamic columns + data types is that I can >> insert what ever I want into dynamic columns and still have the safety net >> of knowing the type. A lot of people are afraid of thrift and don't want to >> dive deep, which is fine. I prefer to understand things at a deep level, >> and use a tool to the fullest extent. >> >> peter >> >> >> On Tue, Dec 17, 2013 at 8:40 AM, Daneel Yaitskov < >> rtfm.rtfm.r...@gmail.com> wrote: >> >>> Hi, >>> >>> I've read some articles about Cassandra and I noticed an opinion that >>> Thrift protocol >>> has some flaws. Thrift should go away in the nearest futures. >>> >>> But I cannot find any reference answering the question why is it so bad? >>> >>> >>> -- >>> Daneel S. Yaitskov >>> >> >> >
Re: Slow pre-decommission repair
On 17 December 2013 20:56, Robert Coli wrote: > On Tue, Dec 17, 2013 at 11:46 AM, Joel Segerlind wrote: > >> Decommission streams a nodes key to the new node responsible for the >>> range, why would a repair be required beforehand? >>> >> >> I wondered the same thing, but I thought that the one writing that ought >> to know way better than me. >> > > Not always the case... > :) > Do you use vnodes? >>> >> >> Yes. >> > > https://issues.apache.org/jira/browse/CASSANDRA-5220 > Thanks for the info. However, wouldn't this also affect nodetool -pr (although not as much), which I ran on the same node the other day in about 35 min? I cannot understand how it can take 35 min for the primary range, and 25 h for a full repair. > =Rob > >
Re: Why was Thrift defined obsolete?
Realize that there will be more and more new features that come along as cassandra matures. It is an overwhelming certainty that these feature will be available thru the new native interface & CQL. The same level of certainty can't be given to Thrift. Certainly if you have existing applications running against Thrift, then there is no need to worry that Thrift will break or not perform optimally in the future. But going forward, there will be things that you won't be able to use thru Thrift that may solve problems for you. If you are starting now, the recommendation is to use the new native interface and CQL.Just saying... - Original Message -From: "Peter Lin" >;wool...@gmail.com
Re: Why was Thrift defined obsolete?
wouldn't that go against what DataStax has publicly stated about support for Thrift? So far, DataStax has been good about keeping the Thrift API up to date. I'm inclined to trust DataStax mean what they say. If DataStax doesn't have the man power to keep Thrift up to speed with new features, I'm willing to pitch in and I'm guessing so would others. On Tue, Dec 17, 2013 at 4:52 PM, Dave Brosius wrote: > Realize that there will be more and more new features that come along as > cassandra matures. It is an overwhelming certainty that these feature will > be available thru the new native interface & CQL. The same level of > certainty can't be given to Thrift. Certainly if you have existing > applications running against Thrift, then there is no need to worry that > Thrift will break or not perform optimally in the future. But going > forward, there will be things that you won't be able to use thru Thrift > that may solve problems for you. If you are starting now, the > recommendation is to use the new native interface and CQL. > > Just saying... > > > > > *- Original Message -* > *From:* "Peter Lin" > *Sent:* Tue, December 1 7, 2013 16:28 > *Subject:* Re: Why was Thrift defined obsolete? > > > For me, it's best to now both and use both where each is strong. that way > you get the most out of cassandra. > > I am bias in favor of thrift, since I've been contributing to hector and > ported hector to C# over the summer. > > > On Tue, Dec 17, 2013 at 4:20 PM, Edward Capriolo wrote: > >> Also you are going to encounter code that uses thirft/hector >> thrift/asyanax and if you work on a codebase that was designed before CQL >> you still need to support it. Th ere are some concepts people have employed >> in those tools like VirtualKeyspaces etc that have not made their way into >> CQL. >> >> >> On Tue, Dec 17, 2013 at 9:00 AM, Peter Lin wrote: >> >>> >>> That's just mis-information by people that don't understand thrift. >>> The thrift drivers are still much more mature than the java drivers >>> right now. DataStax has stated on multiple occasions thrift isn't going any >>> where. CQL is fine if people only want to use SQL-like language. Search the >>> cassandra mailing list and you'll see this topic comes up regularly. >>> My bias perspective is "use thrift" to use 100% of the features that >>> Cassandra provides, or use pure CQL and limit yourself to 90%. >>> >>> The main downside of using pure CQL (ie no thrift) is that once you've >>> defined a default type for column values, you can't stick arbitrary data >>> into dynamic columns. Cassandra will tell you when you try to insert a Date >>> when the default value type is int. Basically, it will throw an exception. >>> In contrast, if you use thrift, you can insert what ever type you want and >>> have total control over what goes into dynamic columns. >>> For me, the biggest value of dynamic columns + data types is that I can >>> insert what ever I want into dynamic columns and still have the safety net >>> of knowing the type. A lot of people are afraid of thrift and don't want to >>> dive deep, which is fine. I prefer to understand things at a deep level, >>> and use a tool to the fullest extent. >>> >>> peter >>> >>> >>> On Tue, Dec 17, 2013 at 8:40 AM, Daneel Yaitskov < >>> rtfm.rtfm.r...@gmail.com> wrote: >>> Hi, I've read some articles about Cassandra and I noticed an opinion that Thrift protocol has some flaws. Thrift should go away in the nearest futures. But I cannot find any reference answering the question why is it so bad? -- Daneel S. Yaitskov >>>
Re: Why was Thrift defined obsolete?
On Tue, Dec 17, 2013 at 1:52 PM, Dave Brosius wrote: > Realize that there will be more and more new features that come along as > cassandra matures. It is an overwhelming certainty that these feature will > be available thru the new native interface & CQL. The same level of > certainty can't be given to Thrift. Certainly if you have existing > applications running against Thrift, then there is no need to worry that > Thrift will break or not perform optimally in the future. But going > forward, there will be things that you won't be able to use thru Thrift > that may solve problems for you. If you are starting now, the > recommendation is to use the new native interface and CQL. Put another way... http://en.wikipedia.org/wiki/Separate_but_equal " (quoting from http://en.wikipedia.org/wiki/Brown_v._Board_of_Educationdecision) We conclude that, in the field of public education, the doctrine of "separate but equal" has no place. Separate educational facilities are inherently unequal. " s/public education/Cassandra APIs/ =Rob
Astyanax TokenAware connection pool
Hello everyone,
I was using astyanax connection pool defined as this:
ipSeeds = "LOAD_BALANCER_HOST:9160";
conPool.setSeeds(ipSeeds)
.setDiscoveryType(NodeDiscoveryType.TOKEN_AWARE)
.setConnectionPoolType(ConnectionPoolType.TOKEN_AWARE);
However, my cluster have 4 nodes and I have 8 client machines connecting on
it. LOAD_BALANCER_HOST forwards requests to one of my four nodes.
On a client node, I have:
$netstat -an | grep 9160 | awk '{print $5}' | sort |uniq -c
235 node1:9160
680 node2:9160
4 node3:9160
4 node4:9160
So although the ConnectionPoolType is TOKEN_AWARE, my client seems to be
connecting mainly to node2, sometimes to node1, but almost never to nodes 3
and 4.
Question is:
Why is this happening? Shouldn't a token aware connection pool query the
ring for the node list and connect to all the active nodes using round
robin algorithm?
Best regards,
Marcelo Valle.
Re: Unable to create collection inside collection
> Could anybody suggest me how do I achieve it in Cassandra. It’s not supported. You may want to model the feeschedule as a table. Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 12/12/2013, at 11:09 pm, Santosh Shet wrote: > Hi, > > I am not able to create collection inside another collection in Cassandra. > Please find screenshot below > > > > In the above screenshot, I am trying to create column named feeschedule with > type Map and Map have values which is of type List. > > Could anybody suggest me how do I achieve it in Cassandra. > My Cassandra version details are given below: > > cqlsh version- cqlsh 4.1.0 > Cassandra version – 2.0.2 > > Thanks in advance, > > Regards > Santosh Shet > Software Engineer | VistaOne Solutions > Direct India : +91 80 30273829 | Mobile India : +91 8105720582 > Skype : santushet
Re: Unbalanced ring with C* 2.0.3 and vnodes after adding additional nodes
> Node: 4 CPU, 6 GB RAM, virtual appliance > > Cassandra: 3 GB Heap, vnodes 256 FWIW that’s a very low powered node. > Maybe we forgot necessary actions during or after cluster expanding process. > We are open for every idea. Where the nodes in the seed list when they joined the cluster? If so they do not bootstrap. The extra writes in nodes 5 and 6 could be from Read Repair writing to them. Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 12/12/2013, at 11:49 pm, Andreas Finke wrote: > Hi, > > after adding 2 more nodes to a 4 nodes cluster (before) we are experiencing > high load on both new nodes. After doing some investigation we found out the > following: > > - High cpu load on vm5+6 > - Higher data load on vm5+6 > - Write requests are evenly distributed to all 6 nodes by our client > application (opscenter -> metrics -> WriteRequests) > - Local writes are as twice as much in vm5 +6 (vm1-4: ~2800/s, vm5-6: ~6800/s) > - Nodetool output: > > UN vm1 9.51 GB256 20,7% 13fa7bb7-19cb-44f5-af83-71a72e04993a X1 > > UN vm2 9.41 GB256 20,0% b71c2d3d-4721-4dde-a418-802f1af4b7a1 D1 > > UN vm3 9.37 GB256 18,9% 8ce4c419-d79c-4ef1-b3fd-8936bff3e44f X1 > > > UN vm4 9.23 GB256 19,5% 17974f20-5756-4eba-a377-52feed3a1b10 D1 > > UN vm5 15.95 GB 256 10,7% 0c6db9ea-4c60-43f6-a12e-51a7d76f8e80 X1 > > UN vm6 14.86 GB 256 10,2% f64d1909-dd96-442b-b602-efee29eee0a0 D1 > > > > Although the ownership is lower on vm5-6 (which already is not right) the > data load is way higher. > > > > Some cluster facts: > > > > Node: 4 CPU, 6 GB RAM, virtual appliance > > Cassandra: 3 GB Heap, vnodes 256 > > Schema: Replication strategy network, RF:2 > > > > Has anyone an idea what could be the cause for the unbalancing. Maybe we > forgot necessary actions during or after cluster expanding process. We are > open for every idea. > > > > Regards > > Andi >
Re: Cassandra pytho pagination
CQL3 and thrift do not support an offset clause, so you can only really support
next / prev page calls to the database.
> I am trying to use xget with column_count and buffer_size parameters. Can
> someone explain me, how does it work? From doc, my understanding is that, I
> can do something like,
What client are you using ?
xget is not a standard cassandra function.
Cheers
-
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 13/12/2013, at 4:56 am, Kumar Ranjan wrote:
> Hey Folks,
>
> I need some ideas about support implementing of pagination on the browser,
> from the backend. So python code (backend) gets request from frontend with
> page=1,2,3,4 and so on and count_per_page=50.
>
> I am trying to use xget with column_count and buffer_size parameters. Can
> someone explain me, how does it work? From doc, my understanding is that, I
> can do something like,
>
>
> total_cols is total columns for that key.
> count is what user sends me.
> .xget('Twitter_search', hh, column_count=total_cols, buffer_size=count):
>
> Is my understanding correct? because its not working for page 2 and so on?
> Please enlighten me with suggestions.
>
> Thanks.
>
Re: Cassandra data update for a row
> 'twitter_row_key': OrderedDict([('411186035495010304', u'{"score": 0, "tid":
> 411186035495010304, "created_at": "Thu Dec 12 17:29:24 + 2013",
> "favorite": 0, "retweet": 0, "approved": "true"}'),])
>
> How can I set approved to 'false' ??
>
>
It looks like the value of the 411186035495010304 column is a string, to
cassandra that’s an opaque typer we do not make partial updates to.
If you need to update the values individually they need to be stored in
columns.
Cheers
-
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 13/12/2013, at 8:18 am, Kumar Ranjan wrote:
> Hey Folks,
>
> I have a row like this. 'twitter_row_key' is the row key and
> 411186035495010304 is column. Rest is values for 411186035495010304 column.
> See below.
>
> 'twitter_row_key': OrderedDict([('411186035495010304', u'{"score": 0, "tid":
> 411186035495010304, "created_at": "Thu Dec 12 17:29:24 + 2013",
> "favorite": 0, "retweet": 0, "approved": "true"}'),])
>
> How can I set approved to 'false' ??
>
>
>
> When I try insert for row key 'twitter_row_key' and column
> 411186035495010304, it overwrites the whole data and new row becomes like this
>
> 'twitter_row_key': OrderedDict([('411186035495010304', u'{"approved":
> "true"}'),])
>
>
>
> Any thoughts guys?
>
Re: Get all the data for x number of seconds from CQL?
> Is it possible to get all the data for last 5 seconds or 10 seconds or 30 > seconds by using the id column? Not using the current table. Try this CREATE TABLE test1 ( day integer, timestamp integer, count integer, record_name text, record_valueblob, PRIMARY KEY (day, timestamp, record_name) ) Store the day as MMDD and the timestamp as before, you can then do queries like select * from test1 where day = 20131218 and timestamp > X and timestamp < y; Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 13/12/2013, at 11:28 am, Techy Teck wrote: > Below is my CQL table - > > CREATE TABLE test1 ( > id text, > record_name text, > record_value blob, > PRIMARY KEY (id, record_name) > ) > > here id column will have data like this - > > timestamp.count > > And here timestamp is in milliseconds but rounded up to nearest seconds. So > as an example, data in `id column` will be like this - > > 138688293.1 > > And a single row in the above table will be like this - > > 138688293.1 | event_name | hello-world > > Now my question is - > > Is it possible to get all the data for last 5 seconds or 10 seconds or 30 > seconds by using the id column? > > I am running Cassandra 1.2.9
Re: Write performance with 1.2.12
> With a single node I get 3K for cassandra 1.0.12 and 1.2.12. So I suspect > there is some network chatter. I have started looking at the sources, hoping > to find something. 1.2 is pretty stable, I doubt there is anything in there that makes it run slower than 1.0. It’s probably something in your configuration or network. Compare the local write time from nodetool cfhistograms and the request latency from nodetool proxyhistograms. Writes request latency should be a bit below 1ms and local write latency should be around .5 ms or better. if there is a wider difference between the two it’s wait time + network time. As a general rule you should get around 3k to 4k writes per second per core. Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 13/12/2013, at 8:06 pm, Rahul Menon wrote: > Quote from > http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2 > > "Murmur3Partitioner is NOT compatible with RandomPartitioner, so if you’re > upgrading and using the new cassandra.yaml file, be sure to change the > partitioner back to RandomPartitioner" > > > On Thu, Dec 12, 2013 at 10:57 PM, srmore wrote: > > > > On Thu, Dec 12, 2013 at 11:15 AM, J. Ryan Earl wrote: > Why did you switch to RandomPartitioner away from Murmur3Partitioner? Have > you tried with Murmur3? > > # partitioner: org.apache.cassandra.dht.Murmur3Partitioner > partitioner: org.apache.cassandra.dht.RandomPartitioner > > > Since I am comparing between the two versions I am keeping all the settings > same. I see > Murmur3Partitioner has some performance improvement but then switching back to > RandomPartitioner should not cause performance to tank, right ? or am I > missing something ? > > Also, is there an easier way to update the data from RandomPartitioner to > Murmur3 ? (upgradesstable ?) > > > > > On Fri, Dec 6, 2013 at 10:36 AM, srmore wrote: > > > > On Fri, Dec 6, 2013 at 9:59 AM, Vicky Kak wrote: > You have passed the JVM configurations and not the cassandra configurations > which is in cassandra.yaml. > > Apologies, was tuning JVM and that's what was in my mind. > Here are the cassandra settings http://pastebin.com/uN42GgYT > > > The spikes are not that significant in our case and we are running the > cluster with 1.7 gb heap. > > Are these spikes causing any issue at your end? > > There are no big spikes, the overall performance seems to be about 40% low. > > > > > > On Fri, Dec 6, 2013 at 9:10 PM, srmore wrote: > > > > On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak wrote: > Hard to say much without knowing about the cassandra configurations. > > The cassandra configuration is > -Xms8G > -Xmx8G > -Xmn800m > -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC > -XX:+CMSParallelRemarkEnabled > -XX:SurvivorRatio=4 > -XX:MaxTenuringThreshold=2 > -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly > > > Yes compactions/GC's could skipe the CPU, I had similar behavior with my > setup. > > Were you able to get around it ? > > > -VK > > > On Fri, Dec 6, 2013 at 7:40 PM, srmore wrote: > We have a 3 node cluster running cassandra 1.2.12, they are pretty big > machines 64G ram with 16 cores, cassandra heap is 8G. > > The interesting observation is that, when I send traffic to one node its > performance is 2x more than when I send traffic to all the nodes. We ran > 1.0.11 on the same box and we observed a slight dip but not half as seen with > 1.2.12. In both the cases we were writing with LOCAL_QUORUM. Changing CL to > ONE make a slight improvement but not much. > > The read_Repair_chance is 0.1. We see some compactions running. > > following is my iostat -x output, sda is the ssd (for commit log) and sdb is > the spinner. > > avg-cpu: %user %nice %system %iowait %steal %idle > 66.460.008.950.010.00 24.58 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz > avgqu-sz await svctm %util > sda 0.0027.60 0.00 4.40 0.00 256.0058.18 > 0.012.55 1.32 0.58 > sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.000.00 0.00 0.00 > sda2 0.0027.60 0.00 4.40 0.00 256.0058.18 > 0.012.55 1.32 0.58 > sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.000.00 0.00 0.00 > sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.000.00 0.00 0.00 > dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.000.00 0.00 0.00 > dm-1 0.00 0.00 0.00 0.60 0.00 4.80 8.00 > 0.005.33 2.67 0.16 > dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.000.00 0.00 0.00 > dm-3 0.00 0.00 0.00 24.8
Dimensional SUM, COUNT, & DISTINCT in C* (replacing Acunu)
We are seeking to replace Acunu in our technology stack / platform. It is the only component in our stack that is not open source. In preparation, over the last few weeks I’ve migrated Virgil to CQL. The vision is that Virgil could receive a REST request to upsert/delete data (hierarchical JSON to support collections). Virgil would lookup the dimensions/aggregations for that table, add the key to the pertinent dimensional tables (e.g. DISTINCT), incorporate values into aggregations (e.g. SUMs) and increment/decrement relevant counters (COUNT). (using additional CF’s) This seems straight-forward, but appears to require a read-before-write. (e.g. read the current value of a SUM, incorporate the new value, then use the lightweight transactions of C* 2.0 to conditionally update the value.) Before I go down this path, I was wondering if anyone is designing/working on the same, perhaps at a lower level? (CQL?) Is there any intent to support aggregations/filters (COUNT, SUM, DISTINCT, etc) at the CQL level? If so, is there a preliminary design? I can see a lower-level approach, which would leverage the commit logs (and mem/sstables) and perform the aggregation during read-operations (and flush/compaction). thoughts? i'm open to all ideas. -brian -- Brian ONeill Chief Architect, Health Market Science (http://healthmarketscience.com) mobile:215.588.6024 blog: http://brianoneill.blogspot.com/ twitter: @boneill42
Re: Bulkoutputformat
> Request did not complete within rpc_timeout. The node is overloaded and did not return in time. Check the logs for errors or excessive JVM GC and try selecting less data. Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 14/12/2013, at 10:06 am, varun allampalli wrote: > Thanks Rahul..article was insightful > > > On Fri, Dec 13, 2013 at 12:25 AM, Rahul Menon wrote: > Here you go > > http://thelastpickle.com/blog/2013/01/11/primary-keys-in-cql.html > > > On Fri, Dec 13, 2013 at 7:19 AM, varun allampalli > wrote: > Hi Aaron, > > It seems like you answered the question here. > > https://groups.google.com/forum/#!topic/nosql-databases/vjZA5vdycWA > > Can you give me the link to the blog which you mentioned > > http://thelastpickle.com/2013/01/11/primary-keys-in-cql/ > > Thanks in advance > Varun > > > On Thu, Dec 12, 2013 at 3:36 PM, varun allampalli > wrote: > Thanks Aaron, I was able to generate sstables and load using sstableloader. > But after loading the tables when I do a select query I get this, the table > has only one record. Is there anything I am missing or any logs I can look > at. > > Request did not complete within rpc_timeout. > > > On Wed, Dec 11, 2013 at 7:58 PM, Aaron Morton wrote: > If you don’t need to use Hadoop then try the SSTableSimpleWriter and > sstableloader , this post is a little old but still relevant > http://www.datastax.com/dev/blog/bulk-loading > > Otherwise AFAIK BulkOutputFormat is what you want from hadoop > http://www.datastax.com/docs/1.1/cluster_architecture/hadoop_integration > > Cheers > > - > Aaron Morton > New Zealand > @aaronmorton > > Co-Founder & Principal Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com > > On 12/12/2013, at 11:27 am, varun allampalli wrote: > >> Hi All, >> >> I want to bulk insert data into cassandra. I was wondering of using >> BulkOutputformat in hadoop. Is it the best way or using driver and doing >> batch insert is the better way. >> >> Are there any disandvantages of using bulkoutputformat. >> >> Thanks for helping >> >> Varun > > > > >
Re: Improving write performance in Cassandra and a few related issues...
> rite throughput is remaining at around 460 pkts/sec or sometimes even falling > below that rate as against the expected rate of around 920 pkts/sec. Is it > some kind of limitation of Cassandra or am I doing something wrong??? There is nothing in cassandra that would make that happen. Double check your client. >I also see an > increase in packet drops when I try to store the packets from both the hosts > into the same keyspace. The packets are getting collected properly followed > by intervals in which they are being dropped in both the systems, at the same > time. Could this be some kind of a buffer issue??? What do you mean by packet drops ? Do you mean dropped messages in cassandra ? > Also, can write throughput be increased by distributing the write requests > between the 2 Cassandra nodes instead of sending the requests to a single > node? Currently, I dont see any improvement even if I distribute the write > requests to different hosts. How can I improve the write performance overall? Normally we expect 3k to 4k non counter writes per core per node, if you are not seeing that it may be configuration or the client. Hope that helps. - Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 15/12/2013, at 7:51 pm, Krishna Chaitanya wrote: > Hello, >I am a newbie to the Cassandra world and have a few doubts which I > wanted to clarify. I am having a software switch that stores netflow packets > into a shared memory segment and a daemon that reads that memory segment and > stores them into a 2-node Cassandra cluster. Currently, I am storing the > packets from 2 hosts into 2 different keyspaces, hence only writes and no > reads. The write throughput is coming to around 460 pkts/sec in each of the > keyspaces. But, when I try to store the packets into the same keyspace, I > observe that the write throughput is remaining at around 460 pkts/sec or > sometimes even falling below that rate as against the expected rate of around > 920 pkts/sec. Is it some kind of limitation of Cassandra or am I doing > something wrong??? >I also see an > increase in packet drops when I try to store the packets from both the hosts > into the same keyspace. The packets are getting collected properly followed > by intervals in which they are being dropped in both the systems, at the same > time. Could this be some kind of a buffer issue??? > The write requests from both the systems are > sent to the same node which is also the seed node. I am mostly using the > default Cassandra configuration with replication_factor set to 1 and without > durable_writes. The systems are i5s with 4 gb RAM. The data model is: each > second is the row key with all the packets collected in that second as the > columns. Also, can write throughput be increased by distributing the write > requests between the 2 Cassandra nodes instead of sending the requests to a > single node? Currently, I dont see any improvement even if I distribute the > write requests to different hosts. How can I improve the write performance > overall? > > Thanks. > -- > Regards, > BNSK.
Re: Cassandra 1.2 : OutOfMemoryError: unable to create new native thread
Try using jstack to see if there are a lot of threads there. Are you using vNodea and Hadoop ? https://issues.apache.org/jira/browse/CASSANDRA-6169 Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 17/12/2013, at 2:48 am, Maciej Miklas wrote: > the cassandra-env.sh has option > JVM_OPTS="$JVM_OPTS -Xss180k" > > it will give this error if you start cassandra with java 7. So increase the > value, or remove option. > > Regards, > Maciej > > > On Mon, Dec 16, 2013 at 2:37 PM, srmore wrote: > What is your thread stack size (xss) ? try increasing that, that could help. > Sometimes the limitation is imposed by the host provider (e.g. amazon ec2 > etc.) > > Thanks, > Sandeep > > > On Mon, Dec 16, 2013 at 6:53 AM, Oleg Dulin wrote: > Hi guys! > > I beleive my limits settings are correct. Here is the output of "ulimits -a": > > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 1547135 > max locked memory (kbytes, -l) unlimited > max memory size (kbytes, -m) unlimited > open files (-n) 10 > pipe size(512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 32768 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > However, I just had a couple of cassandra nodes go down over the weekend for > no apparent reason with the following error: > > java.lang.OutOfMemoryError: unable to create new native thread >at java.lang.Thread.start0(Native Method) >at java.lang.Thread.start(Thread.java:691) >at > java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) > >at > java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017) > >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163) > >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >at java.lang.Thread.run(Thread.java:722) > > Any input is greatly appreciated. > -- > Regards, > Oleg Dulin > http://www.olegdulin.com > > > >
Re: Cassandra 1.1.6 - Disk usage and Load displayed in ring doesn't match
-tmp- files will sit in the data dir, if there was an error creating them during compaction or flushing to disk they will sit around until a restart. Check the logs for errors to see if compaction was failing on something. Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 17/12/2013, at 12:28 pm, Narendra Sharma wrote: > No snapshots. > > I restarted the node and now the Load in ring is in sync with the disk usage. > Not sure what caused it to go out of sync. However, the Live SStable count > doesn't match exactly with the number of data files on disk. > > I am going through the Cassandra code to understand what could be the reason > for the mismatch in the sstable count and also why there is no reference of > some of the data files in system.log. > > > > > On Mon, Dec 16, 2013 at 2:45 PM, Arindam Barua wrote: > > > Do you have any snapshots on the nodes where you are seeing this issue? > > Snapshots will link to sstables which will cause them not be deleted. > > > > -Arindam > > > > From: Narendra Sharma [mailto:narendra.sha...@gmail.com] > Sent: Sunday, December 15, 2013 1:15 PM > To: user@cassandra.apache.org > Subject: Cassandra 1.1.6 - Disk usage and Load displayed in ring doesn't match > > > > We have 8 node cluster. Replication factor is 3. > > > > For some of the nodes the Disk usage (du -ksh .) in the data directory for CF > doesn't match the Load reported in nodetool ring command. When we expanded > the cluster from 4 node to 8 nodes (4 weeks back), everything was okay. Over > period of last 2-3 weeks the disk usage has gone up. We increased the RF from > 2 to 3 2 weeks ago. > > > > I am not sure if increasing the RF is causing this issue. > > > > For one of the nodes that I analyzed: > > 1. nodetool ring reported load as 575.38 GB > > > > 2. nodetool cfstats for the CF reported: > > SSTable count: 28 > > Space used (live): 572671381955 > > Space used (total): 572671381955 > > > > > > 3. 'ls -1 *Data* | wc -l' in the data folder for CF returned > > 46 > > > > 4. 'du -ksh .' in the data folder for CF returned > > 720G > > > > The above numbers indicate that there are some sstables that are obsolete and > are still occupying space on disk. What could be wrong? Will restarting the > node help? The cassandra process is running for last 45 days with no > downtime. However, because the disk usage is high, we are not able to run > full compaction. > > > > Also, I can't find reference to each of the sstables on disk in the > system.log file. For eg I have one data file on disk as (ls -lth): > > 86G Nov 20 06:14 > > > > I have system.log file with first line: > > INFO [main] 2013-11-18 09:41:56,120 AbstractCassandraDaemon.java (line 101) > Logging initialized > > > > The 86G file must be a result of some compaction. I see no reference of data > file in system.log file between 11/18 to 11/25. What could be the reason for > that? The only reference is dated 11/29 when the file was being streamed to > another node (new node). > > > > How can I identify the obsolete files and remove them? I am thinking about > following. Let me know if it make sense. > > 1. Restart the node and check the state. > > 2. Move the oldest data files to another location (to another mount point) > > 3. Restart the node again > > 4. Run repair on the node so that it can get the missing data from its peers. > > > > > > I compared the numbers of a healthy node for the same CF: > > 1. nodetool ring reported load as 662.95 GB > > > > 2. nodetool cfstats for the CF reported: > > SSTable count: 16 > > Space used (live): 670524321067 > > Space used (total): 670524321067 > > > > 3. 'ls -1 *Data* | wc -l' in the data folder for CF returned > > 16 > > > > 4. 'du -ksh .' in the data folder for CF returned > > 625G > > > > > > -Naren > > > > > > > > -- > Narendra Sharma > > Software Engineer > > http://www.aeris.com > > http://narendrasharma.blogspot.com/ > > > > > > > -- > Narendra Sharma > Software Engineer > http://www.aeris.com > http://narendrasharma.blogspot.com/ >
Re: various Cassandra performance problems when CQL3 is really used
> * select id from table where token(id) > token(some_value) and > secondary_index = other_val limit 2 allow filtering; > > Filtering absolutely kills the performance. On a table populated with 130.000 > records, single node Cassandra server (on my i7 notebook, 2GB of JVM heap) > and secondary index built on column with low cardinality of its value set > this query takes 156 seconds to finish. Yes, this is why you have to add allow_filtering. You are asking the nodes to read all the data that matches and filter in memory, that’s a SQL type operation. Your example query is somewhat complex and I doubt it could get decent performance, what does the query plan look like? IMHO you need to do further de-normalisation, you will get the best performance when you select rows by their full or part primary key. > By the way, the performance is order of magnitude better if this patch is > applied: That looks like it’s tuned to your specific need, it would ignore the max results included in the query. > * select id from table; > > As we saw in the trace log, the query - although it queries just row ids - > scans all columns of all the rows and (probably) compares TTL with current > time (?) (we saw hundreds of thousands of gettimeofday(2)). This means that > if the table somehow mixes wide and narrow rows, the performance suffers > horribly. Select all rows from a table requires a range scan, which reads all rows from all nodes. It should never be used production. Not sure what you mean by “scans all columns from all rows” a select by column name will use a SliceByNamesReadCommand which will only read the required columns from each SSTable (it normally short circuits though and read from less). if there is a TTL the ExpiringColumn.localExpirationTime must be checked, if there is no TTL it will no be checked. > As Cassandra checks all the columns in selects, performance suffers badly if > the collection is of any interesting size. This is not true, could you provide an example where you think this is happening ? > Additionally, we saw various random irreproducible freezes, high CPU > consumption when nothing happens (even with trace log level set no activity > was reported) and highly inpredictable performance characteristics after > nodetool flush and/or major compaction. What was the HW platform and what was the load ? Typically freezes in the server correlate to JVM GC, the JVM GC can also be using the CPU. If you have wide rows or make large reads you may run into more JVM GC issues. nodetool flush will (as it says) flush all the tables to disk, if you have a lot tables and/or a lot of secondary indexes this can cause the switch lock to be held preventing write threads from progressing. Once flush threads stop waiting on the flush queue the lock will be released. See the help for memtable_flush_queue_size in the yaml file. major compaction is not recommended to be used in production. If you are seeing it cause performance problems I would guess it is related to JVM GC and/or the disk IO is not able to keep up. When used it creates a single SSTable for each table which will not be compacted again until (default) 3 other large SSTables are created or you run major compaction again. For this reason it is not recommended. > Conclusions: > > - do not use collections > - do not use secondary indexes > - do not use filtering > - have your rows as narrow as possible if you need any kind of all row keys > traversal These features all have a use, but it looks like you leaned on them heavily while creating a relational model. Specially the filtering, you have to explicitly enable it to prevent the client sending queries that will take a long time. The only time row key traversal is used normally is reading data through hadoop. You should always strive to read row(s) from a table by the full or partial primary key. > With these conclusions in mind, CQL seems redundant, plain old thrift may be > used, joins should be done client side and/or all indexes need to be handled > manually. Correct? No. CQL provide a set of functionality not present in the thrift API. Joins and indexes should generally be handled by denormlaising the data during writes. It sounds like your data model was too relational, you need to denormalise and read rows by primary key. Secondary indexes are useful when you have a query pattern that is used infrequently. Hope that helps. - Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 18/12/2013, at 3:47 am, Ondřej Černoš wrote: > Hi all, > > we are reimplementing a legacy interface of an inventory-like service > (currently built on top of mysql) on Cassandra and I thought I would share > some findings with the list. The interface semantics is given and cannot be > changed. We chose Cassandra due to its multiple datacenter c
Re: Improving write performance in Cassandra and a few related issues...
Thanks for the reply. By packet drops I mean, the client is not able to read the shared memory as fast as the software switch is writing into it.. I doubt its the issue with the client but can you in particular issues that could cause this type of scenario? Also, I would like to know if in general , distribution of write requests to different Casaandra nodes instead of only to one, leads to increased write performance in Cassandra. Is there any particular way in which write performance can be measured, preferably from the client??? On Dec 18, 2013 8:30 AM, "Aaron Morton" wrote: > rite throughput is remaining at around 460 pkts/sec or sometimes even > falling below that rate as against the expected rate of around 920 > pkts/sec. Is it some kind of limitation of Cassandra or am I doing > something wrong??? > > There is nothing in cassandra that would make that happen. Double check > your client. > >I also see an > increase in packet drops when I try to store the packets from both the > hosts into the same keyspace. The packets are getting collected properly > followed by intervals in which they are being dropped in both the systems, > at the same time. Could this be some kind of a buffer issue??? > > What do you mean by packet drops ? > > Do you mean dropped messages in cassandra ? > > Also, can write throughput be increased by distributing the write requests > between the 2 Cassandra nodes instead of sending the requests to a single > node? Currently, I dont see any improvement even if I distribute the write > requests to different hosts. How can I improve the write performance > overall? > > Normally we expect 3k to 4k non counter writes per core per node, if you > are not seeing that it may be configuration or the client. > > Hope that helps. > > - > Aaron Morton > New Zealand > @aaronmorton > > Co-Founder & Principal Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com > > On 15/12/2013, at 7:51 pm, Krishna Chaitanya > wrote: > > Hello, >I am a newbie to the Cassandra world and have a few doubts which I > wanted to clarify. I am having a software switch that stores netflow > packets into a shared memory segment and a daemon that reads that memory > segment and stores them into a 2-node Cassandra cluster. Currently, I am > storing the packets from 2 hosts into 2 different keyspaces, hence only > writes and no reads. The write throughput is coming to around 460 pkts/sec > in each of the keyspaces. But, when I try to store the packets into the > same keyspace, I observe that the write throughput is remaining at around > 460 pkts/sec or sometimes even falling below that rate as against the > expected rate of around 920 pkts/sec. Is it some kind of limitation of > Cassandra or am I doing something wrong??? >I also see an > increase in packet drops when I try to store the packets from both the > hosts into the same keyspace. The packets are getting collected properly > followed by intervals in which they are being dropped in both the systems, > at the same time. Could this be some kind of a buffer issue??? > The write requests from both the systems are > sent to the same node which is also the seed node. I am mostly using the > default Cassandra configuration with replication_factor set to 1 and > without durable_writes. The systems are i5s with 4 gb RAM. The data model > is: each second is the row key with all the packets collected in that > second as the columns. Also, can write throughput be increased by > distributing the write requests between the 2 Cassandra nodes instead of > sending the requests to a single node? Currently, I dont see any > improvement even if I distribute the write requests to different hosts. > How can I improve the write performance overall? > > Thanks. > -- > Regards, > BNSK > *. * > > >
