Re: Why so many SSTables?
I've just opened a new JIRA: CASSANDRA-4142 I've double checked numbers, 7747 seems to be array list object's capacity (Eclipse Memory Analyzer displays "java.lang.Object[7747] @ 0x7d3f3f798"). Actually there are 5757 browsable entries in EMA therefore each object is about 140 KB (size varies between 143088 bits and 143768 bits). We have no pending compactions tasks. Our cluster is currently under-loaded. Our goal is to handle hundreds of tera bytes which explains 1 TB per node. Our need is to archive data so our cluster is not running under a read-heavy load. @Dave Brosius: I made a mistake. To be correct 786 MB is 47% of *leak suspects* as reported by Eclipse Memory Analyser. Our Cassandra nodes are pretty standard: 10 GB of ram with Xmx set to 2,5 GB. Regards, Romain
super column
Can a super column contain a super column? I mean is nesting of supercolumns possible? Regards, Puneet :)
Re: super column
No. Maybe that a super column can contain composite columns but I'm not sure. Paolo On Apr 12, 2012 12:15 PM, "puneet loya" wrote: > Can a super column contain a super column? > > I mean is nesting of supercolumns possible? > > Regards, > > Puneet :) >
Re: super column
what is composite columns? super column under it can contain just multiple columns.. will composite columns be useful? On Thu, Apr 12, 2012 at 3:50 PM, Paolo Bernardi wrote: > No. > Maybe that a super column can contain composite columns but I'm not sure. > > Paolo > On Apr 12, 2012 12:15 PM, "puneet loya" wrote: > >> Can a super column contain a super column? >> >> I mean is nesting of supercolumns possible? >> >> Regards, >> >> Puneet :) >> >
Re: super column
http://www.datastax.com/dev/blog/introduction-to-composite-columns-part-1 hope this help. maki 2012/4/12 puneet loya : > what is composite columns? > > super column under it can contain just multiple columns.. will composite > columns be useful? > > > On Thu, Apr 12, 2012 at 3:50 PM, Paolo Bernardi wrote: >> >> No. >> Maybe that a super column can contain composite columns but I'm not sure. >> >> Paolo >> >> On Apr 12, 2012 12:15 PM, "puneet loya" wrote: >>> >>> Can a super column contain a super column? >>> >>> I mean is nesting of supercolumns possible? >>> >>> Regards, >>> >>> Puneet :) > >
Re: Repair Process Taking too long
I also noticed that if I use the -pr option, the repair process went down from 30 hours to 9 hours. Is the -pr option safe to use if I want to run repair processes in parallel on nodes that are not replication peers? thanks On Thu, Apr 12, 2012 at 12:06 AM, Frank Ng wrote: > Thank you for confirming that the per node data size is most likely > causing the long repair process. I have tried a repair on smaller column > families and it was significantly faster. > > On Wed, Apr 11, 2012 at 9:55 PM, aaron morton wrote: > >> If you have 1TB of data it will take a long time to repair. Every bit of >> data has to be read and a hash generated. This is one of the reasons we >> often suggest that around 300 to 400Gb per node is a good load in the >> general case. >> >> Look at nodetool compactionstats .Is there a validation compaction >> running ? If so it is still building the merkle hash tree. >> >> Look at nodetool netstats . Is it streaming data ? If so all hash trees >> have been calculated. >> >> Cheers >> >> >> - >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 12/04/2012, at 2:16 AM, Frank Ng wrote: >> >> Can you expand further on your issue? Were you using Random Patitioner? >> >> thanks >> >> On Tue, Apr 10, 2012 at 5:35 PM, David Leimbach wrote: >> >>> I had this happen when I had really poorly generated tokens for the >>> ring. Cassandra seems to accept numbers that are too big. You get hot >>> spots when you think you should be balanced and repair never ends (I think >>> there is a 48 hour timeout). >>> >>> >>> On Tuesday, April 10, 2012, Frank Ng wrote: >>> I am not using tier-sized compaction. On Tue, Apr 10, 2012 at 12:56 PM, Jonathan Rhone wrote: > Data size, number of nodes, RF? > > Are you using size-tiered compaction on any of the column families > that hold a lot of your data? > > Do your cassandra logs say you are streaming a lot of ranges? > zgrep -E "(Performing streaming repair|out of sync)" > > > On Tue, Apr 10, 2012 at 9:45 AM, Igor wrote: > >> On 04/10/2012 07:16 PM, Frank Ng wrote: >> >> Short answer - yes. >> But you are asking wrong question. >> >> >> I think both processes are taking a while. When it starts up, >> netstats and compactionstats show nothing. Anyone out there successfully >> using ext3 and their repair processes are faster than this? >> >> On Tue, Apr 10, 2012 at 10:42 AM, Igor wrote: >> >>> Hi >>> >>> You can check with nodetool which part of repair process is slow - >>> network streams or verify compactions. use nodetool netstats or >>> compactionstats. >>> >>> >>> On 04/10/2012 05:16 PM, Frank Ng wrote: >>> Hello, I am on Cassandra 1.0.7. My repair processes are taking over 30 hours to complete. Is it normal for the repair process to take this long? I wonder if it's because I am using the ext3 file system. thanks >>> >>> >> >> > > > -- > Jonathan Rhone > Software Engineer > > *TinyCo* > 800 Market St., Fl 6 > San Francisco, CA 94102 > www.tinyco.com > > >> >> >
Re: Cassandra running out of memory?
Hello Aaron, Thank you for getting back to me. I will change to m1.large first to see how long it will take Cassandra node to die (if at all). If again not happy I will try more memory. I just want to test it step by step and see what the differences are. I will also change the cassandra-env file back to defaults. Is there an absolute minimum requirement for Cassandra in terms of memory? I might be wrong, but from my understanding we shouldn't have any problems given the amount of data we store per day (currently approximately 2-2.5G / day). Thank you in advance, Bill On Wed, Apr 11, 2012 at 7:33 PM, aaron morton wrote: > 'system_memory_in_mb' (3760) and the 'system_cpu_cores' (1) according to > our nodes' specification. We also changed the 'MAX_HEAP_SIZE' to 2G and the > 'HEAP_NEWSIZE' to 200M (we think the second is related to the Garbage > Collection). > > It's best to leave the default settings unless you know what you are doing > here. > > In case you find this useful, swap is off and unevictable memory seems to > be very high on all 3 servers (2.3GB, we usually observe the amount of > unevictable memory on other Linux servers of around 0-16KB) > > Cassandra locks the java memory so it cannot be swapped out. > > The problem is that the node we hit from our thrift interface dies > regularly (approximately after we store 2-2.5G of data). Error message: > OutOfMemoryError: Java Heap Space and according to the log it in fact used > all of the allocated memory. > > The easiest solution will be to use a larger EC2 instance. > > People normally use an m1.xlarge with 16Gb of ram (you would also try an > m1.large). > > If you are still experimenting I would suggest using the larger instances > so you can make some progress. Once you have a feel for how things work you > can then try to match the instances to your budget. > > Hope that helps. > > - > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 11/04/2012, at 1:54 AM, Vasileios Vlachos wrote: > > Hello, > > We are experimenting a bit with Cassandra lately (version 1.0.7) and we > seem to have some problems with memory. We use EC2 as our test environment > and we have three nodes with 3.7G of memory and 1 core @ 2.4G, all running > Ubuntu server 11.10. > > The problem is that the node we hit from our thrift interface dies > regularly (approximately after we store 2-2.5G of data). Error message: > OutOfMemoryError: Java Heap Space and according to the log it in fact used > all of the allocated memory. > > The nodes are under relatively constant load and store about 2000-4000 row > keys a minute, which are batched through the Trift interface in 10-30 row > keys at once (with about 50 columns each). The number of reads is very low > with around 1000-2000 a day and only requesting the data of a single row > key. The is currently only one used column family. > > The initial thought was that something was wrong in the cassandra-env.sh > file. So, we specified the variables 'system_memory_in_mb' (3760) and the > 'system_cpu_cores' (1) according to our nodes' specification. We also > changed the 'MAX_HEAP_SIZE' to 2G and the 'HEAP_NEWSIZE' to 200M (we think > the second is related to the Garbage Collection). Unfortunately, that did > not solve the issue and the node we hit via thrift keeps on dying regularly. > > In case you find this useful, swap is off and unevictable memory seems to > be very high on all 3 servers (2.3GB, we usually observe the amount of > unevictable memory on other Linux servers of around 0-16KB) (We are not > quite sure how the unevictable memory ties into Cassandra, its just > something we observed while looking into the problem). The CPU is pretty > much idle the entire time. The heap memory is clearly being reduced once in > a while according to nodetool, but obviously grows over the limit as time > goes by. > > Any ideas? Thanks in advance. > > Bill > > >
Re: Repair Process Taking too long
On Thu, Apr 12, 2012 at 4:06 PM, Frank Ng wrote: > I also noticed that if I use the -pr option, the repair process went down > from 30 hours to 9 hours. Is the -pr option safe to use if I want to run > repair processes in parallel on nodes that are not replication peers? There is pretty much two use case for repair: 1) to rebuild a node: if say a node has lost some data due to a hard drive corruption or the like and you want to to rebuild what's missing 2) the periodic repairs to avoid problem with deleted data coming back from the dead (basically: http://wiki.apache.org/cassandra/Operations#Frequency_of_nodetool_repair) In case 1) you want to run 'nodetool repair' (without -pr) against the node to rebuild. In case 2) (which I suspect is the case your talking now), you *want* to use 'nodetool repair -pr' on *every* node of the cluster. I.e. that's the most efficient way to do it. The only reason not to use -pr in this case would be that it's not available because you're using an old version of Cassandra. And yes, it's is safe to run with -pr in parallel on nodes that are not replication peers. -- Sylvain > > thanks > > > On Thu, Apr 12, 2012 at 12:06 AM, Frank Ng wrote: >> >> Thank you for confirming that the per node data size is most likely >> causing the long repair process. I have tried a repair on smaller column >> families and it was significantly faster. >> >> On Wed, Apr 11, 2012 at 9:55 PM, aaron morton >> wrote: >>> >>> If you have 1TB of data it will take a long time to repair. Every bit of >>> data has to be read and a hash generated. This is one of the reasons we >>> often suggest that around 300 to 400Gb per node is a good load in the >>> general case. >>> >>> Look at nodetool compactionstats .Is there a validation compaction >>> running ? If so it is still building the merkle hash tree. >>> >>> Look at nodetool netstats . Is it streaming data ? If so all hash trees >>> have been calculated. >>> >>> Cheers >>> >>> >>> - >>> Aaron Morton >>> Freelance Developer >>> @aaronmorton >>> http://www.thelastpickle.com >>> >>> On 12/04/2012, at 2:16 AM, Frank Ng wrote: >>> >>> Can you expand further on your issue? Were you using Random Patitioner? >>> >>> thanks >>> >>> On Tue, Apr 10, 2012 at 5:35 PM, David Leimbach >>> wrote: I had this happen when I had really poorly generated tokens for the ring. Cassandra seems to accept numbers that are too big. You get hot spots when you think you should be balanced and repair never ends (I think there is a 48 hour timeout). On Tuesday, April 10, 2012, Frank Ng wrote: > > I am not using tier-sized compaction. > > > On Tue, Apr 10, 2012 at 12:56 PM, Jonathan Rhone > wrote: >> >> Data size, number of nodes, RF? >> >> Are you using size-tiered compaction on any of the column families >> that hold a lot of your data? >> >> Do your cassandra logs say you are streaming a lot of ranges? >> zgrep -E "(Performing streaming repair|out of sync)" >> >> >> On Tue, Apr 10, 2012 at 9:45 AM, Igor wrote: >>> >>> On 04/10/2012 07:16 PM, Frank Ng wrote: >>> >>> Short answer - yes. >>> But you are asking wrong question. >>> >>> >>> I think both processes are taking a while. When it starts up, >>> netstats and compactionstats show nothing. Anyone out there >>> successfully >>> using ext3 and their repair processes are faster than this? >>> >>> On Tue, Apr 10, 2012 at 10:42 AM, Igor wrote: Hi You can check with nodetool which part of repair process is slow - network streams or verify compactions. use nodetool netstats or compactionstats. On 04/10/2012 05:16 PM, Frank Ng wrote: > > Hello, > > I am on Cassandra 1.0.7. My repair processes are taking over 30 > hours to complete. Is it normal for the repair process to take this > long? > I wonder if it's because I am using the ext3 file system. > > thanks >>> >>> >> >> >> >> -- >> Jonathan Rhone >> Software Engineer >> >> TinyCo >> 800 Market St., Fl 6 >> San Francisco, CA 94102 >> www.tinyco.com >> > >>> >>> >> >
Re: solr query for string match in CQL
Never mind. Double quotes within the single quotes worked: select * from solr where solr_query='body:"sixty eight million nine hundred forty three thousand four hundred twenty four"'; On Thu, Apr 12, 2012 at 11:42 AM, A J wrote: > What is the syntax for a string match in CQL for solr_query ? > > cqlsh:wiki> select * from solr where solr_query='body:sixty eight > million nine hundred forty three thousand four hundred twenty four'; > Request did not complete within rpc_timeout. > > url encoding just returns without retrieving the row present: > cqlsh:wiki> select count(*) from solr where > solr_query='body:%22sixty%20eight%20million%20nine%20hundred%20forty%20three%20thousand%20four%20hundred%20twenty%20four%22' > ; > count > --- > 0 > > I have exactly one row matching this string that I can retrieve > through direct solr query. > > > Thanks.
Off-heap row cache and mmapped sstables
Hi, Cassandra issues an mlockall [1] before mmap-ing sstables to prevent the kernel from paging out heap space in favor of memory-mapped sstables. I was wondering, what happens to the off-heap row cache (saved or unsaved)? Is it possible that the kernel pages out off-heap row cache in favor of resident mmap-ed sstable pages? Thanks, Omid [1] http://pubs.opengroup.org/onlinepubs/007908799/xsh/mlockall.html
Trying to avoid super columns
I am currently working on a data model where the purpose is to look up
multiple products for given days of the year. Right now, that model
involves the usage of a super column family. e.g.
"2012-04-12": {
"product_id_1": {
price: 12.44,
tax: 1.00,
fees: 3.00,
},
"product_id_2": {
price: 50.00,
tax: 4.00,
fees: 10.00
}
}
I should note that for a given day/key, we are expecting in the range of 2
million to 4 million products (subcolumns).
With this model, I am able to retrieve any of the products for a given day
using hector's MultigetSuperSliceQuery.
I am looking into changing this model to use Composite column names. How
would I go about modeling this? My initial thought is to migrate the above
model into something more like the following.
"2012-04-12": {
"product_id_1:price": 12.44,
"product_id_1:tax": 1.00,
"product_id_1:fees": 3.00,
"product_id_2:price": 50.00,
"product_id_2:tax": 4.00,
"product_id_2:fees": 10.00,
}
The one thing that stands out to me with this approach is the number of
additonal columns that will be created for a single key. Will the increase
in columns, create new issues I will need to deal with?
Are there any other thoughts about if I should actually move forward (or
not) with migration this super column family to the model with the
component column names?
Thanks,
Phil
Re: Trying to avoid super columns
If you want to reduce the number of columns, you could pack all the data
for a product into one column, as in
composite column name-> product_id_1:12.44:1.00:3.00
On 04/12/2012 03:03 PM, Philip Shon wrote:
I am currently working on a data model where the purpose is to look up
multiple products for given days of the year. Right now, that model
involves the usage of a super column family. e.g.
"2012-04-12": {
"product_id_1": {
price: 12.44,
tax: 1.00,
fees: 3.00,
},
"product_id_2": {
price: 50.00,
tax: 4.00,
fees: 10.00
}
}
I should note that for a given day/key, we are expecting in the range
of 2 million to 4 million products (subcolumns).
With this model, I am able to retrieve any of the products for a given
day using hector's MultigetSuperSliceQuery.
I am looking into changing this model to use Composite column names.
How would I go about modeling this? My initial thought is to migrate
the above model into something more like the following.
"2012-04-12": {
"product_id_1:price": 12.44,
"product_id_1:tax": 1.00,
"product_id_1:fees": 3.00,
"product_id_2:price": 50.00,
"product_id_2:tax": 4.00,
"product_id_2:fees": 10.00,
}
The one thing that stands out to me with this approach is the number
of additonal columns that will be created for a single key. Will the
increase in columns, create new issues I will need to deal with?
Are there any other thoughts about if I should actually move forward
(or not) with migration this super column family to the model with the
component column names?
Thanks,
Phil
Re: Repair Process Taking too long
Thanks for the clarification. I'm running repairs as in case 2 (to avoid deleted data coming back). On Thu, Apr 12, 2012 at 10:59 AM, Sylvain Lebresne wrote: > On Thu, Apr 12, 2012 at 4:06 PM, Frank Ng wrote: > > I also noticed that if I use the -pr option, the repair process went down > > from 30 hours to 9 hours. Is the -pr option safe to use if I want to run > > repair processes in parallel on nodes that are not replication peers? > > There is pretty much two use case for repair: > 1) to rebuild a node: if say a node has lost some data due to a hard > drive corruption or the like and you want to to rebuild what's missing > 2) the periodic repairs to avoid problem with deleted data coming back > from the dead (basically: > http://wiki.apache.org/cassandra/Operations#Frequency_of_nodetool_repair) > > In case 1) you want to run 'nodetool repair' (without -pr) against the > node to rebuild. > In case 2) (which I suspect is the case your talking now), you *want* > to use 'nodetool repair -pr' on *every* node of the cluster. I.e. > that's the most efficient way to do it. The only reason not to use -pr > in this case would be that it's not available because you're using an > old version of Cassandra. And yes, it's is safe to run with -pr in > parallel on nodes that are not replication peers. > > -- > Sylvain > > > > > > thanks > > > > > > On Thu, Apr 12, 2012 at 12:06 AM, Frank Ng wrote: > >> > >> Thank you for confirming that the per node data size is most likely > >> causing the long repair process. I have tried a repair on smaller > column > >> families and it was significantly faster. > >> > >> On Wed, Apr 11, 2012 at 9:55 PM, aaron morton > >> wrote: > >>> > >>> If you have 1TB of data it will take a long time to repair. Every bit > of > >>> data has to be read and a hash generated. This is one of the reasons we > >>> often suggest that around 300 to 400Gb per node is a good load in the > >>> general case. > >>> > >>> Look at nodetool compactionstats .Is there a validation compaction > >>> running ? If so it is still building the merkle hash tree. > >>> > >>> Look at nodetool netstats . Is it streaming data ? If so all hash trees > >>> have been calculated. > >>> > >>> Cheers > >>> > >>> > >>> - > >>> Aaron Morton > >>> Freelance Developer > >>> @aaronmorton > >>> http://www.thelastpickle.com > >>> > >>> On 12/04/2012, at 2:16 AM, Frank Ng wrote: > >>> > >>> Can you expand further on your issue? Were you using Random Patitioner? > >>> > >>> thanks > >>> > >>> On Tue, Apr 10, 2012 at 5:35 PM, David Leimbach > >>> wrote: > > I had this happen when I had really poorly generated tokens for the > ring. Cassandra seems to accept numbers that are too big. You get > hot > spots when you think you should be balanced and repair never ends (I > think > there is a 48 hour timeout). > > > On Tuesday, April 10, 2012, Frank Ng wrote: > > > > I am not using tier-sized compaction. > > > > > > On Tue, Apr 10, 2012 at 12:56 PM, Jonathan Rhone > > wrote: > >> > >> Data size, number of nodes, RF? > >> > >> Are you using size-tiered compaction on any of the column families > >> that hold a lot of your data? > >> > >> Do your cassandra logs say you are streaming a lot of ranges? > >> zgrep -E "(Performing streaming repair|out of sync)" > >> > >> > >> On Tue, Apr 10, 2012 at 9:45 AM, Igor wrote: > >>> > >>> On 04/10/2012 07:16 PM, Frank Ng wrote: > >>> > >>> Short answer - yes. > >>> But you are asking wrong question. > >>> > >>> > >>> I think both processes are taking a while. When it starts up, > >>> netstats and compactionstats show nothing. Anyone out there > successfully > >>> using ext3 and their repair processes are faster than this? > >>> > >>> On Tue, Apr 10, 2012 at 10:42 AM, Igor > wrote: > > Hi > > You can check with nodetool which part of repair process is slow > - > network streams or verify compactions. use nodetool netstats or > compactionstats. > > > On 04/10/2012 05:16 PM, Frank Ng wrote: > > > > Hello, > > > > I am on Cassandra 1.0.7. My repair processes are taking over 30 > > hours to complete. Is it normal for the repair process to take > this long? > > I wonder if it's because I am using the ext3 file system. > > > > thanks > > > >>> > >>> > >> > >> > >> > >> -- > >> Jonathan Rhone > >> Software Engineer > >> > >> TinyCo > >> 800 Market St., Fl 6 > >> San Francisco, CA 94102 > >> www.tinyco.com > >> > > > >>> > >>> > >> > > >
Re: Why so many SSTables?
>From my experience I would strongly advise against leveled compaction for your use-case. But you should certainly test and see for yourself! I have ~1TB on a node with ~13GB of heap. I ended up with 30k SSTables. I raised the SSTable size to 100MB but that didn't prove to be sufficient and I did it too late. I also had issues along the way where I ran out of heap space (before going to 13GB) and the flushing that then happens caused lots of small SSTables to be produced. Loading up 30k SSTables at start-up took 30+ minutes. With 30k SSTables I had 5 levels, if I remember correctly, that means everything goes through 5 compactions in the long run and the resulting cpu load is just "phenomenal". I was using snappy compressor, maybe that exacerbated the problem. At some point I overloaded thing with writes and had a compaction backlog of over 3000. I stopped all data loading and after several days it was still compacting away and not about to end. I eventually switched to size tiered compaction and life is back to normal... Of course, leveled compaction may work wonderfully for others, this is just what I experienced. If I were to try it again, I'd watch my SSTable count and abort the experiment if I end up with >10k and I'd watch my compaction backlog and abort if it goes above 100. - Thorsten On 4/12/2012 12:54 AM, Romain HARDOUIN wrote: > > I've just opened a new JIRA: CASSANDRA-4142 > > I've double checked numbers, 7747 seems to be array list object's > capacity (Eclipse Memory Analyzer displays "java.lang.Object[7747] @ > 0x7d3f3f798"). > Actually there are 5757 browsable entries in EMA therefore each object > is about 140 KB (size varies between 143088 bits and 143768 bits). > > We have no pending compactions tasks. Our cluster is currently > under-loaded. > > Our goal is to handle hundreds of tera bytes which explains 1 TB per > node. Our need is to archive data so our cluster is not running under > a read-heavy load. > > @Dave Brosius: I made a mistake. To be correct 786 MB is 47% of *leak > suspects* as reported by Eclipse Memory Analyser. > Our Cassandra nodes are pretty standard: 10 GB of ram with Xmx set to > 2,5 GB. > > Regards, > > Romain
RMI/JMX errors, weird
Hello, I'm doing compactions under 0.8.8. Recently, I started seeing a stack trace like one below, and I can't figure out what causes this to appear. The cluster has been in operation for mode than half a year w/o errors like this one. Any help will be appreciated, Thanks Maxim WARNING: Failed to check the connection: java.net.SocketTimeoutException: Read timed out Exception in thread "main" java.io.IOException: Repair command #1: some repair session(s) failed (see log for details). at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1613) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)
Re: Cassandra running out of memory?
It depends on a lot of things: schema size, caches, work load etc. If your are just starting out I would recommend using a machine with 8gb or 16gb total ram. By default cassandra will take about 4gb or 8gb (respectively) for the JVM. Once you have a feel for how things work you should be able to estimate the resources your application will need. Hope that helps. - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 13/04/2012, at 2:19 AM, Vasileios Vlachos wrote: > Hello Aaron, > > Thank you for getting back to me. > > I will change to m1.large first to see how long it will take Cassandra node > to die (if at all). If again not happy I will try more memory. I just want to > test it step by step and see what the differences are. I will also change the > cassandra-env file back to defaults. > > Is there an absolute minimum requirement for Cassandra in terms of memory? I > might be wrong, but from my understanding we shouldn't have any problems > given the amount of data we store per day (currently approximately 2-2.5G / > day). > > Thank you in advance, > > Bill > > > On Wed, Apr 11, 2012 at 7:33 PM, aaron morton wrote: >> 'system_memory_in_mb' (3760) and the 'system_cpu_cores' (1) according to our >> nodes' specification. We also changed the 'MAX_HEAP_SIZE' to 2G and the >> 'HEAP_NEWSIZE' to 200M (we think the second is related to the Garbage >> Collection). > It's best to leave the default settings unless you know what you are doing > here. > >> In case you find this useful, swap is off and unevictable memory seems to be >> very high on all 3 servers (2.3GB, we usually observe the amount of >> unevictable memory on other Linux servers of around 0-16KB) > Cassandra locks the java memory so it cannot be swapped out. > >> The problem is that the node we hit from our thrift interface dies regularly >> (approximately after we store 2-2.5G of data). Error message: >> OutOfMemoryError: Java Heap Space and according to the log it in fact used >> all of the allocated memory. > The easiest solution will be to use a larger EC2 instance. > > People normally use an m1.xlarge with 16Gb of ram (you would also try an > m1.large). > > If you are still experimenting I would suggest using the larger instances so > you can make some progress. Once you have a feel for how things work you can > then try to match the instances to your budget. > > Hope that helps. > > - > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 11/04/2012, at 1:54 AM, Vasileios Vlachos wrote: > >> Hello, >> >> We are experimenting a bit with Cassandra lately (version 1.0.7) and we seem >> to have some problems with memory. We use EC2 as our test environment and we >> have three nodes with 3.7G of memory and 1 core @ 2.4G, all running Ubuntu >> server 11.10. >> >> The problem is that the node we hit from our thrift interface dies regularly >> (approximately after we store 2-2.5G of data). Error message: >> OutOfMemoryError: Java Heap Space and according to the log it in fact used >> all of the allocated memory. >> >> The nodes are under relatively constant load and store about 2000-4000 row >> keys a minute, which are batched through the Trift interface in 10-30 row >> keys at once (with about 50 columns each). The number of reads is very low >> with around 1000-2000 a day and only requesting the data of a single row >> key. The is currently only one used column family. >> >> The initial thought was that something was wrong in the cassandra-env.sh >> file. So, we specified the variables 'system_memory_in_mb' (3760) and the >> 'system_cpu_cores' (1) according to our nodes' specification. We also >> changed the 'MAX_HEAP_SIZE' to 2G and the 'HEAP_NEWSIZE' to 200M (we think >> the second is related to the Garbage Collection). Unfortunately, that did >> not solve the issue and the node we hit via thrift keeps on dying regularly. >> >> In case you find this useful, swap is off and unevictable memory seems to be >> very high on all 3 servers (2.3GB, we usually observe the amount of >> unevictable memory on other Linux servers of around 0-16KB) (We are not >> quite sure how the unevictable memory ties into Cassandra, its just >> something we observed while looking into the problem). The CPU is pretty >> much idle the entire time. The heap memory is clearly being reduced once in >> a while according to nodetool, but obviously grows over the limit as time >> goes by. >> >> Any ideas? Thanks in advance. >> >> Bill > >
Re: Trying to avoid super columns
If this is write once read many data you may get some benefit from packing all
the info for a product into one column, using something like JSON for the
column value.
>> The one thing that stands out to me with this approach is the number of
>> additonal columns that will be created for a single key. Will the increase
>> in columns, create new issues I will need to deal with?
Millions of columns in a row may be ok, depending on the types of queries you
want to run (some background
http://thelastpickle.com/2011/07/04/Cassandra-Query-Plans/)
The more important issue is the byte size of the row. Wide rows take longer to
compact and repair, and I try to avoid rows above a few 10's of MB. By default
rows larger than 64MB require slower compaction.
Compression in 1.X will help where you have lots of repeating column names.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 13/04/2012, at 7:32 AM, Dave Brosius wrote:
> If you want to reduce the number of columns, you could pack all the data for
> a product into one column, as in
>
>
> composite column name-> product_id_1:12.44:1.00:3.00
>
>
>
> On 04/12/2012 03:03 PM, Philip Shon wrote:
>> I am currently working on a data model where the purpose is to look up
>> multiple products for given days of the year. Right now, that model
>> involves the usage of a super column family. e.g.
>>
>> "2012-04-12": {
>> "product_id_1": {
>>price: 12.44,
>>tax: 1.00,
>>fees: 3.00,
>> },
>> "product_id_2": {
>>price: 50.00,
>>tax: 4.00,
>>fees: 10.00
>> }
>> }
>>
>> I should note that for a given day/key, we are expecting in the range of 2
>> million to 4 million products (subcolumns).
>>
>> With this model, I am able to retrieve any of the products for a given day
>> using hector's MultigetSuperSliceQuery.
>>
>>
>> I am looking into changing this model to use Composite column names. How
>> would I go about modeling this? My initial thought is to migrate the above
>> model into something more like the following.
>>
>> "2012-04-12": {
>> "product_id_1:price": 12.44,
>> "product_id_1:tax": 1.00,
>> "product_id_1:fees": 3.00,
>> "product_id_2:price": 50.00,
>> "product_id_2:tax": 4.00,
>> "product_id_2:fees": 10.00,
>> }
>>
>> The one thing that stands out to me with this approach is the number of
>> additonal columns that will be created for a single key. Will the increase
>> in columns, create new issues I will need to deal with?
>>
>> Are there any other thoughts about if I should actually move forward (or
>> not) with migration this super column family to the model with the component
>> column names?
>>
>> Thanks,
>>
>> Phil
>
Re: RMI/JMX errors, weird
Look at the server side logs for errors. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 13/04/2012, at 11:47 AM, Maxim Potekhin wrote: > Hello, > > I'm doing compactions under 0.8.8. > > Recently, I started seeing a stack trace like one below, and I can't figure > out what causes this to appear. > The cluster has been in operation for mode than half a year w/o errors like > this one. > > Any help will be appreciated, > Thanks > > Maxim > > > WARNING: Failed to check the connection: java.net.SocketTimeoutException: > Read timed out > Exception in thread "main" java.io.IOException: Repair command #1: some > repair session(s) failed (see log for details). >at > org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1613) >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >at java.lang.reflect.Method.invoke(Method.java:597) >at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) >at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) >at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) >at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) >at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) >at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) >at > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) >at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) >at > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) >at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) >at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) >at > javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >at java.lang.reflect.Method.invoke(Method.java:597) >at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303) >at sun.rmi.transport.Transport$1.run(Transport.java:159) >at java.security.AccessController.doPrivileged(Native Method) >at sun.rmi.transport.Transport.serviceCall(Transport.java:155) >at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) >at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) >at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) >at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >at java.lang.Thread.run(Thread.java:662) >
insert data in cassandra from .net..
I m able to connect cassandra and fetch rows from the cassandra database. Now i want to insert the data from .net on to cassandra but using insert query of cql is not working because i have fields in my table which has null values for the columns and cassandra would not take null values. So now i have to query it the other way.. i.e, i can use set [key] [column name] = column value; but can i send this command from .net to cassandra. I mean we have execute_cql_query() for interacting in a cql way, is there anything available for the other way? that is the commands we execute on cassandra-cli.bat ?
