Re: C 2.1
Hi Ram. 1) As an Operations DBA, I consider all versions of Cassandra to be alpha. So whether you pick 2.0.10 or 2.1.0 doesn't really matter since you will have to do your own acceptance testing. 2) Data modelling is everything when it comes to a distributed database like Cassandra. You can read my blog post which is a quick way to get up to speed with CQL: Notes on “Getting Started with Time Series Data Modeling” in Cassandra http://jbriggs.com/blog/2014/09/notes-on-getting-started-with-time-series-data-modeling-in-cassandra/ Thanks, James Briggs -- Cassandra/MySQL DBA. Available in San Jose area or remote. From: Ram N To: user@cassandra.apache.org Sent: Saturday, September 13, 2014 3:49 PM Subject: C 2.1 Team, I am pretty new to cassandra (with just 2 weeks of playing around with it on and off) and planning a fresh deployment with 2.1 release. The data-model is pretty simple for my use-case. Questions I have in mind are Is 2.1 a production ready release? Driver selection? I played around with Hector, Astyanax and Java driver? I don't see much activity happening on Hector, For Astyanax - Love the Fluent style of writing code and abstractions, recipes, pooling etc Datastax Java driver - I get too confused with CQL and the underlying storage model. I am also not clear on the indexing structure of columns. Does CQL indexes create a separate CF for the index table? How is it different from maintaining inverted index? Internally both are the same? Does cql stmt to create index, creates a separate CF and has an atomic way of updating/managing them? Which one is better to scale? (something like stargate-core or the ones done by usergrid? or the CQL approach?) On a separate note just curious if I have 1000's of columns in a given row and a fixed set of indexed column (say 30 - 50 columns) which approach should I be taking? Will cassandra scale with these many indexed column? Are there any limits? How much of an impact do CQL indexes create on the system? I am also not sure if these use cases are the right choice for cassandra but would really appreciate any response on these. Thanks. -R
Re: why bloom filter is only for row key?
Thanks DuyHai, I think the trouble of bloom filter on all row keys & column names is memory usage. However, if a CF has only hundreds of columns per row, the number of total columns will be much fewer, so the bloom filter is possible for this condition, right? Is there a good way to adjust bloom filter's property between row keys and row keys+column names automatically or by user's config? Thanks, Philo Yang 2014-09-15 2:45 GMT+08:00 DuyHai Doan : > Hello Philo > > Building bloom filter for column names (what you call column key) is > technically possible but very expensive in term of memory usage. > > The approximate formula to calculate space required by bloom filter can > be found on slide 27 here: > http://fr.slideshare.net/quipo/modern-algorithms-and-data-structures-1-bloom-filters-merkle-trees > > false positive chance = 0.6185 * m/n where m = number of bits for the > filter and n = number of distinct keys > > For example, if you want to index 1 million of rows, each having 100 000 > columns on average, it will end up indexing 100 billions of keys (row keys > & column names) with bloom filter. > > By applying the above formula, m ≈ 4.8 * 10^11 bits ≈ 60Gb to allocate in > RAM just for bloom filter on all row keys & column names ... > > Regards > > Duy Hai DOAN > > On Sun, Sep 14, 2014 at 11:22 AM, Philo Yang wrote: > >> Hi all, >> >> After reading some docs, I find that bloom filter is built on row keys, >> not on column key. Can anyone tell me what is considered for not building >> bloom filter on column key? Is it a good idea to offer a table property >> option between row key and primary key for what boolm filter is built on? >> >> Thanks, >> Philo Yang >> >> >
Re: why bloom filter is only for row key?
On Sun, Sep 14, 2014 at 11:22 AM, Philo Yang wrote: > After reading some docs, I find that bloom filter is built on row keys, > not on column key. Can anyone tell me what is considered for not building > bloom filter on column key? Is it a good idea to offer a table property > option between row key and primary key for what boolm filter is built on? > Here's the nitty gritty of the process which removed the row-level bloom filter on column keys. https://issues.apache.org/jira/browse/CASSANDRA-4885 - "Remove or rework per-row bloom filters" =Rob http://twitter.com/rcolidba
Re: C 2.1
On Sat, Sep 13, 2014 at 3:49 PM, Ram N wrote: > Is 2.1 a production ready release? > https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ > Datastax Java driver - I get too confused with CQL and the underlying > storage model. I am also not clear on the indexing structure of columns. > Does CQL indexes create a separate CF for the index table? How is it > different from maintaining inverted index? Internally both are the same? > Does cql stmt to create index, creates a separate CF and has an atomic way > of updating/managing them? Which one is better to scale? (something like > stargate-core or the ones done by usergrid? or the CQL approach?) > New projects should use CQL. Access to underlying storage via Thrift is likely to eventually be removed from Cassandra. > On a separate note just curious if I have 1000's of columns in a given row > and a fixed set of indexed column (say 30 - 50 columns) which approach > should I be taking? Will cassandra scale with these many indexed column? > Are there any limits? How much of an impact do CQL indexes create on the > system? I am also not sure if these use cases are the right choice for > cassandra but would really appreciate any response on these. Thanks. > Use of the "Secondary Indexes" feature is generally an anti-pattern in Cassandra. 30-50 indexed columns in a row sounds insane to me. However 30-50 column families into which one manually denormalized does not sound too insane to me... =Rob http://twitter.com/rcolidba
Re: Quickly loading C* dataset into memory (row cache)
On Sat, Sep 13, 2014 at 11:48 PM, Paulo Ricardo Motta Gomes < paulo.mo...@chaordicsystems.com> wrote: > Apparently Apple is using Cassandra as a massive multi-DC cache, as per > their announcement during the summit, but probably DSE with in-memory > enabled option. Would love to hear about similar use cases. > There's caches and there's caches. I submit that, thus far, the usage of the term "cache" in this conversation has not been specific enough to enhance understanding. I continue to assert, in a very limited scope, that 6GB of row cache in Cassandra on a system with 7GB of RAM is Doing It Wrong. :D =Rob
Re: why bloom filter is only for row key?
Nice catch Rob On Mon, Sep 15, 2014 at 8:04 PM, Robert Coli wrote: > On Sun, Sep 14, 2014 at 11:22 AM, Philo Yang wrote: > >> After reading some docs, I find that bloom filter is built on row keys, >> not on column key. Can anyone tell me what is considered for not building >> bloom filter on column key? Is it a good idea to offer a table property >> option between row key and primary key for what boolm filter is built on? >> > > Here's the nitty gritty of the process which removed the row-level bloom > filter on column keys. > > https://issues.apache.org/jira/browse/CASSANDRA-4885 - "Remove or rework > per-row bloom filters" > > =Rob > http://twitter.com/rcolidba >
Re: C 2.1
If you’re indexing and querying on that many columns (dozens, or more than a handful), consider DSE/Solr, especially if you need to query on multiple columns in the same query. -- Jack Krupansky From: Robert Coli Sent: Monday, September 15, 2014 11:07 AM To: user@cassandra.apache.org Subject: Re: C 2.1 On Sat, Sep 13, 2014 at 3:49 PM, Ram N wrote: Is 2.1 a production ready release? https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ Datastax Java driver - I get too confused with CQL and the underlying storage model. I am also not clear on the indexing structure of columns. Does CQL indexes create a separate CF for the index table? How is it different from maintaining inverted index? Internally both are the same? Does cql stmt to create index, creates a separate CF and has an atomic way of updating/managing them? Which one is better to scale? (something like stargate-core or the ones done by usergrid? or the CQL approach?) New projects should use CQL. Access to underlying storage via Thrift is likely to eventually be removed from Cassandra. On a separate note just curious if I have 1000's of columns in a given row and a fixed set of indexed column (say 30 - 50 columns) which approach should I be taking? Will cassandra scale with these many indexed column? Are there any limits? How much of an impact do CQL indexes create on the system? I am also not sure if these use cases are the right choice for cassandra but would really appreciate any response on these. Thanks. Use of the "Secondary Indexes" feature is generally an anti-pattern in Cassandra. 30-50 indexed columns in a row sounds insane to me. However 30-50 column families into which one manually denormalized does not sound too insane to me... =Rob http://twitter.com/rcolidba
Re: C 2.1
Jack, Using Solr or an external search/indexing service is an option but increases the complexity of managing different systems. I am curious to understand the impact of having wide-rows on a separate CF for inverted index purpose which if I understand correctly is what Rob's response, having a separate CF for index is better than using the default Secondary index option. Would be great to understand the design decision to go with present implementation on Secondary Index when the alternative is better? Looking at JIRAs is still confusing to come up with the why :) --R On Mon, Sep 15, 2014 at 11:17 AM, Jack Krupansky wrote: > If you’re indexing and querying on that many columns (dozens, or more > than a handful), consider DSE/Solr, especially if you need to query on > multiple columns in the same query. > > -- Jack Krupansky > > *From:* Robert Coli > *Sent:* Monday, September 15, 2014 11:07 AM > *To:* user@cassandra.apache.org > *Subject:* Re: C 2.1 > > On Sat, Sep 13, 2014 at 3:49 PM, Ram N wrote: > >> Is 2.1 a production ready release? >> > > https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ > > >> Datastax Java driver - I get too confused with CQL and the >> underlying storage model. I am also not clear on the indexing structure of >> columns. Does CQL indexes create a separate CF for the index table? How is >> it different from maintaining inverted index? Internally both are the same? >> Does cql stmt to create index, creates a separate CF and has an atomic way >> of updating/managing them? Which one is better to scale? (something like >> stargate-core or the ones done by usergrid? or the CQL approach?) >> > > New projects should use CQL. Access to underlying storage via Thrift is > likely to eventually be removed from Cassandra. > > >> On a separate note just curious if I have 1000's of columns in a given >> row and a fixed set of indexed column (say 30 - 50 columns) which approach >> should I be taking? Will cassandra scale with these many indexed column? >> Are there any limits? How much of an impact do CQL indexes create on the >> system? I am also not sure if these use cases are the right choice for >> cassandra but would really appreciate any response on these. Thanks. >> > > Use of the "Secondary Indexes" feature is generally an anti-pattern in > Cassandra. 30-50 indexed columns in a row sounds insane to me. However > 30-50 column families into which one manually denormalized does not sound > too insane to me... > > =Rob > http://twitter.com/rcolidba >
Issues during Multi-DC setup across AWS regions + VPC setup
We are trying to add new data center in us-east. Servers in each DC are running inside VPC. We currently have a cluster in us-west and all servers are running 2.0.7. The two DCs are talking via VPN. listen_address and broadcast_address have private ip. Our endpoint_snitch is GossipingPropertyFileSnitch. All the required ports are open across VPC and we could telnet from one DC to the other. We see this exception when we add a new node in the new dc(us-east) INFO [HANDSHAKE-/10.0.51.81] 2014-09-15 16:38:26,896 OutboundTcpConnection.java (line 386) Handshaking version with /10.0.51.81 DEBUG [WRITE-/10.0.52.81] 2014-09-15 16:38:29,038 OutboundTcpConnection.java (line 333) Target max version is -2147483648; no version information yet, will retry TRACE [HANDSHAKE-/10.0.52.81] 2014-09-15 16:38:29,038 OutboundTcpConnection.java (line 393) Cannot handshake version with / 10.0.52.81 java.nio.channels.AsynchronousCloseException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412) at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:203) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.io.InputStream.read(InputStream.java:101) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.cassandra.net.OutboundTcpConnection$1.run(OutboundTcpConnection.java:387) What are we missing here? Thanks Dinesh
Cassandra, vnodes, and spark
Hello. http://stackoverflow.com/questions/19969329/why-not-enable-virtual-node-in-an-hadoop-node/19974621#19974621 Based on this stackoverflow question, vnodes effect the number of mappers Hadoop needs to spawn. Which in then affect performance. With the spark connector for cassandra would the same situation happen? Would vnodes affect performance in a similar situation to Hadoop?
Re: Cassandra, vnodes, and spark
Sorry. Trigger finger on the send. Would vnodes affect performance for spark in a similar fashion for spark. On Monday, September 15, 2014, Eric Plowe wrote: > Hello. > > > http://stackoverflow.com/questions/19969329/why-not-enable-virtual-node-in-an-hadoop-node/19974621#19974621 > > Based on this stackoverflow question, vnodes effect the number of mappers > Hadoop needs to spawn. Which in then affect performance. > > With the spark connector for cassandra would the same situation happen? > Would vnodes affect performance in a similar situation to Hadoop? >
Re: C 2.1
Ram,
The reason secondary indexes are not recommended is that since
they can't use the partition key, the values have to be fetched from
all nodes. So you have higher latency, and likely timeouts.
The C* solutions are:
a) use a denormalized ("materialized") table
b) use a clustered index if all the data related to the row key is
in the same partition (read my blog link from this thread for more)
That's the price of using distributed systems.
Oh, and then there's the need to rewrite the data access layer
of your entire existing app. :)
AOL and Blizzard talked about porting a couple apps to Cassandra
at the conference last week, but they sounded like trivial user-db
("UDB") apps, and even then Patrick was usually credited with the
data modelling.
I haven't heard of anybody porting a 100+ table Oracle or MySQL
app to C* yet. I'm sure it's been done, but most of the
apps written for C* are greenfield or v2.0 rewrites.
Thanks, James Briggs
--
Cassandra/MySQL DBA. Available in San Jose area or remote.
From: Ram N
To: user
Sent: Monday, September 15, 2014 1:34 PM
Subject: Re: C 2.1
Jack,
Using Solr or an external search/indexing service is an option but increases
the complexity of managing different systems. I am curious to understand the
impact of having wide-rows on a separate CF for inverted index purpose which if
I understand correctly is what Rob's response, having a separate CF for index
is better than using the default Secondary index option.
Would be great to understand the design decision to go with present
implementation on Secondary Index when the alternative is better? Looking at
JIRAs is still confusing to come up with the why :)
--R
On Mon, Sep 15, 2014 at 11:17 AM, Jack Krupansky
wrote:
If you’re indexing and querying on that many columns (dozens, or more than
a handful), consider DSE/Solr, especially if you need to query on multiple
columns in the same query.
>
>-- Jack
Krupansky
>
>From: Robert Coli
>Sent: Monday, September 15, 2014 11:07 AM
>To: user@cassandra.apache.org
>Subject: Re: C 2.1
>
>On Sat, Sep 13, 2014 at 3:49 PM, Ram N wrote:
>
>Is 2.1 a production ready release?
>
>https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
>
>
> Datastax Java driver - I get too confused with CQL and the underlying
> storage model. I am also not clear on the indexing structure of columns.
> Does CQL indexes create a separate CF for the index table? How is it
> different from maintaining inverted index? Internally both are the same?
> Does cql stmt to create index, creates a separate CF and has an atomic way
> of updating/managing them? Which one is better to scale? (something like
> stargate-core or the ones done by usergrid? or the CQL approach?)
>
>New projects should use CQL. Access to underlying storage via Thrift is
likely to eventually be removed from Cassandra.
>
>On a separate note just curious if I have 1000's of columns in a given row
>and a fixed set of indexed column (say 30 - 50 columns) which approach
>should I be taking? Will cassandra scale with these many indexed column? Are
>there any limits? How much of an impact do CQL indexes create on the system?
>I am also not sure if these use cases are the right choice for cassandra but
>would really appreciate any response on these. Thanks.
>
>Use of the "Secondary Indexes" feature is generally an anti-pattern in
Cassandra. 30-50 indexed columns in a row sounds insane to me. However 30-50
column families into which one manually denormalized does not sound too insane
to me...
>
>=Rob
>http://twitter.com/rcolidba
Re: Cassandra, vnodes, and spark
As hadoop* again sorry.. On Monday, September 15, 2014, Eric Plowe wrote: > Sorry. Trigger finger on the send. > > Would vnodes affect performance for spark in a similar fashion for spark. > > On Monday, September 15, 2014, Eric Plowe > wrote: > >> Hello. >> >> >> http://stackoverflow.com/questions/19969329/why-not-enable-virtual-node-in-an-hadoop-node/19974621#19974621 >> >> Based on this stackoverflow question, vnodes effect the number of mappers >> Hadoop needs to spawn. Which in then affect performance. >> >> With the spark connector for cassandra would the same situation happen? >> Would vnodes affect performance in a similar situation to Hadoop? >> >
Re: 2.0.10 debian/ubuntu apt release?
On 09/12/2014 04:34 PM, Michael Shuler wrote: I'll have 2.0.10 deb/rpm packages in the repos on Monday, barring any issues. Just a quick update - I had a few issues with the Windows 2.0.10 release, which finally succeeded a few minutes ago, so I'll push to the repositories tomorrow, so we can update versions in documentation, etc. -- Michael
hs_err_pid3013.log, out of memory?
Hi there, I just encountered an error which left a log '/hs_err_pid3013.log'. So is there a way to solve this? # > # There is insufficient memory for the Java Runtime Environment to > continue. > # Native memory allocation (malloc) failed to allocate 12288 bytes for > committing reserved memory. > # Possible reasons: > # The system is out of physical RAM or swap space > # In 32 bit mode, the process size limit was hit > # Possible solutions: > # Reduce memory load on the system > # Increase physical memory or swap space > # Check if swap backing store is full > # Use 64 bit Java on a 64 bit OS > # Decrease Java heap size (-Xmx/-Xms) > # Decrease number of Java threads > # Decrease Java thread stack sizes (-Xss) > # Set larger code cache with -XX:ReservedCodeCacheSize= > # This output file may be truncated or incomplete. > # > # Out of Memory Error (os_linux.cpp:2718), pid=3013, tid=140294995289856 > # > # JRE version: Java(TM) SE Runtime Environment (7.0_45-b18) (build > 1.7.0_45-b18) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.45-b08 mixed mode > linux-amd64 compressed oops) > # Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > > --- T H R E A D --- > > Current thread (0x7f994ace8000): JavaThread "HANDSHAKE-/10.0.10.9" > daemon [_thread_new, id=3274, stack(0x7f98f957a000,0x7f98f95bb000)] > > Stack: [0x7f98f957a000,0x7f98f95bb000], sp=0x7f98f95b9800, > free space=254k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > V [libjvm.so+0x992c8a] VMError::report_and_die()+0x2ea > V [libjvm.so+0x49319b] report_vm_out_of_memory(char const*, int, > unsigned long, char const*)+0x9b > V [libjvm.so+0x81310e] os::Linux::commit_memory_impl(char*, unsigned > long, bool)+0xfe > V [libjvm.so+0x8131cc] os::pd_commit_memory(char*, unsigned long, > bool)+0xc > V [libjvm.so+0x80b3ba] os::commit_memory(char*, unsigned long, bool)+0x2a > V [libjvm.so+0x8116ed] os::pd_create_stack_guard_pages(char*, unsigned > long)+0x6d > V [libjvm.so+0x94693e] JavaThread::create_stack_guard_pages()+0x5e > V [libjvm.so+0x94d3d4] JavaThread::run()+0x34 > V [libjvm.so+0x815288] java_start(Thread*)+0x108 > > > --- P R O C E S S --- >
Re: hs_err_pid3013.log, out of memory?
On Mon, Sep 15, 2014 at 5:55 PM, Yatong Zhang wrote: > I just encountered an error which left a log '/hs_err_pid3013.log'. So is > there a way to solve this? > > # There is insufficient memory for the Java Runtime Environment to >> continue. >> # Native memory allocation (malloc) failed to allocate 12288 bytes for >> committing reserved memory. >> > Use less heap memory? You haven't specified under which circumstances this occurred, so I can only conjecture that it is likely being caused by "writing too fast". Write more slowly. =Rob
Re: Cassandra, vnodes, and spark
On Mon, Sep 15, 2014 at 4:57 PM, Eric Plowe wrote: > Based on this stackoverflow question, vnodes effect the number of mappers > Hadoop needs to spawn. Which in then affect performance. > > With the spark connector for cassandra would the same situation happen? > Would vnodes affect performance in a similar situation to Hadoop? > I don't know what specifically Spark does here, but if it has the same locality expectations as Hadoop generally, my belief would be : "yes." =Rob
Re: C 2.1
On Mon, Sep 15, 2014 at 1:34 PM, Ram N wrote: > Would be great to understand the design decision to go with present > implementation on Secondary Index when the alternative is better? Looking > at JIRAs is still confusing to come up with the why :) > http://mail-archives.apache.org/mod_mbox/incubator-cassandra-user/201405.mbox/%3CCAEDUwd1i2BwJ-PAFE1qhjQFZ=qz2va_vxwo_jdycms8evkb...@mail.gmail.com%3E (I really should formalize this guy into a blog post, recursive mailing list links are kinda lol...) =Rob
Re: hs_err_pid3013.log, out of memory?
It's during the startup. I tried to upgrade cassandra from 2.0.7 to 2.0.10, but looks like cassandra could not start again. Also I found the following log at '/var/log/messages': Sep 16 09:06:59 storage6 kernel: INFO: task java:4971 blocked for more than > 120 seconds. > Sep 16 09:06:59 storage6 kernel: Tainted: G --- > H 2.6.32-431.el6.x86_64 #1 > Sep 16 09:06:59 storage6 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Sep 16 09:06:59 storage6 kernel: java D 0003 0 > 4971 1 0x0080 > Sep 16 09:06:59 storage6 kernel: 88042b591c98 0082 > 81ed4ff0 8803b4f01540 > Sep 16 09:06:59 storage6 kernel: 88042b591c68 810af370 > 88042b591ca0 8803b4f01540 > Sep 16 09:06:59 storage6 kernel: 8803b4f01af8 88042b591fd8 > fbc8 8803b4f01af8 > Sep 16 09:06:59 storage6 kernel: Call Trace: > Sep 16 09:06:59 storage6 kernel: [] ? > exit_robust_list+0x90/0x160 > Sep 16 09:06:59 storage6 kernel: [] exit_mm+0x95/0x180 > Sep 16 09:06:59 storage6 kernel: [] do_exit+0x15f/0x870 > Sep 16 09:06:59 storage6 kernel: [] > do_group_exit+0x58/0xd0 > Sep 16 09:06:59 storage6 kernel: [] > get_signal_to_deliver+0x1f6/0x460 > Sep 16 09:06:59 storage6 kernel: [] do_signal+0x75/0x800 > Sep 16 09:06:59 storage6 kernel: [] ? > wake_up_new_task+0xd9/0x130 > Sep 16 09:06:59 storage6 kernel: [] ? do_fork+0x13d/0x480 > Sep 16 09:06:59 storage6 kernel: [] ? > sys_futex+0x7b/0x170 > Sep 16 09:06:59 storage6 kernel: [] > do_notify_resume+0x90/0xc0 > Sep 16 09:06:59 storage6 kernel: [] int_signal+0x12/0x17 > Sep 16 09:06:59 storage6 kernel: INFO: task java:4972 blocked for more > than 120 seconds. > Sep 16 09:06:59 storage6 kernel: Tainted: G --- > H 2.6.32-431.el6.x86_64 #1 > Sep 16 09:06:59 storage6 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Sep 16 09:06:59 storage6 kernel: java D 0 > 4972 1 0x0080 > Sep 16 09:06:59 storage6 kernel: 8803b4d7fc98 0082 > 81ed6d78 8803b4cf1500 > Sep 16 09:06:59 storage6 kernel: 8803b4d7fc68 810af370 > 8803b4d7fca0 8803b4cf1500 > Sep 16 09:06:59 storage6 kernel: 8803b4cf1ab8 8803b4d7ffd8 > fbc8 8803b4cf1ab8 > Sep 16 09:06:59 storage6 kernel: Call Trace: > Sep 16 09:06:59 storage6 kernel: [] ? > exit_robust_list+0x90/0x160 > Sep 16 09:06:59 storage6 kernel: [] exit_mm+0x95/0x180 > Sep 16 09:06:59 storage6 kernel: [] do_exit+0x15f/0x870 > Sep 16 09:06:59 storage6 kernel: [] ? > wake_up_state+0x10/0x20 > Sep 16 09:06:59 storage6 kernel: [] > do_group_exit+0x58/0xd0 > Sep 16 09:06:59 storage6 kernel: [] > get_signal_to_deliver+0x1f6/0x460 > Sep 16 09:06:59 storage6 kernel: [] do_signal+0x75/0x800 > Sep 16 09:06:59 storage6 kernel: [] ? > __switch_to+0x1ac/0x320 > Sep 16 09:06:59 storage6 kernel: [] ? > thread_return+0x4e/0x76e > Sep 16 09:06:59 storage6 kernel: [] ? > sys_futex+0x7b/0x170 > Sep 16 09:06:59 storage6 kernel: [] > do_notify_resume+0x90/0xc0 > Sep 16 09:06:59 storage6 kernel: [] int_signal+0x12/0x17 > Sep 16 09:06:59 storage6 kernel: INFO: task java:4973 blocked for more > than 120 seconds. > On Tue, Sep 16, 2014 at 9:00 AM, Robert Coli wrote: > On Mon, Sep 15, 2014 at 5:55 PM, Yatong Zhang wrote: > >> I just encountered an error which left a log '/hs_err_pid3013.log'. So is >> there a way to solve this? >> >> # There is insufficient memory for the Java Runtime Environment to >>> continue. >>> # Native memory allocation (malloc) failed to allocate 12288 bytes for >>> committing reserved memory. >>> >> > Use less heap memory? > > You haven't specified under which circumstances this occurred, so I can > only conjecture that it is likely being caused by "writing too fast". > > Write more slowly. > > =Rob > >
Re: Cassandra, vnodes, and spark
Interesting. The way I understand the spark connector is that it's basically a client executing a cql query and filling a spark rdd. Spark will then handle the partitioning of data. Again, this is my understanding, and it maybe incorrect. On Monday, September 15, 2014, Robert Coli wrote: > On Mon, Sep 15, 2014 at 4:57 PM, Eric Plowe > wrote: > >> Based on this stackoverflow question, vnodes effect the number of mappers >> Hadoop needs to spawn. Which in then affect performance. >> >> With the spark connector for cassandra would the same situation happen? >> Would vnodes affect performance in a similar situation to Hadoop? >> > > I don't know what specifically Spark does here, but if it has the same > locality expectations as Hadoop generally, my belief would be : "yes." > > =Rob > >
Trying to understand cassandra gc logs
I understand that cassandra uses ParNew GC for New Gen and CMS for Old Gen
(tenured). I'm trying to interpret in the logs when a Full GC happens and
what kind of Full GC is used. It never says "Full GC" or anything like that.
But I see that whenever there's a line like
2014-09-15T18:04:17.197-0700: 117485.192: [CMS-concurrent-mark-start]
the count of full GCs increases from
{Heap after GC invocations=158459 (full 931):
to a line like:
{Heap before GC invocations=158459 (full 932):
See the highlighted lines in the gclog output below. So, apparently there was
a full GC between those two lines. Between those lines it also has two lines,
such as:
2014-09-15T18:04:17.197-0700: 117485.192: Total time for which application
threads were stopped: 0.0362080 seconds
2014-09-15T18:04:17.882-0700: 117485.877: Total time for which application
threads were stopped: 0.0129660 seconds
Also, the full count (932 above) is always exactly half the number (1864) FGC
returned by jstat, as in
dc1-cassandra01.dc01 /var/log/cassandra> sudo jstat -gcutil 28511
S0 S1 E O P YGC YGCTFGCFGCT GCT
55.82 0.00 82.45 45.02 59.76 165772 5129.728 1864 320.247 5449.975
So, I am apparently correct that "(full 932)" is the count of Full GCs. I'm
perplexed by the log output, though.
I also see lines mentioning "concurrent mark-sweep" that do not appear to
correspond to full GCs. So, my questions are: Is CMS used also for full GCs?
If not, what kind of gc is done? The logs don't say.Lines saying "Total
time for which application threads were stopped" appear twice per full gc; why?
Apparently, even our Full GCs are fast. 99% of them finish within 0.18
seconds; 99.9% finish within 0.5 seconds (which may be too slow for some of our
clients).
Here below is some log output, with interesting parts highlighted in grey or
yellow. Thanks, Don
{Heap before GC invocations=158458 (full 931):
par new generation total 1290240K, used 1213281K [0x0005bae0,
0x00061260, 0x00061260)
eden space 1146880K, 100% used [0x0005bae0, 0x000600e0,
0x000600e0)
from space 143360K, 46% used [0x000600e0, 0x000604ed87c0,
0x000609a0)
to space 143360K, 0% used [0x000609a0, 0x000609a0,
0x00061260)
concurrent mark-sweep generation total 8003584K, used 5983572K
[0x00061260, 0x0007fae0, 0x0007fae0)
concurrent-mark-sweep perm gen total 44820K, used 26890K [0x0007fae0,
0x0007fd9c5000, 0x0008)
2014-09-15T18:04:17.131-0700: 117485.127: [GCBefore GC:
Statistics for BinaryTreeDictionary:
Total Free Space: 197474318
Max Chunk Size: 160662270
Number of Blocks: 3095
Av. Block Size: 63804
Tree Height: 32
Before GC:
Statistics for BinaryTreeDictionary:
Total Free Space: 2285026
Max Chunk Size: 2279936
Number of Blocks: 8
Av. Block Size: 285628
Tree Height: 5
2014-09-15T18:04:17.133-0700: 117485.128: [ParNew
Desired survivor size 73400320 bytes, new threshold 1 (max 1)
- age 1: 44548776 bytes, 44548776 total
: 1213281K->49867K(1290240K), 0.0264540 secs] 7196854K->6059170K(9293824K)After
GC:
Statistics for BinaryTreeDictionary:
Total Free Space: 195160244
Max Chunk Size: 160662270
Number of Blocks: 3093
Av. Block Size: 63097
Tree Height: 32
After GC:
Statistics for BinaryTreeDictionary:
Total Free Space: 2285026
Max Chunk Size: 2279936
Number of Blocks: 8
Av. Block Size: 285628
Tree Height: 5
, 0.0286700 secs] [Times: user=0.37 sys=0.01, real=0.03 secs]
Heap after GC invocations=158459 (full 931):
par new generation total 1290240K, used 49867K [0x0005bae0,
0x00061260, 0x00061260)
eden space 1146880K, 0% used [0x0005bae0, 0x0005bae0,
0x000600e0)
from space 143360K, 34% used [0x000609a0, 0x00060cab2e18,
0x00061260)
to space 143360K, 0% used [0x000600e0, 0x000600e0,
0x000609a0)
concurrent mark-sweep generation total 8003584K, used 6009302K
[0x00061260, 0x0007fae0, 0x0007fae0)
concurrent-mark-sweep perm gen total 44820K, used 26890K [0x0007fae0,
0x0007fd9c5000, 0x0008)
}
2014-09-15T18:04:17.161-0700: 117485.156: Total time for which application
threads were stopped: 0.0421350 seconds
2014-09-15T18:04:17.173-0700: 117485.168: [GC [1 CMS-initial-mark:
6009302K(8003584K)] 6059194K(9293824K), 0.0231840 secs] [Times: user=0.03
sys=0.00, real=0.03 secs]
2014-09-15T18:04:17.197-0700: 117485.192: Total time for which application
threads were stopped: 0.0362080 seconds
2014-09-15T18:04:17.197-0700: 117485.192: [CMS-concurrent-mark-start]
2014-09-15T18:04:17.681-0700: 117485.677: [CMS-concurrent-mark: 0.484/0.484
Re: Cassandra, vnodes, and spark
Look into the source code of the Spark connector. CassandraRDD try to find all token ranges (even when using vnodes) for each node (endpoint) and create RDD partition to match this distribution of token ranges. Thus data locality is guaranteed On Tue, Sep 16, 2014 at 4:39 AM, Eric Plowe wrote: > Interesting. The way I understand the spark connector is that it's > basically a client executing a cql query and filling a spark rdd. Spark > will then handle the partitioning of data. Again, this is my understanding, > and it maybe incorrect. > > > On Monday, September 15, 2014, Robert Coli wrote: > >> On Mon, Sep 15, 2014 at 4:57 PM, Eric Plowe wrote: >> >>> Based on this stackoverflow question, vnodes effect the number of >>> mappers Hadoop needs to spawn. Which in then affect performance. >>> >>> With the spark connector for cassandra would the same situation happen? >>> Would vnodes affect performance in a similar situation to Hadoop? >>> >> >> I don't know what specifically Spark does here, but if it has the same >> locality expectations as Hadoop generally, my belief would be : "yes." >> >> =Rob >> >>
