Re: Query With Limit Clause

[shalom sagges]

Thanks a lot for the info :)

On Tue, Nov 6, 2018 at 11:11 AM DuyHai Doan  wrote:

> Cassandra will execute such request using a Partition Range Scan.
>
> See more details here http://www.doanduyhai.com/blog/?p=13191, chapter E
> Cluster Read Path (look at the formula of Concurrency Factor)
>
>
>
> On Tue, Nov 6, 2018 at 8:21 AM shalom sagges 
> wrote:
>
>> Hi All,
>>
>> If I run for example:
>> select * from myTable limit 3;
>>
>> Does Cassandra do a full table scan regardless of the limit?
>>
>> Thanks!
>>
>

Cassandra 2.1 bootstrap - No streaming progress from one node

[Steinmaurer, Thomas]

Hello,

while bootstrapping a new node into an existing cluster, a node which is acting 
as source for streaming got restarted unfortunately. Since then, from nodetool 
netstats I don't see any progress for this particular node anymore.

E.g.:

/X.X.X.X
Receiving 94 files, 260.09 GB total. Already received 26 files, 69.33 
GB total

Basically, it is stuck at 69.33GB for hours. Is Cassandra (2.1 in our case) not 
doing any resume here, in case there have been e.g. connectivity troubles or in 
our case, Cassandra on the node acting as stream source got restarted?

Can I force the joining node to recover connection to X.X.X.X or do I need to 
restart the bootstrap via restart on the new node from scratch?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freist?dterstra?e 313

Cassandra crashed during major compaction

[Gabriel Giussi]

After a bulk load of writes to existing partition keys (with a higher
timestamp), I wanted to free disk space, suspecting that rows will be in
the highest levels and it would take a time until they were compacted.
I've started a major compaction, and the disk usage went from ~30% to ~40%
(as expected) but after ~10 hs Cassandra has crashed (*) and disk usage
continues being ~40%, even after restart.

How could I remove SSTables created during the failed compaction?

(*) From the logs, I understand that the JVM runs out of java heap space.
However, I don't understand why there are multiple OutOfMemory errors, it
shouldn't crashed with the first one?

ERROR [MessagingService-Incoming-/x.x.x.x] 2018-11-06 22:28:35,251
CassandraDaemon.java:207 - Exception in thread
Thread[MessagingService-Incoming-/x.x.x.x,5,main]
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:403)
~[apache-cassandra-3.0.13.jar:3.0.13]
at 
org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:341)
~[apache-cassandra-3.0.13.jar:3.0.13]
at 
org.apache.cassandra.db.ReadResponse$Serializer.deserialize(ReadResponse.java:382)
~[apache-cassandra-3.0.13.jar:3.0.13]
--
at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
~[apache-cassandra-3.0.13.jar:3.0.13]
INFO  [HintsDispatcher:63] 2018-11-06 22:34:49,678
HintsDispatchExecutor.java:271 - Finished hinted handoff of file
9584b93c-f86e-464f-a9ba-3dd33134b7af-1541550621736-1.hints to endpoint
/y.y.y.y: 9584b93c-f86e-464f-a9ba-3dd33134b7af, partially
ERROR [MessagingService-Incoming-/z.z.z.z] 2018-11-06 22:37:11,099
CassandraDaemon.java:207 - Exception in thread
Thread[MessagingService-Incoming-/z.z.z.z,5,main]
java.lang.OutOfMemoryError: Java heap space
INFO  [ScheduledTasks:1] 2018-11-06 22:37:34,716 StatusLogger.java:56
- Sampler   0 0  0
0 0

ERROR [MessagingService-Incoming-/y.y.y.y] 2018-11-06 22:39:39,860
CassandraDaemon.java:207 - Exception in thread
Thread[MessagingService-Incoming-/y.y.y.y,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/z.z.z.z] 2018-11-06 22:41:52,690
CassandraDaemon.java:207 - Exception in thread
Thread[MessagingService-Incoming-/z.z.z.z,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/a.a.a.a] 2018-11-06 22:42:23,498
CassandraDaemon.java:207 - Exception in thread
Thread[MessagingService-Incoming-/a.a.a.a,5,main]
java.lang.OutOfMemoryError: Java heap space
.

ERROR [MessagingService-Incoming-/x.x.x.x] 2018-11-06 23:42:53,947
JVMStabilityInspector.java:140 - JVM state determined to be unstable.
Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:403)
~[apache-cassandra-3.0.13.jar:3.0.13]
at 
org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:341)
~[apache-cassandra-3.0.13.jar:3.0.13]
at 
org.apache.cassandra.db.ReadResponse$Serializer.deserialize(ReadResponse.java:382)
~[apache-cassandra-3.0.13.jar:3.0.13]
--
at 
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
~[apache-cassandra-3.0.13.jar:3.0.13]
at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
~[apache-cassandra-3.0.13.jar:3.0.13]
ERROR [SharedPool-Worker-6] 2018-11-06 23:42:53,947
JVMStabilityInspector.java:140 - JVM state determined to be unstable.
Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR [SharedPool-Worker-12] 2018-11-06 23:42:53,947
JVMStabilityInspector.java:140 - JVM state determined to be unstable.
Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR [SharedPool-Worker-18] 2018-11-06 23:42:53,948
JVMStabilityInspector.java:140 - JVM state determined to be unstable.
Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space


Thanks.

RE: Cassandra 2.1 bootstrap - No streaming progress from one node

[Durity, Sean R]

I would wipe the new node and bootstrap again. I do not know of any way to 
resume the streaming that was previously in progress.


Sean Durity
From: Steinmaurer, Thomas 
Sent: Wednesday, November 07, 2018 5:13 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Cassandra 2.1 bootstrap - No streaming progress from one 
node

Hello,

while bootstrapping a new node into an existing cluster, a node which is acting 
as source for streaming got restarted unfortunately. Since then, from nodetool 
netstats I don't see any progress for this particular node anymore.

E.g.:

/X.X.X.X
Receiving 94 files, 260.09 GB total. Already received 26 files, 69.33 
GB total

Basically, it is stuck at 69.33GB for hours. Is Cassandra (2.1 in our case) not 
doing any resume here, in case there have been e.g. connectivity troubles or in 
our case, Cassandra on the node acting as stream source got restarted?

Can I force the joining node to recover connection to X.X.X.X or do I need to 
restart the bootstrap via restart on the new node from scratch?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: Multiple cluster for a single application

[Eric Stevens]

We are engaging in both strategies at the same time:

1) We call it functional sharding - we write to clusters targeted according
to the type of data being written.  Because different data types often have
different workloads this has the nice side effect of being able to tune
each cluster according to its workload.  Your ability to grow in this
dimension is limited by the number of business object types you're
recording.

2) We write to clusters sharded by time.  Our objects are network security
events, so there's always an element of time.  We encode that time into
deterministic object IDs so that we are able to identify in the read path
which shard to direct the request to by extracting the time component.
This basic idea should be able to work any time you're able to use
surrogate keys instead of natural keys.  If you are using natural keys, you
may be facing an unpleasant migration should you need to increase the
number of shards in this dimension.

Our reason for engaging in the second strategy was not purely Cassandra's
fault, rather we were using DSE with a search workload, and the cost of
rebuilding Solr indexes on streaming operations (such as adding nodes to an
existing cluster) required enough resources that we found it prohibitive.
That's because the bootstrapping node was also taking a production write
workload, and we didn't want to run our cluster with enough overhead that a
node could bootstrap and take production workload at the same time.

For vanilla Cassandra workloads we have run clusters with quite a bit more
nodes than 100 without any appreciable trouble.  Curious if you can share
documents about clusters over 100 nodes causing troubles for users.  I'm
wondering if it's related to node failure rate combined with vnodes meaning
that several concurrent node failures cause a part of the ring to go
offline too reliably.

On Mon, Nov 5, 2018 at 7:38 AM onmstester onmstester
 wrote:

> Hi,
>
> One of my applications requires to create a cluster with more than 100
> nodes, I've read documents recommended to use clusters with less than 50 or
> 100 nodes (Netflix got hundreds of clusters with less 100 nodes on each).
> Is it a good idea to use multiple clusters for a single application, just
> to decrease maintenance problems and system complexity/performance?
> If So, which one of below policies is more suitable to distribute data
> among clusters and Why?
> 1. each cluster' would be responsible for a specific partial set of tables
> only (table sizes are almost equal so easy calculations here) for example
> inserts to table X would go to cluster Y
> 2. shard data at loader level by some business logic grouping of data, for
> example all rows with some column starting with X would go to cluster Y
>
> I would appreciate sharing your experiences working with big clusters,
> problem encountered and solutions.
>
> Thanks in Advance
>
> Sent using Zoho Mail 
>
>
>

Re: Multiple cluster for a single application

[Jonathan Haddad]

Interesting approach Eric, thanks for sharing that.

Regarding this:

> I've read documents recommended to use clusters with less than 50 or 100
nodes (Netflix got hundreds of clusters with less 100 nodes on each).

Not sure where you read that, but it's nonsense.  We work with quite a few
clusters that are several hundred nodes each.  Your problems can get a bit
amplified, for instance dynamic snitch can make a cluster perform
significantly worse than if you just flat out disable it, which is what I
usually recommend.

I'm curious how you arrived at the estimate of needing > 100 nodes.  Is
that due to space constraints or performance ones?



On Wed, Nov 7, 2018 at 12:52 PM Eric Stevens  wrote:

> We are engaging in both strategies at the same time:
>
> 1) We call it functional sharding - we write to clusters targeted
> according to the type of data being written.  Because different data types
> often have different workloads this has the nice side effect of being able
> to tune each cluster according to its workload.  Your ability to grow in
> this dimension is limited by the number of business object types you're
> recording.
>
> 2) We write to clusters sharded by time.  Our objects are network security
> events, so there's always an element of time.  We encode that time into
> deterministic object IDs so that we are able to identify in the read path
> which shard to direct the request to by extracting the time component.
> This basic idea should be able to work any time you're able to use
> surrogate keys instead of natural keys.  If you are using natural keys, you
> may be facing an unpleasant migration should you need to increase the
> number of shards in this dimension.
>
> Our reason for engaging in the second strategy was not purely Cassandra's
> fault, rather we were using DSE with a search workload, and the cost of
> rebuilding Solr indexes on streaming operations (such as adding nodes to an
> existing cluster) required enough resources that we found it prohibitive.
> That's because the bootstrapping node was also taking a production write
> workload, and we didn't want to run our cluster with enough overhead that a
> node could bootstrap and take production workload at the same time.
>
> For vanilla Cassandra workloads we have run clusters with quite a bit more
> nodes than 100 without any appreciable trouble.  Curious if you can share
> documents about clusters over 100 nodes causing troubles for users.  I'm
> wondering if it's related to node failure rate combined with vnodes meaning
> that several concurrent node failures cause a part of the ring to go
> offline too reliably.
>
> On Mon, Nov 5, 2018 at 7:38 AM onmstester onmstester
>  wrote:
>
>> Hi,
>>
>> One of my applications requires to create a cluster with more than 100
>> nodes, I've read documents recommended to use clusters with less than 50 or
>> 100 nodes (Netflix got hundreds of clusters with less 100 nodes on each).
>> Is it a good idea to use multiple clusters for a single application, just
>> to decrease maintenance problems and system complexity/performance?
>> If So, which one of below policies is more suitable to distribute data
>> among clusters and Why?
>> 1. each cluster' would be responsible for a specific partial set of
>> tables only (table sizes are almost equal so easy calculations here) for
>> example inserts to table X would go to cluster Y
>> 2. shard data at loader level by some business logic grouping of data,
>> for example all rows with some column starting with X would go to cluster Y
>>
>> I would appreciate sharing your experiences working with big clusters,
>> problem encountered and solutions.
>>
>> Thanks in Advance
>>
>> Sent using Zoho Mail 
>>
>>
>>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Multiple cluster for a single application

[Ben Slater]

I tend to recommend an approach similar to Eric’s functional sharding
although I describe it at quality of service sharding - group your small,
hot data into one cluster and your large, cooler data into another so you
can provision infrastructure and tune according. I guess it depends on you
management environment but if you app functionality allows your to split
into multiple clusters (ie all your data is not all in one giant table)
then I would generally look to split. Splitting also gives you the
advantage of making it harder to have an outage that brings everything down.

Cheers
Ben

On Thu, 8 Nov 2018 at 08:44 Jonathan Haddad  wrote:

> Interesting approach Eric, thanks for sharing that.
>
> Regarding this:
>
> > I've read documents recommended to use clusters with less than 50 or 100
> nodes (Netflix got hundreds of clusters with less 100 nodes on each).
>
> Not sure where you read that, but it's nonsense.  We work with quite a few
> clusters that are several hundred nodes each.  Your problems can get a bit
> amplified, for instance dynamic snitch can make a cluster perform
> significantly worse than if you just flat out disable it, which is what I
> usually recommend.
>
> I'm curious how you arrived at the estimate of needing > 100 nodes.  Is
> that due to space constraints or performance ones?
>
>
>
> On Wed, Nov 7, 2018 at 12:52 PM Eric Stevens  wrote:
>
>> We are engaging in both strategies at the same time:
>>
>> 1) We call it functional sharding - we write to clusters targeted
>> according to the type of data being written.  Because different data types
>> often have different workloads this has the nice side effect of being able
>> to tune each cluster according to its workload.  Your ability to grow in
>> this dimension is limited by the number of business object types you're
>> recording.
>>
>> 2) We write to clusters sharded by time.  Our objects are network
>> security events, so there's always an element of time.  We encode that time
>> into deterministic object IDs so that we are able to identify in the read
>> path which shard to direct the request to by extracting the time
>> component.  This basic idea should be able to work any time you're able to
>> use surrogate keys instead of natural keys.  If you are using natural keys,
>> you may be facing an unpleasant migration should you need to increase the
>> number of shards in this dimension.
>>
>> Our reason for engaging in the second strategy was not purely Cassandra's
>> fault, rather we were using DSE with a search workload, and the cost of
>> rebuilding Solr indexes on streaming operations (such as adding nodes to an
>> existing cluster) required enough resources that we found it prohibitive.
>> That's because the bootstrapping node was also taking a production write
>> workload, and we didn't want to run our cluster with enough overhead that a
>> node could bootstrap and take production workload at the same time.
>>
>> For vanilla Cassandra workloads we have run clusters with quite a bit
>> more nodes than 100 without any appreciable trouble.  Curious if you can
>> share documents about clusters over 100 nodes causing troubles for users.
>> I'm wondering if it's related to node failure rate combined with vnodes
>> meaning that several concurrent node failures cause a part of the ring to
>> go offline too reliably.
>>
>> On Mon, Nov 5, 2018 at 7:38 AM onmstester onmstester
>>  wrote:
>>
>>> Hi,
>>>
>>> One of my applications requires to create a cluster with more than 100
>>> nodes, I've read documents recommended to use clusters with less than 50 or
>>> 100 nodes (Netflix got hundreds of clusters with less 100 nodes on each).
>>> Is it a good idea to use multiple clusters for a single application,
>>> just to decrease maintenance problems and system complexity/performance?
>>> If So, which one of below policies is more suitable to distribute data
>>> among clusters and Why?
>>> 1. each cluster' would be responsible for a specific partial set of
>>> tables only (table sizes are almost equal so easy calculations here) for
>>> example inserts to table X would go to cluster Y
>>> 2. shard data at loader level by some business logic grouping of data,
>>> for example all rows with some column starting with X would go to cluster Y
>>>
>>> I would appreciate sharing your experiences working with big clusters,
>>> problem encountered and solutions.
>>>
>>> Thanks in Advance
>>>
>>> Sent using Zoho Mail 
>>>
>>>
>>>
>
> --
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade
>
-- 


*Ben Slater*

*Chief Product Officer *

   


Read our latest technical blog posts here
.

This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain