Hint file getting stuck

[Paul Chandler]

Hi all

We keep having a problem with hint files on one of our Cassandra nodes (v 
3.11.6 ), there keeps being the following error messages repeated for same file.

INFO [HintsDispatcher:25] 2021-11-02 08:55:29,830 
HintsDispatchExecutor.java:289 - Finished hinted handoff of file 
72a18469-b7d2-499a-aed3-fd4e2cda9678-1635838529279-1.hints to endpoint 
/10.29.49.210: 72a18469-b7d2-499a-aed3-fd4e2cda9678, partially
INFO [HintsDispatcher:24] 2021-11-02 08:55:39,812 
HintsDispatchExecutor.java:289 - Finished hinted handoff of file 
72a18469-b7d2-499a-aed3-fd4e2cda9678-1635838529279-1.hints to endpoint 
/10.29.49.210: 72a18469-b7d2-499a-aed3-fd4e2cda9678, partially
INFO [HintsDispatcher:25] 2021-11-02 08:55:49,822 
HintsDispatchExecutor.java:289 - Finished hinted handoff of file 
72a18469-b7d2-499a-aed3-fd4e2cda9678-1635838529279-1.hints to endpoint 
/10.29.49.210: 72a18469-b7d2-499a-aed3-fd4e2cda9678, partially

On the receiving node ( cassandra0 ) we see the CPU shoot up, this is how 
notice we have a problem.

This has happened serval times with different files, and we find the only way 
to stop this is to delete the offending hint files.

The cluster can be a bit overloaded, and this is what is causing the hint files 
to be generated in the first place, we are working to get that stopped,  
However the question I don’t know the answer to is what causing this 
“partially” hint processing and how can we stop it happening?

Thanks

Paul

RE: One big giant cluster or several smaller ones?

[Durity, Sean R]

For memory-sake, you do not want “too many” tables in a single cluster (~200 is 
a reasonable rule of thumb). But I don’t see a major concern with a few very 
large tables in the same cluster. The client side, at least in Java, could get 
large (memory-wise) holding a Cluster object for multiple clusters.

I agree with Jeff: a cluster per app is the cleanest separation we have seen. 
Multi-tenant leads to many more potential problems. Multi-cluster per app seems 
unnecessarily complex.



Sean Durity – Staff Systems Engineer, Cassandra

From: S G 
Sent: Saturday, November 13, 2021 9:58 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: One big giant cluster or several smaller ones?

I think 1 cluster per large table should be preferred, rather than per 
application.
Example, what if there is a large application that requires several big tables, 
each many 10s of tera-bytes in size?
Is it still recommended to have 1 cluster for that app ?


On Fri, Nov 12, 2021 at 2:01 PM Jeff Jirsa 
mailto:jji...@gmail.com>> wrote:
Oh sorry - a cluster per application makes sense. Sharding within an 
application makes sense to avoid very very very large clusters (think: 
~thousand nodes). 1 cluster per app/use case.

On Fri, Nov 12, 2021 at 1:39 PM S G 
mailto:sg.online.em...@gmail.com>> wrote:
Thanks Jeff.
Any side-effect on the client config from small clusters perspective?

Like several smaller clusters means more CassandraClient objects on the client 
side but I guess number of connections shall remain the same as number of 
physical nodes will most likely remain the same only. So I think client side 
would not see any major issue.


On Fri, Nov 12, 2021 at 11:46 AM Jeff Jirsa 
mailto:jji...@gmail.com>> wrote:
Most people are better served building multiple clusters and spending their 
engineering time optimizing for maintaining multiple clusters, vs spending 
their engineering time learning how to work around the sharp edges that make 
large shared clusters hard.

Large multi-tenant clusters give you less waste and a bit more elasticity (one 
tenant can burst and use spare capacity that would typically be left for the 
other tenants). However, one bad use case / table can ruin everything (one bad 
read that generates GC hits all use cases), and eventually certain 
mechanisms/subsystems dont scale past certain points (e.g. schema - large 
schemas and large clusters are much harder than small schemas and small 
clusters)




On Fri, Nov 12, 2021 at 11:31 AM S G 
mailto:sg.online.em...@gmail.com>> wrote:
Hello,

Is there any case where we would prefer one big giant cluster (with multiple 
large tables) over several smaller clusters?
Apart from some management overhead of multiple Cassandra Clients, it seems 
several smaller clusters are always better than a big one:

  1.  Avoids SPOF for all tables
  2.  Helps debugging (less noise from all tables in the logs)
  3.  Traffic spikes on one table do not affect others if they are in different 
tables.
  4.  We can scale tables independently of each other - so colder data can be 
in a smaller cluster (more data/node) while hotter data can be on a bigger 
cluster (less data/node)

It does not mean that every table should be in its own cluster.
But large ones can be moved to their own dedicated clusters (like those more 
than a few terabytes).
And smaller ones can be clubbed together in one or few clusters.

Please share any recommendations for the above from actual production 
experiences.
Thanks for helping !



INTERNAL USE

RE: Separating storage and processing

[Durity, Sean R]

We have apps like this, also. For straight Cassandra, I think it is just the 
nature of how it works. DataStax provides some interesting solutions in 
different directions: BigNode (for handling 10-20 TB nodes) or Astra 
(cloud-based/container-driven solution that DOES separate read, write, and 
storage into separately scaled aspects of Cassandra). I suppose that you could 
do some similar work on your own with k8cassandra and StarGate.

Sean Durity – Staff Systems Engineer, Cassandra

From: onmstester onmstester 
Sent: Monday, November 15, 2021 12:56 AM
To: user 
Subject: [EXTERNAL] Separating storage and processing

Hi,
In our Cassandra cluster, because of big rows in input data/data model with TTL 
of several months, we ended up using almost 80% of storage (5TB per node), but 
having less than 20% of CPU usage which almost all of it would be writing rows 
to memtables and compacting sstables, so a lot of CPU capacity wasted.
I wonder if there is anything we can do to solve this problem using Cassandra 
or should migrate from Cassandra to something that separates storage and 
processing (currently i'm not aware of anything as satble as cassandra)?


Sent using Zoho Mail 
[zoho.com]




INTERNAL USE

Re: Separating storage and processing

[abe]

It sounds like you can downsize your cluster but increase your drive capacity. 
Depending on how your cluster is deployed, it’s very possible that disks larger 
than 5TB per node are available. Could you reduce the number of nodes and 
increase your disk sizes?

—
Abe

Re: Separating storage and processing

[onmstester onmstester]

I can, but i thought with 5TB per node already violated best practices (1-2 TB 
per node) and won't be a good idea to 2X or 3X that?


Sent using https://www.zoho.com/mail/





 On Mon, 15 Nov 2021 20:55:53 +0330   wrote 


It sounds like you can downsize your cluster but increase your drive capacity. 
Depending on how your cluster is deployed, it’s very possible that disks larger 
than 5TB per node are available. Could you reduce the number of nodes and 
increase your disk sizes? 
 
— 
Abe

Re: Separating storage and processing

[abe]

> I can, but i thought with 5TB per node already violated best practices (1-2 
> TB per node) and won't be a good idea to 2X or 3X that?

The main downside of larger disks is that it takes longer to replace a host 
that goes down, since there’s less network capacity to move data from surviving 
instances to the new, replacement instances. The longer it takes to replace a 
host, the longer the time window when further failure may cause unavailability 
(for example: if you’re running in a 3-instance cluster, one node goes down and 
requires replacement, any additional nodes going down will cause downtime for 
reads that require a quorum).

These are some of the main factors to consider here. You can always bump the 
disk capacity for one instance, measure replacement times, then decide whether 
to increase disk capacity across the cluster.

Re: Separating storage and processing

[onmstester onmstester]

Thank You


Sent using https://www.zoho.com/mail/





 On Tue, 16 Nov 2021 10:00:19 +0330   wrote 


> I can, but i thought with 5TB per node already violated best practices (1-2 
>TB per node) and won't be a good idea to 2X or 3X that?


The main downside of larger disks is that it takes longer to replace a host 
that goes down, since there’s less network capacity to move data from surviving 
instances to the new, replacement instances. The longer it takes to replace a 
host, the longer the time window when further failure may cause unavailability 
(for example: if you’re running in a 3-instance cluster, one node goes down and 
requires replacement, any additional nodes going down will cause downtime for 
reads that require a quorum).



These are some of the main factors to consider here. You can always bump the 
disk capacity for one instance, measure replacement times, then decide whether 
to increase disk capacity across the cluster.