Re: disable debug message on read repair

2020-03-10 Thread Gil Ganz
That's one option, I wish I there was a way to disable just that and not
the entire debug log level, there are some things there I would like to
keep.

On Sun, Mar 8, 2020 at 6:41 PM Jeff Jirsa  wrote:

> There are likely two log configs - one for debug.log and one for
> system.log. Disable the debug.log one, or change
> org.apache.cassandra.service to log at INFO instead
>
> Nobody needs to see every digest mismatch and that someone thought this
> was a good idea is amazing to me. Someone should jira that to be a trace.
>
>
> On Mar 8, 2020, at 3:25 AM, Gil Ganz  wrote:
>
> 
> Thanks Shalom, I know why these read repairs are happening, and they will
> continue to happen for some time, even if I will run a full repair.
> I would like to disable these warning messages.
>
> On Sun, Mar 8, 2020 at 10:19 AM Shalom Sagges 
> wrote:
>
>> Hi Gil,
>>
>> You can run a full repair on your cluster. But if these messages come
>> back again, you need to check what's causing these data inconsistencies.
>>
>>
>> On Sun, Mar 8, 2020 at 10:11 AM Gil Ganz  wrote:
>>
>>> Hey all
>>> I have a lot of debug message about read repairs in my debug log :
>>>
>>> DEBUG [ReadRepairStage:346] 2020-03-08 08:09:12,959
>>> ReadCallback.java:242 - Digest mismatch:
>>> org.apache.cassandra.service.DigestMismatchException: Mismatch for key
>>> DecoratedKey(-28476014476640,
>>> 000400871130303a33613a37643a33613a62643a383000)
>>> (38d3509295a283326a71887113fc vs 033cbba98c43e7ba6f15c0ba462f5fcc)
>>> at
>>> org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92)
>>> ~[apache-cassandra-3.11.5.jar:3.11.5]
>>> at
>>> org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233)
>>> ~[apache-cassandra-3.11.5.jar:3.11.5]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> [na:1.8.0_201]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> [na:1.8.0_201]
>>> at
>>> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
>>> [apache-cassandra-3.11.5.jar:3.11.5]
>>> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_201]
>>>
>>> Version is 3.11.5, anyone knows how to disable just these warnings?
>>> Thanks
>>> Gil
>>>
>>>


Re: disable debug message on read repair

2020-03-10 Thread Paul Chandler
Hi Gil,

All the logging is controlled via logback. You can change the level of any type 
of message.

Take a look here for some more details: 
https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/configuration/configLoggingLevels.html
 


Thanks 

Paul Chandler
www.redshots.com

> On 10 Mar 2020, at 08:56, Gil Ganz  wrote:
> 
> That's one option, I wish I there was a way to disable just that and not the 
> entire debug log level, there are some things there I would like to keep.
> 
> On Sun, Mar 8, 2020 at 6:41 PM Jeff Jirsa  > wrote:
> There are likely two log configs - one for debug.log and one for system.log. 
> Disable the debug.log one, or change org.apache.cassandra.service to log at 
> INFO instead 
> 
> Nobody needs to see every digest mismatch and that someone thought this was a 
> good idea is amazing to me. Someone should jira that to be a trace.
> 
> 
>> On Mar 8, 2020, at 3:25 AM, Gil Ganz > > wrote:
>> 
>> 
>> Thanks Shalom, I know why these read repairs are happening, and they will 
>> continue to happen for some time, even if I will run a full repair. 
>> I would like to disable these warning messages.
>> 
>> On Sun, Mar 8, 2020 at 10:19 AM Shalom Sagges > > wrote:
>> Hi Gil, 
>> 
>> You can run a full repair on your cluster. But if these messages come back 
>> again, you need to check what's causing these data inconsistencies. 
>> 
>> 
>> On Sun, Mar 8, 2020 at 10:11 AM Gil Ganz > > wrote:
>> Hey all
>> I have a lot of debug message about read repairs in my debug log :
>> 
>> DEBUG [ReadRepairStage:346] 2020-03-08 08:09:12,959 ReadCallback.java:242 - 
>> Digest mismatch:
>> org.apache.cassandra.service.DigestMismatchException: Mismatch for key 
>> DecoratedKey(-28476014476640, 
>> 000400871130303a33613a37643a33613a62643a383000) 
>> (38d3509295a283326a71887113fc vs 033cbba98c43e7ba6f15c0ba462f5fcc)
>> at 
>> org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92)
>>  ~[apache-cassandra-3.11.5.jar:3.11.5]
>> at 
>> org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233)
>>  ~[apache-cassandra-3.11.5.jar:3.11.5]
>> at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>  [na:1.8.0_201]
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>  [na:1.8.0_201]
>> at 
>> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
>>  [apache-cassandra-3.11.5.jar:3.11.5]
>> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_201]
>> 
>> Version is 3.11.5, anyone knows how to disable just these warnings?
>> Thanks
>> Gil
>> 



Re: slurm for cluster job scheduling and coordination

2020-03-10 Thread Patrick McFadin
Carl,

Slurm might be a nice way to keep things you already have built in some
sort of control plane. Things already built meaning terraform, ansible,
salt, chef,  wrote:

> Between repairs, rolling restarts, scheduled maintenance bounces, backups,
> upgrades, etc there are lots of cluster-wide tasks that would be nice to be
> scheduled and viewed.
>
> Slurm appears to have some features that support this but might be
> heavyweight considering its primary application is supercomputer job
> scheduling.
>
> I'd like something kinda independent of cloud vendor, container/VM/metal
> strategies, or specific "cloud os".
>
> Anyone tried slurm or have an alternative? I really don't want to write
> yet-another-scheduler.
>


Re: slurm for cluster job scheduling and coordination

2020-03-10 Thread John Sanda
>
> I've been working towards organizing an effort around using Kubernetes for
> cluster management. There is a lot of work to do but this could be
> something really important to tackle as a community if you(or anyone else)
> are interested in getting involved.
>

This is a big area of interest for me and would be more than happy to be
involved.

On Tue, Mar 10, 2020 at 12:41 PM Patrick McFadin  wrote:

> Carl,
>
> Slurm might be a nice way to keep things you already have built in some
> sort of control plane. Things already built meaning terraform, ansible,
> salt, chef,  Cassandra cluster, but that doesn't mean it's not used.
>
> I've been working towards organizing an effort around using Kubernetes for
> cluster management. There is a lot of work to do but this could be
> something really important to tackle as a community if you(or anyone else)
> are interested in getting involved.
>
> Patrick
>
> On Mon, Mar 9, 2020 at 9:34 AM Carl Mueller
>  wrote:
>
>> Between repairs, rolling restarts, scheduled maintenance bounces,
>> backups, upgrades, etc there are lots of cluster-wide tasks that would be
>> nice to be scheduled and viewed.
>>
>> Slurm appears to have some features that support this but might be
>> heavyweight considering its primary application is supercomputer job
>> scheduling.
>>
>> I'd like something kinda independent of cloud vendor, container/VM/metal
>> strategies, or specific "cloud os".
>>
>> Anyone tried slurm or have an alternative? I really don't want to write
>> yet-another-scheduler.
>>
>

-- 

- John