Do we have to setup the reaper on one of the node where Cassandra cluster is running? We are using a separate node where we have the connectivity to the Cassandra cluster .
We have tried with the certificate settings in /usr/local/bin/cassandra-reaper We have put below in /usr/local/bin/cassandra-reaper JVM_OPTS="$JVM_OPTS -Dssl.enable=true -Djavax.net.ssl.keyStore=/etc/dse/cassandra/keystores/server-keystore.jks -Djavax.net.ssl.keyStorePassword=xxxxxx -Djavax.net.ssl.trustStore=/etc/dse/cassandra/keystores/server-truststore.jks -Djavax.net.ssl.trustStorePassword=xxxxxxx" But still getting below error: Exception in thread "main" java.lang.IllegalStateException: Cannot initialize SSL Context at com.datastax.driver.core.JdkSSLOptions.makeDefaultContext(JdkSSLOptions.java:81) at com.datastax.driver.core.JdkSSLOptions.<init>(JdkSSLOptions.java:49) at com.datastax.driver.core.JdkSSLOptions$Builder.build(JdkSSLOptions.java:128) at systems.composable.dropwizard.cassandra.ssl.JDKSSLOptionsFactory.build(JDKSSLOptionsFactory.java:15) at java.util.Optional.map(Optional.java:215) at systems.composable.dropwizard.cassandra.CassandraFactory.build(CassandraFactory.java:477) at systems.composable.dropwizard.cassandra.CassandraFactory.build(CassandraFactory.java:447) at io.cassandrareaper.storage.CassandraStorage.<init>(CassandraStorage.java:140) at io.cassandrareaper.ReaperApplication.initializeStorage(ReaperApplication.java:235) at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:140) at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:67) at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43) at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:85) at io.dropwizard.cli.Cli.run(Cli.java:75) at io.dropwizard.Application.run(Application.java:79) at io.cassandrareaper.ReaperApplication.main(ReaperApplication.java:87) Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext) at java.security.Provider$Service.newInstance(Provider.java:1617) at sun.security.jca.GetInstance.getInstance(GetInstance.java:236) at sun.security.jca.GetInstance.getInstance(GetInstance.java:164) at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156) at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96) at com.datastax.driver.core.JdkSSLOptions.makeDefaultContext(JdkSSLOptions.java:79) ... 15 more Caused by: java.security.PrivilegedActionException: java.io.FileNotFoundException: /etc/dse/cassandra/keystores/server-keystore.jks (No such file or directory) at java.security.AccessController.doPrivileged(Native Method) at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.getKeyManagers(SSLContextImpl.java:822) at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.<clinit>(SSLContextImpl.java:758) at sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:913) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.security.Provider$Service.newInstance(Provider.java:1595) ... 20 more Caused by: java.io.FileNotFoundException: /etc/dse/cassandra/keystores/server-keystore.jks (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(FileInputStream.java:93) at sun.security.ssl.SSLContextImpl$DefaultManagersHolder$2.run(SSLContextImpl.java:826) at sun.security.ssl.SSLContextImpl$DefaultManagersHolder$2.run(SSLContextImpl.java:823) On 24 May 2018 at 14:12, Dennis Lovely <d...@aegisco.com> wrote: > looks like you're connecting to a service listening on SSL but you don't > have the CA used in your truststore > > On Thu, May 24, 2018 at 1:58 PM, Surbhi Gupta <surbhi.gupt...@gmail.com> > wrote: > >> Getting below error: >> >> Caused by: sun.security.validator.ValidatorException: PKIX path building >> failed: sun.security.provider.certpath.SunCertPathBuilderException: >> unable to find valid certification path to requested target >> >> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) >> >> at sun.security.validator.PKIXValidator.engineValidate(PKIXVali >> dator.java:302) >> >> at sun.security.validator.Validator.validate(Validator.java:260) >> >> at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustMana >> gerImpl.java:324) >> >> at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509Trust >> ManagerImpl.java:281) >> >> at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X50 >> 9TrustManagerImpl.java:136) >> >> at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHa >> ndshaker.java:1501) >> >> ... 20 common frames omitted >> >> Any thought? >> >> On 24 May 2018 at 10:35, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote: >> >>> Another question, We use 9142 cqlsh port in one of the datacenter and on >>> other datacenter we use 9042 port. >>> How should we configure this ? >>> >>> On 24 May 2018 at 10:22, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote: >>> >>>> What is the impact of >>>> PARALLEL - all replicas at the same time ? >>>> Will it make repair faster,? >>>> Do we expect more CPU , Load and memory usage in case if we use >>>> Parallel , compare to other settings ? >>>> >>>> >>>> >>>> On 21 May 2018 at 22:55, Alexander Dejanovski <a...@thelastpickle.com> >>>> wrote: >>>> >>>>> You won't be able to have less segments than vnodes, so just use 256 >>>>> segments per node, use parallel as repair parallelism, and set intensity >>>>> to >>>>> 1. >>>>> >>>>> You apparently have more than 3TB per node, and that kind of density >>>>> is always challenging when it comes to run "fast" repairs. >>>>> >>>>> Cheers, >>>>> >>>>> Le mar. 22 mai 2018 à 07:28, Surbhi Gupta <surbhi.gupt...@gmail.com> >>>>> a écrit : >>>>> >>>>>> We are on Dse 4.8.15 and it is cassandra 2.1. >>>>>> What are the best configuration to use for reaper for 144 nodes with >>>>>> 256 vnodes and it shows around 532TB data when we start opscenter >>>>>> repairs. >>>>>> >>>>>> We need to finish repair soon. >>>>>> >>>>>> On Mon, May 21, 2018 at 10:53 AM Alexander Dejanovski < >>>>>> a...@thelastpickle.com> wrote: >>>>>> >>>>>>> Hi Subri, >>>>>>> >>>>>>> Reaper might indeed be your best chance to reduce the overhead of >>>>>>> vnodes there. >>>>>>> The latest betas include a new feature that will group vnodes >>>>>>> sharing the same replicas in the same segment. This will allow to have >>>>>>> less >>>>>>> segments than vnodes, and is available with Cassandra 2.2 and onwards >>>>>>> (the >>>>>>> improvement is especially beneficial with Cassandra 3.0+ as such token >>>>>>> ranges will be repaired in a single session). >>>>>>> >>>>>>> We have a gitter that you can join if you want to ask questions. >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> Le lun. 21 mai 2018 à 15:29, Surbhi Gupta <surbhi.gupt...@gmail.com> >>>>>>> a écrit : >>>>>>> >>>>>>>> Thanks Abdul >>>>>>>> >>>>>>>> On Mon, May 21, 2018 at 6:28 AM Abdul Patel <abd786...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> We have a paramater in reaper yaml file called >>>>>>>>> repairManagerSchrdulingIntervalSeconds default is 10 seconds , >>>>>>>>> i tested with 8,6,5 seconds and found 5 seconds optimal for my >>>>>>>>> environment >>>>>>>>> ..you go down further but it will have cascading effects in cpu and >>>>>>>>> memory >>>>>>>>> consumption. >>>>>>>>> So test well. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Monday, May 21, 2018, Surbhi Gupta <surbhi.gupt...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks a lot for your inputs, >>>>>>>>>> Abdul, how did u tune reaper? >>>>>>>>>> >>>>>>>>>> On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad < >>>>>>>>>> j...@jonhaddad.com> wrote: >>>>>>>>>> >>>>>>>>>>> FWIW the largest deployment I know about is a single reaper >>>>>>>>>>> instance managing 50 clusters and over 2000 nodes. >>>>>>>>>>> >>>>>>>>>>> There might be bigger, but I either don’t know about it or can’t >>>>>>>>>>> remember. >>>>>>>>>>> >>>>>>>>>>> On Sun, May 20, 2018 at 10:04 AM Abdul Patel < >>>>>>>>>>> abd786...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I recently tested reaper and it actually helped us alot. Even >>>>>>>>>>>> with our small footprint 18 node reaper takes close to 6 >>>>>>>>>>>> hrs.<intially took >>>>>>>>>>>> 13 hrs ,i was able to tune it 50%>. But it really depends on >>>>>>>>>>>> number nodes. >>>>>>>>>>>> For example if you have 4 nodes then it runs on 4*256<vnodes> =1024 >>>>>>>>>>>> segements , so for your env. Ut will be 256*144 close to 36k >>>>>>>>>>>> segements. >>>>>>>>>>>> Better test on poc box how much time it takes and then proceed >>>>>>>>>>>> further ..i have tested so far in 1 dc only , we can actually have >>>>>>>>>>>> seperate >>>>>>>>>>>> reaper instance handling seperate dc but havent tested it yet. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Sunday, May 20, 2018, Surbhi Gupta <surbhi.gupt...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> We have a cluster with 144 nodes( 3 datacenter) with 256 >>>>>>>>>>>>> Vnodes . >>>>>>>>>>>>> When we tried to start repairs from opscenter then it showed >>>>>>>>>>>>> 1.9Million ranges to repair . >>>>>>>>>>>>> And even after doing compaction and strekamthroughput to 0 , >>>>>>>>>>>>> opscenter is not able to help us much to finish repair in 9 days >>>>>>>>>>>>> timeframe . >>>>>>>>>>>>> >>>>>>>>>>>>> What is your thought on Reaper ? >>>>>>>>>>>>> Do you think , Reaper might be able to help us in this >>>>>>>>>>>>> scenario ? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> Surbhi >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>> Jon Haddad >>>>>>>>>>> http://www.rustyrazorblade.com >>>>>>>>>>> twitter: rustyrazorblade >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>> ----------------- >>>>>>> Alexander Dejanovski >>>>>>> France >>>>>>> @alexanderdeja >>>>>>> >>>>>>> Consultant >>>>>>> Apache Cassandra Consulting >>>>>>> http://www.thelastpickle.com >>>>>>> >>>>>>> >>>>>>> -- >>>>> ----------------- >>>>> Alexander Dejanovski >>>>> France >>>>> @alexanderdeja >>>>> >>>>> Consultant >>>>> Apache Cassandra Consulting >>>>> http://www.thelastpickle.com >>>>> >>>> >>>> >>> >> >
