Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue

2017-10-13 Thread Ravindra Jaju
I've faced this issue before, and I'd hazard a guess that like me, it could be related to the IOPS settings. AWS' IO performance is highly variable, and you get good performance in bursts, in very low volumes, and generally only initially for long runs. Which kind of makes sense, unless you reserv

Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue

2017-10-13 Thread Nathan Fisher
It sounds like you have a memory leak. I would look at addressing that before any performance tricks. On Fri, 13 Oct 2017 at 05:35, wrote: > Following Daniel Compton's suggestion, I turned on logging for GC. I don't > see it happening more often, but the slow down does seem related to the > momen

Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue

2017-10-12 Thread lawrence . krubner
Following Daniel Compton's suggestion, I turned on logging for GC. I don't see it happening more often, but the slow down does seem related to the moment when the app hits the maximum memory allowed. It had been running with 4G, so I increased that to 7G, so it goes longer now before it hits 98

Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue

2017-10-12 Thread Nathan Fisher
Hi! Can you change one of the variables? Specifically can you replicate this on your local machine? If it happens locally then I would focus on something in the JVM eco-system. If you can't replicate it locally then it's possibly AWS specific. It sounds like you're using a t2.large or m4.xlarge.

Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue

2017-10-12 Thread lawrence . krubner
Daniel Compton, good suggestion. I've increased the memory to see if I can postpone the GCs, and I'll log that more carefully. On Wednesday, October 11, 2017 at 8:35:44 PM UTC-4, Daniel Compton wrote: > > Without more information it's hard to tell, but this looks a like it could > be a garbage

Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue

2017-10-12 Thread lawrence . krubner
Justin Smith, thanks, I've created an Uberjar that I now run under Supervisord on an EC2 instance, and I set the JVM options via the command that Supervisord calls. On Wednesday, October 11, 2017 at 6:58:52 PM UTC-4, Justin Smith wrote: > > a small thing here, if memory usage is important you

Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue

2017-10-11 Thread Daniel Compton
Without more information it's hard to tell, but this looks a like it could be a garbage collection issue. Can you run your test again and add some logging/monitoring to show each garbage collection? If my hunch is right, you'll see garbage collections getting more and more frequent until they take

Re: possibly a Clojure question or possibly an AWS question: slow writes to durable-queue

2017-10-11 Thread Justin Smith
a small thing here, if memory usage is important you should be building and running an uberjar instead of using lein on the server (this also has other benefits), and if you are doing that your project.clj jvm-opts are not used, you have to configure your java command line in aws instead On Wed, O

possibly a Clojure question or possibly an AWS question: slow writes to durable-queue

2017-10-11 Thread lawrence . krubner
I can't figure out if this is a Clojure question or an AWS question. And if it is a Clojure question, I can't figure out if it is more of a general JVM question, or if it is specific to some library such as durable-queue. I can redirect my question elsewhere, if people think this is an AWS quest