Re: Attempting to Push Aurora to ~11 requests/second

2015-03-19 Thread Maxim Khutornenko
Same here. A single fork ran for 4 hours and I stopped it after 20K tasks. Running 30 forks now for about an hour or so, no failure so far (though vagrant IO ops stalled to a crawl). On Thu, Mar 19, 2015 at 3:25 PM, Bill Farner wrote: > I've been running 10 forks of your script against a schedule

Re: Attempting to Push Aurora to ~11 requests/second

2015-03-19 Thread Bill Farner
I've been running 10 forks of your script against a scheduler in the vagrant environment for a little over an hour, and have not had luck at inducing a crash yet. Are you able to repro there? Can you post the full scheduler log file from a repro of this? -=Bill On Thu, Mar 19, 2015 at 4:33 AM,

Re: Attempting to Push Aurora to ~11 requests/second

2015-03-19 Thread Ryan Orr
https://gist.github.com/ryanorr/b3736c03a9919f9b87dc Just a simple bash script. I run 10-15 of them in the background and it crashes usually in less than 30 mins as described. Thanks for looking into it. On Wed, Mar 18, 2015 at 7:19 PM, Bill Farner wrote: > I'm slightly doubtful that GC is the

Re: Attempting to Push Aurora to ~11 requests/second

2015-03-18 Thread Bill Farner
I'm slightly doubtful that GC is the issue (based on background from Ryan in IRC), but i could be wrong. Trying out a sample script would help us confirm or deny, though. -=Bill On Wed, Mar 18, 2015 at 11:17 AM, Maxim Khutornenko wrote: > This time gap of over 2 seconds suggests there may have

Re: Attempting to Push Aurora to ~11 requests/second

2015-03-18 Thread Maxim Khutornenko
This time gap of over 2 seconds suggests there may have been an intensive GC and/or IO operation that paused processing long enough for the ZK session timeout to expire: I0316 18:22:55.396108 20795 replica.cpp:508] Replica received write request for position 22669 I0316 18:22:57.621 THREAD1905 org

Re: Attempting to Push Aurora to ~11 requests/second

2015-03-18 Thread Bill Farner
If you can provide a script to trigger this in the vagrant environment it will be a tremendous help in finding the cause. On Wednesday, March 18, 2015, Ryan Orr wrote: > We're attempting to get Aurora to handle 11 job requests a second. We > realize we're going to be limited by the resource inte

Attempting to Push Aurora to ~11 requests/second

2015-03-18 Thread Ryan Orr
We're attempting to get Aurora to handle 11 job requests a second. We realize we're going to be limited by the resource intensive nature of the CLI and will be giving Herc a shot here soon; however we seem to get the scheduler to crash in less than an hour while submitting about 2 jobs/second. The