Wow.. that does say the mesh will hold a request until an app is available, but that is not how Heroku is currently operating..
I'll write this up a bit better and send it off to Heroku support... but here is how to duplicate... --- Created a new app http://h12-test2.heroku.com/ Simple sinatra app. Responds to two paths / and /wait (/wait sleeps for 300 seconds) === h12-test2 Web URL: http://h12-test2.heroku.com/ Git Repo: [email protected]:h12-test2.git Dynos: 2 Workers: 0 Repo size: 1M Slug size: 1M Stack: bamboo-ree-1.8.7 Data size: (empty) Addons: Expanded Logging, Shared Database 5MB 2 Dynos available... If i have one tab open with /wait, according to Heroku.. I should not get a time out on / if I just hit reload and reload in my browser... but I do... 2011-02-16T12:07:03-08:00 heroku[web.2]: State changed from starting to up 2011-02-16T12:07:04-08:00 heroku[web.1]: State changed from starting to up 2011-02-16T12:09:44-08:00 heroku[router]: Error H12 (Request timeout) - > GET h12-test2.heroku.com/wait dyno=web.2 queue=0 wait=0ms service=0ms bytes=0 2011-02-16T12:09:45-08:00 heroku[router]: GET h12-test2.heroku.com/ favicon.ico dyno=web.1 queue=0 wait=0ms service=7ms bytes=221 2011-02-16T12:09:45-08:00 app[web.1]: xx.223.127.66, 10.110.34.145 - - [16/Feb/2011 12:09:45] "GET /favicon.ico HTTP/1.1" 404 18 0.0009 2011-02-16T12:09:45-08:00 heroku[nginx]: GET /favicon.ico HTTP/1.1 | xx.223.127.66 | 252 | http | 404 2011-02-16T12:09:50-08:00 heroku[router]: Error H12 (Request timeout) - > GET h12-test2.heroku.com/ dyno=web.2 queue=0 wait=0ms service=0ms bytes=0 2011-02-16T12:09:50-08:00 heroku[nginx]: GET / HTTP/1.1 | xx. 223.127.66 | 3364 | http | 502 2011-02-16T12:10:01-08:00 heroku[router]: GET h12-test2.heroku.com/ dyno=web.1 queue=0 wait=0ms service=1ms bytes=238 2011-02-16T12:10:01-08:00 app[web.1]: xx.223.127.66, 10.101.29.42 - - [16/Feb/2011 12:10:01] "GET / HTTP/1.0" 200 59 0.0004 2011-02-16T12:10:01-08:00 heroku[nginx]: GET / HTTP/1.1 | xx. 223.127.66 | 269 | http | 200 2011-02-16T12:10:20-08:00 heroku[router]: Error H12 (Request timeout) - > GET h12-test2.heroku.com/favicon.ico dyno=web.2 queue=0 wait=0ms service=0ms bytes=0 2011-02-16T12:10:20-08:00 heroku[router]: GET h12-test2.heroku.com/ favicon.ico dyno=web.1 queue=0 wait=0ms service=8ms bytes=221 2011-02-16T12:10:20-08:00 app[web.1]: xx.223.127.66, 10.110.34.145 - - [16/Feb/2011 12:10:20] "GET /favicon.ico HTTP/1.0" 404 18 0.0005 2011-02-16T12:10:20-08:00 heroku[nginx]: GET /favicon.ico HTTP/1.1 | xx.223.127.66 | 3364 | http | 502 2011-02-16T12:10:21-08:00 heroku[nginx]: GET /favicon.ico HTTP/1.1 | xx.223.127.66 | 252 | http | 404 2011-02-16T12:10:33-08:00 heroku[router]: Error H12 (Request timeout) - > GET h12-test2.heroku.com/ dyno=web.2 queue=0 wait=0ms service=0ms bytes=0 2011-02-16T12:10:33-08:00 heroku[nginx]: GET / HTTP/1.1 | xx. 223.127.66 | 3364 | http | 502 2011-02-16T12:10:44-08:00 app[web.1]: xx.223.127.66, 10.108.18.220 - - [16/Feb/2011 12:10:44] "GET / HTTP/1.0" 200 59 0.0004 The dyno web.2 is busy, web.1 is open... yet requests get sent to web. 2 and H12 timeout. Why? -tim On Feb 16, 2:12 pm, Neil Middleton <[email protected]> wrote: > Although the symptoms that you are seeing may not indicate it the two systems > are the same. There is a queue backlog, and the dynos pick up the next > request from that backlog when they become idle, as described > here:http://devcenter.heroku.com/articles/key-concepts-performance(esp the > part about backlog). > > I've seen many instances on my applications which indicate this to be an > accurate description. > > Although I have no real clue about the application you have regarding number > of dynos etc, it would appear that what you are seeing could be a different > problem. > > Neil Middletonhttp://about.me/neilmiddleton > > > > > > > > On Wednesday, 16 February 2011 at 18:01, Tim W wrote: > > It is not identical to what Heroku is providing.. The Heroku mesh > > seems to blindly sends a request to a dyno, no matter the current > > status of the dyno. The queue is at the dyno level. Passenger holds > > back the request until a process is available.. > > > With passenger you do not end up in the situation noted below, where > > as with Heroku you do.. > > (Request Y gets served ok with passenger, with Heroku, request Y gets > > the H12 error) > > > Quoted from passenger docs (this is what happens if you have that > > feature turned off on passenger and what always happens with Heroku): > > ---------------------------------------------------------------------- > > The situation looks like this: > > > Backend process A: [* ] (1 request in queue) > > Backend process B: [*** ] (3 requests in queue) > > Backend process C: [*** ] (3 requests in queue) > > Backend process D: [*** ] (3 requests in queue) > > Each process is currently serving short-running requests. > > > Phusion Passenger will forward the next request to backend process A. > > A will now have 2 items in its queue. We’ll mark this new request with > > an X: > > > Backend process A: [*X ] (2 request in queue) > > Backend process B: [*** ] (3 requests in queue) > > Backend process C: [*** ] (3 requests in queue) > > Backend process D: [*** ] (3 requests in queue) > > > Assuming that B, C and D still aren’t done with their current request, > > the next HTTP request - let’s call this Y - will be forwarded to > > backend process A as well, because it has the least number of items in > > its queue: > > > Backend process A: [*XY ] (3 requests in queue) > > Backend process B: [*** ] (3 requests in queue) > > Backend process C: [*** ] (3 requests in queue) > > Backend process D: [*** ] (3 requests in queue) > > > But if request X happens to be a long-running request that needs 60 > > seconds to complete, then we’ll have a problem. Y won’t be processed > > for at least 60 seconds. It would have been a better idea if Y was > > forward to processes B, C or D instead, because they only have short- > > living requests in their queues. > > > On Feb 16, 12:50 pm, Neil Middleton <[email protected]> wrote: > > > Is this not identical to what Heroku provides though? Your global queue > > > is your applications dynos and the routing mesh will send requests to > > > whichever dynos are idle. The wait being the backlog. > > > > The only difference I can see is that Passenger won't, by default, spit > > > back any requests that take longer than 30 seconds. > > > > Neil Middletonhttp://about.me/neilmiddleton > > > > On Wednesday, 16 February 2011 at 17:46, Tim W wrote: > > > > Passenger... imho handles this better then Heroku > > > > >http://www.modrails.com/documentation/Users%20guide%20Apache.html#Pas... > > > > > > > If global queuing is turned on, then Phusion Passenger will use a > > > > > > global queue that’s shared between all backend processes. If an > > > > > > HTTP request comes in, and all the backend processes are still > > > > > > busy, then Phusion Passenger will wait until at least one backend > > > > > > process is done, and will then forward the request to that > > > > > > process.<< > > > > > default is on > > > > > On Feb 16, 12:36 pm, Neil Middleton <[email protected]> wrote: > > > > > AFAIK Passenger does have a similar concept with running processes > > > > > (having a default of six running processes, which are comparable to 6 > > > > > dynos). > > > > > > The situation you describe should have the same results on Passenger > > > > > as Heroku. More info on Passenger > > > > > here:http://www.modrails.com/documentation/Users%20guide%20Apache.html#_re... > > > > > > Neil Middletonhttp://about.me/neilmiddleton > > > > > > On Wednesday, 16 February 2011 at 17:29, Tim W wrote: > > > > > > I guess I am just used to using passenger which uses a global queue, > > > > > > making a single long running request a non issue. > > > > > > > On Feb 16, 11:57 am, Neil Middleton <[email protected]> > > > > > > wrote: > > > > > > > Is it, but you have a healthy dyno. If the dyno crashes, or hangs > > > > > > > somehow, it gets removed. > > > > > > > > Neil Middletonhttp://about.me/neilmiddleton > > > > > > > > On Wednesday, 16 February 2011 at 16:55, Tim W wrote: > > > > > > > > Thanks, I will give rack-timeout a try. > > > > > > > > > So what it seems like is that the routing mesh is not as > > > > > > > > sophisticated > > > > > > > > as Heroku leads on? > > > > > > > > > On Feb 16, 11:45 am, Neil Middleton <[email protected]> > > > > > > > > wrote: > > > > > > > > > The dyno is still running the long request, successfully. > > > > > > > > > It's only the routing mesh that's returned the timeout error > > > > > > > > > back to the user. Therefore, the dynos still in your 'grid' > > > > > > > > > and ready for new requests. > > > > > > > > > > I blogged about something very similar a couple of weeks > > > > > > > > > back:http://neilmiddleton.com/avoiding-zombie-dynos-with-heroku > > > > > > > > > > Neil Middletonhttp://about.me/neilmiddleton > > > > > > > > > > On Wednesday, 16 February 2011 at 16:42, Tim W wrote: > > > > > > > > > > The Heroku website claims: > > > > > > > > > > >http://heroku.com/how/dyno_grid_last#3 > > > > > > > > > > "If a dyno is unresponsive for any reason (user bugs, long > > > > > > > > > > requests, > > > > > > > > > > or high load), other requests will be routed around it." > > > > > > > > > > > In my experience, this does not seem to be the case. We > > > > > > > > > > have several > > > > > > > > > > admin features in our app that when requested with certain > > > > > > > > > > params, it > > > > > > > > > > can take longer then 30s to run. (I am working on ways to > > > > > > > > > > get these in > > > > > > > > > > check and in the background). When a user trips one of > > > > > > > > > > these long > > > > > > > > > > running requests, Heroku appears to queue additional > > > > > > > > > > requests to this > > > > > > > > > > dyno and those requests time out, even though there are > > > > > > > > > > plenty of > > > > > > > > > > other dynos available to handle that request. > > > > > > > > > > > Is the statement on the Heroku website true or false? It > > > > > > > > > > does not > > > > > > > > > > appear that Heroku actively monitors the dynos to see if > > > > > > > > > > they are busy > > > > > > > > > > with a long running request. Is there a better way to > > > > > > > > > > handle this > > > > > > > > > > situation? > > > > > > > > > > > Thanks.. > > > > > > > > > > -tim > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > Google Groups "Heroku" group. > > > > > > > > > > To post to this group, send email to > > > > > > > > > > [email protected]. > > > > > > > > > > To unsubscribe from this group, send email to > > > > > > > > > > [email protected]. > > > > > > > > > > For more options, visit this group > > > > > > > > > > athttp://groups.google.com/group/heroku?hl=en. > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the > > > > > > > > Google Groups "Heroku" group. > > > > > > > > To post to this group, send email to [email protected]. > > > > > > > > To unsubscribe from this group, send email to > > > > > > > > [email protected]. > > > > > > > > For more options, visit this group > > > > > > > > athttp://groups.google.com/group/heroku?hl=en. > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "Heroku" group. > > > > > > To post to this group, send email to [email protected]. > > > > > > To unsubscribe from this group, send email to > > > > > > [email protected]. > > > > > > For more options, visit this group > > > > > > athttp://groups.google.com/group/heroku?hl=en. > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "Heroku" group. > > > > To post to this group, send email to [email protected]. > > > > To unsubscribe from this group, send email to > > > > [email protected]. > > > > For more options, visit this group > > > > athttp://groups.google.com/group/heroku?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Heroku" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/heroku?hl=en. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
