As an alternative to polling to get the Redis state, you could consider HTTP server-side-events.
—Paul > On Sep 29, 2015, at 5:41 PM, Jonathan Vanasco <[email protected]> wrote: > > we have a similar situation. `pyramid` is the main web application. when a > user uploads a file, it is handed off to `celery` via `redis`, which handles > all the image resizing and uploading onto S3. > > the solution we went with: > > • short polling on a configurable delay. We default to 2500ms, and half or > double it (manually) depending on average server load. > > • we use a VERY STRIPPED DOWN pyramid view. It's part of the normal app, but > inherits from a different base class (we use class based views) and has a > very lean integration with pyramid. it basically has 20% the overhead of our > other views. this allows for the short polling to even be 100s of ms if we > want. > > • just to be safe and bust aggressive caching, we send a nonce with each > poll. (originally we put the nonce in the url, but that had a wee bit of > overhead in pyramid) > > Since the "answer" is going to be in redis, you'll have a blazing fast lookup. > > • a streamlined view/route gets it done rather perfomant. > • if things slow down, you can pull it out and build a dedicated pyramid > service (it won't be serving other routes, so the memory footprint will be > smaller and you can run way more instances) > • if things still don't scale well, then you can think about nodejs. we have > some twisted services and would probably choose nodejs for this task over > twisted -- it's better suited and you'll be abandoning enough of your > codebase to work in Twisted, that it's worth starting such a small project > from scratch in nodejs. > > You could definitely do long polling instead. short polling is just super > simple to get up and running. > > -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at http://groups.google.com/group/pylons-discuss > <http://groups.google.com/group/pylons-discuss>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/pylons-discuss. For more options, visit https://groups.google.com/d/optout.
