Hello all,
Long time lurker here wishing to first thank all you for your
wonderful work and a magnificent community. I have learned a great
deal from this list and the book in a very very short order of time.
Finally I may have something to contribute back to all you wonderful
folks. Though not as feature rich as celery or even specifically
built for this purpose I have found a combination that has proven
successful for me and wish to share it with you all.
I don't see a lot of talk about Redis on this list but Redis, BSON
(binary implementation of json) and HotQueue together (all available
both via pip and elsewhere on the interwebs) allow one to do many cool
things with a great deal of performance and reliability. Redis keeps
all of its data in memory but persists it to disk. See redis.io for
more details and a fun javascript terminal to test redis out right
there.
Aside from having some really useful data types, Redis when combined
with HotQueue and BSON turns your plain old cache server (I use Redis
in lieu of Membase/Memcached) into an asynchronous task/messaging
monster. HotQueue has very easy to follow documentation and plays
very nicely with web2py however it was intended only as a message
queue.
BSON will help us with our transformation of HotQueue. BSON allows us
to serialize our data (lists, strings, more) as documents/dicts into a
string. This string can be pushed to a queue (into Redis). A
separate worker will pull the string from the queue, de-serialize it
and put it into the database or whatever else you care to do. With
some creativity, your dict or list of dicts can turn go from plain old
lists and dicts into an {'action': 'data' } task queue or {'userid':
'message to user' } async buffer for your DB.
So if you wanted you could just install Redis and have a cache/queue/
messaging system all in one. Redis also has a pub/sub system which
seems powerful but haven't touched it yet. For my project the less
software the better so this combination was a good fit for me.
To recap you can put a list, dict, etc. containing your data,
serialize your data into a string with BSON and put them into a Redis
HotQueue. Write a worker (run it multiple times to have more
processes handling your async data) to run in the background pulling
from your HotQueue and inserting your messages into your db flavor of
choice or handling the data however else you wish. Simple async.
goodness which should drastically outperform disk-based RDBMS as a
queue and save your db some pain.
I hope this proves to be useful to someone looking for a queueing
solution and thank you all again for this product. I will contribute
back in any way I can.
-David
