Thank you David,
do you have code we can peek at?
Massimo
On Aug 9, 9:42 pm, web2py_is_the_shiz <yanosh...@gmail.com> wrote:
> Hello all,
>
> Long time lurker here wishing to first thank all you for your
> wonderful work and a magnificent community. I have learned a great
> deal from this list and the book in a very very short order of time.
>
> Finally I may have something to contribute back to all you wonderful
> folks. Though not as feature rich as celery or even specifically
> built for this purpose I have found a combination that has proven
> successful for me and wish to share it with you all.
>
> I don't see a lot of talk about Redis on this list but Redis, BSON
> (binary implementation of json) and HotQueue together (all available
> both via pip and elsewhere on the interwebs) allow one to do many cool
> things with a great deal of performance and reliability. Redis keeps
> all of its data in memory but persists it to disk. See redis.io for
> more details and a fun javascript terminal to test redis out right
> there.
>
> Aside from having some really useful data types, Redis when combined
> with HotQueue and BSON turns your plain old cache server (I use Redis
> in lieu of Membase/Memcached) into an asynchronous task/messaging
> monster. HotQueue has very easy to follow documentation and plays
> very nicely with web2py however it was intended only as a message
> queue.
>
> BSON will help us with our transformation of HotQueue. BSON allows us
> to serialize our data (lists, strings, more) as documents/dicts into a
> string. This string can be pushed to a queue (into Redis). A
> separate worker will pull the string from the queue, de-serialize it
> and put it into the database or whatever else you care to do. With
> some creativity, your dict or list of dicts can turn go from plain old
> lists and dicts into an {'action': 'data' } task queue or {'userid':
> 'message to user' } async buffer for your DB.
>
> So if you wanted you could just install Redis and have a cache/queue/
> messaging system all in one. Redis also has a pub/sub system which
> seems powerful but haven't touched it yet. For my project the less
> software the better so this combination was a good fit for me.
>
> To recap you can put a list, dict, etc. containing your data,
> serialize your data into a string with BSON and put them into a Redis
> HotQueue. Write a worker (run it multiple times to have more
> processes handling your async data) to run in the background pulling
> from your HotQueue and inserting your messages into your db flavor of
> choice or handling the data however else you wish. Simple async.
> goodness which should drastically outperform disk-based RDBMS as a
> queue and save your db some pain.
>
> I hope this proves to be useful to someone looking for a queueing
> solution and thank you all again for this product. I will contribute
> back in any way I can.
>
> -David
