[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-09 Thread cyan
Very nice. Thank you!

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-09 Thread Anthony
Here are the __init__ and __call__ functions for the CacheInRam class: http://code.google.com/p/web2py/source/browse/gluon/cache.py#133 http://code.google.com/p/web2py/source/browse/gluon/cache.py#165 Looks like the both acquire and release locks during the course of the function call, but they d

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-09 Thread cyan
> Thanks for confirming this. The only kind of updates would be adding new >> key-value pairs. Having confirmed that this sort of updates will be >> consistent on the cache, I am now a little worried about the performance of >> doing so, given the mention of the mutex-lock design of the cache.

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-09 Thread Anthony
> > Thanks for confirming this. The only kind of updates would be adding new > key-value pairs. Having confirmed that this sort of updates will be > consistent on the cache, I am now a little worried about the performance of > doing so, given the mention of the mutex-lock design of the cache. >

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-09 Thread cyan
> I've looked deeper into this, and in web2py doc: >> http://www.web2py.com/examples/static/epydoc/web2py.gluon.cache.CacheInRam-class.html, >> >> it mentions: This is implemented as global (per process, shared by all >> threads) dictionary. A mutex-lock mechanism avoid conflicts. >> >> Does

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-08 Thread Anthony
> > I've looked deeper into this, and in web2py doc: > http://www.web2py.com/examples/static/epydoc/web2py.gluon.cache.CacheInRam-class.html, > > it mentions: This is implemented as global (per process, shared by all > threads) dictionary. A mutex-lock mechanism avoid conflicts. > > Does this

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-08 Thread Anthony
> > user_data = cache.ram('user_data', lambda:dict(), time_expire=None) >> >> # add the data from this user, this should also update the cached dict? >> user_data[this_user_id] = submitted_data >> > > The above would not update the dict in the cache -- you'd have to do that > explicitly: > Scrat

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-08 Thread cyan
> Regarding the pros and cons of this approach (in comparison to David's >> database approach), I wonder what are the potential pitfalls/risks of the >> cache approach. For examples, but not limited to, >> >> 1. Is there any consistency issue for the data stored in web2py cache? >> > > Yes, thi

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-08 Thread villas
In my opinion, using a DB is easier, more logical, transparent and makes your data more accessible. The function that is used to process the data on arrival can check whether all pieces of data are present. If so, then after saving the last piece it can then go on to migrate them to your ma

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-07 Thread Anthony
> > Regarding the pros and cons of this approach (in comparison to David's > database approach), I wonder what are the potential pitfalls/risks of the > cache approach. For examples, but not limited to, > > 1. Is there any consistency issue for the data stored in web2py cache? > Yes, this could

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-07 Thread cyan
> In general, the approach David suggests ( > https://groups.google.com/d/msg/web2py/4cBbis_B1i0/KkKnNwUw8lcJ) is > probably preferable, but below are answers to your questions... > > user_data = cache.ram('user_data', lambda:dict(), time_expire=None) >> >> # add the data from this user, this sh

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-07 Thread cyan
> Databases are specially designed for keeping persistent data - so there is > your answer! :) > > I suggest: > >1. Write the data initially to a transitional Sqlite DB on disk. >2. Once all the data pieces have arrived, migrate the completed data >to your main DB and delete all

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-07 Thread Anthony
In general, the approach David suggests (https://groups.google.com/d/msg/web2py/4cBbis_B1i0/KkKnNwUw8lcJ) is probably preferable, but below are answers to your questions... user_data = cache.ram('user_data', lambda:dict(), time_expire=None) > > # add the data from this user, this should also upd

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-07 Thread cyan
> If you just need to store data for a single user across requests, you can > store it in the session (each user has a separate session). But if you need > to store data from multiple users and then process it all together, you can > store it in the cache: http://web2py.com/books/default/chapt

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-07 Thread villas
Hi Cyan, Databases are specially designed for keeping persistent data - so there is your answer! :) I suggest: 1. Write the data initially to a transitional Sqlite DB on disk. 2. Once all the data pieces have arrived, migrate the completed data to your main DB and delete all the

Re: [web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-07 Thread Anthony
> > If you just need to store data for a single user across requests, you can > store it in the session (each user has a separate session). But if you need > to store data from multiple users and then process it all together, you can > store it in the cache: http://web2py.com/books/default/chap

Re: [web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-07 Thread cyan
> That might be just slightly unsafe, don't you think, if a piece of data > gets evicted before it's needed? I'd prefer to limit caching to data that > can be recreated when it's not found in the cache. > > An alternative might be a memory-based SQLite database, taking care not to > let it lea

Re: [web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-07 Thread Jonathan Lundell
On May 7, 2012, at 10:35 AM, Anthony wrote: > If you just need to store data for a single user across requests, you can > store it in the session (each user has a separate session). But if you need > to store data from multiple users and then process it all together, you can > store it in the ca

[web2py] Re: How to keep and manipulate data in memory with web2py

2012-05-07 Thread Anthony
If you just need to store data for a single user across requests, you can store it in the session (each user has a separate session). But if you need to store data from multiple users and then process it all together, you can store it in the cache: http://web2py.com/books/default/chapter/29/4#ca