Hi Alex,
On Aug 7, 6:54 pm, Alex Willmer <a...@moreati.org.uk> wrote:
> On Aug 7, 5:26 pm, GZ <zyzhu2...@gmail.com> wrote:
>
> > I am wondering if there is a module that can persist a stream of
> > objects without having to load everything into memory. (For this
> > reason, I think Pickle is out, too, because it needs everything to be
> > in memory.)
>
> From the pickle docs it looks like you could do something like:
>
> try:
> import cPickle as pickle
> except ImportError
> import pickle
>
> file_obj = open('whatever', 'wb')
> p = pickle.Pickler(file_obj)
>
> for x in stream_of_objects:
> p.dump(x)
> p.memo.clear()
>
> del p
> file_obj.close()
>
> then later
>
> file_obj = open('whatever', 'rb')
> p = pickle.Unpickler(file_obj)
>
> while True:
> try:
> x = p.load()
> do_something_with(x)
> except EOFError:
> break
>
> Your loading loop could be wrapped in a generator function, so only
> one object should be held in memory at once.
This totally works!
Thanks!
--
http://mail.python.org/mailman/listinfo/python-list
