Yeah fully implementing streaming may require some non trivial effort.
But from your initial description it appeared that you have enough
memory to process even the largest blobs in isolation. So maybe
there's a memory leak somewhere that can be detected with a profiler
and fixed by changing some cache settings / DataContext scope without
rewriting Cayenne?
Andrus
On May 28, 2010, at 2:32 PM, mgarg...@escholar.com wrote:
Hi Tore,
I finally got around to looking at your code in the jira
attachments. It looks like it will need a little bit of updating to
get
it 3.0 ready, but at the end of the day it still pulls the whole thing
into memory at some point. I might ping you over the next week, if
you
don't mind, about this as I quickly (i.e. panic-mode) try to cobble
something together. I'm just learning cayenne so a lot of the
internals
are still not very clear to me.
Thanks.
-Mike
From:
Tore Halset <hal...@pvv.ntnu.no>
To:
user@cayenne.apache.org
Date:
05/25/2010 07:58 AM
Subject:
Re: Blobs in the DataContext
Hello.
I tried to implement support for streaming blobs back in 2006.
Sorry, but
it never got completed. I still think it is a nice feature. If you
want to
work on this issue, you might want to take a look at
streaming_blob_try2.zip from https://issues.apache.org/jira/browse/CAY-316
Regards,
- Tore.
On 21. mai 2010, at 23.27, mgarg...@escholar.com wrote:
Hi,
I'm using cayenne to store large files in BLOBs as a process
runs.
The first step of the process is storing large files (~ 600MB) and
they
are ending up in the DB just fine, then we run some tasks and get
some
output files, and then store the large output files (~ 500MB) to
the DB.
The output files are not making it into the DB. In fact it appears
that
the whole program is just sitting and waiting, for what, i have no
idea
and after you try and spawn another thread in the program it throws
an
out
of memory exception. I was trying to figure out why the larger input
files got persisted fine, but the large output files cause a
problem and
the only thing I could think of was that when the BLOBs are created
they
are cached in the DataContext and are never cleared eventually just
causing the memory to be exhausted. Is this possible? Anything else
anyone can think of?
note: I'm also compressing the stream in memory as I'm adding it to
the
byte[], but still... it works for the input files. also, each of
these
phases of the process is followed by a commit, so all the input files
are
committed together and all the output files should be committed
together
as well, but this never happens.
Thank you for any help you may be able to provide.
-Mike
