Thanks for the new method, it's more easier to create a flow now.
I have tried to use WeakReference/SoftReference to replace the rest property
in LazyFlow, both worked even in low memory scenario (-Xmx128m). The problem
is that the resolved Flow cannot be reused, it'll cause NPE since the
references may have been GCed, though it's rare that the Flow will be used
again. Can Tapestry introduce a new type of flow which is particular for
large dataset?
Another interesting thing is creating a new object in WeakReference seems
much faster than usual situation, at least on Mac.
Thanks,
Richard
Oct 14, 2011 at 4:49 PM, Howard Lewis Ship <hls...@gmail.com> wrote:
> On Fri, Oct 14, 2011 at 1:22 PM, Richard Yunhua Sang
> <yunhua.s...@gmail.com> wrote:
> > Tapestry 5 version: 5.3-beta19
> >
> > I love the API of tapestry-func, Today when I use tapestry-func to
> process a
> > file with 3 million lines, I get following exception:
> >
> > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> > at java.util.Arrays.copyOfRange(Arrays.java:3209)
> > at java.lang.String.<init>(String.java:216)
> > at java.io.BufferedReader.readLine(BufferedReader.java:331)
> > at java.io.BufferedReader.readLine(BufferedReader.java:362)
> > at org.apache.commons.io.LineIterator.hasNext(LineIterator.java:97)
> > at org.apache.tapestry5.func.LazyIterator.next(LazyIterator.java:33)
> > at org.apache.tapestry5.func.LazyFlow.resolve(LazyFlow.java:78)
> > at org.apache.tapestry5.func.LazyFlow.isEmpty(LazyFlow.java:61)
> > at
> org.apache.tapestry5.func.AbstractFlow$1.hasNext(AbstractFlow.java:63)
> > at org.apache.tapestry5.func.AbstractFlow.each(AbstractFlow.java:103)
> > at org.apache.tapestry5.func.AbstractFlow.each(AbstractFlow.java:32)
> > at
> com.datastream.services.flow.FileProcessor.main(FileProcessor.java:29)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
> >
> > Java code: (using common.io)
> >
> > public static void main(String[] args) {
> > try {
> > final LineIterator lineIterator =
> > FileUtils.lineIterator(FileUtils.getFile("c:\\tmp\\largefile.txt"));
> > Iterable<String> lineIterable = new Iterable<String>() {
> > public Iterator<String> iterator() {
> > return lineIterator;
> > }
> > };
> > F.flow(lineIterable)
> > .each(new Worker<String>() {
> > public void work(String s) {
> > }
> > });
> > LineIterator.closeQuietly(lineIterator);
> > } catch (IOException e) {
> > e.printStackTrace();
> > }
> >
> > }
> >
> > I have looked at the code, looks like Tapestry created a lot of
> > chained LazyFlow objects that used up memories. Is it a bug?
> >
>
> Certainly, that's not the intention; it's supposed to work that if you
> don't retain the head of the Flow, it should be able to release nodes
> to the GC. Unfortunately, there's a problem: the head is accessible to
> the GC on the stack as you invoke the each() method. Clojure handles
> this using semantics not available to Java (it sets the local variable
> to null AFTER pushing it onto the stack). I'm not sure there's a
> solution to this in pure Java ... I'm racking my brains right now to
> think of an approach involving WeakReferences that would get the job
> done.
>
> BTW, I'm adding a static F.flow(Iterable) method cause why not?
>
> > Thanks,
> > Richard
> >
>
>
>
> --
> Howard M. Lewis Ship
>
> Creator of Apache Tapestry
>
> The source for Tapestry training, mentoring and support. Contact me to
> learn how I can get you up and productive in Tapestry fast!
>
> (971) 678-5210
> http://howardlewisship.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org
> For additional commands, e-mail: users-h...@tapestry.apache.org
>
>
