I'm curious what Guava (the google Java functional programming
library) does in this situation? I'm not sure they even try for
laziness.
On Tue, Oct 18, 2011 at 5:57 PM, Richard Yunhua Sang
<yunhua.s...@gmail.com> wrote:
> Thanks for the new method, it's more easier to create a flow now.
>
> I have tried to use WeakReference/SoftReference to replace the rest property
> in LazyFlow, both worked even in low memory scenario (-Xmx128m). The problem
> is that the resolved Flow cannot be reused, it'll cause NPE since the
> references may have been GCed, though it's rare that the Flow will be used
> again. Can Tapestry introduce a new type of flow which is particular for
> large dataset?
>
> Another interesting thing is creating a new object in WeakReference seems
> much faster than usual situation, at least on Mac.
>
> Thanks,
> Richard
>
> Oct 14, 2011 at 4:49 PM, Howard Lewis Ship <hls...@gmail.com> wrote:
>
>> On Fri, Oct 14, 2011 at 1:22 PM, Richard Yunhua Sang
>> <yunhua.s...@gmail.com> wrote:
>> > Tapestry 5 version: 5.3-beta19
>> >
>> > I love the API of tapestry-func, Today when I use tapestry-func to
>> process a
>> > file with 3 million lines, I get following exception:
>> >
>> > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>> > at java.util.Arrays.copyOfRange(Arrays.java:3209)
>> > at java.lang.String.<init>(String.java:216)
>> > at java.io.BufferedReader.readLine(BufferedReader.java:331)
>> > at java.io.BufferedReader.readLine(BufferedReader.java:362)
>> > at org.apache.commons.io.LineIterator.hasNext(LineIterator.java:97)
>> > at org.apache.tapestry5.func.LazyIterator.next(LazyIterator.java:33)
>> > at org.apache.tapestry5.func.LazyFlow.resolve(LazyFlow.java:78)
>> > at org.apache.tapestry5.func.LazyFlow.isEmpty(LazyFlow.java:61)
>> > at
>> org.apache.tapestry5.func.AbstractFlow$1.hasNext(AbstractFlow.java:63)
>> > at org.apache.tapestry5.func.AbstractFlow.each(AbstractFlow.java:103)
>> > at org.apache.tapestry5.func.AbstractFlow.each(AbstractFlow.java:32)
>> > at
>> com.datastream.services.flow.FileProcessor.main(FileProcessor.java:29)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
>> >
>> > Java code: (using common.io)
>> >
>> > public static void main(String[] args) {
>> > try {
>> > final LineIterator lineIterator =
>> > FileUtils.lineIterator(FileUtils.getFile("c:\\tmp\\largefile.txt"));
>> > Iterable<String> lineIterable = new Iterable<String>() {
>> > public Iterator<String> iterator() {
>> > return lineIterator;
>> > }
>> > };
>> > F.flow(lineIterable)
>> > .each(new Worker<String>() {
>> > public void work(String s) {
>> > }
>> > });
>> > LineIterator.closeQuietly(lineIterator);
>> > } catch (IOException e) {
>> > e.printStackTrace();
>> > }
>> >
>> > }
>> >
>> > I have looked at the code, looks like Tapestry created a lot of
>> > chained LazyFlow objects that used up memories. Is it a bug?
>> >
>>
>> Certainly, that's not the intention; it's supposed to work that if you
>> don't retain the head of the Flow, it should be able to release nodes
>> to the GC. Unfortunately, there's a problem: the head is accessible to
>> the GC on the stack as you invoke the each() method. Clojure handles
>> this using semantics not available to Java (it sets the local variable
>> to null AFTER pushing it onto the stack). I'm not sure there's a
>> solution to this in pure Java ... I'm racking my brains right now to
>> think of an approach involving WeakReferences that would get the job
>> done.
>>
>> BTW, I'm adding a static F.flow(Iterable) method cause why not?
>>
>> > Thanks,
>> > Richard
>> >
>>
>>
>>
>> --
>> Howard M. Lewis Ship
>>
>> Creator of Apache Tapestry
>>
>> The source for Tapestry training, mentoring and support. Contact me to
>> learn how I can get you up and productive in Tapestry fast!
>>
>> (971) 678-5210
>> http://howardlewisship.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org
>> For additional commands, e-mail: users-h...@tapestry.apache.org
>>
>>
>
--
Howard M. Lewis Ship
Creator of Apache Tapestry
The source for Tapestry training, mentoring and support. Contact me to
learn how I can get you up and productive in Tapestry fast!
(971) 678-5210
http://howardlewisship.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org
For additional commands, e-mail: users-h...@tapestry.apache.org
