Hi Michaël,
The fence is not a general selection solution. It is a very specialized
one. With my new Extract Layer in Fence plugin, I can create smaller
subsets of the million point layer in seconds. Selection will not do this
at all.
I am unable to select a million points without an out of memory error,
even with 750MB allocated. The dataset uses only 230 MB committed memory
when loaded, so the selection is using much more memory than the data
itself. I think the issue is that both a collection and a hashmap is used,
but I wouldn't want to try to come up with the same functionality without a
map. Perhaps you have some ideas there. For me, it is enough to use a
mechanism like the fence that is not bound by memory and does not require a
redraw, in those special cases for which selection was not designed.
As to which use cases are "normal "and which are "extreme", this is a moving
target. Personally, I am happy when I see people stretching the limits of
JUMP. Nothing makes inefficient code apparent like extreme cases. That
reminds me of a story.
In the early days of the Macintosh, before everyone had figured out how to
program event-driven GUIs, there was a common problem: slow drawing due to
multiple redundant redraws of the same regions. This was a notoriously
difficult problem to debug, until someone came up with a specialized control
panel. It's name was "Sludge", and what it did was to slow down the Mac's
processing by running excessive interrupts. With the Sludge control panel,
you could dial down the speed of your Mac until you could watch it draw the
individual pixels. Only then, could you see which routines were wasting
time by redrawing the same parts over and over again. I wish such a gadget
could be done today, but I doubt if it would be possible to grab that much
control over the PC's internals. That is why I prize so highly the
datasets that slow the processor down to human speeds.
regards,
Larry
On 10/8/07, Michaël Michaud <[EMAIL PROTECTED]> wrote:
>
> Hi Larry,
>
> May be I did not understand your solution, but I think like SS, that
> selecting with a fence is not a general solution for the selection
> problem. A selection can be the result of a query or any set of features
> without any particular spatial relationship.
> I wonder why collections containing selection are so "huge". Basically,
> a selection should be a set of handles (or a little more to be able to
> manage geometry parts).
> In the case of a large point layer without attribute, the selection size
> may be comparable to the feature layer size, but it should stay smaller,
> and it is really a worst case.
> Maybe there are other reasons why the selection is so large and slow :
> hard copy of features, cache, undo/redo management ?
>
> Anyway, thanks for the continuous improvement of OpenJUMP performances.
>
> Michaël
>
> Larry Becker a écrit :
>
> > Hi SS,
> >
> > I agree that the problem with selection is the large collections
> > being generated. "Parts" are only part of the problem and it is a bit
> > late to stop supporting this capability. Optimizations are always
> > possible, but only if you have a clear understanding of what you are
> > optimizing and use benchmarks for metrics.
> >
> > Larry
> >
> > On 10/8/07, *Sunburned Surveyor* <[EMAIL PROTECTED]
> > <mailto:[EMAIL PROTECTED]>> wrote:
> >
> > This sounds like a great solution Larry, but I was wondering if you
> > might be able to confirm why selection requires more memory than a
> > fence.
> >
> > Is it because when you use "selection" a bunch of Java collections
> > are
> > created, whereas when you use a fence, you are just creating a
> > geometry? Is it because with the fence no data structures for the
> > features within the fence are created until you actually want to do
> > something?
> >
> > The only problem I see with using a fence is that it wouldn't allow
> > complex selections. (For example, if I use a fence all the features
> > are the fence are in the "quasi-selection".I can't go in an remove
> > some items from the quasi-selection. Does this make sense?)
> >
> > I worked with some of the selection code recently, and it seemed
> very
> > complex to me. There were so many collections being used in the code
> I
> > had to write the purpose of each one down on a scratch pad so I
> could
> > keep track. It seems that most of this complexity was a result of
> > OpenJUMP's ability to select parts of Features, and not being
> > restricted to selection of whole features.
> >
> > I wonder if we couldn't come up with a much simpler selection system
> > that only selected whole features. We could probably do this using a
> > single collection that stored only the FID of a Feature, or an
> object
> > reference to the Feature. A collection of a couple million FID
> values
> > would only be a couple of MB correct?
> >
> > Perhaps your idea with the fence is best, but I'm thinking it might
> be
> > better to address the complexity in the selection system itself.
> >
> > I wonder how much people are using OpenJUMP's ability to select
> > "parts" of features.
> >
> > Just my thoughts. I don't think it would take me long to cook up the
> > simple "whole-feature-selection-only" system I am talking about.
> >
> > The Sunburned Surveyor
> >
> > On 10/8/07, Stefan Steiniger <[EMAIL PROTECTED]
> > <mailto:[EMAIL PROTECTED]>> wrote:
> > > simple solution and well thought :)
> > >
> > > we should go for
> > >
> > > stefan
> > >
> > > Larry Becker wrote:
> > >
> > > > Hi,
> > > >
> > > > Recent work with Extract Layers by Geometry Type, and
> improving
> > > > OpenJump's support for very large selections, has convinced me
> > that
> > > > JUMP needs a mechanism for dealing with subsets of large layers.
> > > > Although the selection mechanism is very flexible, it is very
> > > > expensive in terms of memory and processing. To seem what I
> mean,
> > > > consider Arnd's million point layer problem. To break the
> > layer into
> > > > more manageable pieces, he tried to select some of the points
> > and move
> > > > them to another layer. When this is done with such a large
> > selection,
> > > > the UI must build gigantic data structures. The result is an
> > > > extremely slow response or an out of memory error.
> > > >
> > > > An approach that might work for some problems is to use the
> > Fence tool
> > > > to indicate a subset of the data that can be used in place of a
> > > > selection. The advantage is that no data structures are
> > needed and
> > > > there are no selection handles to draw. I was thinking of
> > producing
> > > > another Layer Extract plugin that would extract features in
> > the Fence
> > > > to a new layer. By coding support methods in a utility class,
> we
> > > > could gradually add support for "features in fence" subsets to
> the
> > > > other tools.
> > > >
> > > > What do you think?
> > > >
> > > > regards,
> > > >
> > > > Larry Becker
> > > >
> > > > --
> > > > http://amusingprogrammer.blogspot.com/
> > > >
> > >
> >
> >------------------------------------------------------------------------
> > > >
> > >
> >
> >-------------------------------------------------------------------------
> > > >This SF.net email is sponsored by: Splunk Inc.
> > > >Still grepping through log files to find problems? Stop.
> > > >Now Search log events and configuration files using AJAX and a
> > browser.
> > > >Download your FREE copy of Splunk now >> http://get.splunk.com/
> > > >
> > >
> >
> >------------------------------------------------------------------------
> > > >
> > > >_______________________________________________
> > > >Jump-pilot-devel mailing list
> > > >Jump-pilot-devel@lists.sourceforge.net
> > <mailto:Jump-pilot-devel@lists.sourceforge.net>
> > > >https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
> > <https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel>
> > > >
> > > >
> > >
> > >
> > >
> >
> -------------------------------------------------------------------------
> > > This SF.net email is sponsored by: Splunk Inc.
> > > Still grepping through log files to find problems? Stop.
> > > Now Search log events and configuration files using AJAX and a
> > browser.
> > > Download your FREE copy of Splunk now >> http://get.splunk.com/
> > > _______________________________________________
> > > Jump-pilot-devel mailing list
> > > Jump-pilot-devel@lists.sourceforge.net
> > <mailto:Jump-pilot-devel@lists.sourceforge.net>
> > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
> > >
> >
> >
> -------------------------------------------------------------------------
> > This SF.net email is sponsored by: Splunk Inc.
> > Still grepping through log files to find problems? Stop.
> > Now Search log events and configuration files using AJAX and a
> > browser.
> > Download your FREE copy of Splunk now >> http://get.splunk.com/
> > _______________________________________________
> > Jump-pilot-devel mailing list
> > Jump-pilot-devel@lists.sourceforge.net
> > <mailto:Jump-pilot-devel@lists.sourceforge.net>
> > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
> > <https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel>
> >
> >
> >
> >
> > --
> > http://amusingprogrammer.blogspot.com/
> >
> >------------------------------------------------------------------------
> >
> >-------------------------------------------------------------------------
> >This SF.net email is sponsored by: Splunk Inc.
> >Still grepping through log files to find problems? Stop.
> >Now Search log events and configuration files using AJAX and a browser.
> >Download your FREE copy of Splunk now >> http://get.splunk.com/
> >
> >------------------------------------------------------------------------
> >
> >_______________________________________________
> >Jump-pilot-devel mailing list
> >Jump-pilot-devel@lists.sourceforge.net
> >https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
> >
> >
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems? Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Jump-pilot-devel mailing list
> Jump-pilot-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
>
--
http://amusingprogrammer.blogspot.com/
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Jump-pilot-devel mailing list
Jump-pilot-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
