> Hi Paolo, > > Nice to hear you, you surely have a good experience about > database-access. > I like the idea of an embedded database. > I have made some tests in the past with jump+hsqldb, but I didn't get > any memory improvement (I have not much dbms knowledge and I may have > done mistakes leading to some sorts of memory leaks...:-( > I have also made some tests showing that accessing a whole dataset by > feature id (randomly) is quite slow and making a query returning the > whole dataset returns a resultset which doesn't fit into > memory (may be > the database I tested lacked a kind of 'cursor' capability or > may be I > didn't know how to use it). > Projet-sigle made some work on H2 which is also a very nice embeded > database announcing very good performances. I never tried > derby but as > far as performance is concerned, I read some bad results compared to > hsqldb or H2. > Your opinion about those databases, how database access can > compare to > file access, and how to access data in an efficient way are > welcome (I > mean in a way that is fast and does not use much memory).
Hi Michaël, I'm sorry but I don't have any experience with HSQL, H2, or Derby, so I can't tell how they would perform in JUMP, both from a speed and a memory-use point of view. This aside, I found that having a "real", albeit embedded, database would be useful for OJ. Shape files are good to exchange data, but they suffer from the limitations of the DBF format (field name length, data types, etc.). Also I consider Shape files only as a _transport_ format, not as something one should actually _work_ upon. Instead engines like HSQL and others are able to manage any data type and can create files that can be exchanged without any loss of information. So a PostgreSQL user could save a few layers to the embedded DB, send the relevant files to an Oracle user and then he can load those layers in JUMP and save them as native Oracle tables. You can do the same with Shape files, but you risk to loose data and/or data structure. But what Landon (A.K.A. - The Sunburned Surveyor) is trying to achieve is probably a different thing. For sure OJ should be able to, at least, _display_ large Shape files without consuming too much memory, and this have to be addressed. Since those embedded DB engines already have capacities to balance memory use, they _may_ be a solution, but the requirement of not been dependant from another piece of software is a good one, and it may justify the burden of writing such a functionality from scratch. Bye Paolo Rizzi > > Michaël > > P.Rizzi Ag.Mobilità Ambiente a écrit : > > > What about using a Java embedded database like HSQL or Derby??? > > They already can manage in-memory / persistence of any kind of data > > and they do it very well I believe... > > You should only keep each Feature's primary keys and, > probably, it's > > bounds. > > The db engine will take care of everything else. > > Also this kind of engine usually create simple files that can be > > copied around > > so they effectively also are a binary format. > > Also, beeing used in many places, there're lots of > utilities and software > > able to use them. The Derby engine has been recently integrated into > > Java 6 (aka Mustang). > > But maybe there're other aspects that I can't see... > > Bye > > Paolo Rizzi > > > > > -------------------------------------------------------------- > ---------- > > *Da:* [EMAIL PROTECTED] per conto di > > Sunburned Surveyor > > *Inviato:* ven 30/03/2007 19.18 > > *A:* List for discussion of JPP development and use. > > *Oggetto:* Re: [JPP-Devel] Fwd: [Freegis-list] Binary > Format For Features > > > > David, > > There are some reasons why I would like to stay away from a system > > based on an existing file format like ESRI's Shapefile, or > on a system > > that requires an external database. I'm looking for a > simpler solution > > that can be used to suck in and store Features from numerous file > > formats. In essence the binary format for Features will be > similar to > > what is done currently in ESRI's shapefile format, but hopefully it > > will be a little more elegant. (For example, doesn't a > Shapefile use > > little-endian and big-endian format, while Java natively > supports only > > big-endian binary data?) > > I'm curious why the OGC didn't define a binary format for features > > when they were coming up with WKB. (Unless they did, and I > just missed > > it.) > > The Sunburned Surveyor > > > > On 3/30/07, *David Zwiers* <[EMAIL PROTECTED] > > <mailto:[EMAIL PROTECTED]>> wrote: > > > > Sounds a lot like some similar problems that have been > addressed, > > and for the most part solved. For most cases you could probably > > use an indexed shapefile reader (not in OJ) or leverage a > > container such as Pico or Hibernate (think Geotools has > some code, > > or atleast a design in this space – talk to Jody Garnett). > > > > HTH, > > > > David > > > > > -------------------------------------------------------------- > ---------- > > > > *From:* [EMAIL PROTECTED] > > <mailto:[EMAIL PROTECTED]> > > [mailto:[EMAIL PROTECTED] > > <mailto:[EMAIL PROTECTED]>] *On > > Behalf Of *Sunburned Surveyor > > *Sent:* March 30, 2007 9:16 AM > > *To:* List for discussion of JPP development and use.; > > User-friendly Desktop Internet GIS > > *Subject:* [JPP-Devel] Fwd: [Freegis-list] Binary > Format For Features > > > > > > > > ---------- Forwarded message ---------- > > From: *Sunburned Surveyor *<[EMAIL PROTECTED] > > <mailto:[EMAIL PROTECTED]>> > > Date: Mar 30, 2007 9:15 AM > > Subject: Re: [Freegis-list] Binary Format For Features > > To: Frank Warmerdam <[EMAIL PROTECTED] > <mailto:[EMAIL PROTECTED]>> > > > > Frank, > > > > Thank you for your comments. I am going to forward your > e-mail to > > the JPP Developer and GeoTools lists. Please see my > comments below. > > > > You wrote: "For instance, the header appears to be a feature > > header, rather than file > > header, is that right? Shouldn't there be some sort of > file level > > header > > or is it intended that a file is just a collection of features > > appended?" > > > > I should have explained this better. For my immediate purposes I > > only need to store a single feature in a binary file. > (This has to > > do with a system I'm trying to design for OpenJUMP that will > > allows us to access features stored in permanent memory > instead of > > in RAM.) However, I realize an important and practical > use of this > > format would be storage of multiple features in a single binary > > file. I would be more than willing to expand the format > > specification to include this. In fact, I think it > might be a good > > idea to specify a way to do both. Specify a single feature in a > > binary file, and multiple features in a single file. (Or perhaps > > you would use the same format, and would only store a single > > feature in a file when needed???) > > > > You wrote: "I'd like some options for efficient access. The most > > obvious thing needed > > for this is an index to the features relating feature id with > > offset in > > the file. This also gives the opportunity to add a > spatial index, > > perhaps > > as a seperate file, at some point in the future." > > > > I agree. This is something that would be important for > storage of > > multiple features in a single file format and should be > included. > > I left it out because I was only thinking of storing a single > > feature initially. > > > > You wrote: "This also gives the opportunity to add a spatial > > index, perhaps > > as a seperate file, at some point in the future." > > > > This would also be a good idea if we are storing > multiple features > > in a single file. An index for attribute values coulb > be stored in > > the same way. These could be future additions to the > format as you > > suggest. I do, however, think there is wisdom in storing the > > feature geomtries separately. I'll be using JTS in WKB > format, but > > there may be other types of geometry definitions that > people would > > like to use. If we can keep the storage of feature geometries > > segragrated I think the format will be more flexible. > > > > You wrote: "Is there a good place to discuss this?" > > > > I think we can have the discussion on the JPP Developer Mailing > > List for the time being. You can subscribe here: > > > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > <https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel> > > > > I'm open to other forums for discussion if you have a > suggestion. > > > > You wrote: "I think it would be great if a Java > implementation of > > this could be done > > in a way that would easily drop into Geotools without license > > conflicts. > > I think Geotools is LGPL, is that right?" > > > > I'm open to negotiation on the license for the parser. I don't > > think I'd have a problem with the LGPL, but I'd need to get a > > little more familiar with its terms. > > > > Landon (A.K.A. - The Sunburned Surveyor) > > > > > > On 3/30/07, *Frank Warmerdam* < [EMAIL PROTECTED] > > <mailto:[EMAIL PROTECTED]>> wrote: > > > > Sunburned Surveyor wrote: > >> I'm currently talking with some of the other OpenJUMP developers > > about a > >> simple binary format that we can use to store Features in > a way that > >> complements the storage of geometries in the OpenGIS > >> Well-Known-Binary WKB format. I thought this might be of > interest to > >> other open source GIS projects, so I am sending a brief message > > here. > >> I'd be willing to work with developers from other projects on a > > format > >> that could be used by other applications. > >> > >> I've put a rough draft of the format here: > >> > http://thejumppilotproject.pbwiki.com/OpenJUMP-Binary-Feature-Format > >> > >> Let me know if your interested, or if you have any suggestions. I > > hope > >> to finalize the format in the next week or two. In the next month > > or two > >> I hope to have a working parser for the binary format in Java > > that will > >> be released under the GPL through the SurveyOS SourceForge Project. > > > > Landon, > > > > I am interested in this effort. I think a binary > "simple features" > > file format > > with the potential for efficient access is interesting, and > > valuable, and I > > might like to implement support for it in OGR as well. > > > > I reviewed > > > http://thejumppilotproject.pbwiki.com/OpenJUMP-Binary-Feature-Format > > and what is there looks good, but it seems incomplete. > > > > For instance, the header appears to be a feature header, rather > > than file > > header, is that right? Shouldn't there be some sort of > file level > > header > > or is it intended that a file is just a collection of features > > appended? > > > > I'd like some options for efficient access. The most > obvious thing > > needed > > for this is an index to the features relating feature id with > > offset in > > the file. This also gives the opportunity to add a > spatial index, > > perhaps > > as a seperate file, at some point in the future. > > > > Is there a good place to discuss this? > > > > PS. I think it would be great if a Java implementation of this > > could be done > > in a way that would easily drop into Geotools without license > > conflicts. > > I think Geotools is LGPL, is that right? > > > > Best regards, > > -- > > > ---------------------------------------+---------------------- > ---------------- > > > > I set the clouds in motion - turn up | Frank Warmerdam, > > [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > > light and sound - activate the windows | > > http://pobox.com/~warmerdam <http://pobox.com/%7Ewarmerdam> > > and watch the world go round - Rush | President OSGeo, > > http://osgeo.org <http://osgeo.org/> > > > > > > > > > > > -------------------------------------------------------------- > ----------- > > Take Surveys. Earn Cash. Influence the Future of IT > > Join SourceForge.net's Techsay panel and you'll get the > chance to > > share your > > opinions on IT & business topics through brief > surveys-and earn cash > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge &CID=DEVDEV > > <http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV> > _______________________________________________ > Jump-pilot-devel mailing list > Jump-pilot-devel@lists.sourceforge.net > <mailto:Jump-pilot-devel@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > <https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel> > > >------------------------------------------------------------------------ > >------------------------------------------------------------------------- >Take Surveys. Earn Cash. Influence the Future of IT >Join SourceForge.net's Techsay panel and you'll get the chance to share your >opinions on IT & business topics through brief surveys-and earn cash >http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > >------------------------------------------------------------------------ > >_______________________________________________ >Jump-pilot-devel mailing list >Jump-pilot-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Jump-pilot-devel mailing list Jump-pilot-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Jump-pilot-devel mailing list Jump-pilot-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
