Shapefile readers often use the rownum in the file as the FID -- you need to define a standard for including a surrogate key column (Oracle this is the PK, SDE/PGDB it's the OID).
Probably need a metadata portion and content portion for the file format. Have you thought about binary GML? There was a paper on this a couple years back. David -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Rahkonen Jukka Sent: April 4, 2007 12:18 AM To: List for discussion of JPP development and use. Subject: Re: [JPP-Devel] Questions about binary format for features... Hi, When planning this new binary feature format, could you try to think if it could be made to support transactions (delete, update, insert) through transactional Web Feature Service? There has been discussion on Geoserver user list that with shapefiles are not actually suitable for WFS-T backend. The problem seems to be somewhere how the shapefiles and WFS server handle feature ID's. They must of course match to make transactions succeed. Don't ask me what it would need from the binary format. I guess that at least an extremely logical fid generation system, so that the WFS service can be made to generate correct fid's for new features, and that the fid's are not changing every now and then so the updates won't go to wrong features. Excerpt from Geoserver discussion: > The ID's for shapefile features seem to be autogenerated: featurename + > "." + index. So, while features are inserted in a shapefile, it is very > easy to "generate" a new ID and return it (just the number of elements + > 1). However, this method will fail hopelessly in case features are > deleted. Then the entire numbering scheme will break. So, mapping (to an > existing property?) would be a better solution, but who is responsible > for generating the new ID? It opens up a lot of issues, and they're not > efficient enough. >> The problem with shapefiles ( and any other file based data source ) >> is that they are not meant to do transactions Regards, -Jukka Rahkonen- ________________________________ Lähettäjä: [EMAIL PROTECTED] puolesta: Sunburned Surveyor Lähetetty: ke 4.4.2007 7:05 Vastaanottaja: Frank Warmerdam Kopio: List for discussion of JPP development and use. Aihe: Re: [JPP-Devel] Questions about binary format for features... Frank, I've done some more work on BOFF. (Binary Open Feature Format). I've modified the rough draft of the file format to support multiple features and feature geometries. I've also defined a default geometry file that used WKB for geometry definitions. I've added a Feature-To-Geometry Map file and Geometry-To-Feature file that will allow us to support multiple geometries for each feature record, or multiple features for each geometry record. I've also added index files for the feature and geometry files. You can find all the information on the format here: http://thejumppilotproject.pbwiki.com/Binary%20Open%20Feature%20Format I will begin work on the reader/writer for the .fea file that I hope to use in my FeatureCache for OpenJUMP. I can then work on a reader/writer for the other parts of the file format. As I work on the reader/writer I'm sure I will discover some changes that need to be made to the file format. I will make these changes as I work. When I've got a working writer/reader and a final file format I'll write a spec for the format that will be published at the SurveyOS Project. I'll also release the Java code for the reader/writer there. The Sunburned Surveyor On 4/2/07, Sunburned Surveyor <[EMAIL PROTECTED]> wrote: Frank, I think we can work together towards a solution that will work for the both of us. I'll revisit the specification I posted earlier and make some changes that it supports multiple features. I will also sketch up a format for a binary index. For my needs in OpenJUMP I can just use the same specification, only my files will only contain a single feature. You wrote: "I'm essentially interested in a replacement format for Shapefiles with none of their limitations for field types and with a geometry that is directly the simple features geometry model instead of a relatively close analog." This is basically my goal as well. However, I'd like to keep the feature geometries in a separate file so that we can "swap" geometry models if necessary. We can make the default geometry format OGC WKB, and I'll write the Java parser/writer to work using that format for feature geometries. The features will store only an identifier for there respective geometries. This will allow different geometry models to be supported. I'm thinking a "binary open feature format" will have 4 separate binary files. [1] The .fea file will contain the feautures/feature attributes. [2] The .find or .fin will contain the index for the features in the .fea file. [3] The .geom or .geo file will contain the featue geometries. [4] The .gind or .gin will contain the index for the features geometries stored in the .geom file. Would it be too much to add a fifth file for some simple metadata? You wrote: " I'm essentially interested in a replacement format for Shapefiles with none of their limitations for field types..." Would there be a way to allow addition of custom data types for attributes? I'm not sure exactlty how this would work. We could have a text file where developers could match an ID stored as the attribute type with the specification for the format. If we did this I could make the parser pluggable so feature attributes with a custom data type could be parsed without the need for recompilation of the parser. But perhaps this is too complex and we should stick with something simple for now. Let me know what you think. Landon On 4/2/07, Frank Warmerdam <[EMAIL PROTECTED] > wrote: Sunburned Surveyor wrote: > This message is directed mostly at Frank, but I included the JPP mailing > list in case other JPP Developers want to comment. > > I've been thinking some more about a binary format for the exchange of > features, and I have been taking a look at the API for Java's > DataInputStream class and DataOutputStream class. They contain methods > that can read and write primitive Java datatypes. They can even read and > write String objects as seqences of Unicode character values with a > character count. At first glance it looked like this was everything we > needed to create a parser and writer for a binary feature exchance format. > > However, I'm concerned about how interoperability with other programming > languages of a binary file written with these Java classes. For example, > if you create a file with a computer that has a 64-bit microprocessor > does the writeInt() method write an integer to the file using 4 bytes or > 8 bytes? (The API documentation says it only uses 4 bytes.) I also think > we would want to stay away from the writeUTF() and readUTF() methods, > since it sounds like they write a funky character encoding that other > programming languages might have trouble with. I think we'll want to > write our own character counter byte and the individual unicode > character values with writeChar() or writeChars() method instead. > > Do you guys have any comments on this? Landon, UTF-8 is fine with other languages, and is essentially ASCII when no unusual characters are used. For binary types it is important to be very precise about their sizes the the format specification. You mentioned earlier that you current interest is in the a "single feature" format, which is of little interest to me. I'm essentially interested in a replacement format for Shapefiles with none of their limitations for field types and with a geometry that is directly the simple features geometry model instead of a relatively close analog. It is also important to me that it be indexable, though I'm happy to handle the indexes as separate files that are specified later. Best regards, -- ---------------------------------------+-------------------------------------- I set the clouds in motion - turn up | Frank Warmerdam, [EMAIL PROTECTED] light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush | President OSGeo, http://osgeo.org <http://osgeo.org/> ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Jump-pilot-devel mailing list Jump-pilot-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Jump-pilot-devel mailing list Jump-pilot-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel
