Would you be willing to move the CF stuff on top of GDBM? It's a much better fit for it, and will give you everything you do with SQLite today, without the extra overhead (SQLite is also not very efficient if you open multiple databases).
Regards, Elias On 11 July 2014 12:21, David B. Lamkins <dlamk...@gmail.com> wrote: > On Thu, 2014-07-10 at 18:52 -0500, Blake McBride wrote: > > > > > > > The component file system is for APL-only applications. The databse > > API is to communicate with the rest of the world. It depends on what > > you need. > > > > and... > > > > > The problem is that you are creating a component file system that > > _depends_ on features of SQLite and is _needlessly_ dependent on those > > features. I understand it works on other databases, but it is an > > unreasonable burden on those systems. You could just as easily create > > the same system that doesn't depend on particulars of SQLite. That > > would work everywhere in a way that is reasonable to any of the > > systems. > > > > Ahhh... OK. Perhaps I've had an "aha!" moment. :) > > If I understand you correctly, generalizing the current component file > implementation to work with databases other than SQLite is problematic > for all of the reasons you've discussed: poor generality, poor use of > resources, and - in general - a poor fit to SQL. > > Therefore, here's my proposal: > > I'll drop all further work on generalizing the current component file > API to work with SQL databases. IOW, the component file system will > remain - literally - a component *file* system. The quick patch that I > worked up last night will never become part of my code. > > Programmers who want a way to persist data for an APL-only single-user > application and a shallow learning curve and no sysadmin or dbadmin > concerns will (I hope) appreciate and use the simple component-file API. > Programmers who need the functionality of a real database - and all that > it offers in terms of access control, concurrency and remote access - > can use lib_sql with a PostgreSQL backend. > > The presence of SQLite will remain an incidental implementation detail > of the component file library. All further concerns specific to the use > of SQLite as an implementation vehicle will be treated as out-of-scope > for the project. > > The current component file code can serve as a reference implementation > to validate the design choices not implicitly required by the ISO APL > Annex. There are a number of such choices that I was forced to make in > the absence of detailed guidance from the spec. For example: the choice > to ensure that component IDs assigned by CF_APPEND are always > nondecreasing for a given component file. There are many other detailed > choices that had to be made. If you have a concern with how I've > implemented the API to meet the spec, we can work out the details > together. > > Meanwhile, if someone feels that it's important to layer a component > file API on top of lib_sql *and* allow a choice of database backends, > they may either start from scratch or fork the existing code. > Personally, I don't see the need for this particular layering of > abstractions; for anything more complex than an integer-indexed file of > APL objects, I'd prefer to work directly with lib_sql. But if someone > else sees value in the layered approach, by all means pursue it as a > separate project. > > Meanwhile, I'll continue working on design sketches for a component file > system that doesn't build upon SQLite. The primary reason for doing so > will be to increase performance while not requiring use of explicit > transactions to reduce single-record insert times below the 100ms mark. > Of course the content of the component file will need to remain reliable > in the face of software and hardware failures; that's one of the > implementation attributes that I got "for free" by choosing SQLite as a > short-term implementation tactic. An alternative implementation will be > challenging, no doubt, but worth the effort in the long run. Whether > such an implementation should be done as a native library or as a hybrid > is not clear to me at this time. > > The take-away from all of this should be the following: if you use the > existing component file library (and I certainly hope that you will do > so in cases where it best serves your needs), you can be assured that > any future work that I do with the component file library will preserve > your data. Any change in the file format will be accompanied by a > utility workspace to migrate your existing component files. > > > > > This is the most I've been involved with a community project. Fun! > > > > Indeed. :) > > Also, it's worth once again acknowledging Jüergen's work in making all > of this possible. All of these little libraries and utilities are > penny-ante stuff compared to the core interpreter. I just want to say, > again, that I am grateful for the ability to work with a full-featured > open-source APL interpreter for the first time ever. If you look at the > history of free APL systems, they've traditionally been thin on the > ground, undermaintained and not very interesting. > > A lot of technologies have developed or matured in the past 30 years to > get to the point where we are today with free APL: Unicode, Linux, > Emacs, gcc, fast hires video and the x86 hardware platform being > foremost among a plethora of candidates. None of us would be here on > this mailing list today, however, had Jüergen not put in an immense > amount of design and implementation effort to create GNU APL. I > sincerely appreciate his continuing support and involvement. > >
