Hi Greg. Thanks for reviewing. On Tue, 2011-02-15, Greg Stein wrote: > On Tue, Feb 15, 2011 at 10:06, Julian Foad <julian.f...@wandisco.com> wrote: > >... > > * ==== Operating Procedures ==== > > * > > * The steps should be carried out in the order specified. (See rationale.) > > * > > * * To add a pristine, do the following inside an SDB txn: > > * * Add the table row, and set the refcount as desired. If a row > > * already exists, add the desired refcount to its refcount, and > > * preferably verify the old row matches the new metadata. > > * * Create the file. Creation should be fs-atomic, e.g. by moving a > > * new file into place, so as never to orphan a partial file. If a > > * file already exists, preferably leave it rather than replace it, > > * and optionally verify it matches the new metadata (e.g. length). > > * > > * * To remove a pristine, do the following inside an SDB txn: > > * * First, check refcount == 0, and abort if not. > > * * Delete the table row. > > * * Delete the file or move it away. (If not present, log a > > * consistency error but, in a release build, return success.) > > * > > * * To query a pristine's existence or SDB metadata, the reader must: > > * * Ensure no pristine-remove txn is in progress while querying it. > > What does "pristine-remove txn" mean?
It means the SDB txn defined under "To remove a pristine" above. The obvious way to ensure it's not in progress would be to do this query in an SDB txn with a suitable lock (and I think the default lock type that you get in a txn would be sufficient). > > * > > * * To read a pristine text, the reader must: > > * * Ensure no pristine-remove txn is in progress while querying and > > * opening it. > > * * Ensure the pristine text remains in the store continuously from > > * opening it for the duration of the read. (Perhaps by ensuring > > * refcount remains >= 1 and/or by cooperating with the clean-up > > * code.) > > Taking out a WC lock would prevent a pristine from being deleted, right? Yes, that would work, because of the rule below that requires an exclusive WC lock in order to purge an unreferenced text. (Originally I didn't say that here because I'm trying to keep the spec as two separate layers, first presenting this low-level spec for maintaining consistency between the files and the PRISTINE table rows in the store, and secondly under the heading "Reference Counting" a higher level spec for maintaining the references to pristines, which layers on top of this one. The latter is where WC locks are involved. However, a cross-reference here would be helpful.) Is a WC lock on the root of the WC an exclusive lock, in that it could not co-exist with any other WC lock whether on the root or deeper? I'm unsure of the semantics of WC locks. > >... > > * ==== Notes ==== > > * > > * * This procedure can leave orphaned pristine files (files without a > > * corresponding SDB row) if Subvsersion crashes. The Pristine Store > > * will still operate correctly. It should be easy to teach "svn cleanup" > > * to safely delete these. ### Do we need to define the clean-up > > * procedure here? > > * > > * * This specification is conceptually simple, but requires completing disk > > * operations within SDB transactions, which may make it too inefficient > > * in practice. An alternative specification could use the Work Queue to > > * enable more efficient processing of multiple transactions. > > Note that my initial design for the pristine inserted a row which > effectively said "we know about this pristine, but it hasn't been > written yet". The file would be put into place, then the row would get > tweaked to say "it is now there". That avoids the disk I/O within a > sqlite txn. That would indeed avoid disk I/O within any txn during "add". The "remove" operation would go through the same states in reverse, I assume. But this seems to imply an extra complexity. How would the whole thing work - locking for "add", "remove" and read operations; crash recovery? Maybe it's straightforward but I'll have to write it out in detail if you think it's worth it. I would suggest first we need to assess whether a file move within a txn is in fact a problem. > >... > > * === Invariants in a Valid WC DB State === > > * > > * * No pristine text, even if refcount == 0, will be deleted from the store > > * as long as any process holds any WC lock in this WC. > > * > > * The following conditions are always true outside of a SQL txn: > > * > > * * The 'checksum' column in each NODES table row is either NULL or > > * references a primary key in the 'pristine' table. > > * > > * * The 'refcount' column in each PRISTINE table row is equal to the > > * number of NODES table rows whose 'checksum' column references this > > * pristine row. > > ACTUAL_NODE also holds references. Shouldn't these count towards the checksum? One day, yes, but for 1.7 no. We don't use that column yet. > > * > > * The following conditions are always true > > * outside of a SQL txn, > > * when the Work Queue is empty: > > * (### ?) when no WC locks are held by any process: > > * > > * * The 'refcount' column in a PRISTINE table row equals the number of > > * NODES table rows whose 'checksum' column references that pristine > > row. > > * It may be zero. > > hmm? duplicate text? Could be... I was losing concentration at this end of the doc. > > * > > * ==== Operating Procedures ==== > > * > > * The steps should be carried out in the order specified. > > * > > * * To add a pristine text reference to the WC, obtain the text and its > > * checksum, and then do this while holding a WC lock: > > * * Add the pristine text to the Pristine Store, setting the desired > > * refcount >= 1. > > * * Add the reference(s) in the NODES table. > > * > > * * To remove a pristine text reference from the WC, do this while holding > > * a WC lock: > > * * Remove the reference(s) in the NODES table. > > * * Decrement the pristine text's 'refcount' column. > > * > > * * To purge an unreferenced pristine text, do this with an *exclusive* > > * WC lock: > > * * Check refcount == 0; skip if not. > > * * Remove it from the pristine store. > > This seems a bit duplicative of the "Operating Procedures" in the first > section. This is supposed to be the higher layer of spec, defining how to add a new pristine text to the WC ALONG WITH ADDING A REFERENCE TO IT in the NODES table, in terms of the lower layer "add this text to the store" operation. - Julian
