I'd like to discuss this topic a bit further, to try and understand things more.
On Thu, Oct 18, 2012 at 7:04 PM, Julian Foad <julianf...@btopenworld.com> wrote: > Philip Martin wrote: >> Johan Corveleyn <jcor...@gmail.com> writes: >> >>> With this new "exclusive" mode it's probably quite clear (no >>> parallellism possible at all), but because of the fuzzyness of the >>> non-exclusive behavior this leaves me wondering: "which things does >>> this exclusive mode make impossible, which were still possible under >>> non-exclusive mode"? What does one lose by enabling exclusive mode? >> >> A write operation like "svn up" opens the working copy and then executes >> a number of SQLite write transactions. SQLite will block other reads >> while a write transaction is in progress. With shared, non-exclusive, >> locking read operations can run between the write transactions. There >> is a timeout (10s for Subversion) and while the whole write operation >> may take a long time the individual write transactions are generally >> shorter that the timeout. >> >> So from an SQLite perspective most read operations may run during a >> write operation but may block before proceeding. From a Subversion >> perspective the read operation will still fail to run if there is a >> workqueue item. So in 1.7 read operations only work if they run between >> write transactions when the workqueue is empty. >> >> With 1.8 exclusive locking the SQLite write lock is not dropped after >> the first write transaction but it help until the whole write operation >> is over and the database is closed. No read operations can run while >> the SQLite write lock is held. >> > Would it be useful to copy this explanation somewhere more discoverable -- > somewhere in the source tree, or linked from the release notes page? > > - Julian +1, some clear documentation around this would be quite welcome I think, at least for developers of svn (GUI) clients, IDE plugins, wrappers, scripts, ... But first, some more questions about the "old" non-exclusive mode: - This 10 second timeout, for a read operation that is blocked by some write transaction: is this implemented somewhere in the svn source code (if so, where?), or is this some configuration setting for the sqlite connection or something like that? - If a read operation (with a "shared lock" I presume) squeezes in between write transactions of a large write operation, and that read takes a long time, I suppose the next write transaction of that large operation will also block, with the same timeout? - Does this also work for two writers? Or do they block out each other with the working copy lock (wc_locks table)? - With your remark about the workqueue making the read fail, is this how a read operation may proceed? * Try to get a shared sqlite lock for reading. Block for max. 10s if the db is locked for writing (error out if timed out). * When db reading is allowed (shared lock acquired), first read the work_queue. If it's not empty: error out. * Finally, read what we want to read, and get out. (and the above sequence may be part of a larger read operation that e.g. does a query per directory, so this sequence may be repeated several times, perhaps interleaved with some large write operation that intermittently takes a write lock for things done per directory or somesuch) - All things considered, the "old" non-exclusive mode sounds quite non-deterministic. I mean, isn't this very hard for large GUI clients, IDE plugins, ... to handle this? Then Bert said: > Note that with the current state we could remove the limitation of being > able to open the database while there are workqueue operations. In 1.7 we > still have workqueue operations that change the database which makes this > unsafe (like the old loggy code was), but for 1.8 these cases are > removed/resolved. > > The remaining problem with enabling that, is that we should introduce a new > check point for outstanding operations. Maybe at the point where we obtain a > working copy lock? If that could be implemented, I guess the non-exclusive mode would become more stable. If our code takes care of splitting up large write operations into not-too-large individual transactions, then it sounds like concurrency (one writer, many readers) should work quite well. I'm wondering if we can more clearly define the two different modes, with accurately described behavior: the non-exclusive mode which maximizes the ability to do things concurrently (at the cost of some performance), and the exclusive mode which maximizes single-threaded performance, but allows no concurrency at all. Are there things that can be done to make the non-exclusive mode more concurrency-friendly (like Bert's suggestion about the workqueue limitation)? -- Johan
