Re: wc_db performance (was: wc_db API discussion)

2011-03-17 Thread Paul Burba
On Tue, Mar 15, 2011 at 6:22 PM, Paul Burba wrote: > On Mon, Mar 14, 2011 at 2:36 PM, Hyrum K Wright wrote: >> On Sat, Mar 12, 2011 at 6:47 AM, Stefan Sperling wrote: >>> On Fri, Mar 11, 2011 at 10:43:46PM -0500, Greg Stein wrote: 2011/3/11 Branko Čibej : >... > For the second tas

Re: wc_db performance (was: wc_db API discussion)

2011-03-15 Thread Paul Burba
On Mon, Mar 14, 2011 at 2:36 PM, Hyrum K Wright wrote: > On Sat, Mar 12, 2011 at 6:47 AM, Stefan Sperling wrote: >> On Fri, Mar 11, 2011 at 10:43:46PM -0500, Greg Stein wrote: >>> 2011/3/11 Branko Čibej : >>> >... >>> > For the second task, I think the first order of business is to change >>> > t

Re: wc_db performance

2011-03-15 Thread Philip Martin
Greg Stein writes: > Why can't we simply put restrictions on those callbacks? I'm unclear > on this part. We seem to be unsure if such a restriction is a good idea. Suppose a client calls proplist, what can it do when it sees a property of interest? Can it delete the property? Modify the valu

Re: wc_db performance

2011-03-15 Thread Philip Martin
Branko Čibej writes: > On 15.03.2011 15:34, Philip Martin wrote: >> So with the temporary table approach the callback really has to use a >> separate database connection to read/write the database from within the >> callback. >> >> However I think that is also the case if we were to avoid the tab

Re: wc_db performance

2011-03-15 Thread Greg Stein
On Tue, Mar 15, 2011 at 10:34, Philip Martin wrote: >... > So with the temporary table approach the callback really has to use a > separate database connection to read/write the database from within the > callback. > > However I think that is also the case if we were to avoid the table and > simpl

Re: wc_db performance

2011-03-15 Thread Branko Čibej
On 15.03.2011 15:34, Philip Martin wrote: > Philip Martin writes: > >> Branko Čibej writes: >> >>> The temporary table is: >>> >>> * in a different database, so the read lock on it does not affect >>> the main wc-db; >>> * per-connection, so even the same process using a different >

Re: wc_db performance

2011-03-15 Thread Philip Martin
Philip Martin writes: > Branko Čibej writes: > >> The temporary table is: >> >> * in a different database, so the read lock on it does not affect >> the main wc-db; >> * per-connection, so even the same process using a different >> database connection will not even see that t

Re: wc_db performance

2011-03-15 Thread Philip Martin
Branko Čibej writes: > The temporary table is: > > * in a different database, so the read lock on it does not affect > the main wc-db; > * per-connection, so even the same process using a different > database connection will not even see that temporary table. OK, so the proce

Re: wc_db performance

2011-03-15 Thread Stefan Sperling
On Tue, Mar 15, 2011 at 12:52:42PM +, Philip Martin wrote: > Yes, by the time we invoke the callbacks we have a read-lock. If we > simply ran a single transaction and made callbacks within the > transaction then it would have the same effect. I don't see what the > temporary table achieves.

Re: wc_db performance

2011-03-15 Thread Branko Čibej
On 15.03.2011 11:26, Philip Martin wrote: > Branko Čibej writes: > >> The restriction that you may not invoke a callback from within a sqlite >> transaction remains. That's what the temporary results tables are for -- >> they live outside transactions on the main wc-db. > I don't understand the te

Re: wc_db performance

2011-03-15 Thread Philip Martin
Stefan Sperling writes: > On Tue, Mar 15, 2011 at 10:26:15AM +, Philip Martin wrote: >> Branko Čibej writes: >> >> > The restriction that you may not invoke a callback from within a sqlite >> > transaction remains. That's what the temporary results tables are for -- >> > they live outside t

Re: wc_db performance

2011-03-15 Thread Stefan Sperling
On Tue, Mar 15, 2011 at 10:26:15AM +, Philip Martin wrote: > Branko Čibej writes: > > > The restriction that you may not invoke a callback from within a sqlite > > transaction remains. That's what the temporary results tables are for -- > > they live outside transactions on the main wc-db. >

Re: wc_db performance

2011-03-15 Thread Philip Martin
Branko Čibej writes: > The restriction that you may not invoke a callback from within a sqlite > transaction remains. That's what the temporary results tables are for -- > they live outside transactions on the main wc-db. I don't understand the temporary table approach. Taking the temp__node_pr

Re: wc_db performance

2011-03-14 Thread Branko Čibej
On 14.03.2011 17:13, Stefan Sperling wrote: > On Mon, Mar 14, 2011 at 05:05:39PM +0100, Stefan Sperling wrote: >> The concern seems to be that invoking a notification callback could >> attempt to modify DB state, correct? Which would deadlock if you invoke >> the callback after starting a transacti

Re: wc_db performance (was: wc_db API discussion)

2011-03-14 Thread Hyrum K Wright
On Sat, Mar 12, 2011 at 6:47 AM, Stefan Sperling wrote: > On Fri, Mar 11, 2011 at 10:43:46PM -0500, Greg Stein wrote: >> 2011/3/11 Branko Čibej : >> >... >> > For the second task, I think the first order of business is to change >> > the wc-db tree crawler to do one query instead of zillions, or a

Re: wc_db performance

2011-03-14 Thread Stefan Sperling
On Mon, Mar 14, 2011 at 05:05:39PM +0100, Stefan Sperling wrote: > The concern seems to be that invoking a notification callback could > attempt to modify DB state, correct? Which would deadlock if you invoke > the callback after starting a transaction to tweak changelist information > but before t

Re: wc_db performance

2011-03-14 Thread Stefan Sperling
On Mon, Mar 14, 2011 at 10:53:48AM -0400, C. Michael Pilato wrote: > > On 12.03.2011 13:47, Stefan Sperling wrote: > >> For others who want to jump in and help, here is a list of places > >> where the node walker is still being used. I'm not sure if we can > >> eliminate it everywhere before releas

Re: wc_db performance

2011-03-14 Thread C. Michael Pilato
> On 12.03.2011 13:47, Stefan Sperling wrote: >> For others who want to jump in and help, here is a list of places >> where the node walker is still being used. I'm not sure if we can >> eliminate it everywhere before release, but each of these should >> be looked at to see whether we can use an al

Re: wc_db performance (was: wc_db API discussion)

2011-03-13 Thread Paul Burba
On Sat, Mar 12, 2011 at 7:47 AM, Stefan Sperling wrote: > On Fri, Mar 11, 2011 at 10:43:46PM -0500, Greg Stein wrote: >> 2011/3/11 Branko Čibej : >> >... >> > For the second task, I think the first order of business is to change >> > the wc-db tree crawler to do one query instead of zillions, or a

Re: wc_db performance (was: wc_db API discussion)

2011-03-12 Thread Branko Čibej
Thanks, Stefan, this looks very promising. I won't make any promises about grabbing some of that code, given how much time I have on my hands ... but I'm glad that at least the proplist bits can be reused. -- Brane On 12.03.2011 13:47, Stefan Sperling wrote: > On Fri, Mar 11, 2011 at 10:43:46PM -

Re: wc_db performance (was: wc_db API discussion)

2011-03-12 Thread Stefan Sperling
On Fri, Mar 11, 2011 at 10:43:46PM -0500, Greg Stein wrote: > 2011/3/11 Branko Čibej : > >... > > For the second task, I think the first order of business is to change > > the wc-db tree crawler to do one query instead of zillions, or at least, > > where several queries are required, to do them all

Re: wc_db performance (was: wc_db API discussion)

2011-03-11 Thread Greg Stein
2011/3/11 Branko Čibej : >... > For the second task, I think the first order of business is to change > the wc-db tree crawler to do one query instead of zillions, or at least, > where several queries are required, to do them all in one transaction. stsp has been working this recently. Killing the

Re: wc_db performance (was: wc_db API discussion)

2011-03-11 Thread Greg Stein
2011/3/11 Branko Čibej : > On 12.03.2011 01:29, Greg Stein wrote: >... >> So. Not a premature optimization, but a design choice. > > Six of one, half a dozen of the other. ACTUAL is just another op-depth > with a few extra attributes, so let's compromise and call it an > implementation choice, rath

Re: wc_db performance (was: wc_db API discussion)

2011-03-11 Thread Branko Čibej
On 12.03.2011 01:11, Mark Phippard wrote: > I am glad you sent this because I was getting ready to send an email > to see if anyone is looking into the suggestions you have made here. > I think we have to get this work done soon. We cannot release with > performance like it is. How do we define t

Re: wc_db performance (was: wc_db API discussion)

2011-03-11 Thread Branko Čibej
On 12.03.2011 01:29, Greg Stein wrote: > 2011/3/11 Branko Čibej : >> This comment is somewhat orthogonal to the API discussions, but as I've >> noted before ... after my relatively brief sojourn in wc-db, I came to >> the conclusion that having separate NODES and ACTUAL_NODE tables is >> going to b

wc_db performance (was: wc_db API discussion)

2011-03-11 Thread Greg Stein
2011/3/11 Branko Čibej : > This comment is somewhat orthogonal to the API discussions, but as I've > noted before ... after my relatively brief sojourn in wc-db, I came to > the conclusion that having separate NODES and ACTUAL_NODE tables is > going to be a perpetual impediment to really speeding u