Re: [PATCH] svnadmin build-repcache command

2020-04-01 Thread Nathan Hartman
On Wed, Apr 1, 2020 at 3:54 PM Stefan Sperling wrote: > On Wed, Apr 01, 2020 at 10:50:12PM +0300, Denis Kovalchuk wrote: > > Because the root cause of the problem was unclear to me I investigated > it. > > > > svn_fs_fs__fixup_expanded_size() function exists due to issue SVN-4554 > [1]. > > In th

Re: [PATCH] svnadmin build-repcache command

2020-04-01 Thread Stefan Sperling
On Wed, Apr 01, 2020 at 10:50:12PM +0300, Denis Kovalchuk wrote: > > > This seems to fix it. > > > > > > I don't understand why this is necessary but the regular code path for > > > access to node revision data in fsfs also seems to apply this always. > > > > > > Can an fsfs expert confirm? > > > >

Re: [PATCH] svnadmin build-repcache command

2020-04-01 Thread Denis Kovalchuk
> > This seems to fix it. > > > > I don't understand why this is necessary but the regular code path for > > access to node revision data in fsfs also seems to apply this always. > > > > Can an fsfs expert confirm? > > Since this fix looks reasonable to me and we're trying to move the release > pro

Re: [PATCH] svnadmin build-repcache command

2020-04-01 Thread Stefan Sperling
On Wed, Apr 01, 2020 at 06:40:27PM +0200, Stefan Sperling wrote: > This seems to fix it. > > I don't understand why this is necessary but the regular code path for > access to node revision data in fsfs also seems to apply this always. > > Can an fsfs expert confirm? Since this fix looks reasona

Re: [PATCH] svnadmin build-repcache command

2020-04-01 Thread Stefan Sperling
On Wed, Apr 01, 2020 at 06:25:12PM +0200, Stefan Sperling wrote: > On Wed, Apr 01, 2020 at 11:45:04AM -0400, Nathan Hartman wrote: > > On Wed, Apr 1, 2020 at 8:05 AM Daniel Shahaf > > wrote: > > > The failure is 100% reproducible with «./svnadmin_tests.py > > > --fs-type=fsfs --fsfs-version=6 74»

Re: [PATCH] svnadmin build-repcache command

2020-04-01 Thread Stefan Sperling
On Wed, Apr 01, 2020 at 11:45:04AM -0400, Nathan Hartman wrote: > On Wed, Apr 1, 2020 at 8:05 AM Daniel Shahaf wrote: > > The failure is 100% reproducible with «./svnadmin_tests.py > > --fs-type=fsfs --fsfs-version=6 74» on Linux. > > I'm experimenting with this also. So far I have nothing defini

Re: [PATCH] svnadmin build-repcache command

2020-04-01 Thread Nathan Hartman
On Wed, Apr 1, 2020 at 8:05 AM Daniel Shahaf wrote: > The failure is 100% reproducible with «./svnadmin_tests.py > --fs-type=fsfs --fsfs-version=6 74» on Linux. I'm experimenting with this also. So far I have nothing definitive but I can confirm that it is 100% reproducible on Linux with the abov

Re: [PATCH] svnadmin build-repcache command

2020-04-01 Thread Daniel Shahaf
Daniel Shahaf wrote on Wed, 01 Apr 2020 10:26 +: > Stefan Sperling wrote on Wed, 01 Apr 2020 12:18 +0200: > > On Wed, Apr 01, 2020 at 09:59:41AM +, Daniel Shahaf wrote: > > > Stefan Sperling wrote on Wed, 01 Apr 2020 11:13 +0200: > > > > https://ci.apache.org/builders/svn-x64-macosx-f

Re: [PATCH] svnadmin build-repcache command

2020-04-01 Thread Stefan Sperling
On Wed, Apr 01, 2020 at 10:26:40AM +, Daniel Shahaf wrote: > If you haven't already, re-run that builder using subversion-bot to > check if the bug is reproducible. Can you please do this yourself when you find time for it? You provide a lot of good ideas, while at the same time asking others

Re: [PATCH] svnadmin build-repcache command

2020-04-01 Thread Daniel Shahaf
Stefan Sperling wrote on Wed, 01 Apr 2020 12:18 +0200: > On Wed, Apr 01, 2020 at 09:59:41AM +, Daniel Shahaf wrote: > > Stefan Sperling wrote on Wed, 01 Apr 2020 11:13 +0200: > > > https://ci.apache.org/builders/svn-x64-macosx-fsfs-v6/builds/2941/steps/Test%20ra_local%2Bfsfs-v6/logs/faillog >

Re: [PATCH] svnadmin build-repcache command

2020-04-01 Thread Stefan Sperling
On Wed, Apr 01, 2020 at 09:59:41AM +, Daniel Shahaf wrote: > Stefan Sperling wrote on Wed, 01 Apr 2020 11:13 +0200: > > https://ci.apache.org/builders/svn-x64-macosx-fsfs-v6/builds/2941/steps/Test%20ra_local%2Bfsfs-v6/logs/faillog > > > > The svnadmin build-repcache test runs into a checksum m

Re: [PATCH] svnadmin build-repcache command

2020-04-01 Thread Denis Kovalchuk
> There's another test failure related to this patch. > > This time on Mac OS X: > https://ci.apache.org/builders/svn-x64-macosx-fsfs-v6/builds/2941/steps/Test%20ra_local%2Bfsfs-v6/logs/faillog > > The svnadmin build-repcache test runs into a checksum mismatch. > > Any ideas? I started to handle t

Re: [PATCH] svnadmin build-repcache command

2020-04-01 Thread Daniel Shahaf
Stefan Sperling wrote on Wed, 01 Apr 2020 11:13 +0200: > https://ci.apache.org/builders/svn-x64-macosx-fsfs-v6/builds/2941/steps/Test%20ra_local%2Bfsfs-v6/logs/faillog > > The svnadmin build-repcache test runs into a checksum mismatch. > > Any ideas? Note the "actual" checksum is the checksum of

Re: [PATCH] svnadmin build-repcache command

2020-04-01 Thread Stefan Sperling
On Tue, Mar 31, 2020 at 12:48:21PM +0200, Stefan Sperling wrote: > On Tue, Mar 31, 2020 at 01:35:54PM +0300, Denis Kovalchuk wrote: > > > This has caused a test failure on the windows buildbot: > > > > > > https://ci.apache.org/builders/svn-windows-local/builds/3392/steps/Test%20fsfs%2Blocal/logs/f

Re: [PATCH] svnadmin build-repcache command

2020-03-31 Thread Stefan Sperling
On Tue, Mar 31, 2020 at 01:35:54PM +0300, Denis Kovalchuk wrote: > > This has caused a test failure on the windows buildbot: > > > > https://ci.apache.org/builders/svn-windows-local/builds/3392/steps/Test%20fsfs%2Blocal/logs/faillog > > > > Can you please check what we should do about that? > > I

Re: [PATCH] svnadmin build-repcache command

2020-03-31 Thread Denis Kovalchuk
> This has caused a test failure on the windows buildbot: > > https://ci.apache.org/builders/svn-windows-local/builds/3392/steps/Test%20fsfs%2Blocal/logs/faillog > > Can you please check what we should do about that? I missed SkipUnless test decorator. The build_repcache() test should be skipped i

Re: [PATCH] svnadmin build-repcache command

2020-03-31 Thread Stefan Sperling
On Tue, Mar 31, 2020 at 10:55:07AM +0200, Stefan Sperling wrote: > On Tue, Mar 24, 2020 at 07:53:21PM +0300, Denis Kovalchuk wrote: > > Introduce 'svnadmin build-repcache' command. > > Committed in r1875921. Thanks! This has caused a test failure on the windows buildbot: https://ci.apache.org/bu

Re: [PATCH] svnadmin build-repcache command

2020-03-31 Thread Stefan Sperling
On Tue, Mar 24, 2020 at 07:53:21PM +0300, Denis Kovalchuk wrote: > Introduce 'svnadmin build-repcache' command. Committed in r1875921. Thanks! I will now nominate both patches for backport to 1.14.x.

Re: [PATCH] rep-cache.db insert optimization (was: Re: [PATCH] svnadmin build-repcache command)

2020-03-31 Thread Stefan Sperling
On Mon, Mar 30, 2020 at 11:40:38PM +0300, Denis Kovalchuk wrote: > rep-cache.db insert optimization. Committed in r1875918. Thank you!

Re: [PATCH] svnadmin build-repcache command

2020-03-30 Thread Daniel Shahaf
Denis Kovalchuk wrote on Mon, 30 Mar 2020 23:48 +0300: > Speaking of the possible starvations, I would like to note, that since > the current 'build-repcache' implementation uses one SQL transaction > per revision, it seems to me that it should behave similarly to > concurrent commits, where a rep-

Re: [PATCH] rep-cache.db insert optimization (was: Re: [PATCH] svnadmin build-repcache command)

2020-03-30 Thread Daniel Shahaf
Denis Kovalchuk wrote on Mon, 30 Mar 2020 23:40 +0300: > > > > We do have an extra call when a duplicate rep is attempted to be > > inserted into the rep-cache. When does that happen? I know it will > > happen when called from build-repcache; what other use-cases trigger the > > codepath you chan

Re: [PATCH] svnadmin build-repcache command

2020-03-30 Thread Denis Kovalchuk
> > Denis, would you like to look into this? It looks like an easy, > localized fix, and your build-repcache patch will benefit from it, too. Thank you for the suggestion. I'll keep it in mind, but as I already have two outgoing patches, I would rather try not to add one more just now, to keep

Re: [PATCH] rep-cache.db insert optimization (was: Re: [PATCH] svnadmin build-repcache command)

2020-03-30 Thread Denis Kovalchuk
> > We do have an extra call when a duplicate rep is attempted to be > inserted into the rep-cache. When does that happen? I know it will > happen when called from build-repcache; what other use-cases trigger the > codepath you changed? Committing two files with equal contents > sequentially doe

Re: [PATCH] svnadmin build-repcache command

2020-03-28 Thread Daniel Shahaf
Daniel Shahaf wrote on Sat, 28 Mar 2020 21:54 +: > Stefan Sperling wrote on Sat, 28 Mar 2020 11:32 +0100: > > On Fri, Mar 27, 2020 at 10:58:48PM +, Daniel Shahaf wrote: > > > Well, I guess the documentation, once written, should point out that if > > > any single revision build-repcache o

Re: [PATCH] svnadmin build-repcache command

2020-03-28 Thread Daniel Shahaf
Stefan Sperling wrote on Sat, 28 Mar 2020 11:32 +0100: > On Fri, Mar 27, 2020 at 10:58:48PM +, Daniel Shahaf wrote: > > Stefan Sperling wrote on Fri, 27 Mar 2020 14:41 +0100: > > > On Thu, Mar 26, 2020 at 08:32:38PM +, Daniel Shahaf wrote: > > > > What I'm worried about is that we'll ap

Setting up a cross-minor-version over-the-wire interoperability buildbot (was: Re: [PATCH] svnadmin build-repcache command)

2020-03-28 Thread Daniel Shahaf
Stefan Sperling wrote on Sat, 28 Mar 2020 11:32 +0100: > On Fri, Mar 27, 2020 at 10:58:48PM +, Daniel Shahaf wrote: > > This reminds me that we don't have good test coverage for on-disk and > > over-the-wire interoperability with older versions (other than > > upgrade_tests_data/, which is wc-o

Re: [PATCH] svnadmin build-repcache command

2020-03-28 Thread Stefan Sperling
On Fri, Mar 27, 2020 at 10:58:48PM +, Daniel Shahaf wrote: > Stefan Sperling wrote on Fri, 27 Mar 2020 14:41 +0100: > > On Thu, Mar 26, 2020 at 08:32:38PM +, Daniel Shahaf wrote: > > > What I'm worried about is that we'll apply the "backwards compatibility > > > until 2.0" promise to someth

Re: [PATCH] svnadmin build-repcache command

2020-03-27 Thread Daniel Shahaf
Stefan Sperling wrote on Fri, 27 Mar 2020 14:41 +0100: > On Thu, Mar 26, 2020 at 08:32:38PM +, Daniel Shahaf wrote: > > What I'm worried about is that we'll apply the "backwards compatibility > > until 2.0" promise to something we shouldn't. As I said, issues of > > this nature can't generally

Re: [PATCH] svnadmin build-repcache command

2020-03-27 Thread Stefan Sperling
On Thu, Mar 26, 2020 at 08:32:38PM +, Daniel Shahaf wrote: > What I'm worried about is that we'll apply the "backwards compatibility > until 2.0" promise to something we shouldn't. As I said, issues of > this nature can't generally be found by tests; they can only be found > by code review. I

Re: [PATCH] svnadmin build-repcache command

2020-03-26 Thread Daniel Shahaf
Stefan Sperling wrote on Thu, 26 Mar 2020 10:18 +0100: > On Wed, Mar 25, 2020 at 09:56:23PM +, Daniel Shahaf wrote: > > Nathan Hartman wrote on Wed, 25 Mar 2020 17:30 -0400: > > > FYI, I tested 1.14.0-rc1 today, but I'm not rushing to sign (yet) > > > because I'm waiting to see if we'll get D

Re: [PATCH] rep-cache.db insert optimization (was: Re: [PATCH] svnadmin build-repcache command)

2020-03-26 Thread Daniel Shahaf
> > Yes, INSERT OR IGNORE supported by all versions. I plan to start a separate > > thread about this topic. > > > > As mentioned above, I think we can optimize the insert statement for the > case of a constraint violation. Now we use the 'FAIL' conflict > resolution algorithm [1] and explicitly

[PATCH] rep-cache.db insert optimization (was: Re: [PATCH] svnadmin build-repcache command)

2020-03-26 Thread Denis Kovalchuk
> > Well, maybe. The code _appears_ to take a roundabout approach, but I'm >> not sure that's actually a bug: there might be a difference between >> «INSERT OR IGNORE» on the one hand, and the current behaviour («INSERT >> OR FAIL» plus a C check) on the other. >> >> Is «INSERT OR IGNORE» supporte

Re: [PATCH] svnadmin build-repcache command

2020-03-26 Thread Stefan Sperling
On Wed, Mar 25, 2020 at 09:56:23PM +, Daniel Shahaf wrote: > Nathan Hartman wrote on Wed, 25 Mar 2020 17:30 -0400: > > FYI, I tested 1.14.0-rc1 today, but I'm not rushing to sign (yet) > > because I'm waiting to see if we'll get Denis's patch (or a later > > version of it) into 1.14. I don't s

Re: [PATCH] svnadmin build-repcache command

2020-03-25 Thread Daniel Shahaf
Nathan Hartman wrote on Wed, 25 Mar 2020 17:30 -0400: > On Wed, Mar 25, 2020 at 9:17 AM Stefan Sperling wrote: > > I rolled 1.14.0-rc1 this morning, and I didn't notice that you had > > posted this updated patch yesterday. Sorry about the bad timing. > > The 1.14.0-rc1 release does not have your p

Re: [PATCH] svnadmin build-repcache command

2020-03-25 Thread Daniel Shahaf
> > I'm proposing to modify fs_has_rep_sharing() to return True when running > > on FSFS with rep-sharing enabled and to return False on other backends. > > That's separate from your test's decorators. > > > > I think that this might be unrelated to this patch, but also I may be > misunderstandi

Re: [PATCH] svnadmin build-repcache command

2020-03-25 Thread Nathan Hartman
On Wed, Mar 25, 2020 at 9:17 AM Stefan Sperling wrote: > I rolled 1.14.0-rc1 this morning, and I didn't notice that you had > posted this updated patch yesterday. Sorry about the bad timing. > The 1.14.0-rc1 release does not have your patch. But until .0 has been > released we can still decide to

Re: [PATCH] svnadmin build-repcache command

2020-03-25 Thread Stefan Sperling
On Tue, Mar 24, 2020 at 07:53:21PM +0300, Denis Kovalchuk wrote: > + /** @since New in 1.14. */ > + SVN_ERRDEF(SVN_ERR_FS_REP_SHARING_NOT_ALLOWED, > + SVN_ERR_FS_CATEGORY_START + 69, > + "Rep-sharing is not allowed.") > + > + /** @since New in 1.14. */ > + SVN_ERRDEF(SVN

Re: [PATCH] svnadmin build-repcache command

2020-03-24 Thread Denis Kovalchuk
> Well, maybe. The code _appears_ to take a roundabout approach, but I'm > not sure that's actually a bug: there might be a difference between > «INSERT OR IGNORE» on the one hand, and the current behaviour («INSERT > OR FAIL» plus a C check) on the other. > > Is «INSERT OR IGNORE» supported by al

Re: [PATCH] svnadmin build-repcache command

2020-03-20 Thread Daniel Shahaf
Denis Kovalchuk wrote on Fri, 20 Mar 2020 22:54 +0300: > > STMT_SET_REP uses «INSERT OR FAIL». Therefore, this call will fail if > > any of the reps in REV are in rep-cache.db. In particular, I expect it > > will fail in the common case that _all_ of the reps in REV are already > in > the datab

Re: [PATCH] svnadmin build-repcache command

2020-03-20 Thread Denis Kovalchuk
Thanks for your replies. > Add: > > /* See svn_fs_fs__build_rep_cache(). */ Fixed. > STMT_SET_REP uses «INSERT OR FAIL». Therefore, this call will fail if > any of the reps in REV are in rep-cache.db. In particular, I expect it > will fail in the common case that _all_ of the reps in REV are

Re: [PATCH] svnadmin build-repcache command

2020-03-18 Thread Julian Foad
Denis Kovalchuk wrote: I would like to suggest an implementation of ‘svnadmin build-repcache’ command Thank you for contributing this enhancement! I agree this is a useful and appropriate feature. - Julian

Re: [PATCH] svnadmin build-repcache command

2020-03-18 Thread Daniel Shahaf
Denis Kovalchuk wrote on Wed, 18 Mar 2020 17:32 +0300: > +++ subversion/include/private/svn_fs_fs_private.h(working copy) > @@ -353,6 +353,16 @@ Please use the -p option to diff. > +typedef struct svn_fs_fs__ioctl_build_rep_cache_input_t > +{ > + svn_revnum_t start_rev; > + svn_revnum_t end