Re: svn commit: r1825787 - /subversion/trunk/subversion/tests/libsvn_subr/utf-test.c

2018-03-03 Thread James McCoy
On Sat, Mar 03, 2018 at 10:21:37PM +0100, Branko Čibej wrote: > On 03.03.2018 22:03, james...@apache.org wrote: > > Author: jamessan > > Date: Sat Mar 3 21:03:09 2018 > > New Revision: 1825787 > > > > URL: http://svn.apache.org/viewvc?rev=1825787&view=rev > > Log: > > * subversion/tests/libsvn_sub

Re: svn commit: r1825787 - /subversion/trunk/subversion/tests/libsvn_subr/utf-test.c

2018-03-03 Thread Branko Čibej
On 03.03.2018 22:03, james...@apache.org wrote: > Author: jamessan > Date: Sat Mar 3 21:03:09 2018 > New Revision: 1825787 > > URL: http://svn.apache.org/viewvc?rev=1825787&view=rev > Log: > * subversion/tests/libsvn_subr/utf-test.c: > (test_utf_conversions): Move maxlen loop after declarations

Re: libsvn_subr/utf-test failure due to unaligned memory access

2018-03-03 Thread Branko Čibej
On 03.03.2018 20:18, James McCoy wrote: > This test has been failing for some time on Debian's alpha buildd[0]. > One of the alpha porters looked into it and gave a thorough diagnosis[1] > of the issue which basically boils down to: using a char* as > apr_uint16_t*/apr_int32_t* is going to result i

libsvn_subr/utf-test failure due to unaligned memory access

2018-03-03 Thread James McCoy
This test has been failing for some time on Debian's alpha buildd[0]. One of the alpha porters looked into it and gave a thorough diagnosis[1] of the issue which basically boils down to: using a char* as apr_uint16_t*/apr_int32_t* is going to result in unaligned access. In a (much belated) reply[2

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Philip Martin
Branko Čibej writes: > In other words ... if we wanted to make authz changes have immediate > effect, we'd need a better (faster, or at least non-blocking) way to > determine that the rules changed than reading the authz file, even if Relatively easy to do for in-repository authz. We could make

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Branko Čibej
On 03.03.2018 19:15, James McCoy wrote: > On Sat, Mar 03, 2018 at 06:54:06PM +0100, Branko Čibej wrote: >> P.S.: Running tests now with the patched 1.10.x, will vote on the >> backport as soon as that's done. If it's approved, I believe we have to >> move our expected release date from 28th March t

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread James McCoy
On Sat, Mar 03, 2018 at 06:54:06PM +0100, Branko Čibej wrote: > P.S.: Running tests now with the patched 1.10.x, will vote on the > backport as soon as that's done. If it's approved, I believe we have to > move our expected release date from 28th March to 4th April? According to the flowchart post

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Branko Čibej
On 03.03.2018 18:46, Philip Martin wrote: > Branko Čibej writes: > >> So if I understand this debate correctly: The authz code is so much >> faster now that parsing the authz file and performing the authz lookups >> beats calculating its MD5 checksum? > More that reading/checksumming is still too

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Philip Martin
Branko Čibej writes: > So if I understand this debate correctly: The authz code is so much > faster now that parsing the authz file and performing the authz lookups > beats calculating its MD5 checksum? More that reading/checksumming is still too slow to be done repeatedly. 1.9 reads the file o

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Stefan Sperling
On Sat, Mar 03, 2018 at 06:01:23PM +0100, Branko Čibej wrote: > On 03.03.2018 17:44, Stefan Sperling wrote: > > On Sat, Mar 03, 2018 at 04:32:35PM +, Philip Martin wrote: > >> Stefan Sperling writes: > >> > >>> Which leads me to believe that r1778923 may have been based on wrong > >>> assumpti

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Philip Martin
Philip Martin writes: > r1825778 contains a revert of a change not in 1.10, so doesn't merge > cleanly to 1.10. The reverse merge of -c-r1779188,-1778923 from trunk > does merge cleanly to 1.10. However, r1825736 is already nominated, so I just need to add r1825778 to that :-) -- Philip

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Philip Martin
Philip Martin writes: > Stefan Sperling writes: > >> I think our best course of action is to revert the change on trunk >> and in 1.10.x. Could you do that? (I could do it, too. I'm just asking >> you since you've probably already prepared it in a local copy.) > > OK, r1825778. > > Hmm, it doesn

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Branko Čibej
On 03.03.2018 17:44, Stefan Sperling wrote: > On Sat, Mar 03, 2018 at 04:32:35PM +, Philip Martin wrote: >> Stefan Sperling writes: >> >>> Which leads me to believe that r1778923 may have been based on wrong >>> assumptions about performance. The new authz is not fast enough to >>> significant

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Philip Martin
Stefan Sperling writes: > I think our best course of action is to revert the change on trunk > and in 1.10.x. Could you do that? (I could do it, too. I'm just asking > you since you've probably already prepared it in a local copy.) OK, r1825778. Hmm, it doesn't merge cleanly into 1.10... -- P

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Stefan Sperling
On Sat, Mar 03, 2018 at 04:32:35PM +, Philip Martin wrote: > Stefan Sperling writes: > > > Which leads me to believe that r1778923 may have been based on wrong > > assumptions about performance. The new authz is not fast enough to > > significantly reduce per-request overhead. > > My testing

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

2018-03-03 Thread Stefan Sperling
On Fri, Mar 02, 2018 at 05:15:46PM +0300, Evgeny Kotkov wrote: > Unless I am missing something, this might be worth considering before the > 1.10 GA release. Evgeny, you were entirely right about calling this out as a release blocker. I am sorry for having suggested otherwise.

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Philip Martin
Stefan Sperling writes: > Which leads me to believe that r1778923 may have been based on wrong > assumptions about performance. The new authz is not fast enough to > significantly reduce per-request overhead. My testing so far was with a very small authz file -- only a handful of rules and alias

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Stefan Sperling
On Sat, Mar 03, 2018 at 04:27:36PM +0100, Stefan Sperling wrote: > Right now I have no idea how to make the current code in trunk any > faster without reverting r1778923. Thinking about this some more, I think reverting r1778923 is the only way to make it faster. The authz cache key is based on t

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Philip Martin
Philip Martin writes: > Here are the cache stats before/after a run once the timing has > stabilised: > > gets : 11770766, 4388248 hits (37.28%) > sets : 7309448 (99.01% of misses) > failures: 0 > used : 777 MB (94.56%) of 822 MB data cache / 958 MB total cache memory > 2225228 entries (99.45%) o

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Stefan Sperling
On Sat, Mar 03, 2018 at 03:10:46PM +0100, Stefan Sperling wrote: > Regardless, could you try one more test run with this diff against > trunk to see if it has any impact already? My new diff doesn't help, it brings the memory problem back. I have done some more digging with APR POOL DEBUG. It is

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Philip Martin
Stefan Sperling writes: > Regardless, could you try one more test run with this diff against > trunk to see if it has any impact already? That's slower than current trunk: 6.2s 4.1s 4.5s 4.7s 4.7s 4.6s 4.7s and memory use is back to several GB. -- Philip

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

2018-03-03 Thread Branko Čibej
On 02.03.2018 19:21, Stefan Sperling wrote: > I am just questioning the usefulness of halting the presses and restarting > the soak for another month for something that isn't a security / data > corruption issue. It's a potential DOS that only needs read access. Falls under security by my definiti

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Philip Martin
Philip Martin writes: > Note an odd effect in the above numbers. The second run for 1.11 is > always the fastest. The first run in each case is the slowest, the > Subversion cache is cold. The second run is faster, the Subversion > cache is hot. In 1.9 subsequent runs are comparable to the sec

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Stefan Sperling
On Sat, Mar 03, 2018 at 01:06:49PM +, Philip Martin wrote: > Stefan Sperling writes: > > > I may have found a way to store an svn_repos_t object on the connection. > > Does it help? (authz_tests pass) > > No significant improvement. > > I am timing > > time svn log http://localhost:/

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Paul Hammant
> Yes, the problem seems to be that mod_authz_svn has no way of storing per-connection state at present. Please excuse the interjection here: My eyes spied "per connection" and I wondered if something I'm doing elsewhere that is closer (but not exactly) "per user" might help in terms of thinking.

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Philip Martin
Stefan Sperling writes: > I may have found a way to store an svn_repos_t object on the connection. > Does it help? (authz_tests pass) No significant improvement. I am timing time svn log http://localhost:/repos/asf/subversion \ --username pm --password mp > /dev/null and I realised

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Stefan Sperling
On Sat, Mar 03, 2018 at 11:43:31AM +0100, Stefan Sperling wrote: > On Fri, Mar 02, 2018 at 09:40:28PM +, Philip Martin wrote: > > Philip Martin writes: > > > > > Again, 1.10 would be nearly twice as fast, but now rereading the authz > > > removes most of that gain. > > > > I think I see the

Re: Potential regression: high server-side memory consumption during import

2018-03-03 Thread Stefan Sperling
On Fri, Mar 02, 2018 at 09:40:28PM +, Philip Martin wrote: > Philip Martin writes: > > > Again, 1.10 would be nearly twice as fast, but now rereading the authz > > removes most of that gain. > > I think I see the underlying problem: the authz code now incorporates a > cache based on the md5