How did you upgrade your server from RHEL 6 to RHEL 8? Did you run
"svnadmin upgrade" on the repo? And just how *big* is the repo? Does
your client setup work with a notably smaller repo? And do the
problems happen if you use svn:// rather than https:// ? Because I've
been dealing with Subversion for a long time, and I have up on the
httpd integration years ago, The complex integration with an
overmuscled and constantly evolving, feature evolving webserver was
not something I bother with anymore, and it has big problems with big
repos, including its own repo over at http://svn.apache.org/repos/asf,
which can't be reliably synced anymore because it's gotten so large.
On Fri, May 10, 2024 at 4:17 PM Williams, James P. {Jim} (JSC-CD4)[KBR
Wyle Services, LLC] via users <[email protected]> wrote:
>
> I'm upgrading an Apache HTTP server of our SVN repos on RedHat Enterprise
> Linux 8. Using Subversion 1.14.1, svn checkout of even a small, simple repo
> with about 150 files hangs about 90% of the time, crashes 5%, and succeeds
> 5%. Given enough time, the hangs eventually time out after checking out much
> of the repo. A debugger shows the following stack during the hang.
>
>
>
> #0 epoll_wait /usr/lib64/libc.so.6
>
> #1 impl_pollset_poll /usr/lib64/libapr-1.so.0
>
> #2 serf_context_run /usr/lib64/libserf-1.so.0
>
> #3 svn_ra_serf.context_run /usr/lib64/libsvn_ra_serf-1.so.0
>
> #4 finish_report /usr/lib64/libsvn_ra_serf-1.so.0
>
> #5 svn_wc_crawl_revisions5 /usr/lib64/libsvn_wc-1.so.0
>
> #6 update_internal.isra /usr/lib64/libsvn_client-1.so.0
>
> #7 svn_client.update_internal /usr/lib64/libsvn_client-1.so.0
>
> #8 svn_client.checkout_internal /usr/lib64/libsvn_client-1.so.0
>
> #9 svn_client_checkout3 /usr/lib64/libsvn_client-1.so.0
>
> #10 svn_cl.checkout
>
> #11 sub_main
>
> #12 main
>
>
>
> strace shows repeated calls to epoll_wait about 1 sec apart.
>
>
>
> When the checkout crashes, it's a SIGSEGV with this stack,
>
>
>
> #0 apr_pool_create_ex (libapr-1.so.0)
>
> #1 svn_pool_create_ex (libsvn_subr-1.so.0)
>
> #2 update_opened (libsvn_ra_serf-1.so.0)
>
> #3 expat_start (libsvn_ra_serf-1.so.0)
>
> #4 expat_start_handler (libsvn_subr-1.so.0)
>
> #5 doContent (libexpat.so.1)
>
> #6 contentProcessor (libexpat.so.1)
>
> #7 XML_ParseBuffer (libexpat.so.1)
>
> #8 svn_xml_parse (libsvn_subr-1.so.0)
>
> #9 expat_response_handler (libsvn_ra_serf-1.so.0)
>
> #10 process_buffer.isra.9 (libsvn_ra_serf-1.so.0)
>
> #11 finish_report (libsvn_ra_serf-1.so.0)
>
> #12 svn_wc_crawl_revisions5 (libsvn_wc-1.so.0)
>
> #13 update_internal.isra.0 (libsvn_client-1.so.0)
>
> #14 svn_client__update_internal (libsvn_client-1.so.0)
>
> #15 svn_client__checkout_internal (libsvn_client-1.so.0)
>
> #16 svn_client_checkout3 (libsvn_client-1.so.0)
>
> #17 svn_cl__checkout (svn)
>
> #18 sub_main (svn)
>
> #19 main (svn)
>
> #20 __libc_start_main (libc.so.6)
>
> #21 _start (svn)
>
>
>
> or this one,
>
>
>
> #0 apr_allocator_alloc (libapr-1.so.0)
>
> #1 serf_bucket_mem_alloc (libserf-1.so.0)
>
> #2 serf_bucket_response_create (libserf-1.so.0)
>
> #3 serf.process_connection (libserf-1.so.0)
>
> #4 serf_event_trigger (libserf-1.so.0)
>
> #5 serf_context_run (libserf-1.so.0)
>
> #6 svn_ra_serf.context_run (libsvn_ra_serf-1.so.0)
>
> #7 finish_report (libsvn_ra_serf-1.so.0)
>
> #8 svn_wc_crawl_revisions5 (libsvn_wc-1.so.0)
>
> #9 update_internal.isra (libsvn_client-1.so.0)
>
> #10 svn_client.update_internal (libsvn_client-1.so.0)
>
> #11 svn_client.checkout_internal (libsvn_client-1.so.0)
>
> #12 svn_client_checkout3 (libsvn_client-1.so.0)
>
> #13 svn_cl.checkout (svn)
>
> #14 sub_main (svn)
>
> #15 main (svn)
>
>
>
> After a failure, I'm left with a half-checked out working copy with many
> locks. I can complete it with svn cleanup and another svn checkout, but
> that's not realistic for our CI/CD or general use. Server logs show no
> indication of a problem; the server appears healthy.
>
>
>
> I've tried a million things before submitting this bug report, read half a
> million posts and searches, but haven't been able to get past this. I'd sure
> appreciate any ideas you have on the way forward. Here's a bit more about my
> system.
>
>
>
> svn, version 1.14.1 (r1886195)
>
> * ra_svn : Module for accessing a repository using the svn network
> protocol.
>
> - with Cyrus SASL authentication
>
> - handles 'svn' scheme
>
> * ra_local : Module for accessing a repository on local disk.
>
> - handles 'file' scheme
>
> * ra_serf : Module for accessing a repository via WebDAV protocol using
> serf.
>
> - using serf 1.3.9 (compiled with 1.3.9)
>
> - handles 'http' scheme
>
> - handles 'https' scheme
>
>
>
> The following authentication credential caches are available:
>
>
>
> * Plaintext cache in /home/me/.subversion
>
> * Gnome Keyring
>
> * GPG-Agent
>
> svn 1.10.2 was failing the same way before we upgraded to 1.14.1 as a
> possible fix.
> Checking out to a local disk succeeds more often, but still hangs and
> crashes. Checking out to an NFS drive just makes it worse.
>
>
>
> And here's more about our Apache.
>
>
>
> Server version: Apache/2.4.37 (Red Hat Enterprise Linux)
>
> Server built: Aug 30 2023 11:01:53
>
> Server's Module Magic Number: 20120211:83
>
> Server loaded: APR 1.6.3, APR-UTIL 1.6.1
>
> Compiled using: APR 1.6.3, APR-UTIL 1.6.1
>
> Architecture: 64-bit
>
> Server MPM: worker
>
> threaded: yes (fixed thread count)
>
> forked: yes (variable process count)
>
> Server compiled with....
>
> -D APR_HAS_SENDFILE
>
> -D APR_HAS_MMAP
>
> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>
> -D APR_USE_SYSVSEM_SERIALIZE
>
> -D APR_USE_PTHREAD_SERIALIZE
>
> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>
> -D APR_HAS_OTHER_CHILD
>
> -D AP_HAVE_RELIABLE_PIPED_LOGS
>
> -D DYNAMIC_MODULE_LIMIT=256
>
> -D HTTPD_ROOT="/etc/httpd"
>
> -D SUEXEC_BIN="/usr/sbin/suexec"
>
> -D DEFAULT_PIDLOG="run/httpd.pid"
>
> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>
> -D DEFAULT_ERRORLOG="logs/error_log"
>
> -D AP_TYPES_CONFIG_FILE="conf/mime.types"
>
> -D SERVER_CONFIG_FILE="conf/httpd.conf"
>
> Access to this server from a browser with SVN and ViewVC pages seems to work.
> Authentication is over Kerberos with mod_auth_gssapi.
> Authorization uses AuthzSVNAccessFile and an access file.
> SSL is used with SSLCryptoDevice set to builtin, based on this.
> I've tried all three MPMs with no change, based on another post: prefork,
> worker, and event.
> We've had Apache running on RedHat 6 with these repos for many years.
>
>
>
> I'd be glad to provide additional details or run more tests. Thanks for any
> ideas you have, and for supporting this software.
>
>
>
> Jim
