On Mon, 22 May 2023, 17:58 Mateusz Kempski, <mkemp...@xtm-intl.com.invalid> wrote:
> @Yann Ylavic: > There seems to be no difference in configuration file except for some > comments: > ``` > diff -Bbde /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.rocky > 8a > # See the httpd.conf(5) man page for more information on this > configuration, > # and httpd.service(8) on using and configuring the httpd service. > # > . > ``` > I will paste complete Rocky config for completeness at the end of this > message. > EnableMMAP and EnableSendfile are set to on, differently to defaults > specified in comments. This is default system configuration present on > both Rocky 8 and Centos 7. I removed httpd and all config files and > installed it again to make sure. > ``` > # Defaults if commented: EnableMMAP On, EnableSendfile Off > # > #EnableMMAP off > EnableSendfile on > ``` > > Both systems return: > ``` > ulimit -n > 1024 > ``` > There is no custom config for httpd: > ``` > stat /etc/systemd/system/httpd.service.d > stat: cannot stat ‘/etc/systemd/system/httpd.service.d’: No such file > or directory > ``` > systemd itself seems to not apply any limit on both systems: > ``` > grep LimitNOFILE /etc/systemd/system.conf > #DefaultLimitNOFILE= > ``` > > After enabling trace there is a lot of messenges about too many open > connections and killing keepalive connections. Example snippet: > ``` > [Mon May 22 11:57:40.146451 2023] [mpm_event:debug] [pid 7710:tid > 139800984155904] event.c(1808): Too many open connections (73), not > accepting new conns in > this process > [Mon May 22 11:57:40.146455 2023] [mpm_event:trace1] [pid 7710:tid > 139800984155904] event.c(1811): Idle workers: 0 > [Mon May 22 11:57:40.146457 2023] [mpm_event:trace1] [pid 7710:tid > 139800984155904] event.c(1574): All workers are busy or dying, will > close 2 keep-alive co > nnections > [Mon May 22 11:57:40.146526 2023] [mpm_event:debug] [pid 7379:tid > 139801403561728] event.c(492): AH00457: Accepting new connections > again: 50 active conns ( > 27 lingering/0 clogged/0 suspended), 7 idle workers > [Mon May 22 11:57:40.146686 2023] [mpm_event:debug] [pid 7511:tid > 139800849938176] event.c(1808): Too many open connections (86), not > accepting new conns in > this process > [Mon May 22 11:57:40.146698 2023] [mpm_event:trace1] [pid 7511:tid > 139800849938176] event.c(1811): Idle workers: 11 > [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7511:tid > 139800849938176] event.c(1574): All workers are busy or dying, will > close 34 keep-alive c > onnections > [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7644:tid > 139801336452864] event.c(1574): All workers are busy or dying, will > close 2 keep-alive co > nnections > [Mon May 22 11:57:40.146812 2023] [mpm_event:debug] [pid 7511:tid > 139800849938176] event.c(492): AH00457: Accepting new connections > again: 85 active conns ( > 52 lingering/0 clogged/0 suspended), 6 idle workers > [Mon May 22 11:57:40.146955 2023] [mpm_event:debug] [pid 7710:tid > 139800984155904] event.c(492): AH00457: Accepting new connections > again: 73 active conns ( > 58 lingering/0 clogged/0 suspended), 9 idle workers > [Mon May 22 11:57:40.148250 2023] [mpm_event:debug] [pid 7162:tid > 139800841545472] event.c(492): AH00457: Accepting new connections > again: 48 active conns ( > 25 lingering/0 clogged/0 suspended), 2 idle workers > [Mon May 22 11:57:40.148562 2023] [mpm_event:debug] [pid 7162:tid > 139800841545472] event.c(1808): Too many open connections (48), not > accepting new conns in > this process > [Mon May 22 11:57:40.148573 2023] [mpm_event:trace1] [pid 7162:tid > 139800841545472] event.c(1811): Idle workers: 0 > [Mon May 22 11:57:40.148921 2023] [mpm_event:debug] [pid 7710:tid > 139800984155904] event.c(1808): Too many open connections (70), not > accepting new conns in > this process > [Mon May 22 11:57:40.148930 2023] [mpm_event:trace1] [pid 7710:tid > 139800984155904] event.c(1811): Idle workers: 1 > [Mon May 22 11:57:40.149594 2023] [mpm_event:debug] [pid 7511:tid > 139800849938176] event.c(1808): Too many open connections (69), not > accepting new conns in > this process > [Mon May 22 11:57:40.149603 2023] [mpm_event:trace1] [pid 7511:tid > 139800849938176] event.c(1811): Idle workers: 1 > [Mon May 22 11:57:40.149630 2023] [mpm_event:debug] [pid 7710:tid > 139800984155904] event.c(492): AH00457: Accepting new connections > again: 61 active conns ( > 38 lingering/0 clogged/0 suspended), 2 idle workers > [Mon May 22 11:57:40.149776 2023] [mpm_event:debug] [pid 7710:tid > 139800984155904] event.c(1808): Too many open connections (63), not > accepting new conns in > this process > [Mon May 22 11:57:40.149782 2023] [mpm_event:trace1] [pid 7710:tid > 139800984155904] event.c(1811): Idle workers: 0 > [Mon May 22 11:57:40.149882 2023] [mpm_event:debug] [pid 7710:tid > 139800984155904] event.c(492): AH00457: Accepting new connections > again: 61 active conns ( > 38 lingering/0 clogged/0 suspended), 2 idle workers > [Mon May 22 11:57:40.149913 2023] [mpm_event:debug] [pid 7511:tid > 139800849938176] event.c(492): AH00457: Accepting new connections > again: 63 active conns ( > 40 lingering/0 clogged/0 suspended), 2 idle workers > [Mon May 22 11:57:40.150210 2023] [mpm_event:debug] [pid 7511:tid > 139800849938176] event.c(1808): Too many open connections (62), not > accepting new conns in > this process > [Mon May 22 11:57:40.150216 2023] [mpm_event:trace1] [pid 7511:tid > 139800849938176] event.c(1811): Idle workers: 0 > ``` > > @Deepak Goel: > 1. Here are result with previously mentioned KeepAlive parameters: > Rocky 8: > ``` > ab -k -t 900 -c 1000 -n 1000000 http://rocky/ > This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ > Licensed to The Apache Software Foundation, http://www.apache.org/ > > Benchmarking 10.1.3.11 (be patient) > Completed 100000 requests > Completed 200000 requests > Completed 300000 requests > Completed 400000 requests > Completed 500000 requests > Completed 600000 requests > Completed 700000 requests > Completed 800000 requests > Completed 900000 requests > Completed 1000000 requests > Finished 1000000 requests > > > Server Software: Apache/2.4.37 > Server Hostname: 10.1.3.11 > Server Port: 80 > > Document Path: / > Document Length: 7620 bytes > > Concurrency Level: 1000 > Time taken for tests: 17.287 seconds > Complete requests: 1000000 > Failed requests: 98218 > (Connect: 0, Receive: 0, Length: 88397, Exceptions: 9821) > Write errors: 0 > Keep-Alive requests: 911603 > Total transferred: 7214426142 bytes > HTML transferred: 6946414860 bytes > Requests per second: 57845.81 [#/sec] (mean) > Time per request: 17.287 [ms] (mean) > Time per request: 0.017 [ms] (mean, across all concurrent requests) > Transfer rate: 407543.30 [Kbytes/sec] received > > Connection Times (ms) > min mean[+/-sd] median max > Connect: 0 4 59.9 0 3015 > Processing: 0 13 14.5 13 432 > Waiting: 0 13 14.1 13 432 > Total: 0 17 63.2 15 3036 > > Percentage of the requests served within a certain time (ms) > 50% 15 > 66% 17 > 75% 17 > 80% 18 > 90% 21 > 95% 23 > 98% 28 > 99% 36 > 100% 3036 (longest request) > ``` > Centos 7: > ``` > ab -k -t 900 -c 1000 -n 1000000 http://10.1.3.3/ > This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ > Licensed to The Apache Software Foundation, http://www.apache.org/ > > Benchmarking 10.1.3.3 (be patient) > Completed 100000 requests > Completed 200000 requests > Completed 300000 requests > Completed 400000 requests > Completed 500000 requests > Completed 600000 requests > Completed 700000 requests > Completed 800000 requests > Completed 900000 requests > Completed 1000000 requests > Finished 1000000 requests > > > Server Software: Apache/2.4.6 > Server Hostname: 10.1.3.3 > Server Port: 80 > > Document Path: / > Document Length: 7620 bytes > > Concurrency Level: 1000 > Time taken for tests: 20.731 seconds > Complete requests: 1000000 > Failed requests: 14 > (Connect: 0, Receive: 0, Length: 14, Exceptions: 0) > Write errors: 0 > Keep-Alive requests: 999986 > Total transferred: 7913889204 bytes > HTML transferred: 7619893320 bytes > Requests per second: 48236.94 [#/sec] (mean) > Time per request: 20.731 [ms] (mean) > Time per request: 0.021 [ms] (mean, across all concurrent requests) > Transfer rate: 372794.72 [Kbytes/sec] received > > Connection Times (ms) > min mean[+/-sd] median max > Connect: 0 0 13.1 0 1025 > Processing: 2 21 7.9 20 821 > Waiting: 0 20 7.8 20 821 > Total: 2 21 15.7 20 1244 > > Percentage of the requests served within a certain time (ms) > 50% 20 > 66% 22 > 75% 23 > 80% 24 > 90% 25 > 95% 27 > 98% 29 > 99% 30 > 100% 1244 (longest request) > ``` > 3. iotop results: > Centos 7 idle: > ``` > Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s > Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s > ``` > Rocky 8 idle: > ``` > Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s > Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s > ``` > Centos 7 during test: > ``` > Total DISK READ : 0.00 B/s | Total DISK WRITE : 4.56 M/s > Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s > ``` > Rocky 8 during test: > ``` > Total DISK READ : 0.00 B/s | Total DISK WRITE : 5.35 M/s > The DISK WRITE is higher in Rocky as compared to Centos. Is it possible for you to test on Ubuntu 18/20 as well? ( Seems Rocky algo's are taking a performance hit ) > Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s > ``` > I ran top command without parameters on both systems - looks like > Centos defaults to KiB and Rocky to MiB. > > Complete httpd.conf: > ``` > # > # This is the main Apache HTTP server configuration file. It contains the > # configuration directives that give the server its instructions. > # See <URL:http://httpd.apache.org/docs/2.4/> for detailed information. > # In particular, see > # <URL:http://httpd.apache.org/docs/2.4/mod/directives.html> > # for a discussion of each configuration directive. > # > # See the httpd.conf(5) man page for more information on this > configuration, > # and httpd.service(8) on using and configuring the httpd service. > # > # Do NOT simply read the instructions in here without understanding > # what they do. They're here only as hints or reminders. If you are > unsure > # consult the online docs. You have been warned. > # > # Configuration and logfile names: If the filenames you specify for many > # of the server's control files begin with "/" (or "drive:/" for Win32), > the > # server will use that explicit path. If the filenames do *not* begin > # with "/", the value of ServerRoot is prepended -- so 'log/access_log' > # with ServerRoot set to '/www' will be interpreted by the > # server as '/www/log/access_log', where as '/log/access_log' will be > # interpreted as '/log/access_log'. > > # > # ServerRoot: The top of the directory tree under which the server's > # configuration, error, and log files are kept. > # > # Do not add a slash at the end of the directory path. If you point > # ServerRoot at a non-local disk, be sure to specify a local disk on the > # Mutex directive, if file-based mutexes are used. If you wish to share > the > # same ServerRoot for multiple httpd daemons, you will need to change at > # least PidFile. > # > ServerRoot "/etc/httpd" > > # > # Listen: Allows you to bind Apache to specific IP addresses and/or > # ports, instead of the default. See also the <VirtualHost> > # directive. > # > # Change this to Listen on specific IP addresses as shown below to > # prevent Apache from glomming onto all bound IP addresses. > # > #Listen 12.34.56.78:80 > Listen 80 > > # > # Dynamic Shared Object (DSO) Support > # > # To be able to use the functionality of a module which was built as a DSO > you > # have to place corresponding `LoadModule' lines at this location so the > # directives contained in it are actually available _before_ they are used. > # Statically compiled modules (those listed by `httpd -l') do not need > # to be loaded here. > # > # Example: > # LoadModule foo_module modules/mod_foo.so > # > Include conf.modules.d/*.conf > > # > # If you wish httpd to run as a different user or group, you must run > # httpd as root initially and it will switch. > # > # User/Group: The name (or #number) of the user/group to run httpd as. > # It is usually good practice to create a dedicated user and group for > # running httpd, as with most system services. > # > User apache > Group apache > > # 'Main' server configuration > # > # The directives in this section set up the values used by the 'main' > # server, which responds to any requests that aren't handled by a > # <VirtualHost> definition. These values also provide defaults for > # any <VirtualHost> containers you may define later in the file. > # > # All of these directives may appear inside <VirtualHost> containers, > # in which case these default settings will be overridden for the > # virtual host being defined. > # > > # > # ServerAdmin: Your address, where problems with the server should be > # e-mailed. This address appears on some server-generated pages, such > # as error documents. e.g. ad...@your-domain.com > # > ServerAdmin root@localhost > > # > # ServerName gives the name and port that the server uses to identify > itself. > # This can often be determined automatically, but we recommend you specify > # it explicitly to prevent problems during startup. > # > # If your host doesn't have a registered DNS name, enter its IP address > here. > # > #ServerName www.example.com:80 > > # > # Deny access to the entirety of your server's filesystem. You must > # explicitly permit access to web content directories in other > # <Directory> blocks below. > # > <Directory /> > AllowOverride none > Require all denied > </Directory> > > # > # Note that from this point forward you must specifically allow > # particular features to be enabled - so if something's not working as > # you might expect, make sure that you have specifically enabled it > # below. > # > > # > # DocumentRoot: The directory out of which you will serve your > # documents. By default, all requests are taken from this directory, but > # symbolic links and aliases may be used to point to other locations. > # > DocumentRoot "/var/www/html" > > # > # Relax access to content within /var/www. > # > <Directory "/var/www"> > AllowOverride None > # Allow open access: > Require all granted > </Directory> > > # Further relax access to the default document root: > <Directory "/var/www/html"> > # > # Possible values for the Options directive are "None", "All", > # or any combination of: > # Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI > MultiViews > # > # Note that "MultiViews" must be named *explicitly* --- "Options All" > # doesn't give it to you. > # > # The Options directive is both complicated and important. Please see > # http://httpd.apache.org/docs/2.4/mod/core.html#options > # for more information. > # > Options Indexes FollowSymLinks > > # > # AllowOverride controls what directives may be placed in .htaccess > files. > # It can be "All", "None", or any combination of the keywords: > # Options FileInfo AuthConfig Limit > # > AllowOverride None > > # > # Controls who can get stuff from this server. > # > Require all granted > </Directory> > > # > # DirectoryIndex: sets the file that Apache will serve if a directory > # is requested. > # > <IfModule dir_module> > DirectoryIndex index.html > </IfModule> > > # > # The following lines prevent .htaccess and .htpasswd files from being > # viewed by Web clients. > # > <Files ".ht*"> > Require all denied > </Files> > > # > # ErrorLog: The location of the error log file. > # If you do not specify an ErrorLog directive within a <VirtualHost> > # container, error messages relating to that virtual host will be > # logged here. If you *do* define an error logfile for a <VirtualHost> > # container, that host's errors will be logged there and not here. > # > ErrorLog "logs/error_log" > > # > # LogLevel: Control the number of messages logged to the error_log. > # Possible values include: debug, info, notice, warn, error, crit, > # alert, emerg. > # > LogLevel warn > > <IfModule log_config_module> > # > # The following directives define some format nicknames for use with > # a CustomLog directive (see below). > # > LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" > \"%{User-Agent}i\"" combined > LogFormat "%h %l %u %t \"%r\" %>s %b" common > > <IfModule logio_module> > # You need to enable mod_logio.c to use %I and %O > LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" > \"%{User-Agent}i\" %I %O" combinedio > </IfModule> > > # > # The location and format of the access logfile (Common Logfile Format). > # If you do not define any access logfiles within a <VirtualHost> > # container, they will be logged here. Contrariwise, if you *do* > # define per-<VirtualHost> access logfiles, transactions will be > # logged therein and *not* in this file. > # > #CustomLog "logs/access_log" common > > # > # If you prefer a logfile with access, agent, and referer information > # (Combined Logfile Format) you can use the following directive. > # > CustomLog "logs/access_log" combined > </IfModule> > > <IfModule alias_module> > # > # Redirect: Allows you to tell clients about documents that used to > # exist in your server's namespace, but do not anymore. The client > # will make a new request for the document at its new location. > # Example: > # Redirect permanent /foo http://www.example.com/bar > > # > # Alias: Maps web paths into filesystem paths and is used to > # access content that does not live under the DocumentRoot. > # Example: > # Alias /webpath /full/filesystem/path > # > # If you include a trailing / on /webpath then the server will > # require it to be present in the URL. You will also likely > # need to provide a <Directory> section to allow access to > # the filesystem path. > > # > # ScriptAlias: This controls which directories contain server scripts. > # ScriptAliases are essentially the same as Aliases, except that > # documents in the target directory are treated as applications and > # run by the server when requested rather than as documents sent to the > # client. The same rules about trailing "/" apply to ScriptAlias > # directives as to Alias. > # > ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" > > </IfModule> > > # > # "/var/www/cgi-bin" should be changed to whatever your ScriptAliased > # CGI directory exists, if you have that configured. > # > <Directory "/var/www/cgi-bin"> > AllowOverride None > Options None > Require all granted > </Directory> > > <IfModule mime_module> > # > # TypesConfig points to the file containing the list of mappings from > # filename extension to MIME-type. > # > TypesConfig /etc/mime.types > > # > # AddType allows you to add to or override the MIME configuration > # file specified in TypesConfig for specific file types. > # > #AddType application/x-gzip .tgz > # > # AddEncoding allows you to have certain browsers uncompress > # information on the fly. Note: Not all browsers support this. > # > #AddEncoding x-compress .Z > #AddEncoding x-gzip .gz .tgz > # > # If the AddEncoding directives above are commented-out, then you > # probably should define those extensions to indicate media types: > # > AddType application/x-compress .Z > AddType application/x-gzip .gz .tgz > > # > # AddHandler allows you to map certain file extensions to "handlers": > # actions unrelated to filetype. These can be either built into the > server > # or added with the Action directive (see below) > # > # To use CGI scripts outside of ScriptAliased directories: > # (You will also need to add "ExecCGI" to the "Options" directive.) > # > #AddHandler cgi-script .cgi > > # For type maps (negotiated resources): > #AddHandler type-map var > > # > # Filters allow you to process content before it is sent to the client. > # > # To parse .shtml files for server-side includes (SSI): > # (You will also need to add "Includes" to the "Options" directive.) > # > AddType text/html .shtml > AddOutputFilter INCLUDES .shtml > </IfModule> > > # > # Specify a default charset for all content served; this enables > # interpretation of all content as UTF-8 by default. To use the > # default browser choice (ISO-8859-1), or to allow the META tags > # in HTML content to override this choice, comment out this > # directive: > # > AddDefaultCharset UTF-8 > > <IfModule mime_magic_module> > # > # The mod_mime_magic module allows the server to use various hints from > the > # contents of the file itself to determine its type. The MIMEMagicFile > # directive tells the module where the hint definitions are located. > # > MIMEMagicFile conf/magic > </IfModule> > > # > # Customizable error responses come in three flavors: > # 1) plain text 2) local redirects 3) external redirects > # > # Some examples: > #ErrorDocument 500 "The server made a boo boo." > #ErrorDocument 404 /missing.html > #ErrorDocument 404 "/cgi-bin/missing_handler.pl" > #ErrorDocument 402 http://www.example.com/subscription_info.html > # > > # > # EnableMMAP and EnableSendfile: On systems that support it, > # memory-mapping or the sendfile syscall may be used to deliver > # files. This usually improves server performance, but must > # be turned off when serving from networked-mounted > # filesystems or if support for these functions is otherwise > # broken on your system. > # Defaults if commented: EnableMMAP On, EnableSendfile Off > # > #EnableMMAP off > EnableSendfile on > > # Supplemental configuration > # > # Load config files in the "/etc/httpd/conf.d" directory, if any. > IncludeOptional conf.d/*.conf > > <IfModule mpm_event_module> > ThreadsPerChild 25 > StartServers 3 > ServerLimit 120 > MinSpareThreads 75 > MaxSpareThreads 3000 > MaxRequestWorkers 3000 > MaxConnectionsPerChild 0 > </IfModule> > ``` > > Mateusz Kempski > Linux System Administrator > XTM International Ltd. > Email: mkemp...@xtm-intl.com > xtm.cloud > > > > > On Mon, 22 May 2023 at 11:16, Deepak Goel <deic...@gmail.com> wrote: > > > > Hi > > > > 1. Please post the test results completely. (Sorry but, saying there is > no difference does not help). > > 2. The memory used in Rocky is comparatively higher than Centos (About > 30Meg or so). Also the buff/cache is approx 10 times higher in Rocky than > in Centos. > > 3. Please also post iotop results too. > > > > P.S: I hope the top commands is run identical for both Rocky and Centos. > > > > Deepak > > "The greatness of a nation can be judged by the way its animals are > treated - Mahatma Gandhi" > > > > +91 73500 12833 > > deic...@gmail.com > > > > Facebook: https://www.facebook.com/deicool > > LinkedIn: www.linkedin.com/in/deicool > > > > "Plant a Tree, Go Green" > > > > Make In India : http://www.makeinindia.com/home > > > > > > > > On Mon, May 22, 2023 at 4:28 PM Mateusz Kempski > <mkemp...@xtm-intl.com.invalid> wrote: > >> > >> I tested again with settings: > >> ``` > >> KeepAliveTimeout 300 > >> MaxKeepAliveRequests 0 > >> ``` > >> but there was no difference in results barring normal diffs run-to-run. > >> > >> Below is the top of top from both servers when idle and during test. > >> > >> Rocky 8 no load: > >> ``` > >> top - 10:49:23 up 4 min, 1 user, load average: 3.27, 2.56, 1.10 > >> Tasks: 254 total, 2 running, 252 sleeping, 0 stopped, 0 zombie > >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 > si, 0.0 st > >> MiB Mem : 15824.7 total, 10682.4 free, 283.5 used, 4858.8 > buff/cache > >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15259.3 avail > Mem > >> ``` > >> Rocky 8 during test: > >> ``` > >> top - 10:50:29 up 5 min, 1 user, load average: 4.33, 2.80, 1.28 > >> Tasks: 232 total, 2 running, 230 sleeping, 0 stopped, 0 zombie > >> %Cpu(s): 13.7 us, 16.9 sy, 0.0 ni, 63.9 id, 0.0 wa, 0.0 hi, 5.4 > si, 0.1 st > >> MiB Mem : 15824.7 total, 9863.0 free, 529.3 used, 5432.3 > buff/cache > >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15012.2 avail > Mem > >> ``` > >> Centos 7 no load: > >> ``` > >> top - 10:52:17 up 0 min, 1 user, load average: 0.00, 0.00, 0.00 > >> Tasks: 201 total, 1 running, 200 sleeping, 0 stopped, 0 zombie > >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 > si, 0.0 st > >> KiB Mem : 16264300 total, 15831896 free, 297124 used, 135280 > buff/cache > >> KiB Swap: 0 total, 0 free, 0 used. 15720740 avail > Mem > >> ``` > >> Centos 7 during test: > >> ``` > >> top - 10:53:21 up 1 min, 1 user, load average: 0.62, 0.16, 0.05 > >> Tasks: 218 total, 3 running, 215 sleeping, 0 stopped, 0 zombie > >> %Cpu(s): 17.6 us, 18.9 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.0 hi, 3.1 > si, 0.1 st > >> KiB Mem : 16264300 total, 14973128 free, 503104 used, 788068 > buff/cache > >> KiB Swap: 0 total, 0 free, 0 used. 15459544 avail > Mem > >> ``` > >> > >> On Mon, 22 May 2023 at 10:34, Deepak Goel <deic...@gmail.com> wrote: > >> > > >> > Hi > >> > > >> > I can see about 8000+ requests have timed out in 'Rocky'. This is > mostly due to Apache, which is unable to handle the load. Is it possible to > increase the parameter "KeepAliveTimeout" (and other KeepAlive parameters). > >> > > >> > Is it also possible for you to post the hardware utilisations for the > 2 different servers (Centos & Rocky)? > >> > > >> > Deepak > >> > "The greatness of a nation can be judged by the way its animals are > treated - Mahatma Gandhi" > >> > > >> > +91 73500 12833 > >> > deic...@gmail.com > >> > > >> > Facebook: https://www.facebook.com/deicool > >> > LinkedIn: www.linkedin.com/in/deicool > >> > > >> > "Plant a Tree, Go Green" > >> > > >> > Make In India : http://www.makeinindia.com/home > >> > > >> > > >> > On Mon, May 22, 2023 at 3:49 PM Mateusz Kempski > <mkemp...@xtm-intl.com.invalid> wrote: > >> >> > >> >> Hi all, > >> >> I have two identical VMs - 16GB RAM, 16 vCPUs. One is fresh Centos 7 > >> >> install, the other is fresh Rocky 8. I installed httpd (on Centos 7 > >> >> it's version 2.4.6 and on Rocky 8 it's 2.4.37), configured them to > >> >> point to the same static default html file and enabled mpm event on > >> >> Centos (mpm event is default on Rocky). Then I added following > options > >> >> to default config on both servers: > >> >> ``` > >> >> <IfModule mpm_event_module> > >> >> ThreadsPerChild 25 > >> >> StartServers 3 > >> >> ServerLimit 120 > >> >> MinSpareThreads 75 > >> >> MaxSpareThreads 3000 > >> >> MaxRequestWorkers 3000 > >> >> MaxConnectionsPerChild 0 > >> >> </IfModule> > >> >> ``` > >> >> After this is done I performed ab tests with keepalive using > different > >> >> Centos 7 VM in the same local network. Can you help me understand > >> >> these results? On Centos I am able to complete 1 milion requests at > >> >> 1000 concurrent connections with little to no errors, however with > >> >> version 2.4.37 on Rocky I get a lot of failed requests due to length > >> >> and exceptions. Served content is static so I am assuming this is > >> >> because keepalive connections are closed by the server. I tried > >> >> various configurations of KeeAaliveTimeout, MaxKeepAliveRequests, > >> >> AsyncRequestWorkerFactor and other mpm_event options but I got no > >> >> better results. The only thing that seems to improve the stability of > >> >> keepalive connections is setting threads and servers to the moon for > >> >> example: > >> >> ``` > >> >> <IfModule mpm_event_module> > >> >> ThreadsPerChild 50 > >> >> StartServers 120 > >> >> ServerLimit 120 > >> >> MinSpareThreads 6000 > >> >> MaxSpareThreads 6000 > >> >> MaxRequestWorkers 6000 > >> >> MaxConnectionsPerChild 0 > >> >> </IfModule> > >> >> ``` > >> >> However it stills throws more errors (~8k) than 2.4.6 on the first > set > >> >> of settings. This problem occurs only when using keepalive. There are > >> >> no errors when using ab without -k option, although speed is lower. > I > >> >> can replicate this issue on newest httpd build from source (2.4.57). > >> >> What is causing this difference of behavior? How can I achieve > >> >> performance from 2.4.6 on 2.4.37 / 2.4.57 without throwing much more > >> >> resources at httpd? These errors seems to be a root cause of a > problem > >> >> we have with 502 errors thrown from downstream reverse proxy server > >> >> when httpd kills keepalive connection prematurely and the proxy is > >> >> trying to use this connection. > >> >> > >> >> Below results of ab tests: > >> >> > >> >> Centos 7 VM: > >> >> ``` > >> >> ab -k -t 900 -c 1000 -n 1000000 http://centos/ > >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, > http://www.zeustech.net/ > >> >> Licensed to The Apache Software Foundation, http://www.apache.org/ > >> >> > >> >> Benchmarking 10.1.3.3 (be patient) > >> >> Completed 100000 requests > >> >> Completed 200000 requests > >> >> Completed 300000 requests > >> >> Completed 400000 requests > >> >> Completed 500000 requests > >> >> Completed 600000 requests > >> >> Completed 700000 requests > >> >> Completed 800000 requests > >> >> Completed 900000 requests > >> >> Completed 1000000 requests > >> >> Finished 1000000 requests > >> >> > >> >> > >> >> Server Software: Apache/2.4.6 > >> >> Server Hostname: 10.1.3.3 > >> >> Server Port: 80 > >> >> > >> >> Document Path: / > >> >> Document Length: 7620 bytes > >> >> > >> >> Concurrency Level: 1000 > >> >> Time taken for tests: 15.285 seconds > >> >> Complete requests: 1000000 > >> >> Failed requests: 67 > >> >> (Connect: 0, Receive: 0, Length: 67, Exceptions: 0) > >> >> Write errors: 0 > >> >> Keep-Alive requests: 990567 > >> >> Total transferred: 7919057974 bytes > >> >> HTML transferred: 7619489460 bytes > >> >> Requests per second: 65422.95 [#/sec] (mean) > >> >> Time per request: 15.285 [ms] (mean) > >> >> Time per request: 0.015 [ms] (mean, across all concurrent > requests) > >> >> Transfer rate: 505945.41 [Kbytes/sec] received > >> >> > >> >> Connection Times (ms) > >> >> min mean[+/-sd] median max > >> >> Connect: 0 0 4.7 0 1042 > >> >> Processing: 3 15 16.8 13 467 > >> >> Waiting: 0 14 16.2 13 433 > >> >> Total: 3 15 17.7 13 1081 > >> >> > >> >> Percentage of the requests served within a certain time (ms) > >> >> 50% 13 > >> >> 66% 15 > >> >> 75% 16 > >> >> 80% 17 > >> >> 90% 21 > >> >> 95% 23 > >> >> 98% 32 > >> >> 99% 44 > >> >> 100% 1081 (longest request) > >> >> ``` > >> >> > >> >> Rocky 8 VM: > >> >> ``` > >> >> ab -k -t 900 -c 1000 -n 1000000 http://rocky/ > >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, > http://www.zeustech.net/ > >> >> Licensed to The Apache Software Foundation, http://www.apache.org/ > >> >> > >> >> Benchmarking 10.1.3.11 (be patient) > >> >> Completed 100000 requests > >> >> Completed 200000 requests > >> >> Completed 300000 requests > >> >> Completed 400000 requests > >> >> Completed 500000 requests > >> >> Completed 600000 requests > >> >> Completed 700000 requests > >> >> Completed 800000 requests > >> >> Completed 900000 requests > >> >> Completed 1000000 requests > >> >> Finished 1000000 requests > >> >> > >> >> > >> >> Server Software: Apache/2.4.37 > >> >> Server Hostname: 10.1.3.11 > >> >> Server Port: 80 > >> >> > >> >> Document Path: / > >> >> Document Length: 7620 bytes > >> >> > >> >> Concurrency Level: 1000 > >> >> Time taken for tests: 19.101 seconds > >> >> Complete requests: 1000000 > >> >> Failed requests: 93159 > >> >> (Connect: 0, Receive: 0, Length: 85029, Exceptions: 8130) > >> >> Write errors: 0 > >> >> Keep-Alive requests: 912753 > >> >> Total transferred: 7248228337 bytes > >> >> HTML transferred: 6973694460 bytes > >> >> Requests per second: 52352.12 [#/sec] (mean) > >> >> Time per request: 19.101 [ms] (mean) > >> >> Time per request: 0.019 [ms] (mean, across all concurrent > requests) > >> >> Transfer rate: 370566.51 [Kbytes/sec] received > >> >> > >> >> Connection Times (ms) > >> >> min mean[+/-sd] median max > >> >> Connect: 0 4 54.4 0 3022 > >> >> Processing: 0 15 20.6 13 1400 > >> >> Waiting: 0 13 17.7 12 1400 > >> >> Total: 0 19 59.0 13 3048 > >> >> > >> >> Percentage of the requests served within a certain time (ms) > >> >> 50% 13 > >> >> 66% 15 > >> >> 75% 17 > >> >> 80% 19 > >> >> 90% 24 > >> >> 95% 41 > >> >> 98% 53 > >> >> 99% 106 > >> >> 100% 3048 (longest request) > >> >> ``` > >> >> > >> >> --- > >> >> Mateusz Kempski > >> >> > >> >> --------------------------------------------------------------------- > >> >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > >> >> For additional commands, e-mail: users-h...@httpd.apache.org > >> >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > >> For additional commands, e-mail: users-h...@httpd.apache.org > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > For additional commands, e-mail: users-h...@httpd.apache.org > >
