can you please post top, iotop results for both apache 18/20? On Tue, 23 May 2023, 17:33 Mateusz Kempski, <mkemp...@xtm-intl.com.invalid> wrote:
> On Ubuntu 20 with apache in stock settings + my mpm event config: > ``` > ab -k -t 900 -c 1000 -n 1000000 http://ubuntu20/ > This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ > Licensed to The Apache Software Foundation, http://www.apache.org/ > > Benchmarking 10.1.3.3 (be patient) > Completed 100000 requests > Completed 200000 requests > Completed 300000 requests > Completed 400000 requests > Completed 500000 requests > Completed 600000 requests > Completed 700000 requests > Completed 800000 requests > Completed 900000 requests > Completed 1000000 requests > Finished 1000000 requests > > > Server Software: Apache/2.4.41 > Server Hostname: 10.1.3.3 > Server Port: 80 > > Document Path: / > Document Length: 10918 bytes > > Concurrency Level: 1000 > Time taken for tests: 19.528 seconds > Complete requests: 1000000 > Failed requests: 234371 > (Connect: 0, Receive: 0, Length: 231339, Exceptions: 3032) > Write errors: 0 > Keep-Alive requests: 768663 > Total transferred: 8630779988 bytes > HTML transferred: 8392262634 bytes > Requests per second: 51207.31 [#/sec] (mean) > Time per request: 19.528 [ms] (mean) > Time per request: 0.020 [ms] (mean, across all concurrent requests) > Transfer rate: 431600.59 [Kbytes/sec] received > > Connection Times (ms) > min mean[+/-sd] median max > Connect: 0 7 69.0 0 3022 > Processing: 0 12 22.6 9 2028 > Waiting: 0 6 15.8 4 2023 > Total: 0 19 73.8 9 3038 > > Percentage of the requests served within a certain time (ms) > 50% 9 > 66% 13 > 75% 16 > 80% 19 > 90% 30 > 95% 46 > 98% 56 > 99% 215 > 100% 3038 (longest request) > ``` > > On Ubuntu 18 with apache in stock settings + my mpm event config. > There is ~15 thousands length fails on the first run after apache > restart then it stays in 100-1k range: > ``` > ab -k -t 900 -c 1000 -n 1000000 http://ubuntu18/ > This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ > Licensed to The Apache Software Foundation, http://www.apache.org/ > > Benchmarking 10.1.3.3 (be patient) > Completed 100000 requests > Completed 200000 requests > Completed 300000 requests > Completed 400000 requests > Completed 500000 requests > Completed 600000 requests > Completed 700000 requests > Completed 800000 requests > Completed 900000 requests > Completed 1000000 requests > Finished 1000000 requests > > > Server Software: Apache/2.4.29 > Server Hostname: 10.1.3.3 > Server Port: 80 > > Document Path: / > Document Length: 10918 bytes > > Concurrency Level: 1000 > Time taken for tests: 17.221 seconds > Complete requests: 1000000 > Failed requests: 636 > (Connect: 0, Receive: 0, Length: 636, Exceptions: 0) > Write errors: 0 > Keep-Alive requests: 990158 > Total transferred: 11220454865 bytes > HTML transferred: 10911056152 bytes > Requests per second: 58067.59 [#/sec] (mean) > Time per request: 17.221 [ms] (mean) > Time per request: 0.017 [ms] (mean, across all concurrent requests) > Transfer rate: 636274.18 [Kbytes/sec] received > > Connection Times (ms) > min mean[+/-sd] median max > Connect: 0 0 17.4 0 1020 > Processing: 0 17 11.3 17 377 > Waiting: 0 8 9.0 9 368 > Total: 0 17 21.6 17 1389 > > Percentage of the requests served within a certain time (ms) > 50% 17 > 66% 18 > 75% 18 > 80% 19 > 90% 20 > 95% 23 > 98% 28 > 99% 36 > 100% 1389 (longest request) > ``` > > Mateusz Kempski > On Mon, 22 May 2023 at 13:05, Deepak Goel <deic...@gmail.com> wrote: > > > > > > > > On Mon, 22 May 2023, 17:58 Mateusz Kempski, > <mkemp...@xtm-intl.com.invalid> wrote: > >> > >> @Yann Ylavic: > >> There seems to be no difference in configuration file except for some > comments: > >> ``` > >> diff -Bbde /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.rocky > >> 8a > >> # See the httpd.conf(5) man page for more information on this > configuration, > >> # and httpd.service(8) on using and configuring the httpd service. > >> # > >> . > >> ``` > >> I will paste complete Rocky config for completeness at the end of this > message. > >> EnableMMAP and EnableSendfile are set to on, differently to defaults > >> specified in comments. This is default system configuration present on > >> both Rocky 8 and Centos 7. I removed httpd and all config files and > >> installed it again to make sure. > >> ``` > >> # Defaults if commented: EnableMMAP On, EnableSendfile Off > >> # > >> #EnableMMAP off > >> EnableSendfile on > >> ``` > >> > >> Both systems return: > >> ``` > >> ulimit -n > >> 1024 > >> ``` > >> There is no custom config for httpd: > >> ``` > >> stat /etc/systemd/system/httpd.service.d > >> stat: cannot stat ‘/etc/systemd/system/httpd.service.d’: No such file > >> or directory > >> ``` > >> systemd itself seems to not apply any limit on both systems: > >> ``` > >> grep LimitNOFILE /etc/systemd/system.conf > >> #DefaultLimitNOFILE= > >> ``` > >> > >> After enabling trace there is a lot of messenges about too many open > >> connections and killing keepalive connections. Example snippet: > >> ``` > >> [Mon May 22 11:57:40.146451 2023] [mpm_event:debug] [pid 7710:tid > >> 139800984155904] event.c(1808): Too many open connections (73), not > >> accepting new conns in > >> this process > >> [Mon May 22 11:57:40.146455 2023] [mpm_event:trace1] [pid 7710:tid > >> 139800984155904] event.c(1811): Idle workers: 0 > >> [Mon May 22 11:57:40.146457 2023] [mpm_event:trace1] [pid 7710:tid > >> 139800984155904] event.c(1574): All workers are busy or dying, will > >> close 2 keep-alive co > >> nnections > >> [Mon May 22 11:57:40.146526 2023] [mpm_event:debug] [pid 7379:tid > >> 139801403561728] event.c(492): AH00457: Accepting new connections > >> again: 50 active conns ( > >> 27 lingering/0 clogged/0 suspended), 7 idle workers > >> [Mon May 22 11:57:40.146686 2023] [mpm_event:debug] [pid 7511:tid > >> 139800849938176] event.c(1808): Too many open connections (86), not > >> accepting new conns in > >> this process > >> [Mon May 22 11:57:40.146698 2023] [mpm_event:trace1] [pid 7511:tid > >> 139800849938176] event.c(1811): Idle workers: 11 > >> [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7511:tid > >> 139800849938176] event.c(1574): All workers are busy or dying, will > >> close 34 keep-alive c > >> onnections > >> [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7644:tid > >> 139801336452864] event.c(1574): All workers are busy or dying, will > >> close 2 keep-alive co > >> nnections > >> [Mon May 22 11:57:40.146812 2023] [mpm_event:debug] [pid 7511:tid > >> 139800849938176] event.c(492): AH00457: Accepting new connections > >> again: 85 active conns ( > >> 52 lingering/0 clogged/0 suspended), 6 idle workers > >> [Mon May 22 11:57:40.146955 2023] [mpm_event:debug] [pid 7710:tid > >> 139800984155904] event.c(492): AH00457: Accepting new connections > >> again: 73 active conns ( > >> 58 lingering/0 clogged/0 suspended), 9 idle workers > >> [Mon May 22 11:57:40.148250 2023] [mpm_event:debug] [pid 7162:tid > >> 139800841545472] event.c(492): AH00457: Accepting new connections > >> again: 48 active conns ( > >> 25 lingering/0 clogged/0 suspended), 2 idle workers > >> [Mon May 22 11:57:40.148562 2023] [mpm_event:debug] [pid 7162:tid > >> 139800841545472] event.c(1808): Too many open connections (48), not > >> accepting new conns in > >> this process > >> [Mon May 22 11:57:40.148573 2023] [mpm_event:trace1] [pid 7162:tid > >> 139800841545472] event.c(1811): Idle workers: 0 > >> [Mon May 22 11:57:40.148921 2023] [mpm_event:debug] [pid 7710:tid > >> 139800984155904] event.c(1808): Too many open connections (70), not > >> accepting new conns in > >> this process > >> [Mon May 22 11:57:40.148930 2023] [mpm_event:trace1] [pid 7710:tid > >> 139800984155904] event.c(1811): Idle workers: 1 > >> [Mon May 22 11:57:40.149594 2023] [mpm_event:debug] [pid 7511:tid > >> 139800849938176] event.c(1808): Too many open connections (69), not > >> accepting new conns in > >> this process > >> [Mon May 22 11:57:40.149603 2023] [mpm_event:trace1] [pid 7511:tid > >> 139800849938176] event.c(1811): Idle workers: 1 > >> [Mon May 22 11:57:40.149630 2023] [mpm_event:debug] [pid 7710:tid > >> 139800984155904] event.c(492): AH00457: Accepting new connections > >> again: 61 active conns ( > >> 38 lingering/0 clogged/0 suspended), 2 idle workers > >> [Mon May 22 11:57:40.149776 2023] [mpm_event:debug] [pid 7710:tid > >> 139800984155904] event.c(1808): Too many open connections (63), not > >> accepting new conns in > >> this process > >> [Mon May 22 11:57:40.149782 2023] [mpm_event:trace1] [pid 7710:tid > >> 139800984155904] event.c(1811): Idle workers: 0 > >> [Mon May 22 11:57:40.149882 2023] [mpm_event:debug] [pid 7710:tid > >> 139800984155904] event.c(492): AH00457: Accepting new connections > >> again: 61 active conns ( > >> 38 lingering/0 clogged/0 suspended), 2 idle workers > >> [Mon May 22 11:57:40.149913 2023] [mpm_event:debug] [pid 7511:tid > >> 139800849938176] event.c(492): AH00457: Accepting new connections > >> again: 63 active conns ( > >> 40 lingering/0 clogged/0 suspended), 2 idle workers > >> [Mon May 22 11:57:40.150210 2023] [mpm_event:debug] [pid 7511:tid > >> 139800849938176] event.c(1808): Too many open connections (62), not > >> accepting new conns in > >> this process > >> [Mon May 22 11:57:40.150216 2023] [mpm_event:trace1] [pid 7511:tid > >> 139800849938176] event.c(1811): Idle workers: 0 > >> ``` > >> > >> @Deepak Goel: > >> 1. Here are result with previously mentioned KeepAlive parameters: > >> Rocky 8: > >> ``` > >> ab -k -t 900 -c 1000 -n 1000000 http://rocky/ > >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, > http://www.zeustech.net/ > >> Licensed to The Apache Software Foundation, http://www.apache.org/ > >> > >> Benchmarking 10.1.3.11 (be patient) > >> Completed 100000 requests > >> Completed 200000 requests > >> Completed 300000 requests > >> Completed 400000 requests > >> Completed 500000 requests > >> Completed 600000 requests > >> Completed 700000 requests > >> Completed 800000 requests > >> Completed 900000 requests > >> Completed 1000000 requests > >> Finished 1000000 requests > >> > >> > >> Server Software: Apache/2.4.37 > >> Server Hostname: 10.1.3.11 > >> Server Port: 80 > >> > >> Document Path: / > >> Document Length: 7620 bytes > >> > >> Concurrency Level: 1000 > >> Time taken for tests: 17.287 seconds > >> Complete requests: 1000000 > >> Failed requests: 98218 > >> (Connect: 0, Receive: 0, Length: 88397, Exceptions: 9821) > >> Write errors: 0 > >> Keep-Alive requests: 911603 > >> Total transferred: 7214426142 bytes > >> HTML transferred: 6946414860 bytes > >> Requests per second: 57845.81 [#/sec] (mean) > >> Time per request: 17.287 [ms] (mean) > >> Time per request: 0.017 [ms] (mean, across all concurrent > requests) > >> Transfer rate: 407543.30 [Kbytes/sec] received > >> > >> Connection Times (ms) > >> min mean[+/-sd] median max > >> Connect: 0 4 59.9 0 3015 > >> Processing: 0 13 14.5 13 432 > >> Waiting: 0 13 14.1 13 432 > >> Total: 0 17 63.2 15 3036 > >> > >> Percentage of the requests served within a certain time (ms) > >> 50% 15 > >> 66% 17 > >> 75% 17 > >> 80% 18 > >> 90% 21 > >> 95% 23 > >> 98% 28 > >> 99% 36 > >> 100% 3036 (longest request) > >> ``` > >> Centos 7: > >> ``` > >> ab -k -t 900 -c 1000 -n 1000000 http://10.1.3.3/ > >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, > http://www.zeustech.net/ > >> Licensed to The Apache Software Foundation, http://www.apache.org/ > >> > >> Benchmarking 10.1.3.3 (be patient) > >> Completed 100000 requests > >> Completed 200000 requests > >> Completed 300000 requests > >> Completed 400000 requests > >> Completed 500000 requests > >> Completed 600000 requests > >> Completed 700000 requests > >> Completed 800000 requests > >> Completed 900000 requests > >> Completed 1000000 requests > >> Finished 1000000 requests > >> > >> > >> Server Software: Apache/2.4.6 > >> Server Hostname: 10.1.3.3 > >> Server Port: 80 > >> > >> Document Path: / > >> Document Length: 7620 bytes > >> > >> Concurrency Level: 1000 > >> Time taken for tests: 20.731 seconds > >> Complete requests: 1000000 > >> Failed requests: 14 > >> (Connect: 0, Receive: 0, Length: 14, Exceptions: 0) > >> Write errors: 0 > >> Keep-Alive requests: 999986 > >> Total transferred: 7913889204 bytes > >> HTML transferred: 7619893320 bytes > >> Requests per second: 48236.94 [#/sec] (mean) > >> Time per request: 20.731 [ms] (mean) > >> Time per request: 0.021 [ms] (mean, across all concurrent > requests) > >> Transfer rate: 372794.72 [Kbytes/sec] received > >> > >> Connection Times (ms) > >> min mean[+/-sd] median max > >> Connect: 0 0 13.1 0 1025 > >> Processing: 2 21 7.9 20 821 > >> Waiting: 0 20 7.8 20 821 > >> Total: 2 21 15.7 20 1244 > >> > >> Percentage of the requests served within a certain time (ms) > >> 50% 20 > >> 66% 22 > >> 75% 23 > >> 80% 24 > >> 90% 25 > >> 95% 27 > >> 98% 29 > >> 99% 30 > >> 100% 1244 (longest request) > >> ``` > >> 3. iotop results: > >> Centos 7 idle: > >> ``` > >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s > >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s > >> ``` > >> Rocky 8 idle: > >> ``` > >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s > >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s > >> ``` > >> Centos 7 during test: > >> ``` > >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 4.56 M/s > >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s > >> ``` > >> Rocky 8 during test: > >> ``` > >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 5.35 M/s > > > > > > The DISK WRITE is higher in Rocky as compared to Centos. > > > > Is it possible for you to test on Ubuntu 18/20 as well? ( Seems Rocky > algo's are taking a performance hit ) > >> > >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s > >> ``` > >> I ran top command without parameters on both systems - looks like > >> Centos defaults to KiB and Rocky to MiB. > >> > >> Complete httpd.conf: > >> ``` > >> # > >> # This is the main Apache HTTP server configuration file. It contains > the > >> # configuration directives that give the server its instructions. > >> # See <URL:http://httpd.apache.org/docs/2.4/> for detailed information. > >> # In particular, see > >> # <URL:http://httpd.apache.org/docs/2.4/mod/directives.html> > >> # for a discussion of each configuration directive. > >> # > >> # See the httpd.conf(5) man page for more information on this > configuration, > >> # and httpd.service(8) on using and configuring the httpd service. > >> # > >> # Do NOT simply read the instructions in here without understanding > >> # what they do. They're here only as hints or reminders. If you are > unsure > >> # consult the online docs. You have been warned. > >> # > >> # Configuration and logfile names: If the filenames you specify for many > >> # of the server's control files begin with "/" (or "drive:/" for > Win32), the > >> # server will use that explicit path. If the filenames do *not* begin > >> # with "/", the value of ServerRoot is prepended -- so 'log/access_log' > >> # with ServerRoot set to '/www' will be interpreted by the > >> # server as '/www/log/access_log', where as '/log/access_log' will be > >> # interpreted as '/log/access_log'. > >> > >> # > >> # ServerRoot: The top of the directory tree under which the server's > >> # configuration, error, and log files are kept. > >> # > >> # Do not add a slash at the end of the directory path. If you point > >> # ServerRoot at a non-local disk, be sure to specify a local disk on the > >> # Mutex directive, if file-based mutexes are used. If you wish to > share the > >> # same ServerRoot for multiple httpd daemons, you will need to change at > >> # least PidFile. > >> # > >> ServerRoot "/etc/httpd" > >> > >> # > >> # Listen: Allows you to bind Apache to specific IP addresses and/or > >> # ports, instead of the default. See also the <VirtualHost> > >> # directive. > >> # > >> # Change this to Listen on specific IP addresses as shown below to > >> # prevent Apache from glomming onto all bound IP addresses. > >> # > >> #Listen 12.34.56.78:80 > >> Listen 80 > >> > >> # > >> # Dynamic Shared Object (DSO) Support > >> # > >> # To be able to use the functionality of a module which was built as a > DSO you > >> # have to place corresponding `LoadModule' lines at this location so the > >> # directives contained in it are actually available _before_ they are > used. > >> # Statically compiled modules (those listed by `httpd -l') do not need > >> # to be loaded here. > >> # > >> # Example: > >> # LoadModule foo_module modules/mod_foo.so > >> # > >> Include conf.modules.d/*.conf > >> > >> # > >> # If you wish httpd to run as a different user or group, you must run > >> # httpd as root initially and it will switch. > >> # > >> # User/Group: The name (or #number) of the user/group to run httpd as. > >> # It is usually good practice to create a dedicated user and group for > >> # running httpd, as with most system services. > >> # > >> User apache > >> Group apache > >> > >> # 'Main' server configuration > >> # > >> # The directives in this section set up the values used by the 'main' > >> # server, which responds to any requests that aren't handled by a > >> # <VirtualHost> definition. These values also provide defaults for > >> # any <VirtualHost> containers you may define later in the file. > >> # > >> # All of these directives may appear inside <VirtualHost> containers, > >> # in which case these default settings will be overridden for the > >> # virtual host being defined. > >> # > >> > >> # > >> # ServerAdmin: Your address, where problems with the server should be > >> # e-mailed. This address appears on some server-generated pages, such > >> # as error documents. e.g. ad...@your-domain.com > >> # > >> ServerAdmin root@localhost > >> > >> # > >> # ServerName gives the name and port that the server uses to identify > itself. > >> # This can often be determined automatically, but we recommend you > specify > >> # it explicitly to prevent problems during startup. > >> # > >> # If your host doesn't have a registered DNS name, enter its IP address > here. > >> # > >> #ServerName www.example.com:80 > >> > >> # > >> # Deny access to the entirety of your server's filesystem. You must > >> # explicitly permit access to web content directories in other > >> # <Directory> blocks below. > >> # > >> <Directory /> > >> AllowOverride none > >> Require all denied > >> </Directory> > >> > >> # > >> # Note that from this point forward you must specifically allow > >> # particular features to be enabled - so if something's not working as > >> # you might expect, make sure that you have specifically enabled it > >> # below. > >> # > >> > >> # > >> # DocumentRoot: The directory out of which you will serve your > >> # documents. By default, all requests are taken from this directory, but > >> # symbolic links and aliases may be used to point to other locations. > >> # > >> DocumentRoot "/var/www/html" > >> > >> # > >> # Relax access to content within /var/www. > >> # > >> <Directory "/var/www"> > >> AllowOverride None > >> # Allow open access: > >> Require all granted > >> </Directory> > >> > >> # Further relax access to the default document root: > >> <Directory "/var/www/html"> > >> # > >> # Possible values for the Options directive are "None", "All", > >> # or any combination of: > >> # Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI > MultiViews > >> # > >> # Note that "MultiViews" must be named *explicitly* --- "Options All" > >> # doesn't give it to you. > >> # > >> # The Options directive is both complicated and important. Please > see > >> # http://httpd.apache.org/docs/2.4/mod/core.html#options > >> # for more information. > >> # > >> Options Indexes FollowSymLinks > >> > >> # > >> # AllowOverride controls what directives may be placed in .htaccess > files. > >> # It can be "All", "None", or any combination of the keywords: > >> # Options FileInfo AuthConfig Limit > >> # > >> AllowOverride None > >> > >> # > >> # Controls who can get stuff from this server. > >> # > >> Require all granted > >> </Directory> > >> > >> # > >> # DirectoryIndex: sets the file that Apache will serve if a directory > >> # is requested. > >> # > >> <IfModule dir_module> > >> DirectoryIndex index.html > >> </IfModule> > >> > >> # > >> # The following lines prevent .htaccess and .htpasswd files from being > >> # viewed by Web clients. > >> # > >> <Files ".ht*"> > >> Require all denied > >> </Files> > >> > >> # > >> # ErrorLog: The location of the error log file. > >> # If you do not specify an ErrorLog directive within a <VirtualHost> > >> # container, error messages relating to that virtual host will be > >> # logged here. If you *do* define an error logfile for a <VirtualHost> > >> # container, that host's errors will be logged there and not here. > >> # > >> ErrorLog "logs/error_log" > >> > >> # > >> # LogLevel: Control the number of messages logged to the error_log. > >> # Possible values include: debug, info, notice, warn, error, crit, > >> # alert, emerg. > >> # > >> LogLevel warn > >> > >> <IfModule log_config_module> > >> # > >> # The following directives define some format nicknames for use with > >> # a CustomLog directive (see below). > >> # > >> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" > >> \"%{User-Agent}i\"" combined > >> LogFormat "%h %l %u %t \"%r\" %>s %b" common > >> > >> <IfModule logio_module> > >> # You need to enable mod_logio.c to use %I and %O > >> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" > >> \"%{User-Agent}i\" %I %O" combinedio > >> </IfModule> > >> > >> # > >> # The location and format of the access logfile (Common Logfile > Format). > >> # If you do not define any access logfiles within a <VirtualHost> > >> # container, they will be logged here. Contrariwise, if you *do* > >> # define per-<VirtualHost> access logfiles, transactions will be > >> # logged therein and *not* in this file. > >> # > >> #CustomLog "logs/access_log" common > >> > >> # > >> # If you prefer a logfile with access, agent, and referer information > >> # (Combined Logfile Format) you can use the following directive. > >> # > >> CustomLog "logs/access_log" combined > >> </IfModule> > >> > >> <IfModule alias_module> > >> # > >> # Redirect: Allows you to tell clients about documents that used to > >> # exist in your server's namespace, but do not anymore. The client > >> # will make a new request for the document at its new location. > >> # Example: > >> # Redirect permanent /foo http://www.example.com/bar > >> > >> # > >> # Alias: Maps web paths into filesystem paths and is used to > >> # access content that does not live under the DocumentRoot. > >> # Example: > >> # Alias /webpath /full/filesystem/path > >> # > >> # If you include a trailing / on /webpath then the server will > >> # require it to be present in the URL. You will also likely > >> # need to provide a <Directory> section to allow access to > >> # the filesystem path. > >> > >> # > >> # ScriptAlias: This controls which directories contain server > scripts. > >> # ScriptAliases are essentially the same as Aliases, except that > >> # documents in the target directory are treated as applications and > >> # run by the server when requested rather than as documents sent to > the > >> # client. The same rules about trailing "/" apply to ScriptAlias > >> # directives as to Alias. > >> # > >> ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" > >> > >> </IfModule> > >> > >> # > >> # "/var/www/cgi-bin" should be changed to whatever your ScriptAliased > >> # CGI directory exists, if you have that configured. > >> # > >> <Directory "/var/www/cgi-bin"> > >> AllowOverride None > >> Options None > >> Require all granted > >> </Directory> > >> > >> <IfModule mime_module> > >> # > >> # TypesConfig points to the file containing the list of mappings from > >> # filename extension to MIME-type. > >> # > >> TypesConfig /etc/mime.types > >> > >> # > >> # AddType allows you to add to or override the MIME configuration > >> # file specified in TypesConfig for specific file types. > >> # > >> #AddType application/x-gzip .tgz > >> # > >> # AddEncoding allows you to have certain browsers uncompress > >> # information on the fly. Note: Not all browsers support this. > >> # > >> #AddEncoding x-compress .Z > >> #AddEncoding x-gzip .gz .tgz > >> # > >> # If the AddEncoding directives above are commented-out, then you > >> # probably should define those extensions to indicate media types: > >> # > >> AddType application/x-compress .Z > >> AddType application/x-gzip .gz .tgz > >> > >> # > >> # AddHandler allows you to map certain file extensions to "handlers": > >> # actions unrelated to filetype. These can be either built into the > server > >> # or added with the Action directive (see below) > >> # > >> # To use CGI scripts outside of ScriptAliased directories: > >> # (You will also need to add "ExecCGI" to the "Options" directive.) > >> # > >> #AddHandler cgi-script .cgi > >> > >> # For type maps (negotiated resources): > >> #AddHandler type-map var > >> > >> # > >> # Filters allow you to process content before it is sent to the > client. > >> # > >> # To parse .shtml files for server-side includes (SSI): > >> # (You will also need to add "Includes" to the "Options" directive.) > >> # > >> AddType text/html .shtml > >> AddOutputFilter INCLUDES .shtml > >> </IfModule> > >> > >> # > >> # Specify a default charset for all content served; this enables > >> # interpretation of all content as UTF-8 by default. To use the > >> # default browser choice (ISO-8859-1), or to allow the META tags > >> # in HTML content to override this choice, comment out this > >> # directive: > >> # > >> AddDefaultCharset UTF-8 > >> > >> <IfModule mime_magic_module> > >> # > >> # The mod_mime_magic module allows the server to use various hints > from the > >> # contents of the file itself to determine its type. The > MIMEMagicFile > >> # directive tells the module where the hint definitions are located. > >> # > >> MIMEMagicFile conf/magic > >> </IfModule> > >> > >> # > >> # Customizable error responses come in three flavors: > >> # 1) plain text 2) local redirects 3) external redirects > >> # > >> # Some examples: > >> #ErrorDocument 500 "The server made a boo boo." > >> #ErrorDocument 404 /missing.html > >> #ErrorDocument 404 "/cgi-bin/missing_handler.pl" > >> #ErrorDocument 402 http://www.example.com/subscription_info.html > >> # > >> > >> # > >> # EnableMMAP and EnableSendfile: On systems that support it, > >> # memory-mapping or the sendfile syscall may be used to deliver > >> # files. This usually improves server performance, but must > >> # be turned off when serving from networked-mounted > >> # filesystems or if support for these functions is otherwise > >> # broken on your system. > >> # Defaults if commented: EnableMMAP On, EnableSendfile Off > >> # > >> #EnableMMAP off > >> EnableSendfile on > >> > >> # Supplemental configuration > >> # > >> # Load config files in the "/etc/httpd/conf.d" directory, if any. > >> IncludeOptional conf.d/*.conf > >> > >> <IfModule mpm_event_module> > >> ThreadsPerChild 25 > >> StartServers 3 > >> ServerLimit 120 > >> MinSpareThreads 75 > >> MaxSpareThreads 3000 > >> MaxRequestWorkers 3000 > >> MaxConnectionsPerChild 0 > >> </IfModule> > >> ``` > >> > >> Mateusz Kempski > >> Linux System Administrator > >> XTM International Ltd. > >> Email: mkemp...@xtm-intl.com > >> xtm.cloud > >> > >> > >> > >> > >> On Mon, 22 May 2023 at 11:16, Deepak Goel <deic...@gmail.com> wrote: > >> > > >> > Hi > >> > > >> > 1. Please post the test results completely. (Sorry but, saying there > is no difference does not help). > >> > 2. The memory used in Rocky is comparatively higher than Centos > (About 30Meg or so). Also the buff/cache is approx 10 times higher in Rocky > than in Centos. > >> > 3. Please also post iotop results too. > >> > > >> > P.S: I hope the top commands is run identical for both Rocky and > Centos. > >> > > >> > Deepak > >> > "The greatness of a nation can be judged by the way its animals are > treated - Mahatma Gandhi" > >> > > >> > +91 73500 12833 > >> > deic...@gmail.com > >> > > >> > Facebook: https://www.facebook.com/deicool > >> > LinkedIn: www.linkedin.com/in/deicool > >> > > >> > "Plant a Tree, Go Green" > >> > > >> > Make In India : http://www.makeinindia.com/home > >> > > >> > > >> > > >> > On Mon, May 22, 2023 at 4:28 PM Mateusz Kempski > <mkemp...@xtm-intl.com.invalid> wrote: > >> >> > >> >> I tested again with settings: > >> >> ``` > >> >> KeepAliveTimeout 300 > >> >> MaxKeepAliveRequests 0 > >> >> ``` > >> >> but there was no difference in results barring normal diffs > run-to-run. > >> >> > >> >> Below is the top of top from both servers when idle and during test. > >> >> > >> >> Rocky 8 no load: > >> >> ``` > >> >> top - 10:49:23 up 4 min, 1 user, load average: 3.27, 2.56, 1.10 > >> >> Tasks: 254 total, 2 running, 252 sleeping, 0 stopped, 0 zombie > >> >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 > si, 0.0 st > >> >> MiB Mem : 15824.7 total, 10682.4 free, 283.5 used, 4858.8 > buff/cache > >> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15259.3 > avail Mem > >> >> ``` > >> >> Rocky 8 during test: > >> >> ``` > >> >> top - 10:50:29 up 5 min, 1 user, load average: 4.33, 2.80, 1.28 > >> >> Tasks: 232 total, 2 running, 230 sleeping, 0 stopped, 0 zombie > >> >> %Cpu(s): 13.7 us, 16.9 sy, 0.0 ni, 63.9 id, 0.0 wa, 0.0 hi, 5.4 > si, 0.1 st > >> >> MiB Mem : 15824.7 total, 9863.0 free, 529.3 used, 5432.3 > buff/cache > >> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15012.2 > avail Mem > >> >> ``` > >> >> Centos 7 no load: > >> >> ``` > >> >> top - 10:52:17 up 0 min, 1 user, load average: 0.00, 0.00, 0.00 > >> >> Tasks: 201 total, 1 running, 200 sleeping, 0 stopped, 0 zombie > >> >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 > si, 0.0 st > >> >> KiB Mem : 16264300 total, 15831896 free, 297124 used, 135280 > buff/cache > >> >> KiB Swap: 0 total, 0 free, 0 used. 15720740 > avail Mem > >> >> ``` > >> >> Centos 7 during test: > >> >> ``` > >> >> top - 10:53:21 up 1 min, 1 user, load average: 0.62, 0.16, 0.05 > >> >> Tasks: 218 total, 3 running, 215 sleeping, 0 stopped, 0 zombie > >> >> %Cpu(s): 17.6 us, 18.9 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.0 hi, 3.1 > si, 0.1 st > >> >> KiB Mem : 16264300 total, 14973128 free, 503104 used, 788068 > buff/cache > >> >> KiB Swap: 0 total, 0 free, 0 used. 15459544 > avail Mem > >> >> ``` > >> >> > >> >> On Mon, 22 May 2023 at 10:34, Deepak Goel <deic...@gmail.com> wrote: > >> >> > > >> >> > Hi > >> >> > > >> >> > I can see about 8000+ requests have timed out in 'Rocky'. This is > mostly due to Apache, which is unable to handle the load. Is it possible to > increase the parameter "KeepAliveTimeout" (and other KeepAlive parameters). > >> >> > > >> >> > Is it also possible for you to post the hardware utilisations for > the 2 different servers (Centos & Rocky)? > >> >> > > >> >> > Deepak > >> >> > "The greatness of a nation can be judged by the way its animals > are treated - Mahatma Gandhi" > >> >> > > >> >> > +91 73500 12833 > >> >> > deic...@gmail.com > >> >> > > >> >> > Facebook: https://www.facebook.com/deicool > >> >> > LinkedIn: www.linkedin.com/in/deicool > >> >> > > >> >> > "Plant a Tree, Go Green" > >> >> > > >> >> > Make In India : http://www.makeinindia.com/home > >> >> > > >> >> > > >> >> > On Mon, May 22, 2023 at 3:49 PM Mateusz Kempski > <mkemp...@xtm-intl.com.invalid> wrote: > >> >> >> > >> >> >> Hi all, > >> >> >> I have two identical VMs - 16GB RAM, 16 vCPUs. One is fresh > Centos 7 > >> >> >> install, the other is fresh Rocky 8. I installed httpd (on Centos > 7 > >> >> >> it's version 2.4.6 and on Rocky 8 it's 2.4.37), configured them to > >> >> >> point to the same static default html file and enabled mpm event > on > >> >> >> Centos (mpm event is default on Rocky). Then I added following > options > >> >> >> to default config on both servers: > >> >> >> ``` > >> >> >> <IfModule mpm_event_module> > >> >> >> ThreadsPerChild 25 > >> >> >> StartServers 3 > >> >> >> ServerLimit 120 > >> >> >> MinSpareThreads 75 > >> >> >> MaxSpareThreads 3000 > >> >> >> MaxRequestWorkers 3000 > >> >> >> MaxConnectionsPerChild 0 > >> >> >> </IfModule> > >> >> >> ``` > >> >> >> After this is done I performed ab tests with keepalive using > different > >> >> >> Centos 7 VM in the same local network. Can you help me understand > >> >> >> these results? On Centos I am able to complete 1 milion requests > at > >> >> >> 1000 concurrent connections with little to no errors, however with > >> >> >> version 2.4.37 on Rocky I get a lot of failed requests due to > length > >> >> >> and exceptions. Served content is static so I am assuming this is > >> >> >> because keepalive connections are closed by the server. I tried > >> >> >> various configurations of KeeAaliveTimeout, MaxKeepAliveRequests, > >> >> >> AsyncRequestWorkerFactor and other mpm_event options but I got no > >> >> >> better results. The only thing that seems to improve the > stability of > >> >> >> keepalive connections is setting threads and servers to the moon > for > >> >> >> example: > >> >> >> ``` > >> >> >> <IfModule mpm_event_module> > >> >> >> ThreadsPerChild 50 > >> >> >> StartServers 120 > >> >> >> ServerLimit 120 > >> >> >> MinSpareThreads 6000 > >> >> >> MaxSpareThreads 6000 > >> >> >> MaxRequestWorkers 6000 > >> >> >> MaxConnectionsPerChild 0 > >> >> >> </IfModule> > >> >> >> ``` > >> >> >> However it stills throws more errors (~8k) than 2.4.6 on the > first set > >> >> >> of settings. This problem occurs only when using keepalive. There > are > >> >> >> no errors when using ab without -k option, although speed is > lower. I > >> >> >> can replicate this issue on newest httpd build from source > (2.4.57). > >> >> >> What is causing this difference of behavior? How can I achieve > >> >> >> performance from 2.4.6 on 2.4.37 / 2.4.57 without throwing much > more > >> >> >> resources at httpd? These errors seems to be a root cause of a > problem > >> >> >> we have with 502 errors thrown from downstream reverse proxy > server > >> >> >> when httpd kills keepalive connection prematurely and the proxy is > >> >> >> trying to use this connection. > >> >> >> > >> >> >> Below results of ab tests: > >> >> >> > >> >> >> Centos 7 VM: > >> >> >> ``` > >> >> >> ab -k -t 900 -c 1000 -n 1000000 http://centos/ > >> >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > >> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, > http://www.zeustech.net/ > >> >> >> Licensed to The Apache Software Foundation, > http://www.apache.org/ > >> >> >> > >> >> >> Benchmarking 10.1.3.3 (be patient) > >> >> >> Completed 100000 requests > >> >> >> Completed 200000 requests > >> >> >> Completed 300000 requests > >> >> >> Completed 400000 requests > >> >> >> Completed 500000 requests > >> >> >> Completed 600000 requests > >> >> >> Completed 700000 requests > >> >> >> Completed 800000 requests > >> >> >> Completed 900000 requests > >> >> >> Completed 1000000 requests > >> >> >> Finished 1000000 requests > >> >> >> > >> >> >> > >> >> >> Server Software: Apache/2.4.6 > >> >> >> Server Hostname: 10.1.3.3 > >> >> >> Server Port: 80 > >> >> >> > >> >> >> Document Path: / > >> >> >> Document Length: 7620 bytes > >> >> >> > >> >> >> Concurrency Level: 1000 > >> >> >> Time taken for tests: 15.285 seconds > >> >> >> Complete requests: 1000000 > >> >> >> Failed requests: 67 > >> >> >> (Connect: 0, Receive: 0, Length: 67, Exceptions: 0) > >> >> >> Write errors: 0 > >> >> >> Keep-Alive requests: 990567 > >> >> >> Total transferred: 7919057974 bytes > >> >> >> HTML transferred: 7619489460 bytes > >> >> >> Requests per second: 65422.95 [#/sec] (mean) > >> >> >> Time per request: 15.285 [ms] (mean) > >> >> >> Time per request: 0.015 [ms] (mean, across all concurrent > requests) > >> >> >> Transfer rate: 505945.41 [Kbytes/sec] received > >> >> >> > >> >> >> Connection Times (ms) > >> >> >> min mean[+/-sd] median max > >> >> >> Connect: 0 0 4.7 0 1042 > >> >> >> Processing: 3 15 16.8 13 467 > >> >> >> Waiting: 0 14 16.2 13 433 > >> >> >> Total: 3 15 17.7 13 1081 > >> >> >> > >> >> >> Percentage of the requests served within a certain time (ms) > >> >> >> 50% 13 > >> >> >> 66% 15 > >> >> >> 75% 16 > >> >> >> 80% 17 > >> >> >> 90% 21 > >> >> >> 95% 23 > >> >> >> 98% 32 > >> >> >> 99% 44 > >> >> >> 100% 1081 (longest request) > >> >> >> ``` > >> >> >> > >> >> >> Rocky 8 VM: > >> >> >> ``` > >> >> >> ab -k -t 900 -c 1000 -n 1000000 http://rocky/ > >> >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > >> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, > http://www.zeustech.net/ > >> >> >> Licensed to The Apache Software Foundation, > http://www.apache.org/ > >> >> >> > >> >> >> Benchmarking 10.1.3.11 (be patient) > >> >> >> Completed 100000 requests > >> >> >> Completed 200000 requests > >> >> >> Completed 300000 requests > >> >> >> Completed 400000 requests > >> >> >> Completed 500000 requests > >> >> >> Completed 600000 requests > >> >> >> Completed 700000 requests > >> >> >> Completed 800000 requests > >> >> >> Completed 900000 requests > >> >> >> Completed 1000000 requests > >> >> >> Finished 1000000 requests > >> >> >> > >> >> >> > >> >> >> Server Software: Apache/2.4.37 > >> >> >> Server Hostname: 10.1.3.11 > >> >> >> Server Port: 80 > >> >> >> > >> >> >> Document Path: / > >> >> >> Document Length: 7620 bytes > >> >> >> > >> >> >> Concurrency Level: 1000 > >> >> >> Time taken for tests: 19.101 seconds > >> >> >> Complete requests: 1000000 > >> >> >> Failed requests: 93159 > >> >> >> (Connect: 0, Receive: 0, Length: 85029, Exceptions: 8130) > >> >> >> Write errors: 0 > >> >> >> Keep-Alive requests: 912753 > >> >> >> Total transferred: 7248228337 bytes > >> >> >> HTML transferred: 6973694460 bytes > >> >> >> Requests per second: 52352.12 [#/sec] (mean) > >> >> >> Time per request: 19.101 [ms] (mean) > >> >> >> Time per request: 0.019 [ms] (mean, across all concurrent > requests) > >> >> >> Transfer rate: 370566.51 [Kbytes/sec] received > >> >> >> > >> >> >> Connection Times (ms) > >> >> >> min mean[+/-sd] median max > >> >> >> Connect: 0 4 54.4 0 3022 > >> >> >> Processing: 0 15 20.6 13 1400 > >> >> >> Waiting: 0 13 17.7 12 1400 > >> >> >> Total: 0 19 59.0 13 3048 > >> >> >> > >> >> >> Percentage of the requests served within a certain time (ms) > >> >> >> 50% 13 > >> >> >> 66% 15 > >> >> >> 75% 17 > >> >> >> 80% 19 > >> >> >> 90% 24 > >> >> >> 95% 41 > >> >> >> 98% 53 > >> >> >> 99% 106 > >> >> >> 100% 3048 (longest request) > >> >> >> ``` > >> >> >> > >> >> >> --- > >> >> >> Mateusz Kempski > >> >> >> > >> >> >> > --------------------------------------------------------------------- > >> >> >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > >> >> >> For additional commands, e-mail: users-h...@httpd.apache.org > >> >> >> > >> >> > >> >> --------------------------------------------------------------------- > >> >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > >> >> For additional commands, e-mail: users-h...@httpd.apache.org > >> >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > >> For additional commands, e-mail: users-h...@httpd.apache.org > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > For additional commands, e-mail: users-h...@httpd.apache.org > >
