The default open file limit of 1024 is extremely low, given modern resources and kernel architectures. The reason that this hasn't changed change is because doing so would break legacy programs that use the select(2) system call in hard to debug ways. So instead programs that want to opt-in to a higher open file limit are expected to bump their soft limit to their hard limit on startup. Details on this are very well explained in a blogpost by the systemd author[1]. There's also a similar change done by the Go language[2].
So this starts bumping postmaster and pgbench its soft open file limit to the hard open file limit. Doing so is especially useful for the AIO work that Andres is doing, because io_uring consumes a lot of file descriptors. But even without looking at AIO there is a large number of reports from people that require changing their soft file limit before starting postgres, sometimes falling back to lowering max_files_per_process when they fail to do so[3-8]. It's also not all that strange to fail at setting the soft open file limit because there are multiple places where one can configure such limits and usually only one of them is effective (which one depends on how Postgres is started). [1]: https://0pointer.net/blog/file-descriptor-limits.html [2]: https://github.com/golang/go/issues/46279 [3]: https://serverfault.com/questions/785330/getting-too-many-open-files-error-for-postgres [4]: https://serverfault.com/questions/716982/how-to-raise-max-no-of-file-descriptors-for-daemons-running-on-debian-jessie [5]: https://www.postgresql.org/message-id/flat/CAKtc8vXh7NvP_qWj8EqqorPY97bvxSaX3h5u7a9PptRFHW5x7g%40mail.gmail.com [6]: https://www.postgresql.org/message-id/flat/113ce31b0908120955w77029099i7ececc053084095a%40mail.gmail.com [7]: https://github.com/abiosoft/colima/discussions/836 [8]: https://www.postgresql.org/message-id/flat/29663.1007738957%40sss.pgh.pa.us#2079ec9e2d8b251593812a3711bfe9e9
From ecf7fc5f34b4e025eaa4dda809874f18dfdb24b4 Mon Sep 17 00:00:00 2001 From: Jelte Fennema-Nio <github-t...@jeltef.nl> Date: Tue, 11 Feb 2025 19:15:36 +0100 Subject: [PATCH v1] Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup The default open file limit of 1024 is extremely low, given modern resources and kernel architectures. The reason that this hasn't changed change is because doing so would break legacy programs that use the select(2) system call in hard to debug ways. So instead programs that want to opt-in to a higher open file limit are expected to bump their soft limit to their hard limit on startup. Details on this are very well explained in a blogpost by the systemd author[1]. There's also a similar change done by the Go language[2]. So this starts bumping postmaster and pgbench its soft open file limit to the hard open file limit. Doing so is especially useful for the AIO work that Andres is doing, because io_uring consumes a lot of file descriptors. But even without looking at AIO there is a large number of reports from people that require changing their soft file limit before starting postgres, sometimes falling back to lowering max_files_per_process when they fail to do so[3-8]. It's also not all that strange to fail at setting the soft open file limit because there are multiple places where one can configure such limits and usually only one of them is effective (which one depends on how Postgres is started). [1]: https://0pointer.net/blog/file-descriptor-limits.html [2]: https://github.com/golang/go/issues/46279 [3]: https://serverfault.com/questions/785330/getting-too-many-open-files-error-for-postgres [4]: https://serverfault.com/questions/716982/how-to-raise-max-no-of-file-descriptors-for-daemons-running-on-debian-jessie [5]: https://www.postgresql.org/message-id/flat/CAKtc8vXh7NvP_qWj8EqqorPY97bvxSaX3h5u7a9PptRFHW5x7g%40mail.gmail.com [6]: https://www.postgresql.org/message-id/flat/113ce31b0908120955w77029099i7ececc053084095a%40mail.gmail.com [7]: https://github.com/abiosoft/colima/discussions/836 [8]: https://www.postgresql.org/message-id/flat/29663.1007738957%40sss.pgh.pa.us#2079ec9e2d8b251593812a3711bfe9e9 --- src/backend/postmaster/postmaster.c | 22 ++++++++++++++++++++++ src/bin/pgbench/pgbench.c | 5 +++++ 2 files changed, 27 insertions(+) diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index bb22b13adef..3c2fbebfa62 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -72,6 +72,7 @@ #include <ctype.h> #include <sys/stat.h> #include <sys/socket.h> +#include <sys/resource.h> #include <fcntl.h> #include <sys/param.h> #include <netdb.h> @@ -406,6 +407,8 @@ static DNSServiceRef bonjour_sdref = NULL; /* * postmaster.c - function prototypes */ + +static void ConfigureFileLimit(void); static void CloseServerPorts(int status, Datum arg); static void unlink_external_pid_file(int status, Datum arg); static void getInstallationPaths(const char *argv0); @@ -573,6 +576,8 @@ PostmasterMain(int argc, char *argv[]) /* Begin accepting signals. */ sigprocmask(SIG_SETMASK, &UnBlockSig, NULL); + ConfigureFileLimit(); + /* * Options setup */ @@ -1393,6 +1398,23 @@ PostmasterMain(int argc, char *argv[]) abort(); /* not reached */ } +static void +ConfigureFileLimit(void) +{ +#ifdef HAVE_GETRLIMIT + struct rlimit rlim; + + if (getrlimit(RLIMIT_NOFILE, &rlim) != 0) + { + elog(WARNING, "could not get open file limit: %m"); + return; + } + + rlim.rlim_cur = rlim.rlim_max; + if (setrlimit(RLIMIT_NOFILE, &rlim) != 0) + elog(WARNING, "could not set open file limit: %m"); +#endif +} /* * on_proc_exit callback to close server's listen sockets diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c index f303bdeec8d..161b231d591 100644 --- a/src/bin/pgbench/pgbench.c +++ b/src/bin/pgbench/pgbench.c @@ -6794,6 +6794,11 @@ main(int argc, char **argv) #ifdef HAVE_GETRLIMIT if (getrlimit(RLIMIT_NOFILE, &rlim) == -1) pg_fatal("getrlimit failed: %m"); + + rlim.rlim_cur = rlim.rlim_max; + if (setrlimit(RLIMIT_NOFILE, &rlim) == -1) + pg_fatal("getrlimit failed: %m"); + if (rlim.rlim_cur < nclients + 3) { pg_log_error("need at least %d open files, but system limit is %ld", base-commit: c366d2bdba7c3b9b2cca1429d4535866e231ca55 -- 2.43.0
