andygrove opened a new pull request, #2794: URL: https://github.com/apache/datafusion-ballista/pull/2794
# Which issue does this PR close? No single issue. Split out of #2792. Related: #2520 (performance review tracking). # Rationale for this change A task holds a file per shuffle partition it reads or writes, plus its spill files, so a stage with a few hundred partitions needs thousands of descriptors per executor. With the common soft limit of 1024, TPC-H at 128 or more partitions fails with `Too many open files`. The JVM raises its soft limit to the hard limit at startup, so Spark executors never see this. # What changes are included in this PR? The executor raises `RLIMIT_NOFILE` to the hard limit at startup (capped at 10240 on macOS, which rejects a higher soft limit even when the hard limit is unlimited) and logs the old and new values. Unix only. No test: the call changes process-wide state, so a unit test would alter the limit for every other test in the binary. # Are there any user-facing changes? The executor process starts with a higher open-file soft limit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
