> On Feb 5, 2019, at 12:56 PM, Iain Barker <iain.bar...@oracle.com> wrote:
>
> Hi everyone,
>
> We just updated our application from DPDK 17.11.4 (LTS) to DPDK 18.11 (LTS)
> and we noticed a regression.
>
> Our host platform is providing 2MB huge pages, so for 8GB reservation this
> means 4000 pages are allocated.
>
> This worked fine in the prior LTS, but after upgrading DPDK what we are
> seeing is that select() on an fd is failing.
>
> select() works fine when the process starts up, but does not work after DPDK
> has been initialized.
>
> We did some investigation and found in the DPDK patches linked below, the
> hugepage tracking mechanism was changed from mmap to an array of file
> descriptors, and the rlimit for fd's is raised from the default to allow more
> fd's to be open.
>
> https://mails.dpdk.org/archives/dev/2018-September/110890.html
> https://mails.dpdk.org/archives/dev/2018-September/110889.html
>
> The problem is that the GNU C library (glibc) has a limit for the maximum fd
> passed to select(), and is hard-coded in the POSIX header file and libc at
> 1024 (and probably many other OS libraries too as a result).
>
> Raising the rlimit for fd >1024 has undefined results, per the manpage:
>
> http://man7.org/linux/man-pages/man2/select.2.html
> An fd_set is a fixed size buffer. Executing FD_CLR() or FD_SET()
> with a value of fd that is negative or is equal to or larger than
> FD_SETSIZE will result in undefined behavior. Moreover, POSIX
> requires fd to be a valid file descriptor.
>
> The Linux kernel allows file descriptor sets of arbitrary size,
> determining the length of the sets to be checked from the value of
> nfds. However, in the glibc implementation, the fd_set type is fixed
> in size.
>
> Specifically, libc's header include/sys/select.h has an array of fd's which
> is FD_SETSIZE deep.
> __fd_mask fds_bits[__FD_SETSIZE / __NFDBITS];
>
> and usr/include/linux/posix_types.h is hard-coded with
> #define __FD_SETSIZE 1024
>
> As this define and array are in libc, they are used in many libraries on a
> Linux system. So to use setsize >1024 means recompiling OS libraries and any
> other package that needs to use FDs, or ensuring that no library used by the
> application ever calls select() on an fd set. That seems an unreasonable
> burden...
>
> Any thoughts?
Would poll work here instead?
>
> thanks,
> Iain
Regards,
Keith
