Launchpad has imported 13 comments from the remote bug at https://support.ntp.org/bugs/show_bug.cgi?id=1071.
If you reply to an imported comment from within Launchpad, your comment will be sent to the remote bug automatically. Read more about Launchpad's inter-bugtracker facilities at https://help.launchpad.net/InterBugTracking. ------------------------------------------------------------------------ On 2008-09-19T01:40:42+00:00 Jw-ntp wrote: Hi: We're running into the same problem on a redhat system that was previously reported against ubuntu. I don't think I could write a better bug report, so I'll just include/reference it here: http://www.mail-archive.com/ubuntu-bugs@lists.ubuntu.com/msg858457.html $ uname -a Linux xxxxx 2.6.24-17-generic #1 SMP Thu May 1 13:57:17 UTC 2008 x86_64 GNU/Linux $ lsb_release -rd Description: Ubuntu 8.04 Release: 8.04 $ apt-cache policy ntp ntp: Installed: 1:4.2.4p4+dfsg-3ubuntu2 Candidate: 1:4.2.4p4+dfsg-3ubuntu2 Version table: *** 1:4.2.4p4+dfsg-3ubuntu2 0 500 http://gb.archive.ubuntu.com hardy/main Packages 100 /var/lib/dpkg/status On a system which has any more than 1134 interface addresses (counting both IPv4 and IPv6 address thus: 'ip addr ls | grep inet | wc -l') ntpd fails with a segmentation fault. I run a large network for a university (20,000 people), and we routinely operate open-source routers with several thousand interface addresses (every client machine lives in it's own /30 subnet). Whilst examining the problem I used this script to add interface addresses: $ cat breakit.sh #!/bin/sh set -e dev="vlan252" sudo ifdown ${dev} # clears all old addr naddr=`ip add ls | grep inet | wc -l` # number of addr already on system echo "Pre-existing addresses on system: ${naddr}" : ${nbreakit:=1217} # number of addr required to break ntp sudo ifup ${dev} total=${naddr} for I in `seq 1 5` ; do for J in `seq 0 255` ; do sudo ip addr add 10.0.${I}.${J}/32 dev ${dev} total=$((${total}+1)) if [ ${total} -ge ${nbreakit} ]; then break ; fi done if [ ${total} -ge ${nbreakit} ]; then break ; fi done echo "Added extra addresses: $((${total}-${naddr}))" echo "Total addresses on system now: $(ip add ls | grep inet | wc -l)" exit 0 $ grep -B1 -A2 vlan252 /etc/network/interfaces #auto vlan252 iface vlan252 inet manual vlan-raw-device eth0 I could then add the desired number of interfaces by doing: $ nbreakit=1135 ../breakit.sh and then run ntpd [1][2] under gdb by doing: sudo sh -c "ulimit -n 8192 ; gdb --args /usr/sbin/ntpd -n -d -D3 -p /var/run/ntpd.pid -u 115:126 -g" [1] You can't use '-d -D3' on the standard ubuntu package - debugging is disabled. [2] It is necessary to raise the number of open files ulimit of 1024 in order to run ntpd with this many addresses. In order to help me understand what was going wrong with the standard package, I rebuilt it with debugging enabled and symbols not stripped: $ mkdir work $ cd work/ $ apt-get source ntp Reading package lists... Done Building dependency tree Reading state information... Done NOTICE: 'ntp' packaging is maintained in the 'Svn' version control system at: svn://svn.debian.org/pkg-ntp/ntp/ Need to get 3120kB of source archives. Get: 1 http://gb.archive.ubuntu.com hardy/main ntp 1:4.2.4p4+dfsg-3ubuntu2 (dsc) [1034B] Get: 2 http://gb.archive.ubuntu.com hardy/main ntp 1:4.2.4p4+dfsg-3ubuntu2 (tar) [2835kB] Get: 3 http://gb.archive.ubuntu.com hardy/main ntp 1:4.2.4p4+dfsg-3ubuntu2 (diff) [284kB] Fetched 3120kB in 0s (7433kB/s) dpkg-source: extracting ntp in ntp-4.2.4p4+dfsg dpkg-source: unpacking ntp_4.2.4p4+dfsg.orig.tar.gz dpkg-source: applying ./ntp_4.2.4p4+dfsg-3ubuntu2.diff.gz $ cd ntp-4.2.4p4+dfsg/ $ cp -p debian/rules{,.orig} $ vi debian/rules $ diff -u debian/rules{.orig,} --- debian/rules.orig 2008-05-28 17:52:21.000000000 +0100 +++ debian/rules 2008-05-28 17:53:21.000000000 +0100 @@ -21,7 +21,7 @@ ./configure CFLAGS='$(CFLAGS)' \ --prefix=/usr \ --enable-all-clocks --enable-parse-clocks --enable-SHM \ - --disable-debugging --sysconfdir=/var/lib/ntp \ + --enable-debugging --sysconfdir=/var/lib/ntp \ --with-sntp=no \ --enable-linuxcaps \ --disable-dependency-tracking @@ -104,7 +104,7 @@ dh_installlogcheck -a dh_installchangelogs -a dh_perl -a - dh_strip -a + #dh_strip -a dh_compress -a dh_fixperms -a dh_installdeb -a $ dpkg-buildpackage -us -uc ... dpkg-buildpackage: binary and diff upload (original source NOT included) $ sudo dpkg -i ../ntp_4.2.4p4+dfsg-3ubuntu2_amd64.deb Once one has 1135 interface addresses on the system you get this segmentation fault: 0x000000000040c9d2 in update_interfaces (port=<value optimized out>, receiver=0, data=0x0) at ntp_io.c:769 769 ISC_LIST_UNLINK_TYPE(inter_list, interface, link, struct interface); If one increases the number of interface addresses to 1215 you get a different segmentation fault: update_interfaces (port=<value optimized out>, receiver=0, data=0x0) at ntp_io.c:1325 1325 if (!(interf->flags & (INT_WILDCARD|INT_MCASTIF))) { and if one increases it to 1216 or more you finally get this one: 0x000000000040ba7f in add_interface (interface=0x7d13c0) at ntp_io.c:756 756 ISC_LIST_APPEND(inter_list, interface, link); I had hoped to work around the problem by renaming the devices to 'vlan252:foo', since in ntpd/ntp_io.c:address_okay() the -L flag causes ntpd to ignore "virtual addresses" - this turns out to mean addresses on interfaces which contain a ':' in their name. Unfortunately, since the segmentation fault occurs during interface enumeration (building a linked list of all interface+address) this doesn't help. In the past, earlier versions of ntpd did not use a linked list for this purpose, but rather a fixed-size array of 512. By simply increasing the size of the array I was able to run with large numbers of addresses. It seems to me that, whilst the linked-list code ought to be made to work correctly, it is in fact an unnecessary precaution (and overhead) to bind all addresses on a linux system - only the root user could bind a more specific address than * on port 123, and if one has root the game is over anyway. I haven't submitted this bug to the upstream package maintainer (http://www.ntp.org/bugs.html) partly because firefox won't let me see their bugs site (invalid SSL certificate), and partly because it might be better coming via the distribution. I note that ubuntu has the currently up-to-date version of ntp (4.2.4p4). Reply at: https://bugs.launchpad.net/ntp/+bug/235793/comments/2 ------------------------------------------------------------------------ On 2008-09-19T03:04:56+00:00 Mayer-r wrote: Can you say why you need that many addresses on one system? I haven't looked recently at what's going on but I would recommend that you turn off rescanning by setting -u 0 on the command line. The only way that I can think of that might cause a linked list to fail is if it ran out of memory but then I don't see why that would happen where you say since it would be in the malloc() call. Does BIND9 run okay on this system? Danny Reply at: https://bugs.launchpad.net/ntp/+bug/235793/comments/3 ------------------------------------------------------------------------ On 2008-09-19T03:12:03+00:00 Jw-ntp wrote: Hi Danny: The systems in question are part of an outbound mail farm running eCelerity ( http://messagesystems.com/ ). Clients get one or more ip addresses depending on their volume/rate. I'll see if -u 0 helps. We don't run bind on those boxes, but I can try and see if it starts... Thanks! Reply at: https://bugs.launchpad.net/ntp/+bug/235793/comments/4 ------------------------------------------------------------------------ On 2008-09-19T03:47:55+00:00 Stenn wrote: Subject: Segmentation fault in ntpd when system has more than 1134 interface addresses While I can appreciate that this is a rare case, the bottom line is we seem to have a bug and it should be fixed. We've been meaning to upgrade and use a more "stock" libisc/, perhaps reporting some bugs along the way. This looks like an opportunity... -- Harlan Stenn <st...@ntp.org> http://ntpforum.isc.org - be a member! Reply at: https://bugs.launchpad.net/ntp/+bug/235793/comments/5 ------------------------------------------------------------------------ On 2008-09-19T14:42:28+00:00 Jw-ntp wrote: -u 0 didn't make any difference. We'll try to see if bind starts. We're happy to test a patch, if one comes available. Reply at: https://bugs.launchpad.net/ntp/+bug/235793/comments/6 ------------------------------------------------------------------------ On 2008-12-06T01:46:09+00:00 Norbert-eder wrote: It is clear, that your additional options don't solve the problem. You must use a BIG U I used -U 0 and it solved this Problem. I opened the Bug-ID 1102. bye, Nobsi Reply at: https://bugs.launchpad.net/ntp/+bug/235793/comments/7 ------------------------------------------------------------------------ On 2009-05-07T21:21:38+00:00 Dave Hart wrote: Thanks to a few hours of cooperative debugging with Sandu Adrian <dex...@d3xt3r01.tk> today, the faulty code has been identified. In ntpd/ntp_io.c the last line of add_fd_to_list is FD_SET(fd, &activefds). This (and the preceding maxactivefd maintenance) should be protected by a check that fd < FD_SETSIZE. If you add code to check for that and msyslog a message and then exit, you will no longer crash ntpd starting with many interfaces. Fixing the actual crash is easy. The tougher part is figuring out how we are going to enumerate more interfaces than FD_SETSIZE, in other words, how to defer opening sockets for each until after enumeration, so that some mechanism can be used to select a subset of interfaces less numerous than FD_SETSIZE that ntpd can use. I suggest we immediately add code to terminate with an error if we are about to try to add an fd >= FD_SETSIZE with FD_SET. We could also add code to add_interface to catch corruption of inter_list.head sooner, as it helped to track this issue down. That depends on the first entry in inter_list never being removed but I think that's a safe assumption. Reply at: https://bugs.launchpad.net/ntp/+bug/235793/comments/8 ------------------------------------------------------------------------ On 2009-05-08T01:49:31+00:00 Dave Hart wrote: I have the two fixes described in Comment #6 ready, which fixes the crash reported. Note that ntpd will simply refuse to start with more interfaces than around FD_SETSIZE (1024 on Linux), so this is not a solution to the overall problem, but it does fix the bug that caused corruption and the fault. pogo:~hart/ntp-stable-784-jjy Reply at: https://bugs.launchpad.net/ntp/+bug/235793/comments/9 ------------------------------------------------------------------------ On 2009-05-08T07:51:59+00:00 Stenn wrote: Jeff, Please check 4.2.4p7-RC6 (or later) and mark this bug as VERIFIED or REOPENED, as appropriate. Dave Hart and Sandu, thanks for your work in getting the root cause identified and resolved. Reply at: https://bugs.launchpad.net/ntp/+bug/235793/comments/10 ------------------------------------------------------------------------ On 2009-05-09T04:47:05+00:00 Dave Hart wrote: See Bug #1180 regarding a fix for starting ntpd with more than 1000 interfaces Reply at: https://bugs.launchpad.net/ntp/+bug/235793/comments/11 ------------------------------------------------------------------------ On 2009-05-09T17:22:57+00:00 Mayer-r wrote: Just note that FD_SETSIZE can be any value and it may default 1024 on SOME flavors of Linux and be something totally different on other O/S's but it could be anything if overridden so this check looks right. Danny Reply at: https://bugs.launchpad.net/ntp/+bug/235793/comments/13 ------------------------------------------------------------------------ On 2009-10-15T16:13:23+00:00 Kostecke-8 wrote: Please mark this bug as VERIFIED if you agree that it is fixed. Or reopen it if further work is required. Reply at: https://bugs.launchpad.net/ntp/+bug/235793/comments/18 ------------------------------------------------------------------------ On 2012-08-07T05:27:33+00:00 Stenn wrote: *** Bug 1102 has been marked as a duplicate of this bug. *** Reply at: https://bugs.launchpad.net/ntp/+bug/235793/comments/19 ** Changed in: ntp Importance: Unknown => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/235793 Title: Segmentation fault in ntpd when system has more than 1134 interface addresses To manage notifications about this bug go to: https://bugs.launchpad.net/ntp/+bug/235793/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
