Launchpad has imported 8 comments from the remote bug at http://bugs.ntp.org/show_bug.cgi?id=2224.
If you reply to an imported comment from within Launchpad, your comment will be sent to the remote bug automatically. Read more about Launchpad's inter-bugtracker facilities at https://help.launchpad.net/InterBugTracking. ------------------------------------------------------------------------ On 2012-06-11T13:37:37+00:00 Ktamateas wrote: Hello everyone, I want to file a bug. Run command: ntpd --user=ntpd:ntpd --logfile=/var/log/ntpd.log It runs for some seconds and then is segfaults. Happens only when I use both the --user and --logfile parameters. Happens only when I have configured it with --enable-clockctl alone. If configured with both --enable-clockctl and --enable-linuxcaps it works OK. /var/log/ntpd.log: -rw-r--r-- 1 ntpd ntpd 21957 Jun 11 14:49 /var/log/ntpd.log /etc/passwd: ntpd:x:10:17:ntpd:/dev/null:/bin/false /etc/group: ntpd:x:1008: /etc/ntpd.conf is empty. Strace gives in the end: http://pastebin.com/Bujn2MNn With more advanced debugging I got: http://pastebin.com/YNWBrRJG When runs in normal manner, strace gives: http://pastebin.com/2JpzK4jh In my humble opinion, the error occurs when ntpd tries to do something with the network interfaces. My machine is a kernel 2.6.35.14 with glibc 2.14.1. ntpd - NTP daemon program - Ver. 4.2.6p5 Greetings. Reply at: https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/0 ------------------------------------------------------------------------ On 2012-06-11T17:10:44+00:00 Dave Hart wrote: Thanks for the report. The additional debugging paste is short enough to include directly in the comments: ==24767== Invalid read of size 8 ==24767== at 0x411048: input_handler (ntp_io.c:3621) ==24767== by 0x414B84: ntpdmain (ntpd.c:1078) ==24767== by 0x406448: main (ntpd.c:356) ==24767== Address 0x5e897f0 is 0 bytes inside a block of size 32 free'd ==24767== at 0x4C26649: free (in /lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==24767== by 0x411072: input_handler (ntp_io.c:3619) ==24767== by 0x414B84: ntpdmain (ntpd.c:1078) ==24767== by 0x406448: main (ntpd.c:356) The code in question is: #ifdef HAS_ROUTING_SOCKET /* * scan list of asyncio readers - currently only used for routing sockets */ asyncio_reader = asyncio_reader_list; while (asyncio_reader != NULL) { if (FD_ISSET(asyncio_reader->fd, &fds)) { ++select_count; (asyncio_reader->receiver)(asyncio_reader); /*3619 */ } asyncio_reader = asyncio_reader->link; /* 3621 */ } #endif /* HAS_ROUTING_SOCKET */ line 3619 is calling process_routing_msgs() which, after root is dropped, is noticing a failed read or other error and removing the entry from asyncio_reader_list and free()ing it, triggering the valgrind catch. I bet can be worked around by adding -U 0 to the command line to disable dynamic interface updates, I suspect (I could be wrong, too). To patch it, we need to add a "next_asyncio_reader" local variable of the same type as asyncio_reader, and assign to it asyncio_reader->link before if (FD_ISSET(..., and change the asyncio_reader assignment to use the saved next_asyncio_reader. I will get that ready for ntp-dev, and am requesting 4.2.6 blocking in case we do another release of that stable version. Reply at: https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/1 ------------------------------------------------------------------------ On 2012-06-11T18:12:06+00:00 Dave Hart wrote: Ready in: ~hart/ntp-dev-2224 Reply at: https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/2 ------------------------------------------------------------------------ On 2012-06-12T06:23:04+00:00 Ktamateas wrote: (In reply to comment #2) > Ready in: > > ~hart/ntp-dev-2224 How can I see the code difference? Reply at: https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/3 ------------------------------------------------------------------------ On 2012-06-12T07:56:53+00:00 Dave Hart wrote: Created attachment 883 pending patch for Bug 2224 With a bit of luck it'll be in 4.2.7p280 before too long. Reply at: https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/4 ------------------------------------------------------------------------ On 2012-06-12T08:24:23+00:00 Ktamateas wrote: (In reply to comment #4) > Created attachment 883 [details] > pending patch for Bug 2224 > > With a bit of luck it'll be in 4.2.7p280 before too long. We consider 4.2.7p XXX stable revisions ? Reply at: https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/5 ------------------------------------------------------------------------ On 2012-06-12T08:50:50+00:00 Dave Hart wrote: (In reply to comment #5) > (In reply to comment #4) > > Created attachment 883 [details] > > pending patch for Bug 2224 > > > > With a bit of luck it'll be in 4.2.7p280 before too long. > > We consider 4.2.7p XXX stable revisions ? Some of we do. I've always preferred ntp-dev snapshots to -stable releases as the -stable releases are years apart and a lot of interesting changes happen in between. Speaking of which, it's been 2.5 years since 4.2.7 forked from 4.2.6. I'm anxious, as are others, to get 4.2.8 out as the next stable release, rather than spend more time patching 4.2.6. Given the relatively small impact of this bug, I'm still inclined to prefer avoiding another 4.2.6 release. Reply at: https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/6 ------------------------------------------------------------------------ On 2012-06-13T05:18:00+00:00 Stenn wrote: nosebleed, Thanks for the report. Please check 4.2.7p280 and mark this bug as VERIFIED or REOPENED, as appropriate. If there is another 4.2.6pX release this patch will be backported. Dave, thanks for your work on this. Reply at: https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/7 ** Changed in: ntp Status: Unknown => Fix Released ** Changed in: ntp Importance: Unknown => High -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to ntp in Ubuntu. https://bugs.launchpad.net/bugs/1481388 Title: NTP : Use-after-free in routing socket code after dropping root Status in NTP: Fix Released Status in ntp package in Ubuntu: Fix Released Status in ntp source package in Precise: In Progress Status in ntp source package in Trusty: In Progress Status in ntp source package in Vivid: In Progress Status in ntp source package in Wily: Fix Released Status in ntp package in Debian: Unknown Bug description: [Impact] * User experienced repeated segfaults at the same instruction pointer i/o error on routing socket No buffer space available - disabling segfault at 31 ip 0000000000000031 sp 00007ffff9f11788 error 14 in libpthread-2.15.so[7f967a5d9000+18000] The remove_ and delete_ functions remove the current element from the asyncio_reader_list, and free it, respectively. We then return back to the loop at the top, wherein the asyncio_reader variable still points at the now-freed element, whose contents are (in theory) now scrambled by having link pointers, etc, from internal malloc state overlaying the data. [Test Case] You can easily reproduce the bug by : - Lowering the sysctl value net.core.rmem_max $ sysctl -w net.core.wmem_max=<LOWER_VALUE> This sets the max OS send buffer size for all types of connections. - Adding multiple network interfaces and static routes. [Regression Potential] None expected since the fix is already available upstream (https://github.com/ntp-project/ntp.git) and Debian package. If after installing the patch, user are receiving this kind of message in /var/log/syslog : "routing socket reports: No buffer space available". The next step, would be to increase the "net.core.rmem_max" and "net.core.wmem_max" values equally until the "routing socket reports: No buffer space available" message no longer showed up. [Other Info] NTP upstream (https://github.com/ntp-project/ntp.git) [Bug 2224] Use-after-free in routing socket code after dropping root. - Commit: d6df9d3 [Bug 2890] Ignore ENOBUFS on routing netlink socket. - Commit: db47bd4 The use-after-free bug has been fix in Debian release (closes: #795315) Will submit the ignore-ENOBUFS-on-routing-netlink-socket in Debian in the next days. [Original Description] We have 1 server (among hundreds) that its ntp service is crashing. A few minute/seconds after a start attempts we can see the following in syslog: ntpd[2729]: peers refreshed ntpd[2729]: Listening on routing socket on fd #49 for interface updates ntpd[2729]: i/o error on routing socket No buffer space available - disabling kernel: [157516.495224] ntpd[2729]: segfault at 31 ip 0000000000000031 sp 00007ffff9f11788 error 14 in libpthread-2.15.so[7f967a5d9000+18000] OS: Ubuntu 12.04.4 LTS Kernel: 3.11.0-19-generic I tried to compare it to other servers, and the only thing I could find that is different is that while it's up (before it crashes) I can see the following when running "lsof | grep ntp": ntpd 2729 ntp 49u sock 0,7 0t0 2473952565 can't identify protocol. To manage notifications about this bug go to: https://bugs.launchpad.net/ntp/+bug/1481388/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
