> On 28 Aug 2017, at 11:27, Julien Charbon <j...@freebsd.org> wrote: > > On 8/28/17 10:25 AM, Ben RUBSON wrote: >>> On 16 Aug 2017, at 11:02, Ben RUBSON <ben.rub...@gmail.com> wrote: >>> >>>> On 15 Aug 2017, at 23:33, Julien Charbon <j...@freebsd.org> wrote: >>>> >>>> On 8/11/17 11:32 AM, Ben RUBSON wrote: >>>>>> On 08 Aug 2017, at 13:33, Julien Charbon <j...@freebsd.org> wrote: >>>>>> >>>>>> On 8/8/17 10:31 AM, Hans Petter Selasky wrote: >>>>>>> >>>>>>> Suggested fix attached. >>>>>> >>>>>> I agree we your conclusion. Just for the record, more precisely this >>>>>> regression seems to have been introduced with: >>>>>> (...) >>>>>> Thus good catch, and your patch looks good. I am going to just verify >>>>>> the other in_pcbrele_wlocked() calls in TCP stack. >>>>> >>>>> Julien, do you plan to make this fix reach 11.0-p12 ? >>>> >>>> I am checking if your issue is another flavor of the issue fixed by: >>>> >>>> https://svnweb.freebsd.org/base?view=revision&revision=307551 >>>> https://reviews.freebsd.org/D8211 >>>> >>>> This fix in not in 11.0 but in 11.1. Currently I did not found how an >>>> inp in INP_TIMEWAIT state can have been INP_FREED without having its tw >>>> set to NULL already except the issue fixed by r307551. >>>> >>>> Thus could you try to apply this patch: >>>> >>>> https://github.com/freebsd/freebsd/commit/acb5bfda99b753d9ead3529d04f20087c5f7d0a0.patch >>>> >>>> and see if you can still reproduce this issue? >>> >>> Thank you for your answer Julien. >>> Unfortunately, I'm not sure at all how to reproduce the issue. >>> I have other servers which are 100% identical to this one, same workload, >>> same some-months uptime, but they did not trigger the bug yet. >>> >>> If other network stack experts (I'm not) agree with your analysis, >>> we could then certainly go further with D8211 / r307551. >>> >>> One thing that perhaps might help : >>> # netstat -an | grep TIME_WAIT$ | wc -l >>> 468 >>> >>> Note that due to this running bug, sendmail has lots of difficulties to >>> send outgoing mails. >>> As soon as I run the above netstat command, I receive a lot of stacked >>> mails (more than 20 this time). >>> As if netstat was able to somehow help... >>> >>> Number of TIME_WAIT connections however does not decrease, but increases. >>> >>>> And in the spirit of r307551 fix and based on Hans patch I will also >>>> propose to add a kernel log describing the issue instead of starting an >>>> infinite loop when INVARIANT is not set. >>> >>> Which should then never be triggered :) >>> Good idea I think ! >> >> What about : >> D8211/r307551 >> + Hans' patch >> + Julien's idea of a kernel log (sort of "We should not be here but we are") > > I did this change and I am testing it
Good news ! > on your side did you try this patch applied on 11.0? > > https://github.com/freebsd/freebsd/commit/acb5bfda99b753d9ead3529d04f20087c5f7d0a0.patch Yes, patch applied and running correctly, however hard to say whether or not it solves this issue, as there is no easy way to reproduce it. >> And backporting all this to 11.0 (and so to 11.1 too) ? >> >> As this bug can impact every FreeBSD machine / server, >> leading to an unavailable / unreachable system (this is how mine ended), >> sounds like it could inevitably be a good thing, for production stability >> purpose. > > The main fix for your issue is (I believe): > > Fix a double-free when an inp transitions to INP_TIMEWAIT state > after having been dropped. > https://svnweb.freebsd.org/base?view=revision&revision=307551 > > This fix has been MFC-ed on both stable/11, stable/10 and is already > included in 11.1 and will be in 10.4. To push in 11.0 release directly, > I guess you have to promote this change to an Errata (never did that > myself): > > https://www.freebsd.org/security/notices.html > https://www.freebsd.org/security/security.html#reporting Mail sent to FreeBSD Security Team ! Many thanks, let's stay tuned ! Ben _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
