Re: clang 3.2 RC2 miscompiles libgcc?

[David Chisnall]

On 7 Jan 2013, at 23:21, Dimitry Andric wrote:

> This is at least the direction I'm looking at.  It seems that in some
> cases with __builtin_eh_return(), llvm does not see that registers can
> be clobbered, and it doesn't save and restore them.

Do you mean that some registers were clobbered by a prior call?  
__builtin_eh_return() doesn't return, so whether it clobbers anything or not 
isn't something that should matter.  The preceding call is 
__builtin_frob_return_addr, which seems to be a no-op, so it shouldn't clobber 
any registers either...

David
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: clang 3.2 RC2 miscompiles libgcc?

[Stefan Farfeleder]

On Tue, Jan 08, 2013 at 12:21:12AM +0100, Dimitry Andric wrote:
> On 2013-01-06 17:03, Stefan Farfeleder wrote:
> > On Sun, Jan 06, 2013 at 03:59:59PM +0100, Dimitry Andric wrote:
> ...
> > The bug also affects ports software, e.g., I also experienced strange
> > rtorrent segfaults that are now gone.
> 
> Can you please try the attached patch, which is a very horrid, atrocious
> hack, and will only work for amd64.  Then rebuild libgcc with clang, and
> please try if this fixes at least some of the crashes...
> 
> This is at least the direction I'm looking at.  It seems that in some
> cases with __builtin_eh_return(), llvm does not see that registers can
> be clobbered, and it doesn't save and restore them.
> 
> After a lot of splitting up of unwind-dw2.c, I arrived at _Unwind_Resume
> which when compiled by clang caused the crashes, but when compiled by
> gcc ran OK.

Hi Dimitry,

your patch seems to work just fine. No crashes whatsoever so far. Thank
you.

Stefan
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: clang 3.2 RC2 miscompiles libgcc?

[Dimitry Andric]

On 2013-01-08 09:08, David Chisnall wrote:

On 7 Jan 2013, at 23:21, Dimitry Andric wrote:

This is at least the direction I'm looking at.  It seems that in some
cases with __builtin_eh_return(), llvm does not see that registers can
be clobbered, and it doesn't save and restore them.

Do you mean that some registers were clobbered by a prior call?  
__builtin_eh_return() doesn't return, so whether it clobbers anything or not 
isn't something that should matter.  The preceding call is 
__builtin_frob_return_addr, which seems to be a no-op, so it shouldn't clobber 
any registers either...


No, I mean that gcc seems to take great care in saving and restoring
almost all important registers in a function, if that function contains
a call to __builtin_eh_return.

If you look at expand_eh_return() in contrib/gcc/except.c, you can see
that it sets the special variable 'current_function_calls_eh_return'.

This influences the code generation all over the place, and specifically
the saving of registers in contrib/gcc/config/i386/i386.c:

==
/* Return 1 if we need to save REGNO.  */
static int
ix86_save_reg (unsigned int regno, int maybe_eh_return)
{
  if (pic_offset_table_rtx
  && regno == REAL_PIC_OFFSET_TABLE_REGNUM
  && (regs_ever_live[REAL_PIC_OFFSET_TABLE_REGNUM]
  || current_function_profile
  || current_function_calls_eh_return
  || current_function_uses_const_pool))
{
  if (ix86_select_alt_pic_regnum () != INVALID_REGNUM)
return 0;
  return 1;
}

  if (current_function_calls_eh_return && maybe_eh_return)
{
  unsigned i;
  for (i = 0; ; i++)
{
  unsigned test = EH_RETURN_DATA_REGNO (i);
  if (test == INVALID_REGNUM)
break;
  if (test == regno)
return 1;
}
}
[...]
/* Emit code to save registers in the prologue.  */

static void
ix86_emit_save_regs (void)
{
  unsigned int regno;
  rtx insn;

  for (regno = FIRST_PSEUDO_REGISTER; regno-- > 0; )
if (ix86_save_reg (regno, true))
  {
insn = emit_insn (gen_push (gen_rtx_REG (Pmode, regno)));
RTX_FRAME_RELATED_P (insn) = 1;
  }
}
==

On i386, most registers are touched anyway in _Unwind_Resume, so clang
will already save and restore them.  But on amd64, there are more
registers than local variables, so clang only seems to save a few; not
enough, in any case.  This is why I added the asm statement which
clobbers all those registers, forcing clang to save and restore them.

This fixes most of the crashes I was able to reproduce.  I think I still
have another unrelated issue in libgcc with clang, but this only occurs
when compiling the testcases with gcc 4.7, and very high optimization.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [RFC/RFT] calloutng

[Alexander Motin]

On 06.01.2013 17:23, Marius Strobl wrote:
> On Wed, Dec 26, 2012 at 09:24:46PM +0200, Alexander Motin wrote:
>> On 26.12.2012 01:21, Marius Strobl wrote:
>>> On Tue, Dec 18, 2012 at 11:03:47AM +0200, Alexander Motin wrote:
 Experiments with dummynet shown ineffective support for very short
 tick-based callouts. New version fixes that, allowing to get as many
 tick-based callout events as hz value permits, while still be able to
 aggregate events and generating minimum of interrupts.

 Also this version modifies system load average calculation to fix some
 cases existing in HEAD and 9 branches, that could be fixed with new
 direct callout functionality.

 http://people.freebsd.org/~mav/calloutng_12_17.patch

 With several important changes made last time I am going to delay commit
 to HEAD for another week to do more testing. Comments and new test cases
 are welcome. Thanks for staying tuned and commenting.
>>>
>>> FYI, I gave both calloutng_12_15_1.patch and calloutng_12_17.patch a
>>> try on sparc64 and it at least survives a buildworld there. However,
>>> with the patched kernels, buildworld times seem to increase slightly but
>>> reproducible by 1-2% (I only did four runs but typically buildworld
>>> times are rather stable and don't vary more than a minute for the
>>> same kernel and source here). Is this an expected trade-off (system
>>> time as such doesn't seem to increase)?
>>
>> I don't think build process uses significant number of callouts to 
>> affect results directly. I think this additional time could be result of 
>> the deeper next event look up, done by the new code, that is practically 
>> useless for sparc64, which effectively has no cpu_idle() routine. It 
>> wouldn't affect system time and wouldn't show up in any statistics 
>> (except PMC or something alike) because it is executed inside timer 
>> hardware interrupt handler. If my guess is right, that is a part that 
>> probably still could be optimized. I'll look on it. Thanks.
>>
>>> Is there anything specific to test?
>>
>> Since the most of code is MI, for sparc64 I would mostly look on related 
>> MD parts (eventtimers and timecounters) to make sure they are working 
>> reliably in more stressful conditions.  I still have some worries about 
>> possible deadlock on hardware where IPIs are used to fetch present time 
>> from other CPU.
> 
> Well, I've just learnt two things the hard way:
> a) We really need the mutex in that path.
> b) Assuming that the initial synchronization of the counters is good
>enough and they won't drift considerably accross the CPUs so we can
>always use the local one makes things go south pretty soon after
>boot. At least with your calloutng_12_26.patch applied.

Do you think it means they are not really synchronized for some reason?

> I'm not really sure what to do about that. Earlier you already said
> that sched_bind(9) also isn't an option in case if td_critnest > 1.
> To be honest, I don't really unerstand why using a spin lock in the
> timecounter path makes sparc64 the only problematic architecture
> for your changes. The x86 i8254_get_timecount() also uses a spin lock
> so it should be in the same boat.

The problem is not in using spinlock, but in waiting for other CPU while
spinlock is held. Other CPU may also hold spinlock and wait for
something, causing deadlock. i8254 code uses spinlock just to atomically
access hardware registers, so it causes no problems.

> The affected machines are equipped with a x86-style south bridge
> which exposes a powermanagment unit (intended to be used as a SMBus
> bridge only in these machines) on the PCI bus. Actually, this device
> also includes an ACPI power management timer. However, I've just
> spent a day trying to get that one working without success - it
> just doesn't increment. Probably its clock input isn't connected as
> it's not intended to be used in these machines.
> That south bridge also includes 8254 compatible timers on the ISA/
> LPC side, but are hidden from the OFW device tree. I can hack these
> devices into existence and give it a try, but even if that works this
> likely would use the same code as the x86 i8254_get_timecount() so I
> don't see what would be gained with that.
> 
> The last thing in order to avoid using the tick counter as timecounter
> in the MP case I can think of is that the Broadcom MACs in the affected
> machines also provide a counter driven by a 1 MHz clock. If that's good
> enough for a timecounter I can hook these up (in case these work ...)
> and hack bge(4) to not detach in that case (given that we can't detach
> timecounters ...).

i8254 on x86 is also just a bit above 1MHz.

>> Here is small tool we are using for test correctness and performance of 
>> different user-level APIs: http://people.freebsd.org/~mav/testsleep.c
>>
> 
> I've run Ian's set of tests on a v215 with and without your
> calloutng_12_26.patch and on a v210 (these uses the

Re: [RFC/RFT] calloutng

[Alexander Motin]

On 06.01.2013 18:20, Luigi Rizzo wrote:
> On Sun, Jan 06, 2013 at 04:23:13PM +0100, Marius Strobl wrote:
>> On Wed, Dec 26, 2012 at 09:24:46PM +0200, Alexander Motin wrote:
>>> Here is small tool we are using for test correctness and performance of 
>>> different user-level APIs: http://people.freebsd.org/~mav/testsleep.c
>>>
>>
>> I've run Ian's set of tests on a v215 with and without your
>> calloutng_12_26.patch and on a v210 (these uses the IPI approach)
>> with the latter also applied.
>> I'm not really sure what to make out of the numbers.
>>
>>v215 w/o v215 w/  v210 w/ 
>> --   
>> select  1   1999.61  1 23.87  1 29.97
>> poll1   1999.70  1   1069.61  1   1075.24
>> usleep  1   1999.86  1 23.43  1 28.99
>> nanosleep   1999.92  1 23.28  1 28.66
>> kqueue  1   1000.12  1   1071.13  1   1076.35
>> kqueueto1999.56  1 26.33  1 31.34
>> syscall 1  1.89  1  1.92  1  2.88
>> select300   1999.72300326.08300332.24
>> poll  300   1999.12300   1069.78300   1075.82
>> usleep300   1999.91300325.63300330.94
>> nanosleep 300999.82300 23.25300 26.76
>> kqueue300   1000.14300   1071.06300   1075.96
>> kqueueto  300999.53300 26.32300 31.42
>> syscall   300  1.90300  1.93300  2.89
>> select   3000   3998.18   3000   3176.51   3000   3193.86
>> poll 3000   3999.29   3000   3182.21   3000   3193.12
>> usleep   3000   3998.46   3000   3191.60   3000   3192.50
>> nanosleep3000   1999.79   3000 23.21   3000 27.02
>> kqueue   3000   3000.12   3000   3189.13   3000   3191.96
>> kqueueto 3000   1999.99   3000 26.28   3000 31.91
>> syscall  3000  1.91   3000  1.91   3000  2.90
>> select  3  30990.85  3  31489.18  3  31548.77
>> poll3  30995.25  3  31518.80  3  31487.92
>> usleep  3  30992.00  3  31510.42  3  31475.50
>> nanosleep   3   1999.46  3 38.67  3 41.95
>> kqueue  3  30006.49  3  30991.86  3  30996.77
>> kqueueto3   1999.09  3 41.67  3 46.36
>> syscall 3  1.91  3  1.91  3  2.88
>> select 30 300990.65 30 301864.98 30 301787.01
>> poll   30 300998.09 30 301831.36 30 301741.62
>> usleep 30 300990.80 30 301824.67 30 301770.10
>> nanosleep  30   1999.15 30325.74 30331.01
>> kqueue 30 30.87 30 301031.11 30 300992.28
>> kqueueto   30   1999.39 30328.77 30333.45
>> syscall30  1.91 30  1.91 30  2.88
> 
> the nanosleep and kqueueto tests are probably passing the wrong
> argument to the syscall (meant to be microseconds, but nanosleep
> takes nanosecond so it should probably be multiplied by 1000).

Yes, you are right. I've used it in such way to see what will happen if
I request sub-microsecond interval. In this test these rows are
definitely incorrect.

> I think that for the time being it would be useful to run at least
> one set of tests with kern.timecounter.alloweddeviation=0 so we can
> tell how close we get to the required timeouts

May be just to be sure, because it should not significantly affect
results of the 1us tests, as 5% of 1us is much less then numbers we see
there.

-- 
Alexander Motin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [RFC/RFT] calloutng

[Luigi Rizzo]

On Tue, Jan 08, 2013 at 12:55:27PM +0200, Alexander Motin wrote:
> On 06.01.2013 18:20, Luigi Rizzo wrote:
...
> > I think that for the time being it would be useful to run at least
> > one set of tests with kern.timecounter.alloweddeviation=0 so we can
> > tell how close we get to the required timeouts
> 
> May be just to be sure, because it should not significantly affect
> results of the 1us tests, as 5% of 1us is much less then numbers we see
> there.

to clarify - i don't mind if we are 50-100us (absolute error) off the
requested timeout for short intervals, but i want to be sure
that this error can be achieved also for large requests.

cheers
luigi
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: sysctl -a causes kernel trap 12

[Rick Macklem]

d wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> On 1/7/13 5:33 PM, Brandon Gooch wrote:
> > On Mon, Jan 7, 2013 at 6:09 PM, Xin Li  > > wrote:
> >
> > -BEGIN PGP SIGNED MESSAGE- Hash: SHA256
> >
> > On 01/07/13 16:02, Konstantin Belousov wrote:
> >> On Mon, Jan 07, 2013 at 03:54:23PM -0800, Xin Li wrote:
> >>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA256
> >>>
> >>> Hi,
> >>>
> >>> I've recently (by mid-December I think) noticed that sysctl -a
> >>> can sometimes cause kernel trap 12. Tried enabling INVARIANTS
> >>> and the problem mysteriously disappeared. After some
> >>> experiments on this, it seems that this can be triggered by
> >>> sysctl -a but the system have an 1 in 10 chance to survive.
> >>> When INVARIANTS is enabled however, I can not trigger the
> >>> panic. "sysctl hw" triggers the panic sometimes, but not
> >>> always.
> >>>
> >>> Do anybody have clue on this? The system hangs hard when it
> >>> panics so kernel debugger won't work. When it panics, the
> >>> fault instruction pointer is always 0x20:0x808d61c9,
> >>> which is sys/kern/subr_turnstile.c:297:
> >>>
>  /* Resort td on the list if needed. */ if
>  (!turnstile_adjust_thread(ts, td)) {
>  mtx_unlock_spin(&ts->ts_lock); // 297 // return; }
> >>>
> >>> This sounds like a race condition but I haven't yet able to
> >>> track it down...
> >>
> >> Could you try to isolate the sub-leaf under hw which causes the
> >> panic ? Just shot in the dark, do you have Intel GPU gemified
> >> driver loaded ?
> >
> > It seems that it was not hw itself that causes the problem (I
> > thought about isolating to sub-leaf and at one point believed it
> > was hw.acpi.battery but doing so repeatedly doesn't panic the
> > system, with or without INVARIANTS; doing 'sysctl hw' sometimes
> > causes panic but not always).
> >
> > The laptop is running nVidia but it's using i7-3610QM which does
> > have Intel GPU, but I have not loaded drm2.ko and the panic is
> > reproducible in single user mode.
> >
> > Cheers, - -- Xin LI  > > https://www.delphij.net/ FreeBSD -
> > The Power to Serve! Live free or die
> >
> >
> > Xin, did you compile the NVIDIA driver port with clang? I was
> > having this issue until I set an exception for the NVIDIA driver
> > port to be built with GCC.
> >
> > Actually, I just remembered (and checked), and what I'm actually
> > doing is setting CFLAGS for the NVIDIA driver port to -O0, and
> > building with clang. Clang optimizes out important bits, I didn't
> > investigate deeply enough to determine what was actually going
> > awry...
> 
> Yes my nVidia driver is compiled with clang. I'll try gcc tomorrow,
> thanks for the pointer.
> 
> (Note that the panic was a NULL pointer deference and it's a race
> condition so I doubt if it's compiler related but will try).
> 
I saw this crash once, on an i386 (No X windows setup) with just
the GENERIC kernel config + "options DEBIG_VFS_LOCKS" built with gcc,
so I don't think it would be caused by the NVIDIA driver.

Just one more data point, rick

> Cheers,
> 
> -BEGIN PGP SIGNATURE-
> 
> iQEcBAEBCAAGBQJQ64cNAAoJEG80Jeu8UPuzkLwIAKgjgbcBaow64mWzrJ0XtczM
> 6Xi4mC0Oq5kdomx0y+Vgks/qTvq28Ff/JqyxN8osD959Vwaq0vbQumFiDtbWCr0h
> /MD0w5m7Asdbf8KzJE80il90Vv1/nJrOVdwj3qoNIqtKIomi12CHDKL5zFs2Ja2i
> rscN22SBh8+3vV1rBSE6NGzJ7jPSs/B0o73YuIdwj6bUYMiBqHWWGGTfFStNhc4U
> 1JX6WWsDdiWxNvGPH5lE3HelSgya+WCltD+B6/8mYaByBQbXD+JBOA9AvX97h2kk
> muQROSMOz6OVdGmtM6U/19Bsiv/8kUtItJF2k19Y/eTQEohH/2xBhq+37EG8cqg=
> =b9UF
> -END PGP SIGNATURE-
> ___
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscr...@freebsd.org"
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

-iface option to route(8) is broken

[Ian FREISLICH]

Hi

Adding routes using the -iface option to route(8) doesn't work any
more.  This was useful to select a specific interface for a route
when the remote gateway has the same IP address.  Are there plans
to restore this functionality?

tun0: flags=8051 metric 0 mtu 1492
options=8
inet 41.135.82.18 --> 41.135.70.1 netmask 0x 
Opened by PID 2387

[router] ~ # route add 41.154.2.53 -iface tun0
route: bad address: tun0

It also doesn't work on ngX PPP interfaces.

ng1: flags=88d1 metric 0 mtu 
1490
inet 197.87.27.111 --> 41.135.70.1 netmask 0x 
ng2: flags=88d1 metric 0 mtu 
1490
inet 41.135.82.120 --> 41.135.70.1 netmask 0x 

[router] ~ # route add 41.154.2.53 -iface ng2 
route: bad address: ng2
[router] ~ # route add 41.154.2.53 -interface ng2
route: bad address: ng2

Ian

-- 
Ian Freislich
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: clang 3.2 RC2 miscompiles libgcc?

[Борис Самородов]

08.01.2013 03:21, Dimitry Andric пишет:

> Can you please try the attached patch, which is a very horrid, atrocious
> hack, and will only work for amd64.  Then rebuild libgcc with clang, and
> please try if this fixes at least some of the crashes...

The patch fixed building editors/libreoffice. Without this patch the
port segfaulted while doing cppunittest. The system:
-
% uname -a
FreeBSD BB051.wart.ru 10.0-CURRENT FreeBSD 10.0-CURRENT #0 r245047: Sat
Jan  5 03:19:00 SAMT 2013
b...@bb051.wart.ru:/usr/obj/usr/src/sys/BB64X  amd64

% clang --version
FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
Target: x86_64-unknown-freebsd10.0
Thread model: posix
-

Thanks!
-- 
WBR, Boris Samorodov (bsam)
FreeBSD Committer, http://www.FreeBSD.org The Power To Serve
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

standards/173421: [patch] strptime() accepts formats that should be rejected

[Fabian Keil]

Anyone interested in a PR for a strptime() bug coming with
a test case and a potential fix?

> http://www.freebsd.org/cgi/query-pr.cgi?pr=173421
> 
> >Category:   standards
> >Responsible:freebsd-standards
> >Synopsis:   [patch] strptime() accepts formats that should be rejected
> >Arrival-Date:   Tue Nov 06 14:30:00 UTC 2012

Looks like freebsd-standards@ is not the most active mailing list.

Fabian


signature.asc
Description: PGP signature

Re: -iface option to route(8) is broken

[Julian Elischer]

On 1/8/13 6:42 AM, Ian FREISLICH wrote:

Hi

Adding routes using the -iface option to route(8) doesn't work any
more.  This was useful to select a specific interface for a route
when the remote gateway has the same IP address.  Are there plans
to restore this functionality?


I agree.. thus was a crucial piece of functionality..

who broke it?




tun0: flags=8051 metric 0 mtu 1492
 options=8
 inet 41.135.82.18 --> 41.135.70.1 netmask 0x
 Opened by PID 2387

[router] ~ # route add 41.154.2.53 -iface tun0
route: bad address: tun0

It also doesn't work on ngX PPP interfaces.

ng1: flags=88d1 metric 0 mtu 
1490
 inet 197.87.27.111 --> 41.135.70.1 netmask 0x
ng2: flags=88d1 metric 0 mtu 
1490
 inet 41.135.82.120 --> 41.135.70.1 netmask 0x

[router] ~ # route add 41.154.2.53 -iface ng2
route: bad address: ng2
[router] ~ # route add 41.154.2.53 -interface ng2
route: bad address: ng2

Ian



___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: -iface option to route(8) is broken

[Hiroki Sato]

Julian Elischer  wrote
  in <50ec51a3.6020...@freebsd.org>:

ju> On 1/8/13 6:42 AM, Ian FREISLICH wrote:
ju> > Hi
ju> >
ju> > Adding routes using the -iface option to route(8) doesn't work any
ju> > more.  This was useful to select a specific interface for a route
ju> > when the remote gateway has the same IP address.  Are there plans
ju> > to restore this functionality?
ju>
ju> I agree.. thus was a crucial piece of functionality..
ju>
ju> who broke it?

 Oh, sorry, it seems I accidentally broke it in the previous commit.
 It should be fixed now.  Please let me know if it still does not
 work.

-- Hiroki


pgpc4BlRPFooR.pgp
Description: PGP signature

Re: -iface option to route(8) is broken

[Julian Elischer]

On 1/8/13 9:28 AM, Hiroki Sato wrote:

Julian Elischer  wrote
   in <50ec51a3.6020...@freebsd.org>:

ju> On 1/8/13 6:42 AM, Ian FREISLICH wrote:
ju> > Hi
ju> >
ju> > Adding routes using the -iface option to route(8) doesn't work any
ju> > more.  This was useful to select a specific interface for a route
ju> > when the remote gateway has the same IP address.  Are there plans
ju> > to restore this functionality?
ju>
ju> I agree.. thus was a crucial piece of functionality..
ju>
ju> who broke it?

  Oh, sorry, it seems I accidentally broke it in the previous commit.
  It should be fixed now.  Please let me know if it still does not
  work.


thankyou



-- Hiroki


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"