Re: [review] sendfile / sendfile_sync refactoring

2013-11-28 Thread Konstantin Belousov 
On Wed, Nov 27, 2013 at 11:36:44AM -0800, Adrian Chadd wrote:
> Hi,
> 
> Here's part #2 in my sendfile refactoring. This is a little more
> intrusive than the first patch.
> 
> http://people.freebsd.org/~adrian/netflix/20131126-sendfile-sync-refactor-2.diff
> 
sf_buf.h is for sf buffers, and not for sendfile stuff.  Please do not
pollute this header.  The changes you have to make to if_ti.c illustrate
my point.

The sys/event.h change of adding new kevent type is useless now and 
has no relation to the rest of the patch.

Patch has too many XXX notes for its triviality, some of which are
baseless, e.g. the comment for struct sendfile_sync forward declaration
in sys/file.h.

You extracted all sfs accesses from the sendfile implementation, except
the one in sf_buf_mext().  This is weird, please move the code from
sf_buf_mext() into static function and put it near sf_sync_ref().

While moving the code into sf_sync_syscall_wait(), please use the
opportunity and replace the if (sfs->count != 0) with the while ()
cv_wait(); loop, to avoid spurious wakeups.

I do not see any use for sfs->flags. Also, I do not see any use
for passing the flags masked with SF_SYNC, which is always set, to
sf_sync_alloc().  If the flags are going to be useful later, it should
be added when needed later.

The sf_sync_* stuff in the compat32 code just duplicates the main
syscall.  It should be done in the common code.

> My aim here is to move the sendfile_sync stuff out of uipc_syscalls.c
> and out of sendfile-only code and into something that can be reused
> elsewhere or replaced later on. I've created methods for all the
> sendfile_sync stuff, I've moved it out of the do_sendfile() loop so
> do_sendfile() doesn't specifically implement the completion behaviour.

As is, the patch is sincere nop. Modulo the comments above, I do not
object against it.


pgp_ieq0LfEKW.pgp
Description: PGP signature

Re: Feature request: sticky bit inheritance

2013-11-28 Thread Harald Schmalzbauer 
 Bezüglich Julian Elischer's Nachricht vom 28.11.2013 01:05 (localtime):
> On 11/28/13, 12:17 AM, Harald Schmalzbauer wrote:
>>   Bezüglich Julian Elischer's Nachricht vom 27.11.2013 16:20
>> (localtime):
>>> On 11/27/13, 8:03 PM, Harald Schmalzbauer wrote:
Hello,

 ever since I took a FreeBSD machine into production, acting as any
 kind
 of file server, I have to work arround the problem, that write
 access to
 a directory implies unlinking (deleting) directory contents.
>>> not sure I fully understand what you mean by that..
>>> Do you mean write access implies delete access? yes..
>>>
>>> This can be modified with the nounlink flag.
>> The uunlink flags also prohibits the owner to delete his files as far as
>> I know. I want to prohibt users from deleting “foreign” files, even if
>> the user has write access to the parent directory (and I wanted to
>> explain that I don't understand why anybody would want that a user with
>> write access to a directory can delete files on which the user doesn't
>> have write access).
>
> You can always unlink a file that is not yours if you own the directory.
> because the ability to unlink is purely dependent on the directory.
> You don't change the file, and it may in fact have other links

I have an idea why this kind of permission ist default: It's more
expensive to extra check the file permission copmpared to only check the
directory permission, the only part which will be altered any way. I
guess having the sticky bit set by default would cause extra I/O+check,
which might have been too expensive in the past… So the default was to
do as less work as needed?!?


...
>> I'd need every child directory of directories, who have the sticky bit
>> set, also to have the sticky bit. The same behaviour as with the gid –
>> it's the same as the parent has for new directories.
> "patches accepted" :-)

Besides horrible C skills, I have no idea where and how to start :-(
I hoped somebody else with deeper knowledge is also suffering badly and
someone could at least estimate the effort (in hours) needed to
implement a inhert-stickybit kernconf option, or even better, a sysctl.
Maybe I can pay for it.

Thanks,

-Harry



signature.asc
Description: OpenPGP digital signature

Re: [review] sendfile / sendfile_sync refactoring

2013-11-28 Thread Adrian Chadd 
Hi,

It's definitely a work in progress, as you've observed.

On 28 November 2013 00:20, Konstantin Belousov  wrote:
> On Wed, Nov 27, 2013 at 11:36:44AM -0800, Adrian Chadd wrote:
>> Hi,
>>
>> Here's part #2 in my sendfile refactoring. This is a little more
>> intrusive than the first patch.
>>
>> http://people.freebsd.org/~adrian/netflix/20131126-sendfile-sync-refactor-2.diff
>>
> sf_buf.h is for sf buffers, and not for sendfile stuff.  Please do not
> pollute this header.  The changes you have to make to if_ti.c illustrate
> my point.

Yup. I don't like having it in here either. I may create a new header
file specifically for this.

> The sys/event.h change of adding new kevent type is useless now and
> has no relation to the rest of the patch.

Yeah, ignore that bit. I'm working on adding code for that.

>
> Patch has too many XXX notes for its triviality, some of which are
> baseless, e.g. the comment for struct sendfile_sync forward declaration
> in sys/file.h.
>
> You extracted all sfs accesses from the sendfile implementation, except
> the one in sf_buf_mext().  This is weird, please move the code from
> sf_buf_mext() into static function and put it near sf_sync_ref().

I plan on doing that soon.

> While moving the code into sf_sync_syscall_wait(), please use the
> opportunity and replace the if (sfs->count != 0) with the while ()
> cv_wait(); loop, to avoid spurious wakeups.
>
> I do not see any use for sfs->flags. Also, I do not see any use
> for passing the flags masked with SF_SYNC, which is always set, to
> sf_sync_alloc().  If the flags are going to be useful later, it should
> be added when needed later.

It's just precursor work to add support for other SF_* flags -
specifically, a new kqueue notification flag for completion.

> The sf_sync_* stuff in the compat32 code just duplicates the main
> syscall.  It should be done in the common code.

The main motivation for moving the sendfile_sync setup and teardown
out of do_sendfile() is so do_sendfile() can keep a very little/light
idea of the behaviour of sendfile_sync. I don't mind keeping the code
in there.

Thanks for your feedback. I'll post an updated diff when I've fleshed
out some more of this.



-a
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [HEADSUP!!!] do not upgrade to or past r258632 if you use ZFS + TRIM

2013-11-28 Thread Andriy Gapon 
on 27/11/2013 18:17 Andriy Gapon said the following:
> on 26/11/2013 11:57 Andriy Gapon said the following:
>> Author: avg
>> Date: Tue Nov 26 09:57:14 2013
>> New Revision: 258632
>> URL: http://svnweb.freebsd.org/changeset/base/258632
>>
>> Log:
>>   MFV r255255: 4045 zfs write throttle & i/o scheduler performance work
>>   
>>   illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
>>   
>>   Please note the following changes:
>>   - zio_ioctl has lost its priority parameter and now TRIM is executed
>> with 'now' priority
>>   - some knobs are gone and some new knobs are added; not all of them are
>> exposed as tunables / sysctls yet
>>   
>>   MFC after: 10 days
>>   Sponsored by:  HybridCluster [merge]
> 
> I think that I've introduced a very serious bug when merging this change.
> Please do not upgrade to this revision if you use ZFS with SSDs and have TRIM
> support enabled.
> 
> If you have already upgraded, please disable TRIM support ASAP and roll back 
> to
> a previous version of kernel and then check integrity of your pools.

The issue should be fixed in r258704.
Yes, the bug was that simple and that serious.
My apologies to all who were affected.

-- 
Andriy Gapon
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: panic: double fault with 11.0-CURRENT r258504

2013-11-28 Thread Don Lewis 
On 28 Nov, Konstantin Belousov wrote:
> On Wed, Nov 27, 2013 at 01:11:35PM -0800, Don Lewis wrote:
>> On 27 Nov, Konstantin Belousov wrote:
>> > On Wed, Nov 27, 2013 at 11:35:19AM -0800, Don Lewis wrote:
>> >> On 27 Nov, Konstantin Belousov wrote:
>> >> > On Wed, Nov 27, 2013 at 11:02:57AM -0800, Don Lewis wrote:
>> >> >> On 27 Nov, Konstantin Belousov wrote:
>> >> >> > On Wed, Nov 27, 2013 at 10:33:30AM -0800, Don Lewis wrote:
>> >> >> >> On 27 Nov, Konstantin Belousov wrote:
>> >> >> >> > On Wed, Nov 27, 2013 at 09:41:36AM -0800, Don Lewis wrote:
>> >> >> >> >> On 27 Nov, Konstantin Belousov wrote:
>> >> >> >> >> > On Wed, Nov 27, 2013 at 02:49:12AM -0800, Don Lewis wrote:
>> >> >> >> >> >> 
>> >> >> >> >> > 
>> >> >> >> >> > What is the instruction at cpu_switch+0x9b ?
>> >> >> >> >> 
>> >> >> >> >> movl 0x8(%edx),%eax
>> >> >> >> > So it is line 176 in swtch.s. Is machine still in ddb, or did you
>> >> >> >> > obtained the core ? If yes, please print out the content of words 
>> >> >> >> > at
>> >> >> >> > 0xe4f62bb0 + 4, +8 (*), +16. Please print the content of the word 
>> >> >> >> > at
>> >> >> >> > address (*) + 8.
>> >> >> >> 
>> >> >> >> It is still in ddb.
>> >> >> >> 
>> >> >> >> , though not 
>> >> >> >> in
>> >> >> >> the above order.
>> >> >> > Uhm, sorry, I mistyped the last part of the instructions.
>> >> >> > 
>> >> >> > The new thread pointer is 0xd2f4e000, there is nothing incriminating.
>> >> >> > Please print the word at 0xd2f4e000+0x254 == 0xd2f4e254, which would 
>> >> >> > be
>> >> >> > the address of the new thread pcb. It is load from the pcb + 8 which
>> >> >> > faults.
>> >> >> 
>> >> >> 0xf3d44d60
>> >> > Again, the pointer looks fine, and its tail is 0xd60, which is correct 
>> >> > for
>> >> > the pcb offset in the last page of the thread stack.
>> >> > 
>> >> > Please do 'show thread 0xd2f4e000' before trying below instructions.
>> >> 
>> >> Ok, see below:
>> >>  
>> >> > What happens if you try to read word at 0xf3d44d68 ?
>> >> 
>> >> Nothing bad ...
>> >> 
>> >> 
>> >> 
>> > So the thread structure looks sane, the stack region is in place where
>> > it is supposed to be, all the gathered data looks self-consistent. And,
>> > the access to the faulted address from ddb does not fault.
>> > 
>> > Thread stacks can only be invalidated when the process is swapped out and
>> > kernel stack is written to swap.  Your thread flags indicate that it is
>> > in memory, and TDF_CANSWAP is not set.  I do not believe that our swapout
>> > code would invalidate stack mapping in such situation, otherwise we would
>> > have too many complaints already.
>> > 
>> > Just in case, do you use swap on this box ?
>> 
>> I do.
>> 
>> > And, as the last resort, I do understand that this sounds as giving up,
>> > do you monitor the temperature of the CPUs ? BTW, which CPUs are that,
>> > please show the cpu identification lines from the boot dmesg.
>> 
>> I don't monitor the temperature, but I do hear the CPU fan speed ramping
>> up and down when I'm building ports like this.  Even though I'm pretty
>> much keeping one core busy the whole time, the temperature must drop
>> enough at times to let the fan speed drop.
>> 
>> I can run math/mprime on this machine for a while to see if anything
>> shows up.  I also have a very similar machine (same motherboard but
>> different CPU) that I can move the drive over to and test.
>> 
>> Here's the full dmesg.boot:
>> 
>> Copyright (c) 1992-2013 The FreeBSD Project.
>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>>  The Regents of the University of California. All rights reserved.
>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>> FreeBSD 11.0-CURRENT #63 r258614M: Tue Nov 26 00:29:01 PST 2013
>> d...@scratch.catspoiler.org:/usr/obj/usr/src/sys/GENERICSMB i386
>> FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
>> WARNING: WITNESS option enabled, expect reduced performance.
>> CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 4800+ (2500.06-MHz 686-class 
>> CPU)
>>   Origin = "AuthenticAMD"  Id = 0x60fb1  Family = 0xf  Model = 0x6b  
>> Stepping = 1
>>   
>> Features=0x178bfbff
>>   Features2=0x2001
>>   AMD Features=0xea500800
>>   AMD Features2=0x11f
> 
> The errata list for the Athlon 64 X2 is quite long.  Do you have latest
> BIOS ?  I am not sure if AMD provides standalone firmware update blocks
> for their CPUs.  If any Linux distribution ships updates for AMD CPUs,
> it might be useful to load the update with cpucontrol(8).  Even if we
> do not hit a CPU bug, it would provide me with more certainity that we
> are not chasing ghost.

I haven't figured out how to find the currently installed BIOS version.
The motherboard is Abit, which is no more, but I found an archive of all
of their downloads.  I'll also check into updates fro

Re: panic: double fault with 11.0-CURRENT r258504

2013-11-28 Thread Konstantin Belousov 
On Thu, Nov 28, 2013 at 12:56:37AM -0800, Don Lewis wrote:
> I haven't figured out how to find the currently installed BIOS version.
> The motherboard is Abit, which is no more, but I found an archive of all
> of their downloads.  I'll also check into updates from the Linux world.
I reviewed the 0xF family revision guide, and stumbled upon errata 122,
"TLB Flush Filter May Cause Coherency Problem in Multicore Systems".
You could try the patch at the end of message.

> This machine gets updated every month or so and I've never had stability
> problems with it until just recently.  It's definitely been using clang
> for quite a while without any problems other than the ports mess.
Is it possible to bisect ?

diff --git a/sys/amd64/amd64/initcpu.c b/sys/amd64/amd64/initcpu.c
index 34a362d..87e3f69 100644
--- a/sys/amd64/amd64/initcpu.c
+++ b/sys/amd64/amd64/initcpu.c
@@ -88,6 +88,11 @@ static void
 init_amd(void)
 {
 
+   if (CPUID_TO_FAMILY(cpu_id) == 0x9) {
+   if ((cpu_feature2 & CPUID2_HV) == 0)
+   wrmsr(MSR_HWCR, rdmsr(MSR_HWCR) | (1 << 6));
+   }
+
/*
 * Work around Erratum 721 for Family 10h and 12h processors.
 * These processors may incorrectly update the stack pointer
diff --git a/sys/i386/i386/initcpu.c b/sys/i386/i386/initcpu.c
index 71c57b2..662ea8e 100644
--- a/sys/i386/i386/initcpu.c
+++ b/sys/i386/i386/initcpu.c
@@ -651,6 +651,32 @@ init_transmeta(void)
 }
 #endif
 
+static void
+init_amd(void)
+{
+
+#ifdef CPU_ATHLON_SSE_HACK
+   /*
+* Sometimes the BIOS doesn't enable SSE instructions.
+* According to AMD document 20734, the mobile Duron, the
+* (mobile) Athlon 4 and the Athlon MP support SSE. These
+* correspond to cpu_id 0x66X or 0x67X.
+*/
+   if ((cpu_feature & CPUID_XMM) == 0 && ((cpu_id & ~0xf) == 0x660 ||
+   (cpu_id & ~0xf) == 0x670 || (cpu_id & ~0xf) == 0x680)) {
+   u_int regs[4];
+
+   wrmsr(MSR_HWCR, rdmsr(MSR_HWCR) & ~0x08000);
+   do_cpuid(1, regs);
+   cpu_feature = regs[3];
+   }
+#endif
+   if (CPUID_TO_FAMILY(cpu_id) == 0x9) {
+   if ((cpu_feature2 & CPUID2_HV) == 0)
+   wrmsr(MSR_HWCR, rdmsr(MSR_HWCR) | (1 << 6));
+   }
+}
+
 /*
  * Initialize CR4 (Control register 4) to enable SSE instructions.
  */
@@ -725,26 +751,9 @@ initializecpu(void)
break;
}
break;
-#ifdef CPU_ATHLON_SSE_HACK
case CPU_VENDOR_AMD:
-   /*
-* Sometimes the BIOS doesn't enable SSE instructions.
-* According to AMD document 20734, the mobile
-* Duron, the (mobile) Athlon 4 and the Athlon MP
-* support SSE. These correspond to cpu_id 0x66X
-* or 0x67X.
-*/
-   if ((cpu_feature & CPUID_XMM) == 0 &&
-   ((cpu_id & ~0xf) == 0x660 ||
-(cpu_id & ~0xf) == 0x670 ||
-(cpu_id & ~0xf) == 0x680)) {
-   u_int regs[4];
-   wrmsr(MSR_HWCR, rdmsr(MSR_HWCR) & ~0x08000);
-   do_cpuid(1, regs);
-   cpu_feature = regs[3];
-   }
+   init_amd();
break;
-#endif
case CPU_VENDOR_CENTAUR:
init_via();
break;


pgpPoMHiK0OZf.pgp
Description: PGP signature

Re: panic: double fault with 11.0-CURRENT r258504

2013-11-28 Thread Peter Holm 
On Thu, Nov 28, 2013 at 09:56:10AM +0200, Konstantin Belousov wrote:
> On Wed, Nov 27, 2013 at 01:11:35PM -0800, Don Lewis wrote:
> > On 27 Nov, Konstantin Belousov wrote:
> > > On Wed, Nov 27, 2013 at 11:35:19AM -0800, Don Lewis wrote:
> > >> On 27 Nov, Konstantin Belousov wrote:
> > >> > On Wed, Nov 27, 2013 at 11:02:57AM -0800, Don Lewis wrote:
> > >> >> On 27 Nov, Konstantin Belousov wrote:
> > >> >> > On Wed, Nov 27, 2013 at 10:33:30AM -0800, Don Lewis wrote:
> > >> >> >> On 27 Nov, Konstantin Belousov wrote:
> > >> >> >> > On Wed, Nov 27, 2013 at 09:41:36AM -0800, Don Lewis wrote:
> > >> >> >> >> On 27 Nov, Konstantin Belousov wrote:
> > >> >> >> >> > On Wed, Nov 27, 2013 at 02:49:12AM -0800, Don Lewis wrote:
> > >> >> >> >> >> 
> > >> >> >> >> > 
> > >> >> >> >> > What is the instruction at cpu_switch+0x9b ?
> > >> >> >> >> 
> > >> >> >> >> movl 0x8(%edx),%eax
> > >> >> >> > So it is line 176 in swtch.s. Is machine still in ddb, or did you
> > >> >> >> > obtained the core ? If yes, please print out the content of 
> > >> >> >> > words at
> > >> >> >> > 0xe4f62bb0 + 4, +8 (*), +16. Please print the content of the 
> > >> >> >> > word at
> > >> >> >> > address (*) + 8.
> > >> >> >> 
> > >> >> >> It is still in ddb.
> > >> >> >> 
> > >> >> >> , though not 
> > >> >> >> in
> > >> >> >> the above order.
> > >> >> > Uhm, sorry, I mistyped the last part of the instructions.
> > >> >> > 
> > >> >> > The new thread pointer is 0xd2f4e000, there is nothing 
> > >> >> > incriminating.
> > >> >> > Please print the word at 0xd2f4e000+0x254 == 0xd2f4e254, which 
> > >> >> > would be
> > >> >> > the address of the new thread pcb. It is load from the pcb + 8 which
> > >> >> > faults.
> > >> >> 
> > >> >> 0xf3d44d60
> > >> > Again, the pointer looks fine, and its tail is 0xd60, which is correct 
> > >> > for
> > >> > the pcb offset in the last page of the thread stack.
> > >> > 
> > >> > Please do 'show thread 0xd2f4e000' before trying below instructions.
> > >> 
> > >> Ok, see below:
> > >>  
> > >> > What happens if you try to read word at 0xf3d44d68 ?
> > >> 
> > >> Nothing bad ...
> > >> 
> > >> 
> > >> 
> > > So the thread structure looks sane, the stack region is in place where
> > > it is supposed to be, all the gathered data looks self-consistent. And,
> > > the access to the faulted address from ddb does not fault.
> > > 
> > > Thread stacks can only be invalidated when the process is swapped out and
> > > kernel stack is written to swap.  Your thread flags indicate that it is
> > > in memory, and TDF_CANSWAP is not set.  I do not believe that our swapout
> > > code would invalidate stack mapping in such situation, otherwise we would
> > > have too many complaints already.
> > > 
> > > Just in case, do you use swap on this box ?
> > 
> > I do.
> > 
> > > And, as the last resort, I do understand that this sounds as giving up,
> > > do you monitor the temperature of the CPUs ? BTW, which CPUs are that,
> > > please show the cpu identification lines from the boot dmesg.
> > 
> > I don't monitor the temperature, but I do hear the CPU fan speed ramping
> > up and down when I'm building ports like this.  Even though I'm pretty
> > much keeping one core busy the whole time, the temperature must drop
> > enough at times to let the fan speed drop.
> > 
> > I can run math/mprime on this machine for a while to see if anything
> > shows up.  I also have a very similar machine (same motherboard but
> > different CPU) that I can move the drive over to and test.
> > 
> > Here's the full dmesg.boot:
> > 
> > Copyright (c) 1992-2013 The FreeBSD Project.
> > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> > The Regents of the University of California. All rights reserved.
> > FreeBSD is a registered trademark of The FreeBSD Foundation.
> > FreeBSD 11.0-CURRENT #63 r258614M: Tue Nov 26 00:29:01 PST 2013
> > d...@scratch.catspoiler.org:/usr/obj/usr/src/sys/GENERICSMB i386
> > FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
> > WARNING: WITNESS option enabled, expect reduced performance.
> > CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 4800+ (2500.06-MHz 686-class 
> > CPU)
> >   Origin = "AuthenticAMD"  Id = 0x60fb1  Family = 0xf  Model = 0x6b  
> > Stepping = 1
> >   
> > Features=0x178bfbff
> >   Features2=0x2001
> >   AMD Features=0xea500800
> >   AMD Features2=0x11f
> 
> The errata list for the Athlon 64 X2 is quite long.  Do you have latest
> BIOS ?  I am not sure if AMD provides standalone firmware update blocks
> for their CPUs.  If any Linux distribution ships updates for AMD CPUs,
> it might be useful to load the update with cpucontrol(8).  Even if we
> do not hit a CPU bug, it would provide me with more certainity that we
> are not chasing ghost.
> 
> Another things to try, in vain, is to compile kernel wi

Re: panic: double fault with 11.0-CURRENT r258504

2013-11-28 Thread Don Lewis 
On 28 Nov, Konstantin Belousov wrote:
> On Thu, Nov 28, 2013 at 12:56:37AM -0800, Don Lewis wrote:
>> I haven't figured out how to find the currently installed BIOS version.
>> The motherboard is Abit, which is no more, but I found an archive of all
>> of their downloads.  I'll also check into updates from the Linux world.
> I reviewed the 0xF family revision guide, and stumbled upon errata 122,
> "TLB Flush Filter May Cause Coherency Problem in Multicore Systems".
> You could try the patch at the end of message.

I'll give it a try after my current test run, but it will probably be a
few days before I have results.

>> This machine gets updated every month or so and I've never had stability
>> problems with it until just recently.  It's definitely been using clang
>> for quite a while without any problems other than the ports mess.
> Is it possible to bisect ?

Yes, though that could take quite a while since each iteration will take
about two days.

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

[RFC] how to get the size of a malloc(9) block ?

2013-11-28 Thread Luigi Rizzo 
in porting some linux kernel code to FreeBSD we
stumbled upon ksize(), which returns the
actual size of a kmalloc() block.

We could easily implement it as the first part
of realloc(9) -- see kern/kern_malloc.c

Would it make sense to add this to the malloc(9) API ?
The userspace equivalent seems to be
malloc_usable_size(3) which according to the
manpage appeared in FreeBSD 7.0

cheers
luigi
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [RFC] how to get the size of a malloc(9) block ?

2013-11-28 Thread jb 
Luigi Rizzo  iet.unipi.it> writes:

> 
> in porting some linux kernel code to FreeBSD we
> stumbled upon ksize(), which returns the
> actual size of a kmalloc() block.
> 
> We could easily implement it as the first part
> of realloc(9) -- see kern/kern_malloc.c
> 
> Would it make sense to add this to the malloc(9) API ?
> The userspace equivalent seems to be
> malloc_usable_size(3) which according to the
> manpage appeared in FreeBSD 7.0

Hi,

The implementation of ksize() depends on (has non-standard behavior across)
architectures, memory allocators, page sizes, C libs, etc.

It means, ksize() exports its implementation details to the caller, and this
disqualifies it, regardless whether in kernel or user spaces.

This leads to dangerous and conflicting assertions:

malloc_usable_size(3) on Linux:
NOTES
   The value returned by malloc_usable_size()  may  be  greater  than  the
   requested  size of the allocation because of alignment and minimum size
   constraints.  Although the excess  bytes  can  be  overwritten  by  the
   application without ill effects, this is not good programming practice:
   the number of excess bytes in an allocation depends on  the  underlying
   implementation.

   The main use of this function is for debugging and introspection.

Other sources:
...
The caller may use this additional memory, even though a smaller amount of
memory was initially specified with the kmalloc call.
...

This is hair-raising:
http://lwn.net/Articles/319686/

Result ? The same code works here, but may not elsewhere.
It follows, you should remove malloc_usable_size() from user space instead.

jb


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [RFC] how to get the size of a malloc(9) block ?

2013-11-28 Thread Luigi Rizzo 
On Thu, Nov 28, 2013 at 01:33:41PM +, jb wrote:
> Luigi Rizzo  iet.unipi.it> writes:
> 
> > 
> > in porting some linux kernel code to FreeBSD we
> > stumbled upon ksize(), which returns the
> > actual size of a kmalloc() block.
> > 
> > We could easily implement it as the first part
> > of realloc(9) -- see kern/kern_malloc.c
> > 
> > Would it make sense to add this to the malloc(9) API ?
> > The userspace equivalent seems to be
> > malloc_usable_size(3) which according to the
> > manpage appeared in FreeBSD 7.0
> 
> Hi,
> 
> The implementation of ksize() depends on (has non-standard behavior across)
> architectures, memory allocators, page sizes, C libs, etc.
> 
> It means, ksize() exports its implementation details to the caller, and this
> disqualifies it, regardless whether in kernel or user spaces.
> 
> This leads to dangerous and conflicting assertions:
> 
> malloc_usable_size(3) on Linux:
> NOTES
>The value returned by malloc_usable_size()  may  be  greater  than  the
>requested  size of the allocation because of alignment and minimum size
>constraints.  Although the excess  bytes  can  be  overwritten  by  the
>application without ill effects, this is not good programming practice:
>the number of excess bytes in an allocation depends on  the  underlying
>implementation.
> 
>The main use of this function is for debugging and introspection.

This is the same exact text we have in the FreeBSD manpage,
and exactly the behaviour of ksize() in the linux kernel
(and exactly the semantics that our realloc(9) relies upon).

While I personally prefer that applications call realloc() directly,
there might be cases where the decision is best left to the application
(or kernel code): e.g. say you would like to get some extra space,
but would rather do without it than risk a malloc() failure or
having to sleep for the allocation.

And there is also a (minor) issue of portability of code.

But I don't understand why you find ksize()/malloc_usable_size() dangerous.

Of course the result depends on the underlying implementation,
but this happens in a ton of places including compiler behaviour
and system calls.
The danger is when the programmer makes assumptions that do not match
reality, but the purpose of this call seems to be exactly the opposite:
avoid making assumptions.

cheers
luigi
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [RFC] how to get the size of a malloc(9) block ?

2013-11-28 Thread jb 
Luigi Rizzo  iet.unipi.it> writes:

> ... 
> But I don't understand why you find ksize()/malloc_usable_size() dangerous.
> ...

The original crime is commited when *usable size* (an implementation detail)
is exported (leaked) to the caller.
To be blunt, when a caller requests memory of certain size, and its request is
satisfied, then it is not its business to learn details beyond that (and they
should not be offered as well).
The API should be sanitized, in kernel and user space.
Otherwise, all kind of charlatans will try to play hair-raising games with it.
If the caller wants to track the *requested size* programmatically, it is its
business to do it and it can be done very easily.

Some of these guys got it perfectly right:
http://stackoverflow.com/questions/5813078/is-it-possible-to-find-the-memory-allocated-to-the-pointer-without-searching-fo

jb





___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: lang/gcc not build

2013-11-28 Thread Volodymyr Kostyrko 

26.11.2013 19:47, Alexander Panyushkin wrote:

#portmaster lang/gcc
[...]
cd /usr/ports/lang/gcc/work/gcc-4.6.4 ; contrib/gcc_update --touch
configure: loading site script /usr/ports/Templates/config.site
checking build system type... x86_64-portbld-freebsd10.0
checking host system type... x86_64-portbld-freebsd10.0
checking target system type... x86_64-portbld-freebsd10.0
checking for a BSD-compatible install... /usr/bin/install -c -o root -g
wheel
checking whether ln works... yes
checking whether ln -s works... yes
checking for a sed that does not truncate output... (cached) /usr/bin/sed
checking for gawk... (cached) /usr/bin/awk
checking for gcc... gcc46
checking for C compiler default output file name...
configure: error: in `/usr/ports/lang/gcc/work/build':
configure: error: C compiler cannot create executables
See `config.log' for more details.
===>  Script "../gcc-4.6.4/configure" failed unexpectedly.
Please report the problem to ger...@freebsd.org [maintainer] and attach the
"/usr/ports/lang/gcc/work/build/config.log" including the output of the
failure
of your make command. Also, it might be a good idea to provide an overview
of all packages installed on your system (e.g. a /usr/local/sbin/pkg-static
info -g -Ea).
*** Error code 1

#
/etc/make.conf

..if ${.CURDIR:N*/ports/lang/gcc*} == ""
CFLAGS= -O2 -march=athlon64-sse3 -mtune=athlon64-sse3 -pipe -Wformat
CPPFLAGS+= -D_FORTIFY_SOURCE=2
USE_GCC=4.6
..endif


Added this to my make.conf and tested - WFM. The problem is not there. 
Can you post a sample of /usr/ports/lang/gcc/work/build/config.log as 
log suggests?


PS: CFLAGS= is a bit rough thing to have, CFLAGS+= would be much better.

PPS: clang is mature enough and works correctly with CPUTYPE?=native so 
-march and -mtune can be substituted for that one.


--
Sphinx of black quartz, judge my vow.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

problems with pkg2ng

2013-11-28 Thread Nilton Jose Rizzo 


Hi all,

 in the near past, when I update ports repository via svn, I always use
 pkg2ng to upgrade (or update?) the database, but today I can not do it.

 The command pkg2ng show this error message:

root@valfenda:/home2/rizzo # pkg2ng
pkg: PACKAGESITE in pkg.conf is deprecated. Please create a repository
configuration file
Converting packages from /var/db/pkg
pkg: PACKAGESITE in pkg.conf is deprecated. Please create a repository
configuration file
pkg: fstat() failed for(/usr/local/etc/cups/cupsd.conf.N): No such file or
directory

but when I search in /usr/ports/UPDATING I can not found anything about this
and no man pages about pkg2ng

root@valfenda:/home2/rizzo # man pkg2ng
No manual entry for pkg2ng
root@valfenda:/home2/rizzo #

Can Someone help me?

MyBox:

root@valfenda:/home2/rizzo # pkg -v
1.2_1
root@valfenda:/home2/rizzo # svn info /usr/src
Caminho: /usr/src
Working Copy Root Path: /usr/src
URL: svn://svn.freebsd.org/base/head
Relative URL: ^/head
Raiz do Repositório: svn://svn.freebsd.org/base
UUID do repositório: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Revisão: 258666
Tipo de Nó: diretório
Agendado: normal
Autor da Última Mudança: glebius
Revisão da Última Mudança: 258666
Data da Última Mudança: 2013-11-26 18:27:57 -0200 (Ter, 26 Nov 2013)

root@valfenda:/home2/rizzo # svn info /usr/ports
Caminho: /usr/ports
Working Copy Root Path: /usr/ports
URL: svn://svn.freebsd.org/ports/head
Relative URL: ^/head
Raiz do Repositório: svn://svn.freebsd.org/ports
UUID do repositório: 35697150-7ecd-e111-bb59-0022644237b5
Revisão: 334975
Tipo de Nó: diretório
Agendado: normal
Autor da Última Mudança: kwm
Revisão da Última Mudança: 334975
Data da Última Mudança: 2013-11-26 19:15:14 -0200 (Ter, 26 Nov 2013)

root@valfenda:/home2/rizzo # uname -a
FreeBSD valfenda 11.0-CURRENT FreeBSD 11.0-CURRENT #9 r258666: Wed Nov 27
00:06:27 BRST 2013 rizzo@valfenda:/usr/obj/usr/src/sys/VALFENDA  amd64

TIA,

Rizzo

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

problems with pkg2ng

2013-11-28 Thread Nilton Jose Rizzo 

  Hi all, 

 in the near past, when I update ports repository via svn, I always use
 pkg2ng to upgrade (or update?) the database, but today I can not do it.

 The command pkg2ng show this error message:

root@valfenda:/home2/rizzo # pkg2ng
pkg: PACKAGESITE in pkg.conf is deprecated. Please create a repository
configuration file
Converting packages from /var/db/pkg
pkg: PACKAGESITE in pkg.conf is deprecated. Please create a repository
configuration file
pkg: fstat() failed for(/usr/local/etc/cups/cupsd.conf.N): No such file or
directory

but when I search in /usr/ports/UPDATING I can not found anything about this
and no man pages about pkg2ng 

root@valfenda:/home2/rizzo # man pkg2ng
No manual entry for pkg2ng
root@valfenda:/home2/rizzo # 

Can Someone help me?

MyBox:

root@valfenda:/home2/rizzo # pkg -v
1.2_1
root@valfenda:/home2/rizzo # svn info /usr/src
Caminho: /usr/src
Working Copy Root Path: /usr/src
URL: svn://svn.freebsd.org/base/head
Relative URL: ^/head
Raiz do Repositório: svn://svn.freebsd.org/base
UUID do repositório: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Revisão: 258666
Tipo de Nó: diretório
Agendado: normal
Autor da Última Mudança: glebius
Revisão da Última Mudança: 258666
Data da Última Mudança: 2013-11-26 18:27:57 -0200 (Ter, 26 Nov 2013)

root@valfenda:/home2/rizzo # svn info /usr/ports
Caminho: /usr/ports
Working Copy Root Path: /usr/ports
URL: svn://svn.freebsd.org/ports/head
Relative URL: ^/head
Raiz do Repositório: svn://svn.freebsd.org/ports
UUID do repositório: 35697150-7ecd-e111-bb59-0022644237b5
Revisão: 334975
Tipo de Nó: diretório
Agendado: normal
Autor da Última Mudança: kwm
Revisão da Última Mudança: 334975
Data da Última Mudança: 2013-11-26 19:15:14 -0200 (Ter, 26 Nov 2013)

root@valfenda:/home2/rizzo # uname -a
FreeBSD valfenda 11.0-CURRENT FreeBSD 11.0-CURRENT #9 r258666: Wed Nov 27
00:06:27 BRST 2013 rizzo@valfenda:/usr/obj/usr/src/sys/VALFENDA  amd64




TIA,

Rizzo
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

request for help: MFC net80211 fixes from -HEAD to -10

2013-11-28 Thread Adrian Chadd 
hi all,

I'd like a developer or two to organise the MFC of anything that's in
net80211 on -HEAD back to -10 before 10.0-REL.

There's a few critical fixes that need to go in but I just don't have
the time to do it myself. :(

Thanks!




-adrian
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: request for help: MFC net80211 fixes from -HEAD to -10

2013-11-28 Thread Glen Barber 
On Thu, Nov 28, 2013 at 06:54:56PM -0800, Adrian Chadd wrote:
> I'd like a developer or two to organise the MFC of anything that's in
> net80211 on -HEAD back to -10 before 10.0-REL.
> 
> There's a few critical fixes that need to go in but I just don't have
> the time to do it myself. :(

Depending on what the fixes are, you're likely too late at this point.

What are the fixes, specifically?

Glen



pgpOjh9PkYu_i.pgp
Description: PGP signature

Re: request for help: MFC net80211 fixes from -HEAD to -10

2013-11-28 Thread Adrian Chadd 
On 28 November 2013 19:04, Glen Barber  wrote:
> On Thu, Nov 28, 2013 at 06:54:56PM -0800, Adrian Chadd wrote:
>> I'd like a developer or two to organise the MFC of anything that's in
>> net80211 on -HEAD back to -10 before 10.0-REL.
>>
>> There's a few critical fixes that need to go in but I just don't have
>> the time to do it myself. :(
>
> Depending on what the fixes are, you're likely too late at this point.
>
> What are the fixes, specifically?

I forget the details off-hand. There's a specific fix for incorrectly
handling return values when queuing frames during power save or scan
that results in traffic drops/errors. But there's been a few.



-a
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: problems with pkg2ng

2013-11-28 Thread Baptiste Daroussin 
On Thu, Nov 28, 2013 at 11:05:21PM -0300, Nilton Jose Rizzo wrote:
> 
> 
> Hi all,
> 
>  in the near past, when I update ports repository via svn, I always use
>  pkg2ng to upgrade (or update?) the database, but today I can not do it.
> 
>  The command pkg2ng show this error message:
> 
> root@valfenda:/home2/rizzo # pkg2ng
> pkg: PACKAGESITE in pkg.conf is deprecated. Please create a repository
> configuration file
> Converting packages from /var/db/pkg
> pkg: PACKAGESITE in pkg.conf is deprecated. Please create a repository
> configuration file
> pkg: fstat() failed for(/usr/local/etc/cups/cupsd.conf.N): No such file or
> directory
> 
> but when I search in /usr/ports/UPDATING I can not found anything about this
> and no man pages about pkg2ng
> 
> root@valfenda:/home2/rizzo # man pkg2ng
> No manual entry for pkg2ng
> root@valfenda:/home2/rizzo #
> 
> Can Someone help me?
> 

Those are "just" warnings, nothing to worry about I'll add an UPDATING entry
about those as soon as I can.

regards,
Bapt


pgpS_kAaSFyRP.pgp
Description: PGP signature

Re: problems with pkg2ng

2013-11-28 Thread Matthew Seaman 
On 29/11/2013 02:02, Nilton Jose Rizzo wrote:
>  in the near past, when I update ports repository via svn, I always use
>  pkg2ng to upgrade (or update?) the database, but today I can not do it.

Uh?  pkg2ng is a one time thing.  Its only use is when converting a
system that had previously used pkg_tools to pkg(8).  That happens once,
and after that, you never need it again.

>  The command pkg2ng show this error message:
> 
> root@valfenda:/home2/rizzo # pkg2ng
> pkg: PACKAGESITE in pkg.conf is deprecated. Please create a repository
> configuration file
> Converting packages from /var/db/pkg
> pkg: PACKAGESITE in pkg.conf is deprecated. Please create a repository
> configuration file
> pkg: fstat() failed for(/usr/local/etc/cups/cupsd.conf.N): No such file or
> directory

This is not pkg2ng specific, but comes from the underlying calls to
various pkg(8) actions.

As it says, the format of pkg.conf has changed with this update, and it
is instructing you on how to modify your pkg.conf so it conforms to the
new norms.  See pkg.conf(5) for details on creating a repository
configuration file.

The fstat() message is a different thing again -- looks like one of the
files owned by the cups port has been deleted.  You can fix that by
forcing a reinstall of that package, but it's pretty innocuous really.

> but when I search in /usr/ports/UPDATING I can not found anything about this
> and no man pages about pkg2ng 

A lot of people seem confused and/or alarmed by this update.  Yes, there
have been a number of problems reported, which are being dealt with.
Whether this warrants an entry in UPDATING I don't know, but the general
tenor of any such entry would be "modify your configuration files as
shown in the output of pkg(8)."

No, there isn't a man page for pkg2ng.  It was never thought necessary.
 However, if anyone wants to contribute a page for pkg2ng I'm sure it
would be gratefully received.

Cheers,

Matthew

-- 
Dr Matthew J Seaman MA, D.Phil.

PGP: http://www.infracaninophile.co.uk/pgpkey
JID: matt...@infracaninophile.co.uk



signature.asc
Description: OpenPGP digital signature