Turn libc.so into an ld script
Hi,
There has been quite a while since the SSP glue has been committed in
the tree. Yet there has always been a gloomy corner case since then
that was reported from time to time, mainly by port maintainers, which
has been hard to reproduce. This is the main showstopper to enable SSP
for ports by default.
On i386 for PIC objects, gcc uses the __stack_chk_fail_local hidden
symbol instead of calling __stack_chk_fail directly [1]. This happen
not only with our gcc-4.2.1 but also with the latest gcc-4.8. If you
want the very nasty details, see [2].
OTOH the problem doesn't exist on other architectures. It also doesn't
exist with Clang as the latter will somehow manage to create the
function in the object file at compile time (contrary to only
referencing it through a symbol that will be brought in at link time).
In a perfect world, when an object file is compiled with
-fstack-protector, it will be linked into a binary or a DSO with this
same flag as well, so GCC will add libssp_nonshared.a to the linker
command-line. Unfortunately, we don't control softwares in ports and we
may have such broken DSO. This is the whole point of this patch.
I wrote a specific test that exhibits the error:
http://people.freebsd.org/~jlh/twisted_ssp_linktime_fail.shar
If you run "make main" on i386, it will fail. More details at [3].
So the attached patch turns libc.so into an ld script which will
automatically _propose_ libssp_nonshared.a along with the real libc DSO
to the linker. It is important to understand that the object file
contained in this library will be pulled in the resulting binary
_only if_ the linker notices one of its symbols is needed (i.e. one of
the SSP symbol is missing). Otherwise nothing is changed, except a
slight theorical overhead that I wasn't able to measure on my Core 2
developement machine with -j 4:
x current
+ current_libc_ldscript
+--+
| ++ x+ xx +x|
|||_M__M___AA___|__| |
+--+
N Min MaxMedian AvgStddev
x 4 9130 9227 9136 9157 46.740418
+ 4 9126 9207 9132 9148 39.420807
Any objection to the patch?
Thanks for reading,
[1] See comment here if you wonder why:
sed -n 19460,+3p src/contrib/gcc/config/i386/i386.c
[2] When compiling a source file to an object file, if you use something
which is external to the compilation unit, GCC doesn't know yet if
this symbol will be inside or outside the DSO. So it expects the
worst case and routes the symbol through the GOT, which means
additional space and extra relocation for rtld(1).
Declaring a symbol has hidden tells GCC to use the optimal route (no
GOT), but on the other hand this means the symbol has to be provided
in the same DSO (namely libssp_nonshared.a).
On i386, GCC actually uses an hidden symbol for SSP in PIC objects
to save PIC register setup, as said in [1].
[3] As abstractly explained in [2], the problem shows up as long as you
compile a PIC (or PIE) object but you don't link it directly with
libssp_nonshared.a.
So in the test I gave, you can also trigger the problem by setting
"BIN_CFLAGS= -fstack-protector-all -fPIE" and leaving BIN_LDFLAGS
blank, whatever you did with LIB_{CFLAGS,LDFLAGS}.
This won't happen without -fPIE here, because a non-hidden symbol
will be emitted in that case.
--
Jeremie Le Hen
Scientists say the world is made up of Protons, Neutrons and Electrons.
They forgot to mention Morons.
Index: lib/libc/Makefile
===
--- lib/libc/Makefile (revision 250307)
+++ lib/libc/Makefile (working copy)
@@ -33,6 +33,12 @@
INSTALL_PIC_ARCHIVE=
PRECIOUSLIB=
+# Only use an ld script in place of the so file on i386, because of
+# the way -fstack-protector works on gcc.
+.if ${MACHINE_CPUARCH} == "i386"
+SHLIB_LDSCRIPT=libc.ldscript
+.endif
+
.ifndef NO_THREAD_STACK_UNWIND
CANCELPOINTS_CFLAGS=-fexceptions
CFLAGS+=${CANCELPOINTS_CFLAGS}
Index: lib/libc/libc.ldscript
===
--- lib/libc/libc.ldscript (revision 0)
+++ lib/libc/libc.ldscript (working copy)
@@ -0,0 +1,2 @@
+/* $FreeBSD$ */
+GROUP ( @@SHLIB@@ @@LIBDIR@@/libssp_nonshared.a )
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Writing a (BSD like) Operating Systems From Scratch
On 24/05/2013 18:57, Welcome, Traiano wrote: > You appear not to realize that to even begin working with one of the existing > projects, you'd best have a solid understanding of OSes to begin with, > which brings up an interesting catch -22 that goes something like: > > "You can't join the club, because you don't know enough. You can't know > enough 'till you join our club". Not at all - it all depends on how strongly you wish to do it and how much time you can devote to it. It won't be easy. Documentation is basically the source, and people have no time to help you or guide you (nor can you expect them to since they are mostly volunteers and have their own lives to live - this is why Julian suggested a uni which is paid to teach you). The usual way things like this work (and which I've taken, as well as most people here that I know) is that you begin at one subsystem, write something in it, then if you feel the need expand to subystems connected to that one, and repeat if necessary. If you wish to be productive, you could ask for a suggestion of such subsystems which could be easy-ish to begin with. > I don't accept the conjecture that modern OSes are too big for one man. > Modern OSes and their associated entourage of userpace and plugin modules > maybe, but not the basic kernel/supervisor program. An OS is as big or small > as you make it. Oh yes they are. The most you can hope for if you try to do it yourself is to write an "example" OS kernel which will do the few things you yourself are able to do, or even realize that they need to be done. The usual way THAT works is that as you go along you find out how the seemingly simple thing you thought needs to be done is only a small part of what is really needed, or completely insufficient, and that to solve the problem correctly, you need another 5 years to work at it - and there is a LOT of problems like that. Ask Matt Dillon [NHF, respect] :) signature.asc Description: OpenPGP digital signature
Re: Writing a (BSD like) Operating Systems From Scratch
"Welcome, Traiano" wrote: > May I ask where you get the divine wisdom to know where I "would be better > working with" ? don't you think that would be best left up to me? etc ... elided As politeness fails: Waste your own time as you want. Don't waste FreeBSD people's time, asking advice how to waste your time, with no benefit to FreeBSD. We're better ignoring that & using send-pr to send patches to FreeBSD on anything else we choose :-) If you want to be a useless lone wolf Not contributing to FreeBSD, do just that, go away & don't freeload for help. I hope instead you'll want to work as a team member & contribute where you think best, but If you want to screw around with an OS on your own - Do It On Your Own. As to who I am, it shouldn't matter as an idea should stand or fall on its merits, but in case it's a genuine question: I'm someone who learnt in 1983 that H/W for OS's evolves faster than small teams can keep up. Who saw a waster in 80s/90's who wanted to rewrite everything, & contributed nothing. Who saw OSs evolve faster than individuals, & knows team work is required, each dividing into specialisms to support the aggregate OS. Cheers, Julian -- Julian Stacey, BSD Unix Linux C Sys Eng Consultant, Munich http://berklix.com Reply below not above, like a play script. Indent old text with "> ". Send plain text. No quoted-printable, HTML, base64, multipart/alternative. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Writing a (BSD like) Operating Systems From Scratch
Jorge Alberto Garcia wrote: > > Hello Joshua, i think you should try XV6, i am pretty one person can > understand it to get a general yet down to the earth felling about a unix > like system; then try with Minix2 > > http://pdos.csail.mit.edu/6.828/2012/xv6.html > > Tnx. > -J.A. Garcia Great to read about XV6, fond memories reading the Lions book in the 80s :-) FYI http://www.minix3.org "MINIX 3 won a grant from the European Research Council for 2.5 million Euro to further research in highly reliable operating systems." Cheers, Julian -- Julian Stacey, BSD Unix Linux C Sys Eng Consultant, Munich http://berklix.com Reply below not above, like a play script. Indent old text with "> ". Send plain text. No quoted-printable, HTML, base64, multipart/alternative. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Turn libc.so into an ld script
I've been running my systems with this modification since Feb 2012 and
have seen no problems beyond file(1) usage on /usr/lib/libc.so in
openssl's configure.
I've taken ports/168010 and ports/138228 for exp-runs. I want to get
(optional) SSP support into ports this year.
I'll start a libc.ld exp-run tomorrow. It will be ran against
9.1-RELEASE since HEAD currently only has 1/2 of the ports tree passing
due to the clang switch.
Bryan
On 5/25/2013 3:06 AM, Jeremie Le Hen wrote:
> Hi,
>
> There has been quite a while since the SSP glue has been committed in
> the tree. Yet there has always been a gloomy corner case since then
> that was reported from time to time, mainly by port maintainers, which
> has been hard to reproduce. This is the main showstopper to enable SSP
> for ports by default.
>
> On i386 for PIC objects, gcc uses the __stack_chk_fail_local hidden
> symbol instead of calling __stack_chk_fail directly [1]. This happen
> not only with our gcc-4.2.1 but also with the latest gcc-4.8. If you
> want the very nasty details, see [2].
>
> OTOH the problem doesn't exist on other architectures. It also doesn't
> exist with Clang as the latter will somehow manage to create the
> function in the object file at compile time (contrary to only
> referencing it through a symbol that will be brought in at link time).
>
> In a perfect world, when an object file is compiled with
> -fstack-protector, it will be linked into a binary or a DSO with this
> same flag as well, so GCC will add libssp_nonshared.a to the linker
> command-line. Unfortunately, we don't control softwares in ports and we
> may have such broken DSO. This is the whole point of this patch.
>
> I wrote a specific test that exhibits the error:
> http://people.freebsd.org/~jlh/twisted_ssp_linktime_fail.shar
> If you run "make main" on i386, it will fail. More details at [3].
>
> So the attached patch turns libc.so into an ld script which will
> automatically _propose_ libssp_nonshared.a along with the real libc DSO
> to the linker. It is important to understand that the object file
> contained in this library will be pulled in the resulting binary
> _only if_ the linker notices one of its symbols is needed (i.e. one of
> the SSP symbol is missing). Otherwise nothing is changed, except a
> slight theorical overhead that I wasn't able to measure on my Core 2
> developement machine with -j 4:
>
> x current
> + current_libc_ldscript
> +--+
> | ++ x+ xx +
> x|
> |||_M__M___AA___|__|
> |
> +--+
> N Min MaxMedian AvgStddev
> x 4 9130 9227 9136 9157 46.740418
> + 4 9126 9207 9132 9148 39.420807
>
>
> Any objection to the patch?
>
> Thanks for reading,
>
>
> [1] See comment here if you wonder why:
> sed -n 19460,+3p src/contrib/gcc/config/i386/i386.c
>
> [2] When compiling a source file to an object file, if you use something
> which is external to the compilation unit, GCC doesn't know yet if
> this symbol will be inside or outside the DSO. So it expects the
> worst case and routes the symbol through the GOT, which means
> additional space and extra relocation for rtld(1).
>
> Declaring a symbol has hidden tells GCC to use the optimal route (no
> GOT), but on the other hand this means the symbol has to be provided
> in the same DSO (namely libssp_nonshared.a).
>
> On i386, GCC actually uses an hidden symbol for SSP in PIC objects
> to save PIC register setup, as said in [1].
>
> [3] As abstractly explained in [2], the problem shows up as long as you
> compile a PIC (or PIE) object but you don't link it directly with
> libssp_nonshared.a.
>
> So in the test I gave, you can also trigger the problem by setting
> "BIN_CFLAGS= -fstack-protector-all -fPIE" and leaving BIN_LDFLAGS
> blank, whatever you did with LIB_{CFLAGS,LDFLAGS}.
>
> This won't happen without -fPIE here, because a non-hidden symbol
> will be emitted in that case.
>
>
>
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>
--
Regards,
Bryan Drewery
signature.asc
Description: OpenPGP digital signature
FreeBSD installers and future direction
I heard there was some discussion at BSDCan about the direction of a future FreeBSD installer. Considering we currently have bsdinstall, pc-sysinstall, and an effort to revive sysinstall, I'd be interested to know what was decided (if anything) and whether I could help make progress towards getting a single really good installer/frontend - instead of the current situation with several, none of which have a much-needed UI for setting up an installation on ZFS. -- Bruce Cran ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD installers and future direction
On Sat, May 25, 2013 at 8:43 AM, Bruce Cran wrote: > > I heard there was some discussion at BSDCan about the direction of a future > FreeBSD installer. Considering we currently have bsdinstall, pc-sysinstall, > and an effort to revive sysinstall, I'd be interested to know what was > decided (if anything) and whether I could help make progress towards getting > a single really good installer/frontend - instead of the current situation > with several, none of which have a much-needed UI for setting up an > installation on ZFS. Hey Bruce, >From my vague recollection, we discussed improving bsdinstall by tying it in with pc-sysinstall, which we've been threatening to do for at least a year. Also, there was much discussion about Devin's bsdconfig perhaps tying in with a Google SoC Project. I think Devin was nominated for most of the work, since he was unable to defend himself :P Cheers, -matt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Creative X-Fi Audio Notebook (SB0950) support?
It's not a X-Fi sound card at all. Its chip is CA0106-WBTLF (from Wikipedia). But it's the only sound card available on ExpressCard, so it's kinda a great choice for laptop users. Is that possible to add its support into FreeBSD OSS? On my FreeBSD 8.4 machine, it's recognized as May 25 12:36:27 elitebook kernel: ugen2.3: at usbus2 May 25 12:36:27 elitebook kernel: uhid0: on usbus2 May 25 12:36:27 elitebook kernel: ukbd0: on usbus2 May 25 12:36:27 elitebook kernel: kbd2 at ukbd0 May 25 12:36:27 elitebook kernel: uaudio0: on usbus2 May 25 12:36:27 elitebook kernel: uaudio0: No playback. May 25 12:36:27 elitebook kernel: uaudio0: Record: 48000 Hz, 2 ch, 16-bit S-LE PCM format, 2x8ms buffer. May 25 12:36:27 elitebook kernel: uaudio0: No MIDI sequencer. May 25 12:36:27 elitebook kernel: pcm6: on uaudio0 May 25 12:36:27 elitebook kernel: uaudio0: HID volume keys found. OSSv4 (including the latest versions in hg) does not work either, -- Zhihao Yuan, ID lichray The best way to predict the future is to invent it. ___ 4BSD -- http://4bsd.biz/ ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD installers and future direction
On 25/05/2013 17:15, Matt Olander wrote: From my vague recollection, we discussed improving bsdinstall by tying it in with pc-sysinstall, which we've been threatening to do for at least a year. Also, there was much discussion about Devin's bsdconfig perhaps tying in with a Google SoC Project. I think Devin was nominated for most of the work, since he was unable to defend himself :P Thanks. From previous discussions with Devin I think he has other plans for the installer that don't involve pc-sysinstall. But since it seems the future is all sh(1) code, I won't be able to contribute. https://wiki.freebsd.org/PCBSDInstallMerge lists a few limitations with pc-sysinstall - are these being fixed? -- Bruce ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Turn libc.so into an ld script
On Sat, May 25, 2013 at 09:03:57AM -0500, Bryan Drewery wrote:
> I've been running my systems with this modification since Feb 2012 and
> have seen no problems beyond file(1) usage on /usr/lib/libc.so in
> openssl's configure.
>
> I've taken ports/168010 and ports/138228 for exp-runs. I want to get
> (optional) SSP support into ports this year.
>
> I'll start a libc.ld exp-run tomorrow. It will be ran against
> 9.1-RELEASE since HEAD currently only has 1/2 of the ports tree passing
> due to the clang switch.
Ok thank you for helping on this :).
I talked with kan@ on IRC, and he suggested that having arch-specific
deviation for this may not be a good idea. As I explained in my
previous e-mail, there is no impact speed-wise or space-wise, so would
you mind to do the exp-run using this patch instead please?
http://people.freebsd.org/~jlh/libc_ldscript_noarch.diff
Thanks,
> On 5/25/2013 3:06 AM, Jeremie Le Hen wrote:
> > Hi,
> >
> > There has been quite a while since the SSP glue has been committed in
> > the tree. Yet there has always been a gloomy corner case since then
> > that was reported from time to time, mainly by port maintainers, which
> > has been hard to reproduce. This is the main showstopper to enable SSP
> > for ports by default.
> >
> > On i386 for PIC objects, gcc uses the __stack_chk_fail_local hidden
> > symbol instead of calling __stack_chk_fail directly [1]. This happen
> > not only with our gcc-4.2.1 but also with the latest gcc-4.8. If you
> > want the very nasty details, see [2].
> >
> > OTOH the problem doesn't exist on other architectures. It also doesn't
> > exist with Clang as the latter will somehow manage to create the
> > function in the object file at compile time (contrary to only
> > referencing it through a symbol that will be brought in at link time).
> >
> > In a perfect world, when an object file is compiled with
> > -fstack-protector, it will be linked into a binary or a DSO with this
> > same flag as well, so GCC will add libssp_nonshared.a to the linker
> > command-line. Unfortunately, we don't control softwares in ports and we
> > may have such broken DSO. This is the whole point of this patch.
> >
> > I wrote a specific test that exhibits the error:
> > http://people.freebsd.org/~jlh/twisted_ssp_linktime_fail.shar
> > If you run "make main" on i386, it will fail. More details at [3].
> >
> > So the attached patch turns libc.so into an ld script which will
> > automatically _propose_ libssp_nonshared.a along with the real libc DSO
> > to the linker. It is important to understand that the object file
> > contained in this library will be pulled in the resulting binary
> > _only if_ the linker notices one of its symbols is needed (i.e. one of
> > the SSP symbol is missing). Otherwise nothing is changed, except a
> > slight theorical overhead that I wasn't able to measure on my Core 2
> > developement machine with -j 4:
> >
> > x current
> > + current_libc_ldscript
> > +--+
> > | ++ x+ xx +
> > x|
> > |||_M__M___AA___|__|
> >|
> > +--+
> > N Min MaxMedian AvgStddev
> > x 4 9130 9227 9136 9157 46.740418
> > + 4 9126 9207 9132 9148 39.420807
> >
> >
> > Any objection to the patch?
> >
> > Thanks for reading,
> >
> >
> > [1] See comment here if you wonder why:
> > sed -n 19460,+3p src/contrib/gcc/config/i386/i386.c
> >
> > [2] When compiling a source file to an object file, if you use something
> > which is external to the compilation unit, GCC doesn't know yet if
> > this symbol will be inside or outside the DSO. So it expects the
> > worst case and routes the symbol through the GOT, which means
> > additional space and extra relocation for rtld(1).
> >
> > Declaring a symbol has hidden tells GCC to use the optimal route (no
> > GOT), but on the other hand this means the symbol has to be provided
> > in the same DSO (namely libssp_nonshared.a).
> >
> > On i386, GCC actually uses an hidden symbol for SSP in PIC objects
> > to save PIC register setup, as said in [1].
> >
> > [3] As abstractly explained in [2], the problem shows up as long as you
> > compile a PIC (or PIE) object but you don't link it directly with
> > libssp_nonshared.a.
> >
> > So in the test I gave, you can also trigger the problem by setting
> > "BIN_CFLAGS= -fstack-protector-all -fPIE" and leaving BIN_LDFLAGS
> > blank, whatever you did with LIB_{CFLAGS,LDFLAGS}.
> >
> > This won't happen without -fPIE here, because a non-hidden symbol
> > will be emitted in that case.
--
Jeremie Le Hen
Performance improvement to strnlen().
Hello FreeBSD devs, I have found a performance improvement to libc's strnlen(). lib/libc/string/strnlen.c is a trivial byte-by-byte implementation, where strlen.c has a smarter word-by-word implementation. I have implemented strnlen similarly to strlen. It runs about 4x as fast, at the cost of a binary codesize increase from 30 bytes to 221. In writing this I needed a test, and there isn't one in tools/regression/lib/libc/string, so I wrote a unit test for strnlen. This really only makes sense for a word-by-word implementation, as it tests each combination of string and limit length, overread characters, and starting alignment. Could someone please review these patches? I am not very experienced in C, and am even less experienced with FreeBSD's style guidelines, so they likely have a bunch of style and idiom issues. Even if one or both of them prove not worth committing, it would still be useful for my learning. If/When these patches prove worthy of submitting, what is the next step after that? Should I submit a PR, or is there some other process? This code is quite similar to the existing strlen.c, and doesn't do anything fancy with e.g. endianness, but I haven't tested it for correctness or performance on anything other than x86... And finally, there is some other low-hanging fruit in the other strn* functions. Would it be worth it for me to give those the same treatment? Thanks, Lee Thomas Test platform: $uname -a FreeBSD LeeDesktop 9.1-STABLE FreeBSD 9.1-STABLE #15 r250831: Mon May 20 17:31:28 EDT 2013 lthomas@LeeDesktop:/usr/obj/usr/src/sys/NOSOUND amd64 $dmesg | grep CPU: CPU: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (2666.81-MHz K8-class CPU) $clang --version FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 Target: x86_64-unknown-freebsd9.1 Thread model: posix My timing test was a file of 1 strings, of uniformly random length, 50% between 0 and 20 bytes long, and 50% between 21 and 1000 bytes long. Each was filled with random generated bytes in the range [1, 255]. The test executables run their strlen on each string in the file 1000 times, and a shell script ran the test programs alternately 100 times. Here are the clang results; gcc results are roughly the same. I will share the actual timing code if someone wants it, but it is pretty ugly C++ and shell and I don't guarantee it working :-). Results: x ./times_bsd_strnlen.txt + ./times_my_strnlen.txt +--+ |+ x| |+ x| |+ x| |+ x| |+ x| |+ x| |+ x| |+ x| |+ x| |+ x| |+ x| |+ x| |+ x| |+ x| |+ x| |+ x| |+ x| |A A| +--+ N Min MaxMedian Avg Stddev x 101 1.8685486 1.8693889 1.8687506 1.8687894 0.0001484903 + 1010.427046830.427794860.427128130.42715597 0.00010681494 Difference at 95.0% confidence -1.44163 +/- 3.56739e-05 -77.1426% +/- 0.00190893% (Student's t, pooled s = 0.000129342) Note: I manually shortened the histogram, as it was 100 lines long of exactly the same. Index: strnlen.c === diff --git a/head/lib/libc/string/strnlen.c b/head/lib/libc/string/strnlen.c --- a/head/lib/libc/string/strnlen.c(revision 250951) +++ b/head/lib/libc/string/strnlen.c(working copy) @@ -1,5 +1,6 @@ /*- - * Copyright (c) 2009 David Schultz + * Copyright (c) 2009, 2010 Xin LI + * Copyright (c) 2013 Lee Thomas * All rig
Re: FreeBSD installers and future direction
On 05/25/13 13:26, Bruce Cran wrote: On 25/05/2013 17:15, Matt Olander wrote: From my vague recollection, we discussed improving bsdinstall by tying it in with pc-sysinstall, which we've been threatening to do for at least a year. Also, there was much discussion about Devin's bsdconfig perhaps tying in with a Google SoC Project. I think Devin was nominated for most of the work, since he was unable to defend himself :P Thanks. From previous discussions with Devin I think he has other plans for the installer that don't involve pc-sysinstall. But since it seems the future is all sh(1) code, I won't be able to contribute. https://wiki.freebsd.org/PCBSDInstallMerge lists a few limitations with pc-sysinstall - are these being fixed? I'm not aware of any movement there (on either side of the table). I'd personally be very suspicious of an all-sh(1) future -- by far the cleanest parts of bsdinstall are in C -- and this is especially true for interacting with geom. That said, since I've lost nearly all of my free time and ability to work on bsdinstall, I won't get in the way of anyone else working on things -Nathan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD installers and future direction
On 26.05.13 01:07, Nathan Whitehorn wrote: > I'm not aware of any movement there (on either side of the table). I'd > personally be very suspicious of an all-sh(1) future -- by far the > cleanest parts of bsdinstall are in C -- and this is especially true for > interacting with geom. That said, since I've lost nearly all of my free > time and ability to work on bsdinstall, I won't get in the way of anyone > else working on things As discussed at BSDCan, I'd be willing to participate in the development and at least implement setting up zpools/zfs and geli/gbde providers. I have done similar things in sh in my ezjail tools and think I can glue the rest together. Scanning through the pc-sysinstall code, I find nothing too fancy there regarding either interaction with zfs nor geom tools. I do not think it is necessary as a back end just for these features. Nathan, is there any design rationale available for the scripts, e.g. on why you chose sh versus C and were you provided with some kind of wish list/requirements in the first place? Any particular mail thread to scan through beforehand? Regards, erdgeist ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD installers and future direction
Please don't turn this into an architecture dependent mess. PCBSD is i386 & AMD64 only. On Sat, May 25, 2013 at 9:01 PM, Dirk Engling wrote: > On 26.05.13 01:07, Nathan Whitehorn wrote: > > > I'm not aware of any movement there (on either side of the table). I'd > > personally be very suspicious of an all-sh(1) future -- by far the > > cleanest parts of bsdinstall are in C -- and this is especially true for > > interacting with geom. That said, since I've lost nearly all of my free > > time and ability to work on bsdinstall, I won't get in the way of anyone > > else working on things > > As discussed at BSDCan, I'd be willing to participate in the development > and at least implement setting up zpools/zfs and geli/gbde providers. I > have done similar things in sh in my ezjail tools and think I can glue > the rest together. > > Scanning through the pc-sysinstall code, I find nothing too fancy there > regarding either interaction with zfs nor geom tools. I do not think it > is necessary as a back end just for these features. > > Nathan, is there any design rationale available for the scripts, e.g. on > why you chose sh versus C and were you provided with some kind of wish > list/requirements in the first place? Any particular mail thread to scan > through beforehand? > > Regards, > > erdgeist > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" > ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD installers and future direction
On May 25, 2013, at 9:15 AM, Matt Olander wrote: > On Sat, May 25, 2013 at 8:43 AM, Bruce Cran wrote: >> >> I heard there was some discussion at BSDCan about the direction of a future >> FreeBSD installer. Considering we currently have bsdinstall, pc-sysinstall, >> and an effort to revive sysinstall, I'd be interested to know what was >> decided (if anything) and whether I could help make progress towards getting >> a single really good installer/frontend - instead of the current situation >> with several, none of which have a much-needed UI for setting up an >> installation on ZFS. > > Hey Bruce, > > From my vague recollection, we discussed improving bsdinstall by tying > it in with pc-sysinstall, which we've been threatening to do for at > least a year. Also, there was much discussion about Devin's bsdconfig > perhaps tying in with a Google SoC Project. > There is indeed a GSoC project to address the situation w/regard to installers. The GSoC project is one submitted by Harsh Bhatt and I've submitted that I wish to Mentor. The project is titled: "Making pc-sysinstall FreeBSD ready by porting it to multiple architectures" The project is designed as such to increase possibilities w/respect to installers. To elaborate… Integrating bsdinstall with pc-sysinstall would introduce a regression because bsdinstall currently supports all platforms, whereas pc-sysinstall has hard-coded assumptions specific to the x86 platform (i386 and amd64 included). If bsdinstall was ever going to be programmed to use pc-sysinstall as a back-end, pc-sysinstall would have to be cleaned up. I'm in no way pushing to integrate bsdinstall with pc-sysinstall… but I *am* saying this is a GSoC project to look out for. The goal of the project is to see if it is even possible to remedy any possibility that tying bsdinstall to pc-sysinstall would introduce a regression in platform support. If the project is successful, PC-BSD should be able to immediately benefit from the fruits thereof (as -- correct me if I'm wrong -- the graphical PC-BSD installer uses pc-sysinstall as a back-end). ASIDE: For what it's worth, it's somewhat convenient to think that in a simple world -- because pc-sysinstall is already the backend of the PC-BSD GUI installer -- FreeBSD would have bsdinstall as the TUI front-end to the same back-end. But however, I'm not naïve and can't imagine that as being clean or elegant unless we can clean up pc-sysinstall. Cleaning up the code is a another major focal-point of the aforementioned GSoC project. With respect to my bsdconfig… (how it relates to installers) Just as there were blockers preventing pc-sysinstall from being integrated into bsdinstall. There are blockers that prevent bsdinstall from being integrated into the larger bsdconfig framework. In the case of pc-sysinstall integrating to bsdinstall, pc-sysinstall doesn't support all the target architectures that bsdinstall does. Meanwhile, in the case of bsdinstall integrating into bsdconfig, bsdinstall doesn't support all the features of bsdconfig. Luckily, introduction of most features to bring bsdinstall on-par with bsdconfig will be easy as we can just make bsdinstall use the subroutine includes from bsdconfig. However, there are other things that just can't be done without plain sweat and labor… For example, making bsdinstall i18n-ready (as bsdconfig is). I expect this to take (with a medium effort) a week or two, aided by the fact that the dialog(1) abstraction API offered by bsdconfig makes most of that transparent (for example, you don't have to worry about whether the "Yes" button says "Ja", "Oui", or "Yes" -- you just call f_dialog_yesno() and it worried about how to call dialog(1) to form the proper i18n-compatible prompt). Another big change to bsdinstall would be to make it *more* modular and rewrite the C components to be in sh. > I think Devin was nominated for most of the work, since he was unable > to defend himself :P > I don't mind being volunteered. In earnest, I figure the code I'm working on is volunteering enough. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD installers and future direction
On May 25, 2013, at 11:26 AM, Bruce Cran wrote: > On 25/05/2013 17:15, Matt Olander wrote: >> From my vague recollection, we discussed improving bsdinstall by tying >> it in with pc-sysinstall, which we've been threatening to do for at >> least a year. Also, there was much discussion about Devin's bsdconfig >> perhaps tying in with a Google SoC Project. >> >> I think Devin was nominated for most of the work, since he was unable >> to defend himself :P > > Thanks. From previous discussions with Devin I think he has other plans for > the installer that don't involve pc-sysinstall. But since it seems the future > is all sh(1) code, I won't be able to contribute. > The future of how these softwares become entangled to produce something better: + bsdinstall + pc-sysinstall + bsdconfig Is in my mind entirely still open as I work to finish up bsdconfig. I was thinking… Perhaps just rewrite bsdinstall from scratch (using the existing code as a template). Have to put pc-sysinstall on the back-burner for any/all considerations until the code is cleaned up and actually usable on all architectures (there's a GSoC project to do exactly that -- I'm the potential mentor; project is pending status). So unless this GSoC goes through and we are able to (as we plan) clean up that mess… Defacto plan is to just rewrite bsdinstall from scratch (again… using the existing code as a template). In said rewrite… I'd *heavily* leverate usr.sbin/bsdconfig/share/*.subr (specifically "dialog.subr" -- the abstraction layer that allows me to have nice i18n-ready dialogs and also gracefully handle dialog in a way that makes my code look very clean). > https://wiki.freebsd.org/PCBSDInstallMerge lists a few limitations with > pc-sysinstall - are these being fixed? > I can see if the GSoC student for the "Making pc-sysinstall FreeBSD ready by porting it to multiple architectures" project is willing to incorporate any of those limitations into his work. But I think the focus of the project should be to clean up the code and make it work on at least one other important non-x86 architecture. Barring that… You're absolutely right in stating that any/all discussions on merging pc-sysinstall with bsdinstall would result in a regression with pc-sysinstall in its current state. -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD installers and future direction
On May 25, 2013, at 4:07 PM, Nathan Whitehorn wrote: > On 05/25/13 13:26, Bruce Cran wrote: >> On 25/05/2013 17:15, Matt Olander wrote: >>> From my vague recollection, we discussed improving bsdinstall by tying >>> it in with pc-sysinstall, which we've been threatening to do for at >>> least a year. Also, there was much discussion about Devin's bsdconfig >>> perhaps tying in with a Google SoC Project. >>> >>> I think Devin was nominated for most of the work, since he was unable >>> to defend himself :P >> >> Thanks. From previous discussions with Devin I think he has other plans for >> the installer that don't involve pc-sysinstall. But since it seems the >> future is all sh(1) code, I won't be able to contribute. >> >> https://wiki.freebsd.org/PCBSDInstallMerge lists a few limitations with >> pc-sysinstall - are these being fixed? >> > > I'm not aware of any movement there (on either side of the table). I'd > personally be very suspicious of an all-sh(1) future -- by far the cleanest > parts of bsdinstall are in C -- and this is especially true for interacting > with geom. So that I can get a feel for the expectations on quality of product… You say the cleanest parts of bsdinstall are in C. What makes the sh parts unclean? Not looking for any particular answer… just want your opinion. I've been working very hard to produce a heavy-lifting "dialog.subr" (see base/head/usr.sbin/bsdconfig/share/dialog.subr) which makes dialog(1) work very nice and clean in sh(1). However, this may not be your concern, and you may instead have some other "cleanliness" insight that I can address in my work. -- Devin > That said, since I've lost nearly all of my free time and ability to work on > bsdinstall, I won't get in the way of anyone else working on things > -Nathan > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD installers and future direction
On May 25, 2013, at 6:01 PM, Dirk Engling wrote: > On 26.05.13 01:07, Nathan Whitehorn wrote: > >> I'm not aware of any movement there (on either side of the table). I'd >> personally be very suspicious of an all-sh(1) future -- by far the >> cleanest parts of bsdinstall are in C -- and this is especially true for >> interacting with geom. That said, since I've lost nearly all of my free >> time and ability to work on bsdinstall, I won't get in the way of anyone >> else working on things > > As discussed at BSDCan, I'd be willing to participate in the development > and at least implement setting up zpools/zfs and geli/gbde providers. I > have done similar things in sh in my ezjail tools and think I can glue > the rest together. > Very Cool! I do encourage you to take a peak at bsdconfig either from HEAD or from ports (in sysutils)… However, I'm getting ready to drastically change the API very soon (as previously threatened). I'm in the home-stretch and I'm trying to get in all the API changes before broaching the idea of taking off the "WITH_BSDCONFIG" build-option requirement. > Scanning through the pc-sysinstall code, I find nothing too fancy there > regarding either interaction with zfs nor geom tools. I do not think it > is necessary as a back end just for these features. > I tend to agree. In fact… the directory structure of pc-sysinstall and even the way it stores the subroutines in *.sh files means I would be compelled to restructure the thing into proper "includes". Since I intend to rework bsdinstall to use bsdconfig's libraries (e.g. "dialog.subr"), it sounds like a really nice path would be to develop: base/head/usr.sbin/bsdconfig/share/zfs.subr base/head/usr.sbin/bsdconfig/share/geom/ base/head/usr.sbin/bsdconfig/share/geom/geli.subr And then bsdinstall could include those. The reason I tell myself that it would be better if they lived in bsdconfig and got imported at runtime by bsdinstall (rather than living in bsdinstall) is that I want to move to brand the utilities in this way: bsdinstall is the program designed to install things (bare metal, jails, etc.) bsdconfig is the program designed to configure things either during installation time or post-installation time bsdinstall would be used on the boot media to install bare metal (but during that installation, made use bsdconfig for things like configuring users, boot services, networking, etc.). It will also be used after installation and running in multi-user land to do things like install jails. So in other words… anytime something could conceivably be wanted by bsdconfig… we should just birth it there and have bsdinstall reach out for it. Seems to make sense… a person installs once but configures many times. Guess I'm saying of course, that there'd be a great use for a zfs and geli library to bsdconfig for managing zfs after booting, etc. > Nathan, is there any design rationale available for the scripts, e.g. on > why you chose sh versus C and were you provided with some kind of wish > list/requirements in the first place? Any particular mail thread to scan > through beforehand? > Can't answer for Nathan (and not sure if you want my opinion), but… I chose 100% sh for bsdconfig because of a few good reasons… I ultimately wanted to (and did) make it scriptable. The scripts are actually `sourced' shell scripts. So… Because bsdconfig is sh, and the script is sh, the script has access to all of the bsdconfig internals, every API call, every variable, and can do nearly anything. There are no frustrating moments where a C aspect doesn't support running in a desired mode because that particular aspect was not made configurable or tunable. Not that this was (or is) a problem at all currently… just that it *was* a problem with sysinstall. For example… sysinstall had a deviceRescan() function but the dispatch.c system did not make *that* particular function available through the system of hard-coded resWords (reserved words) that were allowed in a script. So… if you wanted to have a sysinstall script that (a) formatted some disks with some slices and (b) then formatted those slices with some BSD labels … well… you couldn't (note: this is not to say couldn't get sysinstall to write both slices and labels, just that if you wanted to do this *OUT* of the context of a full install, you couldn't). The problem was that you really needed to call "deviceRescan" after doing the slice action on the bare metal so that sysinstall would pick up the "s1" slice on which you wanted to later write labels on. In the case of sysinstall… a one-line change of adding the deviceRescan() function to the resWord mapping in dispatch.c would have made that functionality available to scripts. bsdconfig doesn't have that problem because the scripts are actually shell scripts and the old sysinstall commands are replicated by shell functions. There is never any need to "map" a function before it becomes available
Re: FreeBSD installers and future direction
On May 25, 2013, at 7:51 PM, Super Bisquit wrote: > Please don't turn this into an architecture dependent mess. PCBSD is i386 & > AMD64 only. > There's a GSoC project (of which I'm potential mentor) to fix that. However, you are entirely right… we can't in all seriousness even think about using pc-sysinstall until it is solid on all architectures as bsdinstall already is. GSoC project is: "Making pc-sysinstall FreeBSD ready by porting it to multiple architectures" -- Devin > > On Sat, May 25, 2013 at 9:01 PM, Dirk Engling wrote: > >> On 26.05.13 01:07, Nathan Whitehorn wrote: >> >>> I'm not aware of any movement there (on either side of the table). I'd >>> personally be very suspicious of an all-sh(1) future -- by far the >>> cleanest parts of bsdinstall are in C -- and this is especially true for >>> interacting with geom. That said, since I've lost nearly all of my free >>> time and ability to work on bsdinstall, I won't get in the way of anyone >>> else working on things >> >> As discussed at BSDCan, I'd be willing to participate in the development >> and at least implement setting up zpools/zfs and geli/gbde providers. I >> have done similar things in sh in my ezjail tools and think I can glue >> the rest together. >> >> Scanning through the pc-sysinstall code, I find nothing too fancy there >> regarding either interaction with zfs nor geom tools. I do not think it >> is necessary as a back end just for these features. >> >> Nathan, is there any design rationale available for the scripts, e.g. on >> why you chose sh versus C and were you provided with some kind of wish >> list/requirements in the first place? Any particular mail thread to scan >> through beforehand? >> >> Regards, >> >> erdgeist >> ___ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" >> > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Performance improvement to strnlen().
On 05/25/2013 10:27 PM, Lee Thomas wrote: > Hello FreeBSD devs, > > I have found a performance improvement to libc's strnlen(). > lib/libc/string/strnlen.c is a trivial byte-by-byte implementation, > where strlen.c has a smarter word-by-word implementation. I have > implemented strnlen similarly to strlen. It runs about 4x as fast, at > the cost of a binary codesize increase from 30 bytes to 221. > > In writing this I needed a test, and there isn't one in > tools/regression/lib/libc/string, so I wrote a unit test for strnlen. > This really only makes sense for a word-by-word implementation, as it > tests each combination of string and limit length, overread characters, > and starting alignment. > > Could someone please review these patches? I am not very experienced in > C, and am even less experienced with FreeBSD's style guidelines, so they > likely have a bunch of style and idiom issues. Even if one or both of > them prove not worth committing, it would still be useful for my learning. > > If/When these patches prove worthy of submitting, what is the next step > after that? Should I submit a PR, or is there some other process? This > code is quite similar to the existing strlen.c, and doesn't do anything > fancy with e.g. endianness, but I haven't tested it for correctness or > performance on anything other than x86... > > And finally, there is some other low-hanging fruit in the other strn* > functions. Would it be worth it for me to give those the same treatment? > > Thanks, > Lee Thomas > > Test platform: > $uname -a > FreeBSD LeeDesktop 9.1-STABLE FreeBSD 9.1-STABLE #15 r250831: > Mon May 20 17:31:28 EDT 2013 > lthomas@LeeDesktop:/usr/obj/usr/src/sys/NOSOUND amd64 > $dmesg | grep CPU: > CPU: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz > (2666.81-MHz K8-class CPU) > $clang --version > FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 > Target: x86_64-unknown-freebsd9.1 > Thread model: posix > > My timing test was a file of 1 strings, of uniformly random length, > 50% between 0 and 20 bytes long, and 50% between 21 and 1000 bytes long. > Each was filled with random generated bytes in the range [1, 255]. The > test executables run their strlen on each string in the file 1000 times, > and a shell script ran the test programs alternately 100 times. Here are > the clang results; gcc results are roughly the same. I will share the > actual timing code if someone wants it, but it is pretty ugly C++ and > shell and I don't guarantee it working :-). > > Results: > > x ./times_bsd_strnlen.txt > + ./times_my_strnlen.txt > +--+ > > |+ x| > |+ x| > |+ x| > |+ x| > |+ x| > |+ x| > |+ x| > |+ x| > |+ x| > |+ x| > |+ x| > |+ x| > |+ x| > |+ x| > |+ x| > |+ x| > |+ x| > |A A| > +--+ > > N Min MaxMedian AvgStddev > x 101 1.8685486 1.8693889 1.8687506 1.8687894 0.0001484903 > + 1010.427046830.427794860.427128130.42715597 0.00010681494 > Difference at 95.0% confidence > -1.44163 +/- 3.56739e-05 > -77.1426% +/- 0.00190893% > (Student's t, pooled s = 0.000129342) > > Note: I manually shortened the histogram, as it was 100 lines long of > exactly the same. > > > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" > > Index: strnlen.c > ==
