On Nov 6, 2010, at 1:33 PM, Devin Teske <dte...@vicor.com> wrote:
>
> On Nov 6, 2010, at 9:24 AM, Nathan Whitehorn wrote:
>
>> On 11/06/10 01:04, Garrett Cooper wrote:
>>> On Fri, Nov 5, 2010 at 10:06 PM, Warner Losh <i...@bsdimp.com> wrote:
>>>>> Just to add to that (because I do find it a novel idea), 1) how
>>>>> are you going to properly prevent man in the middle attacks (SSL, TLS,
>>>>> etc?), and 2) what webserver would you use?
>>>>
>>>> https or ssh.
>>>>
>>>> We're also toying with the idea of having a partition that you could
>>>> 'dd' your certs and keys to (so any system can customize the image
>>>> with keys to make sure you were talking to who you think you are).
>>>> We'd just reserve 1MB of space on partition s3. We'd then check to
>>>> see if there was a tar ball. If so, we'd extract it and do the
>>>> intelligent thing with the keys we find there.
>>>
>>> Wouldn't it be better just to go with a read-write media solution
>>> (USB) like Matt Dillon was suggesting at today then? Then again,
>>> determining the root device to date is still a bit kludgy isn't it?
>>
>> But this breaks badly for people who don't own USB sticks of sufficient
>> size, are installing on machines without USB ports, can't boot from USB,
>> want to install from a shared medium like PXE, are installing on blades
>> with convenient shared CD drives but not USB etc. etc. Everything in the
>> world can boot from CD, and we have to ensure that continues working.
>>
>> I also have mixed feelings about needing to use a web browser to
>> instruct a web app inside a bundled web server to write a config file to
>> be interpreted by shell scripts just in order to run gpart, newfs, and
>> tar. But if you get it working, it's better than sysinstall no matter
>> how baroque.
>> -Nathan
>
>
> I find myself identifying a lot with this sentiment. We have FreeBSD running
> on 1,000+ systems (modestly) within our organization. We've found that some
> older BIOSes don't boot from USB sticks so-well. It's also hit-or-miss in
> varied combinations of old-BIOS and off-brand sticks -- had great luck with
> those super-tiny PNY's that rotate-out, though again, old BIOSes say pre-2003
> P4 or AMD K6, have issues).
>
> And since we're shipping at a rate of at least 100+ servers a year that are
> now coming without any optical drives... our solution has been to purchase
> one of these for each/every field engineer:
>
> http://www.lg.com/us/computer-products/optical-media/LG-external-dvd-burner-GP08LU30.jsp
>
> This great little device has worked wonders for over 30 field engineers
> (modestly). Out of thousands upon thousands of installs with the thing, we've
> really only had a problem with about a half-dozen individual PCs -- but after
> a BIOS upgrade, they tended to work else we slapped internal ATAPI CD-ROM's
> in there and they worked).
>
> Though, to-date what we've really found great-success in, is employing the
> latest-and-greatest ISOLINUX boot-loader w/ their `isohybrid' ISO
> post-processing program which encapsulates the disk-image contents within a
> hardware-emulation structure (you can make your ISO -- when written to USB
> stick -- appear to the BIOS as a ZIP disk, Hard Disk, etc.).
>
> We currently tell all of our field engineers to download a single ISO that is
> built with our custom build-process (which employs ISOLINUX and `isohybrid'
> from http://syslinux.zytor.com/). This ISO can then optionally -- at the
> field engineer's discretion -- either burn that ISO to optical media and use
> the LG AC-Adapterless USB optical drive (linked to above), OR ... they can
> use dd(1) to write the ISO directly to a USB stick (though again... some
> older BIOSes prefer the optical over flash media).
>
> This was, of course, quite a challenge to achieve.
>
> Here's a brief (lol -- it started that way, you got to believe me) list of
> what we did to accomplish installs with one ISO on two mediums:
>
> 1. Entire swaths of sysinstall(8) were patched
>
> Out of the 48 source files that make up sysinstall (that I count currently
> within RELENG_8), we've thus far patched 28 of them and added 1 new file.
>
> We even added a new feature ... ability to install from a directory rather
> than a device. In install.cfg, we have something akin to:
>
> nullfs=/path/to/freebsd/release
> mediaSetNullFS
>
> As implied by the name, a nullfs is done from the directory specified to
> /dist -- where sysinstall(8) expects to find the mounted distribution,
> regardless of what media you select.
>
>
>
> 2. One tiny kernel patch
>
> There's a feature in the kernel for forcing the boot-medium into read-write
> mode (which is not the default... the default is to mount read-only and
> switch it during boot-up). When we're mounting an mfsroot, especially one
> that's been customized to run scripts which build the `install.cfg' script to
> hand-off to sysinstall(8), you _want_ to mount read-write out of the gate.
>
> This feature is enabled by setting the environment variable
> vfs.root.mountfrom.options to "rw" in the loader's Forth name-space (in
> loader.rc for example).
>
> Well, this was broken. : (
>
> So we patched it : )
>
>
>
> 3. One tiny patch to the release(7) Makefile
>
> Since we rely on the release(7) process to compile both our customized
> sysinstall(8) and also our custom mfsroot (explained below), it became a real
> problem that WORLD_FLAGS was not being passed to installworld from within
> `/usr/obj'.
>
> For example, when passing in WORLD_FLAGS="-DWITHOUT_OPENSSL" (see
> src.conf(5)) we kept bombing out on the installworld phase which installs
> your newly compiled object-files from /usr/obj to an installbase of
> /usr/release (we pass CHROOTDIR=/usr/release to `make release').
>
> This was because installworld was descending into a directory like gssapi
> which it shouldn't, had WORLD_FLAGS been properly replicated down the chain
> to installworld.
>
> We fixed this with a one-liner patch to /usr/src/release/Makefile
>
>
>
> 4. A custom dialog(3) based menu system for selecting pre-scripted installs
> of many different flavors
>
>
>
> 5. A custom highly customized mfsroot
>
> We start by patching /usr/src/release/i386/boot_crunch.conf (heavily). We add
> the following utilities:
>
> cat(1), chflags(1), chmod(1), cp(1), kill(1), ln(1), mkdir(1), mv(1),
> rmdir(1), sleep(1), bsdlabel(8) (aka disklabel), init(8), ldconfig(8),
> mount(8), mount_cd9660(8), reboot(8) (aka halt), at(1) (aka atq, atrm,
> batch), awk(1), bzip2(1) (aka bunzip2, bzcat), ee(1), passwd(1), printf(1),
> tar(1), tail(1), uniq(1), chown(8) (aka chgrp), moused(8), mtree(8), pw(8),
> pwd_mkdb(8), sendmail(8) (aka newaliases, mailq), tzsetup(8), vidcontrol(1),
> dialog(1), grep(1), sort(1), and crontab(1)
>
> NOTE: Yes, all those fit into the same amount of disk-space currently
> allocated for the mfsroot, so no patching was necessary to give us a larger
> mfsroot.
>
> First, you might be asking yourself... OMG, why so many utilities added?
>
> With the exception of dialog(1), mount_cd9660(8), and sleep(1), ALL of these
> utilities were added because they are called directly by sysinstall(8).
>
> We wanted to do something absolutely crazy... we wanted to install
> FreeBSD-8.1 amd64 while booted under FreeBSD-8.1 i386. This proved to be a
> challenge because once sysinstall(8) is done laying-down the
> distribution-sets chose, it then does a chroot(2) into the newly-installed OS
> and starts calling all these lovely binaries (most listed above are called
> when performing optional setup routines, like configuring sendmail, setting
> up the mouse, etc. etc.).
>
> Well, by adding all these utilities to the mfsroot and patching sysinstall(8)
> to keep /stand around, we can have sysinstall(8) -- when told to do so --
> only call executables out of /stand after the chroot. This way, all the
> normal optional setup routines can be performed, but now with the 32-bit
> binaries that we trust, rather than the 64-bit binaries which cause the
> system to panic and reboot when invoked after the chroot normally.
>
> This heavy patching affords the user to -- once in our custom dialog(3) based
> menu system -- choose to install either 64-bit or 32-bit despite having
> booted from 32-bit kernel whereas currently sysinstall(8) must be run in an
> environment that can execute 64-bit binaries if you want to install the
> 64-bit distributions.
>
> It also affords us the ability to put legacy OSes on the disk as optional
> installs. Because our patched sysinstall will never call a single binary
> within the installed base, we're safe from problems involving incompatible
> binaries.
>
> Still talking about the customized mfsroot... once the `make release' process
> is finished and we have our highly customized mfsroot, we then patch it some
> more!
>
> We add the binary nullfs.ko kernel module -- to support our custom
> sysinstall(8) feature of mediaSetNullFS -- which will load the nullfs kernel
> module if this VFS type is not already compiled into the kernel (more on that
> later -- see description of our custom boot loader below).
>
> Then we throw a very tiny binary custom init(8) that chain-loads to
> sysinstall(8), but not before simply putting some dots up on the screen for 4
> seconds -- allowing the user to quickly hit Scroll-Lock and peruse the kernel
> messages if necessary before sysinstall launches (this was a real problem in
> 4.x where the scrollback history was not preserved so well).
>
> We add `/install.cfg' (searched-for and read automatically by sysinstall(8)
> on startup) which invokes a script which uses mount_cd9660 to re-mount the
> device we booted off of to `/cdrom'
>
> That last part is important. Remember that I mentioned that we're using
> ISOLINUX's `isohybrid' to post-process the ISO. A very nice effect is that
> our post-processed ISO is probed by the cd9660 geom provider and
> /dev/iso9660/VOLID appears in devfs when probed, despite the fact that we
> booted using zip-disk or hard-disk emulation from the BIOS.
>
> This makes the process of re-mounting the media we booted from (from within
> the mfsroot) a piece of cake, regardless of whether the field engineer wrote
> the ISO to optical media or USB stick.
>
> Lastly, before our custom mfsroot stops interrupting sysinstall(8)'s progress
> (remember, we invoked this custom shell script via `/install.cfg' which used
> mount_cd9660(8) to mount `/dev/iso9660/VOLID' to `/cdrom'), we invoke
> `/cdrom/freebsd/menu/menu' which is an ELF executable based on dialog(3)
> which presents our custom menu to the user for them to choose "Which OS?" to
> install (among which we have both 32-bit variants AND 64-bit variants of
> FreeBSD).
>
>
>
> 6. Because there's always that ONE package you'd like to have pre-installed
> along with your OS, we have an elaborate post-install configuration system.
>
> However, we later realized that (as randi once so eloquently voiced)
> installing Packages from sysinstall(8) is a bad bad idea.
>
> We patched sysinstall(8) to no longer call system executables for
> configuration setups, allowing safe install/configuration of a foreign binary
> system, though we couldn't patch the package installation process for one
> very good reason...
>
> We can't predict what +INSTALL, +POST-INSTALL, +DEINSTALL, +POST-DEINSTALL,
> and in-truth, even +CONTENTS (via @exec/@unexec) were going to do. Some
> developers have even been caught writing these scripts in perl (which is no
> longer part of the base distribution).
>
> So, for that ONE package that we just _had to have_ pre-installed before
> first-boot (for us it's perl), we make use of DIST_LOCAL. sysinstall has a
> feature allowing you to create your own custom distribution-set (local). We
> ended up throwing a few packages in there.
>
> Though to be honest, we felt that perl and kernels deserved special
> attention. Our patched sysinstall(8) also has a selector for our own custom
> kernel (we have in our release, /usr/release/R/stage/kernels/FIS which ends
> up in the ISO at /freebsd/repos/8.1-RELEASE{,-amd64}/kernels/fis.?? w/
> accompanying files). It also has a selector for installing perl -- our
> release has /usr/release/R/stage/trees/perl which is actually an unpacked
> perl-5.10.1_1 package, which ends up on the iso at
> /freebsd/repos/8.1-RELEASE{,-amd64}/perl/perl.?? w/ accompanying dist files).
> I have to say, it's nice being able to install FreeBSD-8.1 _with_ perl (and
> if I later decide to uninstall it, I can, because we retained the /var/db/pkg
> records).
>
> Oh, and before anyone asks, yes, I hand-recoded the +INSTALL, etc. files from
> the packages we converted to distribution-sets into post-install routines
> that are performed automatically.
>
>
>
> 7. And if that all wasn't enough (phew -- yeah, even that was abridged) ...
> now we get to the custom boot loader.
>
> This part isn't as important to the whole "boot from both optical _and_ USB
> stick" schtick, as it is important to our own internal operations.
>
> We have found a common problem to be that we _don't_ currently upgrade our
> Operating System often enough to keep up with hardware demands. We quite
> often find that we can't order a part anymore because it went End-of-Life
> (EOL). Then we're forced to get some new hardware that performs the same
> functionality. This can continue for some time until the operating system
> you're working with too becomes End-of-Life.
>
> If you're good, you'll still be OK... you'll just hire someone to backport
> drivers (HINT: that was my job for several years), from later revisions of
> the OS which _aren't_ EOL'd.
>
> However, that gravy-train comes to an end eventually when the gap between
> your current OS and the next-non-EOL OS is ... oh... say, 4 major revisions
> (compare FreeBSD-4.11 to FreeBSD-8.1).
>
> Well, as an enterprise model, we can't just say "well, sorry, the community
> has stopped supporting our OS, can't help you" as our customers (the ones
> running the 1000+ FreeBSD workstations/pedestals/servers) look to us for
> solutions/upgrades/everything FreeBSD.
>
> The proper solution is to get the customers to upgrade more often -- but our
> customers are the type that don't like to upgrade more than once every few
> years (and as my co-workers point-out, the FreeBSD community has worked with
> us on that in the past -- delaying the EOL of certain releases for example).
>
> However, we found a permanent solution.
>
> By building a custom boot-loader that allows us to select from a list of
> custom kernels and custom mfsroots and change boot parameters visually, we
> were able to solve the problem of not being able to install a particular OS
> because the GENERIC kernel used by the Walnut Creek/FreeBSD Mall CD/DVD's
> couldn't effectively probe the newer hardware.
>
> Of course, the boot-loader alone was only half the solution. Because
> naturally, just slapping a custom kernel and boot-loader onto the disc to get
> around your boot problems won't change the fact that the installer will still
> lay-down the GENERIC kernel. Your very first boot off the internal system
> disk will fail if the hardware is something that the GENERIC kernel can't
> utilize.
>
> The other half of the solution was to have our scripted installs lay down a
> custom kernel and configure that kernel to boot.
>
> FYI: The boot-loader I developed for this is no slouch... it's 1930-lines of
> ANS FICL Forth -- the reverse polish stack-based language for which there is
> an embedded interpreter inside FreeBSD's boot-loader). I did not customize
> the boot-loader itself, so much as I wrote extensible Forth modules capable
> of managing a hierarchical and/or deep menu structure (something far more
> advanced than what is displayed by today's boot-loader).
>
> In the full-picture, this has allowed us to do crazy things... like install
> FreeBSD-4.11 to hardware that is not currently supported by RELENG_4 even in
> it's current state (we've done crazy things like regress drivers from
> FreeBSD-6.x back to 4.x, mainly mpt, isp, mfi, em, fxp, and others as we use
> a lot of varied RAID controllers and NICs).
>
> But none of this could have been possible without the build process that we
> developed to develop the install platform itself.
>
> The build-platform consists of:
>
> autoconf-based configure.in and GNUMakefile.in
> configure script
> config.guess, config.sub, install-sh
>
> Though don't be fooled. The GNUMakefile.in (sorry BSD makefiles -- you don't
> support `override' in the target deps which means length of makefile would
> have tripled if not quadrupled) is a heavy-hitter.
>
> I got deathly tired of ripping open mfsroot's to modify the boot-loader Forth
> modules, only to be forced to close them, deconfigure the md device, run my
> tests, then repeat. I ended up developing the makefile to generate an ISO
> that contains the boot-loader, kernel(s), forth modules, etc. and have the
> menu-ing system load the mfsroot. This opened up the door for a whole new set
> up possibilities.
>
> We are now using vesamenu.c32 from ISOLINUX to throw up an advanced menu of
> options, allowing the user to select from booting memtest86, windiag, dban,
> spinrite, tufftest pro, hdt.c32, seatools, firmware updaters, or ... our
> freebsd installer.
>
> ISOLINUX these days has support for chain-loading (via memdisk module) to ISO
> files. So our makefile makes the ISO containing all the FreeBSD boot stuffs,
> and that chain-loads to mfsroot, and that re-mounts the CD-ROM where all the
> releases are.
>
> Sounds convoluted, but it dramatically increased development time. It also
> allowed the core of development to move off of FreeBSD as the development
> platform. The building of the ISO(s) can be done on any OS, including the
> Forth modules, which kernels to load into the menu, which kernels period, and
> everything about the boot process with the exception of the mfsroot itself,
> can be worked-on on some other OS like Linux, Mac OS X, Windows, or whatever
> as long as it supports CVS, has gmake, and has mkisofs (and perl for
> isohybrid ability -- though that can be disabled via ./configure).
>
> Not to mention that the disc can also install other OSes. We have in the
> works right now, a multi-distro Linux installer being developed on the same
> platform, and an N-Lite based slipstreamed Windows install too. The build
> process is actively allowing development of all three in parallel (FreeBSD
> developers say "make freebsd", linux guys say "make linux", windows guys say
> "make windows", and when we all coalesce to a release-point, we'll say "make
> all" and it will make the all-in-one super-disc capable of installing
> multiple versions of FreeBSD, multiple versions of Linux, and multiple
> versions of Windows -- all scripted installs with minimal user-interaction).
>
> Now... for the best part (which I've been saving the best for last):
>
> I've already started generalizing this stuff and releasing it to the outside
> world:
>
> http://druidbsd.sf.net/
>
> Right now, if you were to pull down the code using anonymous pserver CVS
> access, here's what you would get today:
>
> 1. A development platform already capable of producing an ISO that boots
> FreeBSD either from optical media or USB stick
> 2. The underpinnings of our very own installer that we use to perform
> cross-platform 64-bit while booted under 32-bit
> 3. Many (if not all) of the FreeBSD patches mentioned above
> 4. The excellent build process that solves all the difficult problems of how
> to get FreeBSD booted under ISOLINUX and also the abstraction layer that
> allows you to develop the boot process on platforms other than FreeBSD
> (achieved by building the chain-loaded ISO at compile-time)
> 5. A graphical boot menu system
>
> And I do think I left all the tools in there, so it should come complete with
> memtest86, memtest86+, dban, windiag, seatools, etc. etc. though you can
> compile the ISO without them if size is a concern.
>
> The resulting ISO right now is only 32MB with all the tools enabled, or 24MB
> w/o. You can get it even smaller (sub 16MB) if you rip out all the rescue
> tools that I have added too.
>
> ....
>
> I am so sad that I could not be at the MeetBSD unConference this year. Me and
> a co-worker were going to come down and meet up with you all. We already know
> many of you and many of you know us, but we've been down for the count for a
> few years.
>
> We really wanted to show you what we have been developing for the past 3-5
> years, because we feel it's reached a really stable level where it's worth
> showing. I guess we'll have to work at making an appearance for next year.
>
> It's just with all the mergers, we've not been able to get away from the
> grind-stone (especially with down-sizing). But don't worry, we'll be back in
> the community soon. Many may know us by the name VICOR. Our signatures have
> changed in our e-mail footers, but we are still the same core set of
> developers. Though I do miss the annual release parties we used to throw for
> FreeBSD (gosh, that seems like so long ago now).
Being brief (sadly it would take a millennia to abbreviate the above text
with my phone :/) -- the suggestion was to replace the DVD with a fat USB
image. That was all :).
Thanks!
-Garrett_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
