Re: txt-sysinstall scrapped
On Nov 5, 2010, at 10:17 PM, Josh Paetzel wrote: > On Friday, November 05, 2010 11:48:27 pm Garrett Cooper wrote: > >>> Just to add to that (because I do find it a novel idea), 1) how >>> are you going to properly prevent man in the middle attacks (SSL, TLS, >>> etc?), and 2) what webserver would you use? > > Without putting much analysis into it, we talked about using lighttpd, which > is BSDL. As far as another service, it would be running for the install only > which is in most circumstances something that happens locally. The whole idea makes a huge amount of sense to me. Besides the usual application HTTP servers, there are also quite a few HTTP server libraries that could be embedded into the installer. Even without SSL support, this would be a big win for both local installs and for installs within a controlled network. Supporting installs over the public Internet would be an interesting project, but certainly does not need to be a goal of the first iteration. Can't wait to see this in action! Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD development on Mac OS
On Nov 21, 2010, at 8:44 PM, m...@freebsd.org wrote: > On Sun, Nov 21, 2010 at 8:02 PM, Garrett Cooper wrote: >> On Sun, Nov 21, 2010 at 7:20 PM, Daniel O'Connor >> wrote: >>> >>> On 22/11/2010, at 13:32, m...@freebsd.org wrote: My /etc/exports file on the Mac looks like: /data -maproot=mdf:admin -network 10.211.55.0-mask 255.255.255.0 as that's the IP range that parallels is using for its virtual machines. But when I try to mount from FreeBSD (as root) I get this error: RPCPROG_MNT: RPC: Authentication error; why = Client credential too weak >>> >>> I just tried this on my MBP with parallels and it worked fine - I had >>> maproot=0:0 though. I ran into this same problem a while back but I did eventually get it to work. I can't remember now what the "client credential" in question was; it's a pretty cryptic error message. Here's my /etc/exports on Mac OS: /Users/tim -alldirs -mapall=tim -network 10.123.9.0 -mask 255.255.255.0 (Yes, I've changed the network addressing used by Parallels.) Here's the mount command that works for me on FreeBSD: mount 10.123.9.101:/Users/tim /mnt The FreeBSD VM has nothing at all NFS-related in rc.conf---I recall playing with some options but backing them all out. I've since switched over to using another FreeBSD VM as NFS server, which works well but eats battery. As soon as I can find the time to backup and reformat my Mac disk so I can set aside a case-sensitive partition, I'll probably switch back to having the Mac play NFS server. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: memstick.img is bloated with 7% 2K blocks of nulls
On Feb 11, 2011, at 11:09 AM, Julian H. Stacey wrote: > memstick.img wastes 7% with 2K blocks of nulls. > shown by: > 8f -b 0 -n 2048 -l -f Fr* > http://berklix.com/~jhs/src/bsd/jhs/bin/public/8f/ 8f.c 8f.1 > > ... > The CD & DVD images are not nearly so wasteful, see above. > As near 1G ( 959467520 FreeBSD-8.2-RC3-i386-memstick.img ) it will > soon not fit on 1G sticks. There is a big difference between laying out a read-only filesystem and laying out a read-write disk image. CD/DVD writers do pack file contents one after the other on the image. The current UFS code is designed to leave enough "slack space" to support future file writes. The strategy used by libarchive's recent ISO writer is to concatenate the file bodies into a temp file (with minimal padding between entries to meet alignment requirements) while storing directory information in memory. The final output then consists of the directory information followed by the concatenated file bodies. I suspect a similar strategy could be used to lay out and write a read-only optimized UFS image. A few folks have asked about a UFS writer for libarchive; I think it's probably feasible but I doubt very much of the existing UFS code can be recycled for such a project. Alternatively, of course, is there any way to use isofs instead of ufs for memstick.img? Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: quotas an essential feature? (was: svn commit: r218953 - stable/8/usr.sbin/sysinstall)
On Feb 25, 2011, at 3:46 PM, Steven Hartland wrote: > While I can understand some may want its not something we use on any of > our machines, and I suspect that's the case for many others. > > Given adding it means the kernel will be doing extra work and hence a > drop in performance... Does anyone have benchmark results to measure the performance hit? Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Keeping /etc/localtime up-to-date
On Mar 27, 2011, at 1:31 PM, Doug Barton wrote: > On 03/27/2011 08:38, David Wolfskill wrote: >> So it seems to me that requirements would be: >> * The content of /etc/localtime must provide the appropriate >> "zoneinfo" information, even when/usr/share/zoneinfo/* has been >> modified (or shortly thereafter, in concert with "make installworld"). > > This is more along the lines of something that would be easy to work with in > mergemaster. If I can tell what file in /usr/share/zoneinfo to compare > /etc/localtime to (ideally with fully path), I'm happy to provide a mechanism > in mergemaster to make sure it stays up to date. The best fix is to first add the ability for date(1) to print out the current timezone name. (E.g., "America/Los_Angeles") Then it's trivial for mergemaster to update /etc/localtime; just ask date(1) for the timezone name and copy the correct one over /etc/localtime. Unfortunately, I think it's currently impossible for date(1) to do this because the zoneinfo files don't store that information. This is the real reason Solaris uses a symlink; the value of the symlink gives you the timezone name. FWIW, mergemaster is not the only program that would benefit from a canonical way to obtain the name of the current timezone. Cheers, Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: [ECFT] pkgng 0.1-alpha1: a replacement for pkg_install
II. Package signing. >>> >>> That would be really nice. >> >> Right know we only planned to sign the repo database, so we can trust >> the sah256 of the packages stored in the database. Then if the package >> has the same sha256 as the one in the repo database it is considered >> trusted. >> If we want a per-package signing, we would have a tarball in a tarball. > > I really expected this to have been mentioned already, but this approach > (tarball in a tarball) is taken by Debian packages, and I don't remember > hearing of any issues related to it. I don't think it's worth discounting > from the start without giving some considerationg, but I will defer to the > people actually doing the work. If you use libarchive-style streaming, it's even pretty straightforward to read and extract such things without having to create a bunch of temporary files. You just need to be careful about compression. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: [RFC] Replacing our regex implementation
On May 8, 2011, at 5:29 PM, Gabor Kovesdan wrote: > 2, Optimizations for matching with a fixed pattern heuristic > ... First, I was thinking of putting it into TRE but now I consider a better > solution building a small library, libregexutils or such. It would decouple > this optimization from the vendor code, ... Have you asked the TRE maintainers if they would accept this change? If they would, then getting this change into TRE would benefit a lot more people than just FreeBSD's libc. (BTW, I agree that C++ in libc is a bad idea.) Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: tar has deficient error handling. send-pr fix sent 31 Jan 2011
Thanks for the reminder Julian. I've just committed a fix to trunk that I'd appreciate if you could look at. > Might be nice if someone else would read through tar. More eyes are always better! I'd especially appreciate help improving the test suite for tar. I've put a lot of work into the test suite for libarchive, but the tar front-end has seen less effort in this area. Cheers, Tim On Jun 23, 2011, at 9:20 AM, Julian H. Stacey wrote: > FreeBSD tar has deficient error handling. I sent a fix 31 Jan 2011 > http://www.freebsd.org/cgi/query-pr.cgi?pr=154407 > Please commit the fix or write & commit a better fix. > > As error codes ignored, I wonder if other poor coding may lurk in tar. > Might be nice if someone else would read through tar. > > ports/ : Tar allows some errors in ports pkg-list to go un-noticed. > Using make package, tar fails to see errors, fails to return error > to make, some pkg-list not checked properly & accumulate errors, > A bunch of errors will probably show up when tar is fixed, the > more the longer we do nothing. ( I can't remember which pkg-list > are broken, I've not kept note, The solution is to read tar & > commit a fix to tar ASAP) > > Cheers, > Julian > -- > Julian Stacey, BSD Unix Linux C Sys Eng Consultants Munich http://berklix.com > Reply below, not above; Indent with "> "; Cumulative like a play script. > Format: Plain text. Not HTML, multipart/alternative, base64, quoted-printable. > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: What is going on with ash / sh
On Nov 2, 2011, at 1:28 PM, Mark Saad wrote: > Hackers > What is going on here, if I run the following shell script, what is > the expected output . The script is named xxx > > #!/bin/sh > ps -ax | grep -v grep | grep xxx > > Here is what I see > > # sh xxx > 88318 p0 S+ 0:00.00 sh xxx > 88320 p0 R+ 0:00.00 sh xxx > 88321 p0 R+ 0:00.00 sh xxx > > Can someone explain this ? Here's my understanding: * 'sh xxx' starts (process 88318); let's call this the "parent" process. * It reads and parses the command line 'ps -ax | grep -v grep | grep xxx' * The parent process forks a copy of itself for the last 'grep xxx'. * The fork returns to the parent, the child (pid 88320) is scheduled to run later * The parent forks a copy of itself for the 'grep -v grep' * The fork returns to the parent, the child (pid 88321) is scheduled to run later * The parent runs 'ps -ax', which captures three copies of "sh xxx" (the parent which is waiting on 'ps -ax' to finish, and the two children that have not had a chance to run; note that the two children are both in state 'R' which means they'll run as soon as they get a chance, while the parent process is 'S'leeping waiting for 'ps -ax' to finish) * The two children (which started as copies of 'sh xxx') finally get a chance to run and convert themselves into the respective grep commands (via the exec() system call). The "expected output" is anywhere from one to three copies of 'sh xxx' and maybe a copy of 'grep xxx', depending on what shell you're using, how the shell parses the command line, the order in which it spawns children, and whether the fork returns to the parent or child first. The number of processors can also impact exactly when the child 'sh xxx' processes get a chance to call exec(2). Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: easy way to determine if a stream or fd is seekable
On Nov 16, 2011, at 4:24 PM, Alexander Best wrote: > On Thu Nov 17 11, Joerg Sonnenberger wrote: >> On Wed, Nov 16, 2011 at 01:14:28PM +, Alexander Best wrote: >>> On Wed Nov 16 11, Joerg Sonnenberger wrote: On Tue, Nov 15, 2011 at 08:24:50PM +, Alexander Best wrote: > one of the things i'm missing is an easy way to determine, whether a > stream or > fd is seekable. i checked the dd(1) and hd(1) sources and those tools are > performing so much stuff just to find out if this is the case, and they > still > are doing a very poor job. Isn't the primary issue that FreeBSD doesn't properly report errors for lseek(2)? I think you should start from that and not hack around the fallout... >>> >>> what do you mean? lseek(2) returns -1, when the file descriptor is not >>> seekable. i fired lseek(2) at all sorts of file types (dir, sockets, ...) >>> and it always returned the correct result. >> >> If that were the case, you wouldn't need your flag to detect seek >> support. But e.g. some devices silently ignore seek requests without >> reporting errors. At least that is what I remember from the last time >> this has brought up. > > this is the first time i hear about problems with seek requests. would be nice > to see some examples cases. was this discussed on the mailinglists? or > submitted as a problem report? There was a version of bsdtar that made the mistake of assuming lseek() would return an error. lseek() on a tape drive does not return an error, nor does it actually do anything. After a few experiments, bsdtar stopped using lseek() on FreeBSD for anything other than regular files and block devices. I believe there are other things that do support seeking, but I don't believe there is an accurate mechanism for determining whether lseek() is correctly supported. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: easy way to determine if a stream or fd is seekable
On Nov 17, 2011, at 12:55 PM, Juergen Lock wrote: >> >> After a few experiments, bsdtar stopped using lseek() on >> FreeBSD for anything other than regular files and block >> devices. I believe there are other things that do support >> seeking, but I don't believe there is an accurate mechanism >> for determining whether lseek() is correctly supported. > > Ah is that the reason why my patch never made it into FreeBSD 9? > I'm talking about this thread, where I also commented on seeking > on tape: > > http://docs.freebsd.org/cgi/mid.cgi?20100220101724.GA26604 > (Re: "tar tfv /dev/cd0" speedup patch) > > entire thread here: > http://markmail.org/message/nfznipqik3tuhbqp > > Cheers, > Juergen (who would still like to see a faster "tar tfv /dev/cd0"... :) I would like to see that as well. Take a look at http://libarchive.googlecode.com/svn/trunk/libarchive/archive_read_open_filename.c Especially the comments about detecting "disk-like" devices. I rewrote a bunch of this code to introduce an explicit notion of "strategy" so that we could optimize access to a variety of different devices. This code has a notion of "disk-like" file descriptors and some optimizations for such. There are some comments in there outlining similar optimizations that could be made for "tape-like" or "socket-like" devices. Cheers, Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: easy way to determine if a stream or fd is seekable
On Nov 17, 2011, at 1:48 PM, Alexander Best wrote: > On Thu Nov 17 11, Dieter BSD wrote: >>> lseek() on a tape drive does not return an error, nor does it >>> actually do anything. >> >> IIRC some tape drives can seek, while others cannot. >> Vague memories that it is supposed to be possible to put a >> filesystem on a DECtape and mount the filesystem. > > or how about the following: > > 1) if the file argument we're seeking on is a tape drive, just do a regular > seek operation. > 2) afterwards use ftell() to verify that the seek REALLY happend. if it > didn't, > return -1 and set errno = EBADF. ftell() can't tell whether the seek really happened or not. All it can do is ask the kernel, and the kernel doesn't know. Here is my current understanding: When you call lseek(), the kernel just stores the requested offset in the file descriptor. lseek() always succeeds because storing the requested offset in the file descriptor always succeeds. (Except that the kernel specially checks if the file descriptor is a pipe.) ftell() just obtains the data from the file descriptor. So it always succeeds and always returns the data from the previous seek request, regardless of whether that seek actually did anything. With the next read or write request, the device driver may inspect the offset information in the file descriptor. Or it can ignore it. Almost all tape device drivers ignore it. Filesystems uniformly support it. Disk drivers uniformly support it. Other drivers vary considerably. In short: No, there is no "easy way" to determine if an arbitrary stream or fd is seekable. If you try to create a function that determines this, be certain to include "Don't Know" as a possible return value. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: easy way to determine if a stream or fd is seekable
On Nov 18, 2011, at 2:10 PM, Juergen Lock wrote: > On Fri, Nov 18, 2011 at 12:00:07PM -0800, Tim Kientzle wrote: >> >> On Nov 17, 2011, at 12:55 PM, Juergen Lock wrote: >> >>>> >>>> After a few experiments, bsdtar stopped using lseek() on >>>> FreeBSD for anything other than regular files and block >>>> devices. >>> >>> Ah is that the reason why my patch never made it into FreeBSD 9? >>> …. >>> Cheers, >>> Juergen (who would still like to see a faster "tar tfv /dev/cd0"... :) >> >> I would like to see that as well. >> >> Take a look at >> >> http://libarchive.googlecode.com/svn/trunk/libarchive/archive_read_open_filename.c >> > > Ah so it's `just' a slow release cycle? > > % grep DIOCGMEDIASIZE > /home/ncvs/src/lib/libarchive/archive_read_open_filename.c,v > % > > When will we see this code in FreeBSD? 10.0? 9.1? 8.3? :) Definitely in FreeBSD 10. Unfortunately libarchive 3.0 breaks the API and ABI, so it can't be imported into 9.1 or 8.3 wholesale. It should be possible to back port select pieces, though. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: easy way to determine if a stream or fd is seekable
On Nov 18, 2011, at 12:31 PM, Alexander Best wrote: > On Fri Nov 18 11, Tim Kientzle wrote: >> >> Take a look at >> >> http://libarchive.googlecode.com/svn/trunk/libarchive/archive_read_open_filename.c >> >> Especially the comments about detecting "disk-like" devices. >> I rewrote a bunch of this code to introduce an explicit >> notion of "strategy" so that we could optimize access >> to a variety of different devices. >> >> This code has a notion of "disk-like" file descriptors and >> some optimizations for such. There are some comments >> in there outlining similar optimizations that could be made >> for "tape-like" or "socket-like" devices. > > great you posted that file as reference. i believe most of the stuff done > there > should actually be done within lseek(). Libarchive runs on a lot of systems other than FreeBSD. FreeBSD is not the only Unix-like system with this issue, so that code isn't going to go out of libarchive regardless. If you think those same ideas can be used in dd or hd to speed them up, please send your patches. The key point: You cannot unconditionally call lseek() to skip over data. Instead, treat lseek() as an optimization that can be used under some circumstances. The question then becomes one of figuring out when that optimization can be enabled. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: cron(8) mis-feature with @reboot long after system startup
On Nov 25, 2011, at 2:08 PM, Doug Barton wrote: > On 11/25/2011 08:09, Cy Schubert wrote: >> You're right. Sorry. It was late, after a long night of O/T. > > Actually I was in the same boat, which is why my reply was even grumpier > than usual, sorry. > > Meanwhile I like your suggestion of having cron check that it's within > $time_period before running the @reboot jobs Hmmm… I thought rc.d distinguished between boot-time and non-boot-time starts already. It might be simpler and more accurate to add a crond command-line option (--run-reboot-scripts) and have the rc.d scripts only pass that in when crond is started at boot time. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: cron(8) mis-feature with @reboot long after system startup
>>> Add an option to cron to check lastlog and if within 5 or 10 minutes
>>> of the last reboot, then call run_reboot_jobs().
>>
>> Depending on timestamps might be okay as a temporary quick-and-dirty
>> workaround, but there is likely to be a case where it will also do the
>> wrong thing. ??What if you need to restart cron within the 5-10 minutes?
>>
>> Maybe something like: rc script sets a flag, cron_reboot script checks
>> and resets flag. ??The flag could be a file ("> /tmp/rebooting_system").
>> Better yet, a run-the-reboot-script command line option could be added
>> to cron.
>
> cron(8) already creates a pid file. Maybe it wouldn't be so bad to just check
> the timestamp of that before creating its new pid file along with comparing
> to system boottime so we could make an accurate measure whether it should or
> should not run the @reboot extension at least down to 1 minute.
Timing-based approaches like this tend to be
brittle and fail in strange ways.
Remember that "boot time" basically means "run by /etc/rc".
Is there any way for /etc/rc.d/cron to notice that it's being
run by /etc/rc and pass an extra flag to cron in that case?
(If not, then perhaps /etc/rc should set some kind of marker
when it starts --- such as an environment variable or a
file on disk --- that scripts such as /etc/rc.d/cron can use
to distinguish boot-time startup from manual startup.)
Tim
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
BeagleBone?
Just got a BeagleBone in the mail and so far, it seems like fun: * Under $100 * Relatively modern Cortex-A8 ARM CPU (TI AM3358) * Built-in Ethernet, USB console, etc. So far, I've gotten console access from my FreeBSD laptop and am starting to tinker with a nanobsd-like script to build a bootable SD image. (By copying the MLO and u-boot.img files; nothing FreeBSD-specific yet.) Next step: Compile the arm/uboot boot loader and see if I can get that to load and run. Anyone else tinkering with one of these? Any hints? ;-) Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD has serious problems with focus, longevity, and lifecycle
On Jan 18, 2012, at 2:44 AM, Robert Watson wrote: > > ... perhaps what is really called for is breaking out our .0 release > engineering entirely from .x engineering, with freebsd-update being in the > latter. This is a great idea! In particular, it would allow more people to be involved. There's a practical limit to the number of people that can be involved in any single release process; having multiple groups handling separate releases (for example, we might have a group working just on 8-STABLE releases, so that 8.3, 8.4, etc, weren't competing for resources with the more complex 9.0 release). Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Jumpstart on FreeBSD 9.0
On Feb 20, 2012, at 9:24 AM, ego...@ramattack.net wrote: > > For example... tar -C ... -pxvf does not work with some files (althout you > can mount iso and later do an rsync preserving file flags), The FreeBSD ISOs are now being built with a tool that builds ISO images differently than before. tar has some restrictions in what ISO images it can read, and the FreeBSD ISOs no longer meet those restrictions. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
How does loader(8) decide where to load the kernel?
I have ubldr loading the ELF kernel on BeagleBone and am now trying to untangle some of the hacks I used to get this working. Unfortunately, there's one area of the common loader(8) code that I really don't understand: How does sys/boot/common/load_elf.c determine the physical address at which to load the kernel? __elfN(loadfile) has the following comment: [The file] will be stored at (dest). But that's not really true. For starters, loadfile maps dest through archsw.arch_loadaddr. (This mechanism seems to only be used on ia64 and pc98, though the result is later discarded on those platforms.) Loadfile then passes the value to loadimage which does very strange things: On i386, amd64, powerpc, and arm, loadimage subtracts the dest value from the address declared in the actual ELF headers so that the kernel always gets loaded into low memory. (there's some intermediate bit-twiddling I'm glossing over, but this is the general idea). On other platforms, the dest value passed into loadimage is entirely ignored. So it looks like the arch_loadaddr manipulations used on ia64 and pc98 are in fact not used. In my case, I have a kernel linked against a virtual address of 0xc020 that I want to load into a physical address of 0x8020. I can make it work by having arch_loadaddr return a value of 0x4000 (which then gets subtracted from the virtual address in the ELF file to produce the physical address I need), but this seems really byzantine. How is this *supposed* to work? Anybody know? Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: How does loader(8) decide where to load the kernel?
On May 7, 2012, at 6:57 AM, John Baldwin wrote: > On Saturday, May 05, 2012 1:06:13 am Tim Kientzle wrote: >> I have ubldr loading the ELF kernel on BeagleBone and am now >> trying to untangle some of the hacks I used to get this working. >> >> Unfortunately, there's one area of the common loader(8) code >> that I really don't understand: How does sys/boot/common/load_elf.c >> determine the physical address at which to load the kernel? >> >> __elfN(loadfile) has the following comment: >> >> [The file] will be stored at (dest). >> >> But that's not really true. For starters, loadfile maps dest >> through archsw.arch_loadaddr. (This mechanism seems >> to only be used on ia64 and pc98, though the result is >> later discarded on those platforms.) >> >> Loadfile then passes the value to loadimage which does >> very strange things: >> >> On i386, amd64, powerpc, and arm, loadimage subtracts >> the dest value from the address declared in the actual ELF >> headers so that the kernel always gets loaded into low memory. >> (there's some intermediate bit-twiddling I'm glossing over, but >> this is the general idea). > > The bit twiddling is supposed to be the equivalent of subtracting > KERNBASE from the load address. On both i386 and amd64, there is > a direct mapping of the kernel text such that KERNBASE maps address > 0, etc. By default on i386 KERNBASE is 0xc000. Exactly my problem. This all assumes that you're loading the kernel into low memory. On the AM3358, the DRAM starts at 0x8000 on boot, so I'm trying to find a clean way to convince the loader's ELF code to put the kernel there. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: How does loader(8) decide where to load the kernel?
On May 8, 2012, at 2:39 AM, Andrew Turner wrote: > On Mon, 7 May 2012 22:32:10 -0700 > Tim Kientzle wrote: > >> >> On May 7, 2012, at 6:57 AM, John Baldwin wrote: >>> >>> The bit twiddling is supposed to be the equivalent of subtracting >>> KERNBASE from the load address. On both i386 and amd64, there is >>> a direct mapping of the kernel text such that KERNBASE maps address >>> 0, etc. By default on i386 KERNBASE is 0xc000. >> >> Exactly my problem. This all assumes that you're loading >> the kernel into low memory. >> >> On the AM3358, the DRAM starts at 0x8000 >> on boot, so I'm trying to find a clean way to convince >> the loader's ELF code to put the kernel there. > > I have a script at [1] that builds an image to load the kernel directly > from U-Boot. It figures out where to tell U-Boot to load a kernel by > using readelf to find the value of physaddr and kernbase to use to > calculate what physical addresses to use to load the kernel to and > where the first instruction to execute is. I have a script at [2] that builds a FreeBSD image for BeagleBone that chain loads U-Boot, then ubldr, then loads the ELF kernel from UFS. Works very well, but it relies on some hacks to the common load_elf code that I'd like to find a clean way to generalize. [2] https://github.com/kientzle/freebsd-beaglebone ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: How does loader(8) decide where to load the kernel?
On May 10, 2012, at 5:32 AM, Marcel Moolenaar wrote: > > On May 8, 2012, at 1:32 AM, Tim Kientzle wrote: >>>> On i386, amd64, powerpc, and arm, loadimage subtracts >>>> the dest value from the address declared in the actual ELF >>>> headers so that the kernel always gets loaded into low memory. >>>> (there's some intermediate bit-twiddling I'm glossing over, but >>>> this is the general idea). >>> >>> The bit twiddling is supposed to be the equivalent of subtracting >>> KERNBASE from the load address. On both i386 and amd64, there is >>> a direct mapping of the kernel text such that KERNBASE maps address >>> 0, etc. By default on i386 KERNBASE is 0xc000. >> >> Exactly my problem. This all assumes that you're loading >> the kernel into low memory. >> >> On the AM3358, the DRAM starts at 0x8000 >> on boot, so I'm trying to find a clean way to convince >> the loader's ELF code to put the kernel there. > > Look at what I did for ia64. All that frobbing should be done > in the machine specific implementation of arch_copyin, arch_copyout > and arch_readin. It's a kluge to do it in elf_loadimage. That sounds like a reasonable approach. I've started working down that path… but it looks like I'll have to fix a lot of the FDT code along the way. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: How does loader(8) decide where to load the kernel?
On May 12, 2012, at 4:36 PM, Tim Kientzle wrote: > > On May 10, 2012, at 5:32 AM, Marcel Moolenaar wrote: > >> On May 8, 2012, at 1:32 AM, Tim Kientzle wrote: >>> >>> On the AM3358, the DRAM starts at 0x8000 >>> on boot, so I'm trying to find a clean way to convince >>> the loader's ELF code to put the kernel there. >> >> Look at what I did for ia64. All that frobbing should be done >> in the machine specific implementation of arch_copyin, arch_copyout >> and arch_readin. It's a kluge to do it in elf_loadimage. > > That sounds like a reasonable approach. I've started > working down that path… but it looks like I'll have to fix > a lot of the FDT code along the way. I'm getting close. In particular, I've reworked the FDT code so that it correctly uses copyout() to parse data in the kernel. Of course, in order to use libfdt to actually manipulate the device tree, I have to copyout() the entire blob into a malloc'ed buffer. So now I need to understand how to copyin() the blob back into the kernel address space. Should I overwrite the FDT in the kernel with the edited FDT? That doesn't feel quite right, but it's essentially what the FDT code here was trying to do before. I could also implement something similar to file_loadraw() that would allow me to create a new module from an in-memory buffer. I think that might be a little cleaner. Is there already something like this? Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: How does loader(8) decide where to load the kernel?
On May 15, 2012, at 9:38 AM, Rafal Jaworowski wrote: >> >> Should I overwrite the FDT in the kernel with the >> edited FDT? That doesn't feel quite right, but it's >> essentially what the FDT code here was trying to >> do before. > > A given DTB (loaded dynamically or statically embedded in the kernel) has > some small amount of free space inside it so it is possible to perform > in-place modifications, adding properties / nodes until you run our of this > space (libfdt code will handle cases when exceeding this and would fail > gracefully). The fixup mechanism currently implemented does minor > modifications of the device tree in this fashion. I've fixed the code in sys/boot/fdt/ to do all access through copyout/copyin. I'm waiting for a "make universe" to make sure I didn't break something inadvertently, and will probably commit it tomorrow. Right now, this just copies the DTB to a malloc-ed area, modifies the copy and then writes it back to the same place (either in the kernel or in a separately-loaded blob). It seems to work correctly even when copyout/copyin are doing address translations. > ... if you'd like to modify the blob in a major way you need to relocate it > into a new location with more padding …. create a DTB metadatum (as if the > DTB was loaded dynamically from standalone blob file) with sufficient space, > …. and pass this new copy (as part of regular loader(8) metadata) to the > kernel for processing. The building blocks are there (DTB loaded as metadata) > and should work. Yes, I see how that would work. Is there a real use for this? I could take a look at it once I'm finished with the current ubldr work I'm doing. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: GSoC Project: Automated Kernel Crash Reporting System - Discussion
On May 18, 2012, at 7:51 AM, Mel Flynn wrote: > On 17-5-2012 14:53, Mateusz Guzik wrote: >> On Wed, May 16, 2012 at 11:37:44PM +0300, tza...@it.teithe.gr wrote: > >>> Nice. What about curl over the HTTPS protocol? >>> >> >> curl would be ok, except it's not in the base system. > > For this reason, it's probably best to use tar(1) to package up multiple > files and implement http put support in libfetch(3). You may also need > to implement 305 Use Proxy support. Depends on where the files are coming from. If you have files on disk, then tar(1) might be a good choice. If you're going to have to construct the files, then you can maybe avoid writing them to disk by using libarchive(3) directly instead of going through the tar command-line interface. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Geom_mbr vs. SSD disks (was: proper newts options for SSD disks)
On May 19, 2012, at 11:36 AM, rozhuk...@gmail.com wrote:
> Do not use MBR (or manually do all to align).
> 63 - not 4k aligned.
Right now, the "-a" alignment option for "gpart add" is broken when
used with MBR partitions. It looks like the gpart command uses
it to correctly align the start/end, but then the actual MBR geom code
does another alignment pass that rounds the start/size to
a multiple of gpt_sectors, which defaults to 63.
This seems problematic.
It's tempting to change sys/geom/part/g_part_mbr.c so that
it skips this additional alignment when the geometry has
defaulted. Something like this:
Index: sys/geom/part/g_part_mbr.c
===
--- part/g_part_mbr.c (revision 235597)
+++ part/g_part_mbr.c (working copy)
@@ -211,6 +211,7 @@
start = gpp->gpp_start;
size = gpp->gpp_size;
+ if (sectors != 63 || basetable->gpt_heads != 255) {
if (size < sectors)
return (EINVAL);
if (start % sectors) {
@@ -221,6 +222,7 @@
size = size - (size % sectors);
if (size < sectors)
return (EINVAL);
+ }
if (baseentry->gpe_deleted)
bzero(&entry->ent, sizeof(entry->ent));
I'm not really certain I understand all of the implications
of this change, though.
Tim
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: proper newfs options for SSD disk
On May 22, 2012, at 7:40 AM, Warren Block wrote: > On Tue, 22 May 2012, Matthias Apitz wrote: > >> El día Tuesday, May 22, 2012 a las 07:42:18AM -0600, Warren Block escribió: >> >>> On Tue, 22 May 2012, Matthias Apitz wrote: >>> El día Sunday, May 20, 2012 a las 03:36:01AM +0900, rozhuk...@gmail.com escribió: > > Do not use MBR (or manually do all to align). > 63 - not 4k aligned. To create the above shown partition layout I have not used gpart(8); I just said: # fdisk -I /dev/ada0 # fdisk -B /dev/ada0 ... What is wrong with this procedure? >>> >>> The filesystem partitions end up at locations that aren't even multiples >>> of 4K. This can reduce performance. How much probably depends on the >>> SSD. >> >> But this is then rather a bug in fdisk(8) and not a PEBKAC, or? :-) > > A bug in the design of MBR. Which probably can be forgiven, considering when > it was created and the other problems with it. :) > > gpart's alignment option can be used with MBR slices and bsdlabel partitions. GPart's alignment option doesn't work for MBR slices. It rounds to the requested alignment, and then rounds again to the track size, which defaults to 63 sectors. I'm not convinced this is a bug in the design of MBR. I don't think anything in the MBR design requires that partitions be track-aligned. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: cleaning /usr/obj before copying it to USB key
On Jun 9, 2012, at 7:35 AM, Matthias Apitz wrote:
>
> To use the (booted) USB key later to install other laptops or netbooks I
> enrich the key with /usr/src and /usr/obj as:
>
> # cd /usr
> # cp -Rp src /mnt/usr
> # cp -Rp obj /mnt/usr
>
> my problem is that the both 'cp -Rp ...' commands takes many hours (12
> and six hours) because they are transferring a lot(!!!) of small files;
As someone else pointed out, flash drives are pretty slow
when making a lot of small writes.
You can speed things up a lot by creating the image locally
then copying the image to USB. Here are parts of a shell
script I've been using for something similar:
# Create an empty file IMG_SIZE bytes
dd if=/dev/zero of=${IMG} bs=1 seek=${IMG_SIZE} count=0
# Attach it as a virtual disk device
MD=`mdconfig -a -t vnode -f ${IMG}`
# Partition the virtual disk
gpart create -s MBR ${MD}
gpart add -t freebsd ${MD}
gpart set -a active -i 1 ${MD}
# Format and mount
newfs ${MD}s1 >/dev/null
mount /dev/${MD}s1 /mnt
… copy stuff …
# Unmount and detach the virtual disk
umount /dev/${MD}s1
mdconfig -d -u ${MD}
# Copy the disk image to your flash drive
dd if=${IMG} of=${SDCARD} bs=8m
> I have had a look into /usr/obj and it seems that after 'makeworld' and
> 'makekernel' there are left over a lot of temporary files from the build
> processes...
>
> is there a clean way to remove those files before 'cp -Rp obj /mnt/usr'
> while the result is still useful for another make install with DESTDIR=/mnt ?
You can delete all of the '.o' files using a command like this:
find /usr/obj -name '*.o' | xargs rm
With a little experimenting, you should be able to extend this
to remove most of the intermediate files.
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Better error messages for command not found (was Re: Pull in upstream before 9.1 code freeze?)
On Jul 4, 2012, at 4:41 PM, Jason Hellenthal wrote: > > On Wed, Jul 04, 2012 at 03:59:29PM -0700, Doug Barton wrote: >> On 07/04/2012 15:55, Jason Hellenthal wrote: >>> Seeing as sudo plays a big part of this >> >> No ... not only is sudo not a necessary component, it shouldn't be >> involved at all. The feature works on debian/ubuntu for regular >> userspace commands. >> > > What are they using to authenticate for the install ? do you know ? Huh? What install? Who's talking about install? The version of this I've seen looks like this: $ svn co https://some.url/ svn: Command not found. To use this command, install one of the following packages: devel/subversion devel/subversion-freebsd devel/subversion16 That's all it does: It just prints out a more informative error message. It does not install anything, it requires no special permissions, and does not (as far as I can see) introduce any security or performance problems. The implementation is pretty simple: * A tool for building a database that maps command names to package names. (This would run against a ports tree or package repository. Conceptually, it's pretty similar to how port/package indexes get built today.) * Some way to distribute that database (Probably as part of ISO releases, maybe extend 'portsnap' or 'pkg_add' to update it?) * A program to look up command names in that database and print out the results. * A shell hook to run said program whenever a "command not found" error occurs. As a first prototype, the database could just be a text file and the look up program could be a shell script that uses grep and sed. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Better error messages for command not found (was Re: Pull in upstream before 9.1 code freeze?)
On Jul 4, 2012, at 5:37 PM, Mike Meyer wrote: > On Wed, 04 Jul 2012 17:34:13 -0700 > Doug Barton wrote: >>> As a first prototype, the database could just be a text file >>> and the look up program could be a shell script that uses >>> grep and sed. >> Right-O. The db should almost certainly be updated and distributed as >> part of the (already automated) INDEX generation and distribution process. > > There are already tools in the ports tree that do the > search. Personally, I use portsearch. Very close. Unfortunately, portsearch seems to require an installed /usr/ports tree. A system that generated a database and searched that database would be useful also by people who preferred packages over ports. > Tweaking them to use INDEX (or > tweaking the makefile to generate their database with INDEX) should be > straightforward. INDEX can't be used in its current form because it doesn't have filenames (just port names). But yes, a lot of the key pieces already exist in various places. Just need someone to piece it together and make it actually work. I still think this would be a good addition to the base system and would like to see it included in the standard release ISOs so that new FreeBSD users can more quickly find the software they expect. I'm curious whether the earlier objections were due to misunderstandings about auto-install. Auto-install would be problematic, but the feature being discussed here does not require installation. Just better error messages. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Better error messages for command not found (was Re: Pull in upstream before 9.1 code freeze?)
On Jul 4, 2012, at 6:42 PM, Mike Meyer wrote: > On Wed, 4 Jul 2012 18:03:05 -0700 > Tim Kientzle wrote: >> I'm curious whether the earlier objections were due to >> misunderstandings about auto-install. Auto-install would >> be problematic, but the feature being discussed here does not >> require installation. Just better error messages. > > My objection was not due to misunderstanding about auto-install. I > find the feature annoying - spewing a bunch of crap at me because of a > typo. It annoys me far more often than it actually helps me, because > more often than not the "missing command" is a typo, *not* an attempt > to run a command I don't have. I.e., if I type mmap instead of nmap, I > get: > > mwm@IPGhosterCrawlerI:~$ mmap > No command 'mmap' found, did you mean: > Command 'jmap' from package 'openjdk-6-jdk' (main) > Command 'jmap' from package 'openjdk-7-jdk' (universe) > Command 'gmap' from package 'gmap' (multiverse) > Command 'gmap' from package 'scotch' (universe) > Command 'tmap' from package 'emboss' (universe) > Command 'smap' from package 'slurm-llnl' (universe) > Command 'pmap' from package 'procps' (main) > Command 'moap' from package 'moap' (universe) > Command 'umap' from package 'libunicode-map8-perl' (main) > Command 'map' from package 'sgt-puzzles' (universe) > Command 'amap' from package 'amap-align' (universe) > mmap: command not found I agree this is excessive. In this case, a better UI might be to separate the typo handling from the package reference, so that you would see something like this: $ mmap mmap: command not found Perhaps you meant one of: jmap, gmap, tmap, smap, (8 others) That way, if you did really mean jmap, you could try again and see $ jmap jmap: command not found Try installing one of the following packages: java/openjdk-6-jdk java/openjdk-7-jdk I agree also about the convenience of having a tool that can identify the port/package that would install a particular library (or other file). I too have run into the situation where I need a particular library and can't figure out what the package is called. > bash and zsh already have command_not_found handlers. I don't really > object to that functionality to sh and tcsh. Just *don't turn it on by > default*. I don't think I'd even object to setting those handlers in > /usr/share/skel. How about if it's on by default (in the default /etc/cshrc) but very easy to turn off? The primary point of such a feature is to help people new to FreeBSD. Someone who sits down at a new FreeBSD installation and wants to run 'nmap' can then get up to speed a lot more quickly, since they'll be pointed at the right package right away. > Of course, I might turn around and ask that we add support for command > correction ala zsh to sh & tcsh via those hooks if they get added. > Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
How to Expose Chip-level Ethernet Statistics?
I believe that some of the issues I'm having with this Ethernet driver might be easier to diagnose if I could expose the chip-level statistics counters (especially queue overrun counts). Is there a standard way to do this? I've looked at systat, netstat, and ifconfig but haven't yet found a standard tool that queries this sort of information. (If I could find that, I could figure out which ioctl it used…) Pointers appreciated… In particular, if there's another Ethernet driver that does this well, I can use that for a reference. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: How to Expose Chip-level Ethernet Statistics?
On Aug 4, 2012, at 12:51 PM, Ian Lepore wrote: > On Sat, 2012-08-04 at 12:21 -0700, Tim Kientzle wrote: >> I believe that some of the issues I'm having with this >> Ethernet driver might be easier to diagnose if I could >> expose the chip-level statistics counters (especially queue >> overrun counts). >> >> Is there a standard way to do this? >> > > I don't know if this is exactly what you mean, but have a look at > src/tools/tools/ifinfo, and find some examples of drivers that fill in > that info by grepping for ifmib_iso_8802_3. > > (I really know nothing about this stuff, except that your request > triggered a memory that the atmel if_ate driver gathers some stats that > I've not seen in most other drivers.) Thanks, Ian! That's almost exactly what I'm looking for. Only tricky point: I don't immediately see where the standard MIB allows me to run a function when the sysctl query runs (which will be necessary if I want to expose the on-chip counters). That might lead me to use a separate sysctl tree for this. I'll also take at the atmel if_ate driver…. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SMP Version of tar
On Oct 1, 2012, at 9:51 AM, Brandon Falk wrote: > I would be willing to work on a SMP version of tar (initially just gzip or > something). > > I don't have the best experience in compression, and how to multi-thread it, > but I think I would be able to learn and help out. > > Note: I would like to make this for *BSD under the BSD license. I am aware > that there are already tools to do this (under GPL), but I would really like > to see this existent in the FreeBSD base. > > Anyone interested? Great! First rule: be skeptical. In particular, tar is so entirely disk-bound that many performance optimizations have no impact whatsoever. If you don't do a lot of testing, you can end up wasting a lot of time. There are a few different parallel command-line compressors and decompressors in ports; experiment a lot (with large files being read from and/or written to disk) and see what the real effect is. In particular, some decompression algorithms are actually faster than memcpy() when run on a single processor. Parallelizing such algorithms is not likely to help much in the real world. The two popular algorithms I would expect to benefit most are bzip2 compression and lzma compression (targeting xz or lzip format). For decompression, bzip2 is block-oriented so fits SMP pretty naturally. Other popular algorithms are stream-oriented and less amenable to parallelization. Take a careful look at pbzip2, which is a parallelized bzip2/bunzip2 implementation that's already under a BSD license. You should be able to get a lot of ideas about how to implement a parallel compression algorithm. Better yet, you might be able to reuse a lot of the existing pbzip2 code. Mark Adler's pigz is also worth studying. It's also license-friendly, and is built on top of regular zlib, which is a nice technique when it's feasible. There are three fundamentally different implementation approaches with different complexity/performance issues: * Implement as a stand-alone executable similar to pbzip2. This makes your code a lot simpler and makes it reasonably easy for people to reuse your work. This could work with tar, though it could be slightly slower than the in-process version due to the additional data-copying and process-switch overhead. * Implement within libarchive directly. This would benefit tar and a handful of other programs that use libarchive, but may not be worth the complexity. * Implement as a standalone library with an interface similar to zlib or libbz2 or liblzma. The last would be my personal preference, though it's probably the most complex of all. That would easily support libarchive and you could create a simple stand-alone wrapper around it as well, giving you the best of all worlds. If you could extend the pigz technique, you might be able to build a multi-threaded compression library where the actual compression was handled by an existing single-threaded library. Since zlib, bzlib, and liblzma already have similar interfaces, your layer might require only a thin adapter to handle any of those three. *That* would be very interesting, indeed. Sounds like a fun project. I wish I had time to work on something like this. Cheers, Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SMP Version of tar
> Someone might want to ask if parallelizing tar is even possible. Answer: Yes. Here's a simple parallel version of tar: find . | cpio -o -H ustar | gzip > outfile.tgz There are definitely other approaches. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SMP Version of tar
On Oct 7, 2012, at 10:00 AM, Wojciech Puchar wrote: >> I would be willing to work on a SMP version of tar (initially just gzip or >> something). >> >> I don't have the best experience in compression, and how to multi-thread it, >> but I think I would be able to learn and help out. > > gzip cannot - it is single stream. gunzip commutes with cat, so gzip compression can be multi-threaded by compressing separate blocks and concatenating the result. For proof, look at Mark Adler's pigz program, which does exactly this. GZip decompression is admittedly trickier. > bzip2 - no idea bzip2 is block oriented and can be multi-threaded for both compression and decompression. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SMP Version of tar
On Oct 8, 2012, at 3:21 AM, Wojciech Puchar wrote: >> Not necessarily. If I understand correctly what Tim means, he's talking >> about an in-memory compression of several blocks by several separate >> threads, and then - after all the threads have compressed their > > but gzip format is single stream. dictionary IMHO is not reset every X > kilobytes. > > parallel gzip is possible but not with same data format. Yes, it is. The following creates a compressed file that is completely compatible with the standard gzip/gunzip tools: * Break file into blocks * Compress each block into a gzip file (with gzip header and trailer information) * Concatenate the result. This can be correctly decoded by gunzip. In theory, you get slightly worse compression. In practice, if your blocks are reasonably large (a megabyte or so each), the difference is negligible. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
FreeBSD on RaspberryPi
WARNING: This is still highly experimental and by no means ready for "production use", but some folks might find it interesting. To boot FreeBSD on your RaspberryPi, you'll need: 1) A RaspberryPi. 2) A serial cable similar to this one: www.adafruit.com/products/954 3) An SD card of 2GB or larger Download this 111MB file (~1.6G uncompressed): http://people.freebsd.org/~kientzle/FreeBSD-RPI-B-r242362-2012-10-30.img.xz Uncompress it, dd it onto your SD card, pop it in and apply power. (The serial cable above can also provide power; just leave the red lead disconnected until you get the SD card plugged in.) KNOWN BROKEN STUFF * There's no framebuffer/syscons yet. Hence the need for a serial cable. * The memory is mis-probed (actually a boot loader problem, not a FreeBSD kernel issue), so you'll only get to use 128MB (you might be able to change this for a single boot by breaking into ubldr and editing the FDT by hand) * There has been NO attempt to reduce the footprint of this image. It's a completely stock build of FreeBSD-CURRENT. (Actually, I have turned off sendmail and a few other things in rc.conf, but compensated by building world with full debug enabled.) * I've personally not tried USB or Ethernet and have no idea if they work. HOW TO BUILD YOUR OWN IMAGE The script I used to build this image is at: github.com/kientzle/freebsd-beaglebone (It was originally developed for BeagleBone.) Enjoy! Boot message (edited for length): DRAM: 128 MiB WARNING: Caches not enabled MMC: bcm2835_sdh: 0 Using default environment In:serial Out: serial Err: serial Net: Net Initialization Skipped No ethernet found. Hit any key to stop autoboot: 0 reading uEnv.txt 74 bytes read Importing environment from mmc ... reading ubldr 728201 bytes read ## Starting application at 0x0254 ... Consoles: U-Boot console Compatible API signature found @7b75220 Number of U-Boot devices: 1 FreeBSD/armv6 U-Boot loader, Revision 1.2 (root@fci386.localdomain, Wed Nov 7 01:54:47 PST 2012) DRAM:128MB Device: disk - /boot/kernel/kernel data=0x300238+0x1ec7c syms=[0x4+0x70f80+0x4+0x56d3c] Hit [Enter] to boot immediately, or any other key for command prompt. Booting [/boot/kernel/kernel]... fdt_start: 0x003DC138 fdt_reg_valid(): reg#0 (start: 0x0 size: 0x800) valid! Kernel entry at 0x100100... Kernel args: (null) … Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #3: Wed Nov 7 04:23:11 PST 2012 root@fci386.localdomain:/usr/obj/arm.armv6/usr/src/sys/RPI-B arm CPU: Sheeva 88SV581x rev 7 (Marvell core) Supported features: ARM_ISA THUMB2 JAZELLE ARMv4 Security_Ext WB enabled LABT branch prediction enabled 16KB/32B 4-way instruction cache 16KB/32B 4-way write-back-locking-C data cache real memory = 134217728 (128 MB) avail memory = 125685760 (119 MB) … FreeBSD/arm (raspberry-pi) (ttyu0) login: root FreeBSD 10.0-CURRENT (RPI-B) #3: Wed Nov 7 04:23:11 PST 2012 Welcome to FreeBSD! … root@raspberry-pi:/root # top last pid: 490; load averages: 0.63, 0.30, 0.12up 0+00:01:32 12:47:50 7 processes: 1 running, 6 sleeping CPU: 0.4% user, 0.0% nice, 0.8% system, 2.3% interrupt, 96.5% idle Mem: 59M Active, 7312K Inact, 8136K Wired, 10M Buf, 47M Free Swap: PID USERNAME THR PRI NICE SIZERES STATETIME WCPU COMMAND 488 root1 160 10808K 10280K pause0:00 5.27% csh 490 root1 410 10940K 10080K RUN 0:00 4.84% top 487 root1 80 10980K 10164K wait 0:00 4.31% login 484 root1 -80 9848K 9596K piperd 0:00 0.67% logger 452 root1 420 14724K 2208K select 0:00 0.00% sshd 486 root1 80 1656K 1396K nanslp 0:00 0.00% sleep 483 root1 80 10416K 1128K wait 0:00 0.00% sh ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD on RaspberryPi
On Nov 7, 2012, at 8:42 AM, Lars Engels wrote: > On Wed, Nov 07, 2012 at 08:01:08AM -0800, Tim Kientzle wrote: >> WARNING: This is still highly experimental and by no >> means ready for "production use", but some folks might >> find it interesting. >> >> To boot FreeBSD on your RaspberryPi, you'll need: >> 1) A RaspberryPi. >> 2) A serial cable similar to this one: www.adafruit.com/products/954 >> 3) An SD card of 2GB or larger >> >> Download this 111MB file (~1.6G uncompressed): >> http://people.freebsd.org/~kientzle/FreeBSD-RPI-B-r242362-2012-10-30.img.xz >> > ... > > Are you aware of this? > > http://kernelnomicon.org/?p=178 Yes, of course. The image I posted wouldn't have been possible without their excellent work. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD on RaspberryPi
On Nov 7, 2012, at 8:09 AM, Alexander Yerenkow wrote: > Such experiments was tried by me and others in August; I got framebuffer > worked in rca/hdmi …. How? I haven't seen the drivers for that yet. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD on RaspberryPi
> On Wed, Nov 7, 2012 at 6:01 PM, Tim Kientzle wrote: > WARNING: This is still highly experimental and by no > means ready for "production use", ... > > To boot FreeBSD on your RaspberryPi, you'll need: > 1) A RaspberryPi. > 2) A serial cable similar to this one: www.adafruit.com/products/954 On Nov 8, 2012, at 9:13 AM, Sami Halabi wrote: > > why the console cable is needed ? > As far as I can tell, the code in FreeBSD-CURRENT does not yet support the video out. So you need a serial console cable to interact with it. You might be able to interact via SSH but that requires a little bit more setup (a root password needs to be set and you need to edit /etc/ssh/sshd_config to allow root logins). If someone knows how to get the video out to work, I'm very interested! Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD on RaspberryPi
On Nov 7, 2012, at 8:09 AM, Alexander Yerenkow wrote: > > Such experiments was tried by me and others in August; I got framebuffer > > worked in rca/hdmi; … On Nov 8, 2012, at 11:12 PM, Alexander Yerenkow wrote: > It was plain current with plain RPIB kernel config, and for graphic you > should uncomment there partition about sysconsole; serial then disabled; > Also, if you want ethernet - it's ue device, which also worked, but produced > hangs for me in past (Hans IIRC already fixed.this). > I'll have some time this weekend, feel free to contact me by gtalk or else, I > will play around with my rpi with both serials and vide modes. Alexander, I tried uncommenting the 'sc' entries in the RPI-B kernel config that's in -CURRENT right now. When I do this, I see the rainbow video on HDMI and nothing on the serial console. Can you try this and tell me what I'm missing? Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD on RaspberryPi
On Nov 24, 2012, at 11:13 AM, Alexander Yerenkow wrote: > > > 2012/11/24 Tim Kientzle > On Nov 7, 2012, at 8:09 AM, Alexander Yerenkow wrote: > > > > Such experiments was tried by me and others in August; I got framebuffer > > > worked in rca/hdmi; … > > On Nov 8, 2012, at 11:12 PM, Alexander Yerenkow wrote: > > > It was plain current with plain RPIB kernel config, and for graphic you > > should uncomment there partition about sysconsole; serial then disabled; > > Also, if you want ethernet - it's ue device, which also worked, but > > produced hangs for me in past (Hans IIRC already fixed.this). > > I'll have some time this weekend, feel free to contact me by gtalk or else, > > I will play around with my rpi with both serials and vide modes. > > Alexander, > > I tried uncommenting the 'sc' entries in the RPI-B kernel > config that's in -CURRENT right now. > > All entries, right? Yes, I uncommented all entries. > And how do you booting, via usb-flash, or via SD card? Via SD-card. I'm using the image from here: http://people.freebsd.org/~kientzle/FreeBSD-RPI-B-r242362-2012-10-30.img.xz and swapping out the kernel. > I'll build now CURRENT, will look if I got same. Thanks. I'd appreciate knowing whether it works for you. If you can get it to work, I'd appreciate any details you can give. Cheers, Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD on RaspberryPi
On Nov 24, 2012, at 8:01 PM, Oleksandr Tymoshenko wrote: > > On 2012-11-24, at 4:47 PM, Tim Kientzle wrote: > >> >> On Nov 24, 2012, at 11:13 AM, Alexander Yerenkow wrote: >> >>> >>> >>> 2012/11/24 Tim Kientzle >>> On Nov 7, 2012, at 8:09 AM, Alexander Yerenkow wrote: >>> >>>>> Such experiments was tried by me and others in August; I got framebuffer >>>>> worked in rca/hdmi; … >>> >>> On Nov 8, 2012, at 11:12 PM, Alexander Yerenkow wrote: >>> >>>> It was plain current with plain RPIB kernel config, and for graphic you >>>> should uncomment there partition about sysconsole; serial then disabled; >>>> Also, if you want ethernet - it's ue device, which also worked, but >>>> produced hangs for me in past (Hans IIRC already fixed.this). >>>> I'll have some time this weekend, feel free to contact me by gtalk or >>>> else, I will play around with my rpi with both serials and vide modes. >>> >>> Alexander, >>> >>> I tried uncommenting the 'sc' entries in the RPI-B kernel >>> config that's in -CURRENT right now. >>> >>> All entries, right? >> >> Yes, I uncommented all entries. >> >>> And how do you booting, via usb-flash, or via SD card? >> >> Via SD-card. I'm using the image from here: >> http://people.freebsd.org/~kientzle/FreeBSD-RPI-B-r242362-2012-10-30.img.xz >> and swapping out the kernel. >> >>> I'll build now CURRENT, will look if I got same. >> >> Thanks. I'd appreciate knowing whether it works for you. >> >> If you can get it to work, I'd appreciate any details you can give. > > Tim, > > I'm almost done with getting kernel working with latest raspberry Pi > firmware. Just need > to figure out how to make ubldr pass FDT pointer from u-boot to kernel and > handle > /reserve/ information in ARM machdep code. Let me know if you need help with this. I've worked with the ubldr FDT code recently. > Meanwhile I suggest editing .dts file manually. Fill out "display" node > properties with proper > display resolution and depth. Also add ukbd driver. That should get you > working console. I'll try that. I'm curious: why is this information coming from the DTS? That seems pretty complex; I thought that the console code would query this information via the mailbox interface. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD on RaspberryPi
On Nov 11, 2012, at 12:46 PM, Adrian Chadd wrote: > On 11 November 2012 12:39, Oleksandr Tymoshenko wrote: > >> At the moment HDMI output works only in a sense of video output for simple >> frame buffer. I'm trying to get GPU support ported but not sure how much time >> it will take. Eventually we'd like to get audio support too. > > How's the general, non-video support working out? Are there any random > crashes or panics that people are seeing on R-PI right now? So far, I haven't done very much, but it is stable enough to natively build a bootable kernel; note the build machine on my RPi boot message: FreeBSD 10.0-CURRENT #2: Sun Nov 25 14:23:57 UTC 2012 root@raspberry-pi:/usr/src/sys/arm/compile/RPI-B arm The two biggest obstacles right now are: * Memory. There's a missing piece in the current boot sequence that causes it to always default to 128MB memory. That makes it hard to do much on a generic -CURRENT build. (The native kernel above took a long time; swapping to SDHC is a little slow. ;-) Related: Shipping RPis now have 512MB RAM; I'm pretty excited about that. * Video console. You've probably noticed the exchanges on the list. Apart from that, it's pretty promising. I'm optimistic that we're no more than a couple of weeks from having a FreeBSD image that a lot of folks can just download and use. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD on RaspberryPi
On Nov 25, 2012, at 11:46 PM, Oleksandr Tymoshenko wrote: > > On 2012-11-25, at 9:32 AM, Tim Kientzle wrote: > >> >> On Nov 24, 2012, at 8:01 PM, Oleksandr Tymoshenko wrote: >> >>> >>> On 2012-11-24, at 4:47 PM, Tim Kientzle wrote: >>> >>> > > .. skipped .. > >>> Tim, >>> >>> I'm almost done with getting kernel working with latest raspberry Pi >>> firmware. Just need >>> to figure out how to make ubldr pass FDT pointer from u-boot to kernel and >>> handle >>> /reserve/ information in ARM machdep code. >> >> Let me know if you need help with this. I've worked with >> the ubldr FDT code recently. >> >>> Meanwhile I suggest editing .dts file manually. Fill out "display" node >>> properties with proper >>> display resolution and depth. Also add ukbd driver. That should get you >>> working console. >> >> I'll try that. >> >> I'm curious: why is this information coming from the DTS? >> That seems pretty complex; I thought that the >> console code would query this information via the mailbox >> interface. > > > It's either FDT blob or message box interface. Implementation complexity is > about the same. My thinking: * Display resolution used by kernel has to match what the firmware uses. So the kernel should either get the information from the firmware or from the same place the firmware gets it from. * We want ubldr to remain generic, so I'm reluctant to put things into it that are RaspberryPi-specific. If the firmware is putting the values into the FDT, then having the kernel get it from the FDT is another way for the kernel to get it from the firmware, so that sounds okay. > But since we're getting other variables (like MAC address, memory size) from > FDT I decided > to be consistent and get all of them from there. I don't know about MAC address. Memory size is handled generically by ubldr using a standard interface to U-Boot, so it's not special to RaspberryPi. The FDT editing is just a standard way for ubldr to pass this to the kernel. > The issue I'm facing is that ubldr gets FDT blob > either from file directly or from ELF kernel itself. While on Raspberry Pi to > works as follows: > > - Firmware loads .dtb file from SD card to specified address Does RaspberryPi firmware now load an FDT? Does the firmware now read the FDT to get its values for display resolution, etc? (I don't really like this because a lot of people need to tweak the display settings and it's hard to tell a 6-year-old how to edit and recompile an FDT.) > - Fixes up values like amount of memory, reserved regions, UART and clock > frequencies, > MAC address, display resolution. > - Passes control to next link in boot chain (e.g. U-Boot) To be clear: You say the RPi firmware is already doing this editing? So the ubldr just has to pass the RPi FDT to the kernel? If so, that's a lot simpler. > I'm thinking about adding compile-time constant FDT_BLOB_ADDRESS and arrange > possible > FDT sources in following priority: > > - Check FDT_BLOB_ADDRESS (if defined) > - Check dtb file > - Check ELF kernel > > Does it sound sane enough? If the RPi firmware always loads the FDT at a fixed address and the RPi firmware is using the FDT to configure itself, then it makes a lot of sense. It would be nice to do this without adding RPi-specific code to ubldr. > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" > ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD on RaspberryPi
On Nov 26, 2012, at 10:43 AM, Oleksandr Tymoshenko wrote: > > On 2012-11-26, at 7:05 AM, Tim Kientzle wrote: > >> >> On Nov 25, 2012, at 11:46 PM, Oleksandr Tymoshenko wrote: >> >>> >>> On 2012-11-25, at 9:32 AM, Tim Kientzle wrote: >>> >>>> >>>> On Nov 24, 2012, at 8:01 PM, Oleksandr Tymoshenko wrote: >>>> >>>>> >>>>> On 2012-11-24, at 4:47 PM, Tim Kientzle wrote: >>>>> >>>>> >>> >>> .. skipped .. >>> >>>>> Tim, >>>>> >>>>> I'm almost done with getting kernel working with latest raspberry Pi >>>>> firmware. Just need >>>>> to figure out how to make ubldr pass FDT pointer from u-boot to kernel >>>>> and handle >>>>> /reserve/ information in ARM machdep code. >>>> >>>> Let me know if you need help with this. I've worked with >>>> the ubldr FDT code recently. >>>> >>>>> Meanwhile I suggest editing .dts file manually. Fill out "display" node >>>>> properties with proper >>>>> display resolution and depth. Also add ukbd driver. That should get you >>>>> working console. >>>> >>>> I'll try that. >>>> >>>> I'm curious: why is this information coming from the DTS? >>>> That seems pretty complex; I thought that the >>>> console code would query this information via the mailbox >>>> interface. >>> >>> >>> It's either FDT blob or message box interface. Implementation complexity is >>> about the same. >> >> My thinking: >> * Display resolution used by kernel has to match what the firmware uses. >> So the kernel should either get the information from the firmware or from >> the same place the firmware gets it from. >> * We want ubldr to remain generic, so I'm reluctant to put things into it >> that are RaspberryPi-specific. >> >> If the firmware is putting the values into the FDT, then having the kernel >> get it from the FDT is another way for the kernel to get it from the >> firmware, so that sounds okay. >> >> >>> But since we're getting other variables (like MAC address, memory size) >>> from FDT I decided >>> to be consistent and get all of them from there. >> >> I don't know about MAC address. Memory size is handled generically by ubldr >> using a standard interface to U-Boot, so it's not special to RaspberryPi. >> The FDT editing is just a standard way for ubldr to pass this to the kernel. >> >>> The issue I'm facing is that ubldr gets FDT blob >>> either from file directly or from ELF kernel itself. While on Raspberry Pi >>> to works as follows: >>> >>> - Firmware loads .dtb file from SD card to specified address >> >> Does RaspberryPi firmware now load an FDT? > Yes, when requested by device_tree_address and device_tree parameters in > config.txt > >> >> Does the firmware now read the FDT to get its values for display resolution, >> etc? >> (I don't really like this because a lot of people need to tweak the display >> settings and it's hard to tell a 6-year-old how to edit and recompile an >> FDT.) > No. firmware writes to .dtb display's native resolution (or the one > requested in config.txt) to it. Not vice versa. > That's the way to pass this information to kernel. > >> >>> - Fixes up values like amount of memory, reserved regions, UART and clock >>> frequencies, >>> MAC address, display resolution. >>> - Passes control to next link in boot chain (e.g. U-Boot) >> >> To be clear: You say the RPi firmware is already doing this editing? >> >> So the ubldr just has to pass the RPi FDT to the kernel? If so, that's a >> lot simpler. > Yes > >> >>> I'm thinking about adding compile-time constant FDT_BLOB_ADDRESS and >>> arrange possible >>> FDT sources in following priority: >>> >>> - Check FDT_BLOB_ADDRESS (if defined) >>> - Check dtb file >>> - Check ELF kernel >>> >>> Does it sound sane enough? >> >> If the RPi firmware always loads the FDT at a fixed address >> and the RPi firmware is using the FDT to configure itself, >> then it makes a lot of sense. >> >> It would be nice to do this without adding RPi-specific >> code to ubldr. > > There will be no RPi-specific code. Just one more way to specify location of > generic FDT blob. This all sounds good then. So we can put the FreeBSD .dts file on the MSDOS Boot partition, then: * Firmware will load it into memory and add display information. * ? ubldr will access the FDT and add memory information and MAC address info ? * Kernel will then load it and use it to initialize. Maybe another possibility would be to script ubldr and have it load the FDT from the correct location in memory. ubldr already knows how to load an FDT and how to pass that FDT to the kernel. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Every 20 minutes (less 4 seconds)?
I'm trying to understand why the transmitter for the CPSW ethernet driver just stops sometimes. I just found a very perplexing clue: It happens every 1196 seconds. I added a debug message with an HH:MM:SS timestamp whenever the watchdog sees the transmitter stop and this is what it printed out: 20:30:20 cpsw_tx_watchdog Watchdog resetting driver 20:50:16 cpsw_tx_watchdog Watchdog resetting driver 21:10:12 cpsw_tx_watchdog Watchdog resetting driver 21:30:09 cpsw_tx_watchdog Watchdog resetting driver 21:50:04 cpsw_tx_watchdog Watchdog resetting driver 22:10:01 cpsw_tx_watchdog Watchdog resetting driver 22:29:56 cpsw_tx_watchdog Watchdog resetting driver 22:49:52 cpsw_tx_watchdog Watchdog resetting driver 23:09:49 cpsw_tx_watchdog Watchdog resetting driver 23:29:45 cpsw_tx_watchdog Watchdog resetting driver 23:49:41 cpsw_tx_watchdog Watchdog resetting driver 00:29:33 cpsw_tx_watchdog Watchdog resetting driver 00:49:29 cpsw_tx_watchdog Watchdog resetting driver 01:29:21 cpsw_tx_watchdog Watchdog resetting driver 02:29:09 cpsw_tx_watchdog Watchdog resetting driver 03:28:57 cpsw_tx_watchdog Watchdog resetting driver 04:28:45 cpsw_tx_watchdog Watchdog resetting driver 05:28:31 cpsw_tx_watchdog Watchdog resetting driver (Note: Some 20-minute intervals did seem to get skipped. There was no hang at 01:09, 01:49, 02:09, 02:49, etc. Maybe because the system was less busy then? The network and processor load varied quite a bit over this 9 hour period.) Based on this, it would appear that "something" happens every 1196 seconds and if "something else" happens at that time the transmitter will hang. Does anyone recognize that particular period? Tim P.S. Cron isn't running, nor is there any other particularly periodic process. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: IBM blade server abysmal disk write performances
On Jan 15, 2013, at 6:12 AM, Karim Fodil-Lemelin wrote: > Hi, > > I'm struggling getting FreeBSD 9.1 properly work on an IBM blade server > (HS22). Here is a dd output from Linux CentOS vs FreeBSD 9.1. > > CentOS: > > 10+0 records in > 10+0 records out > 5120 bytes (51 MB) copied, 1.97883 s, 25.9 MB/s > > > FreeBSD 9.1: > > 1+0 records in > 1+0 records out > 512 bytes transferred in 60.024997 secs (85298 bytes/sec) What exactly was the 'dd' command you used? In particular, what block size did you specify? Can you strace the 'dd' command on CentOS to verify that it's using the actual block size you specified? Some programs (I've written at least one) "cheat" by actually doing larger I/O operations than you request. This makes a big difference in performance. So this could reflect optimizations in GNU dd more than any difference in the actual disk I/O. If you want to do a more robust comparison, look for one of the disk benchmarking programs in ports and see if it's available (in the same version) for CentOS. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Testing SIOCADDMULTI?
My next TODO items for this network driver is to implement the SIOCADDMULTI and SIOCDELMULTI ioctls. I'm not quite sure what they do, though, and have no idea how to test them to see if they are working correctly. Any suggestions? Cheers, Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Testing SIOCADDMULTI?
On Jan 26, 2013, at 3:56 PM, Tim Kientzle wrote: > My next TODO items for this network driver is to implement > the SIOCADDMULTI and SIOCDELMULTI ioctls. Looking through other drivers (and net/if.c), I've managed to implement ADDMULTI by adding the multicast ethernet address to the list maintained by the controller. DELMULTI seems trickier. Since if.c does not pass the specific address being removed down to the driver, it looks like I have no choice but to remove every multicast address from the controller and then re-insert all of the ones that are still valid. (This controller doesn't use a hash filter; it uses a list of valid multicast addresses.) Is there a better approach? > I'm not quite sure what they do, though, and have > no idea how to test them to see if they are working > correctly. Would still appreciate any suggestions for how to test these. Cheers, Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Testing SIOCADDMULTI?
On Jan 28, 2013, at 8:09 AM, John Baldwin wrote: > On Sunday, January 27, 2013 1:51:12 am Tim Kientzle wrote: >> >> On Jan 26, 2013, at 3:56 PM, Tim Kientzle wrote: >> >>> My next TODO items for this network driver is to implement >>> the SIOCADDMULTI and SIOCDELMULTI ioctls. >> >> DELMULTI seems trickier. ... >> ... it looks like I have no choice but to remove >> every multicast address from the controller and then >> re-insert all of the ones that are still valid. >> ... >> Is there a better approach? > > You should always reprogram the full table while holding if_maddr_rlock(). Thanks. That's ultimately what I did. I was able to dump the table from the controller to eyeball that the entries looked right but haven't yet done any testing beyond that. >> Would still appreciate any suggestions for how to test these. > > You can write a simple app to listen for UDP packets and have it join a > multicast group and have another machine on the same network write a packet > to > the multicast group. I tried this first, but the test program worked fine even without ADDMULTI/DELMULTI support. Watching tcpdump -e, it appears that IP4 multicast UDP uses broadcast at the Ethernet layer. > However, a simpler test is to toggle the sysctl to enable multicast ping > replies and to ping a multicast address from another machine after joining it > on the test machine using mutest. Ahhh…. I wasn't aware of these tools; I'll take a look. Thanks for the pointer! Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Seeking an extended-support O/S similar to FreeBSD
> > That's no help at all to a bunch of machines that started life on > 4.1 back in 2000 and will continue to run another 10-15 years… So you basically want a group of people to help you maintain FreeBSD 4-STABLE for an indefinite period of time? There seem to be quite a few people still running FreeBSD 4.x-based systems; I'm sure if you ask around you could find a group of people to help you with this. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Performance improvement to strnlen().
>
> Index: strnlen.c
> ===
> diff --git a/head/lib/libc/string/strnlen.c b/head/lib/libc/string/strnlen.c
> --- a/head/lib/libc/string/strnlen.c (revision 250951)
> +++ b/head/lib/libc/string/strnlen.c (working copy)
> @@ -1,5 +1,6 @@
> /*-
> - * Copyright (c) 2009 David Schultz
> + * Copyright (c) 2009, 2010 Xin LI
> + * Copyright (c) 2013 Lee Thomas
> * All rights reserved.
> *
> * Redistribution and use in source and binary forms, with or without
> @@ -27,16 +28,91 @@
> #include
> __FBSDID("$FreeBSD$");
>
> +#include
> +#include
> #include
>
> +/*
> + * Portable strnlen() for 32-bit and 64-bit systems.
> + *
> + * Rationale: it is generally much more efficient to do word length
> + * operations and avoid branches on modern computer systems, as
> + * compared to byte-length operations with a lot of branches.
> + *
> + * The expression:
> + *
> + * ((x - 0x0101) & ~x & 0x8080)
> + *
> + * would evaluate to a non-zero value iff any of the bytes in the
> + * original word is zero.
> + *
> + * On multi-issue processors, we can divide the above expression into:
> + * a) (x - 0x0101)
> + * b) (~x & 0x8080)
> + * c) a & b
> + *
> + * Where, a) and b) can be partially computed in parallel.
> + *
> + * The algorithm above is found on "Hacker's Delight" by
> + * Henry S. Warren, Jr.
> + */
> +
> +/* Magic numbers for the algorithm */
> +#if LONG_BIT == 32
> +static const unsigned long mask01 = 0x01010101;
> +static const unsigned long mask80 = 0x80808080;
> +#elif LONG_BIT == 64
> +static const unsigned long mask01 = 0x0101010101010101;
> +static const unsigned long mask80 = 0x8080808080808080;
> +#else
> +#error Unsupported word size
> +#endif
> +
> +#define LONGPTR_MASK (sizeof(long) - 1)
I would not use this at all; (sizeof(long) - 1) is a common
expression that your audience probably understands
more quickly than "LONGPTR_MASK".
> +
> size_t
> -strnlen(const char *s, size_t maxlen)
> +strnlen(const char *str, size_t maxlen)
> {
> - size_t len;
> + const char *stop, *short_stop;
> + const char *p;
> + const unsigned long *lp;
> + long va, vb;
>
> - for (len = 0; len < maxlen; len++, s++) {
> - if (!*s)
> - break;
> + if (maxlen==0) return 0;
> +
> + stop=str+maxlen;
> + /*
> + * Before trying the hard (unaligned byte-by-byte access) way
> + * to figure out whether there is a nul character, try to see
> + * if there is a nul character is within this accessible word
> + * first.
> + *
> + * p and (p & ~LONGPTR_MASK) must be equally accessible since
> + * they always fall in the same memory page, as long as page
> + * boundaries is integral multiple of word size.
> + */
> + lp = (const unsigned long *)((uintptr_t)str & ~LONGPTR_MASK);
> + lp++;
Have you tested to see whether the extra work you're
doing for the initial segment really gains you much? I
would have used a simpler byte-by-byte check as
follows:
p = s;
while (maxlen-- > 0 && p < (const char *)lp) {
if (*p == '\0')
return (p - s);
}
while (maxlen > 0) {
… complicated test of *lp …
maxlen -= sizeof(*lp);
}
Few points here:
* This form of the initial segment test doesn't get surprised
by a NUL byte just before the string. Yours does a bunch of
extra work in that case.
* Duff's device might help unroll the first loop; would require
testing to see if it was faster. For something this simple, it
might not be.
* Your version tests the first word twice (once in the
initial check and then again at the start of the word-sized
pass). This version doesn't.
* Comparison with zero is often free, so a countdown to zero
is often faster than a count up to a computed limit.
This assumes that 'lp' is pointing to the first aligned word
at or after the beginning of the string.
Your version does not leave lp pointing to
the beginning of the string when the string is initially
already aligned. If the string is already fully aligned, my
version avoids the initial check entirely.
To compute 'lp' as a pointer to the first full word
at or after the beginning of the string:
lp = (const unsigned long *)
(
(
(uintptr_t)str + sizeof(*lp) - 1
)
&
(
sizeof(*lp) - 1
)
);
I've broken this onto multiple lines to illustrate the structure;
the final code would of course be a little more compact.
Also note the use of sizeof(*lp) rather than
sizeof(long) or sizeof(unsigned long); this way,
you guarantee that the sizeof() matches lp even if someone
comes along later and changes the declaration of lp.
> + va = (*lp - mask01);
> + vb = ((~*lp) & mask80);
> + if (va & vb) {
> + /* Check if we have \0 in the first part */
> + short_stop=(con
Re: Beyond Buildworld Dev Summit Working Group Report
On Jun 9, 2013, at 2:02 PM, George Neville-Neil wrote: > Howdy, > > The Beyond Buildworld working group discussed many subjects around our build > system, including > upcoming changes to do a better job of addressing embedded systems, the > integration of bmake, > and the need for better incremental build support. The full notes are on the > wiki and > also pasted at the bottom of this email. > > Best, > George > > https://wiki.freebsd.org/201305DevSummit/Buildworld > > This section includes volunteers or contact points as links. > > • uboot ports [DianeBruce] > • compiler patches vs. gccc on Linux [TimKeintzle] Diane helped me put together a first port for U-Boot (BeagleBone with EABI). Still need to get it committed, then use it as a template for other U-Boot ports. (E.g., RPi, ZedBoard, etc.) If other folks are working on U-Boot patches to boot FreeBSD on other hardware, it would be nice to get those into ports as well. > • ubloader not on ELF [Ian] I looked into making ubldr self-relocating (so that it could just be loaded at an arbitrary address and run without modification). Enabling PIC in the compiler goes a long ways, but the loader libraries rely heavily on static data references to code. These would have to either be rewritten or generated at run-time to make the whole loader able to run at an arbitrary address. > • Crochet with VM images [TimKeintzle], [GlenBarber], [ColinPercival] Per Colin, there's no way for "mere mortals" to upload machine images to Amazon, so there's little point in pursuing that. I did recently add support to Crochet for building VMWare VM images directly. Works pretty well; I've been using it to build throw-away images for testing. This goes a step further than what Glen has recently announced; so far, he's just building the VMDK disk image, Crochet fills in the rest of the VM configuration files. It should be routine to duplicate the approach to support other VM environments (e.g., Parallels, VirtualBox, OFA). My time is limited but I'm happy to assist if someone else wants to work on it. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Beyond Buildworld Dev Summit Working Group Report
On Jun 10, 2013, at 10:02 PM, Glen Barber wrote: > On Mon, Jun 10, 2013 at 09:25:39PM -0700, Tim Kientzle wrote: >>> • Crochet with VM images [TimKeintzle], [GlenBarber], [ColinPercival] >> >> Per Colin, there's no way for "mere mortals" to upload machine >> images to Amazon, so there's little point in pursuing that. >> >> I did recently add support to Crochet for building VMWare VM images >> directly. Works pretty well; I've been using it to build throw-away >> images for testing. This goes a step further than what Glen >> has recently announced; so far, he's just building the VMDK disk >> image, Crochet fills in the rest of the VM configuration files. >> >> It should be routine to duplicate the approach to support other >> VM environments (e.g., Parallels, VirtualBox, OFA). My time is >> limited but I'm happy to assist if someone else wants to work on it. >> > > I do intend to look at how Crochet does this, because I think your > implementation is far more user friendly and cross-application > compatible. > > I'll certainly pick your brain on it, if you don't mind. Key point: VM images are directories which contain a number of different files. The disk image is just one of those. For VMWare (and others I've looked at), the VM directory contains: * Disk image - this can be a straight disk image (which is what Crochet currently uses) or a structured/sparse/compressed "virtual disk" * Disk descriptor -- this specifies the format of the disk image and the parameters of the virtual disk drive. * VM descriptor -- this describes the rest of the virtual hardware. VMWare seems to be the simplest of the ones I've looked at: the descriptor files are lists of key/value pairs and the VMWare software can boot a VM even from a pretty skeletal definition. It didn't take me very many tries to get Crochet to build a VMWare image that VMWare Fusion was comfortable booting. It helps that the VMWare file formats are pretty well documented: In particular: http://sanbarrow.com Parallels and OFA both use more complex XML configuration files -- everything has an ID and there are lots of cross-references. I've toyed with building Parallels VMs but haven't gotten very far yet. qemu-img seems like a nice tool for generating a virtual disk image, but it doesn't currently know how to create compressed VMWare disks. I've skimmed VMWare's documentation and it doesn't look too hard to build a compressed VMWare disk if someone would like to tackle that. ;-) Tim signature.asc Description: Message signed with OpenPGP using GPGMail
'opaque' flag?
Does anyone understand the semantics of the 'opaque' flag? I'm trying to understand an issue with packages built on union file systems. It appears the 'opaque' flag is set on some symlinks, which the package tools then archive. The archived flag is somehow interfering with later installation of the package. I've read McKusick's paper on unionfs which explains the opaque flag for directories created on a unionfs, but I have a bunch of questions: * Why is it appearing on symlinks? * How could the opaque flag interfere with package installs? * Should bsdtar ignore opaque flags? Any input appreciated, Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 'opaque' flag?
Tim Kientzle wrote: Does anyone understand the semantics of the 'opaque' flag? I'm trying to understand an issue with packages built on union file systems. It appears the 'opaque' flag is set on some symlinks, which the package tools then archive. The archived flag is somehow interfering with later installation of the package. I've read McKusick's paper on unionfs which explains the opaque flag for directories created on a unionfs, but I have a bunch of questions: * Why is it appearing on symlinks? * How could the opaque flag interfere with package installs? * Should bsdtar ignore opaque flags? Okay, I now understand the answer to the second question: there's a bug in libarchive restoring symlinks with flags. I'll fix that shortly. The other two questions still puzzle me. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: New FreeBSD package system (a.k.a. Daemon Package System (dps))
Ivan Voras wrote: - A quick test confirms that the current bsdtar will happily ignore any extra data at the end of a tgz/tbz archive, so package metadata can be embedded there, thus conserving existing infrastructure... Not a good idea at all. 1) Keeping everything within the archive makes it possible for people to do surgery on the packages. Being able to extract/modify/rebuild packages using just tar makes it much easier to debug and experiment with new techniques. 2) The existing metadata format is not very pretty, but it can be reasonably extended. (Personally, I don't find XML very pretty, either. ;-) 3) As DES pointed out, the package tools must be able to read the metadata before the files. If you really need a completely separate metadata file, make it the second file in the archive. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: New FreeBSD package system (a.k.a. Daemon Package System (dps))
3) As DES pointed out, the package tools must be able to read the metadata before the files. Actually, the argument is pretty weak. Being able to extract them streamable and access the meta-data easily is fine. The remote access argument is very weak as it doesn't allow e.g. signature checks. I presume you mean that you have to scan the entire package to verify the signature before doing installation? I don't think you do, really. If you can roll back an installation, then you can verify the signature during a streaming install; if the signature fails, you roll back. A good package installer needs to support rollback anyway to do robust dependency handling. I know two relatively straightforward ways to structure the installation process to support rollback. So many ideas, so little time... ;-) Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: New FreeBSD package system (a.k.a. Daemon Package System (dps))
Duane Whitty suggested: > I'm a little out of practice, however, perhaps the routines > that manipulate the ports meta-data could be sufficiently > agnostic about how the data is being manipulated that it > would facilitate experimentation with different > back-ends at a later time Yes. This is an excellent idea. I wrote up some of my own ideas in this direction a few years ago: http://people.freebsd.org/~kientzle/libarchive/libpkg.3.txt The basic idea was, as you say, to provide an abstract interface that separates the data storage from what the tools require. Unfortunately, libarchive (which started as part of a package tools overhaul) has absorbed more time than I expected, so I've not had a chance to get back to these ideas. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: New FreeBSD package system (a.k.a. Daemon Package System (dps))
David Naylor volunteered: Since the installation system is being tackled under a SoC project I am hoping to give the packaging system a go. Wonderful! Be careful about one point: The packaging system as a whole is a big system; much bigger than many people believe. A lot of people (including myself) have set out to rebuild the package system. Few of us have gotten very far. (For example, I built libarchive in order to rework pkg_add. But changing pkg_add's install logic required rethinking dependency handling, which turned out to be a lot more complex than I thought.) A good place to start is with the existing tools. Search the bug reports, work up fixes to some of them and submit those fixes as follow-ups. Start conservatively; don't break compatibility with existing tools until you understand the system more completely. In particular, expect a lot of skepticism about major format changes until you have some actual numbers to back you up. A lot of people on this mailing list have looked into performance issues with the package system; the real problems may not be where you think they are. I have searched the FreeBSD mailing lists and have found little information on the package system. The people who maintain the current codebase are right on this mailing list. Ask away! What do you want to know? Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Using shell commands versus C equivalents
Next step, eliminating the linked list structure in favor or red-black trees, maybe. Due to the way pkg_install works, this most likely is just adding complexity for no gain, it might actually slow it down. Hmmm... the only thing is that it does the linked list traversal a number of times per dependency. I'll take a look in gdb at the size of each dependency and then confer with Kirill (my mentor) about that a bit more. If you tend to look for the same few items over and over, just move items to the front of the list as soon as you find them. Alternatively, if you know you won't look for this item ever again, move it to the end of the list or remove it from the list entirely. Simple tricks like this can greatly speed things up with almost no effort. Better yet, do easy things like this first; if they help, then look at more complex approaches. As Joerg said, though, you're not likely to gain much from this. pkg_install is almost entirely disk bound. I spent a lot of time recently in libarchive/bsdtar optimizing the disk handling; I can share lots of ideas for improving performance of disk-bound operations like this. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Using shell commands versus C equivalents
I spent a lot of time recently in libarchive/bsdtar optimizing the disk handling; I can share lots of ideas for improving performance of disk-bound operations like this. One thing you might find useful: libarchive has an API for "creating things on disk," which handles a lot of trivia (creating parent directories transparently, restoring permissions, users, file flags, etc.). It is, of course, especially useful if you're pulling things out of tar archives, but can be used separately from that. Since libarchive is a standard part of FreeBSD, you should feel free to use it. Tim K ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Using shell commands versus C equivalents
Also, were the bottlenecks seen in pkg_delete and pkg_add, or does it appear to be distributed across the board? The biggest time sink in pkg_add is writing each file to a temp dir then copying it to its final location. There are a couple of strategies for avoiding this (by writing the files directly to their final location), but it basically requires rewriting pkg_add from scratch. I prototyped this a long time ago and found about a 3x speedup. (Parts of that prototype eventually became libarchive.) I haven't looked closely at pkg_delete, but I doubt there's much that can be done to speed it up; once you've examined the dependency information to determine what can be deleted, actually removing the files is a pretty straightforward operation. The two operations that people focus on performance issues have been index rebuilding (which requires inspecting every port in /usr/ports) and update (which requires inspecting every installed port). The modular Xorg is especially going to stress updates, since it greatly increases the number of ports on the average system. One useful tool: "truss" can include timing information that can give a lot of insight into where a program is really spending time. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Rearchitecting pkg_install (was Re: Using shell commands versus C equivalents)
I've also seen a 3x speedup by using my reimplementation of pkg_add using my package management library, libpkg (http://libpkg.berlios.de). It is not production ready yet as if it fails partway through an installation it won't clean up and installed files. I *think* a good way to do this is to: 1) Write each file to a temporary name (as Peter Jeremy suggested earlier in this thread) 2) Write a +INPROGRESS file to the package dir with the temporary name of each file as it is written. 3) Recursively install dependencies. 4) Rename the files to their final names. This sequence allows you to: * Do conflict checking on the fly. If a single file conflicts at any point, you can detect that, ask the user for a resolution at that point and drop the single file or back out the entire package. * Recover crashed installations. Just scan for all +INPROGRESS files and remove every file mentioned. There are a few additional steps you can take to provide even better failure recovery, but this covers the big concerns. I haven't looked closely at pkg_delete, but I doubt there's much that can be done to speed it up... I ran a quick test ... difference was too small to be noticeable ... Yeah, that's not surprising. 2. I can't go and graft your libs, or do something similar with the current source because I don't want to break production code (pkg_install) of this importance and muck up a lot of user's systems irrevocably. If you can get enough people involved in testing, you may be able to pull off a full rearchitecture. Requires some careful political groundwork, though. ;-) Be friendly, talk about your work in different mailing lists, make snapshot versions of your code available, try to build a reputation as someone who responds quickly to reported problems. I managed to get support for replacing 'tar' this way, which is arguably just as critical as pkg_install. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libelf question
Angus Barrow wrote: I'm trying to develop a BSD licensed version of the ar utility using the new libelf library ... The time shown in arh->ar_date (this is the struct that the libelf library provides for each entry in the ar archive) seems to have a negative value (using the GNU ar the date is Jun 3 13:09 2007 Kai Wang recently added 'ar' support to libarchive/bsdtar; what does 'tar -tv' say? Note: You'll need a recent 7-CURRENT system for this or you can install libarchive/bsdtar from the most recent development snapshot on http://people.freebsd.org/~kientzle/ Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Finding slowdowns in pkg_install (continuations of previous threads)
I'm currently running a gamut of tests (500 tests, per package -- 128 total on my server), and outputting all data to CSV files to interpret later, using another Perl script to interpret calculated averages and standard deviations. Excellent! Much-needed work. Using basic printf(2)'s with clock_gettime(2) I have determined that the majority of the issues are disk-bound (as Tom Kientzle put it). Next question: What are those disk operations and are any of them redundant? The scope of my problem is not to analyze tar,... I've spent the last three years+ doing exactly that. Make sure you're using the newest bsdtar/libarchive, which has some very noticable performance improvements. but I've discovered that a lot of time is spent in reading and interpreting the +CONTENTS and related files (most notably in parsing commands to be honest). Oh? That's interesting. Is data being re-parsed (in which case some structural changes to parse it once and store the results may help)? Or is the parser just slow? Will post more conclusive results tomorrow once all of my results are available. I don't follow ports@ so didn't see these "conclusive results" of yours. I'm very interested, though. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Finding slowdowns in pkg_install (continuations of previous threads)
-I tried ... buffering ... the +CONTENTS file parsing function, and the majority of the time it yielded good results One approach I prototyped sometime back was to use libarchive in pkg_add as follows: * Open the archive * Read +CONTENTS directly into memory (it's guaranteed to always be first in the archive) * Parse all of +CONTENTS at once * Continue scanning the archive, disposing of each file as it appears in the archive. Based on my experience with this, I would suggest you just read all of +CONTENTS directly into memory at once and parse the whole thing in a single shot. fopen(), then fstat() to get the size, then allocate a buffer and read the whole thing, then fclose(). You can then parse it all at once. As a bonus, your parser then becomes a nice little bit of reusable code that reads a block of memory and returns a structure describing the package metadata. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Finding slowdowns in pkg_install (continuations of previous threads)
4. CSV files available at: http://students.washington.edu/youshi10/posted/atk-results.tgz. I've posted HTML results of the interpreted spreadsheet on <http://students.washington.edu/posted/atk.htm>. I'll provide commentary tomorrow after I get some sleep. I think the second one should be: <http://students.washington.edu/youshi10/posted/atk.htm> Unfortunately, I get "Permission Denied" here for both of those URLs. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Tar output mode for installworld
I saw this project suggestion on www.freebsd.org:
<http://www.freebsd.org/projects/ideas/#p-taroutmode>
and thought I'd contribute a couple of ideas and notes:
This is easy to implement using a trick that I stumbled
across a few years ago. The idea is to just build
a description of the final archive in a nice verbose
text format such as:
E.g.,
bin/sh file /usr/obj/usr/src/bin/sh
bin/sh uname root
bin/sh gname wheel
rescue/mkdir hardlink rescue/rescue
bin/sh mode 0666
bin/rcp mode 04666
Here a "file" entry indicates where the body of a file
is located, "uname", "gname", "mode", etc, specify
various pieces of metadata.
The key point is that this file doesn't have to get
written in order. You can write single lines of this
information at any point in the install process to a
file:
echo ${TARGETFILE} uname ${OWNER} >>${INSTALLLISTFILE}
echo ${TARGETFILE} file ${SRCFILE} >> ${INSTALLLISTFILE}
This should be easy to splice into the install process,
just add some conditionals on whether ${INSTALLLISTFILE}
is defined or not.
Once you've built this file, just sort it to bring all
the data for each entry together. (As a bonus, sorting
will put bin before bin/sh.)
From there, it's a simple exercise to write a program that
reads this input and spits out a tar file using libarchive.
(Start from the sample code in the archive_write manpage.)
Incorporating mtree input might be a bit trickier; one option
would be to modify mtree itself to spit out a condensed
format like this as output.
Given these details, I'm sure there are lots of people
who could get 90% of this working over a weekend. There
are a bunch of odd corner cases to finish up the last 10%,
but I don't see anything especially tricky.
The end result would be a way to build install CDs without
having to run as root. Building an install tarfile could
even benefit regular "installworld," since the final install
could be done with a minimal toolkit (no need for awk,
sed, or other complex commands, just /bin/sh, make, and tar).
Tim Kientzle
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Finding slowdowns in pkg_install (continuations of previous threads)
The following blog post has all of my commentary on the results I
have:
<http://blogs.freebsdish.org/gcooper/2007/07/14/modifications-to-pkg_install-the-positive-and-negative-implications/>.
I tried to unroll strcmp a bit by checking for the first character of the
> command, then run strcmp ...
There's a somewhat more straightforward optimization that
relies on this same idea:
switch(cmd[0]) {
case 'c':
/* Commands that start with 'c' */
if (strcmp(cmd, 'cwd') == 0)
return (CMD_CWD);
/* FALLTHROUGH */
case 'd':
/* Commands that start with 'd' */
etc
/* FALLTHROUGH */
default:
/* Unrecognized command. */
}
This is a little cleaner and easier to read
and may even be faster than the code you
presented in your blog. Note that the fall through
ensures that all unrecognized commands end up at
the same place. If unrecognized commands are
very rare (they should be), then the fallthrough
is not a performance issue.
/** malloc buffer large enough to hold +CONTENTS **/
while(!feof(file_p)) {
/** add content via fgetc **/
}
Yuck. Try this instead:
struct stat st;
int fd;
char *buff;
fd = open(file);
fstat(fd, &st);
buff = malloc(st.st_size + 1);
read(fd, buff, st.st_size);
buff[st.st_size] = '\0';
close(fd);
Plus some error checking, of course. You can
use stdio if you prefer:
FILE *f;
f = fopen(file, "r");
fstat(fileno(f), &st);
buff = malloc(st.st_size + 1);
fread(buff, 1, st.st_size, f);
buff[st.st_size] = '\0';
fclose(f);
Either way, this is a lot more efficient than
tens of thousands of calls to fgetc().
Cheers,
Tim Kientzle
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Tar output mode for installworld
On 2007-07-14 13:20, Tim Kientzle <[EMAIL PROTECTED]> wrote: <http://www.freebsd.org/projects/ideas/#p-taroutmode> This is easy to implement using a ... text format such as: bin/sh file /usr/obj/usr/src/bin/sh bin/sh uname root rescue/mkdir hardlink rescue/rescue bin/sh mode 0666 bin/rcp mode 04666 Giorgos Keramidas responds: A packaging list for one of the distributions I've built at work includes stuff like: # Misc command-line tools in @prefix@/bin/... d none @prefix@/bin 0755 root bin f none @prefix@/bin/progname-2.0 0755 root bin s none @prefix@/bin/progname=progname-2.0 My concern is that "all of the bits for one file" may not always be available at a single place in the build. Some programs use the "afterinstall" hook to set flags on the installed binary, for example. My version allows different information about a file to be output at different points in the build process. A simple 'sort' then collects information about each file together, making it easy to then build the final archive. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Tar output mode for installworld
Joerg Sonnenberger wrote: On Sat, Jul 14, 2007 at 01:20:15PM -0700, Tim Kientzle wrote: This is easy to implement using a trick that I stumbled across a few years ago. The idea is to just build a description of the final archive in a nice verbose text format such as: ...which is done by NetBSD for the unprivileged release building via build.sh. Anyone interested in working on this should possibly have a look there. Interesting. I hadn't looked at NetBSD's unprivileged release build before. I just skimmed through some of it. Parts of it seems a little roundabout, but I like the use of a modified mtree format as the specification format: usr/bin/su type=file mode=04555 uname=root gname=wheel time=1057493599.102665 I would make two changes: 1) Add a contents= option. With this, DESTDIR is mostly irrelevant, since the path can just refer to the file in the build tree. 2) Allow multiple lines for any file. This complicates the consumer of this file a bit, but simplifies the build process, since you can emit separate information as you find it. For example, the regular install process could install rcp by emitting: bin/rcp type=file mode=04555 contents=/usr/obj/usr/src/bin/rcp and then a later point in the build process could add the flag by adding the line: bin/rcp flags=schg Hmmm... It would be easy for libarchive to parse this format and then bsdtar's existing "archive conversion" feature could trivially generate a tar.gz output from such a specification. Another idea would be to create an install program that accepted this format as the command line: mtree_install usr/bin/su type=file mode=04555 uname=root ... Then the build logic gets scary simple: A regular build sets INSTALL=mtree_install, a tar.gz build sets INSTALL=echo. (Hmmm... Maybe a tad more complex than this, but close.) (You could also modify the specification file so that the file specs mimicked 'install' options.) Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Tar output mode for installworld
This is easy to implement ... just build
a description of the final archive in a nice verbose
text format such as:
...which is done by NetBSD for the unprivileged release building via
build.sh. Anyone interested in working on this should possibly have a
look there.
Here's a rough implementation of my core idea. Add the
attached archive_read_support_format_ntree.c to libarchive
and patch archive_read_support_format_all.c, then
rebuild libarchive and bsdtar. You'll then be able
to read, extract, etc, a format I'm calling "ntree"
for now. This similar to NetBSD's "metalog" format, except:
1) First line must be "#%ntree". This is used as a file signature.
2) Blank lines and lines beginning with '#' are ignored.
3) All other lines have the following format:
= = ...
Where key is one of:
time: decimal seconds since beginning of epoch
gid,uid: decimal group/user ID
gname,uname: textual group/user name
mode: octal
type: as in mtree, defaults to 'file'
content: name of a file on disk
E.g.,
#%ntree
bin/echo uid=0 gid=0 group=wheel contents=my/bin/echo
I think this should form a reasonable basis against which
to implement tar output for installworld. I would actually
suggest building the specification file at buildworld time, not
at installworld time. You could then create a tarball with
tar -czf system.tgz @specification.ntree
or install directly from the specification file using
tar -xvpf specification.ntree -C ${DESTDIR}
Some work still remains:
* Should allow multiple (consecutive) lines for a single file.
* Need to support more keys, especially "flags" and "link".
* Need to find a way to encode hardlinks.
* Need to decide how/whether to reconcile this with mtree. (This
could be extended to read regular mtree files, though it's unclear how
to auto-detect the standard mtree format.)
* Need to implement a test suite for this format and add it to
libarchive_test.
* Should be able to write these files. (A libarchive writer
could even accumulate various hashes and include them, though
the body per se would be lost.)
Feedback appreciated,
Tim Kientzle
Index: archive_read_support_format_all.c
===
--- archive_read_support_format_all.c (revision 398)
+++ archive_read_support_format_all.c (working copy)
@@ -35,6 +35,7 @@
archive_read_support_format_cpio(a);
archive_read_support_format_empty(a);
archive_read_support_format_iso9660(a);
+ archive_read_support_format_ntree(a);
archive_read_support_format_tar(a);
archive_read_support_format_zip(a);
return (ARCHIVE_OK);
/*-
* Copyright (c) 2003-2007 Tim Kientzle
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
*notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
*notice, this list of conditions and the following disclaimer in the
*documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR
* IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
* IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
* NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
* THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include "archive_platform.h"
__FBSDID("$FreeBSD$");
#ifdef HAVE_SYS_STAT_H
#include
#endif
#ifdef HAVE_ERRNO_H
#include
#endif
#ifdef HAVE_FCNTL_H
#include
#endif
#include
/* #include */ /* See archive_platform.h */
#ifdef HAVE_STDLIB_H
#include
#endif
#ifdef HAVE_STRING_H
#include
#endif
#include "archive.h"
#include "archive_entry.h"
#include "archive_private.h"
#include "archive_read_private.h"
struct ntree {
struct archive_stringline;
size_t buffsize;
char*buff;
off_toffset;
int fd;
int bid;
int filetype;
};
static int bid(struct archive_read *);
static int cleanup(struct archive_read *);
static int parse_setting(struct archive_read *, struct ntree *,
Re: Tar output mode for installworld
Paul Schenkeveld wrote: ... read, extract, etc, a format I'm calling "ntree" for now. ... lines have the following format: = = ... Where key is one of: time ... gid,uid ... gname,uname ... mode ... content ... ... I've been playing a lot with automating and customizing the build/distribute/install process of FreeBSD ... Yes, there are a lot of interesting games you can play if you can build a description of a .tgz file and then handily generate a .tgz file from that description. I was pleased that bsdtar's already-existing "archive conversion" feature made this so simple to implement. For the $CUSTOMIZATIONS to work it would be very nice to extend your proposal with the following features: - Allow scattered lines describing attributes of the same file but retain the order in which they appear so that $CUSTOMIZATIONS can override attributes set by make installworld. I already intend to support multiple lines for the same file to allow attributes to be specified separately: file1 type=file mode=04666 file1 uname=root gname=wheel file1 flags=noschg It's trivial to ensure that later options override earlier ones. The tricky part is the word "scattered." This either requires that libarchive read the entire description file into memory up front (awkward, but not particularly unreasonable) or that we have a separate tool that can move those scattered lines so they occur together. (My initial idea was that /usr/bin/sort would suffice, but I don't think it will. In particular, tar requires that hardlinks follow the object being linked to, which is not something that /usr/bin/sort is going to preserve.) - Implement something like: remove Hmmm... Interesting idea. It would also be interesting to extend the tar format with a "whiteout" entry that erases an entry already on disk. Hmmm does this raise any security issues? I'll have to think about that one. Question: Should file1 type=file file1 remove only remove the file from this archive (i.e., not create file1) or should it also look on disk and remove file1 if it's there. Obviously, the latter can only be carried through a .tgz installation file if the tar format supports whiteouts. - Implement something like: move= to allow $CUSTOMIZATIONS to move things around. A special case here should be observed: /bin/foo ... ... /bin/foo move=/bin/foo.orig /bin/foo ... Implementable if libarchive read the entire description into memory up front, but otherwise very tricky. Looks like it will be mandatory for the libarchive support to read the entire description into memory so it can preprocess it before returning complete entries. Having a file describing everything that gets installed would also benefit later upgrades to a system. One of my questions: Does my proposed format suffice for these other purposes? If not, what other features would be required? Is it worth trying to design a single format that handles these various cases? In my experiments I've implemented a manifest that gets shipped with the tarball.tgs ... includes md5 and/or sha256 ... There is some real value to having md5 and/or sha256 checks in generic tar archives as well. I have a couple of ideas for supporting this using pax extensions, but it's tricky to implement robustly, so it's not going to happen for a little while. (In particular, the error check in the tar archive should include all of the headers, and you want to avoid a separate scan of the file to collect an error check before archiving it.) Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Tar output mode for installworld
Ulrich Spoerlein wrote:
On Sat, 14.07.2007 at 23:28:05 -0700, Tim Kientzle wrote:
#%ntree
bin/echo uid=0 gid=0 group=wheel contents=my/bin/echo
... create a tarball with
tar -czf system.tgz @specification.ntree
or install directly from the specification file using
tar -xvpf specification.ntree -C ${DESTDIR}
This would be the perfect basis on which to build a live/install release
CD. You boot it up ... [do] the fdisk/bsdlabel/gmirror/zfs stuff ...
> [and] then kick of the install through tar.
Simple and elegant. It would also do away with those base.aa, base.ab,
etc. madness.
I'm confused. base.aa, etc, are a tar file, so I don't
entirely understand how this would be different? The
current installer does the equivalent of
cat base.* | tar -xf -
I can see one advantage and one disadvantage of installing
a specification file (which references other files) instead:
Plus: The specification file can re-use the existing
files on CD, so you don't have, e.g., one copy of /bin/sh
on the live CD and another buried in base.tgz. This
could save space.
Minus: Installing a specification file this way would
be slower because you then have to read a lot of small
files off of CD.
Or have I missed something?
Tim Kientzle
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Tar output mode for installworld
Being able to record a series of incremental changes in a filesystem hierarchy, and then roll them back as required. That would be exceedingly useful, and I think your 'ntree' format has virtually everything necessary to do that. The most obvious missing bit I can see is creating a backup of a file before overwriting it with different content. Is this something that requires changes to the specification file format, or just a feature of the tool that uses the specification file? If the former, what do you envision would be required? Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Tar output mode for installworld
Ulrich Spoerlein wrote: On 7/15/07, Tim Kientzle <[EMAIL PROTECTED]> wrote: Plus: The specification file can re-use the existing files on CD, ... That is exactly what I was referring to above. And AFAIK DragonflyBSD does it in a similar way. They simply copy the live CD onto the HDD. Minus: Installing a specification file this way would be slower because you then have to read a lot of small files off of CD. True, but couldn't we optimize the ISO layout so it will be a near sequential read of the CD? Hmmm... This might work. The prototype 'ntree' support I posted should be sufficient for people to experiment with these ideas. Cheers, Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Useful tools missing from /rescue
Yar Tikhiy wrote: Hi all, I've had to use /rescue recently and felt lack of a few basic tools in it, namely pgrep(1), head(1), tail(1), tee(1), and a text filter, e.g., sed(1). Well, in fact most functionality of pgrep(1), head(1), tail(1), and even tee(1) can be emulated if one has sed(1), but the tools are so tiny and convenient that it's a pity not to have them all handy during hard times. In addition, there are chflags and chmod in /rescue, but there's no chown in it, so the toolset is a bit incomplete. Oh, my. chown was definitely an oversight. That should have been in there. My opinions: * head, tail, and tee are no-brainers; just do it. * sed would be a nice addition. (I atttempted to fit 'vi' in there, but curses is rather finicky; 'sed' would be more useful.) * I personally would not add pgrep/pkill because I never use them. Of course, it's hard to get worked up about 45k, so feel free to ignore the above. Cheers, Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Useful tools missing from /rescue
Giorgos Keramidas wrote: ... [chown] is still almost 4x the size of chmod: > $ ls -ld chown > -rwxrwxr-x1 keramida users - 550624 Sep 3 03:06 chown $ ls -ld chmod -rwxrwxr-x 1 keramida users - 165884 Sep 3 03:08 chmod getpwuid() pulls in DNS, NIS, and a bunch of other network routines, which are quite large. chmod doesn't need to look up user or group names. Tim Kientzle P.S. Interesting experiment: for every function in libc, create a statically linked and stripped binary that references that one function and look at the sizes. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: dump problems
Danny Braniss wrote: [dump] has been around since the beginin of time, or at least since Unix V6 :-), and it has been hacked ever since. but now that most of you never heard of 9track tapes, etc, I was wondering if there is a point in hacking at it again. pros: dump/restore has never failed me till now. cons: there are other programs tar/cpio/gtar/etc, but they each have their nits. I think there is real value in a backup/restore option that is specifically designed for UFS volumes. In particular, it's the only way to be comfortably certain that all UFS-specific attributes (ACLs, extended attributes, etc) are correctly backed-up and restored. Tar, cpio, and other similar programs are widely used for purposes other than whole-system backup. As such, they have to balance requirements that simply aren't of interest to dump/restore. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Some FreeBSD performance Issues
Dan Nelson wrote: In the last episode (Nov 08), Randall Hyde said: It appears that character-at-a-time file I/O is *exceptionally* slow. ... reasonable, though not stellar, performance under Windows and Linux. However, with the port to FreeBSD I'm seeing a three-orders-of-magnitude performance loss. ... What timings does "dd if=/usr/include/sys/socket.h of=/dev/null ibs=1 obs=64k" report? It takes about .4 sec on my non-idle dual pIII-900 system. Try truss'ing your program as it runs; maybe the program is doing some extra syscalls you aren't aware of? You should also carefully do an strace or similar on Windows and Linux as well. You may find that you're doing a system call per byte on FreeBSD but not on those other systems. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Architectures with strict alignment?
Ivan Voras wrote: Which of the architectures FreeBSD supports (if any) have strict memory alignment requirements? (in the sense that accessing a 32-bit integer not aligned on a 32-bit address results in a hardware trap/exception). I believe ARM has such requirements (at least, GCC for Arm does some very intriguing structure padding). Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: dlopen(), atexit() crash on FreeBSD (testcase included)
Markus Hoenicka wrote: Alexander Kabaev writes: > As designed. atexit should not be used by shared objects that do not > expect themselves to live until actual exit() happens. ELF provides > proper _init/_fini sections to support shared object > initialization/destruction. > That is, the only real solution to this problem is to convince the Firebird folks to remove their atexit() calls from the client libraries? I suspect they never considered that their dynamic library might be used via dlopen()/dlclose(). The real question is whether they're interesting in supporting this model. If the Firebird folks aren't interested in having their library be accessible in that fashion, then you have little choice but to simply forgo unloading this particular library. It's a bit unfortunate that there is no standard way to remove an atexit() registration. It would probably be easier to convince the Firebird folks to remove the registration as part of their cleanup routines (and you could then invoke those cleanup routines manually for that case). Also, I'm wondering how other OSes handle this. I don't see this code crash on Linux, contrary to its design as you say. I would be curious to see the results of running your sample program (with lots of extra fprint(stderr...) calls, of course) on Linux to see whether it calls the registered exit function at dlclose time or never. I suspect the answer is "never." If I'm right, the Firebird folks might be easier to convince (since it means that their atexit() registration isn't really reliable anywhere). Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: dlopen(), atexit() crash on FreeBSD (testcase included)
Also, I'm wondering how other OSes handle this. I don't see this code crash on Linux, contrary to its design as you say. I would be curious to see the results of running your sample program ... on Linux to see whether it calls the registered exit function at dlclose time or never. Linux pulls hidden atexit symbol into every binary that uses it ... Linux calls atexit entries at object unload time. Solaris implements a libc callback from ld.so.1 ... Solaris calls atexit callback when removing it too. Interesting. So the consensus here (at least for Linux and Solaris, anyone know about Mac OS?) seems to be that atexit() can be called from a dynamically loaded library and that functions registered this way will be called at library unload time. And FreeBSD doesn't implement this behavior. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Squeezing out some 70 bytes out of the boot2 loader
Adrian Penisoara wrote: Hello I am trying to hack in some symlink support into the [sys/boot/i386/]boot2 bootloader (for my project [1]) and I seem to fall short of about 69 bytes: ... [1] I'm trying to get support for /boot being mounted as a separate FS and as such I would need to have a "self-pointing" symlink (e.g. "boot -> ." ) Just because there is a symlink doesn't mean boot2 has to follow it. I haven't looked at the boot2 code, but I presume there's the equivalent of "chdir boot" in there. What if you just ignored failure (if there's no "boot" entry or "boot" isn't a dir, assume the boot files are in the root of the FS). Wouldn't that work just as well and require less space? Cheers, Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: encrypted executables
Thiago Damas wrote: And if you make a wrapper, and execute like a shell script: #!/usr/local/bin/mysecyritywrapper <...encryted code goes where...> In this way. it'll be hard to use truss, ktrace, strace etc... No, not really. All of those tools can trace through to sub-processes, so whenever the code gets decrypted and starts executing (whether it's in the main process or in a sub-process), they'll be able to follow the system calls it makes. The key fact about DRM is that it is theoretically impossible. Of course, that's what makes it such an interesting problem in practice. ;-) Cheers, Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: OT: Stream structures in bzlib and zlib
Heiko Wundram (Beenic) wrote: Hey all! I'm currently working on a project using zlib and bzlib, and I'm currently slightly stomped by the fact that both define the input buffer in their stream structure as non-const. I think they're not defined as const just to maintain compatibility with old and/or broken compilers. libarchive treats them as const and has never had any problems from this. Cheers, Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: emulate an end-of-media
Why compress? It's ancient technology and will be vastly outperformed Yes, gzip or bzip2 compress better, but they also: * Are a lot slower. * Use a lot more data memory. * Require a lot more code. Also some nasty person has a software patent on compress... That was over 20 years ago; the patent expired already. Cheers, Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Some versioned storage program?
Xin LI wrote: Ah... Ok I think I should have mentioned the purpose of the system. It is supposed to be used in a CMS system, and the storage program will be used as one auxiliary backend where rendered pages are being stored. I've considered using SVN+Apache behind one or more Squid caches for serving pages: * Easy remote update (via SVN command-line or other tools; TortoiseSVN even makes manual updates easy for most Windows users). * Easy tracking and rollback. * No need to mess with explicit "export" processes; simply committing the change to SVN makes it available. Not sure if this suits your needs or not, but I thought I'd throw the idea out there. Cheers, Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: mtree acl [patch]
Andrew Pantyukhin wrote: ... ACL support in our mtree ... Also here: http://heka.cenkes.org/sat/diffs/mtree_acl.diff Could you give an example of a short mtree file that includes ACLs? I see a few minor style issues (tag names should be sorted on the mtree.5 and mtree.8 man pages, you need to correct a comment that got duplicated in mtree.h), but the idea looks right. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: mtree acl [patch]
Could you give an example of a short mtree file that includes ACLs? Here, the parent dir and "file" have ACLs, "kk" doesn't. I considered "acl=none" for ACL-less files, but I think it'll produce more problems than use cases. # . /set type=file . type=dir \ acl=user::rwx,user:root:rwx,group::r-x,mask::rwx,other::r-x fileacl=user::rw-,group::r--,group:wheel:rwx,mask::rwx,other::r-- kk acl=user::rw-,group::r--,other::r-- .. 'kk' here should not have an 'acl' keyword at all. Just omit it. If someone specifies 'acl' keyword and not 'mode' keyword, then its because they only want to see extended ACL information. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: indent(1) support for gcc(1) 0b prefix
But I would like to use indent(1) to reformat contributed code automatically. Unfortunately, the 0b notation is not supported by that program, and the resulting code looks like this: foo = 0 b00101010; ... then compilation fails, bla bla bla... I can't think of a case (outside of "0x" context) where "...0b..." would be valid C code, let alone better formated as "...0 b...". Hence I see no harm in adding your patch to the base indent(1). Does anyone have an example where "...0 b..." is valid C code? The only case I can think of is when the "b..." is an existing macro, i.e. something like: 1 #include 2 3 #define b0101 + 3 In all of these cases, though, adding a space between the '0' and the 'b' changes the meaning, so is wrong. Indent can change/insert whitespace, but should never do so in a way that changes the meaning of the program. In all of these cases, having indent recognize "0b..." as a single token is the correct behavior. So I don't see any point in having this recognition be tunable. indent already has too many switches. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Adding .db support to pkg_tools
I think this is a really bad idea. The problem with the tools is not with the files. It is that the files need to be parsed on each run, often recursively, and your solution would not help at all. Parsing one file isn't expensive; parsing several hundred files to find one bit of information is expensive. The database(s) should just be a cache of the information stored in the files. Bingo! As long as the .db version can be easily recreated from scratch from the master data stored in the same files as always, it doesn't really matter if the BDB is occasionally corrupted, as long as it can be rebuilt fairly quickly. So, the key points are: * Use the .db file for lookups that can't be done quickly against the existing data. In particular, look for operations where the pkg tools have to scan a lot of files to verify a single fact. * Ensure the .db file can be deleted and rebuilt from scratch from the data in the regular files. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: BDB corrupt - patches
Jeff Anton wrote: Some years ago I mailed patches out to someone regarding Berkeley DB 1.85 btree problems. The two issues which come to mind are... 1) The page split position is improperly computed. ... 2) The record put code has a "last page put to" member ... I'm going to have to dig up these fixes, but presuming I do, who should be alerted? Just file a bug? Recreation is extremely difficult. Definitely file a PR and include the patches. If nothing else, that will help ensure they don't get lost. When you do file that PR, notify this list. Given how heavily the base system uses DB 1.85, I think there will be a lot of interest in anything that improves the stability and reliability of that code. Cheers, Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Debugging rtld
I tried to compile my source tree with -fstack-protector-all, and it happens that rtld breaks with this ... I usually add the CFLAGS+=-g to the rtld-elf/Makefile. Also, you do not need to bring down the whole host by the broken ld.so.1. Do not install it at all, and specify the path to the rtld by the --dynamic-linker switch, see into ld. If you do (accidentally?) break ld.so.1, remember that /rescue is all statically linked and should still work. Single-user boot to /rescue/sh is your friend. Cheers, Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: GSoC Student
Sounds interesting. I'm looking forward to hearing more about your work. Cheers, Tim Kientzle Victor Hugo Bilouro wrote: Hi, I'm Victor Hugo *Bilouro*, GSoC participant. I'll be working on tcptest. It's name came from TCP/IP Regression test suite. As a testing tool it can: *be a conformance test *test a bug already fixed (regression test - one test for each fixed bug) *test a new feature (help protocol developer) *simulate certain cases It will be build on top of pcs.sf.net (Package Construction Set) "PCS is a set of Python modules and objects that make building network protocol code easier for the protocol developer. The core of the system is the pcs module itself which provides the necessary functionality to create classes that implement packets. " My mentor is George V. Neville-Neil, no comments. :-) GSoC started a month ago, by this time, I had almost all environment done, and I'm spending several times reading and often playing with pcs. (cool) Books i bought: (*)TCP/IP Illustrated Volume 1 (far the best) Stevens (*)TCP/IP Guide Charles M.Kazierok (www.tcpipguide.com) (*)Internetworking with tcp/ip I will post on __link__ links, books, status report, futures releases information, news, everything about the project. __link__: http://wiki.freebsd.org/VictorBilouro/TCP-IP_regression_test_suite Regards ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Announcing PathDB
Sounds interesting. There has been a lot of interest within the FreeBSD community on getting a complete set of core development tools that are free of GPL. A debugger is obviously a key part of that. Do you have a Wiki page with more details about what it does and how to start using it? What CPUs does it support? What is required to add support for a new CPU? What executable/debugging formats? What is required to add support for new ones? What languages does it support? What is required to add support for new languages? Does it handle the GDB wire protocol? I would recommend that you put together a FreeBSD port and make that available somewhere. That would make it much easier for people to try it out. Especially for a large project like this, it takes a while to build a user base and even longer to build interest among external developers. Please don't be discouraged by the lack of (constructive) feedback so far. Cheers, Tim Kientzle C. Bergström wrote: PathScale is slowly open sourcing and porting some of our core software technology and thought the BSD community might be interested in PathDB. Months ago we gave a few FBSD developers private access to the source, but never received any feedback. Now we're asking more people to please test and tell us what you think. Source git clone git://git.pathscale.com/path64/debugger.git Note : We have a small patch which makes it build with gcc-4.3+ that should be pushed soon. License (GPLv3, *BUT* if we get a lot of positive feedback from the BSD community we can change this any time - see below) Benefits * Very well designed * Robust and well tested * Similar interface as GDB * Commercially supported * Recently revived roadmap (We're evaluating things like gpu support, reverse debugging, latest dwarf standards, tight integration with compiler for optimized debugging.. etc) Promise - Give us a hand in testing and improving and we'll relicense to a more permissive license. Questions/comments - Say hi on #pathscale - irc.freenode.net or email me directly Thanks Christopher ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
