-reply:<4b7f7f02.1000...@freebsd.org>
Hi > I was looking at your Linux code here and thought > the technique of trying lseek(SEEK_END) might work. Linux 2.6.18, /dev/hda and /dev/sr0, with a CD-RW written by write type TAO lseek(fd, 0, SEEK_END)= 636499968 lseek(fd, -1, SEEK_END)=636499967 lseek(fd, -300k, SEEK_END)= 636192768 and with a CD-ROM media: lseek(fd, 0, SEEK_END)= 618311680 lseek(fd, -1, SEEK_END)=618311679 lseek(fd, -300k, SEEK_END)= 618004480 (No tape drive at hand. Sorry.) Nevertheless: SEEK_END brings you to the first unwritten address of the file object. With a read-only object, this address is not necessarily defined. Further: If you test a CD that was written by write type TAO, then the last two blocks of the track are no readable data blocks. It depends on the implementation of lseek() whether it will allow you to address bytes in those two blocks. (Read will fail, of course. On Linux it even fails shortly before the end because of the old TAO read-ahead bug.) So on CD i would test rather lseek() to an address that is several hundred kB before the end. 300 kB is the traditional fear-padding of CD burn programs on Linux. (On Linux i decide by stat(2). Considered as seekable are S_IFREG and S_IFBLK. Actually tested only with optical drives and USB sticks. To my knowledge ioctl(BLKGETSIZE) works only on S_IFBLK.) Have a nice day :) Thomas ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: ntpd hangs under FBSD 8
On Fri, 19 Feb 2010, Peter Steele wrote: > I posted this originally on the -questions list but did not make any headway. > We have an application where the user can change the date/time via a GUI. One > of the options the user has is to specify that the time is to be synced using > ntp. Our coding worked fine under BSD 7 but since we've moved to BSD 8 we've > encountered a problem where the command that we initiate from the GUI: Just out of curiosity, can you attach to the process via gdb and get a backtrace? This smells like a locked pthread_join I hit in my own code a few weeks ago Cheers, -R. Tyler Ballance -- Jabber: rty...@jabber.org GitHub: http://github.com/rtyler Twitter: http://twitter.com/agentdero Blog: http://unethicalblogger.com pgpkfqopLBnxf.pgp Description: PGP signature
Re: GEOM_ULZMA
On 20 February 2010 04:26, Alex RAY wrote: > :) No, I don`t think about "magically faster", now I near to release FreeBSD > firmware for D-Link DIR-320 router which have only 4MB of flash memory. Maybe > in next time I try to make it for some router with only 2MB of flash. In that > way, > I need not copy of any code. > In ideal embedded systems, if we have code, we must use it everywhere we need > it. Interesting! The Redboot loader that I'm toying with on the ubiquiti hardware supports LZMA as well as GZIP for compressed kernel images; I may have to experiment with that and this module. Thanks! Adrian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: "tar tfv /dev/cd0" speedup patch
On Fri, Feb 19, 2010 at 09:20:30PM -0800, Tim Kientzle wrote:
> Juergen,
Hi!
>
> I was looking at your Linux code here and thought
> the technique of trying lseek(SEEK_END) might work.
> Unfortunately, it doesn't: lseek(fd, 0, SEEK_END) gives
> zero for both /dev/sa0 (a tape drive) and /dev/cd0
> (an optical drive). Are you sure it works on Linux?
>
Yeah that code is Linux-specific, I know it doesn't work on BSDs. :)
Here's some output on Linux after changing O_RDWR to O_RDONLY:
$ ./a.out /dev/sr0
fd=3
lseek(fd, 0, SEEK_SET)=0
lseek(fd, 10240, SEEK_SET)=10240
lseek(fd, 10240, SEEK_CUR)=20480
lseek(fd, 0, SEEK_END)=-2057306112
lseek(fd, 0, SEEK_SET)=0
$ ./a.out /dev/fd0
fd=3
lseek(fd, 0, SEEK_SET)=0
lseek(fd, 10240, SEEK_SET)=10240
lseek(fd, 10240, SEEK_CUR)=20480
lseek(fd, 0, SEEK_END)=1474560
lseek(fd, 0, SEEK_SET)=0
$
Ok /dev/sr0 was a dvd iso and you casted the lseek return value to int...
(And this was on amd64, on i386 your version gets an overflow error for
SEEK_END there too i.e. -1 because on Linux off_t defaults to be a long.)
If I fix that I get:
$ ./a.out /dev/sr0
fd=3
lseek(fd, 0, SEEK_SET)=0
lseek(fd, 10240, SEEK_SET)=10240
lseek(fd, 10240, SEEK_CUR)=20480
lseek(fd, 0, SEEK_END)=2237661184
lseek(fd, 0, SEEK_SET)=0
$
...which matches the size of the iso. (and bsdtar with the patch
also is much faster on /dev/sr0 there than without it, which was how I
originally confirmed the patch is working. There are two 850 MB files
on that iso...)
I'll append my version of the test program below.
Cheers,
Juergen
PS: Seeking on tape is a whole other can of worms, I don't even think
lseek on Linux works as you might expect there... If you really want to
implement this you'd have to try the MTFSR ioctl (at least that's what
its usually called, look for `fsr' in the mt(1) manpage and source),
but since that counts in blocks not bytes you'd have to know the tape's
blocksize too (which can also be variable i.e. depend on how that
particular tape was written, tho I think in case of a tar archive
you can get away with just using the blocksize arg passed to bsdtar
there.) And also I'm not sure how some drives may behave if you use
lots of MTFSR ioctls with small block counts so maybe you should
only use them when the amount of data to skip is big enough so that
switching to `fast forward' is actually worth it. (and continue to
skip over small amounts by doing regular reads.)
---snip---
/* make sure we don't get 32 bit off_t */
#define _FILE_OFFSET_BITS 64
#include
#include
#include
#include
#include
int
main(int argc, char **argv)
{
int fd;
if (argv[1] == NULL) {
fprintf(stderr, "Need to specify a pathname.\n");
exit(1);
}
fd = open(argv[1], O_RDONLY);
printf("fd=%d\n", fd);
printf("lseek(fd, 0, SEEK_SET)=%jd\n",
(intmax_t)lseek(fd, 0, SEEK_SET));
printf("lseek(fd, 10240, SEEK_SET)=%jd\n",
(intmax_t)lseek(fd, 10240, SEEK_SET));
printf("lseek(fd, 10240, SEEK_CUR)=%jd\n",
(intmax_t)lseek(fd, 10240, SEEK_CUR));
printf("lseek(fd, 0, SEEK_END)=%jd\n",
(intmax_t)lseek(fd, 0, SEEK_END));
printf("lseek(fd, 0, SEEK_SET)=%jd\n",
(intmax_t)lseek(fd, 0, SEEK_SET));
return (0);
}
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: "tar tfv /dev/cd0" speedup patch
On Sat, Feb 20, 2010 at 01:00:40AM -0500, Jung-uk Kim wrote: > On Saturday 20 February 2010 12:20 am, Tim Kientzle wrote: > > Juergen, > > > > I was looking at your Linux code here and thought > > the technique of trying lseek(SEEK_END) might work. > > Unfortunately, it doesn't: lseek(fd, 0, SEEK_END) gives > > zero for both /dev/sa0 (a tape drive) and /dev/cd0 > > (an optical drive). Are you sure it works on Linux? > > Can you please try ioctl(fd, BLKGETSIZE64, &some_uint64_var) or > ioctl(fd, BLKGETSIZE, &some_u_long_var)? Yeah I've stumbled across these ioctls in the meantime too, I was just not sure if all Linux versions currently in use already have the 64 bit version i.e. BLKGETSIZE64... (since we don't want the 32 bit version for the same reason we don't want 32 bit off_t. :) Cheers, Juergen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Building FreeBSD on a linux FC11 box.
Hopefully, this is not too ignorant a question. But has anyone every built the FreeBSD sources, both kernel and apps, on a linux platform? I did a google on 'cross-compile freebsd (linux)' and found (mostly) discussions regarding building the linux apps for FreeBSD, but not for building a FreeBSD kernel and world. There was a brief discussion on the mailing list back in 2005, but the poster never reported if he had success or failure. Also, there were discussions about building freebsd amd64 on a freebsd i386. I have done lots of development in linux using cross-compilers (mips embedded systems, powerpc, etc). But not with FreeBSD. I know I'll need to use 'bsdmake' no gnu 'make'. The other worry is kernel-toolchain target. I guess I'll just dive in and swim around and see what happens. Thanks for any pointers, Patrick ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: GEOM_ULZMA
Hi, On Sat, 20 Feb 2010 22:36:02 +0800 Adrian Chadd wrote: > On 20 February 2010 04:26, Alex RAY wrote: > > > :) No, I don`t think about "magically faster", now I near to release > > FreeBSD firmware for D-Link DIR-320 router which have only 4MB of flash > > memory. Maybe in next time I try to make it for some router with only 2MB > > of flash. In that way, > > I need not copy of any code. > > In ideal embedded systems, if we have code, we must use it everywhere we > > need it. > > Interesting! The Redboot loader that I'm toying with on the ubiquiti > hardware supports LZMA as well as GZIP for compressed kernel images; I > may have to experiment with that and this module. No, this module not fore compress/decompress kernel. This module used like geom_uzip (man geom_uzip, man mkuzip), for compressing blocks of filesystem to reducing size of FS image. To use with bootloader You may use lzma utility. > > Thanks! > > > Adrian -- Alex RAY ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: GEOM_ULZMA
Oh I know that! I'm just saying that I may try lzma'ing the kernel and rootfs's to see what kind of savings I get over gzip. :) Adrian On 21 February 2010 02:15, Alex RAY wrote: > Hi, > > On Sat, 20 Feb 2010 22:36:02 +0800 > Adrian Chadd wrote: > >> On 20 February 2010 04:26, Alex RAY wrote: >> >> > :) No, I don`t think about "magically faster", now I near to release >> > FreeBSD firmware for D-Link DIR-320 router which have only 4MB of flash >> > memory. Maybe in next time I try to make it for some router with only 2MB >> > of flash. In that way, >> > I need not copy of any code. >> > In ideal embedded systems, if we have code, we must use it everywhere we >> > need it. >> >> Interesting! The Redboot loader that I'm toying with on the ubiquiti >> hardware supports LZMA as well as GZIP for compressed kernel images; I >> may have to experiment with that and this module. > > No, this module not fore compress/decompress kernel. This module used like > geom_uzip (man geom_uzip, man mkuzip), for compressing blocks of filesystem > to reducing size of FS image. > > To use with bootloader You may use lzma utility. > > >> >> Thanks! >> >> >> Adrian > > > -- > Alex RAY > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" > ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Building FreeBSD on a linux FC11 box.
On Sat, 20 Feb 2010, Patrick Mahan wrote: > > Hopefully, this is not too ignorant a question. But has anyone every > built the FreeBSD sources, both kernel and apps, on a linux platform? > > I did a google on 'cross-compile freebsd (linux)' and found (mostly) > discussions regarding building the linux apps for FreeBSD, but not > for building a FreeBSD kernel and world. There was a brief discussion > on the mailing list back in 2005, but the poster never reported if > he had success or failure. Also, there were discussions about building > freebsd amd64 on a freebsd i386. > > I have done lots of development in linux using cross-compilers (mips > embedded systems, powerpc, etc). But not with FreeBSD. > > I know I'll need to use 'bsdmake' no gnu 'make'. The other worry is > kernel-toolchain target. I guess I'll just dive in and swim around > and see what happens. You might want to ask the Debian GNU/kFreeBSD guys: http://www.debian.org/ports/kfreebsd-gnu/ I bet they've got a good idea :) Cheers, -R. Tyler Ballance -- Jabber: rty...@jabber.org GitHub: http://github.com/rtyler Twitter: http://twitter.com/agentdero Blog: http://unethicalblogger.com pgpub3D3K0Wny.pgp Description: PGP signature
Re: "tar tfv /dev/cd0" speedup patch
Tim and Juergen
I have a couple of suggestions which may help you with what you are
trying to do.
First, though, I can confirm that Unix character special tape
drivers since at least V7 have always ignored seeks. They happily
return the requested offset without feeling the need to actually
*do* anything. The FreeBSD scsi_sa driver continues this tradition.
Historically, all other Unix character special storage devices have
been unable to respond to SEEK_END, presumably because they are
either removable medium devices or hard disks which have movable
"partitions", and no one ever bothered to arrange to query the
medium for size. These devices do support seeking, and in fact it
is readily possibly to seek past the end of the file, just as for
regular files.
Unix block special tape drivers have historically supported seeking.
Somebody decided that FreeBSD doesn't need block devices and tossed
them out; I think that's unfortunate. Last time I looked (long time
ago) Linux had block device tapes, which did seek, and character
special tape devices which did not. (Correct me if I'm wrong, please).
I have a tar-compatible package of my own (tartools) which is about 20
years old but features the ability to fast_skip file data when listing or
skipping unselected files on extract. It uses either seeking or mtio,
and handles the TAPE vs non-TAPE character special problem by simply
checking to see if the device responds to mtio commands. If it does, it's
a tape, and can't seek (but can zip along using mtio).
The last time I compiled it on Linux, the same situation applied, but
that was about 8 years ago, and according to Jung-uk Kim the linux folks
have finally done something about it. The 32 bit version of the ioctl
does appear in the linux emulator in
/usr/src/sys/compat/linux/linux_ioctl.c
(I'm on freebsd 6.4/i386), and is implemented using an "fo_ioctl()"
with argument DIOCGMEDIASIZE, which /usr/include/sys/disk.h defines
as "_IOR('d', 129, u_int)". That appears to be a valid ioct arg,
because several freebsd programs, including bsdlabel.c use
ioctl(f, DIOCGMEDIASIZE, &mediasize)
where mediasize is declared as off_t, which seems wrong on recent
freebsds, where off_t is 8 bytes, but u_int
In any case, an ioctl isn't going to be very portable at this point
(although very handy). There's another way to get medium size if
you need it. Tartools use medium size to enable multi-volume
archives, and until recently required user input by option or "rc"
file to specify volume size. I've been doing extensive upgrades and
bug fixes on tartools the last 3 months, and I've recently added
code which will now get the size of USB sticks, compact flash, etc.
(even floppies) without requiring a command line argument.
It is possible to seek way past the end of a character special file
(and never know it). This isn't a problem for single file archives
(although it might be risky) but if you want multiple volumes you
really need a size (which may be less than the medium size).
You can seek past the end of the medium, but you'll get an error if
you try to read or write there, so it is possible to get an accurate
size using a binary "seek and read" strategy to discover the highest
sector number which is readable).
This is getting a little long, so let me summarize the suggestions.
1. Disambiguate character special tape devices from other character
special devices by issuing an mtio "status" ioctl (MTIOCGET)
and checking the 'mt_type' byte in the status structure. If
it is 0, the device is not a tape and will support seeking
(presuming it is a storage device). It the type is non-zero
then you can mark it non-seekable and either quit there or
program the code to do the skipping via mtio.
2. If you need the size of a device partition or removable
medium, use a binary search algorithm which seeks forward
until a read at the offset fails, then cutting the offset
in half, seeking and reading again, etc. It is usually
possible to converge on a sector withing a couple of dozen
seeks.
On Sat, 20 Feb 2010 01:00:40 -0500
Jung-uk Kim wrote:
> On Saturday 20 February 2010 12:20 am, Tim Kientzle wrote:
> > Juergen,
> >
> > I was looking at your Linux code here and thought
> > the technique of trying lseek(SEEK_END) might work.
> > Unfortunately, it doesn't: lseek(fd, 0, SEEK_END) gives
> > zero for both /dev/sa0 (a tape drive) and /dev/cd0
> > (an optical drive). Are you sure it works on Linux?
>
> Can you please try ioctl(fd, BLKGETSIZE64, &some_uint64_var) or
> ioctl(fd, BLKGETSIZE, &some_u_long_var)?
>
> Jung-uk Kim
> ___
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
>
-
duane.hes...@gmail.com
_
Re: "tar tfv /dev/cd0" speedup patch
Duane H. Hesser wrote: I have a couple of suggestions which may help you with what you are trying to do. First, though, I can confirm that Unix character special tape drivers since at least V7 have always ignored seeks. They happily return the requested offset without feeling the need to actually *do* anything. The FreeBSD scsi_sa driver continues this tradition. Duane, Thank you very, very much for taking the time to write up this information. I'm going to definitely archive your message for future reference. To clarify "what we are trying to do": For some time now, libarchive format handlers have issued "skip" requests internally for things like skipping unnecessary file bodies. Depending on the device you're talking to, these internal skip requests can be translated into seek operations or can be handled by reading and discarding data. As Juergen found when he tried to do "tar tf /dev/cd0", my earlier code was pretty conservative: It only translated skip requests into lseek() operations for regular files. Juergen took a stab at improving that so that operations on raw disk devices would use lseek(). With your information, it should be pretty easy to translate skip requests into mtio operations on tape drives. Libarchive's core already supports rounding skip operations to multiples of the block size, so this should be an easy addition. As it happens, I'm just starting this week to look into multi-volume support for libarchive. I'm hoping to get that into good shape this year. Your notes on size estimation will be quite helpful once I get further into that. Thanks again, and I may be in touch over the next few months with more questions ;-) Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: GEOM_ULZMA
b. f. wrote: The code organization depends on what you want to do with it and how you want to update the code in the future, if your lzma library is third party. LZMA made by Igor Pavlov, and since 4.62 it licensed under Public Domain. So we can use it, if we need. If you never intend to update the lzma code then I guess it's fine to embed it in a big .c file. For a port, it doesn't matter much since it is your own thing. There are stricter rules on maintainability and style if you want it in the base system. So, I think right place for lzma library under sys/contrib directory, if I "promise" >maintain it? There are already long-standing plans to import lzma compression libraries into the base system, by the libarchive maintainers. Libarchive has hooks to support the liblzma library from Lasse Collin's "xz utils" package. When linked against this library, libarchive (and hence bsdtar) has full support for "bare" lzma streams and for Collin's "xz" format (which is a big improvement over bare lzma streams). Like Igor Pavlov's original code, the "xz utils" code is in the public domain. Officially, xz utils is in version 4.999.8beta; Collin has promised to bump the version to 5.0 once he feels the code is sufficiently stable. As far as I can tell, it's actually in pretty good shape now; the changes going in seem to mostly be minor portability fixes. I see no reason to not import the current version into the FreeBSD base system. I plan to do so but may not get to it very soon; I certainly would not complain if someone else beat me to it. ;-) There are other lzma libraries around, of course. In particular, I've seen references to the lzma libraries distributed with lzip. Unfortunately, lzip is GPLv3 so is not a candidate for inclusion in FreeBSD's base system. Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: GEOM_ULZMA
>> The code organization depends on what you want to do with it and how you >> want to update the code in the future, if your lzma library is third party. > >LZMA made by Igor Pavlov, and since 4.62 it licensed under Public Domain. >So we can use it, if we need. > >> >> If you never intend to update the lzma code then I guess it's fine to >> embed it in a big .c file. For a port, it doesn't matter much since it >> is your own thing. There are stricter rules on maintainability and style >> if you want it in the base system. > >So, I think right place for lzma library under sys/contrib directory, if I >"promise" >maintain it? There are already long-standing plans to import lzma compression libraries into the base system, by the libarchive maintainers. So you will probably have to coordinate with them. They have hooks in the current libarchive sources for native lzma compression, but they reportedly have been waiting for related file formats and compression methods to stabilize before enabling them by default. Pavlov has been making significant changes from release to release, so this may not be the easiest code to maintain. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
