Re: Re: Odd ZFS boot module issue on r332158
On 04/09/18 23:33, Allan Jude wrote: On 2018-04-09 19:11, Andrew Gallatin wrote: I updated my main amd64 workstation to r332158 from something much earlier (mid Jan). Upon reboot, all seemed well. However, I later realized that the vmm.ko module was not loaded at boot, because bhyve PCI passthru did not work. My loader.conf looks like (I'm passing a USB interface through): ### vmm_load="YES" opensolaris_load="YES" zfs_load="YES" nvidia_load="YES" nvidia-modeset_load="YES" # Tune ZFS Arc Size - Change to adjust memory used for disk cache vfs.zfs.arc_max="4096M" hint.xhci.2.disabled="1" pptdevs="8/0/0" hw.dmar.enable="0" cuse_load="YES" ### The problem seems "random". I rebooted into single-user to see if somehow, vmm.ko was loaded at boot and something was unloading vmm.ko. However, on this boot it was loaded. I then ^D'ed and continued to multi-user, where X failed to start because this time, the nvidia modules were not loaded. (but nvidia had been loaded on the 1st boot). So it *seems* like different modules are randomly not loaded by the loader, at boot. The ZFS config is: config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p2 ONLINE 0 0 0 da3p2 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 ada1p2 ONLINE 0 0 0 da0p2 ONLINE 0 0 0 cache da2s1d ONLINE 0 0 0 The data drives in the pool are all exactly like this: => 34 9767541101 ada0 GPT (4.5T) 34 6 - free - (3.0K) 40 204800 1 efi (100M) 204840 9763209216 2 freebsd-zfs (4.5T) 9763414056 4096000 3 freebsd-swap (2.0G) 9767510056 31079 - free - (15M) There is about 1.44T used in the pool. I have no idea how ZFS mirrors work, but I'm wondering if somehow this is a 2T problem, and there are issues with blocks on difference sides of the mirror being across the 2T boundary. Sorry to be so vague.. but this is the one machine I *don't* have a serial console on, so I don't have good logs. Drew ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" What makes you think it is related to ZFS? Are there any error messages when the nvidia module did not load? I think it is related to ZFS simply because I'm booting from ZFS and it is not working reliably. Our systems at work, booting from UFS on roughly the same svn rev seem to still load modules reliably from the loader. I know there has been a lot of work on the loader recently, and in a UEFE + UFS context, I've seen it fail to boot the right partition, etc. However, I've never seen it fail to load just some modules. The one difference between what I run at home and what we run at work is ZFS vs UFS. Given that it is a glass console, I have no confidence in my ability to log error messages. However, I could have sworn that I saw something like "io error" when it failed to load vmm.ko (I actually rebooted several times when I was diagnosing it.. at first I thought xhci was holding on to the pass-thru device) I vaguely remembered reading something about this recently. I just tracked it down to the "ZFS i/o error in recent 12.0" thread from last month, and this message in particular: https://lists.freebsd.org/pipermail/freebsd-current/2018-March/068890.html I'm booting via UEFI into a ZFS system with a FS that extends across 2TB.. Is there something like tools/diag/prtblknos for ZFS? Drew ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: buildworld fails: fatal error: 'netpfil/pf/pf.h' file not found
On 9 Apr 2018, at 13:10, Vladimir Zakharov wrote: On Mon, Apr 09, 2018, Kristof Provost wrote: On 9 Apr 2018, at 10:50, Vladimir Zakharov wrote: For several days buildworld fails for me with the following error. Cleaning and rebuilding didn't help. ===> tests/sys/netpfil/pf/ioctl (all) --- validation --- (cd /usr/src/tests/sys/netpfil/pf/ioctl && DEPENDFILE=.depend.validation NO_SUBDIR=1 make -f /usr/src/tests/sys/netpfil/pf/ioctl/Makefile _RECURSING_PROGS=t PROG=validation ) Building /home/obj/usr/src/amd64.amd64/tests/sys/netpfil/pf/ioctl/ validation.o --- validation.o --- In file included from /usr/src/tests/sys/netpfil/pf/ioctl/validation.c:35: /home/obj/usr/src/amd64.amd64/tmp/usr/include/net/pfvar.h:49:10: fatal error: 'netpfil/pf/pf.h' file not found #include ^ It should be fully fixed as of r332358. Thanks for the report. Regards, Kristof ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Odd ZFS boot module issue on r332158
On 10/04/2018 15:27, Andrew Gallatin wrote: > Is there something like tools/diag/prtblknos for ZFS? zdb. It has a manual page, but in the case like this you typically want to run zdb -d[d*] Add d-s until you get all the information you want. It looks like five d-s is needed to get individual blocks reported. -- Andriy Gapon ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Odd ZFS boot module issue on r332158
> On 10 Apr 2018, at 15:27, Andrew Gallatin wrote: > > On 04/09/18 23:33, Allan Jude wrote: >> On 2018-04-09 19:11, Andrew Gallatin wrote: >>> I updated my main amd64 workstation to r332158 from something much >>> earlier (mid Jan). >>> >>> Upon reboot, all seemed well. However, I later realized that the vmm.ko >>> module was not loaded at boot, because bhyve PCI passthru did not >>> work. My loader.conf looks like (I'm passing a USB interface through): >>> >>> ### >>> vmm_load="YES" >>> opensolaris_load="YES" >>> zfs_load="YES" >>> nvidia_load="YES" >>> nvidia-modeset_load="YES" >>> >>> # Tune ZFS Arc Size - Change to adjust memory used for disk cache >>> vfs.zfs.arc_max="4096M" >>> hint.xhci.2.disabled="1" >>> pptdevs="8/0/0" >>> hw.dmar.enable="0" >>> cuse_load="YES" >>> ### >>> >>> The problem seems "random". I rebooted into single-user to >>> see if somehow, vmm.ko was loaded at boot and something >>> was unloading vmm.ko. However, on this boot it was loaded. I then >>> ^D'ed and continued to multi-user, where X failed to start because >>> this time, the nvidia modules were not loaded. (but nvidia had >>> been loaded on the 1st boot). >>> >>> So it *seems* like different modules are randomly not loaded by the >>> loader, at boot. The ZFS config is: >>> >>> config: >>> >>> NAMESTATE READ WRITE CKSUM >>> tankONLINE 0 0 0 >>> mirror-0 ONLINE 0 0 0 >>> ada0p2 ONLINE 0 0 0 >>> da3p2 ONLINE 0 0 0 >>> mirror-1 ONLINE 0 0 0 >>> ada1p2 ONLINE 0 0 0 >>> da0p2 ONLINE 0 0 0 >>> cache >>> da2s1dONLINE 0 0 0 >>> >>> The data drives in the pool are all exactly like this: >>> >>> =>34 9767541101 ada0 GPT (4.5T) >>> 34 6- free - (3.0K) >>> 40 204800 1 efi (100M) >>> 204840 9763209216 2 freebsd-zfs (4.5T) >>> 9763414056 4096000 3 freebsd-swap (2.0G) >>> 9767510056 31079- free - (15M) >>> >>> >>> There is about 1.44T used in the pool. I have no idea >>> how ZFS mirrors work, but I'm wondering if somehow this >>> is a 2T problem, and there are issues with blocks on >>> difference sides of the mirror being across the 2T boundary. >>> >>> Sorry to be so vague.. but this is the one machine I *don't* have >>> a serial console on, so I don't have good logs. >>> >>> Drew >>> >>> ___ >>> freebsd-current@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-current >>> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" >> What makes you think it is related to ZFS? >> Are there any error messages when the nvidia module did not load? > > I think it is related to ZFS simply because I'm booting from ZFS and > it is not working reliably. Our systems at work, booting from UFS on > roughly the same svn rev seem to still load modules reliably from > the loader. I know there has been a lot of work on the loader > recently, and in a UEFE + UFS context, I've seen it fail to boot > the right partition, etc. However, I've never seen it fail to load > just some modules. The one difference between what I run at home > and what we run at work is ZFS vs UFS. > > Given that it is a glass console, I have no confidence in my ability > to log error messages. However, I could have sworn that I saw > something like "io error" when it failed to load vmm.ko > (I actually rebooted several times when I was diagnosing it.. > at first I thought xhci was holding on to the pass-thru device) > > I vaguely remembered reading something about this recently. > I just tracked it down to the "ZFS i/o error in recent 12.0" > thread from last month, and this message in particular: > > https://lists.freebsd.org/pipermail/freebsd-current/2018-March/068890.html > > I'm booting via UEFI into a ZFS system with a FS that > extends across 2TB.. > > Is there something like tools/diag/prtblknos for ZFS? > run zpool scrub first, however, if you were able to load that module manually from OS, there is no reason to suspect the zfs corruption. But if you really are getting IO errors, I would actually suspect that the firmware is is buggy and can not really read past 2TB, so the obvious second suggestion is to check for firmware update. The ZFS reader code does try all block copies before giving up on the block, so the third option you can test is: 1. reboot 2. press esc when the boot menu is up to get to OK prompt 3. enter: start this would load the configured files and you will get the error messages. Also once you have kernel loaded, you can try to load modules manually with load command. If still nothing, the only way to ensure your data is below 2TB line is to create separate parti
Re: Odd ZFS boot module issue on r332158
On 04/10/18 11:25, Andriy Gapon wrote: On 10/04/2018 15:27, Andrew Gallatin wrote: Is there something like tools/diag/prtblknos for ZFS? zdb. It has a manual page, but in the case like this you typically want to run zdb -d[d*] Add d-s until you get all the information you want. It looks like five d-s is needed to get individual blocks reported. Thanks for the instructions! How do I interpret this output: <3:45pm>viserion/gallatin:~>ls -li /boot//kernel/vmm.ko 231484 -r-xr-xr-x 1 root wheel 371168 Apr 7 15:05 /boot//kernel/vmm.ko* <3:46pm>viserion/gallatin:src>sudo zdb -d tank/ROOT/12.0-CURRENT-20180407.104009 231484 Dataset tank/ROOT/12.0-CURRENT-20180407.104009 [ZPL], ID 260, cr_txg 16768182, 19.9G, 292309 objects, rootbp DVA[0]=<0:4565c5a5000:1000> DVA[1]=<1:1f1ba52b000:1000> [L0 DMU objset] fletcher4 uncompressed LE contiguous unique double size=800L/800P birth=16821145L/16821145P fill=292309 cksum=f1c0930f9:12c49ea8e701:e036d6305e6ca:79eca698e0476d9 Object lvl iblk dblk dsize lsize %full type 2314842 128K 128K 392K 384K 100.00 ZFS plain file 168 bonus System attributes dnode flags: USED_BYTES USERUSED_ACCOUNTED dnode maxblkid: 2 path/boot/kernel/vmm.ko uid 0 gid 0 atime Sat Apr 7 17:06:06 2018 mtime Sat Apr 7 15:05:48 2018 ctime Sat Apr 7 15:05:48 2018 crtime Sat Apr 7 14:08:21 2018 gen 16768195 mode100555 size371168 parent 229378 links 1 pflags 4080104 Indirect blocks: 0 L1 1:1f01016c000:1000 2L/1000P F=3 B=16769122/16769122 0 L0 1:1f00f9e3000:2 2L/2P F=1 B=16769122/16769122 2 L0 1:1f00fa03000:2 2L/2P F=1 B=16769122/16769122 4 L0 1:1f00fa23000:2 2L/2P F=1 B=16769122/16769122 segment [, 0006) size 384K Thanks, Drew ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Odd ZFS boot module issue on r332158
On 10/04/2018 22:48, Andrew Gallatin wrote: > On 04/10/18 11:25, Andriy Gapon wrote: >> On 10/04/2018 15:27, Andrew Gallatin wrote: >>> Is there something like tools/diag/prtblknos for ZFS? >> >> zdb. >> >> It has a manual page, but in the case like this you typically want to run >> zdb -d[d*] >> Add d-s until you get all the information you want. >> >> It looks like five d-s is needed to get individual blocks reported. >> > > Thanks for the instructions! > > How do I interpret this output: [snip] > 0 L1 1:1f01016c000:1000 2L/1000P F=3 B=16769122/16769122 > 0 L0 1:1f00f9e3000:2 2L/2P F=1 B=16769122/16769122 > 2 L0 1:1f00fa03000:2 2L/2P F=1 B=16769122/16769122 > 4 L0 1:1f00fa23000:2 2L/2P F=1 B=16769122/16769122 The first number is an offset within the file (hex); Lx is a block level where L0 is a data block, L1 is an indirect block just above data blocks, etc; x:y:z is a (top-level) vdev number, a block offset on disk (hex) and a block size on disk(hex); the rest is not as important. The quoted offsets appear to be just below 2TB. -- Andriy Gapon ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Module compiles looking in /usr/src when alternate src tree is in use
Rodney W. Grimes wrote: > I am having a compile time issue for a patched that compiled fine on my > r329294 system, but now failes to compile with what looks like a wrong > header being included. You may find it helpful to do something like: make -dv -C sys/modules/vmm -V CFLAGS > /tmp/dvo 2>&1 egrep ':.PARSE|/usr/src/sys' /tmp/dvo | grep -B1 usr/src | more The arg to -V doesn't matter btw. You could narrow that down if you know what var -I/usr/src/sys is in (probably CFLAGS but you never know) the above should help find the makefile that is introducing the bogus -I > > > I have wrapped the long line so I can point to a difference between > r329294 and r332262 make log of this file. > > r329294 make output: > > cc -O2 -pipe -DVMM_KEEP_STATS -DSMP -fno-strict-aliasing -Werror -D_KERNEL \ > -DKLD_MODULE -nostdinc -I/usr/src-topo/sys/amd64/vmm \ > -I/usr/src-topo/sys/amd64/vmm/io -I/usr/src-topo/sys/amd64/vmm/intel \ > -I/usr/src-topo/sys/amd64/vmm/amd -I. -I/usr/src-topo/sys -fno-common \ >^ this is what I would > expect ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Module compiles looking in /usr/src when alternate src tree is in use
> Rodney W. Grimes wrote:
>
> > I am having a compile time issue for a patched that compiled fine on my
> > r329294 system, but now failes to compile with what looks like a wrong
> > header being included.
>
> You may find it helpful to do something like:
>
> make -dv -C sys/modules/vmm -V CFLAGS > /tmp/dvo 2>&1
> egrep ':.PARSE|/usr/src/sys' /tmp/dvo | grep -B1 usr/src | more
>
> The arg to -V doesn't matter btw.
> You could narrow that down if you know what var -I/usr/src/sys is in
> (probably CFLAGS but you never know)
> the above should help find the makefile that is introducing the bogus -I
>
Thank you, that does help narrow it down: (I backed up a vew lines
from the first place I saw src/.)
...
Global:.PARSEFILE = bsd.kmod.mk
Global:.PARSEDIR = /usr/src-topo/share/mk
Global:.PARSEFILE = bsd.kmod.mk
Result[] of :U is "/usr/src/sys"
Result[] of :U is "/usr/src/sys"
Global:SYSDIR = ${:U/usr/src/sys:tA}
Global:.PARSEDIR = /usr/src-topo/share/mk
Global:.PARSEFILE = bsd.kmod.mk
Result[] of :U is "/usr/src/sys"
Applying[] :t to "/usr/src/sys"
Result[] of :t is "/usr/src/sys"
Result[] of :U is "/usr/src/sys"
Applying[] :t to "/usr/src/sys"
Result[] of :t is "/usr/src/sys"
Result[] of :U is "/usr/src/sys"
Applying[] :t to "/usr/src/sys"
Result[] of :t is "/usr/src/sys"
Global:.MAKE.MAKEFILES = /usr/src-topo/share/mk/sys.mk
/usr/src-topo/share/mk/local.sys.env.mk /usr/src-topo/share/mk/src.sys.env.mk
/usr/src-topo/share/mk/bsd.mkopt.m
k /usr/src-topo/share/mk/src.sys.obj.mk /usr/src-topo/share/mk/auto.obj.mk
/usr/src-topo/share/mk/bsd.suffixes.mk /usr/src-topo/share/mk/local.sys.mk
/usr/src-topo/sha
re/mk/src.sys.mk /usr/src-topo/sys/modules/vmm/Makefile
/usr/src-topo/share/mk/bsd.kmod.mk /usr/src/sys/conf/kmod.mk
^
Thats gona bust something some day
Global:.PARSEDIR = /usr/src/sys/conf
Global:.PARSEFILE = kmod.mk
Global:.INCLUDEDFROMDIR = /usr/src/sys/conf
Oh my! Ug
So something in bsd.kmod.mk is going very wrong... it looks like it
starts to pull all sorts of stuff from /usr/src/sys!
> >
> > I have wrapped the long line so I can point to a difference between
> > r329294 and r332262 make log of this file.
> >
> > r329294 make output:
> >
> > cc -O2 -pipe -DVMM_KEEP_STATS -DSMP -fno-strict-aliasing -Werror
> > -D_KERNEL \
> > -DKLD_MODULE -nostdinc -I/usr/src-topo/sys/amd64/vmm \
> > -I/usr/src-topo/sys/amd64/vmm/io -I/usr/src-topo/sys/amd64/vmm/intel \
> > -I/usr/src-topo/sys/amd64/vmm/amd -I. -I/usr/src-topo/sys -fno-common \
> >^ this is what I
> > would expect
>
>
--
Rod Grimes rgri...@freebsd.org
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Module compiles looking in /usr/src when alternate src tree is in use
Rodney W. Grimes wrote:
> Thank you, that does help narrow it down: (I backed up a vew lines
> from the first place I saw src/.)
>
> ...
> Global:.PARSEFILE = bsd.kmod.mk
> Global:.PARSEDIR = /usr/src-topo/share/mk
> Global:.PARSEFILE = bsd.kmod.mk
> Result[] of :U is "/usr/src/sys"
> Result[] of :U is "/usr/src/sys"
> Global:SYSDIR = ${:U/usr/src/sys:tA}
That's from the .for loop in bsd.kmod.mk which is clearly inadequate.
Since the tree for some time now, reliably sets SRCTOP
that list should start with ${SRCTOP}/sys
or ${SRCTOP:Uno}/sys just to be sure you ignore it if somehow unset.
> So something in bsd.kmod.mk is going very wrong... it looks like it
> starts to pull all sorts of stuff from /usr/src/sys!
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
