Re: help with crash dump

[Igor V. Ruzanov]

What kernel modules do you load with this kernel? (show the list via 
kldstat)


|Randomly every few weeks / months one of the servers I manage gets a panic. It 
doesn't happen on the same server every time.
|
|I run FreeBSD stable compiled on 2 September 2019 but it did happen with 
previous versions too and if I remember correctly it did happen with FreeBSD 11 
too.
|
|I don't have /usr/lib/debug/kernel files (I delete them to save some space) 
but I can recompile a new kernel / userland if it's needed.
|
|Any idea what the issue is?
|
|kgdb /boot/kernel/kernel ./vmcore.0
|GNU gdb (GDB) 8.3 [GDB v8.3 for FreeBSD]
|Copyright (C) 2019 Free Software Foundation, Inc.
|License GPLv3+: GNU GPL version 3 or later 
|This is free software: you are free to change and redistribute it.
|There is NO WARRANTY, to the extent permitted by law.
|Type "show copying" and "show warranty" for details.
|This GDB was configured as "x86_64-portbld-freebsd12.0".
|Type "show configuration" for configuration details.
|For bug reporting instructions, please see:
|.
|Find the GDB manual and other documentation resources online at:
|.
|
|For help, type "help".
|Type "apropos word" to search for commands related to "word"...
|Reading symbols from /boot/kernel/kernel...
|(No debugging symbols found in /boot/kernel/kernel)
|0x80bff56a in sched_switch ()
|(kgdb) bt
|#0  0x80bff56a in sched_switch ()
|#1  0x80bd9be2 in mi_switch ()
|#2  0x80c29175 in sleepq_catch_signals ()
|#3  0x80c28caf in sleepq_wait_sig ()
|#4  0x80bd960a in _sleep ()
|#5  0x80c64e02 in solisten_dequeue ()
|#6  0x80c6c763 in kern_accept4 ()
|#7  0x80c6ca40 in accept1 ()
|#8  0x810a48b4 in amd64_syscall ()
|#9  
|#10 0x000800ad68ca in ?? ()
|Backtrace stopped: Cannot access memory at address 0x7ffee648
|___
|freebsd-questi...@freebsd.org mailing list
|https://lists.freebsd.org/mailman/listinfo/freebsd-questions
|To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
|
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: help with crash dump

[Christos Chatzaras]

Id Refs AddressSize Name
 1   12 0x8020  2446480 kernel
 21 0x82648000281e0 geom_mirror.ko
 31 0x82a21000 88e8 tmpfs.ko
 41 0x82a2a000258f8 ipfw.ko
 51 0x82a5  acf mac_ntpd.ko
 61 0x82a51000 2940 nullfs.ko




> On 24 Sep 2019, at 11:15, Igor V. Ruzanov  wrote:
> 
> What kernel modules do you load with this kernel? (show the list via 
> kldstat)

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: help with crash dump

[Igor V. Ruzanov]

Is your kernel custom and you were compile it? If yes, could you please 
send kernel config to here?

|Id Refs AddressSize Name
| 1   12 0x8020  2446480 kernel
| 21 0x82648000281e0 geom_mirror.ko
| 31 0x82a21000 88e8 tmpfs.ko
| 41 0x82a2a000258f8 ipfw.ko
| 51 0x82a5  acf mac_ntpd.ko
| 61 0x82a51000 2940 nullfs.ko
|
|
|
|
|> On 24 Sep 2019, at 11:15, Igor V. Ruzanov  wrote:
|> 
|> What kernel modules do you load with this kernel? (show the list via 
|> kldstat)
|
|___
|freebsd-questi...@freebsd.org mailing list
|https://lists.freebsd.org/mailman/listinfo/freebsd-questions
|To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
|
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: help with crash dump

[Christos Chatzaras]

> On 24 Sep 2019, at 12:43, Igor V. Ruzanov  wrote:
> 
> Is your kernel custom and you were compile it? If yes, could you please 
> send kernel config to here?


The kernel is from 12.0-STABLE r351639.

I use the GENERIC config to compile the kernel but with custom src.conf:

cat /etc/src.conf

WITHOUT_AUTHPF=yes
WITHOUT_AUTOFS=yes
WITHOUT_BHYVE=yes
WITHOUT_BLUETOOTH=yes
WITHOUT_BOOTPARAMD=yes
WITHOUT_BOOTPD=yes
WITHOUT_BSDINSTALL=yes
WITHOUT_CCD=yes
WITHOUT_CTM=yes
WITHOUT_CUSE=yes
WITHOUT_CXGBETOOL=yes
WITHOUT_DEBUG_FILES=yes
WITHOUT_FLOPPY=yes
WITHOUT_FREEBSD_UPDATE=yes
WITHOUT_GCOV=yes
WITHOUT_HAST=yes
WITHOUT_HTML=yes
WITHOUT_HYPERV=yes
WITHOUT_I4B=yes
WITHOUT_IPFILTER=yes
WITHOUT_LPR=yes
WITHOUT_MLX5TOOL=yes
WITHOUT_MODULE_DRM=yes
WITHOUT_MODULE_DRM2=yes
WITHOUT_NCP=yes
WITHOUT_NDIS=yes
WITHOUT_OBJC=yes
WITHOUT_PF=yes
WITHOUT_PMC=yes
WITHOUT_PPP=yes
WITHOUT_PROFILE=yes
WITHOUT_RADIUS_SUPPORT=yes
WITHOUT_RBOOTD=yes
WITHOUT_REPRODUCIBLE_BUILD=yes
WITHOUT_SHAREDOCS=yes
WITHOUT_SYSINSTALL=yes
WITHOUT_TESTS=yes
WITHOUT_TFTP=yes
WITHOUT_TIMED=yes
WITHOUT_USB_GADGET_EXAMPLES=yes
WITHOUT_WIRELESS=yes
WITHOUT_WPA_SUPPLICANT_EAPOL=yes
WITHOUT_ZFS=yes
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: help with crash dump

[Igor V. Ruzanov]

Please try to remove 'WITHOUT_REPRODUCIBLE_BUILD' option and recompile the 
kernel to see if it will be stable in your context. The modern FreeBSD 
releses (12.0 for example) is with reproducible build enabled now. As quck 
check, tell uname -a and you will give the following output:
FreeBSD xxx.yourdomain.tld 12.0-RELEASE-p10 FreeBSD 12.0-RELEASE-p10 #0: 
Mon Oct 24 18:49:24 UTC 2016 
r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

And without WITHOUT_REPRODUCIBLE_BUILD you should see the ident: FreeBSD 
xxx.yourdomin.tld 12.0-RELEASE-p10 FreeBSD 12.0-RELEASE-p10 GENERIC  
amd64


|
|> On 24 Sep 2019, at 12:43, Igor V. Ruzanov  wrote:
|> 
|> Is your kernel custom and you were compile it? If yes, could you please 
|> send kernel config to here?
|
|
|The kernel is from 12.0-STABLE r351639.
|
|I use the GENERIC config to compile the kernel but with custom src.conf:
|
|cat /etc/src.conf
|
|WITHOUT_AUTHPF=yes
|WITHOUT_AUTOFS=yes
|WITHOUT_BHYVE=yes
|WITHOUT_BLUETOOTH=yes
|WITHOUT_BOOTPARAMD=yes
|WITHOUT_BOOTPD=yes
|WITHOUT_BSDINSTALL=yes
|WITHOUT_CCD=yes
|WITHOUT_CTM=yes
|WITHOUT_CUSE=yes
|WITHOUT_CXGBETOOL=yes
|WITHOUT_DEBUG_FILES=yes
|WITHOUT_FLOPPY=yes
|WITHOUT_FREEBSD_UPDATE=yes
|WITHOUT_GCOV=yes
|WITHOUT_HAST=yes
|WITHOUT_HTML=yes
|WITHOUT_HYPERV=yes
|WITHOUT_I4B=yes
|WITHOUT_IPFILTER=yes
|WITHOUT_LPR=yes
|WITHOUT_MLX5TOOL=yes
|WITHOUT_MODULE_DRM=yes
|WITHOUT_MODULE_DRM2=yes
|WITHOUT_NCP=yes
|WITHOUT_NDIS=yes
|WITHOUT_OBJC=yes
|WITHOUT_PF=yes
|WITHOUT_PMC=yes
|WITHOUT_PPP=yes
|WITHOUT_PROFILE=yes
|WITHOUT_RADIUS_SUPPORT=yes
|WITHOUT_RBOOTD=yes
|WITHOUT_REPRODUCIBLE_BUILD=yes
|WITHOUT_SHAREDOCS=yes
|WITHOUT_SYSINSTALL=yes
|WITHOUT_TESTS=yes
|WITHOUT_TFTP=yes
|WITHOUT_TIMED=yes
|WITHOUT_USB_GADGET_EXAMPLES=yes
|WITHOUT_WIRELESS=yes
|WITHOUT_WPA_SUPPLICANT_EAPOL=yes
|WITHOUT_ZFS=yes
|___
|freebsd-questi...@freebsd.org mailing list
|https://lists.freebsd.org/mailman/listinfo/freebsd-questions
|To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
|
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Question about bottle neck in storage

[John Fleming]

Is there anyway to see how busy a SAS/Sata controller is vs disks? I
have a R720 with 14 Samsung 860 EVOs in it (its a lab server) in raid
10 ZFS.

When firing off a dd I (bs=1G count=10) seems like the disks never go
above %50 busy. I'm trying to figure out if i'm maxing out SATA 3 BW
or if its something else (like terrible dd options).

my setup is Dell R720 with 2 x LSI 9361 cards. Each card is going to a
dedicated 8 drive board inside the front of the R720. Basically i'm
just saying its not a single SAS cable to 14 drives.

Don't have cpu info hand.. zeon something. DDR3-1600 (128GB)

Both controllers are in 8x slots running PCIe gen 3.

BTW i'm sure this has been asked a million times but what would be
some decent benchmark tests while i'm at it?
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Question about bottle neck in storage

[Pete Wright]

On 9/24/19 8:45 AM, John Fleming wrote:

Is there anyway to see how busy a SAS/Sata controller is vs disks? I
have a R720 with 14 Samsung 860 EVOs in it (its a lab server) in raid
10 ZFS.

When firing off a dd I (bs=1G count=10) seems like the disks never go
above %50 busy. I'm trying to figure out if i'm maxing out SATA 3 BW
or if its something else (like terrible dd options).

my setup is Dell R720 with 2 x LSI 9361 cards. Each card is going to a
dedicated 8 drive board inside the front of the R720. Basically i'm
just saying its not a single SAS cable to 14 drives.

Don't have cpu info hand.. zeon something. DDR3-1600 (128GB)

Both controllers are in 8x slots running PCIe gen 3.


might want to take a look at sysutils/intel-pcm 
(https://github.com/opcm/pcm).  I *think* this should give you metrics 
on PCIe bus utilization among other useful status.


Also, lookup the bandwidth for the PCIe bus and see if your aggregate 
disk throughput on one of the PCIe lanes is saturating the bus (pcm 
should also help here).  You can also run "zpool iostat -v 2" to see per 
disk i/o metrics to help determine if this is an issue.




BTW i'm sure this has been asked a million times but what would be
some decent benchmark tests while i'm at it?


I generally run several tests and then compare results, for example 
bonnie++, iozone, iperf (writing over the wire and to disk) as well as 
some more realistic scripts based on the use-case i'm building a 
solution for.  hope that helps.


-pete

--
Pete Wright
p...@nomadlogic.org
@nomadlogicLA

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Question about bottle neck in storage

[John Fleming]

On Tue, Sep 24, 2019 at 12:05 PM Pete Wright  wrote:
>
>
>
> On 9/24/19 8:45 AM, John Fleming wrote:
> > Is there anyway to see how busy a SAS/Sata controller is vs disks? I
> > have a R720 with 14 Samsung 860 EVOs in it (its a lab server) in raid
> > 10 ZFS.
> >
> > When firing off a dd I (bs=1G count=10) seems like the disks never go
> > above %50 busy. I'm trying to figure out if i'm maxing out SATA 3 BW
> > or if its something else (like terrible dd options).
> >
> > my setup is Dell R720 with 2 x LSI 9361 cards. Each card is going to a
> > dedicated 8 drive board inside the front of the R720. Basically i'm
> > just saying its not a single SAS cable to 14 drives.
> >
> > Don't have cpu info hand.. zeon something. DDR3-1600 (128GB)
> >
> > Both controllers are in 8x slots running PCIe gen 3.
>
> might want to take a look at sysutils/intel-pcm
> (https://github.com/opcm/pcm).  I *think* this should give you metrics
> on PCIe bus utilization among other useful status.
>

ok I can check that, but to be clear what i meant was like have i maxed out
SATA3 bw on the card not so much PCIe bw.

> Also, lookup the bandwidth for the PCIe bus and see if your aggregate
> disk throughput on one of the PCIe lanes is saturating the bus (pcm
> should also help here).  You can also run "zpool iostat -v 2" to see per
> disk i/o metrics to help determine if this is an issue.
>
I think i've looked at that before but i'll check again.

>
> > BTW i'm sure this has been asked a million times but what would be
> > some decent benchmark tests while i'm at it?
>
> I generally run several tests and then compare results, for example
> bonnie++, iozone, iperf (writing over the wire and to disk) as well as
> some more realistic scripts based on the use-case i'm building a
> solution for.  hope that helps.
>
I've done iperf just across network IO. I'm getting 4x25 Gb/sec
so i'm good there i think (Mellonox Connectx4 ethernet mode).

I'll poke around with bonnie++ and iozone. I used bonnie++ once but
didn't really
get what it was telling me but I really didn't put much into it.

Thanks!
> -pete
>
> --
> Pete Wright
> p...@nomadlogic.org
> @nomadlogicLA
>
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Question about bottle neck in storage

[Richard Mackerras]

I’d be interested to know what the actual throughput is you are getting.
Are the disks SATA 7200RPM? What speed is the disk interface?
Do you have just 2 disks or more?

Richard

[Richard Mackerras - Chat @ 
Spike](https://www.spikenow.com/?ref=spike-organic-signature&_ts=6bswo) 
[6bswo]

On September 24, 2019 at 16:30 GMT, Kevin P Neal  wrote:

On Tue, Sep 24, 2019 at 11:45:37AM -0400, John Fleming wrote:
> Don't have cpu info hand.. zeon something. DDR3-1600 (128GB)

See /var/run/dmesg.boot for the output of the dmesg command as of boot
time. It includes the type of CPU, the number of CPUs, the number of
threads, features, disabled features, and so on aside from the type
of memory. I don't think that's available in dmesg?
--
Kevin P. Neal http://www.pobox.com/~kpn/
"Oh, I've heard that paradox a couple of times, but there's something
about a cat dying and I hate to think of such things."
- Dr. Donald Knuth speaking of Schrodinger's cat, December 8, 1999, MIT
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Question about bottle neck in storage

[Warner Losh]

On Tue, Sep 24, 2019 at 5:46 PM John Fleming 
wrote:

> Is there anyway to see how busy a SAS/Sata controller is vs disks? I
> have a R720 with 14 Samsung 860 EVOs in it (its a lab server) in raid
> 10 ZFS.
>
> When firing off a dd I (bs=1G count=10) seems like the disks never go
> above %50 busy. I'm trying to figure out if i'm maxing out SATA 3 BW
> or if its something else (like terrible dd options).
>

Two points to consider here. First, NVMe has lots of queues and needs lots
of concurrent transactions to saturate, so the 50% busy means you are no
where close to saturating the drives. Schedule more I/O too fix that. It's
better to do lots and lots of concurrent DD to different parts of the
drive, or to use fio with the aio kernel option and posixaio I/O scheduling
method.

I use the following script, but often need to increase the number of
threads / jobs to saturate.

; SSD testing: 128k I/O 64 jobs 32 deep queue

[global]
direct=1
rw=randread
refill_buffers
norandommap
randrepeat=0
bs=128k
ioengine=posixaio
iodepth=32
numjobs=64
runtime=60
group_reporting
thread

[ssd128k]

Second, the system's % busy statistics are misleading. They are the %of the
time that a command is outstanding on the drive. 100% busy can be a tiny
percentage of the total bandwidth you can get from the drive.


> my setup is Dell R720 with 2 x LSI 9361 cards. Each card is going to a
> dedicated 8 drive board inside the front of the R720. Basically i'm
> just saying its not a single SAS cable to 14 drives.
>
> Don't have cpu info hand.. zeon something. DDR3-1600 (128GB)
>
> Both controllers are in 8x slots running PCIe gen 3.
>
> BTW i'm sure this has been asked a million times but what would be
> some decent benchmark tests while i'm at it?
>

See above... :)

Warner
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Missing 12 MFCs

[Ed Maste]

On Thu, 19 Sep 2019 at 18:30, David Cross  wrote:
>
> There are a set of commits in head that have not been pulled into the
> stable/12 branch.  Many of these are marked as MFC (and are quite old).

I'm looking at the WITH_PIE / WITH_BIND_NOW changes, and am looking
for re@'s opinion. I'll merge them to stable/12 soon if they're
destined for 12.1, otherwise I will merge them shortly after the
release is done.

> EFI Related:
>
> r344839

I've asked if bcran can take care of this one.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"