On 6/19/2013 20:35, Jeremy Chadwick wrote:
Nope, I see basically the same thing sometimes under ESXi 5.0
Hypervisor (and yes it worries me the implications of something so
broad). Those unites I just haven't been able to isolate on a
server which isn't critical. Lets focus on this server for now
though per your suggestion below.
I'm sorry but I don't understand your first sentence -- the first part
of your sentence says "nope" (I have to assume in reply to my "on bare
metal" part), but then says "I see basically the same thing sometimes
under ESXi" which implies an alternate environment in comparison (i.e.
we *are* talking about bare metal). Consider me confused. :-)
Basically: The issue is extremely similar if not the same root cause, be
it a native or virtual server. This server though is native.
2. We need to know what version of "9.1" you're using, i.e. 9.1-RELEASE.
If you use stable/9 (RELENG_9) we need to see uname -a output (you can
hide the machine name if you want).
Sorry, this ZFS box is 9.1-R P4 (kernel built today):
FreeBSD ilos.dsn 9.1-RELEASE-p4 FreeBSD 9.1-RELEASE-p4 #6: Wed Jun
19 15:31:12 ICT 2013
root@hostname:/usr/obj/usr/src/sys/ATEAMSYSTEMS amd64
I suggest trying stable/9 (and staying with it, for that matter).
The issue is no binary updates and we have a large deploy base, so we've
stuck with -R and use it internally because it's what we deploy.
3. Can we please have dmesg from this machine? The controller and some
other hardware details matter.
Sure take a look at the full log here: http://pastebin.com/k55gVVuU
This includes a boot, then a reboot as I describe (you can see it
logs the All Buffers Synced, etc) then powering back on.
Thanks. I was mainly interested in the storage controller being used
(in this case ahci(4)) and the disks being used (notorious ST3000DM001,
known for excessively parking heads).
Yeah, was not my first choice but then again ... RAIDZ-2 :) HD supply
chain here (Thailand) is weird considering how many are made here (and
can't buy). Smartd screams about them possibly needing a firmware
update (they don't according to Seagate). Had no issues aside from a
failure a month or so again (it's an HD ... it happens).
AFAIK this isn't one of the
controllers that was known for weird "quirky issues" pertaining to
flushing data to disk on shutdown.
I have to ask: is this FreeBSD box running under a HV?
No, native/direct for sure on this one.
If it *is not* running under a HV, could we please get exact motherboard
model and version (including BIOS version)? Sometimes (not always) you
can get this from "kenv | grep smbios."
No problem I built this one personally:
Asus P8B-X BIOS revision 6103
I can also see you're running your own kernel. We'll get to that in a
moment.
It's GENERIC with the following added to the end:
# -- Add Support for nicer console
#
options VESA
options SC_PIXEL_MODE
# -- PF Support
#
device pf
device pflog
device pfsync
# -- Core temperature reporting
#
device coretemp # For Intel CPUs
device smbios
4. Does "sysctl hw.usb.no_shutdown_wait=1" help you?
Weirdly this allowed it to reboot on the first try (without needing
to be reset), but not the second.
I'm not surprised. Pleas re-try with stable/9; Hans has been constantly
working on the USB stack and fixing major bugs.
Got it but probably not going to go this route as it means no more
binary upgrades. While I can reboot it, it is the office NAS here and
so 'testing out' -STABLE I think probably isn't going to happen.
The "Starting background file
system checks in 60 seconds" message appeared ... that only happens
when something is dirty, right?
No it does not. That message is always printed when you use background
fsck, which is the default.
Got it.
I do not advocate using background fsck, because it has been known (and
may still do this -- I do not care to find out, I do not have time for
unreliable filesystem nonsense) to not always fix all filesystem
problems. Meaning: people using background fsck have been known to boot
into single-user and issue "fsck" manually and find issues.
Place background_fsck="no" in /etc/rc.conf. If the machine does not
have a clean filesystem on boot-up, you'll know because the system will
immediately begin fsck (in the foreground actively). You'll recognise
that output if it happens, trust me.
Preaching to the choir, we set this on all servers this one somehow did
not have it set (I think due to ZFS making it unique and not copying our
rc.conf template over properly).
So the second try with just this I could ctrl alt del it and it
responded .. kind of:
http://i.imgur.com/POAIaNg.jpg
Still had to reset it though.
This looks like a chicken-and-egg problem -- you're probably fighting
with background fsck, as the message there indicate "some processes
would not die". I'm just taking a guess though.
Yeah. Even with no background fsck though I still see the issue
(rebooted 4-5 times testing things further).
I even rolled back to the -p3 kernel because this server was not doing
this a month or two ago, and definitely not when it was put into
production. I only just saw/noticed it doing this when rebooting for p4.
I am now going to ask you for more information:
1. "gpart show -p xxx" where xxx is each disk you have in the system
>>>> gpart show -p
=> 34 5860533101 ada0 GPT (2.7T)
34 6 - free - (3.0k)
40 256 ada0p1 freebsd-boot (128k)
296 1752 - free - (876k)
2048 5860530176 ada0p2 freebsd-zfs (2.7T)
5860532224 911 - free - (455k)
=> 34 5860533101 ada1 GPT (2.7T)
34 6 - free - (3.0k)
40 256 ada1p1 freebsd-boot (128k)
296 1752 - free - (876k)
2048 5860530176 ada1p2 freebsd-zfs (2.7T)
5860532224 911 - free - (455k)
=> 34 5860533101 ada2 GPT (2.7T)
34 6 - free - (3.0k)
40 256 ada2p1 freebsd-boot (128k)
296 1752 - free - (876k)
2048 5860530176 ada2p2 freebsd-zfs (2.7T)
5860532224 911 - free - (455k)
=> 34 5860533101 ada3 GPT (2.7T)
34 6 - free - (3.0k)
40 256 ada3p1 freebsd-boot (128k)
296 1752 - free - (876k)
2048 5860530176 ada3p2 freebsd-zfs (2.7T)
5860532224 911 - free - (455k)
=> 34 5860533101 ada4 GPT (2.7T)
34 6 - free - (3.0k)
40 256 ada4p1 freebsd-boot (128k)
296 1752 - free - (876k)
2048 5860530176 ada4p2 freebsd-zfs (2.7T)
5860532224 911 - free - (455k)
=> 34 5860533101 ada5 GPT (2.7T)
34 6 - free - (3.0k)
40 256 ada5p1 freebsd-boot (128k)
296 1752 - free - (876k)
2048 5860530176 ada5p2 freebsd-zfs (2.7T)
5860532224 911 - free - (455k)
Here also with labels so you get an idea of what the scheme here is:
>>>> gpart show -p -l
=> 34 5860533101 ada0 GPT (2.7T)
34 6 - free - (3.0k)
40 256 ada0p1 boot0 (128k)
296 1752 - free - (876k)
2048 5860530176 ada0p2 root0 (2.7T)
5860532224 911 - free - (455k)
=> 34 5860533101 ada1 GPT (2.7T)
34 6 - free - (3.0k)
40 256 ada1p1 boot1 (128k)
296 1752 - free - (876k)
2048 5860530176 ada1p2 root1 (2.7T)
5860532224 911 - free - (455k)
=> 34 5860533101 ada2 GPT (2.7T)
34 6 - free - (3.0k)
40 256 ada2p1 boot2 (128k)
296 1752 - free - (876k)
2048 5860530176 ada2p2 root2 (2.7T)
5860532224 911 - free - (455k)
=> 34 5860533101 ada3 GPT (2.7T)
34 6 - free - (3.0k)
40 256 ada3p1 boot3 (128k)
296 1752 - free - (876k)
2048 5860530176 ada3p2 root3 (2.7T)
5860532224 911 - free - (455k)
=> 34 5860533101 ada4 GPT (2.7T)
34 6 - free - (3.0k)
40 256 ada4p1 boot4 (128k)
296 1752 - free - (876k)
2048 5860530176 ada4p2 root4 (2.7T)
5860532224 911 - free - (455k)
=> 34 5860533101 ada5 GPT (2.7T)
34 6 - free - (3.0k)
40 256 ada5p1 boot5 (128k)
296 1752 - free - (876k)
2048 5860530176 ada5p2 root5 (2.7T)
5860532224 911 - free - (455k)
2. gmirror list
>>>> gmirror list
Geom name: boot
State: COMPLETE
Components: 6
Balance: load
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 3404031251
Providers:
1. Name: mirror/boot
Mediasize: 130560 (127k)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
Consumers:
1. Name: ada0p1
Mediasize: 131072 (128k)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: NONE
GenID: 0
SyncID: 1
ID: 3045005471
2. Name: ada1p1
Mediasize: 131072 (128k)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
State: ACTIVE
Priority: 1
Flags: NONE
GenID: 0
SyncID: 1
ID: 1958473873
3. Name: ada2p1
Mediasize: 131072 (128k)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
State: ACTIVE
Priority: 2
Flags: NONE
GenID: 0
SyncID: 1
ID: 1208229558
4. Name: ada3p1
Mediasize: 131072 (128k)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
State: ACTIVE
Priority: 3
Flags: NONE
GenID: 0
SyncID: 1
ID: 3928010527
5. Name: ada4p1
Mediasize: 131072 (128k)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
State: ACTIVE
Priority: 4
Flags: NONE
GenID: 0
SyncID: 1
ID: 442340132
6. Name: ada5p1
Mediasize: 131072 (128k)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: NONE
GenID: 0
SyncID: 1
ID: 1281187492
3. Any/all details of your gmirror setup or other things you can
think of when you set it up
The only thing is that we use GMIRROR on the partition level because we
use GPT (which is clear from the gpart output I think). I gmirror the
boot partition only in this case as I use ZFS backed swap and ZFS root
for this server.
4. Contents of /etc/fstab
>>>> cat /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
# NOTE: ZFS root is not managed here
/dev/zvol/zroot/swap none swap sw 0 0
5. Contents of /boot/loader.conf
>>>> cat /boot/loader.conf
geom_mirror_load="YES"
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"
aio_load="YES"
if_lagg_load="YES"
6. Contents of /etc/rc.conf
# ---- Don't run FS check and let apps start
#
fsck_y_enable="YES"
background_fsck="NO"
# ---- Power management enables SpeedStep and TurboBoost
#
powerd_enable="YES"
powerd_flags="-a hiadaptive"
# ---- Networking
#
hostname="hostname"
defaultrouter="xxx.xxx.xxx.3"
# -- LACP
ifconfig_em0="up"
ifconfig_em1="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 xxx.xxx.xxx.212/24"
# ---- Services
#
sshd_enable="YES"
smartd_enable="YES"
samba_enable="YES"
zabbix_agentd_enable="YES"
zfs_enable="YES"
apcupsd_enable="YES"
slapd_enable="YES"
slapd_flags='-h "ldapi://%2fvar%2frun%2fopenldap%2fldapi/
ldap://xxx.xxx.xxx.212/ ldap://127.0.0.1/";'
slapd_sockets="/var/run/openldap/ldapi"
# ---- Time Stuff
#
ntpd_enable="YES"
ntpd_sync_on_start="YES"
# ---- Mail
#
postfix_enable="YES"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
7. Contents of /etc/sysctl.conf
kern.maxfiles=25600
kern.maxfilesperproc=16384
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536
8. Contents of /sys/amd64/conf/ATEAMSYSTEMS
See above
5. Does "sysctl hw.acpi.handle_reboot=1" help you?
No change, still responded to a ctrl alt del like above, but like
that still needs to be reset and comes back dirty.
6. Does "sysctl hw.acpi.disable_on_reboot=1" help you?
No change. Same as above, ctrl alt del responds but needs a hard
reset still.
Okay, thank you.
7. If none of the above helps, can you please boot verbose mode and then
when the system "locks up" on "shutdown -r now" take a picture of the
VGA console?
Lots of debug on boot obviously but not much different on shutdown/hang:
http://i.imgur.com/SgzSsoP.jpg
It looks to me like the ACPI layer is still actively working at the time
"all buffers are synced", meaning the actual reboot phase itself never
happens. This to me starts to smell of an ACPI problem, but I do not
have the skill set to debug this, and I'm also grasping at straws.
There are many things that happen during that phase of operation,
particularly the "USB shutdown" phase.
Yeah. Originally I had even my UPS (APC) disconnected, the only USB
device (via a port -- I realize there might be MB virtual ports) was a
Dell KB.
But it all depends on your kernel config, which I've now asked for.
Yeah
--
Adam Strohl
http://www.ateamsystems.com/
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
