Hi,
Here are three logs from the last server watchdog-driven reboot on friday
evening (not that I want you to actually dig into them, it's just to update
this thread with my new findings), with SBD watchdog timeout set to 20
seconds.
1) sar.txt is the output of sar -d -p- 2 (two seconds frequency of disk
statistics pretty printed), starting right before the reboot
2) messages.txt is an extract of the server /var/log/messages starting right
before the reboot, with QLogic driver, scsi layer and SBD verbose loggings
enabled
3) cpu1.txt is the output of sar -P ALL -2 (two seconds frequency of cpu
statistics), filtered by cpu #1, starting right before the reboot
sda is the local drive, sdb and sdc are the same single SAN LUN as seen by
the two FC ports of the server, san is the LUN multipath alias, san_part1 is
the SBD partition, san_part2 is the Oracle partition.
sar.txt shows that somewhere between 17:46.44 and 17:46.46 all reads and
writes to/from the san LUN drops to zero, for both SBD and Oracle
partitions, right until the 17th second of the SBD countdown, at which time
something (3.88 wr/s) seems to get written on the Oracle partition.
%util jumps to 100% as it does iowait%, from cpu1.txt, on 3 of the 24 cpu
cores this server has got (the ones Oracle and SBD were using at the time, I
suppose).
messages.txt shows at 17:46.44 this QLogic driver message that is different
from the rest og QLogic messages:
May 3 17:46:44 server1 kernel: [66588.156113] qla2xxx
[0000:11:00.1]-5816:2: Discard RND Frame -- 1006 02c1 0000.
By the time I started facing these problems, I got gigs of /var/log/messages
from these servers now, and the QLogic driver will write some rare "dropped
frame(s) detected" from time to time during normal server operations, but it
will never write this "Discard RND Frame" message unless there's going to be
an unwanted reboot right after. No scsi layer read and write communication
on sdb and sdc gets recorded by the kernel afterwards, except for a couple
of "device ready" commands. All these info have been shared with the SAN
department already.
Yesterday the SAN department has made a parameter configuration change on
the two Brocade switches (and multipath worked smoothlessly on the servers,
switching paths back and forth as the relative switches got restarted) I
hope this fixes the problem, otherwise we might investigate the switch port
configuration change described in the following link, as our current
configuration seems to apply (8Gb fc, Brocade switches, lots of er_bad_os
port errors, fill word port mode currently set to 1, and random server
problem)
http://loopbackconnector.com/2013/02/14/brocade-8-gb-how-to-talk-when-idle-p
ortcfgfillword/
andrea
----------------------------------------------------------------------
Message: 1
Date: Fri, 3 May 2013 10:17:12 +0200
From: Lars Marowsky-Bree <l...@suse.com>
To: The Pacemaker cluster resource manager
<pacemaker@oss.clusterlabs.org>
Subject: Re: [Pacemaker] Frequent SBD triggered server reboots
Message-ID: <20130503081712.ge3...@suse.de>
Content-Type: text/plain; charset=iso-8859-1
On 2013-05-03T02:49:54, andrea cuozzo <andrea.cuo...@sysma.it> wrote:
> Unfortunately Os and SP version for the Oracle project these clusters
> belong to have been decided several layers over my head, I'll make it
> a point for upgrading to Sp2 anyway, I might get lucky.
Good luck with that!
> the SAN department investigate their side of the problem, I'll take a
> look at trying a different stonith resources, all servers involved
> have some kind of IBM management console. Thanks for your answers to
> my questions and for your time, very much appreciated.
You're missing out on many further fixes since SP1 went out of support.
Not just to sbd, but everything, from kernel to pacemaker to glibc and back.
Since support is obviously irrelevant to your management, you could consider
recompiling sbd from source if you were so inclined, though.
Regards,
Lars
--
Architect Storage/HA
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imend?rffer,
HRB 21284 (AG N?rnberg) "Experience is the name everyone gives to their
mistakes." -- Oscar Wilde
17:46:30 DEV tps rd_sec/s wr_sec/s
avgrq-sz avgqu-sz await svctm %util
17:46:32 sda 7.73 0.00 136.08 17.60 0.05 6.13
4.80 3.71
17:46:32 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:32 sdc 8.25 1.03 90.72 11.12 0.00 0.00
0.00 0.00
17:46:32 san 8.25 1.03 90.72 11.12 0.00 0.00
0.00 0.00
17:46:32 san_part1 1.03 1.03 0.00 1.00 0.00 0.00
0.00 0.00
17:46:32 san_part2 12.37 0.00 90.72 7.33 0.00 0.00
0.00 0.00
17:46:34 sda 2.99 0.00 59.70 20.00 0.03 8.67
5.33 1.59
17:46:34 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:34 sdc 1.00 1.00 0.00 1.00 0.00 0.00
0.00 0.00
17:46:34 san 1.00 1.00 0.00 1.00 0.00 0.00
0.00 0.00
17:46:34 san_part1 1.00 1.00 0.00 1.00 0.00 0.00
0.00 0.00
17:46:34 san_part2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:36 sda 3.02 0.00 112.56 37.33 0.02 5.33
4.00 1.21
17:46:36 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:36 sdc 2.01 1.01 32.16 16.50 0.00 0.00
0.00 0.00
17:46:36 san 2.01 1.01 32.16 16.50 0.00 0.00
0.00 0.00
17:46:36 san_part1 1.01 1.01 0.00 1.00 0.00 0.00
0.00 0.00
17:46:36 san_part2 5.03 0.00 32.16 6.40 0.00 0.00
0.00 0.00
17:46:38 sda 2.96 0.00 63.05 21.33 0.01 4.67
3.33 0.99
17:46:38 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:38 sdc 1.97 0.99 31.53 16.50 0.00 1.00
1.00 0.20
17:46:38 san 1.97 0.99 31.53 16.50 0.00 1.00
1.00 0.20
17:46:38 san_part1 0.99 0.99 0.00 1.00 0.00 2.00
2.00 0.20
17:46:38 san_part2 4.93 0.00 31.53 6.40 0.00 0.00
0.00 0.00
17:46:38 DEV tps rd_sec/s wr_sec/s
avgrq-sz avgqu-sz await svctm %util
17:46:40 sda 3.03 0.00 68.69 22.67 0.01 4.67
3.33 1.01
17:46:40 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:40 sdc 1.01 1.01 0.00 1.00 0.00 0.00
0.00 0.00
17:46:40 san 1.01 1.01 0.00 1.00 0.00 0.00
0.00 0.00
17:46:40 san_part1 1.01 1.01 0.00 1.00 0.00 0.00
0.00 0.00
17:46:40 san_part2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:42 sda 2.96 0.00 59.11 20.00 0.02 5.33
4.00 1.18
17:46:42 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:42 sdc 1.97 0.99 31.53 16.50 0.00 0.00
0.00 0.00
17:46:42 san 1.97 0.99 31.53 16.50 0.00 0.00
0.00 0.00
17:46:42 san_part1 0.99 0.99 0.00 1.00 0.00 0.00
0.00 0.00
17:46:42 san_part2 4.93 0.00 31.53 6.40 0.00 0.00
0.00 0.00
17:46:44 sda 5.08 0.00 178.68 35.20 0.03 5.20
4.40 2.23
17:46:44 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:44 sdc 0.51 0.51 0.00 1.00 0.10 0.00
188.00 9.54
17:46:44 san 0.51 0.51 0.00 1.00 0.20 0.00
188.00 9.54
17:46:44 san_part1 1.02 1.02 0.00 1.00 0.10 0.00
94.00 9.54
17:46:44 san_part2 1.52 0.00 12.18 8.00 0.10 0.00
22.67 3.45
17:46:46 sda 10.00 0.00 140.00 14.00 0.18 18.00
3.20 3.20
17:46:46 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:46 sdc 0.00 0.00 0.00 0.00 1.00 0.00
0.00 100.00
17:46:46 san 0.00 0.00 0.00 0.00 4.00 0.00
0.00 100.00
17:46:46 san_part1 0.00 0.00 0.00 0.00 1.00 0.00
0.00 100.00
17:46:46 san_part2 0.00 0.00 0.00 0.00 3.00 0.00
0.00 100.00
17:46:46 DEV tps rd_sec/s wr_sec/s
avgrq-sz avgqu-sz await svctm %util
17:46:48 sda 6.93 0.00 122.77 17.71 0.04 5.14
4.57 3.17
17:46:48 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:48 sdc 0.00 0.00 0.00 0.00 0.99 0.00
0.00 99.01
17:46:48 san 0.00 0.00 0.00 0.00 3.96 0.00
0.00 99.01
17:46:48 san_part1 0.00 0.00 0.00 0.00 0.99 0.00
0.00 99.01
17:46:48 san_part2 0.00 0.00 0.00 0.00 2.97 0.00
0.00 99.01
17:46:50 sda 7.00 0.00 116.00 16.57 0.04 5.14
4.57 3.20
17:46:50 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:50 sdc 0.00 0.00 0.00 0.00 1.00 0.00
0.00 100.00
17:46:50 san 0.00 0.00 0.00 0.00 4.00 0.00
0.00 100.00
17:46:50 san_part1 0.00 0.00 0.00 0.00 1.00 0.00
0.00 100.00
17:46:50 san_part2 0.00 0.00 0.00 0.00 3.00 0.00
0.00 100.00
17:46:52 sda 7.43 0.00 122.77 16.53 0.04 5.87
5.33 3.96
17:46:52 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:52 sdc 0.00 0.00 0.00 0.00 0.99 0.00
0.00 99.01
17:46:52 san 0.00 0.00 0.00 0.00 3.96 0.00
0.00 99.01
17:46:52 san_part1 0.00 0.00 0.00 0.00 0.99 0.00
0.00 99.01
17:46:52 san_part2 0.00 0.00 0.00 0.00 2.97 0.00
0.00 99.01
17:46:54 sda 7.96 0.00 175.12 22.00 0.04 5.50
4.00 3.18
17:46:54 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:54 sdc 0.00 0.00 0.00 0.00 1.00 0.00
0.00 99.50
17:46:54 san 0.00 0.00 0.00 0.00 3.98 0.00
0.00 99.50
17:46:54 san_part1 0.00 0.00 0.00 0.00 1.00 0.00
0.00 99.50
17:46:54 san_part2 0.00 0.00 0.00 0.00 2.99 0.00
0.00 99.50
17:46:54 DEV tps rd_sec/s wr_sec/s
avgrq-sz avgqu-sz await svctm %util
17:46:56 sda 6.97 0.00 119.40 17.14 0.04 5.14
4.57 3.18
17:46:56 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:56 sdc 0.00 0.00 0.00 0.00 1.00 0.00
0.00 99.50
17:46:56 san 0.00 0.00 0.00 0.00 3.98 0.00
0.00 99.50
17:46:56 san_part1 0.00 0.00 0.00 0.00 1.00 0.00
0.00 99.50
17:46:56 san_part2 0.00 0.00 0.00 0.00 2.99 0.00
0.00 99.50
17:46:58 sda 6.93 0.00 146.53 21.14 0.04 6.29
5.14 3.56
17:46:58 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:46:58 sdc 0.00 0.00 0.00 0.00 0.99 0.00
0.00 99.01
17:46:58 san 0.00 0.00 0.00 0.00 3.96 0.00
0.00 99.01
17:46:58 san_part1 0.00 0.00 0.00 0.00 0.99 0.00
0.00 99.01
17:46:58 san_part2 0.00 0.00 0.00 0.00 2.97 0.00
0.00 99.01
17:47:00 sda 6.80 0.00 120.39 17.71 0.03 5.14
4.57 3.11
17:47:00 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:47:00 sdc 0.49 0.00 3.88 8.00 0.97 0.00
2000.00 97.09
17:47:00 san 0.49 0.00 3.88 8.00 3.88 0.00
2000.00 97.09
17:47:00 san_part1 0.00 0.00 0.00 0.00 0.97 0.00
0.00 97.09
17:47:00 san_part2 0.97 0.00 3.88 4.00 2.91 0.00
1000.00 97.09
17:47:02 sda 7.14 0.00 114.29 16.00 0.05 6.57
5.14 3.67
17:47:02 sdb 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
17:47:02 sdc 0.00 0.00 0.00 0.00 1.02 0.00
0.00 102.04
17:47:02 san 0.00 0.00 0.00 0.00 4.08 0.00
0.00 102.04
17:47:02 san_part1 0.00 0.00 0.00 0.00 1.02 0.00
0.00 102.04
17:47:02 san_part2 0.00 0.00 0.00 0.00 3.06 0.00
0.00 102.04
17:46:30 CPU %user %nice %system %iowait %steal %idle
17:46:32 1 0.00 0.00 0.00 0.00 0.00 100.00
17:46:34 1 0.50 0.00 0.50 0.00 0.00 98.99
17:46:36 1 0.00 0.00 0.00 0.00 0.00 100.00
17:46:38 1 0.00 0.00 0.00 0.00 0.00 100.00
17:46:40 1 0.99 0.00 0.00 0.00 0.00 99.01
17:46:42 1 0.00 0.00 0.48 0.00 0.00 99.52
17:46:44 1 0.00 0.00 0.00 5.61 0.00 94.39
17:46:46 1 0.50 0.00 0.50 99.01 0.00 0.00
17:46:48 1 0.50 0.00 0.00 99.50 0.00 0.00
17:46:50 1 0.00 0.00 0.00 100.00 0.00 0.00
17:46:52 1 0.51 0.00 0.00 99.49 0.00 0.00
17:46:54 1 0.99 0.00 0.99 98.03 0.00 0.00
17:46:56 1 0.00 0.00 0.00 100.00 0.00 0.00
17:46:58 1 0.00 0.00 0.00 100.00 0.00 0.00
17:47:00 1 0.49 0.00 0.98 98.53 0.00 0.00
17:47:02 1 0.00 0.00 0.00 100.00 0.00 0.00
May 3 17:46:43 server1 kernel: [66586.620815] sd 2:0:0:0: [sdc] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:43 server1 kernel: [66586.620820] sd 2:0:0:0: [sdc] CDB: Test Unit
Ready: 00 00 00 00 00 00
May 3 17:46:43 server1 sbd: [12965]: info: Latency: 1 on disk
/dev/mapper/san_part1
May 3 17:46:43 server1 kernel: [66587.157655] sd 2:0:0:0: [sdc] Send:
May 3 17:46:43 server1 kernel: [66587.157660] sd 2:0:0:0: [sdc] CDB: Read(10):
28 00 00 00 00 43 00 00 01 00
May 3 17:46:43 server1 kernel: [66587.157747] sd 2:0:0:0: [sdc] Done: SUCCESS
May 3 17:46:43 server1 kernel: [66587.157751] sd 2:0:0:0: [sdc] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:43 server1 kernel: [66587.157757] sd 2:0:0:0: [sdc] CDB: Read(10):
28 00 00 00 00 43 00 00 01 00
May 3 17:46:44 server1 kernel: [66588.155971] sd 2:0:0:0: [sdc] Send:
May 3 17:46:44 server1 kernel: [66588.155976] sd 2:0:0:0: [sdc] CDB: Read(10):
28 00 00 00 00 43 00 00 01 00
May 3 17:46:44 server1 kernel: [66588.156113] qla2xxx [0000:11:00.1]-5816:2:
Discard RND Frame -- 1006 02c1 0000.
May 3 17:46:44 server1 kernel: [66588.158991] sd 0:1:0:0: [sda] Send:
May 3 17:46:44 server1 kernel: [66588.158997] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 82 00 00 08 00
May 3 17:46:44 server1 kernel: [66588.164066] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:44 server1 kernel: [66588.164071] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:44 server1 kernel: [66588.164075] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 82 00 00 08 00
[
]
May 3 17:46:46 server1 kernel: [66590.363956] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 6f ba 00 00 08 00
May 3 17:46:46 server1 kernel: [66590.367892] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:46 server1 kernel: [66590.367897] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:46 server1 kernel: [66590.367902] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 6f ba 00 00 08 00
May 3 17:46:47 server1 sbd: [12964]: WARN: Latency: No liveness for 4 s
exceeds threshold of 3 s (healthy servants: 0)
May 3 17:46:47 server1 kernel: [66591.186425] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:47 server1 kernel: [66591.186431] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 6f ea 00 00 08 00
May 3 17:46:48 server1 kernel: [66591.612212] sd 1:0:0:0: [sdb] Send:
May 3 17:46:48 server1 kernel: [66591.612217] sd 1:0:0:0: [sdb] CDB: Test Unit
Ready: 00 00 00 00 00 00
May 3 17:46:48 server1 kernel: [66591.612326] sd 1:0:0:0: [sdb] Done: SUCCESS
May 3 17:46:48 server1 kernel: [66591.612332] sd 1:0:0:0: [sdb] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:48 server1 kernel: [66591.612336] sd 1:0:0:0: [sdb] CDB: Test Unit
Ready: 00 00 00 00 00 00
May 3 17:46:48 server1 kernel: [66591.613869] sd 2:0:0:0: [sdc] Send:
May 3 17:46:48 server1 kernel: [66591.613873] sd 2:0:0:0: [sdc] CDB: Test Unit
Ready: 00 00 00 00 00 00
May 3 17:46:48 server1 kernel: [66591.614598] sd 2:0:0:0: [sdc] Done: SUCCESS
May 3 17:46:48 server1 kernel: [66591.614603] sd 2:0:0:0: [sdc] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:48 server1 kernel: [66591.614608] sd 2:0:0:0: [sdc] CDB: Test Unit
Ready: 00 00 00 00 00 00
May 3 17:46:48 server1 sbd: [12964]: WARN: Latency: No liveness for 5 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:46:48 server1 kernel: [66592.166969] sd 0:1:0:0: [sda] Send:
May 3 17:46:48 server1 kernel: [66592.166975] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 82 00 00 08 00
May 3 17:46:48 server1 kernel: [66592.172261] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:48 server1 kernel: [66592.172266] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:48 server1 kernel: [66592.172270] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 82 00 00 08 00
May 3 17:46:48 server1 kernel: [66592.172392] sd 0:1:0:0: [sda] Send:
[
]
May 3 17:46:48 server1 kernel: [66592.364484] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:48 server1 kernel: [66592.364489] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:48 server1 kernel: [66592.364493] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 70 4a 00 00 08 00
May 3 17:46:49 server1 sbd: [12964]: WARN: Latency: No liveness for 6 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:46:49 server1 kernel: [66593.169005] sd 0:1:0:0: [sda] Send:
May 3 17:46:49 server1 kernel: [66593.169010] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 82 00 00 08 00
May 3 17:46:49 server1 kernel: [66593.174256] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:49 server1 kernel: [66593.174262] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
[
]
May 3 17:46:49 server1 kernel: [66593.191032] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:49 server1 kernel: [66593.191037] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:49 server1 kernel: [66593.191041] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 70 7a 00 00 08 00
May 3 17:46:50 server1 sbd: [12964]: WARN: Latency: No liveness for 7 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:46:50 server1 kernel: [66594.171026] sd 0:1:0:0: [sda] Send:
May 3 17:46:50 server1 kernel: [66594.171031] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 82 00 00 08 00
May 3 17:46:50 server1 kernel: [66594.176283] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:50 server1 kernel: [66594.176289] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:50 server1 kernel: [66594.176295] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 82 00 00 08 00
May 3 17:46:50 server1 kernel: [66594.176362] sd 0:1:0:0: [sda] Send:
[
]
May 3 17:46:50 server1 kernel: [66594.423103] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:50 server1 kernel: [66594.423108] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:50 server1 kernel: [66594.423113] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 0c 31 4b ca 00 00 08 00
May 3 17:46:51 server1 sbd: [12964]: WARN: Latency: No liveness for 8 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:46:51 server1 kernel: [66595.173093] sd 0:1:0:0: [sda] Send:
May 3 17:46:51 server1 kernel: [66595.173098] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 82 00 00 08 00
May 3 17:46:51 server1 kernel: [66595.181415] sd 0:1:0:0: [sda] Done: SUCCESS
[
]
May 3 17:46:51 server1 kernel: [66595.195010] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:51 server1 kernel: [66595.195014] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 70 f2 00 00 08 00
May 3 17:46:52 server1 sbd: [12964]: WARN: Latency: No liveness for 9 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:46:52 server1 kernel: [66596.176864] sd 0:1:0:0: [sda] Send:
May 3 17:46:52 server1 kernel: [66596.176870] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 82 00 00 08 00
May 3 17:46:52 server1 kernel: [66596.184427] sd 0:1:0:0: [sda] Done: SUCCESS
[
]
May 3 17:46:52 server1 kernel: [66596.358129] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:52 server1 kernel: [66596.358135] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:52 server1 kernel: [66596.358139] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 71 6a 00 00 08 00
May 3 17:46:53 server1 kernel: [66596.606291] sd 1:0:0:0: [sdb] Send:
May 3 17:46:53 server1 kernel: [66596.606297] sd 1:0:0:0: [sdb] CDB: Test Unit
Ready: 00 00 00 00 00 00
May 3 17:46:53 server1 kernel: [66596.606382] sd 1:0:0:0: [sdb] Done: SUCCESS
May 3 17:46:53 server1 kernel: [66596.606389] sd 1:0:0:0: [sdb] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:53 server1 kernel: [66596.606393] sd 1:0:0:0: [sdb] CDB: Test Unit
Ready: 00 00 00 00 00 00
May 3 17:46:53 server1 kernel: [66596.607412] sd 2:0:0:0: [sdc] Send:
May 3 17:46:53 server1 kernel: [66596.607416] sd 2:0:0:0: [sdc] CDB: Test Unit
Ready: 00 00 00 00 00 00
May 3 17:46:53 server1 kernel: [66596.607544] sd 2:0:0:0: [sdc] Done: SUCCESS
May 3 17:46:53 server1 kernel: [66596.607549] sd 2:0:0:0: [sdc] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:53 server1 kernel: [66596.607554] sd 2:0:0:0: [sdc] CDB: Test Unit
Ready: 00 00 00 00 00 00
May 3 17:46:53 server1 sbd: [12964]: WARN: Latency: No liveness for 10 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:46:53 server1 kernel: [66597.177189] sd 0:1:0:0: [sda] Send:
May 3 17:46:53 server1 kernel: [66597.177195] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 82 00 00 08 00
May 3 17:46:53 server1 kernel: [66597.182417] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:53 server1 kernel: [66597.182424] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:53 server1 kernel: [66597.182430] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 82 00 00 08 00
May 3 17:46:53 server1 kernel: [66597.182516] sd 0:1:0:0: [sda] Send:
May 3 17:46:53 server1 kernel: [66597.182520] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 63 5a ca 00 00 08 0
[
]
May 3 17:46:53 server1 kernel: [66597.200775] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:53 server1 kernel: [66597.200779] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 71 d2 00 00 08 00
May 3 17:46:54 server1 sbd: [12964]: WARN: Latency: No liveness for 11 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:46:54 server1 kernel: [66598.179225] sd 0:1:0:0: [sda] Send:
May 3 17:46:54 server1 kernel: [66598.179231] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 82 00 00 08 00
May 3 17:46:54 server1 kernel: [66598.179258] sd 0:1:0:0: [sda] Send:
May 3 17:46:54 server1 kernel: [66598.179260] sd 0:1:0:0: [sda] CDB:
Write(10): 2a
[
]
May 3 17:46:54 server1 kernel: [66598.351075] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:54 server1 kernel: [66598.351081] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:54 server1 kernel: [66598.351085] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 72 5a 00 00 08 00
May 3 17:46:55 server1 sbd: [12964]: WARN: Latency: No liveness for 12 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:46:55 server1 kernel: [66599.181231] sd 0:1:0:0: [sda] Send:
May 3 17:46:55 server1 kernel: [66599.181236] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 8a 00 00 08 00
May 3 17:46:55 server1 kernel: [66599.186590] sd 0:1:0:0: [sda] Done: SUCCESS
[
]
May 3 17:46:55 server1 kernel: [66599.201501] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:55 server1 kernel: [66599.201506] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:55 server1 kernel: [66599.201511] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 72 72 00 00 08 00
May 3 17:46:56 server1 sbd: [12964]: WARN: Latency: No liveness for 13 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:46:56 server1 kernel: [66600.183276] sd 0:1:0:0: [sda] Send:
May 3 17:46:56 server1 kernel: [66600.183282] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 8a 00 00 08 00
May 3 17:46:56 server1 kernel: [66600.188632] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:56 server1 kernel: [66600.188637] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
[
]
May 3 17:46:56 server1 kernel: [66600.351545] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:56 server1 kernel: [66600.351550] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:56 server1 kernel: [66600.351555] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 72 da 00 00 08 00
May 3 17:46:57 server1 sbd: [12964]: WARN: Latency: No liveness for 14 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:46:57 server1 kernel: [66601.187041] sd 0:1:0:0: [sda] Send:
May 3 17:46:57 server1 kernel: [66601.187048] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 8a 00 00 08 00
May 3 17:46:57 server1 kernel: [66601.194623] sd 0:1:0:0: [sda] Done: SUCCESS
[
]
May 3 17:46:57 server1 kernel: [66601.206090] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 73 0a 00 00 08 00
May 3 17:46:57 server1 kernel: [66601.210040] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:57 server1 kernel: [66601.210045] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:57 server1 kernel: [66601.210050] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 73 0a 00 00 08 00
May 3 17:46:58 server1 kernel: [66601.599530] sd 1:0:0:0: [sdb] Send:
May 3 17:46:58 server1 kernel: [66601.599536] sd 1:0:0:0: [sdb] CDB: Test Unit
Ready: 00 00 00 00 00 00
May 3 17:46:58 server1 kernel: [66601.599647] sd 1:0:0:0: [sdb] Done: SUCCESS
May 3 17:46:58 server1 kernel: [66601.599652] sd 1:0:0:0: [sdb] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:58 server1 kernel: [66601.599656] sd 1:0:0:0: [sdb] CDB: Test Unit
Ready: 00 00 00 00 00 00
May 3 17:46:58 server1 kernel: [66601.600801] sd 2:0:0:0: [sdc] Send:
May 3 17:46:58 server1 kernel: [66601.600805] sd 2:0:0:0: [sdc] CDB: Test Unit
Ready: 00 00 00 00 00 00
May 3 17:46:58 server1 kernel: [66601.601538] sd 2:0:0:0: [sdc] Done: SUCCESS
May 3 17:46:58 server1 kernel: [66601.601543] sd 2:0:0:0: [sdc] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:58 server1 kernel: [66601.601547] sd 2:0:0:0: [sdc] CDB: Test Unit
Ready: 00 00 00 00 00 00
May 3 17:46:58 server1 sbd: [12964]: WARN: Latency: No liveness for 15 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:46:58 server1 kernel: [66602.187357] sd 0:1:0:0: [sda] Send:
May 3 17:46:58 server1 kernel: [66602.187362] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 8a 00 00 08 00
May 3 17:46:58 server1 kernel: [66602.192804] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:58 server1 kernel: [66602.192809] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:58 server1 kernel: [66602.192814] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 8a 00 00 08 00
May 3 17:46:58 server1 kernel: [66602.192940] sd 0:1:0:0: [sda] Send:
[
]
May 3 17:46:58 server1 kernel: [66602.344442] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:58 server1 kernel: [66602.344447] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 73 b2 00 00 08 00
May 3 17:46:59 server1 sbd: [12964]: WARN: Latency: No liveness for 16 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:46:59 server1 kernel: [66603.189394] sd 0:1:0:0: [sda] Send:
May 3 17:46:59 server1 kernel: [66603.189400] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 8a 00 00 08 00
May 3 17:46:59 server1 kernel: [66603.194805] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:59 server1 kernel: [66603.194810] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
[
]
May 3 17:46:59 server1 kernel: [66603.210851] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:46:59 server1 kernel: [66603.210856] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:46:59 server1 kernel: [66603.210860] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 73 e2 00 00 08 00
May 3 17:47:00 server1 sbd: [12964]: WARN: Latency: No liveness for 17 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:47:00 server1 kernel: [66604.191433] sd 0:1:0:0: [sda] Send:
May 3 17:47:00 server1 kernel: [66604.191439] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 8a 00 00 08 00
May 3 17:47:00 server1 kernel: [66604.196850] sd 0:1:0:0: [sda] Done: SUCCESS
[
]
May 3 17:47:00 server1 kernel: [66604.345030] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:47:00 server1 kernel: [66604.345035] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:47:00 server1 kernel: [66604.345040] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 74 4a 00 00 08 00
May 3 17:47:01 server1 sbd: [12964]: WARN: Latency: No liveness for 18 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:47:01 server1 kernel: [66605.193480] sd 0:1:0:0: [sda] Send:
May 3 17:47:01 server1 kernel: [66605.193485] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 8a 00 00 08 00
May 3 17:47:01 server1 kernel: [66605.198912] sd 0:1:0:0: [sda] Done: SUCCESS
[
]
May 3 17:47:01 server1 kernel: [66605.215052] sd 0:1:0:0: [sda] Done: SUCCESS
May 3 17:47:01 server1 kernel: [66605.215058] sd 0:1:0:0: [sda] Result:
hostbyte=DID_OK driverbyte=DRIVER_OK
May 3 17:47:01 server1 kernel: [66605.215062] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 09 7e 74 82 00 00 08 00
May 3 17:47:02 server1 sbd: [12964]: WARN: Latency: No liveness for 19 s
exceeds threshold of 3 s (healthy servants:0)
May 3 17:47:02 server1 kernel: [66606.195505] sd 0:1:0:0: [sda] Send:
May 3 17:47:02 server1 kernel: [66606.195511] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 5b 8a 8a 00 00 08 00
May 3 17:47:02 server1 kernel: [66606.200931] sd 0:1:0:0: [sda] Done: SUCCESS
[
]
May 3 17:47:02 server1 kernel: [66606.311650] sd 0:1:0:0: [sda] Send:
May 3 17:47:02 server1 kernel: [66606.311656] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 72 59 62 00 00 08 00
May 3 17:47:02 server1 kernel: [66606.311693] sd 0:1:0:0: [sda] Send:
May 3 17:47:02 server1 kernel: [66606.311696] sd 0:1:0:0: [sda] CDB:
Write(10): 2a 00 03 72 59 6a 00 00 18 00
---------------REBOOT-----------------------
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
