Re: Reloading iscsi target, iscsi initiator panics

2015-10-15 Thread Edward Tomasz Napierała
On 1014T2316, Frank de Bot (lists) wrote:
> Hello,
> 
> I have a FreeBSD 10.2 server running as iSCSI target. Another FreeBSD
> 10.2 is an initiator and has several targets used.
> 
> When I add a target and reload its config with 'service ctld reload',
> the FreeBSD initiator server panics. It's reproducable (I have a
> coredump, but I think it's clear what the problem is)

How easy it is to reproduce, eg. does it happen about every time?
Could you paste the backtrace?

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"


Re: Reloading iscsi target, iscsi initiator panics

2015-10-15 Thread Frank de Bot (lists)
Edward Tomasz Napierała wrote:
> On 1014T2316, Frank de Bot (lists) wrote:
>> Hello,
>>
>> I have a FreeBSD 10.2 server running as iSCSI target. Another FreeBSD
>> 10.2 is an initiator and has several targets used.
>>
>> When I add a target and reload its config with 'service ctld reload',
>> the FreeBSD initiator server panics. It's reproducable (I have a
>> coredump, but I think it's clear what the problem is)
> 
> How easy it is to reproduce, eg. does it happen about every time?
> Could you paste the backtrace?

The first time I didn't understand what happened, the second time I
could relate it directly to the reloading of ctld on the iSCSI, within
moments the server paniced.

I got 2 backtraces:

First one:

Fatal trap 12: page fault while in kernel mode
cpuid = 1; softdep_deallocate_dependencies: got error 6 while accessing
filesystem
apic id = 01
fault virtual address   = 0x0
fault code  = supervisor read data, page not present
instruction pointer = 0x20:0x808a86b5
stack pointer   = 0x28:0xfe023b5f1200
frame pointer   = 0x28:0xfe023b5f1260
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 42663 (php)
trap number = 12
panic: page fault
cpuid = 1
KDB: stack backtrace:
#0 0x80984e30 at kdb_backtrace+0x60
#1 0x809489e6 at vpanic+0x126
#2 0x809488b3 at panic+0x43
#3 0x80d4aadb at trap_fatal+0x36b
#4 0x80d4addd at trap_pfault+0x2ed
#5 0x80d4a47a at trap+0x47a
#6 0x80d307f2 at calltrap+0x8
#7 0x80b73208 at ffs_blkfree+0x168
#8 0x80b8c747 at handle_workitem_freefrag+0x127
#9 0x80b73e28 at ffs_reallocblks+0xc08
#10 0x80e74117 at VOP_REALLOCBLKS_APV+0xa7
#11 0x809da65b at cluster_write+0x3eb
#12 0x80ba4dcb at ffs_write+0x45b
#13 0x80e72495 at VOP_WRITE_APV+0x145
#14 0x809fc3f9 at vn_write+0x259
#15 0x809fc632 at vn_io_fault_doio+0x22
#16 0x809fa17c at vn_io_fault1+0x7c
#17 0x809f864b at vn_io_fault+0x18b
Uptime: 16d1h28m10s
(da3:iscsi10:0:0:0): Synchronize cache failed
Dumping 1154 out of 8163
MB:..2%..12%..21%..31%..41%..52%..61%..71%..81%..91%


Second:
Device resolver went missing before all of the data could be written to
it; expect data loss.
/dev: got error 6 while accessing filesystem
panic: softdep_deallocate_dependencies: dangling deps
cpuid = 0
KDB: stack backtrace:
#0 0x80984e30 at kdb_backtrace+0x60
#1 0x809489e6 at vpanic+0x126
#2 0x809488b3 at panic+0x43
#3 0x80b866da at softdep_deallocate_dependencies+0x6a
#4 0x809d1d95 at brelse+0x145
#5 0x809e9ee7 at flushbuflist+0x137
#6 0x809e9b5b at bufobj_invalbuf+0x1cb
#7 0x809ec4be at vgonel+0x17e
#8 0x809ec8ec at vgone+0x4c
#9 0x8082b4e8 at devfs_delete+0x218
#10 0x8082b95c at devfs_populate_loop+0x1fc
#11 0x8082b74a at devfs_populate+0x2a
#12 0x8082fa6b at devfs_populate_vp+0x9b
#13 0x808303bc at devfs_lookup+0x2c
#14 0x80e71621 at VOP_LOOKUP_APV+0xa1
#15 0x809e0951 at lookup+0x5a1
#16 0x809e00b4 at namei+0x4d4
#17 0x809f9135 at vn_open_cred+0xd5
Uptime: 38m45s
(da3:iscsi4:0:0:0): Synchronize cache failed
(da4:iscsi5:0:0:0): Synchronize cache failed
Dumping 815 out of 8163 MB:..2%..12%..22%..32%..42%..52%..61%..71%..81%..91%

I've put the whole core.txt.%d files here (with my ip addresses and
hostnames anonymized):
 - http://frankdebot.nl/tmp/core.txt.0
 - http://frankdebot.nl/tmp/core.txt.1


> 
> ___
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
> 

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Reloading iscsi target, iscsi initiator panics

2015-10-15 Thread Edward Tomasz Napierała
On 1015T2005, Frank de Bot (lists) wrote:
> Edward Tomasz Napierała wrote:
> > On 1014T2316, Frank de Bot (lists) wrote:
> >> Hello,
> >>
> >> I have a FreeBSD 10.2 server running as iSCSI target. Another FreeBSD
> >> 10.2 is an initiator and has several targets used.
> >>
> >> When I add a target and reload its config with 'service ctld reload',
> >> the FreeBSD initiator server panics. It's reproducable (I have a
> >> coredump, but I think it's clear what the problem is)
> > 
> > How easy it is to reproduce, eg. does it happen about every time?
> > Could you paste the backtrace?
> 
> The first time I didn't understand what happened, the second time I
> could relate it directly to the reloading of ctld on the iSCSI, within
> moments the server paniced.
> 
> I got 2 backtraces:
> 
> First one:
> 
> Fatal trap 12: page fault while in kernel mode
> cpuid = 1; softdep_deallocate_dependencies: got error 6 while accessing

This is the FFS on the initiator side panicing because the device it's
on went away.  The softupdates code can't handle that very well.

I have no idea why the devices went away and then reappeared, as visible
in the logs.  What has the changed in the ctl.conf?  Do you have any
iscsi-related sysctls set on the initiator side?

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"