RE: New panic in main-n253273-a52d8d4a6c6:

[Michael Jung]

-Original Message-
From: owner-freebsd-curr...@freebsd.org 
[mailto:owner-freebsd-curr...@freebsd.org] On Behalf Of Hans Petter Selasky
Sent: Monday, February 21, 2022 2:48 AM
To: Michael Jung ; freebsd-current 

Subject: Re: New panic in main-n253273-a52d8d4a6c6:

On 2/20/22 21:53, Michael Jung wrote:
> Hi!
>
> The box was quite busy at the time.  The only odd thing I am aware of
> and which I do not think is related is I have not been able to expand
> one of my zpool's.  ZFS sees my added draid2:2d:10c:0s vdev but I
> can't seem to force zpool's expansion - my bet this is somehow related
> to the mirrored special device in the pool even though autoexpand is set to 
> 'on' for the pool.
>
> Not asking anyone to solve this, I plan on putting together a "how to
> reproduce" and post to freebsd-fs@ but wanted to note this oddity.
>
> --mikej
>
> This is a unmodified GENERIC kernel.
>
>
> Unread portion of the kernel message buffer:
> panic: refcount 0xf801cc119284 wraparound cpuid = 7 time =
> 1645382575
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> 0xfe0119a83c80
> vpanic() at vpanic+0x17f/frame 0xfe0119a83cd0
> panic() at panic+0x43/frame 0xfe0119a83d30
> fdclose() at fdclose/frame 0xfe0119a83dc0
> closefp_impl() at closefp_impl+0x77/frame 0xfe0119a83e00
> amd64_syscall() at amd64_syscall+0x12e/frame 0xfe0119a83f30
> fast_syscall_common() at fast_syscall_common+0xf8/frame
> 0xfe0119a83f30
> --- syscall (6, FreeBSD ELF64, sys_close), rip = 0x3309a531b62a, rsp =
> 0x3309a2802bc8, rbp = 0x3309a2803000 ---
> KDB: enter: panic
>

This may be a refcount leak. Are you able to dump "fdp" at:

#16 0x80bad587 in closefp_impl (fdp=0xfe012b7c0430, fd=4,
 fp=0xf801cc119280, td=0xfe00df8d0560, audit=true)
 at /usr/src/sys/kern/kern_descrip.c:1300

frame 16
print /x *fdp

--HPS


 [root@draid /var/crash]# kgdb kernel.full.0 vmcore.0
GNU gdb (GDB) 11.2 [GDB v11.2 for FreeBSD]
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd14.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from kernel.full.0...

Unread portion of the kernel message buffer:
panic: refcount 0xf801cc119284 wraparound
cpuid = 7
time = 1645382575
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe0119a83c80
vpanic() at vpanic+0x17f/frame 0xfe0119a83cd0
panic() at panic+0x43/frame 0xfe0119a83d30
fdclose() at fdclose/frame 0xfe0119a83dc0
closefp_impl() at closefp_impl+0x77/frame 0xfe0119a83e00
amd64_syscall() at amd64_syscall+0x12e/frame 0xfe0119a83f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfe0119a83f30
--- syscall (6, FreeBSD ELF64, sys_close), rip = 0x3309a531b62a, rsp = 
0x3309a2802bc8, rbp = 0x3309a2803000 ---
KDB: enter: panic

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55  __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct 
pcpu,

(kgdb) fram 16
#16 0x80bad587 in closefp_impl (fdp=0xfe012b7c0430, fd=4, 
fp=0xf801cc119280, td=0xfe00df8d0560, audit=true) at 
/usr/src/sys/kern/kern_descrip.c:1300
1300error = closef(fp, td);

(kgdb) print /x *fdp
$1 = {fd_files = 0xfe00df6d9000, fd_map = 0xf80001bb1800, fd_freefile = 
0x4, fd_refcnt = 0x1, fd_holdcnt = 0x1, fd_sx = {lock_object = {lo_name = 
0x8124f6a9, lo_flags = 0x233, lo_data = 0x0,
  lo_witness = 0xf8041eb94000}, sx_lock = 0x1}, fd_kqlist = {tqh_first 
= 0x0, tqh_last = 0x0}, fd_holdleaderscount = 0x0, fd_holdleaderswakeup = 0x0}
(kgdb)




CONFIDENTIALITY NOTE: This message is intended only for the use
of the individual or entity to whom it is addressed and may
contain information that is privileged, confidential, and
exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying
of this communication is strictly prohibited. If you have
received this transmission in error, please notify us by
telephone at (502) 212-4000 or notify us at PAI, Dept. 99,
2101 High Wickham Place, Suite 101, Louisville, KY 40245

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are

Re: New panic in main-n253273-a52d8d4a6c6:

[Hans Petter Selasky]

On 2/21/22 14:07, Michael Jung wrote:

(kgdb) fram 16
#16 0x80bad587 in closefp_impl (fdp=0xfe012b7c0430, fd=4, 
fp=0xf801cc119280, td=0xfe00df8d0560, audit=true) at 
/usr/src/sys/kern/kern_descrip.c:1300
1300error = closef(fp, td);

(kgdb) print /x *fdp
$1 = {fd_files = 0xfe00df6d9000, fd_map = 0xf80001bb1800, fd_freefile = 
0x4, fd_refcnt = 0x1, fd_holdcnt = 0x1, fd_sx = {lock_object = {lo_name = 
0x8124f6a9, lo_flags = 0x233, lo_data = 0x0,
   lo_witness = 0xf8041eb94000}, sx_lock = 0x1}, fd_kqlist = {tqh_first 
= 0x0, tqh_last = 0x0}, fd_holdleaderscount = 0x0, fd_holdleaderswakeup = 0x0}
(kgdb)


Can you also:

print /x *fp

in the same frame?

--HPS

Re: pxeboot binary is too big on FreeBSD (>640KBytes)

[Bjoern A. Zeeb]

On 21 Feb 2022, at 7:43, Hans Petter Selasky wrote:


FYI:

After a lot of digging trying everything, I found that the pxeboot and 
loader.efi was too big simply due to ZFS support.


So I did this after buildworld:

cd /usr/src/stand
make WITHOUT_LOADER_ZFS=YES clean
make WITHOUT_LOADER_ZFS=YES all
make WITHOUT_LOADER_ZFS=YES install

And now it works, with my old GigaByte mainboard!

Why should pxeboot have ZFS support?

https://forums.freebsd.org/threads/problem-with-isc-dhcpd-and-diskless-booting.82199/


Well actually it started with a lua update;  pxeboot 4th is not (or was 
not) too big and still working.


See PR 257018 .

Re: pxeboot binary is too big on FreeBSD (>640KBytes)

[Toomas Soome]

> On 21. Feb 2022, at 09:43, Hans Petter Selasky  wrote:
> 
> FYI:
> 
> After a lot of digging trying everything, I found that the pxeboot and 
> loader.efi was too big simply due to ZFS support.
> 
> So I did this after buildworld:
> 
> cd /usr/src/stand
> make WITHOUT_LOADER_ZFS=YES clean
> make WITHOUT_LOADER_ZFS=YES all
> make WITHOUT_LOADER_ZFS=YES install
> 
> And now it works, with my old GigaByte mainboard!
> 
> Why should pxeboot have ZFS support?
> 
> https://forums.freebsd.org/threads/problem-with-isc-dhcpd-and-diskless-booting.82199/
> 
> —HPS
> 

Well, the feature X can be helpful for recovery purposes. The root cause is not 
the feature X itself, but the size limit. And the unfortunate fact, the size 
limit is not fixed, but depends on the system. Therefore there are two options 
- either to fix the size limit or drop option X from default build — at least 
till the size limit is fixed (or support for BIOS will be dropped).

rgds,
toomas

Re: pxeboot binary is too big on FreeBSD (>640KBytes)

[Simon J. Gerraty]

Toomas Soome  wrote:
> > Why should pxeboot have ZFS support?
> 
> Well, the feature X can be helpful for recovery purposes. The root
> cause is not the feature X itself, but the size limit. And the
> unfortunate fact, the size limit is not fixed, but depends on the
> system. Therefore there are two options - either to fix the size limit
> or drop option X from default build — at least till the size limit is
> fixed (or support for BIOS will be dropped).

Or just build separate variants.
As Bjoern said Lua is probably the straw breaking the cammel's back
but I think it reasonable to assume that a system that has the resources
to support ZFS does not have an ancient BIOS ?

Thus a non-ZFS version could work for older systems
while those without limitation can use the kitchen-sink version.

Re: pxeboot binary is too big on FreeBSD (>640KBytes)

[Toomas Soome]

> On 21. Feb 2022, at 19:18, Simon J. Gerraty  wrote:
> 
> Toomas Soome  wrote:
>>> Why should pxeboot have ZFS support?
>> 
>> Well, the feature X can be helpful for recovery purposes. The root
>> cause is not the feature X itself, but the size limit. And the
>> unfortunate fact, the size limit is not fixed, but depends on the
>> system. Therefore there are two options - either to fix the size limit
>> or drop option X from default build — at least till the size limit is
>> fixed (or support for BIOS will be dropped).
> 
> Or just build separate variants.
> As Bjoern said Lua is probably the straw breaking the cammel's back
> but I think it reasonable to assume that a system that has the resources
> to support ZFS does not have an ancient BIOS ?
> 
> Thus a non-ZFS version could work for older systems
> while those without limitation can use the kitchen-sink version.

It is not even about “ancient” BIOS, the problem is, PXE stack does also need 
resources and it is easier to consume low memory (just as loader does). 

rgds,
toomas

Re: pxeboot binary is too big on FreeBSD (>640KBytes)

[Philipp Ost]

On 2/21/22 18:18, Simon J. Gerraty wrote:

Toomas Soome  wrote:

Why should pxeboot have ZFS support?


Well, the feature X can be helpful for recovery purposes. The root
cause is not the feature X itself, but the size limit. And the
unfortunate fact, the size limit is not fixed, but depends on the
system. Therefore there are two options - either to fix the size limit
or drop option X from default build — at least till the size limit is
fixed (or support for BIOS will be dropped).


Or just build separate variants.
As Bjoern said Lua is probably the straw breaking the cammel's back
but I think it reasonable to assume that a system that has the resources
to support ZFS does not have an ancient BIOS ?


I have several machines in use which are capable of supporting ZFS just 
fine and all of these have a BIOS.


Just saying. ;-)

Re: pxeboot binary is too big on FreeBSD (>640KBytes)

[Alexander Motin]

On 21.02.2022 12:43, Toomas Soome wrote:

On 21. Feb 2022, at 19:18, Simon J. Gerraty  wrote:
Toomas Soome  wrote:

Why should pxeboot have ZFS support?


Well, the feature X can be helpful for recovery purposes. The root
cause is not the feature X itself, but the size limit. And the
unfortunate fact, the size limit is not fixed, but depends on the
system. Therefore there are two options - either to fix the size limit
or drop option X from default build — at least till the size limit is
fixed (or support for BIOS will be dropped).


Or just build separate variants.
As Bjoern said Lua is probably the straw breaking the cammel's back
but I think it reasonable to assume that a system that has the resources
to support ZFS does not have an ancient BIOS ?

Thus a non-ZFS version could work for older systems
while those without limitation can use the kitchen-sink version.


It is not even about “ancient” BIOS, the problem is, PXE stack does also need 
resources and it is easier to consume low memory (just as loader does).


I've recently switched all my lab systems to UEFI PXE.  It "just works" 
for me using normal loader.efi of 872KB.


--
Alexander Motin

RE: New panic in main-n253273-a52d8d4a6c6:

[Michael Jung]

-Original Message-
From: Hans Petter Selasky [mailto:h...@selasky.org]
Sent: Monday, February 21, 2022 8:53 AM
To: Michael Jung ; freebsd-current 

Subject: Re: New panic in main-n253273-a52d8d4a6c6:

On 2/21/22 14:07, Michael Jung wrote:
> (kgdb) fram 16
> #16 0x80bad587 in closefp_impl (fdp=0xfe012b7c0430, fd=4, 
> fp=0xf801cc119280, td=0xfe00df8d0560, audit=true) at 
> /usr/src/sys/kern/kern_descrip.c:1300
> 1300error = closef(fp, td);
>
> (kgdb) print /x *fdp
> $1 = {fd_files = 0xfe00df6d9000, fd_map = 0xf80001bb1800, fd_freefile 
> = 0x4, fd_refcnt = 0x1, fd_holdcnt = 0x1, fd_sx = {lock_object = {lo_name = 
> 0x8124f6a9, lo_flags = 0x233, lo_data = 0x0,
>lo_witness = 0xf8041eb94000}, sx_lock = 0x1}, fd_kqlist = 
> {tqh_first = 0x0, tqh_last = 0x0}, fd_holdleaderscount = 0x0, 
> fd_holdleaderswakeup = 0x0}
> (kgdb)

Can you also:

print /x *fp

in the same frame?

--HPS



Unread portion of the kernel message buffer:
panic: refcount 0xf801cc119284 wraparound
cpuid = 7
time = 1645382575
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe0119a83c80
vpanic() at vpanic+0x17f/frame 0xfe0119a83cd0
panic() at panic+0x43/frame 0xfe0119a83d30
fdclose() at fdclose/frame 0xfe0119a83dc0
closefp_impl() at closefp_impl+0x77/frame 0xfe0119a83e00
amd64_syscall() at amd64_syscall+0x12e/frame 0xfe0119a83f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfe0119a83f30
--- syscall (6, FreeBSD ELF64, sys_close), rip = 0x3309a531b62a, rsp = 
0x3309a2802bc8, rbp = 0x3309a2803000 ---
KDB: enter: panic

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55  __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct 
pcpu,


(kgdb) frame 16
#16 0x80bad587 in closefp_impl (fdp=0xfe012b7c0430, fd=4, 
fp=0xf801cc119280, td=0xfe00df8d0560, audit=true) at 
/usr/src/sys/kern/kern_descrip.c:1300
1300error = closef(fp, td);


(kgdb) print /x *fp
$1 = {f_flag = 0x5, f_count = 0x, f_data = 0xf800158b0400, f_ops = 
0x81b052d0, f_vnode = 0xf800d9156000, f_cred = 0xf801d81be400, 
f_type = 0x1, f_vnread_flags = 0x0, {f_seqcount = {0x0,
  0x0}, f_pipegen = 0x0}, f_nextoff = {0x0, 0x0}, f_vnun = {fvn_cdevpriv = 
0x0, fvn_advice = 0x0}, f_offset = 0x0}
(kgdb)




CONFIDENTIALITY NOTE: This message is intended only for the use
of the individual or entity to whom it is addressed and may
contain information that is privileged, confidential, and
exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying
of this communication is strictly prohibited. If you have
received this transmission in error, please notify us by
telephone at (502) 212-4000 or notify us at PAI, Dept. 99,
2101 High Wickham Place, Suite 101, Louisville, KY 40245

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast, a leader in email security and cyber 
resilience. Mimecast integrates email defenses with brand protection, security 
awareness training, web security, compliance and other essential capabilities. 
Mimecast helps protect large and small organizations from malicious activity, 
human error and technology failure; and to lead the movement toward building a 
more resilient world. To find out more, visit our website.

RE: New panic in main-n253273-a52d8d4a6c6:

[Michael Jung]

Hi:

I was trying to remember what I did that was odd when this crash occurred then 
it
hit me.  You can repeat this panic by doing:

# watch -I -W pts/0

Here is another panic that happened write after issuing "watch" for comparison.

http://mail.mikej.com/core.txt.1

http://mail.mikej.com/info.1

http://mail.mikej.com/vmcore.1

--mikej

-Original Message-
From: Michael Jung
Sent: Monday, February 21, 2022 6:22 PM
To: 'Hans Petter Selasky' ; freebsd-current 

Subject: RE: New panic in main-n253273-a52d8d4a6c6:



-Original Message-
From: Hans Petter Selasky [mailto:h...@selasky.org]
Sent: Monday, February 21, 2022 8:53 AM
To: Michael Jung ; freebsd-current 

Subject: Re: New panic in main-n253273-a52d8d4a6c6:

On 2/21/22 14:07, Michael Jung wrote:
> (kgdb) fram 16
> #16 0x80bad587 in closefp_impl (fdp=0xfe012b7c0430, fd=4, 
> fp=0xf801cc119280, td=0xfe00df8d0560, audit=true) at 
> /usr/src/sys/kern/kern_descrip.c:1300
> 1300error = closef(fp, td);
>
> (kgdb) print /x *fdp
> $1 = {fd_files = 0xfe00df6d9000, fd_map = 0xf80001bb1800, fd_freefile 
> = 0x4, fd_refcnt = 0x1, fd_holdcnt = 0x1, fd_sx = {lock_object = {lo_name = 
> 0x8124f6a9, lo_flags = 0x233, lo_data = 0x0,
>lo_witness = 0xf8041eb94000}, sx_lock = 0x1}, fd_kqlist =
> {tqh_first = 0x0, tqh_last = 0x0}, fd_holdleaderscount = 0x0,
> fd_holdleaderswakeup = 0x0}
> (kgdb)

Can you also:

print /x *fp

in the same frame?

--HPS



Unread portion of the kernel message buffer:
panic: refcount 0xf801cc119284 wraparound cpuid = 7 time = 1645382575
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe0119a83c80
vpanic() at vpanic+0x17f/frame 0xfe0119a83cd0
panic() at panic+0x43/frame 0xfe0119a83d30
fdclose() at fdclose/frame 0xfe0119a83dc0
closefp_impl() at closefp_impl+0x77/frame 0xfe0119a83e00
amd64_syscall() at amd64_syscall+0x12e/frame 0xfe0119a83f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfe0119a83f30
--- syscall (6, FreeBSD ELF64, sys_close), rip = 0x3309a531b62a, rsp = 
0x3309a2802bc8, rbp = 0x3309a2803000 ---
KDB: enter: panic

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55  __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct 
pcpu,


(kgdb) frame 16
#16 0x80bad587 in closefp_impl (fdp=0xfe012b7c0430, fd=4, 
fp=0xf801cc119280, td=0xfe00df8d0560, audit=true) at 
/usr/src/sys/kern/kern_descrip.c:1300
1300error = closef(fp, td);


(kgdb) print /x *fp
$1 = {f_flag = 0x5, f_count = 0x, f_data = 0xf800158b0400, f_ops = 
0x81b052d0, f_vnode = 0xf800d9156000, f_cred = 0xf801d81be400, 
f_type = 0x1, f_vnread_flags = 0x0, {f_seqcount = {0x0,
  0x0}, f_pipegen = 0x0}, f_nextoff = {0x0, 0x0}, f_vnun = {fvn_cdevpriv = 
0x0, fvn_advice = 0x0}, f_offset = 0x0}
(kgdb)




CONFIDENTIALITY NOTE: This message is intended only for the use
of the individual or entity to whom it is addressed and may
contain information that is privileged, confidential, and
exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying
of this communication is strictly prohibited. If you have
received this transmission in error, please notify us by
telephone at (502) 212-4000 or notify us at PAI, Dept. 99,
2101 High Wickham Place, Suite 101, Louisville, KY 40245

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast, a leader in email security and cyber 
resilience. Mimecast integrates email defenses with brand protection, security 
awareness training, web security, compliance and other essential capabilities. 
Mimecast helps protect large and small organizations from malicious activity, 
human error and technology failure; and to lead the movement toward building a 
more resilient world. To find out more, visit our website.

Re: New panic in main-n253273-a52d8d4a6c6:

[Hans Petter Selasky]

On 2/22/22 00:42, Michael Jung wrote:

Hi:

I was trying to remember what I did that was odd when this crash occurred then 
it
hit me.  You can repeat this panic by doing:

# watch -I -W pts/0

Here is another panic that happened write after issuing "watch" for comparison.

http://mail.mikej.com/core.txt.1

http://mail.mikej.com/info.1

http://mail.mikej.com/vmcore.1



I also need your kernel and debug kernel to fully debug this.

1) Do ssh to machine.
2) watch -i -W pts/0 (does not panic over here)

Can you explain what step 1 is? An scp ?

Refcount is -1.
f_count = 0x

f_data = 0xf800158b0400

In your KGDB, can you enter:

info 0x81b052d0

Does the attached patch make any difference?

--HPSdiff --git a/sys/dev/snp/snp.c b/sys/dev/snp/snp.c
index 64e2d0f6453..6eec4a5a632 100644
--- a/sys/dev/snp/snp.c
+++ b/sys/dev/snp/snp.c
@@ -86,6 +86,7 @@ static d_poll_t		snp_poll;
 
 static struct cdevsw snp_cdevsw = {
 	.d_version	= D_VERSION,
+	.d_flags	= D_TRACKCLOSE,
 	.d_open		= snp_open,
 	.d_read		= snp_read,
 	.d_write	= snp_write,

Re: New panic in main-n253273-a52d8d4a6c6:

[Rob Wing]

kinda thinking this might be related to commit f40dd6c8034b ("tty: switch
ttyhook_register to use fget_cap_locked")

fget_unlocked() must have grabbed an extra reference count on the file
pointer that fget_cap_locked() doesn't get

On Mon, Feb 21, 2022 at 4:35 PM Hans Petter Selasky  wrote:

> On 2/22/22 00:42, Michael Jung wrote:
> > Hi:
> >
> > I was trying to remember what I did that was odd when this crash
> occurred then it
> > hit me.  You can repeat this panic by doing:
> >
> > # watch -I -W pts/0
> >
> > Here is another panic that happened write after issuing "watch" for
> comparison.
> >
> > http://mail.mikej.com/core.txt.1
> >
> > http://mail.mikej.com/info.1
> >
> > http://mail.mikej.com/vmcore.1
> >
>
> I also need your kernel and debug kernel to fully debug this.
>
> 1) Do ssh to machine.
> 2) watch -i -W pts/0 (does not panic over here)
>
> Can you explain what step 1 is? An scp ?
>
> Refcount is -1.
> f_count = 0x
>
> f_data = 0xf800158b0400
>
> In your KGDB, can you enter:
>
> info 0x81b052d0
>
> Does the attached patch make any difference?
>
> --HPS

RE: New panic in main-n253273-a52d8d4a6c6:

[Michael Jung]

CONFIDENTIALITY NOTE: This message is intended only for the use
of the individual or entity to whom it is addressed and may
contain information that is privileged, confidential, and
exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying
of this communication is strictly prohibited. If you have
received this transmission in error, please notify us by
telephone at (502) 212-4000 or notify us at: PAI, Dept. 99,
2101 High Wickham Place, Suite 101, Louisville, KY 40245



From: Hans Petter Selasky [mailto:h...@selasky.org]
Sent: Monday, February 21, 2022 8:34 PM
To: Michael Jung ; freebsd-current 

Subject: Re: New panic in main-n253273-a52d8d4a6c6:

On 2/22/22 00:42, Michael Jung wrote:
> Hi:
>
> I was trying to remember what I did that was odd when this crash occurred 
> then it
> hit me. You can repeat this panic by doing:
>
> # watch -I -W pts/0
>
> Here is another panic that happened write after issuing "watch" for 
> comparison.
>
> http://mail.mikej.com/core.txt.1
>
> http://mail.mikej.com/info.1
>
> http://mail.mikej.com/vmcore.1
>

I also need your kernel and debug kernel to fully debug this.

1) Do ssh to machine.
2) watch -i -W pts/0 (does not panic over here)

Can you explain what step 1 is? An scp ?

Refcount is -1.
f_count = 0x

f_data = 0xf800158b0400

In your KGDB, can you enter:

info 0x81b052d0

Does the attached patch make any difference?

--HPS



Patch made no difference, instant panic.

These are will the patch applied.

http://mail.mikej.com/vmcore.2

http://mail.mikej.com/core.txt.2

http://mail.mikej.com/info.2

http://mail.mikej.com/kernel.2

http://mail.mikej.com/kernel.full.2

--mikej

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast, a leader in email security and cyber 
resilience. Mimecast integrates email defenses with brand protection, security 
awareness training, web security, compliance and other essential capabilities. 
Mimecast helps protect large and small organizations from malicious activity, 
human error and technology failure; and to lead the movement toward building a 
more resilient world. To find out more, visit our website.

RE: New panic in main-n253273-a52d8d4a6c6:

[Michael Jung]

CONFIDENTIALITY NOTE: This message is intended only for the use
of the individual or entity to whom it is addressed and may
contain information that is privileged, confidential, and
exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying
of this communication is strictly prohibited. If you have
received this transmission in error, please notify us by
telephone at (502) 212-4000 or notify us at: PAI, Dept. 99,
2101 High Wickham Place, Suite 101, Louisville, KY 40245



From: Michael Jung
Sent: Monday, February 21, 2022 9:06 PM
To: 'Hans Petter Selasky' ; freebsd-current 

Subject: RE: New panic in main-n253273-a52d8d4a6c6:

From: Hans Petter Selasky [mailto:h...@selasky.org]
Sent: Monday, February 21, 2022 8:34 PM
To: Michael Jung 
mailto:mi...@paymentallianceintl.com>>; 
freebsd-current 
mailto:freebsd-current@freebsd.org>>
Subject: Re: New panic in main-n253273-a52d8d4a6c6:

On 2/22/22 00:42, Michael Jung wrote:
> Hi:
>
> I was trying to remember what I did that was odd when this crash occurred 
> then it
> hit me. You can repeat this panic by doing:
>
> # watch -I -W pts/0
>
> Here is another panic that happened write after issuing "watch" for 
> comparison.
>
> http://mail.mikej.com/core.txt.1
>
> http://mail.mikej.com/info.1
>
> http://mail.mikej.com/vmcore.1
>

I also need your kernel and debug kernel to fully debug this.

1) Do ssh to machine.
2) watch -i -W pts/0 (does not panic over here)

Can you explain what step 1 is? An scp ?

Refcount is -1.
f_count = 0x

f_data = 0xf800158b0400

In your KGDB, can you enter:

info 0x81b052d0

Does the attached patch make any difference?

--HPS


So sorry, yes step one, ssh to machine….

Even open a second console on the computer, no SSH and do “watch –I  -W v0” it 
panics.

Example of “watch –I –W v0” for comparison – kernel/kernel.full have not 
changed since .2

http://mail.mikej.com/core.txt.3

http://mail.mikej.com/info.3

http://mail.mikej.com/vmcore.3

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast, a leader in email security and cyber 
resilience. Mimecast integrates email defenses with brand protection, security 
awareness training, web security, compliance and other essential capabilities. 
Mimecast helps protect large and small organizations from malicious activity, 
human error and technology failure; and to lead the movement toward building a 
more resilient world. To find out more, visit our website.

Re: New panic in main-n253273-a52d8d4a6c6:

[Rob Wing]

try the attached patch

On Mon, Feb 21, 2022 at 5:14 PM Michael Jung 
wrote:

>
>
>
>
>
> CONFIDENTIALITY NOTE: This message is intended only for the use
> of the individual or entity to whom it is addressed and may
> contain information that is privileged, confidential, and
> exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby
> notified that any dissemination, distribution or copying
> of this communication is strictly prohibited. If you have
> received this transmission in error, please notify us by
> telephone at (502) 212-4000 or notify us at: PAI, Dept. 99,
> 2101 High Wickham Place, Suite 101, Louisville, KY 40245
>
>
>
> *From:* Michael Jung
> *Sent:* Monday, February 21, 2022 9:06 PM
> *To:* 'Hans Petter Selasky' ; freebsd-current <
> freebsd-current@freebsd.org>
> *Subject:* RE: New panic in main-n253273-a52d8d4a6c6:
>
>
>
> *From:* Hans Petter Selasky [mailto:h...@selasky.org ]
> *Sent:* Monday, February 21, 2022 8:34 PM
> *To:* Michael Jung ; freebsd-current <
> freebsd-current@freebsd.org>
> *Subject:* Re: New panic in main-n253273-a52d8d4a6c6:
>
>
>
> On 2/22/22 00:42, Michael Jung wrote:
> > Hi:
> >
> > I was trying to remember what I did that was odd when this crash
> occurred then it
> > hit me. You can repeat this panic by doing:
> >
> > # watch -I -W pts/0
> >
> > Here is another panic that happened write after issuing "watch" for
> comparison.
> >
> > http://mail.mikej.com/core.txt.1
> >
> > http://mail.mikej.com/info.1
> >
> > http://mail.mikej.com/vmcore.1
> >
>
> I also need your kernel and debug kernel to fully debug this.
>
> 1) Do ssh to machine.
> 2) watch -i -W pts/0 (does not panic over here)
>
> Can you explain what step 1 is? An scp ?
>
> Refcount is -1.
> f_count = 0x
>
> f_data = 0xf800158b0400
>
> In your KGDB, can you enter:
>
> info 0x81b052d0
>
> Does the attached patch make any difference?
>
> --HPS
>
>
>
>
>
> So sorry, yes step one, ssh to machine….
>
>
>
> Even open a second console on the computer, no SSH and do “watch –I  -W
> v0” it panics.
>
>
>
> Example of “watch –I –W v0” for comparison – kernel/kernel.full have not
> changed since .2
>
>
>
> http://mail.mikej.com/core.txt.3
>
>
>
> http://mail.mikej.com/info.3
>
>
>
> http://mail.mikej.com/vmcore.3
>
>
>
>
>
>
>
>
> *Disclaimer*
>
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and may
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by Mimecast, a leader in email security and cyber
> resilience. Mimecast integrates email defenses with brand protection,
> security awareness training, web security, compliance and other essential
> capabilities. Mimecast helps protect large and small organizations from
> malicious activity, human error and technology failure; and to lead the
> movement toward building a more resilient world. To find out more, visit
> our website.
>
diff --git a/sys/kern/tty.c b/sys/kern/tty.c
index ebb32f698e88..e98b2d10b810 100644
--- a/sys/kern/tty.c
+++ b/sys/kern/tty.c
@@ -2083,6 +2083,8 @@ ttyhook_register(struct tty **rtp, struct proc *p, int fd, struct ttyhook *th,
 	FILEDESC_SLOCK(fdp);
 	error = fget_cap_locked(fdp, fd, cap_rights_init_one(&rights, CAP_TTYHOOK),
 	&fp, NULL);
+	if (error == 0 && !fhold(fp))
+		return (EBADF);
 	FILEDESC_SUNLOCK(fdp);
 	if (error != 0)
 		return (error);

Re: New panic in main-n253273-a52d8d4a6c6:

[Rob Wing]

The previous attached patch should work for testing purposes, but it's
incorrect if an error occurs.

The fhold() should be below FILEDESC_SUNLOCK(), this patch addresses that.

On Mon, Feb 21, 2022 at 5:18 PM Rob Wing  wrote:

> try the attached patch
>
> On Mon, Feb 21, 2022 at 5:14 PM Michael Jung <
> mi...@paymentallianceintl.com> wrote:
>
>>
>>
>>
>>
>>
>> CONFIDENTIALITY NOTE: This message is intended only for the use
>> of the individual or entity to whom it is addressed and may
>> contain information that is privileged, confidential, and
>> exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby
>> notified that any dissemination, distribution or copying
>> of this communication is strictly prohibited. If you have
>> received this transmission in error, please notify us by
>> telephone at (502) 212-4000 or notify us at: PAI, Dept. 99,
>> 2101 High Wickham Place, Suite 101, Louisville, KY 40245
>>
>>
>>
>> *From:* Michael Jung
>> *Sent:* Monday, February 21, 2022 9:06 PM
>> *To:* 'Hans Petter Selasky' ; freebsd-current <
>> freebsd-current@freebsd.org>
>> *Subject:* RE: New panic in main-n253273-a52d8d4a6c6:
>>
>>
>>
>> *From:* Hans Petter Selasky [mailto:h...@selasky.org ]
>> *Sent:* Monday, February 21, 2022 8:34 PM
>> *To:* Michael Jung ; freebsd-current <
>> freebsd-current@freebsd.org>
>> *Subject:* Re: New panic in main-n253273-a52d8d4a6c6:
>>
>>
>>
>> On 2/22/22 00:42, Michael Jung wrote:
>> > Hi:
>> >
>> > I was trying to remember what I did that was odd when this crash
>> occurred then it
>> > hit me. You can repeat this panic by doing:
>> >
>> > # watch -I -W pts/0
>> >
>> > Here is another panic that happened write after issuing "watch" for
>> comparison.
>> >
>> > http://mail.mikej.com/core.txt.1
>> >
>> > http://mail.mikej.com/info.1
>> >
>> > http://mail.mikej.com/vmcore.1
>> >
>>
>> I also need your kernel and debug kernel to fully debug this.
>>
>> 1) Do ssh to machine.
>> 2) watch -i -W pts/0 (does not panic over here)
>>
>> Can you explain what step 1 is? An scp ?
>>
>> Refcount is -1.
>> f_count = 0x
>>
>> f_data = 0xf800158b0400
>>
>> In your KGDB, can you enter:
>>
>> info 0x81b052d0
>>
>> Does the attached patch make any difference?
>>
>> --HPS
>>
>>
>>
>>
>>
>> So sorry, yes step one, ssh to machine….
>>
>>
>>
>> Even open a second console on the computer, no SSH and do “watch –I  -W
>> v0” it panics.
>>
>>
>>
>> Example of “watch –I –W v0” for comparison – kernel/kernel.full have not
>> changed since .2
>>
>>
>>
>> http://mail.mikej.com/core.txt.3
>>
>>
>>
>> http://mail.mikej.com/info.3
>>
>>
>>
>> http://mail.mikej.com/vmcore.3
>>
>>
>>
>>
>>
>>
>>
>>
>> *Disclaimer*
>>
>> The information contained in this communication from the sender is
>> confidential. It is intended solely for use by the recipient and others
>> authorized to receive it. If you are not the recipient, you are hereby
>> notified that any disclosure, copying, distribution or taking action in
>> relation of the contents of this information is strictly prohibited and may
>> be unlawful.
>>
>> This email has been scanned for viruses and malware, and may have been
>> automatically archived by Mimecast, a leader in email security and cyber
>> resilience. Mimecast integrates email defenses with brand protection,
>> security awareness training, web security, compliance and other essential
>> capabilities. Mimecast helps protect large and small organizations from
>> malicious activity, human error and technology failure; and to lead the
>> movement toward building a more resilient world. To find out more, visit
>> our website.
>>
>
diff --git a/sys/kern/tty.c b/sys/kern/tty.c
index ebb32f698e88..2c60a7e42580 100644
--- a/sys/kern/tty.c
+++ b/sys/kern/tty.c
@@ -2086,6 +2086,8 @@ ttyhook_register(struct tty **rtp, struct proc *p, int fd, struct ttyhook *th,
 	FILEDESC_SUNLOCK(fdp);
 	if (error != 0)
 		return (error);
+	if (!fhold(fp))
+		return (EBADF);
 	if (fp->f_ops == &badfileops) {
 		error = EBADF;
 		goto done1;

RE: New panic in main-n253273-a52d8d4a6c6:

[Michael Jung]

Rob:

I did not remove Hans earlier patch, but your “latest” patch applied and 
compiled with no errors.

I can now watch and control a TTY (v0) or a SSH session (pts/1) without 
crashing.

Anything else you would like me to try?

--mikej


From: Rob Wing [mailto:rob.fx...@gmail.com]
Sent: Monday, February 21, 2022 9:30 PM
To: Michael Jung 
Cc: Hans Petter Selasky ; freebsd-current 

Subject: Re: New panic in main-n253273-a52d8d4a6c6:

The previous attached patch should work for testing purposes, but it's 
incorrect if an error occurs.

The fhold() should be below FILEDESC_SUNLOCK(), this patch addresses that.

On Mon, Feb 21, 2022 at 5:18 PM Rob Wing 
mailto:rob.fx...@gmail.com>> wrote:
try the attached patch

On Mon, Feb 21, 2022 at 5:14 PM Michael Jung 
mailto:mi...@paymentallianceintl.com>> wrote:




CONFIDENTIALITY NOTE: This message is intended only for the use
of the individual or entity to whom it is addressed and may
contain information that is privileged, confidential, and
exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying
of this communication is strictly prohibited. If you have
received this transmission in error, please notify us by
telephone at (502) 212-4000 or notify us at: PAI, Dept. 99,
2101 High Wickham Place, Suite 101, Louisville, KY 40245


From: Michael Jung
Sent: Monday, February 21, 2022 9:06 PM
To: 'Hans Petter Selasky' mailto:h...@selasky.org>>; 
freebsd-current 
mailto:freebsd-current@freebsd.org>>
Subject: RE: New panic in main-n253273-a52d8d4a6c6:

From: Hans Petter Selasky [mailto:h...@selasky.org]
Sent: Monday, February 21, 2022 8:34 PM
To: Michael Jung 
mailto:mi...@paymentallianceintl.com>>; 
freebsd-current 
mailto:freebsd-current@freebsd.org>>
Subject: Re: New panic in main-n253273-a52d8d4a6c6:

On 2/22/22 00:42, Michael Jung wrote:
> Hi:
>
> I was trying to remember what I did that was odd when this crash occurred 
> then it
> hit me. You can repeat this panic by doing:
>
> # watch -I -W pts/0
>
> Here is another panic that happened write after issuing "watch" for 
> comparison.
>
> http://mail.mikej.com/core.txt.1
>
> http://mail.mikej.com/info.1
>
> http://mail.mikej.com/vmcore.1
>

I also need your kernel and debug kernel to fully debug this.

1) Do ssh to machine.
2) watch -i -W pts/0 (does not panic over here)

Can you explain what step 1 is? An scp ?

Refcount is -1.
f_count = 0x

f_data = 0xf800158b0400

In your KGDB, can you enter:

info 0x81b052d0

Does the attached patch make any difference?

--HPS


So sorry, yes step one, ssh to machine….

Even open a second console on the computer, no SSH and do “watch –I  -W v0” it 
panics.

Example of “watch –I –W v0” for comparison – kernel/kernel.full have not 
changed since .2

http://mail.mikej.com/core.txt.3

http://mail.mikej.com/info.3

http://mail.mikej.com/vmcore.3





Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast, a leader in email security and cyber 
resilience. Mimecast integrates email defenses with brand protection, security 
awareness training, web security, compliance and other essential capabilities. 
Mimecast helps protect large and small organizations from malicious activity, 
human error and technology failure; and to lead the movement toward building a 
more resilient world. To find out more, visit our website.

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast, a leader in email security and cyber 
resilience. Mimecast integrates email defenses with brand protection, security 
awareness training, web security, compliance and other essential capabilities. 
Mimecast helps protect large and small organizations from malicious activity, 
human error and technology failure; and to lead the movement toward building a 
more resilient world. To find out more, visit our website.

Re: New panic in main-n253273-a52d8d4a6c6:

[Rob Wing]

Think that's it...thanks for testing for the patch.

I opened up https://reviews.freebsd.org/D34335 for review.

-Rob

On Mon, Feb 21, 2022 at 6:05 PM Michael Jung 
wrote:

> Rob:
>
>
>
> I did not remove Hans earlier patch, but your “latest” patch applied and
> compiled with no errors.
>
>
>
> I can now watch and control a TTY (v0) or a SSH session (pts/1) without
> crashing.
>
>
>
> Anything else you would like me to try?
>
>
>
> --mikej
>
>
>
>
>
> *From:* Rob Wing [mailto:rob.fx...@gmail.com]
> *Sent:* Monday, February 21, 2022 9:30 PM
> *To:* Michael Jung 
> *Cc:* Hans Petter Selasky ; freebsd-current <
> freebsd-current@freebsd.org>
> *Subject:* Re: New panic in main-n253273-a52d8d4a6c6:
>
>
>
> The previous attached patch should work for testing purposes, but it's
> incorrect if an error occurs.
>
>
>
> The fhold() should be below FILEDESC_SUNLOCK(), this patch addresses that.
>
>
>
> On Mon, Feb 21, 2022 at 5:18 PM Rob Wing  wrote:
>
> try the attached patch
>
>
>
> On Mon, Feb 21, 2022 at 5:14 PM Michael Jung <
> mi...@paymentallianceintl.com> wrote:
>
>
>
>
>
>
>
> CONFIDENTIALITY NOTE: This message is intended only for the use
> of the individual or entity to whom it is addressed and may
> contain information that is privileged, confidential, and
> exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby
> notified that any dissemination, distribution or copying
> of this communication is strictly prohibited. If you have
> received this transmission in error, please notify us by
> telephone at (502) 212-4000 or notify us at: PAI, Dept. 99,
> 2101 High Wickham Place, Suite 101, Louisville, KY 40245
>
>
> *From:* Michael Jung
> *Sent:* Monday, February 21, 2022 9:06 PM
> *To:* 'Hans Petter Selasky' ; freebsd-current <
> freebsd-current@freebsd.org>
> *Subject:* RE: New panic in main-n253273-a52d8d4a6c6:
>
>
>
> *From:* Hans Petter Selasky [mailto:h...@selasky.org ]
> *Sent:* Monday, February 21, 2022 8:34 PM
> *To:* Michael Jung ; freebsd-current <
> freebsd-current@freebsd.org>
> *Subject:* Re: New panic in main-n253273-a52d8d4a6c6:
>
>
>
> On 2/22/22 00:42, Michael Jung wrote:
> > Hi:
> >
> > I was trying to remember what I did that was odd when this crash
> occurred then it
> > hit me. You can repeat this panic by doing:
> >
> > # watch -I -W pts/0
> >
> > Here is another panic that happened write after issuing "watch" for
> comparison.
> >
> > http://mail.mikej.com/core.txt.1
> >
> > http://mail.mikej.com/info.1
> >
> > http://mail.mikej.com/vmcore.1
> >
>
> I also need your kernel and debug kernel to fully debug this.
>
> 1) Do ssh to machine.
> 2) watch -i -W pts/0 (does not panic over here)
>
> Can you explain what step 1 is? An scp ?
>
> Refcount is -1.
> f_count = 0x
>
> f_data = 0xf800158b0400
>
> In your KGDB, can you enter:
>
> info 0x81b052d0
>
> Does the attached patch make any difference?
>
> --HPS
>
>
>
>
>
> So sorry, yes step one, ssh to machine….
>
>
>
> Even open a second console on the computer, no SSH and do “watch –I  -W
> v0” it panics.
>
>
>
> Example of “watch –I –W v0” for comparison – kernel/kernel.full have not
> changed since .2
>
>
>
> http://mail.mikej.com/core.txt.3
>
>
>
> http://mail.mikej.com/info.3
>
>
>
> http://mail.mikej.com/vmcore.3
>
>
>
>
>
>
>
>
>
> *Disclaimer*
>
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and may
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by Mimecast, a leader in email security and cyber
> resilience. Mimecast integrates email defenses with brand protection,
> security awareness training, web security, compliance and other essential
> capabilities. Mimecast helps protect large and small organizations from
> malicious activity, human error and technology failure; and to lead the
> movement toward building a more resilient world. To find out more, visit
> our website.
>
>
>
> *Disclaimer*
>
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and may
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by Mimecast, a leader in email security and cyber
> resilience. Mimecast integrates email defenses with brand protection,
> security awareness training, web security, compliance and other essential
> capabilities