>Number: 148698
>Category: kern
>Synopsis: Panic at boot due to integer overflow computing top->cg_mask
>in smp_topo_none() when mp_ncpus == 32
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Fri Jul 16 23:10:07 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator: Joe Landers
>Release: FreeBSD 8.0 (amd64)
>Organization:
VMware, Inc
>Environment:
Any system with 32 CPUs.
>Description:
Attempting to boot FreeBSD 8.0 (amd64) on machine with 32 CPUs causes the
kernel to panic. panic() calls boot() and boot() calls thread_lock(curthread),
however curthread->td_lock is NULL so the system starts taking repeated page
faults and printing to the screen.
FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
FreeBSD/SMP: 32 package(s) x 1 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
cpu4 (AP): APIC ID: 4
cpu5 (AP): APIC ID: 5
cpu6 (AP): APIC ID: 6
cpu7 (AP): APIC ID: 7
cpu8 (AP): APIC ID: 8
cpu9 (AP): APIC ID: 9
cpu10 (AP): APIC ID: 10
cpu11 (AP): APIC ID: 11
cpu12 (AP): APIC ID: 12
cpu13 (AP): APIC ID: 13
cpu14 (AP): APIC ID: 14
cpu15 (AP): APIC ID: 15
cpu16 (AP): APIC ID: 16
cpu17 (AP): APIC ID: 17
cpu18 (AP): APIC ID: 18
cpu19 (AP): APIC ID: 19
cpu20 (AP): APIC ID: 20
cpu21 (AP): APIC ID: 21
cpu22 (AP): APIC ID: 22
cpu23 (AP): APIC ID: 23
cpu24 (AP): APIC ID: 24
cpu25 (AP): APIC ID: 25
cpu26 (AP): APIC ID: 26
cpu27 (AP): APIC ID: 27
cpu28 (AP): APIC ID: 28
cpu29 (AP): APIC ID: 29
cpu30 (AP): APIC ID: 30
cpu31 (AP): APIC ID: 31
panic: Built bad topology at 0xffffffff8081fc20. CPU mask 0x0 != 0xFFFFFFFF
cpuid = 0
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x18
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80322e28
stack pointer = 0x28:0xffffffff809b0b20
frame pointer = 0x28:0xffffffff809b0b60
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 0 (swapper)
trap number = 12
panic: page fault
cpuid = 0
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x18
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80322e28
stack pointer = 0x28:0xffffffff809b0790
frame pointer = 0x28:0xffffffff809b07d0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 0 (swapper)
trap number = 12
panic: page fault
cpuid = 0
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x18
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80322e28
stack pointer = 0x28:0xffffffff809b0400
frame pointer = 0x28:0xffffffff809b0440
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 0 (swapper)
trap number = 12
panic: page fault
cpuid = 0
kernel trap 12 with interrupts disabled
.. <repeats> ...
When the hardware has 32 CPUs, the computation of top->cg_mask in
smp_topo_none() overflows and becomes zero. This causes smp_topo() to call
panic() with the message about the topology mismatch.
struct cpu_group *
smp_topo_none(void)
{
struct cpu_group *top;
top = &group[0];
top->cg_parent = NULL;
top->cg_child = NULL;
top->cg_mask = (1 << mp_ncpus) - 1; <<<<<<<<<<<<<<<<
top->cg_count = mp_ncpus;
top->cg_children = 0;
top->cg_level = CG_SHARE_NONE;
top->cg_flags = 0;
return (top);
}
0xffffffff805b9743 <smp_topo_none+51>: shl %cl,%eax <<<<<<<<
0xffffffff805b9745 <smp_topo_none+53>: mov %cl,6632169(%rip) #
0xffffffff80c0ca34 <group+20>
0xffffffff805b974b <smp_topo_none+59>: movb $0x0,6632165(%rip) #
0xffffffff80c0ca37 <group+23>
0xffffffff805b9752 <smp_topo_none+66>: sub $0x1,%eax
0xffffffff805b9755 <smp_topo_none+69>: mov %eax,6632149(%rip) #
0xffffffff80c0ca30 <group+16>
0xffffffff805b975b <smp_topo_none+75>: leaveq
0xffffffff805b975c <smp_topo_none+76>: mov $0xffffffff80c0ca20,%rax
0xffffffff805b9763 <smp_topo_none+83>: retq
static int
start_all_aps(void)
{
..
all_cpus |= (1 << cpu); /* record AP in CPU map */
}
..
}
struct cpu_group *
smp_topo(void)
{
/*
* Verify the returned topology.
*/
if (top->cg_count != mp_ncpus)
panic("Built bad topology at %p. CPU count %d != %d",
top, top->cg_count, mp_ncpus);
if (top->cg_mask != all_cpus) <<<<<<
panic("Built bad topology at %p. CPU mask 0x%X != 0x%X",
top, top->cg_mask, all_cpus);
return (top);
}
(gdb) x/xw 0xc00000 + (0xffffffff80c0ca20 & 0x1fffff) + 0x10
0xc0ca30: 0x00000000 // top->cg_mask
(gdb) x/xw 0xc00000 + (0xffffffff80c0ca20 & 0x1fffff) + 0x14
0xc0ca34: 0x00000020 // top->cg_count
(gdb) p &all_cpus
$16 = (cpumask_t *) 0xffffffff80c0c9b8
(gdb) x/xw 0xc00000 + (0xffffffff80c0c9b8 & 0x1fffff)
0xc0c9b8: 0xffffffff // all_cpus
(gdb) p &mp_ncpus
$15 = (int *) 0xffffffff80c0c9b0
(gdb) x/xw 0xc00000 + (0xffffffff80c0c9b0 & 0x1fffff)
0xc0c9b0: 0x00000020 // mp_ncpus
0x121db30: 0xffffff00bfed7000 0x0000000000000104
0x121db40: 0xffffffff80951948 0xffffffff80be7600
0x121db50: 0x0000000000000000 0x0000000000000000
0x121db60: 0xffffffff8121dbb0 0xffffffff8057f41f // boot+47
0xffffffff8057f418 <boot+40>: xor %esi,%esi
0xffffffff8057f41a <boot+42>: callq 0xffffffff80571290 <_thread_lock_flags>
0xffffffff8057f41f <boot+47>: mov %gs:0x0,%rdi
0x121db70: 0xffffffff80951948 0xffffffff80be7600
0x121db80: 0x0000000000000000 0x0000000000000104
0x121db90: 0xffffffff80951948 0xffffffff80be7600
0x121dba0: 0x0000000000000000 0x0000000000000000
0x121dbb0: 0xffffffff8121dcb0 0xffffffff8057fcfc // panic+332
0x121dbc0: 0x0000003000000020 0xffffffff8121dcc0
0x121dbd0: 0xffffffff8121dbe0 0x00000000bfed9560
0x121dbe0: 0xffffffff8121dc70 0xffffffff80c0ca20
0x121dbf0: 0x0000000000000000 0x00000000ffffffff
0x121dc00: 0x0000000000000000 0x0000000000000001
0x121dc10: 0xffffff00bfed94e0 0x0000000200000202
0x121dc20: 0xffffff00bfed94e8 0x0000010280b60920
0x121dc30: 0x0000000000000000 0xffffffff80b60920
0x121dc40: 0xffffffff8121dc80 0xffffffff8056db9b
0x121dc50: 0x00000000023a3c80 0xffffff00bfed94c0
0x121dc60: 0x00000000023a3c80 0xffffff00bfed94c0
0x121dc70: 0xffffff00023b0d00 0xffffffff8098dfe8
0x121dc80: 0xffffffff80bf1240 0x0000000000000000
0x121dc90: 0x0000000000000000 0xffffffff8098dfe8
0x121dca0: 0xffffffff80bf1240 0x0000000000000000
0x121dcb0: 0xffffffff8121dcc0 0xffffffff805b9bc7 // smp_topo+263
0xffffffff80951948 <basefix.2920+424>: "Built bad topology at %p. CPU mask 0x%X
!= 0x%X"
0xffffffff805b9bb9 <smp_topo+249>: mov $0xffffffff80951948,%rdi
0xffffffff805b9bc0 <smp_topo+256>: xor %eax,%eax
0xffffffff805b9bc2 <smp_topo+258>: callq 0xffffffff8057fbb0 <panic>
>How-To-Repeat:
Our engineering team originally came across this issue testing FreeBSD-8.0
(amd64) running on large number of virtual CPUs running as a guest on VMware
ESX.
Subsequently, engineering was able to reproduce this issue on physical
hardware, booting the FreeBSD-8.0 (amd64) install disk on an IBM x3950 (4 node)
Athena M2. The physical system panic()'s identically there as well.
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
_______________________________________________
freebsd-bugs@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
