Re: head -r325997: Fatal trap 12: page fault while in kernel mode (during a buildworld, virtualbox guest context) [2nd example]
[Adding some analysis of where the 2 failures were in
source code terms.]
On 2017-Nov-19, at 9:07 PM, Mark Millard wrote:
> [I got another of these. By the way: amd64 context.
> Again: buildworld was running.]
>
> On 2017-Nov-19, at 5:52 PM, Mark Millard wrote:
>
>> Attempting a dump failed. I'm afraid all for
>> information is the below. The kernel was a
>> non-debug kernel (with debug information).
>>
>> The following is hand typed from a screen shot:
>>
>> Fatal trap 12: page fault while in kernel mode
>> cpuid = 0; apic id = 00
>> fault virtual address = 0xff53f000e2b0
>
> New one: 0x806b49010
>
>> fault code= supervisor read data, page not present
>
> New one: supervisor write data, page not present
>
>> instruction pointer = 0x20:0x80f2b11e
>
> New one: 0x20:0x80f2b21b
>
>> stack pointer = 0x0:0xfe01aeb28970
>
> New one: 0x28:0xfe01aeb28970
>
>> frame pointer = 0x0:0xfe01aeb289f0
>
> New one: 0x28:0xfe01aeb289f0
>
>> code segment = base 0x0, limit 0xf, type 0x1b
>> = DPL 0, pres 1, long 1, def32 0, gran 1
>> processor eflags = interrupt enabled, resume, IOPL = 0
>> current process = 20 (pagedaemon)
>> [ thread pid 20 tid 100089 ]
>> Stopped at pmap_ts_referenced+0x72e: movq (%rcx,rdi,8),%rbx
>
> New one: pmap_ts_referenced+0x82b: movq %rcx,0x10(%rax)
>
>> bd > bt
>> Tracing pid 20 tid 100089 td 0xf80003eb3560
>
> New one:td 0xf80003df6000
>
>> pmap_ts_referenced() at pmap_ts_referenced_0x72e/frame 0xfe01aeb289f0
> New one:
> pmap_ts_referenced() at pmap_ts_referenced_0x82b/frame 0xfe01aeb289f0
>
>> vm_pageout() at vm_pageout+0xdeb/frame 0xfe01aeb28ab0
>
> Correction to original:frame 0xfe01aeb28a70
> (new is the same)
>
>> fork_exit() at fork_exit+0x82/frame 0xfe01aeb28ab0
>> fork_trampoline() at fork_trampoline+0xe/frame 0xfe01aeb28ab0
>> --- trap 0, rip = 0, rsp = 0, rpb = 0 ---
>> db>
>>
>> The prior (cross) buildworld buildkernel had completed fine.
>>
>> Until yesterday, I'd been running -r325700 or before and had not
>> seen such an issue ever before. I'd been using the virtualbox
>> version for a while before this as well.
Taking the case of:
Stopped at pmap_ts_referenced+0x72e: movq (%rcx,rdi,8),%rbx:
80f2b0fc mov%rax,%rsi
80f2b0ff shr$0x1b,%rsi
80f2b103 and$0xff8,%esi
80f2b109 mov(%rcx,%rsi,1),%rcx
80f2b10d and%r10,%rcx
80f2b110 or %r9,%rcx
80f2b113 mov%eax,%edi
80f2b115 shr$0x15,%edi
80f2b118 and$0x1ff,%edi
80f2b11e mov(%rcx,%rdi,8),%rbx
<<===
80f2b122 and%r10,%rbx
80f2b125 or %r9,%rbx
80f2b128 shr$0x9,%rax
80f2b12c and$0xff8,%eax
80f2b131 lea(%rbx,%rax,1),%rsi
80f2b135 mov(%rbx,%rax,1),%rbx
80f2b139 mov%rbx,%rax
80f2b13c and%rdx,%rax
80f2b13f cmp%rdx,%rax
80f2b142 jne80f2b14f
Which, if I understand right, is in the
"small_mappings:" code:
PG_A = pmap_accessed_bit(pmap);
PG_M = pmap_modified_bit(pmap);
PG_RW = pmap_rw_bit(pmap);
pde = pmap_pde(pmap, pv->pv_va);
KASSERT((*pde & PG_PS) == 0,
("pmap_ts_referenced: found a 2mpage in page %p's pv list",
m));
pte = pmap_pde_to_pte(pde, pv->pv_va);
if ((*pte & (PG_M | PG_RW)) == (PG_M | PG_RW))
vm_page_dirty(m);
if ((*pte & PG_A) != 0) {
with the failure being during *pde in:
/* Return a pointer to the PT slot that corresponds to a VA */
static __inline pt_entry_t *
pmap_pde_to_pte(pd_entry_t *pde, vm_offset_t va)
{
pt_entry_t *pte;
pte = (pt_entry_t *)PHYS_TO_DMAP(*pde & PG_FRAME);
return (&pte[pmap_pte_index(va)]);
}
Taking the case of:
New one: pmap_ts_referenced+0x82b: movq %rcx,0x10(%rax)
80f2b1fb lock cmpxchg %rcx,(%rdx)
80f2b200 sete %cl
80f2b203 test %cl,%cl
80f2b205 je 80f2b27d
80f2b207 test %r12,%r12
80f2b20a je 80f2b255
80f2b20c mov0x8(%r12),%rax
80f2b211 test %rax,%rax
80f2b214 je 80f2b255
80f2b216 mov0x10(%r12),%rcx
80f2b21b mov%rcx,0x10(%rax)
<=
80f2b21f mov0x8(%r12),%rax
80f2b224 mov0x10(%r12),%rcx
80f2b229 mov%rax,(%rcx)
Which, if I understand right, appears to be during
the TAILQ_REMOVE of:
PMAP_UNLOCK(pmap);
/* Rotate the PV list if it has more than one entry. */
if (pv != NULL &&
getfsstat / nullsfs / automount bug?
There appears to be a bug in the system call related to getmntinfo(3) /
getfsstat(2)
, when if the "automount" flag is set on "nullfs" mounts, it is not returned on
a getfsstat "WAIT" call. The non-refreshed, non-blocking "MNT_NOWAIT" produces
the
correct result.
I first noticed this when debugging why my nullfs autofs partitions weren't
being
automatically unmounted. (FreeBSD current and release)
The somewhat hacked snippet from automount.c in the example below demonstrates
this -
When called with no parameters, it performs a MNT_WAIT request, otherwise a
MNT_NOWAIT
request is performed:
Cheers, Jamie
| #include
| #include
| #include
|
| int main(int argc, char **argv)
| {
| struct statfs *mntbuf;
| int i, nitems;
|
| nitems = getmntinfo(&mntbuf, (argc == 1) ? MNT_WAIT : MNT_NOWAIT);
| if (nitems <= 0)
| printf ("getmntinfo fail\n");
|
| for (i = 0; i < nitems; i++) {
| if (strcmp(mntbuf[i].f_fstypename, "autofs") == 0) {
| printf("skipping %s, filesystem type is autofs\n",
| mntbuf[i].f_mntonname);
| continue;
| }
|
| if ((mntbuf[i].f_flags & MNT_AUTOMOUNTED) == 0) {
| printf("skipping %s, not automounted\n",
| mntbuf[i].f_mntonname);
| continue;
| }
|
| printf("%s IS automounted!\n",
| mntbuf[i].f_mntonname);
| }
| }
| 14:24 [2] (1) "autofs" root@thompson# df
| Filesystem1K-blocks Used Avail Capacity Mounted on
| /dev/ada1p2 5061628 707264394943615%/
| devfs 11 0 100%/dev
| /dev/ada1p4 5061628 535004412169611%/var
| /dev/ada1p5 978973296 21689416 878966020 2%/usr
|
| 14:24 [2] (2) "autofs" root@thompson# mount
| /dev/ada1p2 on / (ufs, local)
| devfs on /dev (devfs, local, multilabel)
| /dev/ada1p4 on /var (ufs, local, soft-updates)
| /dev/ada1p5 on /usr (ufs, local, soft-updates)
|
| 14:25 [2] (3) "autofs" root@thompson# mkdir /tmp/automounted /tmp/manual
|
| 14:25 [2] (4) "autofs" root@thompson# mount -t nullfs -o ro /usr/src
/tmp/manual
|
| 14:25 [2] (5) "autofs" root@thompson# mount -t nullfs -o ro,automounted
/usr/src /tmp/automounted/
|
| 14:26 [2] (6) "autofs" root@thompson# df
| Filesystem1K-blocks Used Avail Capacity Mounted on
| /dev/ada1p2 5061628 707264394943615%/
| devfs 11 0 100%/dev
| /dev/ada1p4 5061628 535004412169611%/var
| /dev/ada1p5 978973296 21689420 878966016 2%/usr
| /usr/src 978973296 21689420 878966016 2%/tmp/manual
| /usr/src 978973296 21689420 878966016 2%/tmp/automounted
|
| 14:26 [2] (7) "autofs" root@thompson# mount
| /dev/ada1p2 on / (ufs, local)
| devfs on /dev (devfs, local, multilabel)
| /dev/ada1p4 on /var (ufs, local, soft-updates)
| /dev/ada1p5 on /usr (ufs, local, soft-updates)
| /usr/src on /tmp/manual (nullfs, local, read-only)
| /usr/src on /tmp/automounted (nullfs, local, read-only, automounted)
|
| 14:26 [2] (8) "autofs" root@thompson# ./a.out
| skipping /, not automounted
| skipping /dev, not automounted
| skipping /var, not automounted
| skipping /usr, not automounted
| skipping /tmp/manual, not automounted
| skipping /tmp/automounted, not automounted
|
| 14:26 [2] (9) "autofs" root@thompson# ./a.out x
| skipping /, not automounted
| skipping /dev, not automounted
| skipping /var, not automounted
| skipping /usr, not automounted
| skipping /tmp/manual, not automounted
| /tmp/automounted IS automounted!
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Loader.conf problem
I recently upgraded my FreeBSD-Current system today and did not fully understand the impact of changes to loader.conf. My system has a Geli encrypted ZFS pool. In the past whenever I screwed up, I was able to use 'Beadm' into boot the most recent good kernel or enter a shell to reread the UPDATING notes for a clearer understanding of what needs to be done before rebooting. Is my system recoverable or does it require wiping and reloading from a snapshot? If it is recoverable, how to do? Tom -- Public Keys: PGP KeyID = 0x5F22FDC1 GnuPG KeyID = 0x620836CF ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Loader.conf problem
On Mon, Nov 20, 2017 at 1:30 PM, Thomas Laus wrote: > I recently upgraded my FreeBSD-Current system today and did not fully > understand the impact of changes to loader.conf. My system has a Geli > encrypted ZFS pool. In the past whenever I screwed up, I was able to > use 'Beadm' into boot the most recent good kernel or enter a shell to > reread the UPDATING notes for a clearer understanding of what needs to > be done before rebooting. > > Is my system recoverable or does it require wiping and reloading from a > snapshot? If it is recoverable, how to do? > My changes to the system hasn't changed how things work. Or at least shouldn't have changed anything. If they do, that's likely a bug, and likely my fault. The one exception is if you're booting off GELI + ZFS with UEFI based on boot1.efi changes that were in the tree for a few days... Warner ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Emacs and LLD
On 3 November 2017 at 11:29, Tobias Kortkamp wrote: > > My src.conf has WITH_LLD_IS_LD=yes and reading > https://bugs.freebsd.org/214864 leads me to believe that it's somehow > responsible for the problems I have with Emacs. Yes, the emacs build does some rather unusual things and it's perhaps not surprising that it's one of the ports that's giving grief with lld. The exp-run shows the same error you experienced: ./temacs --batch --load loadup bootstrap Fatal error 'Can't allocate initial thread' at line 337 in file /poudriere/jails/headamd64PR214864/usr/src/lib/libthr/thread/thr_init.c (errno = 12) gmake[2]: *** [Makefile:737: bootstrap-emacs] Abort trap (core dumped) I don't yet have any insight into the failure, and hope that someone with knowledge of the emacs build process and emacs internals can take a look. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Loader.conf problem
Warner Losh [i...@bsdimp.com] wrote: > On Mon, Nov 20, 2017 at 1:30 PM, Thomas Laus wrote: > > > I recently upgraded my FreeBSD-Current system today and did not fully > > understand the impact of changes to loader.conf. My system has a Geli > > encrypted ZFS pool. In the past whenever I screwed up, I was able to > > use 'Beadm' into boot the most recent good kernel or enter a shell to > > reread the UPDATING notes for a clearer understanding of what needs to > > be done before rebooting. > > > > Is my system recoverable or does it require wiping and reloading from a > > snapshot? If it is recoverable, how to do? > > > > My changes to the system hasn't changed how things work. Or at least > shouldn't have changed anything. If they do, that's likely a bug, and > likely my fault. > > The one exception is if you're booting off GELI + ZFS with UEFI based on > boot1.efi changes that were in the tree for a few days... > This PC is not using UEFI to boot. After booting I get the following message: gptzfsboot: No ZFS pools located; Cant't boot That is as far as things go. I read the UPDATING file before I builtworld and built a kernel. This was the first boot after doing an installworld and performing mergemaster and then copying the /boot/gptzfsboot file to ada0p1 and ada1p1. Tom -- Public Keys: PGP KeyID = 0x5F22FDC1 GnuPG KeyID = 0x620836CF ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Loader.conf problem
On Mon, Nov 20, 2017 at 2:42 PM, Thomas Laus wrote: > Warner Losh [i...@bsdimp.com] wrote: > > On Mon, Nov 20, 2017 at 1:30 PM, Thomas Laus wrote: > > > > > I recently upgraded my FreeBSD-Current system today and did not fully > > > understand the impact of changes to loader.conf. My system has a Geli > > > encrypted ZFS pool. In the past whenever I screwed up, I was able to > > > use 'Beadm' into boot the most recent good kernel or enter a shell to > > > reread the UPDATING notes for a clearer understanding of what needs to > > > be done before rebooting. > > > > > > Is my system recoverable or does it require wiping and reloading from a > > > snapshot? If it is recoverable, how to do? > > > > > > > My changes to the system hasn't changed how things work. Or at least > > shouldn't have changed anything. If they do, that's likely a bug, and > > likely my fault. > > > > The one exception is if you're booting off GELI + ZFS with UEFI based on > > boot1.efi changes that were in the tree for a few days... > > > This PC is not using UEFI to boot. After booting I get the following > message: > > gptzfsboot: No ZFS pools located; Cant't boot > > That is as far as things go. I read the UPDATING file before I > builtworld and built a kernel. This was the first boot after doing an > installworld and performing mergemaster and then copying the > /boot/gptzfsboot file to ada0p1 and ada1p1. Can you revert this part of the change? Go back to a known-working version of those files? Let me know if that fixes the problem? Bonus points for bisecting which one of the few dozen commits I did that caused this regression... I don't have a GELI + ZFS test bed here. Warner ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Emacs and LLD
On Mon, Nov 20, 2017 at 2:39 PM, Ed Maste wrote: > On 3 November 2017 at 11:29, Tobias Kortkamp wrote: > > > > My src.conf has WITH_LLD_IS_LD=yes and reading > > https://bugs.freebsd.org/214864 leads me to believe that it's somehow > > responsible for the problems I have with Emacs. > > Yes, the emacs build does some rather unusual things and it's perhaps > not surprising that it's one of the ports that's giving grief with > lld. > > The exp-run shows the same error you experienced: > > ./temacs --batch --load loadup bootstrap > Fatal error 'Can't allocate initial thread' at line 337 in file > /poudriere/jails/headamd64PR214864/usr/src/lib/libthr/thread/thr_init.c > (errno = 12) > gmake[2]: *** [Makefile:737: bootstrap-emacs] Abort trap (core dumped) > > I don't yet have any insight into the failure, and hope that someone > with knowledge of the emacs build process and emacs internals can take > a look. > Is temacs still an 'undumped' core dump[*]? Maybe the undumping code isn't playing well with lld's different behavior than ld? Warner [*] For speed, emacs use to compile all its lisp, load it into a memory arena, then take a core dump. The core dump was cleaned up so it could be used as an executable with the now-preloaded lisp code in place. It was a standard thing on 'big iron' that EMACS came from, but always a bit of an 'odd duck' as far as fitting into how Unix works. There used to be knobs to do it differently for things like VMS that simply couldn't easily cope. Maybe one of those needs to be tweaked? It's a tiny bit slower, but with the speed of today's hardware (and most hardware made since ~2000), the optimization likely saves time below the human threshold to detect... ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Loader.conf problem
Warner Losh [i...@bsdimp.com] wrote: > > Can you revert this part of the change? Go back to a known-working version > of those files? Let me know if that fixes the problem? Bonus points for > bisecting which one of the few dozen commits I did that caused this > regression... > > I don't have a GELI + ZFS test bed here. > I have another PC that is operational and is running CURRENT last built about a week ago. I'll copy it's gptzfsboot file to a memory stick and try loading it on the non-functioning PC in the morning. I'll let you know how it comes out. Tom -- Public Keys: PGP KeyID = 0x5F22FDC1 GnuPG KeyID = 0x620836CF ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Loader.conf problem
On Mon, Nov 20, 2017 at 4:31 PM, Thomas Laus wrote: > Warner Losh [i...@bsdimp.com] wrote: > > > > Can you revert this part of the change? Go back to a known-working > version > > of those files? Let me know if that fixes the problem? Bonus points for > > bisecting which one of the few dozen commits I did that caused this > > regression... > > > > I don't have a GELI + ZFS test bed here. > > > I have another PC that is operational and is running CURRENT last > built about a week ago. I'll copy it's gptzfsboot file to a memory > stick and try loading it on the non-functioning PC in the morning. > I'll let you know how it comes out. Please do. I fear I screwed something up in the massive rototilling I did, despite efforts to the contrary, and any help you can give me tracking it down would be greatly appreciated. Warner ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
head -r325700 to -r325997: problem locking/unlocking/ for (at least) amd64?
Ever since I updated from -r325700 to -r325997 I've been having odd, non-repeatable, fairly rare problems on amd64. (I've yet to see if I get similar issues on other platforms.) I've reported a couple of panic for page faults in kernel mode in other list messages. But I'm also getting things like clang failing assertions that do not repeat when the compile is simply retried with the same input files. I only mention locking because of recent head/sys/kern activity between -r325700 and -r325997, the intermittent nature of the observed failures, and the wide variety of contexts (kernel, world). I do not have specific evidence of a specific cause. (I do not see how I'm ever going to get any such evidence.) The following error did not repeat on a simple retry of the compile: --- Sema/SemaDeclAttr.o --- Assertion failed: (I->second && I->first == I->second->getValPtr() && "List invariant broken!"), function AddToUseList, file /usr/src/contrib/llvm/lib/IR/Value.cpp, line 793. c++: error: unable to execute command: Abort trap (core dumped) c++: error: clang frontend command failed due to signal (use -v to see invocation) FreeBSD clang version 5.0.0 (tags/RELEASE_500/final 312559) (based on LLVM 5.0.0svn) Target: powerpc-unknown-freebsd12.0 Thread model: posix InstalledDir: /usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc/tmp/usr/bin c++: note: diagnostic msg: PLEASE submit a bug report to https://bugs.freebsd.org/submit/ and include the crash backtrace, preprocessed source, and associated run script. Building /usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc/lib/clang/libclang/Sema/SemaExceptionSpec.o c++: note: diagnostic msg: PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT: Preprocessed source(s) and associated run script(s) are located at: c++: note: diagnostic msg: /tmp/SemaDeclAttr-6b7ba8.cpp c++: note: diagnostic msg: /tmp/SemaDeclAttr-6b7ba8.sh c++: note: diagnostic msg: *** [Sema/SemaDeclAttr.o] Error code 254 make[6]: stopped in /usr/src/lib/clang/libclang .ERROR_TARGET='Sema/SemaDeclAttr.o' .ERROR_META_FILE='/usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc/lib/clang/libclang/Sema_SemaDeclAttr.o.meta' .MAKE.LEVEL='6' MAKEFILE='' .MAKE.MODE='meta missing-filemon=yes missing-meta=yes silent=yes verbose' _ERROR_CMD='c++ -target powerpc-unknown-freebsd12.0 --sysroot=/usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc/tmp -B/usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc/tmp/usr/bin -O2 -pipe -I/usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc/lib/clang/libclang -I/usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc/lib/clang/libllvm -I/usr/src/contrib/llvm/tools/clang/lib/Driver -I/usr/src/contrib/llvm/tools/clang/include -DCLANG_ENABLE_ARCMT -DCLANG_ENABLE_STATIC_ANALYZER -I/usr/src/lib/clang/include -I/usr/src/contrib/llvm/include -DLLVM_BUILD_GLOBAL_ISEL -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -DLLVM_DEFAULT_TARGET_TRIPLE=\"powerpc-unknown-freebsd12.0\" -DLLVM_HOST_TRIPLE=\"powerpc-unknown-freebsd12.0\" -DDEFAULT_SYSROOT=\"\" -ffunction-sections -fdata-sections -gline-tables-only -fstack-protector-strong -Qunused-arguments -std=c++11 -fno-exceptions -fno-rtti -gline-tables-only -stdlib=libc++ -Wno-c++11-exte nsions -c /usr/src/contrib/llvm/tools/clang/lib/Sema/SemaDeclAttr.cpp -o Sema/SemaDeclAttr.o;' .CURDIR='/usr/src/lib/clang/libclang' .MAKE='make' .OBJDIR='/usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc/lib/clang/libclang' .TARGETS='all' DESTDIR='/usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc/tmp' LD_LIBRARY_PATH='' MACHINE='powerpc' MACHINE_ARCH='powerpc' MAKEOBJDIRPREFIX='' MAKESYSPATH='/usr/src/share/mk' MAKE_VERSION='20171028' PATH='/usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc/tmp/legacy/usr/sbin:/usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc/tmp/legacy/usr/bin:/usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc/tmp/legacy/bin:/usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc/tmp/usr/sbin:/usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin' SRCTOP='/usr/src' OBJTOP='/usr/obj/powerpcvtsc_clang/powerpc.powerpc/usr/src/powerpc.powerpc' .MAKE.MAKEFILES='/usr/src/share/mk/sys.mk /usr/src/share/mk/local.sys.env.mk /usr/src/share/mk/src.sys.env.mk /root/src.configs/src.conf.powerpc-clang-bootstrap.amd64-host /usr/src/share/mk/bsd.mkopt.mk /usr/src/share/mk/src.sys.obj.mk /usr/src/share/mk/auto.obj.mk /usr/src/share/mk/bsd.suffixes.mk /root/src.configs/make.conf /usr/src/share/mk/local.sys.mk /usr/src/share/mk/src.sys.mk /dev/null /usr/src/lib/clang/libclang/Makefile /usr/src/share/mk/src.opts.mk /usr/src/share/mk/bsd.own.mk /usr/src/share/mk/bsd.opts.mk /usr/src/share/mk/bsd.cpu.mk /usr/src/share/mk/bsd.compiler.mk /usr/s
