[Good News Summary: Rebuilding buildworld/buildkernel for rpi2 11.0-CURRENT 292413 from amd64 based on adding -fmax-type-align=4 has so far removed the crashes during the toolchain activity: no more misaligned accesses in libc's _fseeko or elsewhere.]
On 2015-Dec-25, at 12:31 AM, Mark Millard <[email protected]> wrote: > On 2015-Dec-24, at 10:39 PM, Mark Millard <[email protected]> wrote: > >> [I do not know if this partial crash analysis related to on-arm >> clang-associated activity is good enough and appropriate to submit or not.] >> >> The /usr/local/arm-gnueabi-freebsd/bin/ar on the rpi2b involved below came >> from pkg install activity instead of port building. Used as-is. >> >> When I just tried my first from-rpi2b builds (ports for a rpi2b), >> /usr/local/arm-gnueabi-freebsd/bin/ar crashed. I believe that the following >> suggests an alignment error for the type of instructions that memset for 128 >> bytes was translated to (sizeof(mbstate_t)) in the code used by >> /usr/local/arm-gnueabi-freebsd/bin/ar. (But I do not know how to check SCTLR >> bit[1] to be directly sure that alignment was being enforced.) >> >> The crash was a Bus error in /usr/local/arm-gnueabi-freebsd/bin/ar : >> >>> libtool: link: /usr/local/arm-gnueabi-freebsd/bin/ar cru .libs/libgnuintl.a >>> bindtextdom.o dcgettext.o dgettext.o gettext.o finddomain.o hash-string.o >>> loadmsgcat.o localealias.o textdomain.o l10nflist.o explodename.o >>> dcigettext.o dcngettext.o dngettext.o ngettext.o pluralx.o plural-exp.o >>> localcharset.o threadlib.o lock.o relocatable.o langprefs.o localename.o >>> log.o printf.o setlocale.o version.o xsize.o osdep.o intl-compat.o >>> Bus error (core dumped) >>> *** [libgnuintl.la] Error code 138 >> >> It failed in _fseeko doing a memset that turned into uses of "vst1.64 >> {d16-d17}, [r0]" instructions, for an address in register r0 that ended in >> 0xa4, so was not aligned to 8 byte boundaries. From what I read such "VSTn >> (multiple n-element structures)" that have .64 require 8 byte alignment. The >> evidence of the code and register value follow. >> >>> # gdb /usr/local/arm-gnueabi-freebsd/bin/ar >>> /usr/obj/portswork/usr/ports/devel/gettext-tools/work/gettext-0.19.6/gettext-tools/intl/ar.core >>> . . . >>> #0 0x2033adcc in _fseeko (fp=0x20651dcc, offset=<value optimized out>, >>> whence=<value optimized out>, ltest=<value optimized out>) at >>> /usr/src/lib/libc/stdio/fseek.c:299 >>> 299 memset(&fp->_mbstate, 0, sizeof(mbstate_t)); >>> . . . >>> (gdb) x/24i 0x2033adb0 >>> 0x2033adb0 <_fseeko+836>: vmov.i32 q8, #0 ; 0x00000000 >>> 0x2033adb4 <_fseeko+840>: movw r1, #65503 ; 0xffdf >>> 0x2033adb8 <_fseeko+844>: stm r4, {r0, r7} >>> 0x2033adbc <_fseeko+848>: ldrh r0, [r4, #12] >>> 0x2033adc0 <_fseeko+852>: and r0, r0, r1 >>> 0x2033adc4 <_fseeko+856>: strh r0, [r4, #12] >>> 0x2033adc8 <_fseeko+860>: add r0, r4, #216 ; 0xd8 >>> 0x2033adcc <_fseeko+864>: vst1.64 {d16-d17}, [r0] >>> 0x2033add0 <_fseeko+868>: add r0, r4, #200 ; 0xc8 >>> 0x2033add4 <_fseeko+872>: vst1.64 {d16-d17}, [r0] >>> 0x2033add8 <_fseeko+876>: add r0, r4, #184 ; 0xb8 >>> 0x2033addc <_fseeko+880>: vst1.64 {d16-d17}, [r0] >>> 0x2033ade0 <_fseeko+884>: add r0, r4, #168 ; 0xa8 >>> 0x2033ade4 <_fseeko+888>: vst1.64 {d16-d17}, [r0] >>> 0x2033ade8 <_fseeko+892>: add r0, r4, #152 ; 0x98 >>> 0x2033adec <_fseeko+896>: vst1.64 {d16-d17}, [r0] >>> 0x2033adf0 <_fseeko+900>: add r0, r4, #136 ; 0x88 >>> 0x2033adf4 <_fseeko+904>: vst1.64 {d16-d17}, [r0] >>> 0x2033adf8 <_fseeko+908>: add r0, r4, #120 ; 0x78 >>> 0x2033adfc <_fseeko+912>: vst1.64 {d16-d17}, [r0] >>> 0x2033ae00 <_fseeko+916>: add r0, r4, #104 ; 0x68 >>> 0x2033ae04 <_fseeko+920>: vst1.64 {d16-d17}, [r0] >>> 0x2033ae08 <_fseeko+924>: b 0x2033b070 <_fseeko+1540> >>> 0x2033ae0c <_fseeko+928>: cmp r5, #0 ; 0x0 >>> (gdb) info all-registers >>> r0 0x20651ea4 543497892 >>> r1 0xffdf 65503 >>> r2 0x0 0 >>> r3 0x0 0 >>> r4 0x20651dcc 543497676 >>> r5 0x0 0 >>> r6 0x0 0 >>> r7 0x0 0 >>> r8 0x20359df4 540384756 >>> r9 0x0 0 >>> r10 0x0 0 >>> r11 0xbfbfb948 -1077954232 >>> r12 0x2037b208 540520968 >>> sp 0xbfbfb898 -1077954408 >>> lr 0x2035a004 540385284 >>> pc 0x2033adcc 540257740 >>> f0 0 (raw 0x000000000000000000000000) >>> f1 0 (raw 0x000000000000000000000000) >>> f2 0 (raw 0x000000000000000000000000) >>> f3 0 (raw 0x000000000000000000000000) >>> f4 0 (raw 0x000000000000000000000000) >>> f5 0 (raw 0x000000000000000000000000) >>> f6 0 (raw 0x000000000000000000000000) >>> f7 0 (raw 0x000000000000000000000000) >>> fps 0x0 0 >>> cpsr 0x60000010 1610612752 >> >> The syntax in use for vst1.64 instructions does not explicitly have the >> alignment notation. Presuming that the decoding is correct then from what I >> read the following applies: >> >>> Home > NEON and VFP Programming > NEON load and store element and structure >>> instructions > Alignment restrictions in load and store, element and >>> structure instructions >>> >>> . . . When the alignment is not specified in the instruction, the alignment >>> restriction is controlled by the A bit (SCTLR bit[1]): >>> • if the A bit is 0, there are no alignment restrictions (except >>> for strongly ordered or device memory, where accesses must be element >>> aligned or the result is unpredictable) >>> • if the A bit is 1, accesses must be element aligned. >>> If an address is not correctly aligned, an alignment fault occurs. >> >> So if at the time the "A bit" (SCTLR bit[1]) is 1 then the Bus error would >> have the context to happen because of the mis-alignment. >> >> The following shows the make.conf context that explains how >> /usr/local/arm-gnueabi-freebsd/bin/ar came to be invoked: >> >>> # more /etc/make.conf >>> WRKDIRPREFIX=/usr/obj/portswork >>> WITH_DEBUG= >>> WITH_DEBUG_FILES= >>> MALLOC_PRODUCTION= >>> # >>> TO_TYPE=armv6 >>> TOOLS_TO_TYPE=arm-gnueabi >>> CROSS_BINUTILS_PREFIX=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ >>> .if ${.MAKE.LEVEL} == 0 >>> CC=/usr/bin/clang -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a >>> CXX=/usr/bin/clang++ -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a >>> CPP=/usr/bin/clang-cpp -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a >>> .export CC >>> .export CXX >>> .export CPP >>> AS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/as >>> AR=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ar >>> LD=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ld >>> NM=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/nm >>> OBJCOPY=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objcopy >>> OBJDUMP=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objdump >>> RANLIB=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ranlib >>> SIZE=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/size >>> #NO-SUCH: STRINGS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/strings >>> STRINGS=/usr/local/bin/${TOOLS_TO_TYPE}-freebsd-strings >>> .export AS >>> .export AR >>> .export LD >>> .export NM >>> .export OBJCOPY >>> .export OBJDUMP >>> .export RANLIB >>> .export SIZE >>> .export STRINGS >>> .endif >> >> >> Other context: >> >>> # freebsd-version -ku; uname -aKU >>> 11.0-CURRENT >>> 11.0-CURRENT >>> FreeBSD rpi2 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r292413M: Tue Dec 22 >>> 22:02:21 PST 2015 >>> root@FreeBSDx64:/usr/obj/clang/arm.armv6/usr/src/sys/RPI2-NODBG arm >>> 1100091 1100091 >> >> >> >> I will note that world and kernel are my own build of -r292413 (earlier >> experiment) --a build made from an amd64 host context and put in place via >> DESTDIR=. My expectation would be that the amd64 context would not be likely >> to have similar alignment restrictions involved in its ar activity (or other >> activity). That would explain how I got this far using such a clang 3.7 >> related toolchain for targeting an rpi2 before finding such a problem. > > > I realized re-reading the all above that it seems to suggest that the _fseeko > code involved is from /usr/local/arm-gnueabi-freebsd/bin/ar but that was not > my intent. > > libc.so.7 is from my buildworld, including the fseeko implementation: > > Reading symbols from /lib/libc.so.7...Reading symbols from > /usr/lib/debug//lib/libc.so.7.debug...done. > done. > Loaded symbols for /lib/libc.so.7 > > > head/sys/sys/_types.h has: > > /* > * mbstate_t is an opaque object to keep conversion state during multibyte > * stream conversions. > */ > typedef union { > char __mbstate8[128]; > __int64_t _mbstateL; /* for alignment */ > } __mbstate_t; > > suggesting an implicit alignment of the union to whatever the implementation > defines for __int64_t --which need not be 8 byte alignment (in the abstract, > general case). But 8 byte alignment is a possibility as well (in the > abstract). > > But printing *fp in gdb for the fp argument to _fseeko reports the same > not-8-byte aligned address for __mbstate8 that was in r0: > >> (gdb) bt >> #0 0x2033adcc in _fseeko (fp=0x20651dcc, offset=<value optimized out>, >> whence=<value optimized out>, ltest=<value optimized out>) at >> /usr/src/lib/libc/stdio/fseek.c:299 >> #1 0x2033b108 in fseeko (fp=0x20651dcc, offset=18571438587904, whence=0) at >> /usr/src/lib/libc/stdio/fseek.c:82 >> #2 0x00016138 in ?? () >> (gdb) print fp >> $2 = (FILE *) 0x20651dcc >> (gdb) print *fp >> $3 = {_p = 0x2069a240 "", _r = 0, _w = 0, _flags = 5264, _file = 36, _bf = >> {_base = 0x2069a240 "", _size = 32768}, _lbfsize = 0, _cookie = 0x20651dcc, >> _close = 0x20359dfc <__sclose>, >> _read = 0x20359de4 <__sread>, _seek = 0x20359df4 <__sseek>, _write = >> 0x20359dec <__swrite>, _ub = {_base = 0x0, _size = 0}, _up = 0x0, _ur = 0, >> _ubuf = 0x20651e0c "", _nbuf = 0x20651e0f "", _lb = { >> _base = 0x0, _size = 0}, _blksize = 32768, _offset = 0, _fl_mutex = 0x0, >> _fl_owner = 0x0, _fl_count = 0, _orientation = 0, _mbstate = {__mbstate8 = >> 0x20651e34 "", _mbstateL = 0}, _flags2 = 0} > > The overall FILE struct containing the _mbstate field is also not 8-byte > aligned. But the offset from the start of the FILE struct to __mbstate8 is a > multiple of 8 bytes. > > It is my interpretation that there is nothing here to justify the memset > implementation combination: > > SCTLR bit[1]==1 > > mixed with > > vst1.64 instructions > > I.e.: one or both needs to change unless some way for forcing 8-byte > alignment is introduced. > > I have not managed to track down anything that would indicate FreeBSD's > intent for SCTLR bit[1]. I do not even know if it is required by the design > to be constant (once initialized). I have (so far) removed the build tool crashes based on adding -fmax-type-align=4 to avoid the misaligned accesses. Details follow. src.conf on amd64 for the rpi2 targeting buildworld/buildkernel now looks like: > # more ~/src.configs/src.conf.rpi2-clang.amd64-host > TO_TYPE=armv6 > TOOLS_TO_TYPE=arm-gnueabi > FROM_TYPE=amd64 > TOOLS_FROM_TYPE=x86_64 > VERSION_CONTEXT=11.0 > # > KERNCONF=RPI2-NODBG > TARGET=arm > .if ${.MAKE.LEVEL} == 0 > TARGET_ARCH=${TO_TYPE} > .export TARGET_ARCH > .endif > # > WITHOUT_CROSS_COMPILER= > # > # For WITH_BOOT= . . . > # arm-gnueabi-freebsd/bin/ld reports bootinfo.o: relocation R_ARM_MOVW_ABS_NC > against `a local symbol' can not be used when making a shared object; > recompile with -fPIC > WITHOUT_BOOT= > # > WITH_FAST_DEPEND= > WITH_LIBCPLUSPLUS= > WITH_CLANG= > WITH_CLANG_IS_CC= > WITH_CLANG_FULL= > WITH_LLDB= > WITH_CLANG_EXTRAS= > # > WITHOUT_LIB32= > WITHOUT_GCC= > WITHOUT_GNUCXX= > # > NO_WERROR= > MALLOC_PRODUCTION= > #CFLAGS+= -DELF_VERBOSE > # > WITH_DEBUG= > WITH_DEBUG_FILES= > # > # TOOLS_TO_TYPE based on ${TO_TYPE}-xtoolchain-gcc related bintutils... > # > #CROSS_TOOLCHAIN=${TO_TYPE}-gcc > X_COMPILER_TYPE=clang > CROSS_BINUTILS_PREFIX=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ > .if ${.MAKE.LEVEL} == 0 > XCC=/usr/bin/clang -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a > -fmax-type-align=4 > XCXX=/usr/bin/clang++ -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a > -fmax-type-align=4 > XCPP=/usr/bin/clang-cpp -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a > -fmax-type-align=4 > .export XCC > .export XCXX > .export XCPP > XAS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/as > XAR=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ar > XLD=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ld > XNM=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/nm > XOBJCOPY=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objcopy > XOBJDUMP=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objdump > XRANLIB=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ranlib > XSIZE=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/size > #NO-SUCH: XSTRINGS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/strings > XSTRINGS=/usr/local/bin/${TOOLS_TO_TYPE}-freebsd-strings > .export XAS > .export XAR > .export XLD > .export XNM > .export XOBJCOPY > .export XOBJDUMP > .export XRANLIB > .export XSIZE > .export XSTRINGS > .endif > # > # Host compiler stuff: > .if ${.MAKE.LEVEL} == 0 > CC=/usr/bin/clang -B/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin > CXX=/usr/bin/clang++ -B/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin > CPP=/usr/bin/clang-cpp -B/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin > .export CC > .export CXX > .export CPP > AS=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/as > AR=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/ar > LD=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/ld > NM=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/nm > OBJCOPY=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/objcopy > OBJDUMP=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/objdump > RANLIB=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/ranlib > SIZE=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/size > #NO-SUCH: STRINGS=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/strings > STRINGS=/usr/local/bin/${TOOLS_FROM_TYPE}-freebsd-strings > .export AS > .export AR > .export LD > .export NM > .export OBJCOPY > .export OBJDUMP > .export RANLIB > .export SIZE > .export STRINGS > .endif make.conf for during the on-rpi2 port builds now looks like: > $ more /etc/make.conf > WRKDIRPREFIX=/usr/obj/portswork > WITH_DEBUG= > WITH_DEBUG_FILES= > MALLOC_PRODUCTION= > # > TO_TYPE=armv6 > TOOLS_TO_TYPE=arm-gnueabi > CROSS_BINUTILS_PREFIX=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ > .if ${.MAKE.LEVEL} == 0 > CC=/usr/bin/clang -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a > -fmax-type-align=4 > CXX=/usr/bin/clang++ -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a > -fmax-type-align=4 > CPP=/usr/bin/clang-cpp -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a > -fmax-type-align=4 > .export CC > .export CXX > .export CPP > AS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/as > AR=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ar > LD=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ld > NM=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/nm > OBJCOPY=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objcopy > OBJDUMP=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objdump > RANLIB=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ranlib > SIZE=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/size > #NO-SUCH: STRINGS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/strings > STRINGS=/usr/local/bin/${TOOLS_TO_TYPE}-freebsd-strings > .export AS > .export AR > .export LD > .export NM > .export OBJCOPY > .export OBJDUMP > .export RANLIB > .export SIZE > .export STRINGS > .endif === Mark Millard markmi at dsl-only.net _______________________________________________ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-toolchain To unsubscribe, send any mail to "[email protected]"
