Date: Mon, 13 Jun 2016 18:06:48 -0600 From: Ian Lepore <ian@freebsd.org> To: cem@freebsd.org, Alan Somers <asomers@freebsd.org> Cc: Mark Millard <markmi@dsl-only.net>, freebsd-arm <freebsd-arm@freebsd.org>, FreeBSD Current <freebsd-current@freebsd.org> Subject: Re: 11.0 -r301815 "kyua test -k /usr/tests/Kyuafile" on rpi2 [armv7-a/cortex-a7]: broken (24) and failing (59) lists Message-ID: <1465862808.1188.129.camel@freebsd.org> In-Reply-To: <CAG6CVpUWuC2ys0cpfRnfV91MPpNOFU4%2BfMLRLCLkESmm=Nohyw@mail.gmail.com> References: <F86BEFBC-9A3D-48A5-A389-2C97267ADE37@dsl-only.net> <CAOtMX2geyMO1wk9r4iusQ8uS=07DQZWYKFnidY7kCtjjAckymg@mail.gmail.com> <CAG6CVpUWuC2ys0cpfRnfV91MPpNOFU4%2BfMLRLCLkESmm=Nohyw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 2016-06-13 at 11:04 -0700, Conrad Meyer wrote:
> I expect it's because:
>
> 1. bitstr_size() is just bytes (doesn't round up to sizeof(bitstr_t ==
> unsigned long))
This might result in allocating too-few bytes, but that's something
you'd only notice if you have something like valgrind watching out for
accesses beyond the allocated size.
> 2. The userspace version of bit_alloc() uses calloc(bitstr_size(), 1)
> (an array of nmemb=bitstr_size() size=bytes, doesn't have to be
> 'unsigned long' sized or aligned).
> 3. Various bit_* functions access the result as if it's an array of
> 'unsigned long', when it was allocated as a single-byte array (no
> ulong alignment or size).
malloc and related functions (including calloc) always return memory
"...suitably aligned for storage of any type of object." Last time I
checked that basically meant all allocations, even a single byte, are
aligned on a 16-byte boundary on arm.
> 4. ARM isn't as happy about unaligned accesses as x86.
>
That's mostly not the case anymore. Using load/store-doubleword or
load/store-multiple instructions requires 4-byte-aligned values (not a
typo: doubleword access requires word alignment). Everything smaller
than doubleword access in userland these days can be unaligned. The
optimizer can combine adjacent 32-bit accesses into doubleword
-instruction accesses, leading to alignment faults with unaligned data,
but that shouldn't be the case here because malloc'd memory is aligned.
-- Ian
> I'd make the following change (needs sys/param.h, not compile tested)
> and see if it fixes it:
>
> --- sys/bitstring.h (revision 301805)
> +++ sys/bitstring.h (working copy)
> @@ -119,7 +119,8 @@
> static inline bitstr_t *
> bit_alloc(int _nbits)
> {
> - return ((bitstr_t *)calloc(bitstr_size(_nbits), 1));
> + return (calloc(howmany(bitstr_size(_nbits),
> sizeof(bitstr_t)),
> + sizeof(bitstr_t)));
> }
> #endif
>
>
>
>
>
>
> On Mon, Jun 13, 2016 at 10:49 AM, Alan Somers <asomers@freebsd.org>
> wrote:
> > Please open a bug for the bitstring test failures and assign it to
> > me.
> > Also, since I don't have any arm hardware, please provide
> > instructions
> > on how to run this code in a VM, or where I can get access to the
> > hardware.
> >
> > -Alan
> >
> > On Mon, Jun 13, 2016 at 11:29 AM, Mark Millard <markmi@dsl-only.net
> > > wrote:
> > > With the newly less strict alignment requirements "kyua test -k
> > > /usr/tests/Kyuafile" runs to completion, unlike before.
> > >
> > > > ===> Summary
> > > > Results read from /root/.kyua/store/results.usr_tests.20160613
> > > > -080302-120731.db
> > > > Test cases: 5694 total, 54 skipped, 21 expected failures, 24
> > > > broken, 59 failed
> > > > Total time: 8723.243s
> > >
> > >
> > > I only list the one line summaries below. Then I list various
> > > context details.
> > >
> > > > ===> Broken tests
> > > > lib/msun/cexp_test:main -> broken: Received signal 6
> > > > [1.054s]
> > > > lib/msun/ctrig_test:main -> broken: Received signal 6
> > > > [1.074s]
> > > > lib/msun/exponential_test:main -> broken: Received signal 6
> > > > [1.045s]
> > > > lib/msun/fenv_test:main -> broken: Received signal 6
> > > > [1.048s]
> > > > lib/msun/fma_test:main -> broken: Received signal 6 [1.080s]
> > > > lib/msun/invctrig_test:main -> broken: Received signal 6
> > > > [1.091s]
> > > > lib/msun/invtrig_test:main -> broken: Received signal 6
> > > > [1.086s]
> > > > lib/msun/logarithm_test:main -> broken: Received signal 6
> > > > [1.054s]
> > > > lib/msun/lrint_test:main -> broken: Received signal 6
> > > > [1.069s]
> > > > lib/msun/nearbyint_test:main -> broken: Received signal 6
> > > > [1.066s]
> > > > lib/msun/rem_test:main -> broken: Received signal 6 [1.069s]
> > > > lib/msun/trig_test:main -> broken: Received signal 6
> > > > [1.070s]
> > > > sbin/growfs/legacy_test:main -> broken: Reported plan differs
> > > > from actual executed tests [0.459s]
> > > > sys/geom/class/eli/integrity_copy_test:main -> broken: Test
> > > > case timed out [1200.082s]
> > > > sys/geom/class/eli/integrity_hmac_test:main -> broken: Test
> > > > case timed out [600.138s]
> > > > sys/geom/class/eli/onetime_a_test:main -> broken: Test case
> > > > timed out [600.044s]
> > > > sys/sys/bitstring_test:bit_clear -> broken: Test case body
> > > > timed out [300.032s]
> > > > sys/sys/bitstring_test:bit_count -> broken: Premature exit;
> > > > test case received signal 11 (core dumped) [1.080s]
> > > > sys/sys/bitstring_test:bit_ffc -> broken: Premature exit;
> > > > test case received signal 11 (core dumped) [1.077s]
> > > > sys/sys/bitstring_test:bit_ffc_at -> broken: Premature exit;
> > > > test case received signal 11 (core dumped) [1.081s]
> > > > sys/sys/bitstring_test:bit_ffs -> broken: Premature exit;
> > > > test case received signal 11 (core dumped) [1.082s]
> > > > sys/sys/bitstring_test:bit_ffs_at -> broken: Premature exit;
> > > > test case received signal 11 (core dumped) [1.077s]
> > > > sys/sys/bitstring_test:bit_nclear -> broken: Premature exit;
> > > > test case received signal 11 (core dumped) [1.083s]
> > > > sys/sys/bitstring_test:bit_nset -> broken: Premature exit;
> > > > test case received signal 11 (core dumped) [1.079s]
> > >
> > >
> > > > ===> Failed tests
> > > > lib/libc/c063/fstatat_test:fstatat_fd -> failed:
> > > > /usr/src/contrib/netbsd-tests/lib/libc/c063/t_fstatat.c:74:
> > > > memcmp(&st1, &st2, sizeof(st1)) == 0 not met [0.
> > > > 027s]
> > > > lib/libc/nss/gethostby_test:getipnodebyname_getaddrinfo_ipv4
> > > > -> failed: /usr/src/lib/libc/tests/nss/gethostby_test.c:1335:
> > > > run_tests(_hostlist_file, _snapshot
> > > > _file, 2, TEST_GETHOSTBYNAME2_GETADDRINFO, 0) == 0 not met
> > > > [15.315s]
> > > > lib/libc/ssp/ssp_test:fgets -> failed: Test case body
> > > > returned a non-ok exit code, but this is not allowed [0.153s]
> > > > lib/libc/ssp/ssp_test:gets -> failed: Test case body returned
> > > > a non-ok exit code, but this is not allowed [0.158s]
> > > > lib/libc/ssp/ssp_test:memcpy -> failed: atf-check failed; see
> > > > the output of the test for details [0.148s]
> > > > lib/libc/ssp/ssp_test:memmove -> failed: atf-check failed;
> > > > see the output of the test for details [0.147s]
> > > > lib/libc/ssp/ssp_test:memset -> failed: atf-check failed; see
> > > > the output of the test for details [0.147s]
> > > > lib/libc/ssp/ssp_test:read -> failed: Test case body returned
> > > > a non-ok exit code, but this is not allowed [0.154s]
> > > > lib/libc/ssp/ssp_test:readlink -> failed: atf-check failed;
> > > > see the output of the test for details [0.155s]
> > > > lib/libc/ssp/ssp_test:snprintf -> failed: atf-check failed;
> > > > see the output of the test for details [0.149s]
> > > > lib/libc/ssp/ssp_test:sprintf -> failed: atf-check failed;
> > > > see the output of the test for details [0.149s]
> > > > lib/libc/ssp/ssp_test:stpcpy -> failed: atf-check failed; see
> > > > the output of the test for details [0.149s]
> > > > lib/libc/ssp/ssp_test:stpncpy -> failed: atf-check failed;
> > > > see the output of the test for details [0.147s]
> > > > lib/libc/ssp/ssp_test:strcat -> failed: atf-check failed; see
> > > > the output of the test for details [0.147s]
> > > > lib/libc/ssp/ssp_test:strcpy -> failed: atf-check failed; see
> > > > the output of the test for details [0.147s]
> > > > lib/libc/ssp/ssp_test:strncat -> failed: atf-check failed;
> > > > see the output of the test for details [0.147s]
> > > > lib/libc/ssp/ssp_test:strncpy -> failed: atf-check failed;
> > > > see the output of the test for details [0.146s]
> > > > lib/libc/ssp/ssp_test:vsnprintf -> failed: atf-check failed;
> > > > see the output of the test for details [0.150s]
> > > > lib/libc/ssp/ssp_test:vsprintf -> failed: atf-check failed;
> > > > see the output of the test for details [0.148s]
> > > > lib/libc/stdio/printbasic_test:int_within_limits -> failed:
> > > > printf("%tu", (size_t)-1) ==> [18446744073709551615], expected
> > > > [4294967295]<<NEWLINE>> [0.030s]
> > > > lib/libc/stdio/scanfloat_test:infinities_and_nans -> failed:
> > > > /usr/src/lib/libc/tests/stdio/scanfloat_test.c:191:
> > > > fetestexcept(FE_INVALID) == 0 not met [0.031
> > > > s]
> > > > lib/libc/sys/mincore_test:mincore_resid -> failed:
> > > > /usr/src/contrib/netbsd-tests/lib/libc/sys/t_mincore.c:225:
> > > > check_residency(addr, npgs) == 0 not met [0.04
> > > > 0s]
> > > > lib/libc/sys/mincore_test:mincore_shmseg -> failed:
> > > > /usr/src/contrib/netbsd-tests/lib/libc/sys/t_mincore.c:298:
> > > > check_residency(addr, npgs) == 0 not met [0.0
> > > > 29s]
> > > > lib/libc/tls/tls_dynamic_test:t_tls_dynamic -> failed: 15
> > > > checks failed; see output for more details [0.035s]
> > > > lib/libproc/proc_test:symbol_lookup -> failed:
> > > > /usr/src/lib/libproc/tests/proc_test.c:116: state != PS_STOP:
> > > > process has state 4 [0.177s]
> > > > lib/libxo/functional_test:test_02__E -> failed: atf-check
> > > > failed; see the output of the test for details [0.166s]
> > > > lib/libxo/functional_test:test_02__H -> failed: atf-check
> > > > failed; see the output of the test for details [0.168s]
> > > > lib/libxo/functional_test:test_02__HIPx -> failed: atf-check
> > > > failed; see the output of the test for details [0.170s]
> > > > lib/libxo/functional_test:test_02__HP -> failed: atf-check
> > > > failed; see the output of the test for details [0.164s]
> > > > lib/libxo/functional_test:test_02__J -> failed: atf-check
> > > > failed; see the output of the test for details [0.169s]
> > > > lib/libxo/functional_test:test_02__JP -> failed: atf-check
> > > > failed; see the output of the test for details [0.166s]
> > > > lib/libxo/functional_test:test_02__T -> failed: atf-check
> > > > failed; see the output of the test for details [0.168s]
> > > > lib/libxo/functional_test:test_02__X -> failed: atf-check
> > > > failed; see the output of the test for details [0.169s]
> > > > lib/libxo/functional_test:test_02__XP -> failed: atf-check
> > > > failed; see the output of the test for details [0.168s]
> > > > lib/msun/conj_test:main -> failed: 9 tests of 42 failed
> > > > [0.034s]
> > > > lib/msun/ldexp_test:ldexp_denormal -> failed: 4 checks
> > > > failed; see output for more details [0.034s]
> > > > local/kyua/model/metadata_test:override_all_with_set_string ->
> > > > failed: Line 253: disk_space != md.required_disk_space()
> > > > (16777216.00T != 2.00G) [0.047s]
> > > > local/kyua/testers/stacktrace_test:dump__cannot_find_gdb ->
> > > > failed: testers/stacktrace_test.c:281:
> > > > atf_utils_grep_file("execvp failed", "stacktrace") not met
> > > > [0.611s]
> > > > local/kyua/testers/stacktrace_test:dump__gdb_fail -> failed:
> > > > testers/stacktrace_test.c:294: atf_utils_grep_file("foo",
> > > > "stacktrace") not met [0.610s]
> > > > local/kyua/testers/stacktrace_test:dump__gdb_times_out ->
> > > > failed: testers/stacktrace_test.c:311:
> > > > atf_utils_grep_file("foo", "stacktrace") not met [0.614s]
> > > > local/kyua/testers/stacktrace_test:dump__integration ->
> > > > failed: testers/stacktrace_test.c:233:
> > > > atf_utils_grep_file("#0", "stacktrace") not met [0.613s]
> > > > local/kyua/testers/stacktrace_test:dump__ok -> failed:
> > > > testers/stacktrace_test.c:249: atf_utils_grep_file("frame 1",
> > > > "stacktrace") not met [0.614s]
> > > > local/kyua/testers/stacktrace_test:find_core__found__long ->
> > > > failed: Core dumped, but no candidates found [0.606s]
> > > > local/kyua/testers/stacktrace_test:find_core__found__short ->
> > > > failed: Core dumped, but no candidates found [0.603s]
> > > > local/kyua/testers/tap_parser_test:try_parse_plan__insane ->
> > > > failed: testers/tap_parser_test.c:135: 'too long' not matched
> > > > in 'Plan line includes out of range
> > > > numbers' [0.032s]
> > > > sys/geom/class/eli/resize_test:main -> failed: 15 tests of 27
> > > > failed [1.292s]
> > > > sys/kern/pipe/pipe_fstat_bug_test:main -> failed: Returned
> > > > non-success exit status 1 [0.044s]
> > > > usr.bin/lastcomm/legacy_test:main -> failed: 4 tests of 6
> > > > failed [0.151s]
> > > > usr.sbin/rpcbind/addrmerge_test:addrmerge_bindip -> failed: 1
> > > > checks failed; see output for more details [0.035s]
> > > > usr.sbin/rpcbind/addrmerge_test:addrmerge_bindip_rev ->
> > > > failed: 1 checks failed; see output for more details [0.035s]
> > > > usr.sbin/rpcbind/addrmerge_test:addrmerge_localhost_only ->
> > > > failed: 1 checks failed; see output for more details [0.034s]
> > > > usr.sbin/rpcbind/addrmerge_test:addrmerge_one_addr_on_each_subn
> > > > et -> failed: 1 checks failed; see output for more details
> > > > [0.035s]
> > > > usr.sbin/rpcbind/addrmerge_test:addrmerge_one_addr_on_each_subn
> > > > et_rev -> failed: 1 checks failed; see output for more
> > > > details [0.035s]
> > > > usr.sbin/rpcbind/addrmerge_test:addrmerge_point2point ->
> > > > failed: 1 checks failed; see output for more details [0.035s]
> > > > usr.sbin/rpcbind/addrmerge_test:addrmerge_point2point_rev ->
> > > > failed: 1 checks failed; see output for more details [0.033s]
> > > > usr.sbin/rpcbind/addrmerge_test:addrmerge_recvdstaddr ->
> > > > failed: 1 checks failed; see output for more details [0.035s]
> > > > usr.sbin/rpcbind/addrmerge_test:addrmerge_recvdstaddr_rev ->
> > > > failed: 1 checks failed; see output for more details [0.035s]
> > > > usr.sbin/rpcbind/addrmerge_test:addrmerge_singlehomed ->
> > > > failed: 1 checks failed; see output for more details [0.032s]
> > > > usr.sbin/sa/legacy_test:main -> failed: 12 tests of 13 failed
> > > > [0.340s]
> > >
> > >
> > >
> > > Context details:
> > >
> > > > # uname -apKU
> > > > FreeBSD rpi2 11.0-ALPHA3 FreeBSD 11.0-ALPHA3 #0 r301815M: Sat
> > > > Jun 11 23:43:48 PDT 2016 markmi@FreeBSDx64:/usr/obj/clang/a
> > > > rm.armv6/usr/src/sys/RPI2-NODBG ar
> > > > m armv6 1100116 1100116
> > >
> > > In use for the system build was:
> > >
> > > > XCFLAGS+= -march=armv7-a -mcpu=cortex-a7
> > > > XCXXFLAGS+= -march=armv7-a -mcpu=cortex-a7
> > >
> > > I do have a few local /usr/src changes/additions, but nearly all
> > > are only for powerpc and/or powerpc64 contexts:
> > >
> > > > # svnlite status /usr/src/
> > > > M
> > > > /usr/src/contrib/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBui
> > > > lder.cpp
> > > > M /usr/src/lib/csu/powerpc64/Makefile
> > > > ? /usr/src/sys/amd64/include/include
> > > > ? /usr/src/sys/arm/conf/RPI2-NODBG
> > > > ? /usr/src/sys/arm/include/include
> > > > M /usr/src/sys/boot/ofw/Makefile.inc
> > > > M /usr/src/sys/boot/powerpc/Makefile
> > > > M /usr/src/sys/boot/powerpc/Makefile.inc
> > > > M /usr/src/sys/boot/uboot/Makefile.inc
> > > > M /usr/src/sys/conf/Makefile.powerpc
> > > > M /usr/src/sys/conf/kern.mk
> > > > M /usr/src/sys/conf/kmod.mk
> > > > M /usr/src/sys/dev/cxgb/ulp/tom/cxgb_listen.c
> > > > M /usr/src/sys/dev/cxgbe/tom/t4_listen.c
> > > > ? /usr/src/sys/powerpc/conf/GENERIC64-NODBG
> > > > ? /usr/src/sys/powerpc/conf/GENERIC64vtsc
> > > > ? /usr/src/sys/powerpc/conf/GENERIC64vtsc-NODEBUG
> > > > ? /usr/src/sys/powerpc/conf/GENERICvtsc
> > > > ? /usr/src/sys/powerpc/conf/GENERICvtsc-NODEBUG
> > > > ? /usr/src/sys/powerpc/include/include
> > > > M /usr/src/sys/powerpc/ofw/ofw_machdep.c
> > > > M /usr/src/sys/powerpc/powerpc/exec_machdep.c
> > > > ? /usr/src/sys/x86/include/include
> > >
> > > [The cxbg and cxbge changes just remove redundant declarations
> > > that stopped a amd64-gcc based build. But other areas have such
> > > issues and I quit trying eliminating such blocks to that tool
> > > chain. The include/include's are from something making symbolic
> > > links back to the parent include.]
> > >
> > > The kernel configuration was via:
> > >
> > > > # more /usr/src/sys/arm/conf/RPI2-NODBG
> > > > #
> > > > # RPI2 -- Custom configuration for the Raspberry Pi 2
> > > > #
> > > > # For more information on this file, please read the config(5)
> > > > manual page,
> > > > # and/or the handbook section on Kernel Configuration Files:
> > > > #
> > > > # http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/
> > > > kernelconfig-config.html
> > > > #
> > > > # The handbook is also available locally in
> > > > /usr/share/doc/handbook
> > > > # if you've installed the doc distribution, otherwise always
> > > > see the
> > > > # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for
> > > > the
> > > > # latest information.
> > > > #
> > > > # An exhaustive list of options and more detailed explanations
> > > > of the
> > > > # device lines is also present in the ../../conf/NOTES and
> > > > NOTES files.
> > > > # If you are in doubt as to the purpose or necessity of a line,
> > > > check first
> > > > # in NOTES.
> > > > #
> > > >
> > > > ident RPI2-NODBG
> > > >
> > > > include "RPI2"
> > > >
> > > > makeoptions DEBUG=-g # Build kernel with
> > > > gdb(1) debug symbols
> > > > options ALT_BREAK_TO_DEBUGGER
> > > > #options VERBOSE_SYSINIT # Enable verbose
> > > > sysinit messages
> > > >
> > > > options KDB # Enable kernel
> > > > debugger support
> > > >
> > > > # For minimum debugger support (stable branch) use:
> > > > #options KDB_TRACE # Print a stack trace
> > > > for a panic
> > > > options DDB # Enable the kernel
> > > > debugger
> > > >
> > > > nooptions INVARIANTS # Enable calls of extra
> > > > sanity checking
> > > > nooptions INVARIANT_SUPPORT # Extra sanity checks
> > > > of internal structures, required by INVARIANTS
> > > > nooptions WITNESS # Enable checks to
> > > > detect deadlocks and cycles
> > > > nooptions WITNESS_SKIPSPIN # Don't run witness on
> > > > spinlocks for speed
> > > > nooptions DIAGNOSTIC
> > >
> > >
> > > The armv6 11.0 -r301815 system was cross built from amd64, built
> > > using src.conf:
> > >
> > > > # more ~/src.configs/src.conf.rpi2-clang-bootstrap.amd64-host
> > > > TO_TYPE=armv6
> > > > #
> > > > KERNCONF=RPI2-NODBG
> > > > TARGET=arm
> > > > .if ${.MAKE.LEVEL} == 0
> > > > TARGET_ARCH=${TO_TYPE}
> > > > .export TARGET_ARCH
> > > > .endif
> > > > #
> > > > WITH_CROSS_COMPILER=
> > > > WITHOUT_SYSTEM_COMPILER=
> > > > #
> > > > #CPUTYPE=soft
> > > > WITH_LIBSOFT=
> > > > WITH_LIBCPLUSPLUS=
> > > > WITH_BINUTILS_BOOTSTRAP=
> > > > WITH_CLANG_BOOTSTRAP=
> > > > WITH_CLANG=
> > > > WITH_CLANG_IS_CC=
> > > > WITH_CLANG_FULL=
> > > > WITH_CLANG_EXTRAS=
> > > > WITH_LLDB=
> > > > #
> > > > WITH_BOOT=
> > > > WITHOUT_LIB32=
> > > > #
> > > > WITHOUT_ELFTOOLCHAIN_BOOTSTRAP=
> > > > WITHOUT_GCC_BOOTSTRAP=
> > > > WITHOUT_GCC=
> > > > WITHOUT_GCC_IS_CC=
> > > > WITHOUT_GNUCXX=
> > > > #
> > > > NO_WERROR=
> > > > #WERROR=
> > > > MALLOC_PRODUCTION=
> > > > #
> > > > WITH_DEBUG_FILES=
> > > > #
> > > > XCFLAGS+= -march=armv7-a -mcpu=cortex-a7
> > > > XCXXFLAGS+= -march=armv7-a -mcpu=cortex-a7
> > > > # There is no XCPPFLAGS but XCPP ets XCFLAGS content.
> > >
> > > and with the make.conf for the system build being empty:
> > >
> > > > # more ~/src.configs/make.conf
> > > > #
> > >
> > > ===
> > > Mark Millard
> > > markmi at dsl-only.net
> > >
> > > _______________________________________________
> > > freebsd-current@freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > > To unsubscribe, send any mail to "
> > > freebsd-current-unsubscribe@freebsd.org"
> > _______________________________________________
> > freebsd-current@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to "
> > freebsd-current-unsubscribe@freebsd.org"
> _______________________________________________
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "
> freebsd-current-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1465862808.1188.129.camel>