Date: Thu, 3 Jan 2019 02:06:51 -0800 From: Mark Millard <marklmi@yahoo.com> To: Kyle Evans <kevans@freebsd.org>, freebsd-emulation@freebsd.org Cc: freebsd-arm <freebsd-arm@freebsd.org> Subject: Re: Under qemu-aarch64-static "wc /dev/null" gets "Unsupported ancillary data: 1/0" from a sendmsg attempt: because of wrong cmsg_len type in target_cmsghdr Message-ID: <DA582336-A6B1-4026-B48E-603CB491D511@yahoo.com> In-Reply-To: <22184643-4320-4B7C-86DA-A71DF62D4543@yahoo.com> References: <22184643-4320-4B7C-86DA-A71DF62D4543@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[Adding a for-reference note.]
On 2019-Jan-3, at 01:25, Mark Millard <marklmi at yahoo.com> wrote:
> [This note follows the investigation sequence,
> ending with the important conclusions.]
>=20
> My test context here is a poudriere-devel bulk -i for a
> amd64->aarch64 context.
>=20
> wc /dev/null or wc //dev/null does:
>=20
> # wc /dev/null
> Unsupported ancillary data: 1/0
>=20
> that then hangs-up until I ^C to get back to a prompt.
>=20
>=20
> Here is what ktrace/kdump shows the process before the hang through
> when I hit ^C to stop the hang-up:
>=20
> . . .
> 98475 101033 qemu-aarch64-static 0.000340 CALL =
sigprocmask[340](SIG_BLOCK,0x7ffffffe3c80,0x7ffffffe3d80)
> 98475 101033 qemu-aarch64-static 0.000003 RET sigprocmask[340] 0
> 98475 101033 qemu-aarch64-static 0.000001 CALL =
pselect[522](0x6,0,0x7ffffffe3fb0,0,0,0x7ffffffe3d80)
> 98475 101033 qemu-aarch64-static 0.000001 RET pselect[522] 1
> 98475 101033 qemu-aarch64-static 0.000001 CALL =
sigprocmask[340](SIG_SETMASK,0x7ffffffe3c80,0)
> 98475 101033 qemu-aarch64-static 0.000001 RET sigprocmask[340] 0
> 98475 101033 qemu-aarch64-static 0.000042 CALL =
write[4](0x2,0x7ffffffe3480,0x20)
> 98475 101033 qemu-aarch64-static 0.000036 GIO fd 2 wrote 32 bytes
> "Unsupported ancillary data: 1/0
> "
> 98475 101033 qemu-aarch64-static 0.000003 RET write[4] 32/0x20
> 98475 101033 qemu-aarch64-static 0.000001 CALL =
sendmsg[28](0x5,0x7ffffffe3c28,0)
> 98475 101033 qemu-aarch64-static 0.000003 RET sendmsg[28] -1 errno =
22 Invalid argument
> 98475 101033 qemu-aarch64-static 0.000184 CALL close[6](0x3)
> 98475 101033 qemu-aarch64-static 0.000040 RET close[6] 0
> 98475 101033 qemu-aarch64-static 0.000017 CALL close[6](0x7)
> 98475 101033 qemu-aarch64-static 0.000005 RET close[6] 0
> 98475 101033 qemu-aarch64-static 0.000002 CALL =
sigprocmask[340](SIG_BLOCK,0x7ffffffe3c80,0x7ffffffe3d80)
> 98475 101033 qemu-aarch64-static 0.000001 RET sigprocmask[340] 0
> 98475 101033 qemu-aarch64-static 0.000001 CALL =
pselect[522](0x6,0x7ffffffe3dd0,0,0,0,0x7ffffffe3d80)
> 98475 101539 qemu-aarch64-static 0.000089 RET nanosleep[240] 0
> 98475 101539 qemu-aarch64-static 0.000042 CALL =
_umtx_op[454](0x86101f008,UMTX_OP_WAIT_UINT_PRIVATE,0,0,0)
> 98475 101033 qemu-aarch64-static 15.845396 RET pselect[522] -1 errno =
4 Interrupted system call
>=20
> Note the qemu-aarch64 genrated message and the later:
> sendmsg[28] -1 errno 22 Invalid argument
>=20
> The qemu-*-static code that wrote the message is from
> t2h_freebsd_cmsg and is:
>=20
> if ((cmsg->cmsg_level =3D=3D TARGET_SOL_SOCKET) &&
> (cmsg->cmsg_type =3D=3D SCM_RIGHTS)) {
> int *fd =3D (int *)data;
> int *target_fd =3D (int *)target_data;
> int i, numfds =3D len / sizeof(int);
>=20
> for (i =3D 0; i < numfds; i++) {
> fd[i] =3D tswap32(target_fd[i]);
> }
> } else if ((cmsg->cmsg_level =3D=3D TARGET_SOL_SOCKET) &&
> (cmsg->cmsg_type =3D=3D SCM_TIMESTAMP) &&
> (len =3D=3D sizeof(struct timeval))) {
> /* copy struct timeval to host */
> struct timeval *tv =3D (struct timeval *)data;
> struct target_freebsd_timeval *target_tv =3D
> (struct target_freebsd_timeval *)target_data;
> __get_user(tv->tv_sec, &target_tv->tv_sec);
> __get_user(tv->tv_usec, &target_tv->tv_usec);
> } else {
> gemu_log("Unsupported ancillary data: %d/%d\n",
> cmsg->cmsg_level, cmsg->cmsg_type);
> memcpy(data, target_data, len);
> }
>=20
> Well it turns out that qemu_*-static 's code has:
>=20
> struct target_cmsghdr {
> abi_long cmsg_len;
> int32_t cmsg_level;
> int32_t cmsg_type;
> };
>=20
> where for amd64 target_cmsghdr has:
>=20
> (gdb) p/d sizeof(struct target_cmsghdr)
> $2 =3D 16
> (gdb) p/d sizeof(((struct target_cmsghdr *)0)->cmsg_len)=20
> $5 =3D 8
> (gdb) p/d &((struct target_cmsghdr *)0)->cmsg_level
> $4 =3D 8
> (gdb) p/d &((struct target_cmsghdr *)0)->cmsg_type=20
> $1 =3D 12
>=20
> which does not match the amd64 or aarch64 native:
>=20
> struct cmsghdr {
> socklen_t cmsg_len; /* data byte count, =
including hdr */
> int cmsg_level; /* originating protocol =
*/
> int cmsg_type; /* protocol-specific =
type */
> /* followed by u_char cmsg_data[]; */
> }; =20
>=20
> because the cmsghdr's cmsg_len is smaller, even on a 64-bit =
architecture:
>=20
> (gdb) p/d sizeof(((struct cmsghdr *)0)->cmsg_len)
> $6 =3D 4
>=20
> /usr/include/arpa/inet.h:typedef __socklen_t socklen_t;
> /usr/include/netinet/in.h:typedef __socklen_t socklen_t;
> /usr/include/netinet6/in6.h:typedef __socklen_t socklen_t;
> /usr/include/sys/_types.h:typedef __uint32_t __socklen_t;
> /usr/include/sys/socket.h:typedef __socklen_t socklen_t;
> . . .
> /usr/include/netdb.h:typedef __socklen_t socklen_t;
>=20
> so abi_long does not match socklen_t for 64-bit architectures.
>=20
> So code such as in t2h_freebsd_cmsg:
>=20
> cmsg->cmsg_level =3D tswap32(target_cmsg->cmsg_level);
> cmsg->cmsg_type =3D tswap32(target_cmsg->cmsg_type);
>=20
> is not using the correct target offsets when aarch64 is the target
> that it is extracting from (for example).
>=20
> For comparison on a 64-bit architecture:
>=20
> (gdb) p/d sizeof(struct cmsghdr)
> $1 =3D 12
> (gdb) p/d &((struct cmsghdr *)0)->cmsg_level
> $2 =3D 4
> (gdb) p/d &((struct cmsghdr *)0)->cmsg_type=20
> $3 =3D 8
>=20
>=20
> I do not yet have a tested change.
>=20
On aarch64 (like on amd64):
# more cmsghdr_size_offsets.c=20
#include "/usr/include/sys/socket.h" // cmsghdr
#include <stddef.h> // offsetof
#include <stdio.h> // printf
int
main()
{
printf("%lu\n", (unsigned long) sizeof(struct cmsghdr));
printf("cmsg_len %lu\n", (unsigned long) offsetof(struct =
cmsghdr, cmsg_len));
printf("cmsg_level %lu\n", (unsigned long) offsetof(struct =
cmsghdr, cmsg_level));
printf("cmsg_type %lu\n", (unsigned long) offsetof(struct =
cmsghdr, cmsg_type));
return 0;
}
produces:
# ./a.out
12
cmsg_len 0
cmsg_level 4
cmsg_type 8
which qemu-aarch64-static 's target_cmsghdr definitely does not match.
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?DA582336-A6B1-4026-B48E-603CB491D511>