Date: Wed, 6 Jan 2016 09:46:08 -0500 From: Shawn Webb <shawn.webb@hardenedbsd.org> To: Adrian Chadd <adrian.chadd@gmail.com> Cc: freebsd-current <freebsd-current@freebsd.org> Subject: Re: kernel panic by enabling net.inet.ip.random_id Message-ID: <20160106144608.GA71037@mutt-hardenedbsd> In-Reply-To: <CAJ-VmonZO8WzrTMS394AJw8duvbW=%2B2bEfaQDzkkaC5HHcmAxA@mail.gmail.com> References: <20160106015742.GA8405@mutt-hardenedbsd> <CAJ-VmonnHgpCxN%2BVvrP9j%2BtHK=3Yxjz0qa9kd8riSaUEhJnNtg@mail.gmail.com> <20160106021316.GB8405@mutt-hardenedbsd> <CAJ-VmonZO8WzrTMS394AJw8duvbW=%2B2bEfaQDzkkaC5HHcmAxA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--yrj/dFKFPuw6o+aM
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
(kgdb) list *(0xffffffff80b5de9e)
0xffffffff80b5de9e is in ip_fillid (/usr/src/sys/netinet/ip_id.c:237).
warning: Source file is more recent than executable.
232 new_id =3D 0;
233 do {
234 if (new_id !=3D 0)
235 V_random_id_collisions++;
236 arc4rand(&new_id, sizeof(new_id), 0);
237 } while (bit_test(V_id_bits, new_id) || new_id =3D=3D 0);
238 bit_clear(V_id_bits, V_id_array[V_array_ptr]);
239 bit_set(V_id_bits, new_id);
240 V_id_array[V_array_ptr] =3D new_id;
241 V_array_ptr++;
This is the change I made to ip_id.c that caused the underlying kernel
panic:
https://github.com/HardenedBSD/hardenedBSD/commit/52d5a93b92097e7a79be8d2e0=
eb9c1a58b8337d1
Ideally, we should be able to just toggle that variable and all would be
well. But it seems with the VIMAGE work, something is preventing that.
Thanks,
Shawn
On Tue, Jan 05, 2016 at 06:22:34PM -0800, Adrian Chadd wrote:
> try list *(0x[address]) .
>=20
> That line is mtx_unlock(), which makes no sense (as mtx_lock succeeded fi=
ne.)
>=20
>=20
> -a
>=20
>=20
> On 5 January 2016 at 18:13, Shawn Webb <shawn.webb@hardenedbsd.org> wrote:
> > Thanks for the quick reply! Here's some more debugging output:
> >
> > =3D=3D=3D Begin Log =3D=3D=3D
> > (kgdb) bt
> > #0 doadump (textdump=3D0) at pcpu.h:221
> > #1 0xffffffff8037c78b in db_dump (dummy=3D<value optimized out>, dummy=
2=3Dfalse, dummy3=3D0, dummy4=3D0x0) at /usr/src/sys/ddb/db_command.c:533
> > #2 0xffffffff8037c57e in db_command (cmd_table=3D0x0) at /usr/src/sys/=
ddb/db_command.c:440
> > #3 0xffffffff8037c314 in db_command_loop () at /usr/src/sys/ddb/db_com=
mand.c:493
> > #4 0xffffffff8037edab in db_trap (type=3D<value optimized out>, code=
=3D0) at /usr/src/sys/ddb/db_main.c:251
> > #5 0xffffffff80a5c563 in kdb_trap (type=3D12, code=3D0, tf=3D<value op=
timized out>) at /usr/src/sys/kern/subr_kdb.c:654
> > #6 0xffffffff80e6b7e1 in trap_fatal (frame=3D0xfffffe02c33894d0, eva=
=3D<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:829
> > #7 0xffffffff80e6ba2d in trap_pfault (frame=3D0xfffffe02c33894d0, user=
mode=3D<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:684
> > #8 0xffffffff80e6b15f in trap (frame=3D0xfffffe02c33894d0) at /usr/src=
/sys/amd64/amd64/trap.c:435
> > #9 0xffffffff80e4af97 in calltrap () at /usr/src/sys/amd64/amd64/excep=
tion.S:234
> > #10 0xffffffff80b5de9e in ip_fillid (ip=3D0xfffff8000ef8cb88) at /usr/s=
rc/sys/netinet/ip_id.c:237
> > #11 0xffffffff80b6c41b in ip_output (m=3D<value optimized out>, opt=3D<=
value optimized out>, ro=3D<value optimized out>, flags=3D0, imo=3D0x0, inp=
=3D0xfffff8000e66e960) at /usr/src/sys/netinet/ip_output.c:268
> > #12 0xffffffff80bf0612 in udp_send (so=3D<value optimized out>, flags=
=3D<value optimized out>, m=3D<value optimized out>, addr=3D0x0, control=3D=
<value optimized out>, td=3D0xfffff8000ef8cb88) at /usr/src/sys/netinet/udp=
_usrreq.c:1517
> > #13 0xffffffff80aa3872 in sosend_dgram (so=3D0xfffff8000e6422e8, addr=
=3D0x0, uio=3D<value optimized out>, top=3D0xfffff8000ef8cb00, control=3D0x=
0, flags=3D<value optimized out>, td=3D0xffffffff81bef2ec) at /usr/src/sys/=
kern/uipc_socket.c:1164
> > #13 0xffffffff80aa3872 in sosend_dgram (so=3D0xfffff8000e6422e8, addr=
=3D0x0, uio=3D<value optimized out>, top=3D0xfffff8000ef8cb00, control=3D0x=
0, flags=3D<value optimized out>, td=3D0xffffffff81bef2ec) at /usr/src/sys/=
kern/uipc_socket.c:1164
> > #14 0xffffffff80aaa03b in kern_sendit (td=3D0xfffff8000e4cd9c0, s=3D6, =
mp=3D<value optimized out>, flags=3D0, control=3D0x0, segflg=3DUIO_USERSPAC=
E) at /usr/src/sys/kern/uipc_syscalls.c:906
> > #15 0xffffffff80aaa336 in sendit (td=3D0xfffff8000e4cd9c0, s=3D<value o=
ptimized out>, mp=3D0xfffffe02c3389970, flags=3D3980) at /usr/src/sys/kern/=
uipc_syscalls.c:833
> > #16 0xffffffff80aaa1fd in sys_sendto (td=3D0x0, uap=3D<value optimized =
out>) at /usr/src/sys/kern/uipc_syscalls.c:957
> > #17 0xffffffff80e6bfdb in amd64_syscall (td=3D0xfffff8000e4cd9c0, trace=
d=3D0) at subr_syscall.c:135
> > #18 0xffffffff80e4b27b in Xfast_syscall () at /usr/src/sys/amd64/amd64/=
exception.S:394
> > #19 0x000003e339782e8a in ?? ()
> > (kgdb) x/i 0xffffffff80b5de9e
> > 0xffffffff80b5de9e <ip_fillid+142>: movzbl (%rax,%rcx,1),%esi
> > (kgdb) info reg
> > rax 0x0 0
> > rbx 0x0 0
> > rcx 0x0 0
> > rdx 0x0 0
> > rsi 0x0 0
> > rdi 0x0 0
> > rbp 0xfffffe02c3388fe0 0xfffffe02c3388fe0
> > rsp 0xfffffe02c3388fc8 0xfffffe02c3388fc8
> > r8 0x0 0
> > r9 0x0 0
> > r10 0x0 0
> > r11 0x0 0
> > r12 0xffffffff817c0b80 -2122577024
> > r13 0xffffffff817c1470 -2122574736
> > r14 0x1 1
> > r15 0x4 4
> > rip 0xffffffff80a1fae3 0xffffffff80a1fae3 <doadump+51>
> > eflags 0x0 0
> > cs 0x0 0
> > ss 0x0 0
> > ds 0x0 0
> > es 0x0 0
> > fs 0x0 0
> > gs 0x0 0
> > =3D=3D=3D End Log =3D=3D=3D
> >
> > Thanks,
> >
> > Shawn
> >
> > On Tue, Jan 05, 2016 at 06:06:41PM -0800, Adrian Chadd wrote:
> >> looks like a null pointer deference. What's kgdb show at that IP?
> >>
> >>
> >> -a
> >>
> >>
> >> On 5 January 2016 at 17:57, Shawn Webb <shawn.webb@hardenedbsd.org> wr=
ote:
> >> > Hey All,
> >> >
> >> > Here's a kernel panic I'm experiencing by enabling net.inet.ip.rando=
m_id
> >> > at boot.
> >> >
> >> > I'm on latest HEAD on amd64 in bhyve. I'll soon-ish be testing on na=
tive
> >> > hardware with VIMAGE enabled.
> >> >
> >> > =3D=3D=3D Begin Log =3D=3D=3D
> >> > Kernel page fault with the following non-sleepable locks held:
> >> > exclusive sleep mutex ip_id_mtx (ip_id_mtx) r =3D 0 (0xffffffff81c54=
830) locked @ /usr/src/sys/netinet/ip_id.c:227
> >> > stack backtrace:
> >> > #0 0xffffffff80a79620 at witness_debugger+0x70
> >> > #1 0xffffffff80a7a937 at witness_warn+0x3d7
> >> > #2 0xffffffff80e6b887 at trap_pfault+0x57
> >> > #3 0xffffffff80e6b15f at trap+0x4bf
> >> > #4 0xffffffff80e4af97 at calltrap+0x8
> >> > #5 0xffffffff80b6c41b at ip_output+0x16b
> >> > #6 0xffffffff80b68e82 at icmp_reflect+0x5b2
> >> > #7 0xffffffff80b6883f at icmp_error+0x46f
> >> > #8 0xffffffff80beeb12 at udp_input+0x982
> >> > #9 0xffffffff80b69d1d at ip_input+0x17d
> >> > #10 0xffffffff80b08ba1 at netisr_dispatch_src+0x81
> >> > #11 0xffffffff80afecce at ether_demux+0x15e
> >> > #12 0xffffffff80affa14 at ether_nh_input+0x344
> >> > #13 0xffffffff80b08ba1 at netisr_dispatch_src+0x81
> >> > #14 0xffffffff80afefcf at ether_input+0x4f
> >> > #15 0xffffffff8089a5c3 at vtnet_rxq_eof+0x823
> >> > #16 0xffffffff8089b2ce at vtnet_rx_vq_intr+0x4e
> >> > #17 0xffffffff809e9ba6 at intr_event_execute_handlers+0x96
> >> >
> >> >
> >> > Fatal trap 12: page fault while in kernel mode
> >> > cpuid =3D 6; apic id =3D 06
> >> > fault virtual address =3D 0x5bd
> >> > fault code =3D supervisor read data, page not present
> >> > instruction pointer =3D 0x20:0xffffffff80b5de9e
> >> > stack pointer =3D 0x28:0xfffffe02b8d483e0
> >> > frame pointer =3D 0x28:0xfffffe02b8d48410
> >> > code segment =3D base 0x0, limit 0xfffff, type 0x1b
> >> > =3D DPL 0, pres 1, long 1, def32 0, gran 1
> >> > processor eflags =3D interrupt enabled, resume, IOPL =3D 0
> >> > current process =3D 12 (irq265: virtio_pci0)
> >> > [ thread pid 12 tid 100040 ]
> >> > Stopped at ip_fillid+0x8e: movzbl (%rax,%rcx,1),%esi
> >> > =3D=3D=3D End Log =3D=3D=3D
> >> >
> >> > Thanks,
> >> >
> >> > --
> >> > Shawn Webb
> >> > HardenedBSD
> >> >
> >> > GPG Key ID: 0x6A84658F52456EEE
> >> > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6=
EEE
> >
> > --
> > Shawn Webb
> > HardenedBSD
> >
> > GPG Key ID: 0x6A84658F52456EEE
> > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE
--=20
Shawn Webb
HardenedBSD
GPG Key ID: 0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE
--yrj/dFKFPuw6o+aM
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWjSitAAoJEGqEZY9SRW7udgUQALOysjXSlzrANZkBVvAgraNk
hGXfRTZFWacRY9IdB2cskjpDg3xKrKzBaUUKb38QGd0UHOXwJCLKe/5MOwJObagX
cx6eS/NYROcRhgBC3/u/fuFzUwmeICB33CIIzBrTjiTCQFoEV3RkF3zhea+fV6LN
AF4l8tb1f2C5N4xOmUicwmYLzGqzShPg1YYdHFsKpr9iAfivk/zIVidblblQ1ZwE
0ytNWGzJ1l2Jk37UnyeI5n/8ovbUHjmLSa9R60iJoY6rkCNSDBjQacEy6c32QXun
vckPVRGAqS765xooczBU7zJ29CMAr8T7xSA0htzwmI+uj8O4IjfuiCSucfkKoADh
win05LCZfzp9z3JuHLmrL8Cc2TJVbtWBeCLegyz7ZmDQka/q+z1z18VbYvjqvQjC
gf9+WJotEVvN27g9u4Jhqw8+piVFu/dGA6RUeh+n2122NPodDYLQiVnWMeJ8GSa+
j5uaUOKR+ULgFz5qbtPi0EnD1U2wpBc2AIsIWcaerhgk1raLXYEXCUJ08vPiEtd2
b7i/i6if/FWPyk1vPnLwisCjy4Uc52nfd6ep3r/C6EGeQLmHsXsHn79X6G3LlR4J
35rne7yX5yT30Gin+tgjBF/zfZ2J7O1JY6uIVtgJRRGAnJxTdmcZVsqzGXBBEDiS
W29Jo/ezCFHaVJEiiC4W
=4Hot
-----END PGP SIGNATURE-----
--yrj/dFKFPuw6o+aM--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160106144608.GA71037>