Date: Wed, 6 Jan 2016 13:34:34 -0500 From: Shawn Webb <shawn.webb@hardenedbsd.org> To: Adrian Chadd <adrian.chadd@gmail.com> Cc: freebsd-current <freebsd-current@freebsd.org> Subject: Re: kernel panic by enabling net.inet.ip.random_id Message-ID: <20160106183434.GB45169@mutt-hardenedbsd> In-Reply-To: <20160106182453.GA45169@mutt-hardenedbsd> References: <20160106015742.GA8405@mutt-hardenedbsd> <CAJ-VmonnHgpCxN%2BVvrP9j%2BtHK=3Yxjz0qa9kd8riSaUEhJnNtg@mail.gmail.com> <20160106021316.GB8405@mutt-hardenedbsd> <CAJ-VmonZO8WzrTMS394AJw8duvbW=%2B2bEfaQDzkkaC5HHcmAxA@mail.gmail.com> <20160106144608.GA71037@mutt-hardenedbsd> <CAJ-VmonFRzw4SMLuNMartzMT2%2B-4ZnsARvbC_SySaj95aOA_TQ@mail.gmail.com> <20160106182453.GA45169@mutt-hardenedbsd>
next in thread | previous in thread | raw e-mail | index | archive | help
--8P1HSweYDcXXzwPJ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Yup, calling ip_initid() in the SYSINIT works! Thanks for the help.
Thanks,
Shawn
On Wed, Jan 06, 2016 at 01:24:53PM -0500, Shawn Webb wrote:
> That's what gets toggled via the sysctl. I think I figured out that I
> need to call ip_initid() in the SYSINIT. Compiling and testing now.
>=20
> Thanks,
>=20
> Shawn
>=20
> On Wed, Jan 06, 2016 at 10:15:08AM -0800, Adrian Chadd wrote:
> > Why'd you condition the vimage definition? :)
> >=20
> >=20
> >=20
> > -a
> >=20
> >=20
> > On 6 January 2016 at 06:46, Shawn Webb <shawn.webb@hardenedbsd.org> wro=
te:
> > > (kgdb) list *(0xffffffff80b5de9e)
> > > 0xffffffff80b5de9e is in ip_fillid (/usr/src/sys/netinet/ip_id.c:237).
> > > warning: Source file is more recent than executable.
> > >
> > > 232 new_id =3D 0;
> > > 233 do {
> > > 234 if (new_id !=3D 0)
> > > 235 V_random_id_collisions++;
> > > 236 arc4rand(&new_id, sizeof(new_id), 0);
> > > 237 } while (bit_test(V_id_bits, new_id) || new_id =3D=3D=
0);
> > > 238 bit_clear(V_id_bits, V_id_array[V_array_ptr]);
> > > 239 bit_set(V_id_bits, new_id);
> > > 240 V_id_array[V_array_ptr] =3D new_id;
> > > 241 V_array_ptr++;
> > >
> > > This is the change I made to ip_id.c that caused the underlying kernel
> > > panic:
> > > https://github.com/HardenedBSD/hardenedBSD/commit/52d5a93b92097e7a79b=
e8d2e0eb9c1a58b8337d1
> > >
> > > Ideally, we should be able to just toggle that variable and all would=
be
> > > well. But it seems with the VIMAGE work, something is preventing that.
> > >
> > > Thanks,
> > >
> > > Shawn
> > >
> > > On Tue, Jan 05, 2016 at 06:22:34PM -0800, Adrian Chadd wrote:
> > >> try list *(0x[address]) .
> > >>
> > >> That line is mtx_unlock(), which makes no sense (as mtx_lock succeed=
ed fine.)
> > >>
> > >>
> > >> -a
> > >>
> > >>
> > >> On 5 January 2016 at 18:13, Shawn Webb <shawn.webb@hardenedbsd.org> =
wrote:
> > >> > Thanks for the quick reply! Here's some more debugging output:
> > >> >
> > >> > =3D=3D=3D Begin Log =3D=3D=3D
> > >> > (kgdb) bt
> > >> > #0 doadump (textdump=3D0) at pcpu.h:221
> > >> > #1 0xffffffff8037c78b in db_dump (dummy=3D<value optimized out>, =
dummy2=3Dfalse, dummy3=3D0, dummy4=3D0x0) at /usr/src/sys/ddb/db_command.c:=
533
> > >> > #2 0xffffffff8037c57e in db_command (cmd_table=3D0x0) at /usr/src=
/sys/ddb/db_command.c:440
> > >> > #3 0xffffffff8037c314 in db_command_loop () at /usr/src/sys/ddb/d=
b_command.c:493
> > >> > #4 0xffffffff8037edab in db_trap (type=3D<value optimized out>, c=
ode=3D0) at /usr/src/sys/ddb/db_main.c:251
> > >> > #5 0xffffffff80a5c563 in kdb_trap (type=3D12, code=3D0, tf=3D<val=
ue optimized out>) at /usr/src/sys/kern/subr_kdb.c:654
> > >> > #6 0xffffffff80e6b7e1 in trap_fatal (frame=3D0xfffffe02c33894d0, =
eva=3D<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:829
> > >> > #7 0xffffffff80e6ba2d in trap_pfault (frame=3D0xfffffe02c33894d0,=
usermode=3D<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:684
> > >> > #8 0xffffffff80e6b15f in trap (frame=3D0xfffffe02c33894d0) at /us=
r/src/sys/amd64/amd64/trap.c:435
> > >> > #9 0xffffffff80e4af97 in calltrap () at /usr/src/sys/amd64/amd64/=
exception.S:234
> > >> > #10 0xffffffff80b5de9e in ip_fillid (ip=3D0xfffff8000ef8cb88) at /=
usr/src/sys/netinet/ip_id.c:237
> > >> > #11 0xffffffff80b6c41b in ip_output (m=3D<value optimized out>, op=
t=3D<value optimized out>, ro=3D<value optimized out>, flags=3D0, imo=3D0x0=
, inp=3D0xfffff8000e66e960) at /usr/src/sys/netinet/ip_output.c:268
> > >> > #12 0xffffffff80bf0612 in udp_send (so=3D<value optimized out>, fl=
ags=3D<value optimized out>, m=3D<value optimized out>, addr=3D0x0, control=
=3D<value optimized out>, td=3D0xfffff8000ef8cb88) at /usr/src/sys/netinet/=
udp_usrreq.c:1517
> > >> > #13 0xffffffff80aa3872 in sosend_dgram (so=3D0xfffff8000e6422e8, a=
ddr=3D0x0, uio=3D<value optimized out>, top=3D0xfffff8000ef8cb00, control=
=3D0x0, flags=3D<value optimized out>, td=3D0xffffffff81bef2ec) at /usr/src=
/sys/kern/uipc_socket.c:1164
> > >> > #13 0xffffffff80aa3872 in sosend_dgram (so=3D0xfffff8000e6422e8, a=
ddr=3D0x0, uio=3D<value optimized out>, top=3D0xfffff8000ef8cb00, control=
=3D0x0, flags=3D<value optimized out>, td=3D0xffffffff81bef2ec) at /usr/src=
/sys/kern/uipc_socket.c:1164
> > >> > #14 0xffffffff80aaa03b in kern_sendit (td=3D0xfffff8000e4cd9c0, s=
=3D6, mp=3D<value optimized out>, flags=3D0, control=3D0x0, segflg=3DUIO_US=
ERSPACE) at /usr/src/sys/kern/uipc_syscalls.c:906
> > >> > #15 0xffffffff80aaa336 in sendit (td=3D0xfffff8000e4cd9c0, s=3D<va=
lue optimized out>, mp=3D0xfffffe02c3389970, flags=3D3980) at /usr/src/sys/=
kern/uipc_syscalls.c:833
> > >> > #16 0xffffffff80aaa1fd in sys_sendto (td=3D0x0, uap=3D<value optim=
ized out>) at /usr/src/sys/kern/uipc_syscalls.c:957
> > >> > #17 0xffffffff80e6bfdb in amd64_syscall (td=3D0xfffff8000e4cd9c0, =
traced=3D0) at subr_syscall.c:135
> > >> > #18 0xffffffff80e4b27b in Xfast_syscall () at /usr/src/sys/amd64/a=
md64/exception.S:394
> > >> > #19 0x000003e339782e8a in ?? ()
> > >> > (kgdb) x/i 0xffffffff80b5de9e
> > >> > 0xffffffff80b5de9e <ip_fillid+142>: movzbl (%rax,%rcx,1),%esi
> > >> > (kgdb) info reg
> > >> > rax 0x0 0
> > >> > rbx 0x0 0
> > >> > rcx 0x0 0
> > >> > rdx 0x0 0
> > >> > rsi 0x0 0
> > >> > rdi 0x0 0
> > >> > rbp 0xfffffe02c3388fe0 0xfffffe02c3388fe0
> > >> > rsp 0xfffffe02c3388fc8 0xfffffe02c3388fc8
> > >> > r8 0x0 0
> > >> > r9 0x0 0
> > >> > r10 0x0 0
> > >> > r11 0x0 0
> > >> > r12 0xffffffff817c0b80 -2122577024
> > >> > r13 0xffffffff817c1470 -2122574736
> > >> > r14 0x1 1
> > >> > r15 0x4 4
> > >> > rip 0xffffffff80a1fae3 0xffffffff80a1fae3 <doadum=
p+51>
> > >> > eflags 0x0 0
> > >> > cs 0x0 0
> > >> > ss 0x0 0
> > >> > ds 0x0 0
> > >> > es 0x0 0
> > >> > fs 0x0 0
> > >> > gs 0x0 0
> > >> > =3D=3D=3D End Log =3D=3D=3D
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Shawn
> > >> >
> > >> > On Tue, Jan 05, 2016 at 06:06:41PM -0800, Adrian Chadd wrote:
> > >> >> looks like a null pointer deference. What's kgdb show at that IP?
> > >> >>
> > >> >>
> > >> >> -a
> > >> >>
> > >> >>
> > >> >> On 5 January 2016 at 17:57, Shawn Webb <shawn.webb@hardenedbsd.or=
g> wrote:
> > >> >> > Hey All,
> > >> >> >
> > >> >> > Here's a kernel panic I'm experiencing by enabling net.inet.ip.=
random_id
> > >> >> > at boot.
> > >> >> >
> > >> >> > I'm on latest HEAD on amd64 in bhyve. I'll soon-ish be testing =
on native
> > >> >> > hardware with VIMAGE enabled.
> > >> >> >
> > >> >> > =3D=3D=3D Begin Log =3D=3D=3D
> > >> >> > Kernel page fault with the following non-sleepable locks held:
> > >> >> > exclusive sleep mutex ip_id_mtx (ip_id_mtx) r =3D 0 (0xffffffff=
81c54830) locked @ /usr/src/sys/netinet/ip_id.c:227
> > >> >> > stack backtrace:
> > >> >> > #0 0xffffffff80a79620 at witness_debugger+0x70
> > >> >> > #1 0xffffffff80a7a937 at witness_warn+0x3d7
> > >> >> > #2 0xffffffff80e6b887 at trap_pfault+0x57
> > >> >> > #3 0xffffffff80e6b15f at trap+0x4bf
> > >> >> > #4 0xffffffff80e4af97 at calltrap+0x8
> > >> >> > #5 0xffffffff80b6c41b at ip_output+0x16b
> > >> >> > #6 0xffffffff80b68e82 at icmp_reflect+0x5b2
> > >> >> > #7 0xffffffff80b6883f at icmp_error+0x46f
> > >> >> > #8 0xffffffff80beeb12 at udp_input+0x982
> > >> >> > #9 0xffffffff80b69d1d at ip_input+0x17d
> > >> >> > #10 0xffffffff80b08ba1 at netisr_dispatch_src+0x81
> > >> >> > #11 0xffffffff80afecce at ether_demux+0x15e
> > >> >> > #12 0xffffffff80affa14 at ether_nh_input+0x344
> > >> >> > #13 0xffffffff80b08ba1 at netisr_dispatch_src+0x81
> > >> >> > #14 0xffffffff80afefcf at ether_input+0x4f
> > >> >> > #15 0xffffffff8089a5c3 at vtnet_rxq_eof+0x823
> > >> >> > #16 0xffffffff8089b2ce at vtnet_rx_vq_intr+0x4e
> > >> >> > #17 0xffffffff809e9ba6 at intr_event_execute_handlers+0x96
> > >> >> >
> > >> >> >
> > >> >> > Fatal trap 12: page fault while in kernel mode
> > >> >> > cpuid =3D 6; apic id =3D 06
> > >> >> > fault virtual address =3D 0x5bd
> > >> >> > fault code =3D supervisor read data, page not pres=
ent
> > >> >> > instruction pointer =3D 0x20:0xffffffff80b5de9e
> > >> >> > stack pointer =3D 0x28:0xfffffe02b8d483e0
> > >> >> > frame pointer =3D 0x28:0xfffffe02b8d48410
> > >> >> > code segment =3D base 0x0, limit 0xfffff, type 0x1b
> > >> >> > =3D DPL 0, pres 1, long 1, def32 0, gra=
n 1
> > >> >> > processor eflags =3D interrupt enabled, resume, IOPL =3D=
0
> > >> >> > current process =3D 12 (irq265: virtio_pci0)
> > >> >> > [ thread pid 12 tid 100040 ]
> > >> >> > Stopped at ip_fillid+0x8e: movzbl (%rax,%rcx,1),%esi
> > >> >> > =3D=3D=3D End Log =3D=3D=3D
> > >> >> >
> > >> >> > Thanks,
> > >> >> >
> > >> >> > --
> > >> >> > Shawn Webb
> > >> >> > HardenedBSD
> > >> >> >
> > >> >> > GPG Key ID: 0x6A84658F52456EEE
> > >> >> > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5=
245 6EEE
> > >> >
> > >> > --
> > >> > Shawn Webb
> > >> > HardenedBSD
> > >> >
> > >> > GPG Key ID: 0x6A84658F52456EEE
> > >> > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245=
6EEE
> > >
> > > --
> > > Shawn Webb
> > > HardenedBSD
> > >
> > > GPG Key ID: 0x6A84658F52456EEE
> > > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6E=
EE
>=20
> --=20
> Shawn Webb
> HardenedBSD
>=20
> GPG Key ID: 0x6A84658F52456EEE
> GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE
--=20
Shawn Webb
HardenedBSD
GPG Key ID: 0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE
--8P1HSweYDcXXzwPJ
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIaBAEBCAAGBQJWjV46AAoJEGqEZY9SRW7uW3wP6wUsAEIdsKq4u0/UZPgKqvrL
4Btl4BSuFE6IE8dp3/X0442it1R9r+6Kqx4irDbowseZeyjv9v9iFtZQsy+o/NgA
UmokFXanlwR8+L5Rcx7kNH44j/FgYrmh4kHffMFYuhcVi6CxRzhWFDe2aXEFRpfr
dGRCUjiR+Ia/rBrZZ8/+4zlCxjQhO8H8nXghG67czOHLikitlVmwCyDcnW1ehLuD
W2+Xnx06MOQDF+d1Bs8otli9OhEAS8Seef4GVLrnHtawUIczxpEPB273WhbbGiOi
4aOXwNSBXHRzB3RPXlD/jOhFBj187EC113AFSbeyF5bX+aiNzoe3xZSQMDquB3e8
68jtTSZEZJsTP9f7b7xHMns89jEv1Rf2RCHap920fdxH1s5rSrwlKQ43AT4xVJpq
5iWcftM6PEnuzZkDU2CP/QTXHEH74UpqcCUFGx5akR8ZnNzOq4+6Y3AIeSMHECV8
KET5zy1ONedb7VFSpzes5iX8fGqyA2xFUQzRvM/clp2LkcxtLViJfEJE1Ge0zQKK
gZhMC2AZFuVStkkeoUErYI4eMW4cBzh7K2+kpP0R2pgHyTG5QNupfZgXsLORRMbV
7QTmboJZ0lQgOYBY54Pp04TAvtu4fbwUGzMrMgKwlaCRBBm+4l/7U/WZhwGqQGch
txI4ocGHQPZVfYRHNw==
=MBUa
-----END PGP SIGNATURE-----
--8P1HSweYDcXXzwPJ--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160106183434.GB45169>