From owner-freebsd-current@freebsd.org Wed Jan 6 18:34:39 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5732DA65A46 for ; Wed, 6 Jan 2016 18:34:39 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qk0-x22c.google.com (mail-qk0-x22c.google.com [IPv6:2607:f8b0:400d:c09::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0D56D18BB for ; Wed, 6 Jan 2016 18:34:39 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qk0-x22c.google.com with SMTP id q19so101081653qke.3 for ; Wed, 06 Jan 2016 10:34:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=NO03mOdwbi/z6/c6GLf9YclvX2GvygKsMcT6N/oWX4s=; b=2BQNQgvvwHDhb7oIi7nPLd0NZNdF9ZteWFae6tv15yzMjLQQVPjXw3e69cA9ltcYvJ +DpTJGx9KZ8BkEJGDqY+rUa5qGqTPhYVtR/kReo9w+B+ntV9HmQX3pEXcd7dIgSzsNZf wRAN58uWo16xjg7ZbG8C7OiJGwsoG7SBDAy59XZE6FIHZHdVj9sxOinbaukTFUekSVTh HOD8/XZdUaEiF1IgRnAjWvonpCepA/LST+Fj1McodZLfKGOUEnz3JT8Fu71hCdvoTiOO Shjos/YimXxNv6xht5rlgtHvUnTFnxg2AxroqWTbWLNpaJw4+Ve/vf6/h+K5SHKN2+fN gt4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=NO03mOdwbi/z6/c6GLf9YclvX2GvygKsMcT6N/oWX4s=; b=K4leG29FsbwfvbHkViMKCaLneK8EA3+q/qloRahbAyztQPPjbgXe+KupBEjZZR+Jpm EGtEsG/+DFvTm3hsQbff+N7a35VvXGXdecYGBuSo60288Bt5Xh8aI5GOJFDeahIwin7n RjQtco0vcnOjkbfg3BlJnW0TM8cT0qfS+r38bR78f1DU8Bdhzpyr2HYDu+xPptaUDKPB ho5Fn17L4CR/hZyqRZpwqRv9FRNvwjvpPFPGcaqQMTeu+qjh6LkZQpJVzf1VZpZhJwGo acB6+sHdBTC1uukXu0VWeV50SflKLqqUfvnuhajYwtw+LoSThUDjAmPJMAQjel4tJpYG InhQ== X-Gm-Message-State: ALoCoQkeH6GDV4TMRQ7J24BMB+d5Zy5rSeSKNn/jtyicOZGF5IyMZW8IS4O0UvsJLCNYTgbSzbGIwRTcS1h8SqfOSqE/Dcg+DQ== X-Received: by 10.13.196.196 with SMTP id g187mr34162820ywd.47.1452105278134; Wed, 06 Jan 2016 10:34:38 -0800 (PST) Received: from mutt-hardenedbsd ([63.88.83.104]) by smtp.gmail.com with ESMTPSA id h187sm42428561ywc.0.2016.01.06.10.34.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Jan 2016 10:34:36 -0800 (PST) Date: Wed, 6 Jan 2016 13:34:34 -0500 From: Shawn Webb To: Adrian Chadd Cc: freebsd-current Subject: Re: kernel panic by enabling net.inet.ip.random_id Message-ID: <20160106183434.GB45169@mutt-hardenedbsd> References: <20160106015742.GA8405@mutt-hardenedbsd> <20160106021316.GB8405@mutt-hardenedbsd> <20160106144608.GA71037@mutt-hardenedbsd> <20160106182453.GA45169@mutt-hardenedbsd> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="8P1HSweYDcXXzwPJ" Content-Disposition: inline In-Reply-To: <20160106182453.GA45169@mutt-hardenedbsd> X-Operating-System: FreeBSD mutt-hardenedbsd 11.0-CURRENT-HBSD FreeBSD 11.0-CURRENT-HBSD X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jan 2016 18:34:39 -0000 --8P1HSweYDcXXzwPJ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Yup, calling ip_initid() in the SYSINIT works! Thanks for the help. Thanks, Shawn On Wed, Jan 06, 2016 at 01:24:53PM -0500, Shawn Webb wrote: > That's what gets toggled via the sysctl. I think I figured out that I > need to call ip_initid() in the SYSINIT. Compiling and testing now. >=20 > Thanks, >=20 > Shawn >=20 > On Wed, Jan 06, 2016 at 10:15:08AM -0800, Adrian Chadd wrote: > > Why'd you condition the vimage definition? :) > >=20 > >=20 > >=20 > > -a > >=20 > >=20 > > On 6 January 2016 at 06:46, Shawn Webb wro= te: > > > (kgdb) list *(0xffffffff80b5de9e) > > > 0xffffffff80b5de9e is in ip_fillid (/usr/src/sys/netinet/ip_id.c:237). > > > warning: Source file is more recent than executable. > > > > > > 232 new_id =3D 0; > > > 233 do { > > > 234 if (new_id !=3D 0) > > > 235 V_random_id_collisions++; > > > 236 arc4rand(&new_id, sizeof(new_id), 0); > > > 237 } while (bit_test(V_id_bits, new_id) || new_id =3D=3D= 0); > > > 238 bit_clear(V_id_bits, V_id_array[V_array_ptr]); > > > 239 bit_set(V_id_bits, new_id); > > > 240 V_id_array[V_array_ptr] =3D new_id; > > > 241 V_array_ptr++; > > > > > > This is the change I made to ip_id.c that caused the underlying kernel > > > panic: > > > https://github.com/HardenedBSD/hardenedBSD/commit/52d5a93b92097e7a79b= e8d2e0eb9c1a58b8337d1 > > > > > > Ideally, we should be able to just toggle that variable and all would= be > > > well. But it seems with the VIMAGE work, something is preventing that. > > > > > > Thanks, > > > > > > Shawn > > > > > > On Tue, Jan 05, 2016 at 06:22:34PM -0800, Adrian Chadd wrote: > > >> try list *(0x[address]) . > > >> > > >> That line is mtx_unlock(), which makes no sense (as mtx_lock succeed= ed fine.) > > >> > > >> > > >> -a > > >> > > >> > > >> On 5 January 2016 at 18:13, Shawn Webb = wrote: > > >> > Thanks for the quick reply! Here's some more debugging output: > > >> > > > >> > =3D=3D=3D Begin Log =3D=3D=3D > > >> > (kgdb) bt > > >> > #0 doadump (textdump=3D0) at pcpu.h:221 > > >> > #1 0xffffffff8037c78b in db_dump (dummy=3D, = dummy2=3Dfalse, dummy3=3D0, dummy4=3D0x0) at /usr/src/sys/ddb/db_command.c:= 533 > > >> > #2 0xffffffff8037c57e in db_command (cmd_table=3D0x0) at /usr/src= /sys/ddb/db_command.c:440 > > >> > #3 0xffffffff8037c314 in db_command_loop () at /usr/src/sys/ddb/d= b_command.c:493 > > >> > #4 0xffffffff8037edab in db_trap (type=3D, c= ode=3D0) at /usr/src/sys/ddb/db_main.c:251 > > >> > #5 0xffffffff80a5c563 in kdb_trap (type=3D12, code=3D0, tf=3D) at /usr/src/sys/kern/subr_kdb.c:654 > > >> > #6 0xffffffff80e6b7e1 in trap_fatal (frame=3D0xfffffe02c33894d0, = eva=3D) at /usr/src/sys/amd64/amd64/trap.c:829 > > >> > #7 0xffffffff80e6ba2d in trap_pfault (frame=3D0xfffffe02c33894d0,= usermode=3D) at /usr/src/sys/amd64/amd64/trap.c:684 > > >> > #8 0xffffffff80e6b15f in trap (frame=3D0xfffffe02c33894d0) at /us= r/src/sys/amd64/amd64/trap.c:435 > > >> > #9 0xffffffff80e4af97 in calltrap () at /usr/src/sys/amd64/amd64/= exception.S:234 > > >> > #10 0xffffffff80b5de9e in ip_fillid (ip=3D0xfffff8000ef8cb88) at /= usr/src/sys/netinet/ip_id.c:237 > > >> > #11 0xffffffff80b6c41b in ip_output (m=3D, op= t=3D, ro=3D, flags=3D0, imo=3D0x0= , inp=3D0xfffff8000e66e960) at /usr/src/sys/netinet/ip_output.c:268 > > >> > #12 0xffffffff80bf0612 in udp_send (so=3D, fl= ags=3D, m=3D, addr=3D0x0, control= =3D, td=3D0xfffff8000ef8cb88) at /usr/src/sys/netinet/= udp_usrreq.c:1517 > > >> > #13 0xffffffff80aa3872 in sosend_dgram (so=3D0xfffff8000e6422e8, a= ddr=3D0x0, uio=3D, top=3D0xfffff8000ef8cb00, control= =3D0x0, flags=3D, td=3D0xffffffff81bef2ec) at /usr/src= /sys/kern/uipc_socket.c:1164 > > >> > #13 0xffffffff80aa3872 in sosend_dgram (so=3D0xfffff8000e6422e8, a= ddr=3D0x0, uio=3D, top=3D0xfffff8000ef8cb00, control= =3D0x0, flags=3D, td=3D0xffffffff81bef2ec) at /usr/src= /sys/kern/uipc_socket.c:1164 > > >> > #14 0xffffffff80aaa03b in kern_sendit (td=3D0xfffff8000e4cd9c0, s= =3D6, mp=3D, flags=3D0, control=3D0x0, segflg=3DUIO_US= ERSPACE) at /usr/src/sys/kern/uipc_syscalls.c:906 > > >> > #15 0xffffffff80aaa336 in sendit (td=3D0xfffff8000e4cd9c0, s=3D, mp=3D0xfffffe02c3389970, flags=3D3980) at /usr/src/sys/= kern/uipc_syscalls.c:833 > > >> > #16 0xffffffff80aaa1fd in sys_sendto (td=3D0x0, uap=3D) at /usr/src/sys/kern/uipc_syscalls.c:957 > > >> > #17 0xffffffff80e6bfdb in amd64_syscall (td=3D0xfffff8000e4cd9c0, = traced=3D0) at subr_syscall.c:135 > > >> > #18 0xffffffff80e4b27b in Xfast_syscall () at /usr/src/sys/amd64/a= md64/exception.S:394 > > >> > #19 0x000003e339782e8a in ?? () > > >> > (kgdb) x/i 0xffffffff80b5de9e > > >> > 0xffffffff80b5de9e : movzbl (%rax,%rcx,1),%esi > > >> > (kgdb) info reg > > >> > rax 0x0 0 > > >> > rbx 0x0 0 > > >> > rcx 0x0 0 > > >> > rdx 0x0 0 > > >> > rsi 0x0 0 > > >> > rdi 0x0 0 > > >> > rbp 0xfffffe02c3388fe0 0xfffffe02c3388fe0 > > >> > rsp 0xfffffe02c3388fc8 0xfffffe02c3388fc8 > > >> > r8 0x0 0 > > >> > r9 0x0 0 > > >> > r10 0x0 0 > > >> > r11 0x0 0 > > >> > r12 0xffffffff817c0b80 -2122577024 > > >> > r13 0xffffffff817c1470 -2122574736 > > >> > r14 0x1 1 > > >> > r15 0x4 4 > > >> > rip 0xffffffff80a1fae3 0xffffffff80a1fae3 > > >> > eflags 0x0 0 > > >> > cs 0x0 0 > > >> > ss 0x0 0 > > >> > ds 0x0 0 > > >> > es 0x0 0 > > >> > fs 0x0 0 > > >> > gs 0x0 0 > > >> > =3D=3D=3D End Log =3D=3D=3D > > >> > > > >> > Thanks, > > >> > > > >> > Shawn > > >> > > > >> > On Tue, Jan 05, 2016 at 06:06:41PM -0800, Adrian Chadd wrote: > > >> >> looks like a null pointer deference. What's kgdb show at that IP? > > >> >> > > >> >> > > >> >> -a > > >> >> > > >> >> > > >> >> On 5 January 2016 at 17:57, Shawn Webb wrote: > > >> >> > Hey All, > > >> >> > > > >> >> > Here's a kernel panic I'm experiencing by enabling net.inet.ip.= random_id > > >> >> > at boot. > > >> >> > > > >> >> > I'm on latest HEAD on amd64 in bhyve. I'll soon-ish be testing = on native > > >> >> > hardware with VIMAGE enabled. > > >> >> > > > >> >> > =3D=3D=3D Begin Log =3D=3D=3D > > >> >> > Kernel page fault with the following non-sleepable locks held: > > >> >> > exclusive sleep mutex ip_id_mtx (ip_id_mtx) r =3D 0 (0xffffffff= 81c54830) locked @ /usr/src/sys/netinet/ip_id.c:227 > > >> >> > stack backtrace: > > >> >> > #0 0xffffffff80a79620 at witness_debugger+0x70 > > >> >> > #1 0xffffffff80a7a937 at witness_warn+0x3d7 > > >> >> > #2 0xffffffff80e6b887 at trap_pfault+0x57 > > >> >> > #3 0xffffffff80e6b15f at trap+0x4bf > > >> >> > #4 0xffffffff80e4af97 at calltrap+0x8 > > >> >> > #5 0xffffffff80b6c41b at ip_output+0x16b > > >> >> > #6 0xffffffff80b68e82 at icmp_reflect+0x5b2 > > >> >> > #7 0xffffffff80b6883f at icmp_error+0x46f > > >> >> > #8 0xffffffff80beeb12 at udp_input+0x982 > > >> >> > #9 0xffffffff80b69d1d at ip_input+0x17d > > >> >> > #10 0xffffffff80b08ba1 at netisr_dispatch_src+0x81 > > >> >> > #11 0xffffffff80afecce at ether_demux+0x15e > > >> >> > #12 0xffffffff80affa14 at ether_nh_input+0x344 > > >> >> > #13 0xffffffff80b08ba1 at netisr_dispatch_src+0x81 > > >> >> > #14 0xffffffff80afefcf at ether_input+0x4f > > >> >> > #15 0xffffffff8089a5c3 at vtnet_rxq_eof+0x823 > > >> >> > #16 0xffffffff8089b2ce at vtnet_rx_vq_intr+0x4e > > >> >> > #17 0xffffffff809e9ba6 at intr_event_execute_handlers+0x96 > > >> >> > > > >> >> > > > >> >> > Fatal trap 12: page fault while in kernel mode > > >> >> > cpuid =3D 6; apic id =3D 06 > > >> >> > fault virtual address =3D 0x5bd > > >> >> > fault code =3D supervisor read data, page not pres= ent > > >> >> > instruction pointer =3D 0x20:0xffffffff80b5de9e > > >> >> > stack pointer =3D 0x28:0xfffffe02b8d483e0 > > >> >> > frame pointer =3D 0x28:0xfffffe02b8d48410 > > >> >> > code segment =3D base 0x0, limit 0xfffff, type 0x1b > > >> >> > =3D DPL 0, pres 1, long 1, def32 0, gra= n 1 > > >> >> > processor eflags =3D interrupt enabled, resume, IOPL =3D= 0 > > >> >> > current process =3D 12 (irq265: virtio_pci0) > > >> >> > [ thread pid 12 tid 100040 ] > > >> >> > Stopped at ip_fillid+0x8e: movzbl (%rax,%rcx,1),%esi > > >> >> > =3D=3D=3D End Log =3D=3D=3D > > >> >> > > > >> >> > Thanks, > > >> >> > > > >> >> > -- > > >> >> > Shawn Webb > > >> >> > HardenedBSD > > >> >> > > > >> >> > GPG Key ID: 0x6A84658F52456EEE > > >> >> > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5= 245 6EEE > > >> > > > >> > -- > > >> > Shawn Webb > > >> > HardenedBSD > > >> > > > >> > GPG Key ID: 0x6A84658F52456EEE > > >> > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245= 6EEE > > > > > > -- > > > Shawn Webb > > > HardenedBSD > > > > > > GPG Key ID: 0x6A84658F52456EEE > > > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6E= EE >=20 > --=20 > Shawn Webb > HardenedBSD >=20 > GPG Key ID: 0x6A84658F52456EEE > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --=20 Shawn Webb HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --8P1HSweYDcXXzwPJ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIaBAEBCAAGBQJWjV46AAoJEGqEZY9SRW7uW3wP6wUsAEIdsKq4u0/UZPgKqvrL 4Btl4BSuFE6IE8dp3/X0442it1R9r+6Kqx4irDbowseZeyjv9v9iFtZQsy+o/NgA UmokFXanlwR8+L5Rcx7kNH44j/FgYrmh4kHffMFYuhcVi6CxRzhWFDe2aXEFRpfr dGRCUjiR+Ia/rBrZZ8/+4zlCxjQhO8H8nXghG67czOHLikitlVmwCyDcnW1ehLuD W2+Xnx06MOQDF+d1Bs8otli9OhEAS8Seef4GVLrnHtawUIczxpEPB273WhbbGiOi 4aOXwNSBXHRzB3RPXlD/jOhFBj187EC113AFSbeyF5bX+aiNzoe3xZSQMDquB3e8 68jtTSZEZJsTP9f7b7xHMns89jEv1Rf2RCHap920fdxH1s5rSrwlKQ43AT4xVJpq 5iWcftM6PEnuzZkDU2CP/QTXHEH74UpqcCUFGx5akR8ZnNzOq4+6Y3AIeSMHECV8 KET5zy1ONedb7VFSpzes5iX8fGqyA2xFUQzRvM/clp2LkcxtLViJfEJE1Ge0zQKK gZhMC2AZFuVStkkeoUErYI4eMW4cBzh7K2+kpP0R2pgHyTG5QNupfZgXsLORRMbV 7QTmboJZ0lQgOYBY54Pp04TAvtu4fbwUGzMrMgKwlaCRBBm+4l/7U/WZhwGqQGch txI4ocGHQPZVfYRHNw== =MBUa -----END PGP SIGNATURE----- --8P1HSweYDcXXzwPJ--