From owner-freebsd-cloud@freebsd.org Fri Jan 12 09:23:32 2018 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 58BF0E7D13F; Fri, 12 Jan 2018 09:23:32 +0000 (UTC) (envelope-from bhughes@freebsd.org) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 316917FAA7; Fri, 12 Jan 2018 09:23:31 +0000 (UTC) (envelope-from bhughes@freebsd.org) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id EFA7D20BB8; Fri, 12 Jan 2018 04:23:29 -0500 (EST) Received: from frontend1 ([10.202.2.160]) by compute1.internal (MEProxy); Fri, 12 Jan 2018 04:23:29 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=t70e03tqjyV1mIKE82t/0hVNERQtS fR/GmghQQtRWCI=; b=Z1xs8mNiKIvIFaSx0T4iXjGE9ziOjE3PUaxx1/HCoNEBp 2n19epn3ZjBgCxbm5nQp3oEOVoAdIX/aHNsTObVkgPWNVVEDhxfAa18E+msOEyNw rj6eNIo0LOOeA5Q5N/L34xsAAFb4DrKujinOtCt5YOUIQXNRY7zvII5gtTZx7/o8 A1lGJVBYZRtAteuTPChyIM+qyX8d/LVFOp/zRRBaCUvI6lRSlrH4MvBH403EJcc8 IwGnI7S8MWrwuJeG/6tmuNWy4LXVzsFeji6/nAwPj5PRMqroxoHjEnW1VQx/CH1R FKg0oQ+Le8pfhCs4MVlCg0j2iGAu6Znrr/fEmH7EQ== X-ME-Sender: Received: from hayseed.lan (90.89-11-136.nextgentel.com [89.11.136.90]) by mail.messagingengine.com (Postfix) with ESMTPA id 594DE7E51F; Fri, 12 Jan 2018 04:23:29 -0500 (EST) From: "Bradley T. Hughes" Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Panic with recent -CURRENT kernel in EC2 Message-Id: Date: Fri, 12 Jan 2018 10:23:27 +0100 Cc: freebsd-net@freebsd.org To: freebsd-cloud@freebsd.org X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jan 2018 09:23:32 -0000 (Cross-posting to -net, but I'm not subscribed there... yet) I was wondering if anyone else has experienced this panic when dhclient = tries to get an IP address after booting a recent 12.0-CURRENT kernel? = Using the 12.0-CURRENT AMI from early November works, but with the = latest snapshot or when using a kernel built from source I get this: ixv0: mem = 0xf3000000-0xf3003fff,0xf3004000-0xf3007fff at device 3.0 on pci0 ixv0: using 2048 tx descriptors and 2048 rx descriptors ixv0: msix_init qsets capped at 1 ixv0: pxm cpus: 2 queue msgs: 2 admincnt: 1 ixv0: using 1 rx queues 1 tx queues=20 ixv0: Using MSIX interrupts with 2 vectors ixv0: allocated for 1 queues ixv0: allocated for 1 rx queues taskqgroup_attach_cpu: setaffinity failed: 22 taskqgroup_attach_cpu: setaffinity failed: 22 ixv0: Ethernet address: 0a:ad:68:70:ea:78 ixv0: netmap queues/slots: TX 1/2048, RX 1/2048 ... panic: Assertion if_getdrvflags(ifp) =3D=3D i failed at = /usr/src/sys/net/iflib.c:2199 cpuid =3D 3 time =3D 1515673176 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame = 0xfffffe008a8ee600 vpanic() at vpanic+0x19c/frame 0xfffffe008a8ee680 kassert_panic() at kassert_panic+0x126/frame 0xfffffe008a8ee6f0 iflib_init_locked() at iflib_init_locked+0x75f/frame 0xfffffe008a8ee750 iflib_if_ioctl() at iflib_if_ioctl+0x744/frame 0xfffffe008a8ee7b0 ifioctl() at ifioctl+0x1903/frame 0xfffffe008a8ee850 kern_ioctl() at kern_ioctl+0x2c4/frame 0xfffffe008a8ee8b0 sys_ioctl() at sys_ioctl+0x15c/frame 0xfffffe008a8ee980 amd64_syscall() at amd64_syscall+0x79b/frame 0xfffffe008a8eeab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe008a8eeab0 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x80100a4fa, rsp =3D = 0x7fffffffe408, rbp =3D 0x7fffffffe460 --- Thanks in advance :) -- Bradley T. Hughes bhughes@freebsd.org From owner-freebsd-cloud@freebsd.org Fri Jan 12 10:43:40 2018 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 14E67EA4C80 for ; Fri, 12 Jan 2018 10:43:40 +0000 (UTC) (envelope-from 01000160e9f7b2db-51e1b120-17fb-4d5f-9e59-b0835d3f2bbf-000000@amazonses.com) Received: from a8-52.smtp-out.amazonses.com (a8-52.smtp-out.amazonses.com [54.240.8.52]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CF84C82A55 for ; Fri, 12 Jan 2018 10:43:39 +0000 (UTC) (envelope-from 01000160e9f7b2db-51e1b120-17fb-4d5f-9e59-b0835d3f2bbf-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ae7m2yrxjw65l2cqdpjxuucyrvy564tn; d=tarsnap.com; t=1515753812; h=Subject:To:References:Cc:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; bh=UzDhRL672GFAaBZr9f2/SNYeJ2+9AAa6DUIxQa6FVIk=; b=rtDTIqdRCli8aXrS/h5ASEHJirPrIp8uY5cTsEaxTI339Y/m5MXf3yAgRk7Jq36T i6vJ48OBxOjRLcOwb4aTrCzIUXOkej4ZJG9Ut1o6FyAkQKMgCzDqOxSahEoNW6klHDa UDdplwTF/mYvfZhfiypx0lajIMANrsDhrUpTBrx0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=224i4yxa5dv7c2xz3womw6peuasteono; d=amazonses.com; t=1515753812; h=Subject:To:References:Cc:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:Feedback-ID; bh=UzDhRL672GFAaBZr9f2/SNYeJ2+9AAa6DUIxQa6FVIk=; b=ipOGtZQ9/gNr4CDpeBKxTQR3puGIiya20IuuNeGT5++BssaHEFPGp8trB/zWH0LZ dLvezbOKWB31PrWEJBk/klESdPcJDpNWie7p+9QdCoxZ0w/QaCDo04+6jMz9q5/auJJ bzjvrWS5KkP1+urUriHWgcifbATqdXuNXJnlatm0= Subject: Re: Panic with recent -CURRENT kernel in EC2 To: "Bradley T. Hughes" , freebsd-cloud@freebsd.org References: Cc: freebsd-net@freebsd.org, Eric Joyner From: Colin Percival Message-ID: <01000160e9f7b2db-51e1b120-17fb-4d5f-9e59-b0835d3f2bbf-000000@email.amazonses.com> Date: Fri, 12 Jan 2018 10:43:32 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-SES-Outgoing: 2018.01.12-54.240.8.52 Feedback-ID: 1.us-east-1.Lv9FVjaNvvR5llaqfLoOVbo2VxOELl7cjN0AOyXnPlk=:AmazonSES X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jan 2018 10:43:40 -0000 This sounds like it may be related to r327031, which make ixgbe (and thus ixv) use iflib... maybe erj@ will have some ideas here? In case it helps, dhclient sets the MTU, which in the past has caused problems due to drivers resetting -- the fact that sys_ioctl is ending up in iflib_init_locked makes me wonder if this is a similar issue. (If it is, there's a locking problem somewhere...) Colin Percival On 01/12/18 01:23, Bradley T. Hughes wrote: > (Cross-posting to -net, but I'm not subscribed there... yet) > > I was wondering if anyone else has experienced this panic when dhclient tries to get an IP address after booting a recent 12.0-CURRENT kernel? Using the 12.0-CURRENT AMI from early November works, but with the latest snapshot or when using a kernel built from source I get this: > > ixv0: mem 0xf3000000-0xf3003fff,0xf3004000-0xf3007fff at device 3.0 on pci0 > ixv0: using 2048 tx descriptors and 2048 rx descriptors > ixv0: msix_init qsets capped at 1 > ixv0: pxm cpus: 2 queue msgs: 2 admincnt: 1 > ixv0: using 1 rx queues 1 tx queues > ixv0: Using MSIX interrupts with 2 vectors > ixv0: allocated for 1 queues > ixv0: allocated for 1 rx queues > taskqgroup_attach_cpu: setaffinity failed: 22 > taskqgroup_attach_cpu: setaffinity failed: 22 > ixv0: Ethernet address: 0a:ad:68:70:ea:78 > ixv0: netmap queues/slots: TX 1/2048, RX 1/2048 > ... > panic: Assertion if_getdrvflags(ifp) == i failed at /usr/src/sys/net/iflib.c:2199 > cpuid = 3 > time = 1515673176 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe008a8ee600 > vpanic() at vpanic+0x19c/frame 0xfffffe008a8ee680 > kassert_panic() at kassert_panic+0x126/frame 0xfffffe008a8ee6f0 > iflib_init_locked() at iflib_init_locked+0x75f/frame 0xfffffe008a8ee750 > iflib_if_ioctl() at iflib_if_ioctl+0x744/frame 0xfffffe008a8ee7b0 > ifioctl() at ifioctl+0x1903/frame 0xfffffe008a8ee850 > kern_ioctl() at kern_ioctl+0x2c4/frame 0xfffffe008a8ee8b0 > sys_ioctl() at sys_ioctl+0x15c/frame 0xfffffe008a8ee980 > amd64_syscall() at amd64_syscall+0x79b/frame 0xfffffe008a8eeab0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe008a8eeab0 > --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x80100a4fa, rsp = 0x7fffffffe408, rbp = 0x7fffffffe460 --- > > Thanks in advance :) -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid From owner-freebsd-cloud@freebsd.org Fri Jan 12 16:16:21 2018 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 62BF5E6DC33; Fri, 12 Jan 2018 16:16:21 +0000 (UTC) (envelope-from jeffrey.e.pieper@intel.com) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "fmsmga102.fm.intel.com", Issuer "COMODO RSA Organization Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 414E271C73; Fri, 12 Jan 2018 16:16:20 +0000 (UTC) (envelope-from jeffrey.e.pieper@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 12 Jan 2018 08:16:13 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,349,1511856000"; d="scan'208";a="10755483" Received: from orsmsx109.amr.corp.intel.com ([10.22.240.7]) by orsmga006.jf.intel.com with ESMTP; 12 Jan 2018 08:16:13 -0800 Received: from orsmsx111.amr.corp.intel.com ([169.254.12.174]) by ORSMSX109.amr.corp.intel.com ([169.254.11.160]) with mapi id 14.03.0319.002; Fri, 12 Jan 2018 08:16:12 -0800 From: "Pieper, Jeffrey E" To: Colin Percival , "Bradley T. Hughes" , "freebsd-cloud@freebsd.org" CC: "freebsd-net@freebsd.org" , Eric Joyner Subject: Re: Panic with recent -CURRENT kernel in EC2 Thread-Topic: Panic with recent -CURRENT kernel in EC2 Thread-Index: AQHTi4caKgDIaBWA90C4d85+u0sUrqNwk22A///W1QA= Date: Fri, 12 Jan 2018 16:16:12 +0000 Message-ID: <191060CB-F1F8-4E34-8191-99EB7B28597C@intel.com> References: <01000160e9f7ae62-92937548-d46c-4687-87c0-2205119d5132-000000@email.amazonses.com> In-Reply-To: <01000160e9f7ae62-92937548-d46c-4687-87c0-2205119d5132-000000@email.amazonses.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/f.29.0.171205 x-originating-ip: [134.134.172.139] Content-Type: text/plain; charset="utf-8" Content-ID: <0DAB4051CA743F44984DA395EC2DA288@intel.com> Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jan 2018 16:16:21 -0000 SHJt4oCmDQoNCj4gdGFza3Fncm91cF9hdHRhY2hfY3B1OiBzZXRhZmZpbml0eSBmYWlsZWQ6IDIy DQo+IHRhc2txZ3JvdXBfYXR0YWNoX2NwdTogc2V0YWZmaW5pdHkgZmFpbGVkOiAyMg0KDQpJcyB0 aGlzIGEgZnJlc2ggc25hcHNob3Q/IFRoYXQgc2hvdWxkIGhhdmUgYmVlbiBmaXhlZCBpbiBodHRw czovL3N2bndlYi5mcmVlYnNkLm9yZy9iYXNlP3ZpZXc9cmV2aXNpb24mcmV2aXNpb249MzI3MDEz IC4gQ2FuIHlvdSBwbGVhc2UgZmlsZSBhIGJ1ZyBvbiB0aGlzPyBUaGlzIGxvb2tzIGxpa2UgYW4g aWZsaWIgaXNzdWUuIA0KDQpUaGFua3MsDQpKZWZmDQoNCk9uIDEvMTIvMTgsIDI6NDMgQU0sICJv d25lci1mcmVlYnNkLW5ldEBmcmVlYnNkLm9yZyBvbiBiZWhhbGYgb2YgQ29saW4gUGVyY2l2YWwi IDxvd25lci1mcmVlYnNkLW5ldEBmcmVlYnNkLm9yZyBvbiBiZWhhbGYgb2YgY3BlcmNpdmFAdGFy c25hcC5jb20+IHdyb3RlOg0KDQogICAgVGhpcyBzb3VuZHMgbGlrZSBpdCBtYXkgYmUgcmVsYXRl ZCB0byByMzI3MDMxLCB3aGljaCBtYWtlIGl4Z2JlIChhbmQgdGh1cyBpeHYpDQogICAgdXNlIGlm bGliLi4uIG1heWJlIGVyakAgd2lsbCBoYXZlIHNvbWUgaWRlYXMgaGVyZT8NCiAgICANCiAgICBJ biBjYXNlIGl0IGhlbHBzLCBkaGNsaWVudCBzZXRzIHRoZSBNVFUsIHdoaWNoIGluIHRoZSBwYXN0 IGhhcyBjYXVzZWQNCiAgICBwcm9ibGVtcyBkdWUgdG8gZHJpdmVycyByZXNldHRpbmcgLS0gdGhl IGZhY3QgdGhhdCBzeXNfaW9jdGwgaXMgZW5kaW5nDQogICAgdXAgaW4gaWZsaWJfaW5pdF9sb2Nr ZWQgbWFrZXMgbWUgd29uZGVyIGlmIHRoaXMgaXMgYSBzaW1pbGFyIGlzc3VlLiAgKElmDQogICAg aXQgaXMsIHRoZXJlJ3MgYSBsb2NraW5nIHByb2JsZW0gc29tZXdoZXJlLi4uKQ0KICAgIA0KICAg IENvbGluIFBlcmNpdmFsDQogICAgDQogICAgT24gMDEvMTIvMTggMDE6MjMsIEJyYWRsZXkgVC4g SHVnaGVzIHdyb3RlOg0KICAgID4gKENyb3NzLXBvc3RpbmcgdG8gLW5ldCwgYnV0IEknbSBub3Qg c3Vic2NyaWJlZCB0aGVyZS4uLiB5ZXQpDQogICAgPiANCiAgICA+IEkgd2FzIHdvbmRlcmluZyBp ZiBhbnlvbmUgZWxzZSBoYXMgZXhwZXJpZW5jZWQgdGhpcyBwYW5pYyB3aGVuIGRoY2xpZW50DQog ICAgdHJpZXMgdG8gZ2V0IGFuIElQIGFkZHJlc3MgYWZ0ZXIgYm9vdGluZyBhIHJlY2VudCAxMi4w LUNVUlJFTlQga2VybmVsPyBVc2luZw0KICAgIHRoZSAxMi4wLUNVUlJFTlQgQU1JIGZyb20gZWFy bHkgTm92ZW1iZXIgd29ya3MsIGJ1dCB3aXRoIHRoZSBsYXRlc3Qgc25hcHNob3QNCiAgICBvciB3 aGVuIHVzaW5nIGEga2VybmVsIGJ1aWx0IGZyb20gc291cmNlIEkgZ2V0IHRoaXM6DQogICAgPiAN CiAgICA+IGl4djA6IDxJbnRlbChSKSBQUk8vMTBHYkUgVmlydHVhbCBGdW5jdGlvbiBOZXR3b3Jr IERyaXZlcj4gbWVtIDB4ZjMwMDAwMDAtMHhmMzAwM2ZmZiwweGYzMDA0MDAwLTB4ZjMwMDdmZmYg YXQgZGV2aWNlIDMuMCBvbiBwY2kwDQogICAgPiBpeHYwOiB1c2luZyAyMDQ4IHR4IGRlc2NyaXB0 b3JzIGFuZCAyMDQ4IHJ4IGRlc2NyaXB0b3JzDQogICAgPiBpeHYwOiBtc2l4X2luaXQgcXNldHMg Y2FwcGVkIGF0IDENCiAgICA+IGl4djA6IHB4bSBjcHVzOiAyIHF1ZXVlIG1zZ3M6IDIgYWRtaW5j bnQ6IDENCiAgICA+IGl4djA6IHVzaW5nIDEgcnggcXVldWVzIDEgdHggcXVldWVzIA0KICAgID4g aXh2MDogVXNpbmcgTVNJWCBpbnRlcnJ1cHRzIHdpdGggMiB2ZWN0b3JzDQogICAgPiBpeHYwOiBh bGxvY2F0ZWQgZm9yIDEgcXVldWVzDQogICAgPiBpeHYwOiBhbGxvY2F0ZWQgZm9yIDEgcnggcXVl dWVzDQogICAgPiB0YXNrcWdyb3VwX2F0dGFjaF9jcHU6IHNldGFmZmluaXR5IGZhaWxlZDogMjIN CiAgICA+IHRhc2txZ3JvdXBfYXR0YWNoX2NwdTogc2V0YWZmaW5pdHkgZmFpbGVkOiAyMg0KICAg ID4gaXh2MDogRXRoZXJuZXQgYWRkcmVzczogMGE6YWQ6Njg6NzA6ZWE6NzgNCiAgICA+IGl4djA6 IG5ldG1hcCBxdWV1ZXMvc2xvdHM6IFRYIDEvMjA0OCwgUlggMS8yMDQ4DQogICAgPiAuLi4NCiAg ICA+IHBhbmljOiBBc3NlcnRpb24gaWZfZ2V0ZHJ2ZmxhZ3MoaWZwKSA9PSBpIGZhaWxlZCBhdCAv dXNyL3NyYy9zeXMvbmV0L2lmbGliLmM6MjE5OQ0KICAgID4gY3B1aWQgPSAzDQogICAgPiB0aW1l ID0gMTUxNTY3MzE3Ng0KICAgID4gS0RCOiBzdGFjayBiYWNrdHJhY2U6DQogICAgPiBkYl90cmFj ZV9zZWxmX3dyYXBwZXIoKSBhdCBkYl90cmFjZV9zZWxmX3dyYXBwZXIrMHgyYi9mcmFtZSAweGZm ZmZmZTAwOGE4ZWU2MDANCiAgICA+IHZwYW5pYygpIGF0IHZwYW5pYysweDE5Yy9mcmFtZSAweGZm ZmZmZTAwOGE4ZWU2ODANCiAgICA+IGthc3NlcnRfcGFuaWMoKSBhdCBrYXNzZXJ0X3BhbmljKzB4 MTI2L2ZyYW1lIDB4ZmZmZmZlMDA4YThlZTZmMA0KICAgID4gaWZsaWJfaW5pdF9sb2NrZWQoKSBh dCBpZmxpYl9pbml0X2xvY2tlZCsweDc1Zi9mcmFtZSAweGZmZmZmZTAwOGE4ZWU3NTANCiAgICA+ IGlmbGliX2lmX2lvY3RsKCkgYXQgaWZsaWJfaWZfaW9jdGwrMHg3NDQvZnJhbWUgMHhmZmZmZmUw MDhhOGVlN2IwDQogICAgPiBpZmlvY3RsKCkgYXQgaWZpb2N0bCsweDE5MDMvZnJhbWUgMHhmZmZm ZmUwMDhhOGVlODUwDQogICAgPiBrZXJuX2lvY3RsKCkgYXQga2Vybl9pb2N0bCsweDJjNC9mcmFt ZSAweGZmZmZmZTAwOGE4ZWU4YjANCiAgICA+IHN5c19pb2N0bCgpIGF0IHN5c19pb2N0bCsweDE1 Yy9mcmFtZSAweGZmZmZmZTAwOGE4ZWU5ODANCiAgICA+IGFtZDY0X3N5c2NhbGwoKSBhdCBhbWQ2 NF9zeXNjYWxsKzB4NzliL2ZyYW1lIDB4ZmZmZmZlMDA4YThlZWFiMA0KICAgID4gWGZhc3Rfc3lz Y2FsbCgpIGF0IFhmYXN0X3N5c2NhbGwrMHhmYi9mcmFtZSAweGZmZmZmZTAwOGE4ZWVhYjANCiAg ICA+IC0tLSBzeXNjYWxsICg1NCwgRnJlZUJTRCBFTEY2NCwgc3lzX2lvY3RsKSwgcmlwID0gMHg4 MDEwMGE0ZmEsIHJzcCA9IDB4N2ZmZmZmZmZlNDA4LCByYnAgPSAweDdmZmZmZmZmZTQ2MCAtLS0N CiAgICA+IA0KICAgID4gVGhhbmtzIGluIGFkdmFuY2UgOikNCiAgICANCiAgICAtLSANCiAgICBD b2xpbiBQZXJjaXZhbA0KICAgIFNlY3VyaXR5IE9mZmljZXIgRW1lcml0dXMsIEZyZWVCU0QgfCBU aGUgcG93ZXIgdG8gc2VydmUNCiAgICBGb3VuZGVyLCBUYXJzbmFwIHwgd3d3LnRhcnNuYXAuY29t IHwgT25saW5lIGJhY2t1cHMgZm9yIHRoZSB0cnVseSBwYXJhbm9pZA0KICAgIF9fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQogICAgZnJlZWJzZC1uZXRAZnJl ZWJzZC5vcmcgbWFpbGluZyBsaXN0DQogICAgaHR0cHM6Ly9saXN0cy5mcmVlYnNkLm9yZy9tYWls bWFuL2xpc3RpbmZvL2ZyZWVic2QtbmV0DQogICAgVG8gdW5zdWJzY3JpYmUsIHNlbmQgYW55IG1h aWwgdG8gImZyZWVic2QtbmV0LXVuc3Vic2NyaWJlQGZyZWVic2Qub3JnIg0KICAgIA0KDQo= From owner-freebsd-cloud@freebsd.org Fri Jan 12 22:19:44 2018 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37931E7FEB5 for ; Fri, 12 Jan 2018 22:19:44 +0000 (UTC) (envelope-from 01000160ec74f2d4-040abda0-23bb-4a11-a9ab-0b8be244dd4d-000000@amazonses.com) Received: from a8-176.smtp-out.amazonses.com (a8-176.smtp-out.amazonses.com [54.240.8.176]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EFF2C83D4D for ; Fri, 12 Jan 2018 22:19:43 +0000 (UTC) (envelope-from 01000160ec74f2d4-040abda0-23bb-4a11-a9ab-0b8be244dd4d-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ae7m2yrxjw65l2cqdpjxuucyrvy564tn; d=tarsnap.com; t=1515795576; h=Subject:To:References:Cc:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; bh=d2FriLlhqd5YfajjPU09n+miRpyovmpCRGMWKiyIhbA=; b=nY2Zf4EARp+zB29+sWCJ5Sa1RQ8qw7ZqQZmFOGF53LrZhUWL7UAuZ8kxQW8nHaBv pp3ZXYgflQJ2EvVtOxMbeYKI2fEtnXrdv+o1R9mT1r9xLslBzz3AFKVbKsNBKVqyaUK o3BkBfDcyIGaJvrjHZJhLkvuoWKt1H32N/PKkH/A= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=224i4yxa5dv7c2xz3womw6peuasteono; d=amazonses.com; t=1515795576; h=Subject:To:References:Cc:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:Feedback-ID; bh=d2FriLlhqd5YfajjPU09n+miRpyovmpCRGMWKiyIhbA=; b=MZrvrSqsWmdK3GFMV5ARmjmp/ZQQczQ8ALNF7EiOs+cj2PdLuAiQig2I9bQ6bMMm ZQJImUQDlFgGJowpEGza+zN3SG42aTh4p7BfDBNyyUoD8ig4UYog5axklhtCEBpsUQE 2/kvOY8BTx3NtDhuWSttvCovFdMrjZvK4V0bFu5k= Subject: Re: Panic with recent -CURRENT kernel in EC2 To: "Pieper, Jeffrey E" , "Bradley T. Hughes" , "freebsd-cloud@freebsd.org" References: <01000160e9f7ae62-92937548-d46c-4687-87c0-2205119d5132-000000@email.amazonses.com> <191060CB-F1F8-4E34-8191-99EB7B28597C@intel.com> Cc: "freebsd-net@freebsd.org" , Eric Joyner From: Colin Percival Message-ID: <01000160ec74f2d4-040abda0-23bb-4a11-a9ab-0b8be244dd4d-000000@email.amazonses.com> Date: Fri, 12 Jan 2018 22:19:35 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <191060CB-F1F8-4E34-8191-99EB7B28597C@intel.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-SES-Outgoing: 2018.01.12-54.240.8.176 Feedback-ID: 1.us-east-1.Lv9FVjaNvvR5llaqfLoOVbo2VxOELl7cjN0AOyXnPlk=:AmazonSES X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jan 2018 22:19:44 -0000 On 01/12/18 08:16, Pieper, Jeffrey E wrote: > Hrm… > >> taskqgroup_attach_cpu: setaffinity failed: 22 >> taskqgroup_attach_cpu: setaffinity failed: 22 > > Is this a fresh snapshot? That should have been fixed in > https://svnweb.freebsd.org/base?view=revision&revision=327013 . Can you > please file a bug on this? This looks like an iflib issue. I can confirm that this happens with the 12.0-CURRENT r327524 snasphot (on a c4.8xlarge EC2 instance, in case it matters). Colin Percival > On 1/12/18, 2:43 AM, "owner-freebsd-net@freebsd.org on behalf of Colin Percival" wrote: > > This sounds like it may be related to r327031, which make ixgbe (and thus ixv) > use iflib... maybe erj@ will have some ideas here? > > In case it helps, dhclient sets the MTU, which in the past has caused > problems due to drivers resetting -- the fact that sys_ioctl is ending > up in iflib_init_locked makes me wonder if this is a similar issue. (If > it is, there's a locking problem somewhere...) > > Colin Percival > > On 01/12/18 01:23, Bradley T. Hughes wrote: > > (Cross-posting to -net, but I'm not subscribed there... yet) > > > > I was wondering if anyone else has experienced this panic when dhclient > tries to get an IP address after booting a recent 12.0-CURRENT kernel? Using > the 12.0-CURRENT AMI from early November works, but with the latest snapshot > or when using a kernel built from source I get this: > > > > ixv0: mem 0xf3000000-0xf3003fff,0xf3004000-0xf3007fff at device 3.0 on pci0 > > ixv0: using 2048 tx descriptors and 2048 rx descriptors > > ixv0: msix_init qsets capped at 1 > > ixv0: pxm cpus: 2 queue msgs: 2 admincnt: 1 > > ixv0: using 1 rx queues 1 tx queues > > ixv0: Using MSIX interrupts with 2 vectors > > ixv0: allocated for 1 queues > > ixv0: allocated for 1 rx queues > > taskqgroup_attach_cpu: setaffinity failed: 22 > > taskqgroup_attach_cpu: setaffinity failed: 22 > > ixv0: Ethernet address: 0a:ad:68:70:ea:78 > > ixv0: netmap queues/slots: TX 1/2048, RX 1/2048 > > ... > > panic: Assertion if_getdrvflags(ifp) == i failed at /usr/src/sys/net/iflib.c:2199 > > cpuid = 3 > > time = 1515673176 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe008a8ee600 > > vpanic() at vpanic+0x19c/frame 0xfffffe008a8ee680 > > kassert_panic() at kassert_panic+0x126/frame 0xfffffe008a8ee6f0 > > iflib_init_locked() at iflib_init_locked+0x75f/frame 0xfffffe008a8ee750 > > iflib_if_ioctl() at iflib_if_ioctl+0x744/frame 0xfffffe008a8ee7b0 > > ifioctl() at ifioctl+0x1903/frame 0xfffffe008a8ee850 > > kern_ioctl() at kern_ioctl+0x2c4/frame 0xfffffe008a8ee8b0 > > sys_ioctl() at sys_ioctl+0x15c/frame 0xfffffe008a8ee980 > > amd64_syscall() at amd64_syscall+0x79b/frame 0xfffffe008a8eeab0 > > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe008a8eeab0 > > --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x80100a4fa, rsp = 0x7fffffffe408, rbp = 0x7fffffffe460 --- > > > > Thanks in advance :) -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid