From owner-freebsd-current@freebsd.org Sun Apr 22 00:58:22 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60BB7FB2B5C for ; Sun, 22 Apr 2018 00:58:22 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) Received: from sonic315-21.consmr.mail.ne1.yahoo.com (sonic315-21.consmr.mail.ne1.yahoo.com [66.163.190.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D5BF28677C for ; Sun, 22 Apr 2018 00:58:21 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) X-YMail-OSG: TDs_ahAVM1lqMZGuJGOXUPTBt9qrYzE9DSXFnyW0ABcl_mSv2OUI9qP3whscWQL kgTnyxq6PSn7JPc4sZ8p_mcDUDx4LZl3ubaZUCWHKu4ipxCBK4EuO4v9oJ_MJ9FFA_yrjMWOI5WC qhZD5ETnjwaKrhkY48MP.treqkLJhigZvza4wMpuKWtqbqyQc0pJo8aO_X_fvT1iUwEcNPFsHzFZ 0_269R0HUCg5qJ8WmMfbCKuEZmmGoq6Org1hozzP7JuP0502pvJRA9p8zqsvALUW_1K5lLCOaSf2 r5rB_5wvLzEuGffyVwFt_yakBfHk6Pl6vlDC.VlnHqxrJR4N321rX4Alj0fn3rFc8na71ZLXA3pa SSEgaIKcmCevLMqRaLWaI_BJSSjV8t0Gg1hZuR.ooAHKCLi_nI9JO1.10dEFOda1j1vNHc3CMXA8 U8H0lObvXGfNFNWe3Yjwia9yf.ibAXl3tpE.mozm1bStbOozMpL_bKv9TzSH2ffb0OI8S6cj_21d jYttMj5YKhFZyzvzodVToBYBY1Hvty4ITjcDmLgKA2YkJHe1HUJLrvNdS_RQ9Rq5cEg.VWavVYsr O3zQuRtQ- Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.ne1.yahoo.com with HTTP; Sun, 22 Apr 2018 00:58:15 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp410.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID dfaeed34c522db5cb77cf65fb3b5cfa1; Sun, 22 Apr 2018 00:58:12 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.3 \(3445.6.18\)) Subject: Failed buildworld buildkernel: /usr/obj/. . ./arm.armv7/tmp/usr/bin/ld: cannot find -lgcc_s for all_subdir_lib/libdl (a build race?) Message-Id: Date: Sat, 21 Apr 2018 17:58:11 -0700 To: freebsd-arm@freebsd.org, freebsd-toolchain@freebsd.org, FreeBSD Current X-Mailer: Apple Mail (2.3445.6.18) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 00:58:22 -0000 I tried to amd64 -> armv7 cross build head -r332861 and got an error = about -lgcc_s not being found. I backed off to -r332858 for other reasons = (powerpc* related). Retrying the armv7 build then worked. I had first upgraded the amd64 context the first time and had backed off = amd64 first the second time. (The builds are explicit about -mcpu=3Dcortext-a7 = .) (When I think about it, I tend to check https://ci.freebsd.org to pick a version likely to build for the variations that I play with. But armv7 is not covered by https://ci.freebsd.org . armv7 variations are missing = from the report for -r332796's snapshots. But I'm not aware of anything = around that indicates why variations ended up missing for snapshots.) Below is the error report that shows the message and the _ERROR_CMD that was involved. I have no direct clue if it is because of a race relative to gcc_s availability or not. The context involved WITH_META_MODE=3Dyes = and -j28 . I do not see anything in -r332859 to -r332861 that would suggest = anything but a race as a relevant difference, where a re-try found gcc_s already available. --- all_subdir_lib/libdl --- /usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/tmp/usr/bin/ld: cannot = find -lgcc_s --- all_subdir_lib/libbsm --- Building = /usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/lib/libbsm/bsm_notify.o --- sbin/ipf/libipf__L --- Building = /usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/sbin/ipf/libipf/printiphd= r.o --- lib__L --- --- all_subdir_lib/libdl --- cc: error: linker command failed with exit code 1 (use -v to see = invocation) --- all_subdir_lib/libdevinfo --- --- libdevinfo.a --- building static devinfo library --- kerberos5/lib__L --- Building = /usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/kerberos5/lib/libkadm5srv= /set_keys.o --- lib__L --- --- all_subdir_lib/libdl --- *** [libdl.so.1.full] Error code 1 make[5]: stopped in /usr/src/lib/libdl .ERROR_TARGET=3D'libdl.so.1.full' = .ERROR_META_FILE=3D'/usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/lib/l= ibdl/libdl.so.1.full.meta' .MAKE.LEVEL=3D'5' MAKEFILE=3D'' .MAKE.MODE=3D'meta missing-filemon=3Dyes missing-meta=3Dyes silent=3Dyes = verbose' _ERROR_CMD=3D'@echo building shared library libdl.so.1; @rm -f = libdl.so.1 libdl.so; cc -mcpu=3Dcortex-a7 -target = armv7-gnueabihf-freebsd12.0 = --sysroot=3D/usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/tmp = -B/usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/tmp/usr/bin = -Wl,-F,libc.so.7 -Wl,--version-script=3DVersion.map -shared -Wl,-x = -Wl,--fatal-warnings -Wl,--warn-shared-textrel -o libdl.so.1.full = -Wl,-soname,libdl.so.1 `NM=3D'nm' NMFLAGS=3D'' lorder dlfcn.pico | = tsort -q` ;' .CURDIR=3D'/usr/src/lib/libdl' .MAKE=3D'make' .OBJDIR=3D'/usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/lib/libdl' .TARGETS=3D'all' DESTDIR=3D'/usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/tmp' LD_LIBRARY_PATH=3D'' MACHINE=3D'arm' MACHINE_ARCH=3D'armv7' MAKEOBJDIRPREFIX=3D'' MAKESYSPATH=3D'/usr/src/share/mk' MAKE_VERSION=3D'20180222' = PATH=3D'/usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/tmp/legacy/usr/sb= in:/usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/tmp/legacy/usr/bin:/us= r/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/tmp/legacy/bin:/usr/obj/armv= 7_clang/arm.armv7/usr/src/arm.armv7/tmp/usr/sbin:/usr/obj/armv7_clang/arm.= armv7/usr/src/arm.armv7/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin' SRCTOP=3D'/usr/src' OBJTOP=3D'/usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7' .MAKE.MAKEFILES=3D'/usr/src/share/mk/sys.mk = /usr/src/share/mk/local.sys.env.mk /usr/src/share/mk/src.sys.env.mk = /root/src.configs/src.conf.armv7-clang-bootstrap.amd64-host = /usr/src/share/mk/bsd.mkopt.mk /usr/src/share/mk/src.sys.obj.mk = /usr/src/share/mk/auto.obj.mk /usr/src/share/mk/bsd.suffixes.mk = /root/src.configs/make.conf /usr/src/share/mk/local.sys.mk = /usr/src/share/mk/src.sys.mk /dev/null /usr/src/lib/libdl/Makefile = /usr/src/share/mk/bsd.lib.mk /usr/src/share/mk/bsd.init.mk = /usr/src/share/mk/bsd.opts.mk /usr/src/share/mk/bsd.cpu.mk = /usr/src/share/mk/local.init.mk /usr/src/share/mk/src.init.mk = /usr/src/lib/libdl/../Makefile.inc /usr/src/share/mk/bsd.own.mk = /usr/src/share/mk/bsd.compiler.mk /usr/src/share/mk/bsd.linker.mk = /usr/src/share/mk/bsd.libnames.mk /usr/src/share/mk/src.libnames.mk = /usr/src/share/mk/src.opts.mk /usr/src/share/mk/bsd.symver.mk = /usr/src/share/mk/bsd.nls.mk /usr/src/share/mk/bsd.files.mk = /usr/src/share/mk/bsd.incs.mk /usr/src/share/mk/bsd.confs.mk = /usr/src/share/mk/bsd.links.mk /usr/src/share/mk/bsd.dep.mk = /usr/src/share/mk/bsd.clang-analyze.mk /usr/src/share/mk/bsd.obj.mk = /usr/src/share/mk/bsd.subdir.mk /usr/src/share/mk/bsd.sys.mk' .PATH=3D'. /usr/src/lib/libdl /usr/src/lib/libc/gen' 1 error =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) From owner-freebsd-current@freebsd.org Sun Apr 22 01:37:40 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B78B7FB5799 for ; Sun, 22 Apr 2018 01:37:40 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 291BD6F63E for ; Sun, 22 Apr 2018 01:37:39 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w3M1b9BM077992; Sat, 21 Apr 2018 18:37:09 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w3M1b9V8077991; Sat, 21 Apr 2018 18:37:09 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201804220137.w3M1b9V8077991@pdx.rh.CN85.dnsmgr.net> Subject: Re: SCHED_ULE makes 256Mbyte i386 unusable In-Reply-To: <20180421201128.GO6887@kib.kiev.ua> To: Konstantin Belousov Date: Sat, 21 Apr 2018 18:37:09 -0700 (PDT) CC: Rick Macklem , "freebsd-current@freebsd.org" , George Mitchell , Peter X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 01:37:41 -0000 > On Sat, Apr 21, 2018 at 07:21:58PM +0000, Rick Macklem wrote: > > I decided to start a new thread on current related to SCHED_ULE, since I see > > more than just performance degradation and on a recent current kernel. > > (I cc'd a couple of the people discussing performance problems in freebsd-stable > > recently under a subject line of "Re: kern.sched.quantum: Creepy, sadistic scheduler". > > > > When testing a pNFS server on a single core i386 with 256Mbytes using a Dec. 2017 > > current/head kernel, I would see about a 30% performance degradation (elapsed > > run time for a kernel build over NFSv4.1) when the server kernel was built with > > options SCHED_ULE > > instead of > > options SCHED_4BSD > > > > Now, with a kernel from a couple of days ago, the > > options SCHED_ULE > > kernel becomes unusable shortly after starting testing. > > I have seen two variants of this: > > - Became essentially hung. All I could do was ping the machine from the network. > > - Reported "vm_thread_new: kstack allocation failed > > and then any attempt to do anything gets "No more processes". > This is strange. It usually means that you get KVA either exhausted or > severly fragmented. > > Enter ddb, it should be operational since pings are replied. Try to see > where the threads are stuck. > > > with the only difference being a kernel built with > > options SCHED_4BSD > > everything works and performs the same as the Dec 2017 kernel. > > > > I can try rolling back through the revisions, but it would be nice if someone > > could suggest where to start, because it takes a couple of hours to build a > > kernel on this system. > > > > So, something has made things worse for a head/current kernel this winter, rick > > There are at least two potentially relevant changes. > > First is r326758 Dec 11 which bumped KSTACK_PAGES on i386 to 4. My experience with this bump and trying to run a GENERIC -current (meaning it has INVARIANTS and WITNESS) is that you have a very unstable situation until you get above 1G of memory. And NFS is a major kstack consumer. > Second is r332489 Apr 13, which introduced 4/4G KVA/UVA split. > > Consequences of the first one are obvious, it is much harder to find > the place to map the stack. Second change, on the other hand, provides > almost full 4G for KVA and should have mostly compensate for the negative > effects of the first. > > And, I cannot see how changing the scheduler would fix or even affect that > behaviour. > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-current@freebsd.org Sun Apr 22 03:00:38 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4CC37FB89A0 for ; Sun, 22 Apr 2018 03:00:38 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670052.outbound.protection.outlook.com [40.107.67.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D380980A01; Sun, 22 Apr 2018 03:00:37 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM (52.132.66.153) by YQBPR0101MB1106.CANPRD01.PROD.OUTLOOK.COM (52.132.67.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.696.12; Sun, 22 Apr 2018 03:00:36 +0000 Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::893c:efc2:d71f:945a]) by YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::893c:efc2:d71f:945a%13]) with mapi id 15.20.0696.015; Sun, 22 Apr 2018 03:00:35 +0000 From: Rick Macklem To: Tijl Coosemans , "freebsd-current@freebsd.org" , Cy Schubert Subject: Re: i386 hangs during halt "vnodes remaining... 0 time out" Thread-Topic: i386 hangs during halt "vnodes remaining... 0 time out" Thread-Index: AQHT2eVWHLCjuZtjWUejob9lr9/K/6QMF0jg Date: Sun, 22 Apr 2018 03:00:35 +0000 Message-ID: References: Message from Cy Schubert of "Sat, 21 Apr 2018 15:27:51 -0700." <201804212227.w3LMRp5W002975@slippy.cwsent.com>, <201804220255.w3M2t9nP003967@slippy.cwsent.com> In-Reply-To: <201804220255.w3M2t9nP003967@slippy.cwsent.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: FreeBSD.org; dkim=none (message not signed) header.d=none;FreeBSD.org; dmarc=none action=none header.from=uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YQBPR0101MB1106; 7:d1ZtfG5bTcY2/owZuQ7rZEkcTQQzWJNoNRRzA24b+hDhuCpwhaMr7AxzewEOpGGAwVdqLpxnnTO6YCfvhAc+8LRsqGwLbDW2RsZGgYVi1Q+ZLZEfpizqWFncQ/K2nkCkJWX8A8ycTFBrRn0EzRUuSlF3yRyZrGdSTsc8metn6CvTAReANqHA9411s+tmA8LKcUm7ZA6SPXjXbMTqljCRlolWivoGJ9q2SVV+Nhp/SNB4jHWWmVLESn7XNRE0/rWm x-ms-exchange-antispam-srfa-diagnostics: SOS; x-microsoft-antispam: UriScan:(80341031608730); BCL:0; PCL:0; RULEID:(7020095)(4652020)(8989080)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(8990040)(2017052603328)(7153060)(7193020); SRVR:YQBPR0101MB1106; x-ms-traffictypediagnostic: YQBPR0101MB1106: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(80341031608730); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(10201501046)(3002001)(3231232)(944501410)(52105095)(93006095)(93001095)(6041310)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123558120)(20161123562045)(20161123560045)(6072148)(201708071742011); SRVR:YQBPR0101MB1106; BCL:0; PCL:0; RULEID:; SRVR:YQBPR0101MB1106; x-forefront-prvs: 0650714AAA x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(376002)(39380400002)(346002)(396003)(39860400002)(366004)(26005)(55016002)(6246003)(11346002)(110136005)(53936002)(8936002)(8676002)(33656002)(316002)(81166006)(786003)(9686003)(86362001)(74316002)(5890100001)(2501003)(74482002)(186003)(5250100002)(966005)(2900100001)(25786009)(6306002)(478600001)(446003)(229853002)(102836004)(76176011)(6436002)(7696005)(5660300001)(3280700002)(2906002)(6506007)(3660700001)(476003)(305945005); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB1106; H:YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; MLV:sfv; x-microsoft-antispam-message-info: g+7to/vGU6my6HIx5PSwlHekI4rexMJrfIf2oUu28PwYf6uKFyuWlfWENAF5bdgP4I/Zvs6r5vEbqS0bys7mBOGXubvLhQfBTCtrTPCPesrvibtFgHA40gcrbmLCF4Wz9xo2RGVoWmL57ZsyDNJmBr5avkh3ulDWOUAiucDo6+mcR5p63vjzC8o3b1lGGJT8 spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: bd0c1b51-6be4-4e08-e55c-08d5a7fd3886 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: bd0c1b51-6be4-4e08-e55c-08d5a7fd3886 X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Apr 2018 03:00:35.8941 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1106 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 03:00:38 -0000 Cy Schubert wrote: >In message <201804212227.w3LMRp5W002975@slippy.cwsent.com>, Cy Schubert writes: >> In message <20180421234934.10d7dfab@kalimero.tijl.coosemans.org>, Tijl >> Cooseman >> s writes: >> > --MP_/TDsO+CDIra7UXGs=3DvVO3NTB >> > Content-Type: text/plain; charset=3DUS-ASCII >> > Content-Transfer-Encoding: 7bit >> > Content-Disposition: inline >> > >> > On Sat, 21 Apr 2018 21:09:09 +0000 Rick Macklem = wrot >> e: >> > > With a recent head/current kernel (doesn't happen when running a Dec= . >> > > 2017 one), when I do a halt, it gets as far as: >> > > >> > > vnodes remaining... 0 time out >> > > >> > > and that's it (the time out appears several seconds after the first = "0"). >> > > With a Dec. 2017 kernel there would be several "0"s printed. >> > > It appears that it is stuck in the first iteration of the sched_sync= () >> > > loop after it is no longer in SYNCER_RUNNING state. >> > > >> > > Any ideas? rick >> > >> > See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D227404 >> > I have a patch (attached) but haven't been able to test it yet. >> >> I've noticed this as well on my old Penium-M laptop (updated about >> twice a year). I'll try your patch this weekend or early next week. > >Works perfectly. Patch seems to work for my i386 as well. Thanks, rick From owner-freebsd-current@freebsd.org Sun Apr 22 03:21:19 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 46857FB9558 for ; Sun, 22 Apr 2018 03:21:19 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B3095844BF; Sun, 22 Apr 2018 03:21:18 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from spqr.komquats.com ([70.67.125.17]) by shaw.ca with ESMTPA id A5YxfI0OIvB5RA5YzfALrV; Sat, 21 Apr 2018 21:21:17 -0600 X-Authority-Analysis: v=2.3 cv=PvS9kTE3 c=1 sm=1 tr=0 a=VFtTW3WuZNDh6VkGe7fA3g==:117 a=VFtTW3WuZNDh6VkGe7fA3g==:17 a=kj9zAlcOel0A:10 a=Kd1tUaAdevIA:10 a=UqCG9HQmAAAA:8 a=VxmjJ2MpAAAA:8 a=YqRfAJJkAAAA:8 a=hF2rLc1pAAAA:8 a=6I5d2MoRAAAA:8 a=YxBL1-UpAAAA:8 a=sBeMAxy3m3l4eF98PuwA:9 a=CjuIK1q_8ugA:10 a=VkU0TBxsNEYA:10 a=WthIJ1qJ88kA:10 a=7gXAzLPJhVmCkEl4_tsf:22 a=EZxYlxVqKW1-bC4Rwo9g:22 a=O9OM7dhJW_8Hj9EqqvKN:22 a=IjZwj45LgO3ly-622nXo:22 a=Ia-lj3WSrqcvXOmTRaiG:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTPS id AA285DA4; Sat, 21 Apr 2018 20:21:15 -0700 (PDT) Received: from slippy.cwsent.com (localhost [127.0.0.1]) by slippy.cwsent.com (8.15.2/8.15.2) with ESMTP id w3M3LFk8004189; Sat, 21 Apr 2018 20:21:15 -0700 (PDT) (envelope-from Cy.Schubert@cschubert.com) Received: from slippy (cy@localhost) by slippy.cwsent.com (8.15.2/8.15.2/Submit) with ESMTP id w3M3LFFj004186; Sat, 21 Apr 2018 20:21:15 -0700 (PDT) (envelope-from Cy.Schubert@cschubert.com) Message-Id: <201804220321.w3M3LFFj004186@slippy.cwsent.com> X-Authentication-Warning: slippy.cwsent.com: cy owned process doing -bs X-Mailer: exmh version 2.8.0 04/21/2012 with nmh-1.7.1 Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: Rick Macklem cc: Tijl Coosemans , "freebsd-current@freebsd.org" , Cy Schubert Subject: Re: i386 hangs during halt "vnodes remaining... 0 time out" In-Reply-To: Message from Rick Macklem of "Sun, 22 Apr 2018 03:00:35 -0000." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 21 Apr 2018 20:21:15 -0700 X-CMAE-Envelope: MS4wfPGc2okl5FjLSoaVR6iK6qksxdxj027hw1oiDtxm5oz9nku7dVR5Z/oWfje4D9xE/mGfkATKEjxNYqrFdX9OoCEpmt8n858VpBTONuElaTWzzyVbrRxH SFebKIrblGgkTXKojTG6ywjXNAbBa4v+8FXx8nbt1/ZHH1AvnnJBU2nSQpbv8A8NNzuJ/aw6NfpoB3jFrqv22+681vrBCkbZSN5sXoJhbhAfySCFdCXYEz7/ 9vvcJqT+ZvNcd00LI+nigA== X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 03:21:19 -0000 In message , Rick Macklem writes: > Cy Schubert wrote: > >In message <201804212227.w3LMRp5W002975@slippy.cwsent.com>, Cy Schubert > writes: > >> In message <20180421234934.10d7dfab@kalimero.tijl.coosemans.org>, Tijl > >> Cooseman > >> s writes: > >> > --MP_/TDsO+CDIra7UXGs=vVO3NTB > >> > Content-Type: text/plain; charset=US-ASCII > >> > Content-Transfer-Encoding: 7bit > >> > Content-Disposition: inline > >> > > >> > On Sat, 21 Apr 2018 21:09:09 +0000 Rick Macklem w > rot > >> e: > >> > > With a recent head/current kernel (doesn't happen when running a Dec. > >> > > 2017 one), when I do a halt, it gets as far as: > >> > > > >> > > vnodes remaining... 0 time out > >> > > > >> > > and that's it (the time out appears several seconds after the first "0 > "). > >> > > With a Dec. 2017 kernel there would be several "0"s printed. > >> > > It appears that it is stuck in the first iteration of the sched_sync() > >> > > loop after it is no longer in SYNCER_RUNNING state. > >> > > > >> > > Any ideas? rick > >> > > >> > See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227404 > >> > I have a patch (attached) but haven't been able to test it yet. > >> > >> I've noticed this as well on my old Penium-M laptop (updated about > >> twice a year). I'll try your patch this weekend or early next week. > > > >Works perfectly. > Patch seems to work for my i386 as well. > > Thanks, rick Yes, thank you. -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. From owner-freebsd-current@freebsd.org Sun Apr 22 02:55:15 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6BEDFB85DB for ; Sun, 22 Apr 2018 02:55:14 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 655207E1DB; Sun, 22 Apr 2018 02:55:14 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from spqr.komquats.com ([70.67.125.17]) by shaw.ca with ESMTPA id A59hfHv9lvB5RA59jfAIyN; Sat, 21 Apr 2018 20:55:12 -0600 X-Authority-Analysis: v=2.3 cv=PvS9kTE3 c=1 sm=1 tr=0 a=VFtTW3WuZNDh6VkGe7fA3g==:117 a=VFtTW3WuZNDh6VkGe7fA3g==:17 a=kj9zAlcOel0A:10 a=Kd1tUaAdevIA:10 a=VxmjJ2MpAAAA:8 a=YqRfAJJkAAAA:8 a=hF2rLc1pAAAA:8 a=6I5d2MoRAAAA:8 a=YxBL1-UpAAAA:8 a=IPRFU6G7GWTDQhDMTzQA:9 a=CjuIK1q_8ugA:10 a=WthIJ1qJ88kA:10 a=7gXAzLPJhVmCkEl4_tsf:22 a=EZxYlxVqKW1-bC4Rwo9g:22 a=O9OM7dhJW_8Hj9EqqvKN:22 a=IjZwj45LgO3ly-622nXo:22 a=Ia-lj3WSrqcvXOmTRaiG:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTPS id 9B8F8D22; Sat, 21 Apr 2018 19:55:09 -0700 (PDT) Received: from slippy.cwsent.com (localhost [127.0.0.1]) by slippy.cwsent.com (8.15.2/8.15.2) with ESMTP id w3M2t9dW003971; Sat, 21 Apr 2018 19:55:09 -0700 (PDT) (envelope-from Cy.Schubert@cschubert.com) Received: from slippy (cy@localhost) by slippy.cwsent.com (8.15.2/8.15.2/Submit) with ESMTP id w3M2t9nP003967; Sat, 21 Apr 2018 19:55:09 -0700 (PDT) (envelope-from Cy.Schubert@cschubert.com) Message-Id: <201804220255.w3M2t9nP003967@slippy.cwsent.com> X-Authentication-Warning: slippy.cwsent.com: cy owned process doing -bs X-Mailer: exmh version 2.8.0 04/21/2012 with nmh-1.7.1 Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: Tijl Coosemans , Rick Macklem , "freebsd-current@freebsd.org" Subject: Re: i386 hangs during halt "vnodes remaining... 0 time out" In-Reply-To: Message from Cy Schubert of "Sat, 21 Apr 2018 15:27:51 -0700." <201804212227.w3LMRp5W002975@slippy.cwsent.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 21 Apr 2018 19:55:09 -0700 X-CMAE-Envelope: MS4wfD5YdmXUSaxZRAUaq7R5UPBkUyDc/Y4WzY+bSXkYfIoBobTBP6cpe0Jof5QYlYfZs3DVEpcq/D6cgNltWv3lNzx5IGzmqXzLAAYBcslsT/y5GAA2Bizz UwM4DTP8USpeTRzV0dhm1NZ1W75Q6GyR4tHVyURnhyoUUM+9LKIVEv2Z3pOSv7sCSu0MD3f/jxldfROiSt2H9MegwVIvHJRN9p1RH5GZDxAcEdm0HYxN0STx tyKMmLRf/4F02n7GbHMACQ== X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 02:55:15 -0000 In message <201804212227.w3LMRp5W002975@slippy.cwsent.com>, Cy Schubert writes: > In message <20180421234934.10d7dfab@kalimero.tijl.coosemans.org>, Tijl > Cooseman > s writes: > > --MP_/TDsO+CDIra7UXGs=vVO3NTB > > Content-Type: text/plain; charset=US-ASCII > > Content-Transfer-Encoding: 7bit > > Content-Disposition: inline > > > > On Sat, 21 Apr 2018 21:09:09 +0000 Rick Macklem wrot > e: > > > With a recent head/current kernel (doesn't happen when running a Dec. > > > 2017 one), when I do a halt, it gets as far as: > > > > > > vnodes remaining... 0 time out > > > > > > and that's it (the time out appears several seconds after the first "0"). > > > With a Dec. 2017 kernel there would be several "0"s printed. > > > It appears that it is stuck in the first iteration of the sched_sync() > > > loop after it is no longer in SYNCER_RUNNING state. > > > > > > Any ideas? rick > > > > See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227404 > > I have a patch (attached) but haven't been able to test it yet. > > I've noticed this as well on my old Penium-M laptop (updated about > twice a year). I'll try your patch this weekend or early next week. Works perfectly. -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. From owner-freebsd-current@freebsd.org Sun Apr 22 10:42:40 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1BC09FB4DA5; Sun, 22 Apr 2018 10:42:40 +0000 (UTC) (envelope-from oshogbo.vx@gmail.com) Received: from mail-pg0-x22b.google.com (mail-pg0-x22b.google.com [IPv6:2607:f8b0:400e:c05::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 97C136C41F; Sun, 22 Apr 2018 10:42:39 +0000 (UTC) (envelope-from oshogbo.vx@gmail.com) Received: by mail-pg0-x22b.google.com with SMTP id i194so6388903pgd.0; Sun, 22 Apr 2018 03:42:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:from:date:message-id:subject:to; bh=DZsUckeKIQmy8lCoC50LhYw9f9yA4BmIIDjqha9KH10=; b=XPrE7zAd45F6vrhdN6xmkWsDAagdovBrGD5zWXGIZ4J/cyx4md5oP+hdYhUFl5fYYL UvHHMrsiYY6AmwsNhAzNGuZ9kQ/r5fUJT0LlcZjQMw2fyC5C+0+cRp97WyIlMxSeq/En Vma5yWhJf1BVvf41Tc7Dag/Equy0q8kORZ2Rw5icPnsppZ886Zjj7gXFChcf8hAzvDxH KmnBF9hUmOEUyFFEOfpOV6UjJyfV42j3iaUcmNrm80FYieMJY7re1yCF/ln7hYyhvfpU er5OzvzFUl1WuSJWiQyFFM4m59vm5mklONxeTP09X4jKj6qht1XtigNGp/iCmfbCOecl pwwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:from:date:message-id:subject :to; bh=DZsUckeKIQmy8lCoC50LhYw9f9yA4BmIIDjqha9KH10=; b=af2uDLOhSKgnS9cAcik5B8bwsPByandI68dvtF7bJWToHDOqcNqvY7V//ysbCBcv+G GSoYLzfTZ1TpWHGsPZXN2XdfEwhrXfTOr9O3xs3eWCWkBqNeHd50JhpzIT//8CAfuAq0 K2yWos5Uz6sSEWbJKrVue82AG830p/dJSUWZ+aJ0G7dIjMyUFs44wDPimLNl9V7K6rqa L2NmQqCsTM6w0mdij5wWY2b1t4vCacNwd6ALvJzIBUW+5atxXRBmoFl5Mhc6/6v21FsZ lB7921U7yECFKx3EIHpQJDJryQ8n/uMxsEb1aCM8uWrFjyNq2/WFMAA6Lwi71Rn1qASf z84A== X-Gm-Message-State: ALQs6tAEenxXSFzksqnyEq0kz2xa+aUQXtK17yluMBWdO6ZKQhtIH89N IEY/Wi2pfqoajzIgKyNfvqtrIeTjamSzSRXw8lINIRwi X-Google-Smtp-Source: AIpwx49m4sUMrt4h0ZUo/2oHMnLhtaOsPB6s6tu32IFdVNDulIU/PGGFJzAItRCBZbwEQD8fCn8hw3PFzKXS0aLhd0s= X-Received: by 2002:a17:902:70c7:: with SMTP id l7-v6mr16424222plt.165.1524393758466; Sun, 22 Apr 2018 03:42:38 -0700 (PDT) MIME-Version: 1.0 Sender: oshogbo.vx@gmail.com Received: by 10.100.145.137 with HTTP; Sun, 22 Apr 2018 03:42:37 -0700 (PDT) From: Mariusz Zaborski Date: Sun, 22 Apr 2018 12:42:37 +0200 X-Google-Sender-Auth: iy3WP6gJibRLTv8QLKKL-mwHekg Message-ID: Subject: Nvidia issue with CURRENT To: FreeBSD Current , freebsd-desktop@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 10:42:40 -0000 Hello, I upgraded my FreeBSD to CURRENT and nvidia-drvier-390.48. But it's stop working. I tried also nvidia-driver-390.25 without luck as well. I have loaded nvidia-modeset.ko . While I'm rebooting my machine its also core dumping: https://people.freebsd.org/~oshogbo/nvidia-mail.png . I'm attaching also Xorg log. Is this a known issue? Thanks, Mariusz From owner-freebsd-current@freebsd.org Sun Apr 22 12:03:14 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D63A8FBA8D7 for ; Sun, 22 Apr 2018 12:03:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 30DC47EE49 for ; Sun, 22 Apr 2018 12:03:12 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w3MC2gC3080461 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 22 Apr 2018 15:02:45 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w3MC2gC3080461 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w3MC2fOU080460; Sun, 22 Apr 2018 15:02:41 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 22 Apr 2018 15:02:41 +0300 From: Konstantin Belousov To: Rick Macklem Cc: "freebsd-current@freebsd.org" , George Mitchell , Peter Subject: Re: SCHED_ULE makes 256Mbyte i386 unusable Message-ID: <20180422120241.GR6887@kib.kiev.ua> References: <20180421201128.GO6887@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.5 (2018-04-13) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 12:03:14 -0000 On Sat, Apr 21, 2018 at 11:30:55PM +0000, Rick Macklem wrote: > Konstantin Belousov wrote: > >On Sat, Apr 21, 2018 at 07:21:58PM +0000, Rick Macklem wrote: > >> I decided to start a new thread on current related to SCHED_ULE, since I see > >> more than just performance degradation and on a recent current kernel. > >> (I cc'd a couple of the people discussing performance problems in freebsd-stable > >> recently under a subject line of "Re: kern.sched.quantum: Creepy, sadistic scheduler". > >> > >> When testing a pNFS server on a single core i386 with 256Mbytes using a Dec. 2017 > >> current/head kernel, I would see about a 30% performance degradation (elapsed > >> run time for a kernel build over NFSv4.1) when the server kernel was built with > >> options SCHED_ULE > >> instead of > >> options SCHED_4BSD > >> > >> Now, with a kernel from a couple of days ago, the > >> options SCHED_ULE > >> kernel becomes unusable shortly after starting testing. > >> I have seen two variants of this: > >> - Became essentially hung. All I could do was ping the machine from the network. > >> - Reported "vm_thread_new: kstack allocation failed > >> and then any attempt to do anything gets "No more processes". > >This is strange. It usually means that you get KVA either exhausted or > >severly fragmented. > Yes. I reduced the number of nfsd threads from 256->32 and the SCHED_ULE > kernel is working ok now. I haven't done enough to compare performance yet. > Maybe I'll post again when I have some numbers. > > >Enter ddb, it should be operational since pings are replied. Try to see > >where the threads are stuck. > I didn't do this, since reducing the number of kernel threads seems to have fixed > the problem. For the pNFS server, the nfsd threads will spawn additional kernel > threads to do proxies to the mirrored DS servers. > > >> with the only difference being a kernel built with > >> options SCHED_4BSD > >> everything works and performs the same as the Dec 2017 kernel. > >> > >> I can try rolling back through the revisions, but it would be nice if someone > >> could suggest where to start, because it takes a couple of hours to build a > >> kernel on this system. > >> > >> So, something has made things worse for a head/current kernel this winter, rick > > > >There are at least two potentially relevant changes. > > > >First is r326758 Dec 11 which bumped KSTACK_PAGES on i386 to 4. > I've been running this machine with KSTACK_PAGES=4 for some time, so no change. > > >Second is r332489 Apr 13, which introduced 4/4G KVA/UVA split. > Could this change have resulted in the system being able to allocate fewer > kernel threads/stacks for some reason? Well, it could, as anything can be buggy. But the intent of the change was to give 4G KVA, and it did. > > >Consequences of the first one are obvious, it is much harder to find > >the place to map the stack. Second change, on the other hand, provides > >almost full 4G for KVA and should have mostly compensate for the negative > >effects of the first. > > > >And, I cannot see how changing the scheduler would fix or even affect that > >behaviour. > My hunch is that the system was running near its limit for kernel threads/stacks. > Then, somehow, the timing SCHED_ULE caused resulted in the nfsd trying to get > to a higher peak number of threads and hit the limit. > SCHED_4BSD happened to result in timing such that it stayed just below the > limit and worked. > I can think of a couple of things that might affect this: > 1 - If SCHED_ULE doesn't do the termination of kernel threads as quickly, then > they wouldn't terminate and release their resources before more new ones > are spawned. Scheduler has nothing to do with the threads termination. It might select running threads in a way that causes the undesired pattern to appear which might create some amount of backlog for termination, but I doubt it. > 2 - If SCHED_ULE handles the nfsd threads in a more "bursty" way, then the burst > could try and spawn more mirror DS worker threads at about the same time. > > Anyhow, thanks for the help, rick From owner-freebsd-current@freebsd.org Sun Apr 22 12:05:36 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A4A76FBACDE for ; Sun, 22 Apr 2018 12:05:36 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EFB957F07A; Sun, 22 Apr 2018 12:05:35 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w3MC5LUp080766 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 22 Apr 2018 15:05:24 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w3MC5LUp080766 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w3MC5LAu080765; Sun, 22 Apr 2018 15:05:21 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 22 Apr 2018 15:05:21 +0300 From: Konstantin Belousov To: Tijl Coosemans Cc: Rick Macklem , "freebsd-current@freebsd.org" Subject: Re: i386 hangs during halt "vnodes remaining... 0 time out" Message-ID: <20180422120521.GS6887@kib.kiev.ua> References: <20180421234934.10d7dfab@kalimero.tijl.coosemans.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180421234934.10d7dfab@kalimero.tijl.coosemans.org> User-Agent: Mutt/1.9.5 (2018-04-13) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 12:05:36 -0000 On Sat, Apr 21, 2018 at 11:49:34PM +0200, Tijl Coosemans wrote: > On Sat, 21 Apr 2018 21:09:09 +0000 Rick Macklem wrote: > > With a recent head/current kernel (doesn't happen when running a Dec. > > 2017 one), when I do a halt, it gets as far as: > > > > vnodes remaining... 0 time out > > > > and that's it (the time out appears several seconds after the first "0"). > > With a Dec. 2017 kernel there would be several "0"s printed. > > It appears that it is stuck in the first iteration of the sched_sync() > > loop after it is no longer in SYNCER_RUNNING state. > > > > Any ideas? rick > > See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227404 > I have a patch (attached) but haven't been able to test it yet. > Index: sys/kern/vfs_bio.c > =================================================================== > --- sys/kern/vfs_bio.c (revision 332165) > +++ sys/kern/vfs_bio.c (working copy) > @@ -791,9 +791,12 @@ bufspace_daemon(void *arg) > { > struct bufdomain *bd; > > + EVENTHANDLER_REGISTER(shutdown_pre_sync, kthread_shutdown, curthread, > + SHUTDOWN_PRI_LAST); > + > bd = arg; > for (;;) { > - kproc_suspend_check(curproc); > + kthread_suspend_check(); > > /* > * Free buffers from the clean queue until we meet our > @@ -3357,7 +3360,7 @@ buf_daemon() > /* > * This process needs to be suspended prior to shutdown sync. > */ > - EVENTHANDLER_REGISTER(shutdown_pre_sync, kproc_shutdown, bufdaemonproc, > + EVENTHANDLER_REGISTER(shutdown_pre_sync, kthread_shutdown, curthread, > SHUTDOWN_PRI_LAST); > > /* > @@ -3381,7 +3384,7 @@ buf_daemon() > bd_request = 0; > mtx_unlock(&bdlock); > > - kproc_suspend_check(bufdaemonproc); > + kthread_suspend_check(); > > /* > * Save speedupreq for this pass and reset to capture new This looks fine. From owner-freebsd-current@freebsd.org Sun Apr 22 12:24:51 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 192C5FBC324; Sun, 22 Apr 2018 12:24:51 +0000 (UTC) (envelope-from tommi.pernila@gmail.com) Received: from mail-qk0-f182.google.com (mail-qk0-f182.google.com [209.85.220.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B2C6684228; Sun, 22 Apr 2018 12:24:50 +0000 (UTC) (envelope-from tommi.pernila@gmail.com) Received: by mail-qk0-f182.google.com with SMTP id v2so13339812qkh.10; Sun, 22 Apr 2018 05:24:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=rzBV0gGFuB+scnScF5EzWqajqDz5YFzZUpdgy2XddNg=; b=OrTPaZdZLrH8E5obD3D8RstebTGsQoIyMItZRobGjP7j6st/hFBbJppxR4Vmsrclaq ojQcAvgYDy0AoUROmQ36SI6nu5sIJtwNVyN48LwGhWKLCLnJyAQW7o2w8Jic0OXEHoX0 rjeziRRs61agyGyNFPznWFuCW6e8Rtt6xIWTWUuosBCiE3nMKKJc/7N7qq8+UOc+nquY pkH5jLkPT7o9N3KtB7Fir4gHgbb07CHH7tv0DRKs9ZaHdDdS6kqEazZWNTNUQmrcWnZk spgGHVSzDwo2fc7uvnYqeCn7akCJrixai24pjvFBzUt/c27/UtCXMAKpFGSM0/EJDRuJ GcUg== X-Gm-Message-State: ALQs6tCO+5h0sEHWRWvX5gAJsezRLCjwEyEiGOreFLPveXm8BSkBf0Zl 5Vt/Cp8vdZatkXCpa/vRb9kZBvyLqZwrMsoXwLItTA== X-Google-Smtp-Source: AB8JxZpkXTcuChjK4QV+vAMBum5ZFPMcIo/oNnZcCf0JFVEsJx68av2vDGWP5lnJuZEyx2MUJ656LC/dO29L7adFqns= X-Received: by 10.233.230.10 with SMTP id z10mr19331130qkf.365.1524399883697; Sun, 22 Apr 2018 05:24:43 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Tommi Pernila Date: Sun, 22 Apr 2018 12:24:32 +0000 Message-ID: Subject: Re: Nvidia issue with CURRENT To: Mariusz Zaborski Cc: FreeBSD Current , freebsd-desktop@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 12:24:51 -0000 Hi, are you running which version of CURRENT? E.g. Some snapshot or did you compile from source? -Tommi On Sun, 22 Apr 2018 at 13.47, Mariusz Zaborski wrote: > Hello, > > I upgraded my FreeBSD to CURRENT and nvidia-drvier-390.48. But it's > stop working. > I tried also nvidia-driver-390.25 without luck as well. > > I have loaded nvidia-modeset.ko . > > While I'm rebooting my machine its also core dumping: > https://people.freebsd.org/~oshogbo/nvidia-mail.png . > I'm attaching also Xorg log. > > Is this a known issue? > > Thanks, > Mariusz > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > From owner-freebsd-current@freebsd.org Sun Apr 22 12:38:57 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE3E2F86333; Sun, 22 Apr 2018 12:38:57 +0000 (UTC) (envelope-from oshogbo.vx@gmail.com) Received: from mail-pg0-x243.google.com (mail-pg0-x243.google.com [IPv6:2607:f8b0:400e:c05::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2ECB4864AC; Sun, 22 Apr 2018 12:38:57 +0000 (UTC) (envelope-from oshogbo.vx@gmail.com) Received: by mail-pg0-x243.google.com with SMTP id e9so6479851pgr.7; Sun, 22 Apr 2018 05:38:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=PO/lZW/dlnUGHndwyu6NmlU71lQbeS7ys8wMwT8ys/M=; b=P0meq2c4iowMsac/w5vdYm75l1Nj8MlFb8fuOBw6b25uWb+m+EEDPxP8UUgKcTIxtu Jlpxv7GP9RC5Zq856UKbp8nxlrefFnxqdSYNrNBNjdfFIyfwtS+Ir7R0yqxiZt+GHv6v hkz8Gmxr8aTA0pdjkP9HWI7a7+UyzYgwHYb1UaW1BR3dy5KMoG4eumxjppHKrieyNjKG vYlWrvyJmTWT1IEMZCDU2kdkpFGWIGK1xyYqpD74P7gGRn7Jog+RY393k3BAlEgLakNz xcuA/DR2MVMN3xZb/Ijk4igMtA5uBh4Jb55Q+UL6oJsDf4/cKc3va05Qz4Ok4tZaqCPa FpQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=PO/lZW/dlnUGHndwyu6NmlU71lQbeS7ys8wMwT8ys/M=; b=RjOX0o17zfovQ72+Bz5QsHZULoNgbIRjjauLliYA8MFqn78OVE7mr8SfxW4xIfIxVD V8ueWFTMKgrL/LZkhOb+MNTVHTxE063uqwv05A1NIUmeMXHa1sKGEjlFuxN6Cljsn4qj 33sA9ULvFPQlT9WjoVmi9ZPe0Dp3X38zt8dlJXlOGmjKcnBBccguW8h9zDKb5Wp0S+h0 rQupkrRcBXVI9cTExiDKq1wvKKGz940xOXOGWOryANmx8VjelzmPNKgGhb0dNNwH8le7 tGbWhCheAqFBADCbx8Sv86fQBWOqP4QZ+xSrojtH6roYku6aFj937tSlLp+TlCmvnGbM jBqg== X-Gm-Message-State: ALQs6tDjWUgpWjuAQRo7fIpSJUvCjXx6nw4s9GS8CUZ/TqefO4OxmXw3 aAcoAna+XUVaBWZp0Bxuv2cTNryP3NPY96McvxIShxjt X-Google-Smtp-Source: AIpwx49V1nUkvIjhw9AGNIcDOpChwNp0teBzFQGUWNCGfoh0fvfmfnA5SLf4I4PKsNKakG1cO2Q2dUyKDnOoPqVD2Jw= X-Received: by 2002:a17:902:70c7:: with SMTP id l7-v6mr16805160plt.165.1524400736278; Sun, 22 Apr 2018 05:38:56 -0700 (PDT) MIME-Version: 1.0 Sender: oshogbo.vx@gmail.com Received: by 10.100.145.137 with HTTP; Sun, 22 Apr 2018 05:38:55 -0700 (PDT) In-Reply-To: References: From: Mariusz Zaborski Date: Sun, 22 Apr 2018 14:38:55 +0200 X-Google-Sender-Auth: PcDD4yf0GGegdIJkpDuPWVC_o6Y Message-ID: Subject: Re: Nvidia issue with CURRENT To: Tommi Pernila Cc: FreeBSD Current , freebsd-desktop@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 12:38:57 -0000 Hi, Normally I build my CURRENT by myself from Xorg - r332861. But I also tried latest SNAPSHOT. Thanks, Mariusz On 22 April 2018 at 14:24, Tommi Pernila wrote: > Hi, > > are you running which version of CURRENT? > E.g. Some snapshot or did you compile from source? > > -Tommi > > On Sun, 22 Apr 2018 at 13.47, Mariusz Zaborski wrote: >> >> Hello, >> >> I upgraded my FreeBSD to CURRENT and nvidia-drvier-390.48. But it's >> stop working. >> I tried also nvidia-driver-390.25 without luck as well. >> >> I have loaded nvidia-modeset.ko . >> >> While I'm rebooting my machine its also core dumping: >> https://people.freebsd.org/~oshogbo/nvidia-mail.png . >> I'm attaching also Xorg log. >> >> Is this a known issue? >> >> Thanks, >> Mariusz >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" From owner-freebsd-current@freebsd.org Sun Apr 22 13:10:29 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1C6A0F8BD8E for ; Sun, 22 Apr 2018 13:10:29 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660079.outbound.protection.outlook.com [40.107.66.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9EFB36CD9D; Sun, 22 Apr 2018 13:10:28 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM (52.132.66.153) by YQBPR0101MB1249.CANPRD01.PROD.OUTLOOK.COM (52.132.68.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.696.12; Sun, 22 Apr 2018 13:10:27 +0000 Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::893c:efc2:d71f:945a]) by YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::893c:efc2:d71f:945a%13]) with mapi id 15.20.0696.017; Sun, 22 Apr 2018 13:10:27 +0000 From: Rick Macklem To: Konstantin Belousov , Tijl Coosemans CC: "freebsd-current@freebsd.org" Subject: Re: i386 hangs during halt "vnodes remaining... 0 time out" Thread-Topic: i386 hangs during halt "vnodes remaining... 0 time out" Thread-Index: AQHT2bR4fPVXlRRgnkynIGzjEjWSm6QLwdwAgADvGoCAABBIvg== Date: Sun, 22 Apr 2018 13:10:27 +0000 Message-ID: References: <20180421234934.10d7dfab@kalimero.tijl.coosemans.org>, <20180422120521.GS6887@kib.kiev.ua> In-Reply-To: <20180422120521.GS6887@kib.kiev.ua> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YQBPR0101MB1249; 7:Yn/SdKEVl4kiDRDT4UjniuOj2sPltNZ9YTvLzeuVeusnxzmrxHLt/oz0HMWuoaoddGfpow7f39HLKhNmP7ZUpe8RfDId6ku2+4vd4GXO3YE20SbJRfOupMCiIoMOGYe4CWM8vw+zKrxLm6QqV4z7vESrHGVtZ6xi8urwnSO3jkvnXTqFvWNjAsc+YIYBEzrHrZQ8mNlefiDQFufx+Sr+nQDLnWL1JhA13TbhlCXRGmqvJaJhWEJQMzlxn1xwZm40 x-ms-exchange-antispam-srfa-diagnostics: SOS; x-microsoft-antispam: UriScan:(80341031608730); BCL:0; PCL:0; RULEID:(7020095)(4652020)(8989080)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(8990040)(2017052603328)(7153060)(7193020); SRVR:YQBPR0101MB1249; x-ms-traffictypediagnostic: YQBPR0101MB1249: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(80341031608730); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(93006095)(93001095)(3231232)(944501410)(52105095)(3002001)(10201501046)(6041310)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123560045)(20161123558120)(20161123562045)(6072148)(201708071742011); SRVR:YQBPR0101MB1249; BCL:0; PCL:0; RULEID:; SRVR:YQBPR0101MB1249; x-forefront-prvs: 0650714AAA x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(346002)(396003)(39380400002)(376002)(39840400004)(99286004)(478600001)(39060400002)(4326008)(8676002)(81166006)(186003)(33656002)(102836004)(76176011)(59450400001)(7696005)(6506007)(5890100001)(55016002)(25786009)(5250100002)(2900100001)(9686003)(6306002)(229853002)(6246003)(53936002)(14454004)(2906002)(3660700001)(6436002)(74482002)(476003)(446003)(74316002)(8936002)(11346002)(5660300001)(786003)(966005)(316002)(110136005)(3280700002)(86362001)(305945005)(26005); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB1249; H:YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; MLV:ovrnspm; PTR:InfoNoRecords; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: IwLv94Zzl3vMpOLI50TpteZi+hDRXHD2wIyisvSFin7NYKXR01ywyqJzrBtLluR9vCEYI8v+tKPHLdrWvCtryYhE0d6Cw/PxMb/+SXzDvvCHxn56XRl3OkXjGT5Du8fqqkSDt6YohhVk4C6TLOAvAnGsJhH1rnAM2vEkjNvUd74roUUTrX0z3sRYo5yU/D3J spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: d79fab05-c7f5-420c-dcae-08d5a8526a95 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: d79fab05-c7f5-420c-dcae-08d5a8526a95 X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Apr 2018 13:10:27.1002 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1249 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 13:10:29 -0000 Konstantin Belousov wrote: >On Sat, Apr 21, 2018 at 11:49:34PM +0200, Tijl Coosemans wrote: >> On Sat, 21 Apr 2018 21:09:09 +0000 Rick Macklem w= rote: >> > With a recent head/current kernel (doesn't happen when running a Dec. >> > 2017 one), when I do a halt, it gets as far as: >> > >> > vnodes remaining... 0 time out >> > >> > and that's it (the time out appears several seconds after the first "0= "). >> > With a Dec. 2017 kernel there would be several "0"s printed. >> > It appears that it is stuck in the first iteration of the sched_sync() >> > loop after it is no longer in SYNCER_RUNNING state. >> > >> > Any ideas? rick >> >> See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D227404 >> I have a patch (attached) but haven't been able to test it yet. > >> Index: sys/kern/vfs_bio.c >> >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> --- sys/kern/vfs_bio.c (revision 332165) >> +++ sys/kern/vfs_bio.c (working copy) >> @@ -791,9 +791,12 @@ bufspace_daemon(void *arg) >> { >> struct bufdomain *bd; >> >> + EVENTHANDLER_REGISTER(shutdown_pre_sync, kthread_shutdown, curthre= ad, >> + SHUTDOWN_PRI_LAST); >> + >> bd =3D arg; >> for (;;) { >> - kproc_suspend_check(curproc); >> + kthread_suspend_check(); >> >> /* >> * Free buffers from the clean queue until we meet our >> @@ -3357,7 +3360,7 @@ buf_daemon() >> /* >> * This process needs to be suspended prior to shutdown sync. >> */ >> - EVENTHANDLER_REGISTER(shutdown_pre_sync, kproc_shutdown, bufdaemon= proc, >> + EVENTHANDLER_REGISTER(shutdown_pre_sync, kthread_shutdown, curthre= ad, >> SHUTDOWN_PRI_LAST); >> >> /* >> @@ -3381,7 +3384,7 @@ buf_daemon() >> bd_request =3D 0; >> mtx_unlock(&bdlock); >> >> - kproc_suspend_check(bufdaemonproc); >> + kthread_suspend_check(); >> >> /* >> * Save speedupreq for this pass and reset to capture new >This looks fine. For some reason, this thread became two threads, so I'll reply to this one = as well. The patch seems to work fine for me. Thanks, rick= From owner-freebsd-current@freebsd.org Sun Apr 22 13:15:10 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 689B9F915C3 for ; Sun, 22 Apr 2018 13:15:10 +0000 (UTC) (envelope-from tijl@freebsd.org) Received: from mailrelay116.isp.belgacom.be (mailrelay116.isp.belgacom.be [195.238.20.143]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "relay.skynet.be", Issuer "GlobalSign Organization Validation CA - SHA256 - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C97BB6D21D; Sun, 22 Apr 2018 13:15:09 +0000 (UTC) (envelope-from tijl@freebsd.org) X-Belgacom-Dynamic: yes IronPort-PHdr: =?us-ascii?q?9a23=3AerQLsxUOoVfPZGb0Ysxp4S6wLUPV8LGtZVwlr6E/?= =?us-ascii?q?grcLSJyIuqrYYxOCt8tkgFKBZ4jH8fUM07OQ7/i7HzRYqb+681k6OKRWUBEEjc?= =?us-ascii?q?hE1ycBO+WiTXPBEfjxciYhF95DXlI2t1uyMExSBdqsLwaK+i764jEdAAjwOhRo?= =?us-ascii?q?LerpBIHSk9631+ev8JHPfglEnjWwba98IRmssQndqtQdjJd/JKo21hbHuGZDdf?= =?us-ascii?q?5MxWNvK1KTnhL86dm18ZV+7SleuO8v+tBZX6nicKs2UbJXDDI9M2Ao/8LrrgXM?= =?us-ascii?q?TRGO5nQHTGoblAdDDhXf4xH7WpfxtTb6tvZ41SKHM8D6Uaw4VDK/5KptVRTmij?= =?us-ascii?q?oINyQh/W/ZisJ+kr9VrhGjqBxxzIHbfI6bOeFifq7fYd8WWXZNUtpPWyFHH4iy?= =?us-ascii?q?b5EPD+0EPetAsYf9plkOrR+jDgSyA+PvzSRIiWHz3aIg1eQhChzN0Qs8H9IPsn?= =?us-ascii?q?TUqM74OqcIUe+r0qbF0CjNYf1M1Tf68ojIfQksrPeRVrx+dsrRzFMgFwLDjliI?= =?us-ascii?q?p4zlJS2a2foWvGiA8uVsT+Wvi3Qoqwx3vzOhxd8sh5HKi44I0FzI6yp0zJsvKd?= =?us-ascii?q?GmVEJ3f8SoHIZQuiyUMYZ9X9ksTHtyuCkgz70LoZu7fC8Xx5s53xPfcPmHc5SQ?= =?us-ascii?q?4hLkSeaRPS90hHJ7d7K7gBa/6Uugxff4Vsm11VZFsDBFkt7WunAR1hzT6MyHRu?= =?us-ascii?q?Fh8Uem3jaPzB7c6uReLkAyjqrXMZkhwqQ/lpYLsETDGDH5mFnugaKVa0ko4Pak?= =?us-ascii?q?5uv6brn8uJOQK5F4hhvjPqkulMGzGeE4PRIPX2if9+S8zrrj/UjhTbVWj/02kK?= =?us-ascii?q?3ZvYvUJcQBuKG2HRRa0p0+5BqlCDemytsYkWEdLF1ZYBKHk5TpO1bWLfD7Cve/?= =?us-ascii?q?mEiskDZox//dILLhBo7ALnfGkLj7fLZ971RQxxY0zdBatNpoDeQiJ/ToRkb3qN?= =?us-ascii?q?3eRjU0Nwup2OH5QIF+0ZgCWGGFD6uxP6bbsFvO7eUqdbqifogQ7Qr8KfxtzPnp?= =?us-ascii?q?lnI8kFkGNf213JkTQF6iE/lMGGnfZmDj1IRSWVwWtxYzGbS5wGaJViReMi6/?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A2BUHACEidxa/5nK8VFVBx4BHg2DKlMOe?= =?us-ascii?q?iiMSowXAQGBczEBXYZsjBQUgWQuhEgCgkQiNRcBAgEBAQEBAQIBaxwMgjUigkw?= =?us-ascii?q?BBTocIxALDgoJJQ8SGB4GE4R3AxkLqRiHBg2BK4IpBYofhBqCT0IEGIEvBBOFV?= =?us-ascii?q?AKXRywIhVyFZ4JyaItxiTZAh1weATWBUk0wCIJ+giAXiFmFQD0wAY4GKYIdAQE?= X-IPAS-Result: =?us-ascii?q?A2BUHACEidxa/5nK8VFVBx4BHg2DKlMOeiiMSowXAQGBczE?= =?us-ascii?q?BXYZsjBQUgWQuhEgCgkQiNRcBAgEBAQEBAQIBaxwMgjUigkwBBTocIxALDgoJJ?= =?us-ascii?q?Q8SGB4GE4R3AxkLqRiHBg2BK4IpBYofhBqCT0IEGIEvBBOFVAKXRywIhVyFZ4J?= =?us-ascii?q?yaItxiTZAh1weATWBUk0wCIJ+giAXiFmFQD0wAY4GKYIdAQE?= Received: from 153.202-241-81.adsl-dyn.isp.belgacom.be (HELO kalimero.tijl.coosemans.org) ([81.241.202.153]) by relay.skynet.be with ESMTP; 22 Apr 2018 15:15:01 +0200 Received: from kalimero.tijl.coosemans.org (kalimero.tijl.coosemans.org [127.0.0.1]) by kalimero.tijl.coosemans.org (8.15.2/8.15.2) with ESMTP id w3MDF1Ai062374; Sun, 22 Apr 2018 15:15:01 +0200 (CEST) (envelope-from tijl@FreeBSD.org) Date: Sun, 22 Apr 2018 15:15:00 +0200 From: Tijl Coosemans To: Konstantin Belousov Cc: jeff@FreeBSD.org, "freebsd-current@freebsd.org" Subject: Re: i386 hangs during halt "vnodes remaining... 0 time out" Message-ID: <20180422151500.1608af96@kalimero.tijl.coosemans.org> In-Reply-To: <20180422120521.GS6887@kib.kiev.ua> References: <20180421234934.10d7dfab@kalimero.tijl.coosemans.org> <20180422120521.GS6887@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 13:15:10 -0000 On Sun, 22 Apr 2018 15:05:21 +0300 Konstantin Belousov wrote: > On Sat, Apr 21, 2018 at 11:49:34PM +0200, Tijl Coosemans wrote: >> On Sat, 21 Apr 2018 21:09:09 +0000 Rick Macklem wrote: >>> With a recent head/current kernel (doesn't happen when running a Dec. >>> 2017 one), when I do a halt, it gets as far as: >>> >>> vnodes remaining... 0 time out >>> >>> and that's it (the time out appears several seconds after the first "0"). >>> With a Dec. 2017 kernel there would be several "0"s printed. >>> It appears that it is stuck in the first iteration of the sched_sync() >>> loop after it is no longer in SYNCER_RUNNING state. >>> >>> Any ideas? rick >> >> See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227404 >> I have a patch (attached) but haven't been able to test it yet. >> >> Index: sys/kern/vfs_bio.c >> =================================================================== >> --- sys/kern/vfs_bio.c (revision 332165) >> +++ sys/kern/vfs_bio.c (working copy) >> @@ -791,9 +791,12 @@ bufspace_daemon(void *arg) >> { >> struct bufdomain *bd; >> >> + EVENTHANDLER_REGISTER(shutdown_pre_sync, kthread_shutdown, curthread, >> + SHUTDOWN_PRI_LAST); >> + >> bd = arg; >> for (;;) { >> - kproc_suspend_check(curproc); >> + kthread_suspend_check(); >> >> /* >> * Free buffers from the clean queue until we meet our >> @@ -3357,7 +3360,7 @@ buf_daemon() >> /* >> * This process needs to be suspended prior to shutdown sync. >> */ >> - EVENTHANDLER_REGISTER(shutdown_pre_sync, kproc_shutdown, bufdaemonproc, >> + EVENTHANDLER_REGISTER(shutdown_pre_sync, kthread_shutdown, curthread, >> SHUTDOWN_PRI_LAST); >> >> /* >> @@ -3381,7 +3384,7 @@ buf_daemon() >> bd_request = 0; >> mtx_unlock(&bdlock); >> >> - kproc_suspend_check(bufdaemonproc); >> + kthread_suspend_check(); >> >> /* >> * Save speedupreq for this pass and reset to capture new > This looks fine. Thanks for the review. There's just one concern I have. With this patch the bufspace_daemon threads appear to shutdown after the buf_daemon and after the syncer because the event handlers are registered later. Are there any dependencies between these processes that require the bufspace threads to be stopped earlier? From owner-freebsd-current@freebsd.org Sun Apr 22 13:29:07 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65E9DF928F1 for ; Sun, 22 Apr 2018 13:29:07 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E1C8B71F84; Sun, 22 Apr 2018 13:29:06 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w3MDStiC099568 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 22 Apr 2018 16:28:59 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w3MDStiC099568 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w3MDSttO099567; Sun, 22 Apr 2018 16:28:55 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 22 Apr 2018 16:28:55 +0300 From: Konstantin Belousov To: Tijl Coosemans Cc: jeff@FreeBSD.org, "freebsd-current@freebsd.org" Subject: Re: i386 hangs during halt "vnodes remaining... 0 time out" Message-ID: <20180422132855.GU6887@kib.kiev.ua> References: <20180421234934.10d7dfab@kalimero.tijl.coosemans.org> <20180422120521.GS6887@kib.kiev.ua> <20180422151500.1608af96@kalimero.tijl.coosemans.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180422151500.1608af96@kalimero.tijl.coosemans.org> User-Agent: Mutt/1.9.5 (2018-04-13) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 13:29:07 -0000 On Sun, Apr 22, 2018 at 03:15:00PM +0200, Tijl Coosemans wrote: > On Sun, 22 Apr 2018 15:05:21 +0300 Konstantin Belousov wrote: > > On Sat, Apr 21, 2018 at 11:49:34PM +0200, Tijl Coosemans wrote: > >> On Sat, 21 Apr 2018 21:09:09 +0000 Rick Macklem wrote: > >>> With a recent head/current kernel (doesn't happen when running a Dec. > >>> 2017 one), when I do a halt, it gets as far as: > >>> > >>> vnodes remaining... 0 time out > >>> > >>> and that's it (the time out appears several seconds after the first "0"). > >>> With a Dec. 2017 kernel there would be several "0"s printed. > >>> It appears that it is stuck in the first iteration of the sched_sync() > >>> loop after it is no longer in SYNCER_RUNNING state. > >>> > >>> Any ideas? rick > >> > >> See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227404 > >> I have a patch (attached) but haven't been able to test it yet. > >> > >> Index: sys/kern/vfs_bio.c > >> =================================================================== > >> --- sys/kern/vfs_bio.c (revision 332165) > >> +++ sys/kern/vfs_bio.c (working copy) > >> @@ -791,9 +791,12 @@ bufspace_daemon(void *arg) > >> { > >> struct bufdomain *bd; > >> > >> + EVENTHANDLER_REGISTER(shutdown_pre_sync, kthread_shutdown, curthread, > >> + SHUTDOWN_PRI_LAST); > >> + > >> bd = arg; > >> for (;;) { > >> - kproc_suspend_check(curproc); > >> + kthread_suspend_check(); > >> > >> /* > >> * Free buffers from the clean queue until we meet our > >> @@ -3357,7 +3360,7 @@ buf_daemon() > >> /* > >> * This process needs to be suspended prior to shutdown sync. > >> */ > >> - EVENTHANDLER_REGISTER(shutdown_pre_sync, kproc_shutdown, bufdaemonproc, > >> + EVENTHANDLER_REGISTER(shutdown_pre_sync, kthread_shutdown, curthread, > >> SHUTDOWN_PRI_LAST); > >> > >> /* > >> @@ -3381,7 +3384,7 @@ buf_daemon() > >> bd_request = 0; > >> mtx_unlock(&bdlock); > >> > >> - kproc_suspend_check(bufdaemonproc); > >> + kthread_suspend_check(); > >> > >> /* > >> * Save speedupreq for this pass and reset to capture new > > This looks fine. > > Thanks for the review. There's just one concern I have. With this patch > the bufspace_daemon threads appear to shutdown after the buf_daemon and > after the syncer because the event handlers are registered later. Are > there any dependencies between these processes that require the bufspace > threads to be stopped earlier? I think for correctness bufdaemon must stop after the syncer, since syncer operation can cause a situation where bufdaemon help is needed to proceed. Other than this, the stop order is irrelevant, because after syncer finished, there should be no any further filesystem activity. From owner-freebsd-current@freebsd.org Sun Apr 22 13:43:56 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51D7BFA021A for ; Sun, 22 Apr 2018 13:43:56 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670075.outbound.protection.outlook.com [40.107.67.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D18C0748BA for ; Sun, 22 Apr 2018 13:43:55 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM (52.132.66.153) by YQBPR0101MB1699.CANPRD01.PROD.OUTLOOK.COM (52.132.70.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.696.13; Sun, 22 Apr 2018 13:43:53 +0000 Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::893c:efc2:d71f:945a]) by YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::893c:efc2:d71f:945a%13]) with mapi id 15.20.0696.017; Sun, 22 Apr 2018 13:43:53 +0000 From: Rick Macklem To: Konstantin Belousov CC: "freebsd-current@freebsd.org" , "George Mitchell" , Peter Subject: Re: SCHED_ULE makes 256Mbyte i386 unusable Thread-Topic: SCHED_ULE makes 256Mbyte i386 unusable Thread-Index: AQHT2aRaIslt1yc1ckCslUza9Eao6KQLppMAgAA0JP2AANWhgIAAF4Lf Date: Sun, 22 Apr 2018 13:43:53 +0000 Message-ID: References: <20180421201128.GO6887@kib.kiev.ua> , <20180422120241.GR6887@kib.kiev.ua> In-Reply-To: <20180422120241.GR6887@kib.kiev.ua> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YQBPR0101MB1699; 7:K437CgpTt7z7Pq2hwuVylwu8PRWykaEpDOiUPdNkBsv10qZ2QrtcqskvwJVl/khJ96c2NXhmpy83zNm3RHxUPpeerfMk7cgzBDs03ar025n3u+zn08awqx70S/sXXG19BaVlcrGtmH761pDam9DHOJKc9c6xpJZ8XKDX+n8K97QrmnASSs63/tP1lr/n1HxaOrtF5YBSC7Yh8UDgcZ2BaI/z0OZEAJOE2ZInUW/CkqKqxL05BGo37I7b20BxPOr6 x-ms-exchange-antispam-srfa-diagnostics: SOS; x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(8989080)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(8990040)(2017052603328)(7153060)(7193020); SRVR:YQBPR0101MB1699; x-ms-traffictypediagnostic: YQBPR0101MB1699: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(93006095)(93001095)(3231232)(944501410)(52105095)(3002001)(10201501046)(6041310)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123560045)(20161123558120)(20161123562045)(6072148)(201708071742011); SRVR:YQBPR0101MB1699; BCL:0; PCL:0; RULEID:; SRVR:YQBPR0101MB1699; x-forefront-prvs: 0650714AAA x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(376002)(39380400002)(39860400002)(396003)(366004)(346002)(51914003)(86362001)(5660300001)(6246003)(9686003)(26005)(305945005)(8936002)(7696005)(5250100002)(39060400002)(446003)(786003)(53936002)(316002)(476003)(11346002)(74316002)(81166006)(8676002)(186003)(54906003)(99286004)(25786009)(6916009)(6436002)(6506007)(93886005)(33656002)(2906002)(3660700001)(102836004)(3280700002)(74482002)(14454004)(59450400001)(4326008)(55016002)(229853002)(76176011)(478600001)(1411001)(2900100001); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB1699; H:YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; MLV:ovrnspm; PTR:InfoNoRecords; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: OnxlmrpFg7QE1602QIHbLozt4Ow1yqv9NXToiIVc3bGHmsgTlY04ChpY7hM+zINtfiEPaLzTOyqsp0wPVN5Ql4HvKwPkz4EvkpNDNDjkH36oc9B+ZPVa7+FV193pSEaGYdhZWOpXXqLK1fmzf5PpG6HRuRrHD9JWdSpw+UIZtWIBaB0a8WqTfbFPjl4mN1FQ spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 836cacd9-f1f7-47a0-610c-08d5a85716a8 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 836cacd9-f1f7-47a0-610c-08d5a85716a8 X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Apr 2018 13:43:53.8079 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1699 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 13:43:56 -0000 Konstantin Belousov wrote: >On Sat, Apr 21, 2018 at 11:30:55PM +0000, Rick Macklem wrote: >> Konstantin Belousov wrote: >> >On Sat, Apr 21, 2018 at 07:21:58PM +0000, Rick Macklem wrote: >> >> I decided to start a new thread on current related to SCHED_ULE, sinc= e I see >> >> more than just performance degradation and on a recent current kernel= . >> >> (I cc'd a couple of the people discussing performance problems in fre= ebsd-stable >> >> recently under a subject line of "Re: kern.sched.quantum: Creepy, sa= distic scheduler". >> >> >> >> When testing a pNFS server on a single core i386 with 256Mbytes using= a Dec. 2017 >> >> current/head kernel, I would see about a 30% performance degradation = (elapsed >> >> run time for a kernel build over NFSv4.1) when the server kernel was = built with >> >> options SCHED_ULE >> >> instead of >> >> options SCHED_4BSD So, now that I have decreased the number of nfsd kernel threads to 32, it w= orks with both schedulers and with essentially the same performance. (ie. The 30= % performance degradation has disappeared.) >> >> >> >> Now, with a kernel from a couple of days ago, the >> >> options SCHED_ULE >> >> kernel becomes unusable shortly after starting testing. >> >> I have seen two variants of this: >> >> - Became essentially hung. All I could do was ping the machine from t= he network. >> >> - Reported "vm_thread_new: kstack allocation failed >> >> and then any attempt to do anything gets "No more processes". >> >This is strange. It usually means that you get KVA either exhausted or >> >severly fragmented. >> Yes. I reduced the number of nfsd threads from 256->32 and the SCHED_ULE >> kernel is working ok now. I haven't done enough to compare performance y= et. >> Maybe I'll post again when I have some numbers. >> >> >Enter ddb, it should be operational since pings are replied. Try to se= e >> >where the threads are stuck. >> I didn't do this, since reducing the number of kernel threads seems to h= ave fixed >> the problem. For the pNFS server, the nfsd threads will spawn additional= kernel >> threads to do proxies to the mirrored DS servers. >> >> >> with the only difference being a kernel built with >> >> options SCHED_4BSD >> >> everything works and performs the same as the Dec 2017 kernel. >> >> >> >> I can try rolling back through the revisions, but it would be nice if= someone >> >> could suggest where to start, because it takes a couple of hours to b= uild a >> >> kernel on this system. >> >> >> >> So, something has made things worse for a head/current kernel this wi= nter, rick >> > >> >There are at least two potentially relevant changes. >> > >> >First is r326758 Dec 11 which bumped KSTACK_PAGES on i386 to 4. >> I've been running this machine with KSTACK_PAGES=3D4 for some time, so n= o change. W.r.t. Rodney Grimes comments about this (which didn't end up in this messa= ges in the thread): I didn't see any instability when using KSTACK_PAGES=3D4 for this until thi= s cropped up and seemed to be scheduler related (but not really, it seems). I bumped it to KSTACK_PAGES=3D4 because I needed that for the pNFS Metadata Server code. Yes, NFS does use quite a bit of kernel stack. Unfortunately, it isn't one = big item getting allocated on the stack, but many moderate sized ones. (A part of it is multiple instances of "struct vattr", some buried in "stru= ct nfsvattr", that NFS needs to use. I don't think these are large enough to justify mal= loc/free, but it has to use several of them.) One case I did try fixing was about 6 cases where "struct nfsstate" ended u= p on the stack. I changes the code to malloc/free them and then when testing, to my surprise I had a 20% performance hit and shelved the patch. Now that I know that the server was running near its limit, I might try thi= s one again, to see if the performance hit doesn't occur when the machine has ade= quate memory. If the performance hit goes away, I could commit this, but it would= n't have that much effect on the kstack usage. (It's interesting how this p= atch ended up related to the issue this thread discussed.) >> >> >Second is r332489 Apr 13, which introduced 4/4G KVA/UVA split. >> Could this change have resulted in the system being able to allocate few= er >> kernel threads/stacks for some reason? >Well, it could, as anything can be buggy. But the intent of the change >was to give 4G KVA, and it did. Righto. No concern here. I suspect the Dec. 2017 kernel was close to the li= mit (see performance issue that went away, noted above) and any change could have pushed it across the line, I think. >> >> >Consequences of the first one are obvious, it is much harder to find >> >the place to map the stack. Second change, on the other hand, provides >> >almost full 4G for KVA and should have mostly compensate for the negati= ve >> >effects of the first. >> > >> >And, I cannot see how changing the scheduler would fix or even affect t= hat >> >behaviour. >> My hunch is that the system was running near its limit for kernel thread= s/stacks. >> Then, somehow, the timing SCHED_ULE caused resulted in the nfsd trying t= o get >> to a higher peak number of threads and hit the limit. >> SCHED_4BSD happened to result in timing such that it stayed just below t= he >> limit and worked. >> I can think of a couple of things that might affect this: >> 1 - If SCHED_ULE doesn't do the termination of kernel threads as quickly= , then >> they wouldn't terminate and release their resources before more ne= w ones >> are spawned. >Scheduler has nothing to do with the threads termination. It might >select running threads in a way that causes the undesired pattern to >appear which might create some amount of backlog for termination, but >I doubt it. > >> 2 - If SCHED_ULE handles the nfsd threads in a more "bursty" way, then t= he burst >> could try and spawn more mirror DS worker threads at about the sam= e time. >> >> Anyhow, thanks for the help, rick Have a good day, rick= From owner-freebsd-current@freebsd.org Sun Apr 22 14:36:41 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A06AFA459E for ; Sun, 22 Apr 2018 14:36:41 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 864CB81084 for ; Sun, 22 Apr 2018 14:36:40 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w3MEa9JH080703; Sun, 22 Apr 2018 07:36:10 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w3MEa9DY080702; Sun, 22 Apr 2018 07:36:09 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201804221436.w3MEa9DY080702@pdx.rh.CN85.dnsmgr.net> Subject: Re: SCHED_ULE makes 256Mbyte i386 unusable In-Reply-To: To: Rick Macklem Date: Sun, 22 Apr 2018 07:36:09 -0700 (PDT) CC: Konstantin Belousov , "freebsd-current@freebsd.org" , George Mitchell , Peter X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 14:36:41 -0000 > Konstantin Belousov wrote: > >On Sat, Apr 21, 2018 at 11:30:55PM +0000, Rick Macklem wrote: > >> Konstantin Belousov wrote: > >> >On Sat, Apr 21, 2018 at 07:21:58PM +0000, Rick Macklem wrote: > >> >> I decided to start a new thread on current related to SCHED_ULE, since I see > >> >> more than just performance degradation and on a recent current kernel. > >> >> (I cc'd a couple of the people discussing performance problems in freebsd-stable > >> >> recently under a subject line of "Re: kern.sched.quantum: Creepy, sadistic scheduler". > >> >> > >> >> When testing a pNFS server on a single core i386 with 256Mbytes using a Dec. 2017 > >> >> current/head kernel, I would see about a 30% performance degradation (elapsed > >> >> run time for a kernel build over NFSv4.1) when the server kernel was built with > >> >> options SCHED_ULE > >> >> instead of > >> >> options SCHED_4BSD > So, now that I have decreased the number of nfsd kernel threads to 32, it works > with both schedulers and with essentially the same performance. (ie. The 30% > performance degradation has disappeared.) > > >> >> > >> >> Now, with a kernel from a couple of days ago, the > >> >> options SCHED_ULE > >> >> kernel becomes unusable shortly after starting testing. > >> >> I have seen two variants of this: > >> >> - Became essentially hung. All I could do was ping the machine from the network. > >> >> - Reported "vm_thread_new: kstack allocation failed > >> >> and then any attempt to do anything gets "No more processes". > >> >This is strange. It usually means that you get KVA either exhausted or > >> >severly fragmented. > >> Yes. I reduced the number of nfsd threads from 256->32 and the SCHED_ULE > >> kernel is working ok now. I haven't done enough to compare performance yet. > >> Maybe I'll post again when I have some numbers. > >> > >> >Enter ddb, it should be operational since pings are replied. Try to see > >> >where the threads are stuck. > >> I didn't do this, since reducing the number of kernel threads seems to have fixed > >> the problem. For the pNFS server, the nfsd threads will spawn additional kernel > >> threads to do proxies to the mirrored DS servers. > >> > >> >> with the only difference being a kernel built with > >> >> options SCHED_4BSD > >> >> everything works and performs the same as the Dec 2017 kernel. > >> >> > >> >> I can try rolling back through the revisions, but it would be nice if someone > >> >> could suggest where to start, because it takes a couple of hours to build a > >> >> kernel on this system. > >> >> > >> >> So, something has made things worse for a head/current kernel this winter, rick > >> > > >> >There are at least two potentially relevant changes. > >> > > >> >First is r326758 Dec 11 which bumped KSTACK_PAGES on i386 to 4. > >> I've been running this machine with KSTACK_PAGES=4 for some time, so no change. > W.r.t. Rodney Grimes comments about this (which didn't end up in this messages > in the thread): > I didn't see any instability when using KSTACK_PAGES=4 for this until this cropped > up and seemed to be scheduler related (but not really, it seems). > I bumped it to KSTACK_PAGES=4 because I needed that for the pNFS Metadata > Server code. > > Yes, NFS does use quite a bit of kernel stack. Unfortunately, it isn't one big > item getting allocated on the stack, but many moderate sized ones. > (A part of it is multiple instances of "struct vattr", some buried in "struct nfsvattr", > that NFS needs to use. I don't think these are large enough to justify malloc/free, > but it has to use several of them.) > > One case I did try fixing was about 6 cases where "struct nfsstate" ended up on > the stack. I changes the code to malloc/free them and then when testing, to > my surprise I had a 20% performance hit and shelved the patch. > Now that I know that the server was running near its limit, I might try this one > again, to see if the performance hit doesn't occur when the machine has adequate > memory. If the performance hit goes away, I could commit this, but it wouldn't > have that much effect on the kstack usage. (It's interesting how this patch ended > up related to the issue this thread discussed.) Anything we can do to help relieve KSTACK usage, especially on i386 is helpfull. These is a thread back quite some time where someone came up with a compile time static "this functions uses X bytes of local stack" and a bit of clean up was done. We should persue this issue further. My experiece with the i386/KSTACK issues was attempting to do installs from snapshot .iso's, I usually had to change to a custom kernel without INVARIANTS and WITNESS, or reduce KSTACK to 2 and suffer the small stack problem (ie, dont use NFS during install). Neither was very pleasant. I have found it in practical to run the 4 page KSTACK in production VM's using i386 due to memory requirements. I run many very lean i386 VM's with 64MB of memory. I suspect our user base also has many people doing this, and it would be to our advantage to try and reduce our kernel stack needs. > >> >Second is r332489 Apr 13, which introduced 4/4G KVA/UVA split. > >> Could this change have resulted in the system being able to allocate fewer > >> kernel threads/stacks for some reason? > >Well, it could, as anything can be buggy. But the intent of the change > >was to give 4G KVA, and it did. > Righto. No concern here. I suspect the Dec. 2017 kernel was close to the limit > (see performance issue that went away, noted above) and any change could > have pushed it across the line, I think. > > >> > >> >Consequences of the first one are obvious, it is much harder to find > >> >the place to map the stack. Second change, on the other hand, provides > >> >almost full 4G for KVA and should have mostly compensate for the negative > >> >effects of the first. > >> > > >> >And, I cannot see how changing the scheduler would fix or even affect that > >> >behaviour. > >> My hunch is that the system was running near its limit for kernel threads/stacks. > >> Then, somehow, the timing SCHED_ULE caused resulted in the nfsd trying to get > >> to a higher peak number of threads and hit the limit. > >> SCHED_4BSD happened to result in timing such that it stayed just below the > >> limit and worked. > >> I can think of a couple of things that might affect this: > >> 1 - If SCHED_ULE doesn't do the termination of kernel threads as quickly, then > >> they wouldn't terminate and release their resources before more new ones > >> are spawned. > >Scheduler has nothing to do with the threads termination. It might > >select running threads in a way that causes the undesired pattern to > >appear which might create some amount of backlog for termination, but > >I doubt it. > > > >> 2 - If SCHED_ULE handles the nfsd threads in a more "bursty" way, then the burst > >> could try and spawn more mirror DS worker threads at about the same time. > >> > >> Anyhow, thanks for the help, rick > > Have a good day, rick > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-current@freebsd.org Sun Apr 22 14:55:22 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3019CFA5E18 for ; Sun, 22 Apr 2018 14:55:22 +0000 (UTC) (envelope-from tijl@freebsd.org) Received: from mailrelay105.isp.belgacom.be (mailrelay105.isp.belgacom.be [195.238.20.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "relay.skynet.be", Issuer "GlobalSign Organization Validation CA - SHA256 - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 76E90860F9; Sun, 22 Apr 2018 14:55:21 +0000 (UTC) (envelope-from tijl@freebsd.org) X-Belgacom-Dynamic: yes IronPort-PHdr: =?us-ascii?q?9a23=3Ai4P2TxfahRkGaAIvXLX04mNtlGMj4u6mDksu8pMi?= =?us-ascii?q?zoh2WeGdxcu5YR7h7PlgxGXEQZ/co6odzbaO6Oa4ASQp2tWoiDg6aptCVhsI24?= =?us-ascii?q?09vjcLJ4q7M3D9N+PgdCcgHc5PBxdP9nC/NlVJSo6lPwWB6nK94iQPFRrhKAF7?= =?us-ascii?q?Ovr6GpLIj8Swyuu+54Dfbx9HiTahb75+Ngm6oRnMvcQKnIVuLbo8xAHUqXVSYe?= =?us-ascii?q?RWwm1oJVOXnxni48q74YBu/SdNtf8/7sBMSar1cbg2QrxeFzQmLns65Nb3uhnZ?= =?us-ascii?q?TAuA/WUTX2MLmRdVGQfF7RX6XpDssivms+d2xSeXMdHqQb0yRD+v6bpgRh31hy?= =?us-ascii?q?cdLzM3/mHZhNJtgqxYoh2hqRNwzJLbboyOKPpzfL/Rcc8GSWZdQMpcUTFKDIOm?= =?us-ascii?q?b4sICuoMJfhWr4j/p1sKsBCzGw6sBOT0yjBWg3/5x6s60/88GgzBwAwgHtAOsH?= =?us-ascii?q?DPodv1LqcdT/66wbTVwzvNdf9W3i396IfRfx0nvPqCXqpwfNLSxEUyDQ/JkFqd?= =?us-ascii?q?pZH/Mz+LyugBrXKX4/dgWO+hjWMstht/rSK1xsg2j4nEnocVylfZ+ipn2Is1Pt?= =?us-ascii?q?i4SFJjYd6jDZtQqzmWN4toTcMmRGFloCU6xacCuZ66eSgF1o4nxxnFZ/ybcoiI?= =?us-ascii?q?4BbjWPyNLjd/g3JlY6ywhxOo/kim0e3wTM600ExFriZdk9nMsG4C1wDL58WEV/?= =?us-ascii?q?dx5Fmt1DmS2wzJ9O1IPV44mbDGJ5MhzLM8jp8Tvl7CHi/ylkX2lqiWdkA89+i0?= =?us-ascii?q?6uTnYLHmq4SSN49ulA7xLL8hmteiDugiNQgORWeb9fym1LL/5U35XKlKjvoun6?= =?us-ascii?q?nat5DaPtgbpq+6AwBOzIkj7w2yDzij0NsCnHkHKEhJdw6Aj4jsaBnyJ6XbCvGk?= =?us-ascii?q?n12qjDZtj9rLOrr8GZLTZizAl6z9fLV35kp0xw86zNQZ7JVRXOIvOvX2D3Pwtt?= =?us-ascii?q?iQJRg+KAGxyuD8QIFh14EacUyVD6KzC4+UtkWHsLF8a9KQbZMY7W6uY8Mu4OTj?= =?us-ascii?q?2Cc0?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A2A0HAADodxa/5nK8VFcg0krU4EIKIxKj?= =?us-ascii?q?BcBAYFzMQFdhmyODIR2AoJEIjcVAQIBAQEBAQECAWsogjUigkwBBTocIxALDgo?= =?us-ascii?q?JJQ8SGB4GE4R3AxmpLYcGDYErgi6KH4Qagk+HeAKXRywIiCyDF4JyaItxiXaHX?= =?us-ascii?q?DIigVJNMAiCfoMxAQiNFj0wkE0BAQ?= X-IPAS-Result: =?us-ascii?q?A2A0HAADodxa/5nK8VFcg0krU4EIKIxKjBcBAYFzMQFdhmy?= =?us-ascii?q?ODIR2AoJEIjcVAQIBAQEBAQECAWsogjUigkwBBTocIxALDgoJJQ8SGB4GE4R3A?= =?us-ascii?q?xmpLYcGDYErgi6KH4Qagk+HeAKXRywIiCyDF4JyaItxiXaHXDIigVJNMAiCfoM?= =?us-ascii?q?xAQiNFj0wkE0BAQ?= Received: from 153.202-241-81.adsl-dyn.isp.belgacom.be (HELO kalimero.tijl.coosemans.org) ([81.241.202.153]) by relay.skynet.be with ESMTP; 22 Apr 2018 16:55:15 +0200 Received: from kalimero.tijl.coosemans.org (kalimero.tijl.coosemans.org [127.0.0.1]) by kalimero.tijl.coosemans.org (8.15.2/8.15.2) with ESMTP id w3MEtDOL062720; Sun, 22 Apr 2018 16:55:13 +0200 (CEST) (envelope-from tijl@FreeBSD.org) Date: Sun, 22 Apr 2018 16:55:13 +0200 From: Tijl Coosemans To: Konstantin Belousov Cc: jeff@FreeBSD.org, "freebsd-current@freebsd.org" Subject: Re: i386 hangs during halt "vnodes remaining... 0 time out" Message-ID: <20180422165513.711579c3@kalimero.tijl.coosemans.org> In-Reply-To: <20180422132855.GU6887@kib.kiev.ua> References: <20180421234934.10d7dfab@kalimero.tijl.coosemans.org> <20180422120521.GS6887@kib.kiev.ua> <20180422151500.1608af96@kalimero.tijl.coosemans.org> <20180422132855.GU6887@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 14:55:22 -0000 On Sun, 22 Apr 2018 16:28:55 +0300 Konstantin Belousov wrote: > On Sun, Apr 22, 2018 at 03:15:00PM +0200, Tijl Coosemans wrote: >> Thanks for the review. There's just one concern I have. With this patch >> the bufspace_daemon threads appear to shutdown after the buf_daemon and >> after the syncer because the event handlers are registered later. Are >> there any dependencies between these processes that require the bufspace >> threads to be stopped earlier? > > I think for correctness bufdaemon must stop after the syncer, since syncer > operation can cause a situation where bufdaemon help is needed to proceed. > Other than this, the stop order is irrelevant, because after syncer > finished, there should be no any further filesystem activity. A quick way to do that would be to use SHUTDOWN_PRI_LAST + 100 for the event handlers in the patch, like shutdown_conf in kern_shutdown.c already does. Is that acceptable here as well? From owner-freebsd-current@freebsd.org Sun Apr 22 15:04:09 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0497DFA69DE for ; Sun, 22 Apr 2018 15:04:09 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5D59C8683F; Sun, 22 Apr 2018 15:04:08 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w3MF3vct020882 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 22 Apr 2018 18:04:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w3MF3vct020882 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w3MF3vCg020881; Sun, 22 Apr 2018 18:03:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 22 Apr 2018 18:03:57 +0300 From: Konstantin Belousov To: Tijl Coosemans Cc: jeff@FreeBSD.org, "freebsd-current@freebsd.org" Subject: Re: i386 hangs during halt "vnodes remaining... 0 time out" Message-ID: <20180422150357.GV6887@kib.kiev.ua> References: <20180421234934.10d7dfab@kalimero.tijl.coosemans.org> <20180422120521.GS6887@kib.kiev.ua> <20180422151500.1608af96@kalimero.tijl.coosemans.org> <20180422132855.GU6887@kib.kiev.ua> <20180422165513.711579c3@kalimero.tijl.coosemans.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180422165513.711579c3@kalimero.tijl.coosemans.org> User-Agent: Mutt/1.9.5 (2018-04-13) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2018 15:04:09 -0000 On Sun, Apr 22, 2018 at 04:55:13PM +0200, Tijl Coosemans wrote: > On Sun, 22 Apr 2018 16:28:55 +0300 Konstantin Belousov wrote: > > On Sun, Apr 22, 2018 at 03:15:00PM +0200, Tijl Coosemans wrote: > >> Thanks for the review. There's just one concern I have. With this patch > >> the bufspace_daemon threads appear to shutdown after the buf_daemon and > >> after the syncer because the event handlers are registered later. Are > >> there any dependencies between these processes that require the bufspace > >> threads to be stopped earlier? > > > > I think for correctness bufdaemon must stop after the syncer, since syncer > > operation can cause a situation where bufdaemon help is needed to proceed. > > Other than this, the stop order is irrelevant, because after syncer > > finished, there should be no any further filesystem activity. > > A quick way to do that would be to use SHUTDOWN_PRI_LAST + 100 for the > event handlers in the patch, like shutdown_conf in kern_shutdown.c > already does. Is that acceptable here as well? I do not see why not. From owner-freebsd-current@freebsd.org Mon Apr 23 01:40:39 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57F24FB0A83; Mon, 23 Apr 2018 01:40:39 +0000 (UTC) (envelope-from grog@lemis.com) Received: from www.lemis.com (www.lemis.com [208.86.226.86]) by mx1.freebsd.org (Postfix) with ESMTP id 012876DE4D; Mon, 23 Apr 2018 01:40:38 +0000 (UTC) (envelope-from grog@lemis.com) Received: from eureka.lemis.com (lemis.com [192.109.197.81]) by www.lemis.com (Postfix) with ESMTP id 1A59F1B72837; Mon, 23 Apr 2018 01:40:32 +0000 (UTC) Received: by eureka.lemis.com (Postfix, from userid 1004) id 00EDF4494B5; Mon, 23 Apr 2018 11:40:30 +1000 (AEST) Date: Mon, 23 Apr 2018 11:40:30 +1000 From: Greg 'groggy' Lehey To: Mariusz Zaborski Cc: FreeBSD Current , freebsd-desktop@freebsd.org Subject: Re: Nvidia issue with CURRENT Message-ID: <20180423014030.GG31055@eureka.lemis.com> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Az4VpBrmI9+OyhK/" Content-Disposition: inline In-Reply-To: Organization: The FreeBSD Project Phone: +61-3-5346-1370, +61-3-5309-0418 Mobile: 0401 265 606. Use only as instructed. WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 01:40:39 -0000 --Az4VpBrmI9+OyhK/ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Sunday, 22 April 2018 at 12:42:37 +0200, Mariusz Zaborski wrote: > Hello, > > I upgraded my FreeBSD to CURRENT and nvidia-drvier-390.48. But it's > stop working. > I tried also nvidia-driver-390.25 without luck as well. Yes, I've had this trouble as well with -STABLE. It happened some time in the February/March time frame. See http://www.lemis.com/grog/diary-mar2018.php#D-20180324-031830. I haven't reported it yet because I had intended to try the latest version of the driver. At the time that was 390.42, but now it's 390.48. You might like to try that (see http://www.nvidia.com/object/unix.html). > I'm attaching also Xorg log. This seems to have got lost. Greg -- Sent from my desktop computer. Finger grog@FreeBSD.org for PGP public key. See complete headers for address and phone numbers. This message is digitally signed. If your Microsoft mail program reports problems, please read http://lemis.com/broken-MUA --Az4VpBrmI9+OyhK/ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAlrdOY4ACgkQIubykFB6QiOFUACfeprz6lb5pRkgTMjCgkTpMsHO /a4An2QBfj/456mHOFyegwO+yA0+d7x6 =daYu -----END PGP SIGNATURE----- --Az4VpBrmI9+OyhK/-- From owner-freebsd-current@freebsd.org Mon Apr 23 03:21:53 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 985F7FB698B; Mon, 23 Apr 2018 03:21:53 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ua0-x229.google.com (mail-ua0-x229.google.com [IPv6:2607:f8b0:400c:c08::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C5F180708; Mon, 23 Apr 2018 03:21:53 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by mail-ua0-x229.google.com with SMTP id g10so9246383ual.6; Sun, 22 Apr 2018 20:21:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=K5qaCrgA3R+tDrChs7KPMUmaWL8CI1ITy2t0ccUNTfk=; b=nABtsuhfpVn/rEDLFp/AAgLVYdF7u0iGjssTyPSA65iGJPJdebpStT27exVMsxtiej wm6cRHaKjI5il4hvm0B9dnjjLT8+8IrNmWWNva9FxTkz+c0f64mYBmCb69vSlhYWy7cw PYQ/YZW4QQpmXVDsZMr7ayvsw2VAQHcQFViEZjUK5fMw5x7+GIL+547cMSSkS8m3G1RP Ozj1gl+IrwbP6yqfrg3S/SqC/bfBjrMj3Gj1k249Dd3RDEURNWunQP+2IWyXhlAZaZBf IhWxwkFSA3vC/3RnU1gVxjEZMdNoK7CRts7pornLjXB468RpGwpBWGKlsER2UU1Z25ZQ iM+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=K5qaCrgA3R+tDrChs7KPMUmaWL8CI1ITy2t0ccUNTfk=; b=VN541t16asDVilVMJTuqPf3/Sf47KMUliMLOJX3+ZHT6V3nyXcRFFG9j9dwhdJHeZS x17nyo/dcrtfcohNo4nzAMbgeb9KZg7zMXjQ27OnCI97b0x1jU1MgVT2vSadezQgQpIw cuMtGPqte2stO5XSoKjeiHStUhMYkt74aWN4M5MmTIaOXezzTWcYwJ3riYmtSYvlLUo2 0WVV48Cvjg0Br6vXbWCy9Y3PUa+qiZ5qmcg/xtRbKPZ7I5mvCjQwhg69Z0+rJyUOFCOV iF6jMlo6p49v9XazXHaHTstnEapM8KeUGBf4cl3bDeelKGisc0lT83DXDd76Pe3FNTtZ 2Isw== X-Gm-Message-State: ALQs6tBzTNbJfYUpwTvtZFyEVJ/6hUcXevKoGUhrCx3h6dprbT6DF1/7 zCMmFa15cp85SxYIlEMZ6wFvsaYh+44q/77+aEmId/9t X-Google-Smtp-Source: AIpwx4+Mc563sMm29uSFHSKe4IzIYOndNc1s1CcBx3L9hsX/SMPjeFbP3AFo4IHM/bKEzHKs46tIx8/2jxw6W7E8wGc= X-Received: by 10.159.55.46 with SMTP id z43mr14317326uad.117.1524453712384; Sun, 22 Apr 2018 20:21:52 -0700 (PDT) MIME-Version: 1.0 Sender: kob6558@gmail.com Received: by 10.103.81.15 with HTTP; Sun, 22 Apr 2018 20:21:51 -0700 (PDT) In-Reply-To: <20180423014030.GG31055@eureka.lemis.com> References: <20180423014030.GG31055@eureka.lemis.com> From: Kevin Oberman Date: Sun, 22 Apr 2018 20:21:51 -0700 X-Google-Sender-Auth: HZjvlDVegHtiOmImGkQAQFfBDwk Message-ID: Subject: Re: Nvidia issue with CURRENT To: "Greg 'groggy' Lehey" Cc: Mariusz Zaborski , FreeBSD Current , freebsd-desktop@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 03:21:53 -0000 On Sun, Apr 22, 2018 at 6:40 PM, Greg 'groggy' Lehey wrote: > On Sunday, 22 April 2018 at 12:42:37 +0200, Mariusz Zaborski wrote: > > > I'm attaching also Xorg log. > > This seems to have got lost. > > Greg Please be sure that attachments are typed "text/plain". I believe all other MIME type are removed for security reasons. I also believe that text/SOME CHARACTER-SET will also be removed as they can be abused. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 From owner-freebsd-current@freebsd.org Mon Apr 23 07:00:59 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27D22F9B5A2; Mon, 23 Apr 2018 07:00:59 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 816D77025D; Mon, 23 Apr 2018 07:00:58 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from freyja.zeit4.iv.bundesimmobilien.de ([87.138.105.249]) by mail.gmx.com (mrgmx102 [212.227.17.168]) with ESMTPSA (Nemesis) id 0LnlmV-1eTlqt0Dl9-00hyyD; Mon, 23 Apr 2018 09:00:44 +0200 Date: Mon, 23 Apr 2018 09:00:33 +0200 From: "O. Hartmann" To: Mariusz Zaborski Cc: Tommi Pernila , FreeBSD Current , freebsd-desktop@freebsd.org Subject: Re: Nvidia issue with CURRENT Message-ID: <20180423090033.70a5ed85@freyja.zeit4.iv.bundesimmobilien.de> In-Reply-To: References: Organization: Walstatt MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K1:G/8mzDZ4iJ9xuicti1MRrRhqq2cEYoLGxPRObl4MUHYtZt/YTim ZrivWPNX1S7mwNLrQs5NnCJAI5zvD8cVXeKqCNKj6Bs2ynhJuSt+eeahwgG6g1SCnoR/GjU 4XGfeoyGDV9vknoZ+j7zkioW8y9tHiMH6h1rfOtGPT8GTTMyR/JC+qzgCv8HkjYkjUfWaqf rwT6ugX+iNxquY45RfpbQ== X-UI-Out-Filterresults: notjunk:1;V01:K0:inGzY+k/Zo0=:++4VFHV7GKAFues67FTP+/ kRypgaMEVdbbzA4n1L3aSVrjUfVGtoFGhl0unLDApoSNujScpGnACIrXyc9hSbuDGLiHHL64g gEwmcXs2D0dVkCFeJRSh1hqTkaq/Ns2LQYdCH0qe7OA0zKm4ljl6srHSJB8zyhDZYVJFLpj61 F2G5TxhPgSWk4g0sg4JvLDiUt+wy2eh/9AO4o7RfnBjPVYEXX2f9q2bo3HVId4hMsOIaJuyl4 9jHshYnv8Mc5WFHNP9LkLMmlhmg8NUEiZLmEhGeKoiKeIsoAcf/bdBYkjazx38uynC0mXoiP5 geYGksIKc9w3cAkh49KDG0DQNRy+2J7O3lGe3++7OHW+cKaNBbZ0bM/h5dxupBX086xp0XVEP 1KmQJIMgd1NwlXQzyxj4nniVSWUiXV+Zh4WpWQ4FEQ5enol1caSS34l4oULUiHrHgRXP9jBh6 v/XpZi4DwdEEj38s4j+3dYC85N6IU27wZnEJLk3851KT0da7bDpCjwhhKifw5iUDVMcJKoO3T EDpOGFwLDVGnweVTEVGsM/FeOj37QMUW8Mg6cbn0pBOKXwjHzOXIguTf5tgy42eF8Wdwy1wR9 ix62AYcWUSGAoFRAcgya36UyNvceM/td4h/YSFBtKt+h/qZZlu+5NigUl0tItQ04/+w8ASFri moALq9QoEiPZR3Jb0efLzWvGIkMgy37AukRO9uwvupjgdZhtD4bMjEOfTE7FyYxLUYz6aqGgt Hrpakel7lOAIhCmiree7/WJZ1V/eKpr/TKIxx5ZjvOs5KzLyep6jmRrpS9xLTYcM2sa4g6ZYB XrBmLohABit9wlXzujrM5ESkzU/XSFMRaTWdTHU/nUJH6t2u6o= X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 07:00:59 -0000 On Sun, 22 Apr 2018 14:38:55 +0200 Mariusz Zaborski wrote: > Hi, > > Normally I build my CURRENT by myself from Xorg - r332861. > But I also tried latest SNAPSHOT. > > Thanks, > Mariusz All my boxes running with nVidia hardware running most recent CURRENT (compiled this morning on an almost daily basis) and I'm using the lates official driver available from nVidia, 390.48. It happens to be as a natural byproduct of CURRENT that very often the kernel module of the nVidia driver is out of sync so i made it a habit to recompile the module from sources whenever I recompile/install a kernel. In /etc/src.conf , therefore you should add something similar to (like I added to mine): PORTS_MODULES= PORTS_MODULES+= x11/nvidia-driver PORTS_MODULES+= emulators/virtualbox-ose-kmod This is one of the great advantages of having an operating system which you can compile yourself. Regards, oh > > On 22 April 2018 at 14:24, Tommi Pernila wrote: > > Hi, > > > > are you running which version of CURRENT? > > E.g. Some snapshot or did you compile from source? > > > > -Tommi > > > > On Sun, 22 Apr 2018 at 13.47, Mariusz Zaborski > > wrote: > >> > >> Hello, > >> > >> I upgraded my FreeBSD to CURRENT and nvidia-drvier-390.48. But it's > >> stop working. > >> I tried also nvidia-driver-390.25 without luck as well. > >> > >> I have loaded nvidia-modeset.ko . > >> > >> While I'm rebooting my machine its also core dumping: > >> https://people.freebsd.org/~oshogbo/nvidia-mail.png . > >> I'm attaching also Xorg log. > >> > >> Is this a known issue? > >> > >> Thanks, > >> Mariusz > >> _______________________________________________ > >> freebsd-current@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-current > >> To unsubscribe, send any mail to > >> "freebsd-current-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" From owner-freebsd-current@freebsd.org Mon Apr 23 07:47:26 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A1D2BFA34C7 for ; Mon, 23 Apr 2018 07:47:26 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3F6FF77E48 for ; Mon, 23 Apr 2018 07:47:25 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (220-253-154-11.dyn.iinet.net.au [220.253.154.11]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id w3N7kuYg057615 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Mon, 23 Apr 2018 00:46:59 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: SCHED_ULE makes 256Mbyte i386 unusable To: "Rodney W. Grimes" , Rick Macklem Cc: Konstantin Belousov , "freebsd-current@freebsd.org" , George Mitchell , Peter References: <201804221436.w3MEa9DY080702@pdx.rh.CN85.dnsmgr.net> From: Julian Elischer Message-ID: <10a9dcf7-0f50-fd8d-67b4-e5dd8189277b@freebsd.org> Date: Mon, 23 Apr 2018 15:46:51 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <201804221436.w3MEa9DY080702@pdx.rh.CN85.dnsmgr.net> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 07:47:26 -0000 On 22/4/18 10:36 pm, Rodney W. Grimes wrote: >> Konstantin Belousov wrote: >>> On Sat, Apr 21, 2018 at 11:30:55PM +0000, Rick Macklem wrote: >>>> Konstantin Belousov wrote: >>>>> On Sat, Apr 21, 2018 at 07:21:58PM +0000, Rick Macklem wrote: >>>>>> I decided to start a new thread on current related to SCHED_ULE, since I see >>>>>> more than just performance degradation and on a recent current kernel. >>>>>> (I cc'd a couple of the people discussing performance problems in freebsd-stable >>>>>> recently under a subject line of "Re: kern.sched.quantum: Creepy, sadistic scheduler". >>>>>> >>>>>> When testing a pNFS server on a single core i386 with 256Mbytes using a Dec. 2017 >>>>>> current/head kernel, I would see about a 30% performance degradation (elapsed >>>>>> run time for a kernel build over NFSv4.1) when the server kernel was built with >>>>>> options SCHED_ULE >>>>>> instead of >>>>>> options SCHED_4BSD >> So, now that I have decreased the number of nfsd kernel threads to 32, it works >> with both schedulers and with essentially the same performance. (ie. The 30% >> performance degradation has disappeared.) >> >>>>>> Now, with a kernel from a couple of days ago, the >>>>>> options SCHED_ULE >>>>>> kernel becomes unusable shortly after starting testing. >>>>>> I have seen two variants of this: >>>>>> - Became essentially hung. All I could do was ping the machine from the network. >>>>>> - Reported "vm_thread_new: kstack allocation failed >>>>>> and then any attempt to do anything gets "No more processes". >>>>> This is strange. It usually means that you get KVA either exhausted or >>>>> severly fragmented. >>>> Yes. I reduced the number of nfsd threads from 256->32 and the SCHED_ULE >>>> kernel is working ok now. I haven't done enough to compare performance yet. >>>> Maybe I'll post again when I have some numbers. >>>> >>>>> Enter ddb, it should be operational since pings are replied. Try to see >>>>> where the threads are stuck. >>>> I didn't do this, since reducing the number of kernel threads seems to have fixed >>>> the problem. For the pNFS server, the nfsd threads will spawn additional kernel >>>> threads to do proxies to the mirrored DS servers. >>>> >>>>>> with the only difference being a kernel built with >>>>>> options SCHED_4BSD >>>>>> everything works and performs the same as the Dec 2017 kernel. >>>>>> >>>>>> I can try rolling back through the revisions, but it would be nice if someone >>>>>> could suggest where to start, because it takes a couple of hours to build a >>>>>> kernel on this system. >>>>>> >>>>>> So, something has made things worse for a head/current kernel this winter, rick >>>>> There are at least two potentially relevant changes. >>>>> >>>>> First is r326758 Dec 11 which bumped KSTACK_PAGES on i386 to 4. >>>> I've been running this machine with KSTACK_PAGES=4 for some time, so no change. >> W.r.t. Rodney Grimes comments about this (which didn't end up in this messages >> in the thread): >> I didn't see any instability when using KSTACK_PAGES=4 for this until this cropped >> up and seemed to be scheduler related (but not really, it seems). >> I bumped it to KSTACK_PAGES=4 because I needed that for the pNFS Metadata >> Server code. >> >> Yes, NFS does use quite a bit of kernel stack. Unfortunately, it isn't one big >> item getting allocated on the stack, but many moderate sized ones. >> (A part of it is multiple instances of "struct vattr", some buried in "struct nfsvattr", >> that NFS needs to use. I don't think these are large enough to justify malloc/free, >> but it has to use several of them.) >> >> One case I did try fixing was about 6 cases where "struct nfsstate" ended up on >> the stack. I changes the code to malloc/free them and then when testing, to >> my surprise I had a 20% performance hit and shelved the patch. >> Now that I know that the server was running near its limit, I might try this one >> again, to see if the performance hit doesn't occur when the machine has adequate >> memory. If the performance hit goes away, I could commit this, but it wouldn't >> have that much effect on the kstack usage. (It's interesting how this patch ended >> up related to the issue this thread discussed.) > Anything we can do to help relieve KSTACK usage, especially on i386 > is helpfull. These is a thread back quite some time where someone > came up with a compile time static "this functions uses X bytes of > local stack" and a bit of clean up was done. We should persue > this issue further. that was me. use |-Wframe-larger-than||=|¶ and set it to something like 512 bytes > > My experiece with the i386/KSTACK issues was attempting to do installs > from snapshot .iso's, I usually had to change to a custom kernel without > INVARIANTS and WITNESS, or reduce KSTACK to 2 and suffer the small stack > problem (ie, dont use NFS during install). Neither was very pleasant. > > I have found it in practical to run the 4 page KSTACK in production > VM's using i386 due to memory requirements. I run many very lean > i386 VM's with 64MB of memory. I suspect our user base also has > many people doing this, and it would be to our advantage to try > and reduce our kernel stack needs. > > >>>>> Second is r332489 Apr 13, which introduced 4/4G KVA/UVA split. >>>> Could this change have resulted in the system being able to allocate fewer >>>> kernel threads/stacks for some reason? >>> Well, it could, as anything can be buggy. But the intent of the change >>> was to give 4G KVA, and it did. >> Righto. No concern here. I suspect the Dec. 2017 kernel was close to the limit >> (see performance issue that went away, noted above) and any change could >> have pushed it across the line, I think. >> >>>>> Consequences of the first one are obvious, it is much harder to find >>>>> the place to map the stack. Second change, on the other hand, provides >>>>> almost full 4G for KVA and should have mostly compensate for the negative >>>>> effects of the first. >>>>> >>>>> And, I cannot see how changing the scheduler would fix or even affect that >>>>> behaviour. >>>> My hunch is that the system was running near its limit for kernel threads/stacks. >>>> Then, somehow, the timing SCHED_ULE caused resulted in the nfsd trying to get >>>> to a higher peak number of threads and hit the limit. >>>> SCHED_4BSD happened to result in timing such that it stayed just below the >>>> limit and worked. >>>> I can think of a couple of things that might affect this: >>>> 1 - If SCHED_ULE doesn't do the termination of kernel threads as quickly, then >>>> they wouldn't terminate and release their resources before more new ones >>>> are spawned. >>> Scheduler has nothing to do with the threads termination. It might >>> select running threads in a way that causes the undesired pattern to >>> appear which might create some amount of backlog for termination, but >>> I doubt it. >>> >>>> 2 - If SCHED_ULE handles the nfsd threads in a more "bursty" way, then the burst >>>> could try and spawn more mirror DS worker threads at about the same time. >>>> >>>> Anyhow, thanks for the help, rick >> Have a good day, rick >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >> From owner-freebsd-current@freebsd.org Mon Apr 23 07:48:14 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C2198FA35F9 for ; Mon, 23 Apr 2018 07:48:13 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 5AB47785C2 for ; Mon, 23 Apr 2018 07:48:13 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (220-253-154-11.dyn.iinet.net.au [220.253.154.11]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id w3N7lgIn057618 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Mon, 23 Apr 2018 00:47:45 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: SCHED_ULE makes 256Mbyte i386 unusable To: "Rodney W. Grimes" , Rick Macklem Cc: Konstantin Belousov , "freebsd-current@freebsd.org" , George Mitchell , Peter References: <201804221436.w3MEa9DY080702@pdx.rh.CN85.dnsmgr.net> From: Julian Elischer Message-ID: <6f5fbe1e-6da3-c4ed-ddc3-1629ad2d3058@freebsd.org> Date: Mon, 23 Apr 2018 15:47:37 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <201804221436.w3MEa9DY080702@pdx.rh.CN85.dnsmgr.net> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 07:48:14 -0000 On 22/4/18 10:36 pm, Rodney W. Grimes wrote: >> Konstantin Belousov wrote: >>> On Sat, Apr 21, 2018 at 11:30:55PM +0000, Rick Macklem wrote: >>>> Konstantin Belousov wrote: >>>>> On Sat, Apr 21, 2018 at 07:21:58PM +0000, Rick Macklem wrote: >>>>>> I decided to start a new thread on current related to SCHED_ULE, since I see >>>>>> more than just performance degradation and on a recent current kernel. >>>>>> (I cc'd a couple of the people discussing performance problems in freebsd-stable >>>>>> recently under a subject line of "Re: kern.sched.quantum: Creepy, sadistic scheduler". >>>>>> >>>>>> When testing a pNFS server on a single core i386 with 256Mbytes using a Dec. 2017 >>>>>> current/head kernel, I would see about a 30% performance degradation (elapsed >>>>>> run time for a kernel build over NFSv4.1) when the server kernel was built with >>>>>> options SCHED_ULE >>>>>> instead of >>>>>> options SCHED_4BSD >> So, now that I have decreased the number of nfsd kernel threads to 32, it works >> with both schedulers and with essentially the same performance. (ie. The 30% >> performance degradation has disappeared.) >> >>>>>> Now, with a kernel from a couple of days ago, the >>>>>> options SCHED_ULE >>>>>> kernel becomes unusable shortly after starting testing. >>>>>> I have seen two variants of this: >>>>>> - Became essentially hung. All I could do was ping the machine from the network. >>>>>> - Reported "vm_thread_new: kstack allocation failed >>>>>> and then any attempt to do anything gets "No more processes". >>>>> This is strange. It usually means that you get KVA either exhausted or >>>>> severly fragmented. >>>> Yes. I reduced the number of nfsd threads from 256->32 and the SCHED_ULE >>>> kernel is working ok now. I haven't done enough to compare performance yet. >>>> Maybe I'll post again when I have some numbers. >>>> >>>>> Enter ddb, it should be operational since pings are replied. Try to see >>>>> where the threads are stuck. >>>> I didn't do this, since reducing the number of kernel threads seems to have fixed >>>> the problem. For the pNFS server, the nfsd threads will spawn additional kernel >>>> threads to do proxies to the mirrored DS servers. >>>> >>>>>> with the only difference being a kernel built with >>>>>> options SCHED_4BSD >>>>>> everything works and performs the same as the Dec 2017 kernel. >>>>>> >>>>>> I can try rolling back through the revisions, but it would be nice if someone >>>>>> could suggest where to start, because it takes a couple of hours to build a >>>>>> kernel on this system. >>>>>> >>>>>> So, something has made things worse for a head/current kernel this winter, rick >>>>> There are at least two potentially relevant changes. >>>>> >>>>> First is r326758 Dec 11 which bumped KSTACK_PAGES on i386 to 4. >>>> I've been running this machine with KSTACK_PAGES=4 for some time, so no change. >> W.r.t. Rodney Grimes comments about this (which didn't end up in this messages >> in the thread): >> I didn't see any instability when using KSTACK_PAGES=4 for this until this cropped >> up and seemed to be scheduler related (but not really, it seems). >> I bumped it to KSTACK_PAGES=4 because I needed that for the pNFS Metadata >> Server code. >> >> Yes, NFS does use quite a bit of kernel stack. Unfortunately, it isn't one big >> item getting allocated on the stack, but many moderate sized ones. >> (A part of it is multiple instances of "struct vattr", some buried in "struct nfsvattr", >> that NFS needs to use. I don't think these are large enough to justify malloc/free, >> but it has to use several of them.) >> >> One case I did try fixing was about 6 cases where "struct nfsstate" ended up on >> the stack. I changes the code to malloc/free them and then when testing, to >> my surprise I had a 20% performance hit and shelved the patch. >> Now that I know that the server was running near its limit, I might try this one >> again, to see if the performance hit doesn't occur when the machine has adequate >> memory. If the performance hit goes away, I could commit this, but it wouldn't >> have that much effect on the kstack usage. (It's interesting how this patch ended >> up related to the issue this thread discussed.) > Anything we can do to help relieve KSTACK usage, especially on i386 > is helpfull. These is a thread back quite some time where someone > came up with a compile time static "this functions uses X bytes of > local stack" and a bit of clean up was done. We should persue > this issue further. that was me. use |-Wframe-larger-than||=|¶ and set it to something like 512 bytes (obviously you have to make warnings non fatal as well). > > My experiece with the i386/KSTACK issues was attempting to do installs > from snapshot .iso's, I usually had to change to a custom kernel without > INVARIANTS and WITNESS, or reduce KSTACK to 2 and suffer the small stack > problem (ie, dont use NFS during install). Neither was very pleasant. > > I have found it in practical to run the 4 page KSTACK in production > VM's using i386 due to memory requirements. I run many very lean > i386 VM's with 64MB of memory. I suspect our user base also has > many people doing this, and it would be to our advantage to try > and reduce our kernel stack needs. > > >>>>> Second is r332489 Apr 13, which introduced 4/4G KVA/UVA split. >>>> Could this change have resulted in the system being able to allocate fewer >>>> kernel threads/stacks for some reason? >>> Well, it could, as anything can be buggy. But the intent of the change >>> was to give 4G KVA, and it did. >> Righto. No concern here. I suspect the Dec. 2017 kernel was close to the limit >> (see performance issue that went away, noted above) and any change could >> have pushed it across the line, I think. >> >>>>> Consequences of the first one are obvious, it is much harder to find >>>>> the place to map the stack. Second change, on the other hand, provides >>>>> almost full 4G for KVA and should have mostly compensate for the negative >>>>> effects of the first. >>>>> >>>>> And, I cannot see how changing the scheduler would fix or even affect that >>>>> behaviour. >>>> My hunch is that the system was running near its limit for kernel threads/stacks. >>>> Then, somehow, the timing SCHED_ULE caused resulted in the nfsd trying to get >>>> to a higher peak number of threads and hit the limit. >>>> SCHED_4BSD happened to result in timing such that it stayed just below the >>>> limit and worked. >>>> I can think of a couple of things that might affect this: >>>> 1 - If SCHED_ULE doesn't do the termination of kernel threads as quickly, then >>>> they wouldn't terminate and release their resources before more new ones >>>> are spawned. >>> Scheduler has nothing to do with the threads termination. It might >>> select running threads in a way that causes the undesired pattern to >>> appear which might create some amount of backlog for termination, but >>> I doubt it. >>> >>>> 2 - If SCHED_ULE handles the nfsd threads in a more "bursty" way, then the burst >>>> could try and spawn more mirror DS worker threads at about the same time. >>>> >>>> Anyhow, thanks for the help, rick >> Have a good day, rick >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >> From owner-freebsd-current@freebsd.org Mon Apr 23 07:49:41 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92B83FA37B7 for ; Mon, 23 Apr 2018 07:49:41 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2F6317946C for ; Mon, 23 Apr 2018 07:49:40 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (220-253-154-11.dyn.iinet.net.au [220.253.154.11]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id w3N7b8nR057578 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Mon, 23 Apr 2018 00:37:13 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: SCHED_ULE makes 256Mbyte i386 unusable To: Rick Macklem , Konstantin Belousov Cc: "freebsd-current@freebsd.org" , George Mitchell , Peter References: <20180421201128.GO6887@kib.kiev.ua> <20180422120241.GR6887@kib.kiev.ua> From: Julian Elischer Message-ID: Date: Mon, 23 Apr 2018 15:37:03 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 07:49:41 -0000 On 22/4/18 9:43 pm, Rick Macklem wrote: > Konstantin Belousov wrote: >> On Sat, Apr 21, 2018 at 11:30:55PM +0000, Rick Macklem wrote: >>> Konstantin Belousov wrote: >>>> On Sat, Apr 21, 2018 at 07:21:58PM +0000, Rick Macklem wrote: >>>>> I decided to start a new thread on current related to SCHED_ULE, since I see >>>>> more than just performance degradation and on a recent current kernel. >>>>> (I cc'd a couple of the people discussing performance problems in freebsd-stable >>>>> recently under a subject line of "Re: kern.sched.quantum: Creepy, sadistic scheduler". >>>>> >>>>> When testing a pNFS server on a single core i386 with 256Mbytes using a Dec. 2017 >>>>> current/head kernel, I would see about a 30% performance degradation (elapsed >>>>> run time for a kernel build over NFSv4.1) when the server kernel was built with >>>>> options SCHED_ULE >>>>> instead of >>>>> options SCHED_4BSD > So, now that I have decreased the number of nfsd kernel threads to 32, it works > with both schedulers and with essentially the same performance. (ie. The 30% > performance degradation has disappeared.) > >>>>> Now, with a kernel from a couple of days ago, the >>>>> options SCHED_ULE >>>>> kernel becomes unusable shortly after starting testing. >>>>> I have seen two variants of this: >>>>> - Became essentially hung. All I could do was ping the machine from the network. >>>>> - Reported "vm_thread_new: kstack allocation failed >>>>> and then any attempt to do anything gets "No more processes". >>>> This is strange. It usually means that you get KVA either exhausted or >>>> severly fragmented. >>> Yes. I reduced the number of nfsd threads from 256->32 and the SCHED_ULE >>> kernel is working ok now. I haven't done enough to compare performance yet. >>> Maybe I'll post again when I have some numbers. >>> >>>> Enter ddb, it should be operational since pings are replied. Try to see >>>> where the threads are stuck. >>> I didn't do this, since reducing the number of kernel threads seems to have fixed >>> the problem. For the pNFS server, the nfsd threads will spawn additional kernel >>> threads to do proxies to the mirrored DS servers. >>> >>>>> with the only difference being a kernel built with >>>>> options SCHED_4BSD >>>>> everything works and performs the same as the Dec 2017 kernel. >>>>> >>>>> I can try rolling back through the revisions, but it would be nice if someone >>>>> could suggest where to start, because it takes a couple of hours to build a >>>>> kernel on this system. >>>>> >>>>> So, something has made things worse for a head/current kernel this winter, rick >>>> There are at least two potentially relevant changes. >>>> >>>> First is r326758 Dec 11 which bumped KSTACK_PAGES on i386 to 4. >>> I've been running this machine with KSTACK_PAGES=4 for some time, so no change. > W.r.t. Rodney Grimes comments about this (which didn't end up in this messages > in the thread): > I didn't see any instability when using KSTACK_PAGES=4 for this until this cropped > up and seemed to be scheduler related (but not really, it seems). > I bumped it to KSTACK_PAGES=4 because I needed that for the pNFS Metadata > Server code. > > Yes, NFS does use quite a bit of kernel stack. Unfortunately, it isn't one big > item getting allocated on the stack, but many moderate sized ones. > (A part of it is multiple instances of "struct vattr", some buried in "struct nfsvattr", > that NFS needs to use. I don't think these are large enough to justify malloc/free, > but it has to use several of them.) > > One case I did try fixing was about 6 cases where "struct nfsstate" ended up on > the stack. I changes the code to malloc/free them and then when testing, to > my surprise I had a 20% performance hit and shelved the patch. you might try using uma. especially setting up a non-freeing zone, where he system allocates what it needs and then just recycles them. (man uma) > Now that I know that the server was running near its limit, I might try this one > again, to see if the performance hit doesn't occur when the machine has adequate > memory. If the performance hit goes away, I could commit this, but it wouldn't have that much effect on the kstack usage. (It's interesting how this patch ended > up related to the issue this thread discussed.) > >>>> Second is r332489 Apr 13, which introduced 4/4G KVA/UVA split. >>> Could this change have resulted in the system being able to allocate fewer >>> kernel threads/stacks for some reason? >> Well, it could, as anything can be buggy. But the intent of the change >> was to give 4G KVA, and it did. > Righto. No concern here. I suspect the Dec. 2017 kernel was close to the limit > (see performance issue that went away, noted above) and any change could > have pushed it across the line, I think. > >>>> Consequences of the first one are obvious, it is much harder to find >>>> the place to map the stack. Second change, on the other hand, provides >>>> almost full 4G for KVA and should have mostly compensate for the negative >>>> effects of the first. >>>> >>>> And, I cannot see how changing the scheduler would fix or even affect that >>>> behaviour. >>> My hunch is that the system was running near its limit for kernel threads/stacks. >>> Then, somehow, the timing SCHED_ULE caused resulted in the nfsd trying to get >>> to a higher peak number of threads and hit the limit. >>> SCHED_4BSD happened to result in timing such that it stayed just below the >>> limit and worked. >>> I can think of a couple of things that might affect this: >>> 1 - If SCHED_ULE doesn't do the termination of kernel threads as quickly, then >>> they wouldn't terminate and release their resources before more new ones >>> are spawned. >> Scheduler has nothing to do with the threads termination. It might >> select running threads in a way that causes the undesired pattern to >> appear which might create some amount of backlog for termination, but >> I doubt it. >> >>> 2 - If SCHED_ULE handles the nfsd threads in a more "bursty" way, then the burst >>> could try and spawn more mirror DS worker threads at about the same time. >>> >>> Anyhow, thanks for the help, rick > Have a good day, rick > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > From owner-freebsd-current@freebsd.org Mon Apr 23 07:51:05 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2E11FFA3BBE; Mon, 23 Apr 2018 07:51:05 +0000 (UTC) (envelope-from grog@lemis.com) Received: from www.lemis.com (www.lemis.com [208.86.226.86]) by mx1.freebsd.org (Postfix) with ESMTP id AAB1E79A79; Mon, 23 Apr 2018 07:51:04 +0000 (UTC) (envelope-from grog@lemis.com) Received: from eureka.lemis.com (lemis.com [192.109.197.81]) by www.lemis.com (Postfix) with ESMTP id F35B11B72837; Mon, 23 Apr 2018 07:51:02 +0000 (UTC) Received: by eureka.lemis.com (Postfix, from userid 1004) id CAC384494B5; Mon, 23 Apr 2018 17:51:01 +1000 (AEST) Date: Mon, 23 Apr 2018 17:51:01 +1000 From: Greg 'groggy' Lehey To: "O. Hartmann" Cc: Mariusz Zaborski , Tommi Pernila , FreeBSD Current , freebsd-desktop@freebsd.org Subject: Re: Nvidia issue with CURRENT Message-ID: <20180423075101.GH31055@eureka.lemis.com> References: <20180423090033.70a5ed85@freyja.zeit4.iv.bundesimmobilien.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="apbmkPN6Hu/1dI3g" Content-Disposition: inline In-Reply-To: <20180423090033.70a5ed85@freyja.zeit4.iv.bundesimmobilien.de> Organization: The FreeBSD Project Phone: +61-3-5346-1370, +61-3-5309-0418 Mobile: 0401 265 606. Use only as instructed. WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 07:51:05 -0000 --apbmkPN6Hu/1dI3g Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Monday, 23 April 2018 at 9:00:33 +0200, O. Hartmann wrote: > On Sun, 22 Apr 2018 14:38:55 +0200 > Mariusz Zaborski wrote: > >> Hi, >> >> Normally I build my CURRENT by myself from Xorg - r332861. >> But I also tried latest SNAPSHOT. > > All my boxes running with nVidia hardware running most recent CURRENT (compiled > this morning on an almost daily basis) and I'm using the lates official driver > available from nVidia, 390.48. > > It happens to be as a natural byproduct of CURRENT that very often > the kernel module of the nVidia driver is out of sync so i made it a > habit to recompile the module from sources whenever I > recompile/install a kernel. As I commented, I've had this on -STABLE as well. My guess is that this is GPU dependent. I'm using an old card: [ 32.251] Current Operating System: FreeBSD teevee.lemis.com 11.1-STABLE FreeBSD 11.1-STABLE #2 r327971: Mon Jan 15 1 0:55:53 AEDT 2018 grog@teevee.lemis.com:/home/obj/eureka/home/src/FreeBSD/svn/stable/11/sys/GENERIC amd64 ... [ 32.763] (II) NVIDIA dlloader X Driver 390.25 Wed Jan 24 19:00:20 PST 2018 ... [ 33.785] (II) NVIDIA(0): NVIDIA GPU GeForce GT 710 (GK208) at PCI:1:0:0 (GPU-0) [ 33.785] (--) NVIDIA(0): Memory: 2097152 kBytes [ 33.785] (--) NVIDIA(0): VideoBIOS: 80.28.b8.00.45 [ 33.785] (II) NVIDIA(0): Detected PCI Express Link width: 8X > In /etc/src.conf , therefore you should add something similar to (like I added > to mine): > > PORTS_MODULES= > PORTS_MODULES+= x11/nvidia-driver > PORTS_MODULES+= emulators/virtualbox-ose-kmod > > This is one of the great advantages of having an operating system which you can > compile yourself. Yes, but this has nothing to do with the bug. Clearly Marisuz and I have the configuration correct, but something has changed in the last few months. Marisuz, as I commented, your log wasn't appended to the message I received. What is your hardware? Greg -- Sent from my desktop computer. Finger grog@FreeBSD.org for PGP public key. See complete headers for address and phone numbers. This message is digitally signed. If your Microsoft mail program reports problems, please read http://lemis.com/broken-MUA --apbmkPN6Hu/1dI3g Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAlrdkGUACgkQIubykFB6QiMAUwCdF5aaHIoxToO7q9ifkOin7lrp PU8AoLLnfQO7cYk+xP+6Xo6PRwd8M17R =Bih8 -----END PGP SIGNATURE----- --apbmkPN6Hu/1dI3g-- From owner-freebsd-current@freebsd.org Mon Apr 23 07:56:21 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9CAD4FA47B9; Mon, 23 Apr 2018 07:56:21 +0000 (UTC) (envelope-from oshogbo.vx@gmail.com) Received: from mail-lf0-x235.google.com (mail-lf0-x235.google.com [IPv6:2a00:1450:4010:c07::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0635A7A631; Mon, 23 Apr 2018 07:56:21 +0000 (UTC) (envelope-from oshogbo.vx@gmail.com) Received: by mail-lf0-x235.google.com with SMTP id i18-v6so13503669lfc.7; Mon, 23 Apr 2018 00:56:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=00A/y+EFu+bjT8zh+VhZz8El71I3yRZwb+wnAARKW9I=; b=LC33XP84/zirfOUa/OkWg+F9f72eFdWslG0oDJH03uTneSOs2pZ+F4dCLCL+GM+RCu O2l05jws4n96l2LAi6KM2aMeGPPEK+0YS1+KLETfqnfEapfOH8Kc9SNSEp8uGfrXO1ix IBbNbiq4PwSEXEzwSLiSAPmAu1kE5nFwxwZDD6VjWZcBg/oU54f/zvzNe2EVDFUbr+EN Cn0BKcUEh9xQvEnDBfr1+PoDe9i6VptIRApC+pyT88NNs0tThXWoS3GFAmg5ho+fN44I W2JPNeJWPL8twbtUwcJ+SWjav4VhXfM1LsLPoVYjhJeiuFmL4XbVJgW+rY0RfQjsT82r 3f9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=00A/y+EFu+bjT8zh+VhZz8El71I3yRZwb+wnAARKW9I=; b=AFyAu4kav0gUFR32kwA/mn1R14TXeDleRvjF+UIeWjVcUX+uR2Nl35iZohRuy8/Sxj K1EjnQxbdlZzwFWa8RuvPxD8kDQc2Zv6FzSPC/vZluowAQ9y+Hri7ObZZWQX9GpfmaPU dmb/D58Nz0Xsp/CYOZXtjWkXCKmCA6tVXGSdlwN/qjJRz+LwFwYRH72LIi25EDERWkcZ L1UnovpYvq1DqUqZPizDjAG7aPHw8yKCtCZBPbIPo2kc8b/L6U4KtqEzkOZnRpFwE0vm CCwgOUrRnNMNYkgc4CzEuI/fxzHJ2SCYqrQGJ6X72M8ifXYDVDWyAM1kO+H7XWuy9oGX SefQ== X-Gm-Message-State: ALQs6tBN17DkSmQKqfBKZ3APHRtDjJ5SjmQNY5H3ieAsrrnrk/eq1HW9 2+1FGKj0TCdwnfek2CM+OLt5BMqF X-Google-Smtp-Source: AIpwx49FQVMOVWPDaDQrUMhSXpDf3PTuq872onTHRmL4S5Z7Wj+PUFwOJbmeDoHjXLj+eW5NwMg2bg== X-Received: by 10.46.64.77 with SMTP id n74mr13421405lja.6.1524470179208; Mon, 23 Apr 2018 00:56:19 -0700 (PDT) Received: from jarvis ([77.79.224.226]) by smtp.gmail.com with ESMTPSA id v79-v6sm2160395lfa.60.2018.04.23.00.56.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 23 Apr 2018 00:56:18 -0700 (PDT) Sender: Mariusz Zaborski Date: Mon, 23 Apr 2018 09:55:40 +0200 From: Mariusz Zaborski To: Greg 'groggy' Lehey Cc: "O. Hartmann" , Tommi Pernila , FreeBSD Current , freebsd-desktop@freebsd.org Subject: Re: Nvidia issue with CURRENT Message-ID: <20180423075540.GA11937@jarvis> References: <20180423090033.70a5ed85@freyja.zeit4.iv.bundesimmobilien.de> <20180423075101.GH31055@eureka.lemis.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="CE+1k2dSO48ffgeK" Content-Disposition: inline In-Reply-To: <20180423075101.GH31055@eureka.lemis.com> User-Agent: Mutt/1.8.3 (2017-05-23) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 07:56:21 -0000 --CE+1k2dSO48ffgeK Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Apr 23, 2018 at 05:51:01PM +1000, Greg 'groggy' Lehey wrote: > On Monday, 23 April 2018 at 9:00:33 +0200, O. Hartmann wrote: > > On Sun, 22 Apr 2018 14:38:55 +0200 > > Mariusz Zaborski wrote: > > > >> Hi, > >> > >> Normally I build my CURRENT by myself from Xorg - r332861. > >> But I also tried latest SNAPSHOT. > > > > All my boxes running with nVidia hardware running most recent CURRENT (= compiled > > this morning on an almost daily basis) and I'm using the lates official= driver > > available from nVidia, 390.48. > > > > It happens to be as a natural byproduct of CURRENT that very often > > the kernel module of the nVidia driver is out of sync so i made it a > > habit to recompile the module from sources whenever I > > recompile/install a kernel. >=20 > As I commented, I've had this on -STABLE as well. >=20 > My guess is that this is GPU dependent. I'm using an old card: >=20 > [ 32.251] Current Operating System: FreeBSD teevee.lemis.com 11.1-STAB= LE FreeBSD 11.1-STABLE #2 r327971: Mon Jan 15 1 > 0:55:53 AEDT 2018 grog@teevee.lemis.com:/home/obj/eureka/home/src/Fre= eBSD/svn/stable/11/sys/GENERIC amd64 > ... > [ 32.763] (II) NVIDIA dlloader X Driver 390.25 Wed Jan 24 19:00:20 P= ST 2018 > ... > [ 33.785] (II) NVIDIA(0): NVIDIA GPU GeForce GT 710 (GK208) at PCI:1:0= :0 (GPU-0) > [ 33.785] (--) NVIDIA(0): Memory: 2097152 kBytes > [ 33.785] (--) NVIDIA(0): VideoBIOS: 80.28.b8.00.45 > [ 33.785] (II) NVIDIA(0): Detected PCI Express Link width: 8X >=20 > > In /etc/src.conf , therefore you should add something similar to (like = I added > > to mine): > > > > PORTS_MODULES=3D > > PORTS_MODULES+=3D x11/nvidia-driver > > PORTS_MODULES+=3D emulators/virtualbox-ose-kmod > > > > This is one of the great advantages of having an operating system which= you can > > compile yourself. >=20 > Yes, but this has nothing to do with the bug. Clearly Marisuz and I > have the configuration correct, but something has changed in the last > few months. Yea this is a known issue so I rebuild nvidia-driver. I'm just not sure if this is a problem with kernel or with the driver itsel= f. > Marisuz, as I commented, your log wasn't appended to the message I > received. What is your hardware? https://people.freebsd.org/~oshogbo/Xorg.0.log NVIDIA GPU GeForce GTX 1050 Ti Thanks, --=20 Mariusz Zaborski oshogbo//vx | http://oshogbo.vexillium.org FreeBSD commiter | https://freebsd.org Software developer | http://wheelsystems.com If it's not broken, let's fix it till it is!!1 --CE+1k2dSO48ffgeK Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEkD1x0xkJXVVY1Gwf38KEGuLGxWQFAlrdkXoACgkQ38KEGuLG xWSOhRAAjRcD0PiC8F666cnD4wiVMMLdZuO2qFerRYLH9Vi5F4WlJ9SVVmLy/oOY TvZV2gkCRxJi8sHHM16oWmyq92S0kYQ9DoVXrwM/AOYrkLF/HXeqJWU1rk12GCJp QpLSIZMr2Si3GJgocZ3Fn4SQ7o4Oj3b/jABbOhcm0MSfgNNEnxWyx3AsV+XSPQwf 35VOthjbc6U1Ol3sjAuaN200AWFEXCbPoVB/ZrczXwOmmaA9dtMoGCPUQS1mjFPr Zk3v8ULYpNGWXRYj+MtYKrLFDsCiJTk9fP+dBEhnLXv+ZQwyX5S/pGHNM/OmbdV9 x2YAcGYySbNYMLukFN4J0Z8iU4Y+WlHxppsUEg7crScljo9NJiEtReg4JaHmeqTB 8F8H2+jGcQ3rprIxKir9Ip04FN0Exqz5t0OLzx3iyEzE2XIOvjMD8PPORK+wh46z xWQR/N3M4CrVWoYzQLbqttHK71FzpbOeLaT7Ao0+Pk28nzDd0ligiP7oUnOwCqdH zjEGy7+t2oBXF08I6/hqU6fFKew28+DIvWWYiK56m3GwXHp1N7K+JivjSM4nSPKo 0TwtGFBmKHIsR3J/6m3mcPST04kb43fvElQQjhdvcgq7PRiZfpg2YYUrCrWZQ+hu SMUSRZJ0wl+UZNdVMdK1ot8E01qDtbMFlpuCtT+R4ndNy5zR3oY= =IDoE -----END PGP SIGNATURE----- --CE+1k2dSO48ffgeK-- From owner-freebsd-current@freebsd.org Mon Apr 23 08:17:45 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9A92CFA6783; Mon, 23 Apr 2018 08:17:45 +0000 (UTC) (envelope-from grog@lemis.com) Received: from www.lemis.com (www.lemis.com [208.86.226.86]) by mx1.freebsd.org (Postfix) with ESMTP id 3F1577F2F8; Mon, 23 Apr 2018 08:17:44 +0000 (UTC) (envelope-from grog@lemis.com) Received: from eureka.lemis.com (lemis.com [192.109.197.81]) by www.lemis.com (Postfix) with ESMTP id E9C2E1B72837; Mon, 23 Apr 2018 08:17:43 +0000 (UTC) Received: by eureka.lemis.com (Postfix, from userid 1004) id A9ED04494B5; Mon, 23 Apr 2018 18:17:42 +1000 (AEST) Date: Mon, 23 Apr 2018 18:17:42 +1000 From: Greg 'groggy' Lehey To: Mariusz Zaborski Cc: "O. Hartmann" , Tommi Pernila , FreeBSD Current , freebsd-desktop@freebsd.org Subject: Re: Nvidia issue with CURRENT Message-ID: <20180423081742.GI31055@eureka.lemis.com> References: <20180423090033.70a5ed85@freyja.zeit4.iv.bundesimmobilien.de> <20180423075101.GH31055@eureka.lemis.com> <20180423075540.GA11937@jarvis> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="KjSGHOmKKB2VUiQn" Content-Disposition: inline In-Reply-To: <20180423075540.GA11937@jarvis> Organization: The FreeBSD Project Phone: +61-3-5346-1370, +61-3-5309-0418 Mobile: 0401 265 606. Use only as instructed. WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 08:17:45 -0000 --KjSGHOmKKB2VUiQn Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Monday, 23 April 2018 at 9:55:40 +0200, Mariusz Zaborski wrote: > On Mon, Apr 23, 2018 at 05:51:01PM +1000, Greg 'groggy' Lehey wrote: >> On Monday, 23 April 2018 at 9:00:33 +0200, O. Hartmann wrote: >>> On Sun, 22 Apr 2018 14:38:55 +0200 Mariusz Zaborski wrote: >>> In /etc/src.conf , therefore you should add something similar to (like I added >>> to mine): >>> >>> PORTS_MODULES= >>> PORTS_MODULES+= x11/nvidia-driver >>> PORTS_MODULES+= emulators/virtualbox-ose-kmod >>> >>> This is one of the great advantages of having an operating system which you can >>> compile yourself. >> >> Yes, but this has nothing to do with the bug. Clearly Marisuz and I >> have the configuration correct, but something has changed in the last >> few months. > > Yea this is a known issue so I rebuild nvidia-driver. > I'm just not sure if this is a problem with kernel or with the > driver itself. Almost by definition, it's a driver issue. Something in the kernel has changed which makes it no longer work. >> Marisuz, as I commented, your log wasn't appended to the message I >> received. What is your hardware? > > https://people.freebsd.org/~oshogbo/Xorg.0.log A brief scan doesn't show anything very similar to my issues. I'll look again tomorrow when I have time. Did you try the most recent driver? Greg -- Sent from my desktop computer. Finger grog@FreeBSD.org for PGP public key. See complete headers for address and phone numbers. This message is digitally signed. If your Microsoft mail program reports problems, please read http://lemis.com/broken-MUA --KjSGHOmKKB2VUiQn Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iEYEARECAAYFAlrdlqYACgkQIubykFB6QiOJXQCdE0qQOIi9NB+JKtazNBfhdpaU iEYAniicit6XUXrSmMw0ShVtu6bOlR5P =oUDd -----END PGP SIGNATURE----- --KjSGHOmKKB2VUiQn-- From owner-freebsd-current@freebsd.org Mon Apr 23 09:22:44 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B6E4FAB18B; Mon, 23 Apr 2018 09:22:44 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: from mail-lf0-f67.google.com (mail-lf0-f67.google.com [209.85.215.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 11F588C075; Mon, 23 Apr 2018 09:22:43 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: by mail-lf0-f67.google.com with SMTP id g203-v6so13844661lfg.11; Mon, 23 Apr 2018 02:22:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fa2uDLHdcjaR7ROJiiyEpaoTKyr7VGsrsJc72ZFYuXs=; b=FnlfTdXbYZySO0b0Lc66SLVAOSvDSE/Zzanlks9PkciYGtfmf9m3yxoDhh+uEwFP4X aiB+d8L7GDSxCKvlC0OS7hEwQb23JRhW1yTAhvTNKRaw5F2Dqyhf6/rHx0XI+ZS3KyP+ xghlmuHnH2B94TEnEgFSx8rAaNDDUVsZqK9DSDn033RgCyVxHRWurDfGLURZ960CANtA XT7xZuf7XOJfn7hQp0aNxnjwOg/iew/LHGvQWvwDdKLcc6d3pEjfQuxDLJ95QW2gs7m9 PH9gjgOILOpoTo0wBWFq6v6I/4VnxxUhky5hzC4TMpUoBLDcsO75LA1KbafQwMMYessS O+KA== X-Gm-Message-State: ALQs6tCHUurzgC9YZQ1k1S43BdeXGQWiGbV/9Vmpim6G7rqRgfP3Tqs5 TeppvHnTYoCQJ04YEpC4cldlFkpb/qE= X-Google-Smtp-Source: AB8JxZqSKz1galSCo5ZZ55gJ1kRaBoyq1EO7Ha5+GSNIMVaXtaFmoBlr8M+lrVm5cb7V+Mr9C2kxEQ== X-Received: by 2002:a19:4ed8:: with SMTP id u85-v6mr7941768lfk.98.1524473552077; Mon, 23 Apr 2018 01:52:32 -0700 (PDT) Received: from oxy (fw-gw-atm.mimuw.edu.pl. [193.0.96.15]) by smtp.gmail.com with ESMTPSA id q22sm2255096ljj.68.2018.04.23.01.52.31 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 23 Apr 2018 01:52:31 -0700 (PDT) Date: Mon, 23 Apr 2018 10:53:26 +0200 From: Mateusz Piotrowski <0mp@FreeBSD.org> To: "O. Hartmann" Cc: Mariusz Zaborski , Tommi Pernila , FreeBSD Current , freebsd-desktop@freebsd.org Subject: Re: Nvidia issue with CURRENT Message-ID: <20180423105326.63b32e2b@oxy> In-Reply-To: <20180423090033.70a5ed85@freyja.zeit4.iv.bundesimmobilien.de> References: <20180423090033.70a5ed85@freyja.zeit4.iv.bundesimmobilien.de> X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd12.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 09:22:44 -0000 On Mon, 23 Apr 2018 09:00:33 +0200 "O. Hartmann" wrote: >In /etc/src.conf , therefore you should add something similar to (like >I added to mine): > >PORTS_MODULES= >PORTS_MODULES+= x11/nvidia-driver >PORTS_MODULES+= emulators/virtualbox-ose-kmod Shouldn't it go into make.conf(5)? That's what the manual suggests. From owner-freebsd-current@freebsd.org Mon Apr 23 12:22:08 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A75DFFB7A68; Mon, 23 Apr 2018 12:22:08 +0000 (UTC) (envelope-from oshogbo.vx@gmail.com) Received: from mail-lf0-x244.google.com (mail-lf0-x244.google.com [IPv6:2a00:1450:4010:c07::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1B1C66DDA0; Mon, 23 Apr 2018 12:22:08 +0000 (UTC) (envelope-from oshogbo.vx@gmail.com) Received: by mail-lf0-x244.google.com with SMTP id b23-v6so14669314lfg.4; Mon, 23 Apr 2018 05:22:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=SSzkvNE2rME9/g7h4qOZEhCeuJ1hTKbPXURgfraDcA8=; b=EsLmk+TwFn0EEHHUvjC4DPzXuTyw0nLJ7YsFl7L81+e3tIelPYcq+VdWY0fwHiCWKl XzEEX1x9aY9VSaEDpdYtXEYBLtsX4gRmVPPQZ4RHyE22d5IoMfulLAaQv6PKMXR7fMvM 7/0j1tC2STLuIMSZ+hqSJoI1xQVKRVC+mJyJHMtTswdTCSnfTy5VUpCTa+9j0dc7xWuB nfA0gOfdYGzpXUCwTMI+ZHXL83Foh58Cf4UPs74nPbLyr85JauB0wdVaI5EabdxZIC83 v1e/PAMSlIEcPDaRBOjOkfo6zUDTRdeDFG1A8Jvx+g+DSDEdbQJyzVwQHg56XTtA2ctI Mygw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=SSzkvNE2rME9/g7h4qOZEhCeuJ1hTKbPXURgfraDcA8=; b=j7E9zzM4ea4vVVSgzFkfpNoSkc7YLtujDXDgFq8b1J7e3KatCpBCrTyes7cltgl62o x8Z2QGP2YFJ11ARTsSowbTCQpcszsDlh8/ih9ljfIMAtY/xt8WUCbVy08MirKrArTaCo r2Oi35fXoG4RyqNl1uEkf3T20GR/aqLzCRhKcCBL+BXFFTTbU9lbOxjsr40EDTatyWCW +9WSPwOZkICL/DrDWT0/pHqFIPo0xtOLiCp3tQKjylOX4LBQPaQm3TBeKvq53DZvkwWq 8h4R62LAJtzFmD4hRrMtAOrruJQA0V37gRlbHBysL4sG/eHCbiKCzgzMs5EuxdtZqiSN y5Ow== X-Gm-Message-State: ALQs6tCDEVZTcStQIjI+q16AnbWap4k16WrOdCrz8z2PqKmyw8u7VhPZ 0KQtJfYxnjNjEF1G3WUE7lH61jTT X-Google-Smtp-Source: AB8JxZq1EdqOq4esmD3Y3yCX5NK+oU66RQc8VViUVNoMY1BwicnLAQoPIDDAoPAjgt6QxIhQxvuhmA== X-Received: by 2002:a19:1398:: with SMTP id 24-v6mr9127768lft.106.1524486126318; Mon, 23 Apr 2018 05:22:06 -0700 (PDT) Received: from jarvis ([77.79.224.226]) by smtp.gmail.com with ESMTPSA id s3sm2259437ljd.75.2018.04.23.05.22.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 23 Apr 2018 05:22:05 -0700 (PDT) Sender: Mariusz Zaborski Date: Mon, 23 Apr 2018 14:21:27 +0200 From: Mariusz Zaborski To: Greg 'groggy' Lehey Cc: "O. Hartmann" , Tommi Pernila , FreeBSD Current , freebsd-desktop@freebsd.org Subject: Re: Nvidia issue with CURRENT Message-ID: <20180423122127.GA19781@jarvis> References: <20180423090033.70a5ed85@freyja.zeit4.iv.bundesimmobilien.de> <20180423075101.GH31055@eureka.lemis.com> <20180423075540.GA11937@jarvis> <20180423081742.GI31055@eureka.lemis.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180423081742.GI31055@eureka.lemis.com> User-Agent: Mutt/1.8.3 (2017-05-23) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 12:22:08 -0000 On Mon, Apr 23, 2018 at 06:17:42PM +1000, Greg 'groggy' Lehey wrote: > On Monday, 23 April 2018 at 9:55:40 +0200, Mariusz Zaborski wrote: > > On Mon, Apr 23, 2018 at 05:51:01PM +1000, Greg 'groggy' Lehey wrote: > >> On Monday, 23 April 2018 at 9:00:33 +0200, O. Hartmann wrote: > >>> On Sun, 22 Apr 2018 14:38:55 +0200 Mariusz Zaborski wrote: > >>> In /etc/src.conf , therefore you should add something similar to (like I added > >>> to mine): > >>> > >>> PORTS_MODULES= > >>> PORTS_MODULES+= x11/nvidia-driver > >>> PORTS_MODULES+= emulators/virtualbox-ose-kmod > >>> > >>> This is one of the great advantages of having an operating system which you can > >>> compile yourself. > >> > >> Yes, but this has nothing to do with the bug. Clearly Marisuz and I > >> have the configuration correct, but something has changed in the last > >> few months. > > > > Yea this is a known issue so I rebuild nvidia-driver. > > I'm just not sure if this is a problem with kernel or with the > > driver itself. > > Almost by definition, it's a driver issue. Something in the kernel > has changed which makes it no longer work. > > >> Marisuz, as I commented, your log wasn't appended to the message I > >> received. What is your hardware? > > > > https://people.freebsd.org/~oshogbo/Xorg.0.log > > A brief scan doesn't show anything very similar to my issues. I'll > look again tomorrow when I have time. > > Did you try the most recent driver? If you mean the 390.48, then yes. I didn't see any newer then that. Thanks, -- Mariusz Zaborski oshogbo//vx | http://oshogbo.vexillium.org FreeBSD commiter | https://freebsd.org Software developer | http://wheelsystems.com If it's not broken, let's fix it till it is!!1 From owner-freebsd-current@freebsd.org Mon Apr 23 14:50:58 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F03EFA2306 for ; Mon, 23 Apr 2018 14:50:58 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8315F6DFB6 for ; Mon, 23 Apr 2018 14:50:57 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (220-253-154-11.dyn.iinet.net.au [220.253.154.11]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id w3NEonO0059560 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Mon, 23 Apr 2018 07:50:53 -0700 (PDT) (envelope-from julian@freebsd.org) To: freebsd-current From: Julian Elischer Subject: crash on process exit.. current at about r332467 Message-ID: <9479e941-39be-e6e2-869e-aac475c5e33a@freebsd.org> Date: Mon, 23 Apr 2018 22:50:43 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 14:50:58 -0000 back trace at:  http://www.freebsd.org/~julian/bob-crash.png If anyone wants to take a look.. In the exit syscall, while deallocating a vm object. I haven't see references to a similar crash in the last 10 days or so.. But if it rings any bells... From owner-freebsd-current@freebsd.org Mon Apr 23 15:19:28 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3105FA4159; Mon, 23 Apr 2018 15:19:28 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-vk0-x22f.google.com (mail-vk0-x22f.google.com [IPv6:2607:f8b0:400c:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4431376385; Mon, 23 Apr 2018 15:19:28 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by mail-vk0-x22f.google.com with SMTP id g83so6742408vkc.6; Mon, 23 Apr 2018 08:19:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=NeLQuh/WLHM/DyEv6+sKc+CjtjYpa4nlCApF+re+3IQ=; b=nr3SCuyMjQQJtZIc8PsyUyo53HhanlVq48RvmRh+aDhBJAf6WOrVdHgXzYDmGF75vD J9r9HLpHcRdNt+d+s86dZPlggoF3HGnsBINn2BSSGaYqlLTZ1S9jl7MhGo/ZqW4DPhoH OM5+67UFM0R/N4pX5Ue3PP33ZcqwLz3Nk1jBIJHJ+CqkZEdqdX7nvufh8xIcUPbO3lVK nTWLxqGn/42894AHF+PIekWK1SIm9drTr1ZRo5By5PRUnNCCYIOWhXnSyRXsbguEuXVh A4+MkO3Fg1bZ6Ni2ueREmQt0cwHX4cc4AAKnggBRtET14o2sIKyKXHRuePV4FlVWIiug Y/ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=NeLQuh/WLHM/DyEv6+sKc+CjtjYpa4nlCApF+re+3IQ=; b=NBS/NjIq0BRoilGmgf0N+AqlDbjkVrzKcaD0eMOQ9ZRmcLiKOtNutWAm+/FIpIxQuN urzIHT7truT5f/K2WBKJAvwf5iubj33D4G1TwV70xt/Tmeskkzobqo8yeHvc2lkdFeus NAALjWA6Fv9wxsBaFHAOCdxrTM2miLAJvz7NEavH8KtBGr1jxtegEI+/vH+QEpEIDzpT i8zB1F1E7yDPUob3d01TcQ/uHVG4oksvxKvJvlf3rtb+s04tfOGCnxqxqm3KNLtgwSsf 7/S4mf/1CssqItxF9GSFYinM2FUVYgLyVrFdTzKXGFTqVxYiCeElQVFqdUrXViiAvLzc 6dqQ== X-Gm-Message-State: ALQs6tBkdHcHHDvutaLqpfkHH2xmz4vjYTkmB70Ise4L2szyyARGdPMi mLfY0OEzm12UQvV5rzWDD9Yj3MzFjnluYw+vsg3Rrxzw X-Google-Smtp-Source: AIpwx4+/dHqTH8K+MMbuRni54O7qzYr8Neu9ixls5EqZYhNCsA7kc1ees9zQcu6dCgDkMEvGo8X3QeCcvp33SRpjC/4= X-Received: by 10.31.162.5 with SMTP id l5mr13483943vke.155.1524496766780; Mon, 23 Apr 2018 08:19:26 -0700 (PDT) MIME-Version: 1.0 Sender: kob6558@gmail.com Received: by 10.103.81.15 with HTTP; Mon, 23 Apr 2018 08:19:25 -0700 (PDT) In-Reply-To: <20180423105326.63b32e2b@oxy> References: <20180423090033.70a5ed85@freyja.zeit4.iv.bundesimmobilien.de> <20180423105326.63b32e2b@oxy> From: Kevin Oberman Date: Mon, 23 Apr 2018 08:19:25 -0700 X-Google-Sender-Auth: EHkMt9-OSIbVIp1FnU0cgWA0fos Message-ID: Subject: Re: Nvidia issue with CURRENT To: Mateusz Piotrowski <0mp@freebsd.org> Cc: "O. Hartmann" , Mariusz Zaborski , Tommi Pernila , FreeBSD Current , freebsd-desktop@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 15:19:28 -0000 On Mon, Apr 23, 2018 at 1:53 AM, Mateusz Piotrowski <0mp@freebsd.org> wrote: > On Mon, 23 Apr 2018 09:00:33 +0200 > "O. Hartmann" wrote: > > >In /etc/src.conf , therefore you should add something similar to (like > >I added to mine): > > > >PORTS_MODULES= > >PORTS_MODULES+= x11/nvidia-driver > >PORTS_MODULES+= emulators/virtualbox-ose-kmod > > Shouldn't it go into make.conf(5)? That's what the manual suggests. > Ans the man page should be fixed. By putting it into make.conf, it pollutes the environment of ever make(1) run on the system. (Yes, unlikely to ever cause a problem,) In src.conf it only impacts the builds of the system and ports. This is why src.conf was created a few years ago. In fact, it was originally supposed to only impact the build of the system, but the ports Mk files were modified to pull it in, to, much to my annoyance. I liked being able to modify compile options just for the system without them breaking ports builds. Simple rule... any definition used by make(1) only for system builds belongs in /etc/sec.conf. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 From owner-freebsd-current@freebsd.org Mon Apr 23 22:53:22 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 21623FAD4B9; Mon, 23 Apr 2018 22:53:22 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: from mail-lf0-f51.google.com (mail-lf0-f51.google.com [209.85.215.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 63E2C7C745; Mon, 23 Apr 2018 22:53:21 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: by mail-lf0-f51.google.com with SMTP id d20-v6so17393687lfe.3; Mon, 23 Apr 2018 15:53:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IhvOTZPGbhtV0L82TfYSfvwcL0+QdY+OniABJp34Bp4=; b=GZinFJB3qQ3mWHpBLPJONGXQgqtmzgvtUyDK0/myCOjE1gxI5RZ+gYX1MHXSs5vyOB sKVsVEEXHx4H8TQUR2n/uFEO0s5x1W4Bglc68VunCXfAxCSJcufacYhwU4XRwggKD2ZA nD7n9L/Rwh645Dh8NHsw60c5z/pwM2w46eRq6AqJBADt7KGDs/HQnT+07Wm72heowU1a 5FaG3yPU9x8MXv1CUJrRUG+e9jFm2xfqXiTH0tjdtlDImQ4wQP6b/9HqaJ0ooMfjQ93L 8oJnGw+sFc6AAB8LxuUr9MTzL+K/bRfpefePojdbZdxOxC8BND4PBLNbOV8dZ/dm0wNz xwpA== X-Gm-Message-State: ALQs6tBqw6tY3040fj/QzuH+LGgh1THYPAbXR0RT7TntKfFqcZ+QoMlu u8zHZsCSu8UPN2X3+4IrVVo= X-Google-Smtp-Source: AB8JxZr1akirZbusPny4S7PeWCb/544/YbpM1BLJNvVmAKjdtxXd8/tH2fKSQqy3yQG555yLbcRkYA== X-Received: by 10.46.27.82 with SMTP id b79mr2365354ljb.134.1524523994067; Mon, 23 Apr 2018 15:53:14 -0700 (PDT) Received: from oxy (89-76-8-18.dynamic.chello.pl. [89.76.8.18]) by smtp.gmail.com with ESMTPSA id j85-v6sm2992380lfh.5.2018.04.23.15.53.13 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 23 Apr 2018 15:53:13 -0700 (PDT) Date: Tue, 24 Apr 2018 00:54:07 +0200 From: Mateusz Piotrowski <0mp@FreeBSD.org> To: Kevin Oberman Cc: "O. Hartmann" , Mariusz Zaborski , Tommi Pernila , FreeBSD Current , freebsd-desktop@freebsd.org Subject: Re: Nvidia issue with CURRENT Message-ID: <20180424005407.473fd145@oxy> In-Reply-To: References: <20180423090033.70a5ed85@freyja.zeit4.iv.bundesimmobilien.de> <20180423105326.63b32e2b@oxy> X-Mailer: Claws Mail 3.16.0 (GTK+ 2.24.32; amd64-portbld-freebsd12.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 22:53:22 -0000 On Mon, 23 Apr 2018 08:19:25 -0700 Kevin Oberman wrote: >On Mon, Apr 23, 2018 at 1:53 AM, Mateusz Piotrowski <0mp@freebsd.org> >wrote: > >> On Mon, 23 Apr 2018 09:00:33 +0200 >> "O. Hartmann" wrote: >> >> >In /etc/src.conf , therefore you should add something similar to >> >(like I added to mine): >> > >> >PORTS_MODULES= >> >PORTS_MODULES+= x11/nvidia-driver >> >PORTS_MODULES+= >> >emulators/virtualbox-ose-kmod >> >> Shouldn't it go into make.conf(5)? That's what the manual suggests. >> > >Ans the man page should be fixed. By putting it into make.conf, it >pollutes the environment of ever make(1) run on the system. (Yes, >unlikely to ever cause a problem,) > >In src.conf it only impacts the builds of the system and ports. This >is why src.conf was created a few years ago. In fact, it was >originally supposed to only impact the build of the system, but the >ports Mk files were modified to pull it in, to, much to my annoyance. >I liked being able to modify compile options just for the system >without them breaking ports builds. > >Simple rule... any definition used by make(1) only for system builds >belongs in /etc/sec.conf. Done, I've added this information to make.conf(5). Now it's waiting for a review. https://reviews.freebsd.org/D15177 Thanks! Mateusz From owner-freebsd-current@freebsd.org Tue Apr 24 08:56:08 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A4A3FC0762 for ; Tue, 24 Apr 2018 08:56:08 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 071F2871B7 for ; Tue, 24 Apr 2018 08:56:08 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id BB1FFFC0760; Tue, 24 Apr 2018 08:56:07 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 93ED4FC075F; Tue, 24 Apr 2018 08:56:07 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EF2988718F; Tue, 24 Apr 2018 08:56:06 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w3O8tr2c086842 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 24 Apr 2018 11:55:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w3O8tr2c086842 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w3O8trSY086841; Tue, 24 Apr 2018 11:55:53 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 24 Apr 2018 11:55:53 +0300 From: Konstantin Belousov To: current@freebsd.org Cc: net@freebsd.org, menny@mellanox.com Subject: mlx5(4) jumbo receive Message-ID: <20180424085553.GA6887@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.9.5 (2018-04-13) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2018 08:56:08 -0000 Hello, the patch below is of interest for people who use Mellanox Connect-X/4 and 5 ethernet adapters and configure it for jumbo frames. The patch should improve the system behavior on low or fragmented memory situations. See the commit message for detailed description. Also, the patch should not cause performance degradation for the normal 1500 MTU by keeping the same configuration as currently unpatched driver. I am interested in hearing about regressions (or not) caused by the change. commit f25c0c624177a1ca06ca051b0adb842acb66ec11 Author: Konstantin Belousov Date: Wed Nov 22 20:14:53 2017 +0200 mlx5en: Handle jumbo frames without requiring big clusters. The scatter list is formed by the chunks of MCLBYTES each, and larger than default packets are returned to the stack as the mbuf chain. This is an improvements over the original patch by hanss@mellanox.com which solves some busdma issues and sizes WQEs scatter list to only have the minimal sufficient number of elements. In particular, for the default MTU 1500 bytes the receive queue format is not changed comparing to the unpatched driver, avoiding a reported performance regression. Issue: 1065432 Issue: 1184316 Change-Id: Ib63a5ef55abd6ef1ec9b296e2cef88e4604bbd29 Signed-off-by: Konstantin Belousov diff --git a/sys/dev/mlx5/mlx5_en/en.h b/sys/dev/mlx5/mlx5_en/en.h index b5000c32eaf..ff5ef446e34 100644 --- a/sys/dev/mlx5/mlx5_en/en.h +++ b/sys/dev/mlx5/mlx5_en/en.h @@ -80,8 +80,19 @@ #define MLX5E_PARAMS_DEFAULT_LOG_RQ_SIZE 0xa #define MLX5E_PARAMS_MAXIMUM_LOG_RQ_SIZE 0xe -/* freeBSD HW LRO is limited by 16KB - the size of max mbuf */ +#define MLX5E_MAX_RX_SEGS 7 + +#ifndef MLX5E_MAX_RX_BYTES +#define MLX5E_MAX_RX_BYTES MCLBYTES +#endif + +#if (MLX5E_MAX_RX_SEGS == 1) +/* FreeBSD HW LRO is limited by 16KB - the size of max mbuf */ #define MLX5E_PARAMS_DEFAULT_LRO_WQE_SZ MJUM16BYTES +#else +#define MLX5E_PARAMS_DEFAULT_LRO_WQE_SZ \ + MIN(65535, MLX5E_MAX_RX_SEGS * MLX5E_MAX_RX_BYTES) +#endif #define MLX5E_PARAMS_DEFAULT_RX_CQ_MODERATION_USEC 0x10 #define MLX5E_PARAMS_DEFAULT_RX_CQ_MODERATION_USEC_FROM_CQE 0x3 #define MLX5E_PARAMS_DEFAULT_RX_CQ_MODERATION_PKTS 0x20 @@ -530,6 +541,7 @@ struct mlx5e_rq { struct mtx mtx; bus_dma_tag_t dma_tag; u32 wqe_sz; + u32 nsegs; struct mlx5e_rq_mbuf *mbuf; struct ifnet *ifp; struct mlx5e_rq_stats stats; @@ -795,9 +807,12 @@ struct mlx5e_tx_wqe { struct mlx5e_rx_wqe { struct mlx5_wqe_srq_next_seg next; - struct mlx5_wqe_data_seg data; + struct mlx5_wqe_data_seg data[]; }; +/* the size of the structure above must be power of two */ +CTASSERT(powerof2(sizeof(struct mlx5e_rx_wqe))); + struct mlx5e_eeprom { int lock_bit; int i2c_addr; diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c index c2c4b0d7744..25544e561e3 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c @@ -33,9 +33,12 @@ #ifndef ETH_DRIVER_VERSION #define ETH_DRIVER_VERSION "3.4.1" #endif + char mlx5e_version[] = "Mellanox Ethernet driver" " (" ETH_DRIVER_VERSION ")"; +static int mlx5e_get_wqe_sz(struct mlx5e_priv *priv, u32 *wqe_sz, u32 *nsegs); + struct mlx5e_channel_param { struct mlx5e_rq_param rq; struct mlx5e_sq_param sq; @@ -810,6 +813,11 @@ mlx5e_create_rq(struct mlx5e_channel *c, int wq_sz; int err; int i; + u32 nsegs, wqe_sz; + + err = mlx5e_get_wqe_sz(priv, &wqe_sz, &nsegs); + if (err != 0) + goto done; /* Create DMA descriptor TAG */ if ((err = -bus_dma_tag_create( @@ -819,9 +827,9 @@ mlx5e_create_rq(struct mlx5e_channel *c, BUS_SPACE_MAXADDR, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ - MJUM16BYTES, /* maxsize */ - 1, /* nsegments */ - MJUM16BYTES, /* maxsegsize */ + nsegs * MLX5E_MAX_RX_BYTES, /* maxsize */ + nsegs, /* nsegments */ + nsegs * MLX5E_MAX_RX_BYTES, /* maxsegsize */ 0, /* flags */ NULL, NULL, /* lockfunc, lockfuncarg */ &rq->dma_tag))) @@ -834,23 +842,9 @@ mlx5e_create_rq(struct mlx5e_channel *c, rq->wq.db = &rq->wq.db[MLX5_RCV_DBR]; - if (priv->params.hw_lro_en) { - rq->wqe_sz = priv->params.lro_wqe_sz; - } else { - rq->wqe_sz = MLX5E_SW2MB_MTU(priv->ifp->if_mtu); - } - if (rq->wqe_sz > MJUM16BYTES) { - err = -ENOMEM; + err = mlx5e_get_wqe_sz(priv, &rq->wqe_sz, &rq->nsegs); + if (err != 0) goto err_rq_wq_destroy; - } else if (rq->wqe_sz > MJUM9BYTES) { - rq->wqe_sz = MJUM16BYTES; - } else if (rq->wqe_sz > MJUMPAGESIZE) { - rq->wqe_sz = MJUM9BYTES; - } else if (rq->wqe_sz > MCLBYTES) { - rq->wqe_sz = MJUMPAGESIZE; - } else { - rq->wqe_sz = MCLBYTES; - } wq_sz = mlx5_wq_ll_get_size(&rq->wq); @@ -861,7 +855,11 @@ mlx5e_create_rq(struct mlx5e_channel *c, rq->mbuf = malloc(wq_sz * sizeof(rq->mbuf[0]), M_MLX5EN, M_WAITOK | M_ZERO); for (i = 0; i != wq_sz; i++) { struct mlx5e_rx_wqe *wqe = mlx5_wq_ll_get_wqe(&rq->wq, i); +#if (MLX5E_MAX_RX_SEGS == 1) uint32_t byte_count = rq->wqe_sz - MLX5E_NET_IP_ALIGN; +#else + int j; +#endif err = -bus_dmamap_create(rq->dma_tag, 0, &rq->mbuf[i].dma_map); if (err != 0) { @@ -869,8 +867,15 @@ mlx5e_create_rq(struct mlx5e_channel *c, bus_dmamap_destroy(rq->dma_tag, rq->mbuf[i].dma_map); goto err_rq_mbuf_free; } - wqe->data.lkey = c->mkey_be; - wqe->data.byte_count = cpu_to_be32(byte_count | MLX5_HW_START_PADDING); + + /* set value for constant fields */ +#if (MLX5E_MAX_RX_SEGS == 1) + wqe->data[0].lkey = c->mkey_be; + wqe->data[0].byte_count = cpu_to_be32(byte_count | MLX5_HW_START_PADDING); +#else + for (j = 0; j < rq->nsegs; j++) + wqe->data[j].lkey = c->mkey_be; +#endif } rq->ifp = c->ifp; @@ -1809,16 +1814,51 @@ mlx5e_close_channel_wait(struct mlx5e_channel *volatile *pp) free(c, M_MLX5EN); } +static int +mlx5e_get_wqe_sz(struct mlx5e_priv *priv, u32 *wqe_sz, u32 *nsegs) +{ + u32 r, n; + + r = priv->params.hw_lro_en ? priv->params.lro_wqe_sz : + MLX5E_SW2MB_MTU(priv->ifp->if_mtu); + if (r > MJUM16BYTES) + return (-ENOMEM); + + if (r > MJUM9BYTES) + r = MJUM16BYTES; + else if (r > MJUMPAGESIZE) + r = MJUM9BYTES; + else if (r > MCLBYTES) + r = MJUMPAGESIZE; + else + r = MCLBYTES; + + /* + * n + 1 must be a power of two, because stride size must be. + * Stride size is 16 * (n + 1), as the first segment is + * control. + */ + for (n = howmany(r, MLX5E_MAX_RX_BYTES); !powerof2(n + 1); n++) + ; + + *wqe_sz = r; + *nsegs = n; + return (0); +} + static void mlx5e_build_rq_param(struct mlx5e_priv *priv, struct mlx5e_rq_param *param) { void *rqc = param->rqc; void *wq = MLX5_ADDR_OF(rqc, rqc, wq); + u32 wqe_sz, nsegs; + mlx5e_get_wqe_sz(priv, &wqe_sz, &nsegs); MLX5_SET(wq, wq, wq_type, MLX5_WQ_TYPE_LINKED_LIST); MLX5_SET(wq, wq, end_padding_mode, MLX5_WQ_END_PAD_MODE_ALIGN); - MLX5_SET(wq, wq, log_wq_stride, ilog2(sizeof(struct mlx5e_rx_wqe))); + MLX5_SET(wq, wq, log_wq_stride, ilog2(sizeof(struct mlx5e_rx_wqe) + + nsegs * sizeof(struct mlx5_wqe_data_seg))); MLX5_SET(wq, wq, log_wq_sz, priv->params.log_rq_size); MLX5_SET(wq, wq, pd, priv->pdn); diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c b/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c index fb14be43b32..5868b0a2f55 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c @@ -32,21 +32,47 @@ static inline int mlx5e_alloc_rx_wqe(struct mlx5e_rq *rq, struct mlx5e_rx_wqe *wqe, u16 ix) { - bus_dma_segment_t segs[1]; + bus_dma_segment_t segs[rq->nsegs]; struct mbuf *mb; int nsegs; int err; - +#if (MLX5E_MAX_RX_SEGS != 1) + struct mbuf *mb_head; + int i; +#endif if (rq->mbuf[ix].mbuf != NULL) return (0); +#if (MLX5E_MAX_RX_SEGS == 1) mb = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, rq->wqe_sz); if (unlikely(!mb)) return (-ENOMEM); - /* set initial mbuf length */ mb->m_pkthdr.len = mb->m_len = rq->wqe_sz; +#else + mb_head = mb = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, + MLX5E_MAX_RX_BYTES); + if (unlikely(mb == NULL)) + return (-ENOMEM); + + mb->m_len = MLX5E_MAX_RX_BYTES; + mb->m_pkthdr.len = MLX5E_MAX_RX_BYTES; + for (i = 1; i < rq->nsegs; i++) { + if (mb_head->m_pkthdr.len >= rq->wqe_sz) + break; + mb = mb->m_next = m_getjcl(M_NOWAIT, MT_DATA, 0, + MLX5E_MAX_RX_BYTES); + if (unlikely(mb == NULL)) { + m_freem(mb_head); + return (-ENOMEM); + } + mb->m_len = MLX5E_MAX_RX_BYTES; + mb_head->m_pkthdr.len += MLX5E_MAX_RX_BYTES; + } + /* rewind to first mbuf in chain */ + mb = mb_head; +#endif /* get IP header aligned */ m_adj(mb, MLX5E_NET_IP_ALIGN); @@ -54,12 +80,26 @@ mlx5e_alloc_rx_wqe(struct mlx5e_rq *rq, mb, segs, &nsegs, BUS_DMA_NOWAIT); if (err != 0) goto err_free_mbuf; - if (unlikely(nsegs != 1)) { + if (unlikely(nsegs == 0)) { bus_dmamap_unload(rq->dma_tag, rq->mbuf[ix].dma_map); err = -ENOMEM; goto err_free_mbuf; } - wqe->data.addr = cpu_to_be64(segs[0].ds_addr); +#if (MLX5E_MAX_RX_SEGS == 1) + wqe->data[0].addr = cpu_to_be64(segs[0].ds_addr); +#else + wqe->data[0].addr = cpu_to_be64(segs[0].ds_addr); + wqe->data[0].byte_count = cpu_to_be32(segs[0].ds_len | + MLX5_HW_START_PADDING); + for (i = 1; i != nsegs; i++) { + wqe->data[i].addr = cpu_to_be64(segs[i].ds_addr); + wqe->data[i].byte_count = cpu_to_be32(segs[i].ds_len); + } + for (; i < rq->nsegs; i++) { + wqe->data[i].addr = 0; + wqe->data[i].byte_count = 0; + } +#endif rq->mbuf[ix].mbuf = mb; rq->mbuf[ix].data = mb->m_data; @@ -214,6 +254,9 @@ mlx5e_build_rx_mbuf(struct mlx5_cqe64 *cqe, { struct ifnet *ifp = rq->ifp; struct mlx5e_channel *c; +#if (MLX5E_MAX_RX_SEGS != 1) + struct mbuf *mb_head; +#endif int lro_num_seg; /* HW LRO session aggregated packets counter */ uint64_t tstmp; @@ -224,7 +267,26 @@ mlx5e_build_rx_mbuf(struct mlx5_cqe64 *cqe, rq->stats.lro_bytes += cqe_bcnt; } +#if (MLX5E_MAX_RX_SEGS == 1) mb->m_pkthdr.len = mb->m_len = cqe_bcnt; +#else + mb->m_pkthdr.len = cqe_bcnt; + for (mb_head = mb; mb != NULL; mb = mb->m_next) { + if (mb->m_len > cqe_bcnt) + mb->m_len = cqe_bcnt; + cqe_bcnt -= mb->m_len; + if (likely(cqe_bcnt == 0)) { + if (likely(mb->m_next != NULL)) { + /* trim off empty mbufs */ + m_freem(mb->m_next); + mb->m_next = NULL; + } + break; + } + } + /* rewind to first mbuf in chain */ + mb = mb_head; +#endif /* check if a Toeplitz hash was computed */ if (cqe->rss_hash_type != 0) { mb->m_pkthdr.flowid = be32_to_cpu(cqe->rss_hash_result); @@ -447,7 +509,10 @@ mlx5e_rx_cq_comp(struct mlx5_core_cq *mcq) int i = 0; #ifdef HAVE_PER_CQ_EVENT_PACKET - struct mbuf *mb = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, rq->wqe_sz); +#if (MHLEN < 15) +#error "MHLEN is too small" +#endif + struct mbuf *mb = m_gethdr(M_NOWAIT, MT_DATA); if (mb != NULL) { /* this code is used for debugging purpose only */ From owner-freebsd-current@freebsd.org Tue Apr 24 09:55:42 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA7E7FC1E1D for ; Tue, 24 Apr 2018 09:55:42 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3750E762A7 for ; Tue, 24 Apr 2018 09:55:42 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id E2915FC1E1A; Tue, 24 Apr 2018 09:55:41 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CFE90FC1E19; Tue, 24 Apr 2018 09:55:41 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x229.google.com (mail-wr0-x229.google.com [IPv6:2a00:1450:400c:c0c::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4A08E762A5; Tue, 24 Apr 2018 09:55:41 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x229.google.com with SMTP id v60-v6so48966486wrc.7; Tue, 24 Apr 2018 02:55:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=kwZ1k/M1N81U2XUHmG1NDMcVWzp13Dap0jCwfFYPgKM=; b=AAtyO2J+EriwYb6LBTORjfTlbD4y5YvDAXrZL2pXF/HB584vNdzNjWD6DILxNwu2mo +ArZdE62TekhvUavch0QNMx08cfmhNOzDxfwuiLTzoM0cUPy9qDh7pD30vpEFSjjlq+r iQdrcR1p1+BVWo5apAanXrKVN8qCeBDTjh3ld2fXyMF7GUur/7Z3FFXMer4FN5FxXcON 75JlVdc6cdN5/k+lf7Kj+VOMDLgkPwpctB/qD1RYx8de4PxromRAHsPOiS8L2Qfp7EQK RbFcXHeoWwWVfl23UcT4PQdm07RWm4oeS20lTB2Lofd+8IOMlX3hXU2VqSQgFlIhdFPj 1uRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=kwZ1k/M1N81U2XUHmG1NDMcVWzp13Dap0jCwfFYPgKM=; b=jp8UO0yywl5y6ziWosJhCiSYYPPrvEM1gKIewtKLxLREdjF/+c96IilbgO0Ryxo73S gsNRpy89OOXZS37G2KHhdK7dVtcvK/sPsS182XgvVi702MgjL9ENm97CpkxXFueqff+B gfypGT0re6oeVDeYzS+xUEyrPOkc1hR3FtTp6STD0NnhzCiw5Sf4g0XJtfCC3YjyPpKJ fTaiaI2TcSmNjy/lSIh9AsYqSc9OTbL0OtUK8ubuybPlVCgwkBG4xZIV7HgbMfPdR9V0 uliSQsN286yhOzj3nbFZ4siNgqyMmcqZzKzpwiOyYm321dsN0bd2N767AcWXvLz48khf jZIg== X-Gm-Message-State: ALQs6tBfKX1auzw4UIFyVkyAviPsYPuZaPrmWnEXVfIRPVzoGJ1rKc8u wW5PntEiX9Osldc9YvrguTU= X-Google-Smtp-Source: AIpwx4/krP2QzlqEP/MqncMXeXQezpmu9BD3EM8kYHdlkyYfPzLG+tucSQk4JDNm+b3fQHyTwwRaTA== X-Received: by 2002:adf:de0c:: with SMTP id b12-v6mr12855576wrm.131.1524563740371; Tue, 24 Apr 2018 02:55:40 -0700 (PDT) Received: from bens-mac.home (LFbn-1-7077-100.w90-116.abo.wanadoo.fr. [90.116.246.100]) by smtp.gmail.com with ESMTPSA id c141sm15530241wmh.21.2018.04.24.02.55.39 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 24 Apr 2018 02:55:39 -0700 (PDT) Content-Type: text/plain; charset=us-ascii; delsp=yes; format=flowed Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: mlx5(4) jumbo receive From: Ben RUBSON In-Reply-To: <20180424085553.GA6887@kib.kiev.ua> Date: Tue, 24 Apr 2018 11:55:38 +0200 Cc: current@freebsd.org, menny@mellanox.com, net@freebsd.org Content-Transfer-Encoding: 7bit Message-Id: <2AE60706-A9E1-4F47-AB16-AA1292405671@gmail.com> References: <20180424085553.GA6887@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.3124) X-Mailman-Approved-At: Tue, 24 Apr 2018 10:38:04 +0000 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2018 09:55:43 -0000 On 24 Apr, Konstantin Belousov wrote: > Hello, > the patch below is of interest for people who use Mellanox Connect-X/4 > and 5 ethernet adapters and configure it for jumbo frames. Hi Konstantin, Good news ! Do you think your work could be easily ported to Connect-X/3 devices (mlx4 driver) ? Thank you very much, Ben For completeness regarding mlx4, some work has been started on it : https://reviews.freebsd.org/D11560 There's a discussion : https://lists.freebsd.org/pipermail/freebsd-net/2017-June/048293.html Which continues here : https://lists.freebsd.org/pipermail/freebsd-net/2017-September/048906.html From owner-freebsd-current@freebsd.org Tue Apr 24 11:11:02 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5530DF8B2C5 for ; Tue, 24 Apr 2018 11:11:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1419085195 for ; Tue, 24 Apr 2018 11:11:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id C911FF8B2AC; Tue, 24 Apr 2018 11:11:01 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B533CF8B2AB; Tue, 24 Apr 2018 11:11:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 28DB985187; Tue, 24 Apr 2018 11:11:01 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w3OBAojU017230 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 24 Apr 2018 14:10:53 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w3OBAojU017230 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w3OBAowW017229; Tue, 24 Apr 2018 14:10:50 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 24 Apr 2018 14:10:50 +0300 From: Konstantin Belousov To: Ben RUBSON Cc: current@freebsd.org, menyy@mellanox.com, net@freebsd.org Subject: Re: mlx5(4) jumbo receive Message-ID: <20180424111050.GB6887@kib.kiev.ua> References: <20180424085553.GA6887@kib.kiev.ua> <2AE60706-A9E1-4F47-AB16-AA1292405671@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2AE60706-A9E1-4F47-AB16-AA1292405671@gmail.com> User-Agent: Mutt/1.9.5 (2018-04-13) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2018 11:11:02 -0000 On Tue, Apr 24, 2018 at 11:55:38AM +0200, Ben RUBSON wrote: > On 24 Apr, Konstantin Belousov wrote: > > > Hello, > > the patch below is of interest for people who use Mellanox Connect-X/4 > > and 5 ethernet adapters and configure it for jumbo frames. > > Hi Konstantin, > > Good news ! > Do you think your work could be easily ported to Connect-X/3 devices (mlx4 > driver) ? No, it cannot be easily ported. The port might happen, but I do not promise this. > > Thank you very much, > > Ben > > For completeness regarding mlx4, some work has been started on it : > https://reviews.freebsd.org/D11560 > There's a discussion : > https://lists.freebsd.org/pipermail/freebsd-net/2017-June/048293.html > Which continues here : > https://lists.freebsd.org/pipermail/freebsd-net/2017-September/048906.html From owner-freebsd-current@freebsd.org Tue Apr 24 13:44:36 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC0D7FA4BF3 for ; Tue, 24 Apr 2018 13:44:35 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 61E2C866C9 for ; Tue, 24 Apr 2018 13:44:35 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: by mailman.ysv.freebsd.org (Postfix) id 15AE3FA4BEE; Tue, 24 Apr 2018 13:44:35 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4DCEFA4BED; Tue, 24 Apr 2018 13:44:34 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 34E37866C7; Tue, 24 Apr 2018 13:44:33 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w3ODiO7R090978; Tue, 24 Apr 2018 06:44:24 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w3ODiOdA090977; Tue, 24 Apr 2018 06:44:24 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201804241344.w3ODiOdA090977@pdx.rh.CN85.dnsmgr.net> Subject: Re: mlx5(4) jumbo receive In-Reply-To: <20180424085553.GA6887@kib.kiev.ua> To: Konstantin Belousov Date: Tue, 24 Apr 2018 06:44:24 -0700 (PDT) CC: current@freebsd.org, menny@mellanox.com, net@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2018 13:44:36 -0000 > Hello, > the patch below is of interest for people who use Mellanox Connect-X/4 > and 5 ethernet adapters and configure it for jumbo frames. The patch > should improve the system behavior on low or fragmented memory situations. > See the commit message for detailed description. Also, the patch should > not cause performance degradation for the normal 1500 MTU by keeping the > same configuration as currently unpatched driver. > > I am interested in hearing about regressions (or not) caused by the > change. I'll hopefully get to test this early next week, but my schedule is very full until then. I do have a fist full of sytle(9) and other comments though. You can probably ignore my nits on capitalize start of comments, it seems that this may be the style of this code.... > commit f25c0c624177a1ca06ca051b0adb842acb66ec11 > Author: Konstantin Belousov > Date: Wed Nov 22 20:14:53 2017 +0200 > > mlx5en: Handle jumbo frames without requiring big clusters. > > The scatter list is formed by the chunks of MCLBYTES each, and larger > than default packets are returned to the stack as the mbuf chain. > > This is an improvements over the original patch by hanss@mellanox.com > which solves some busdma issues and sizes WQEs scatter list to only > have the minimal sufficient number of elements. In particular, for > the default MTU 1500 bytes the receive queue format is not changed > comparing to the unpatched driver, avoiding a reported performance > regression. > > Issue: 1065432 > Issue: 1184316 > Change-Id: Ib63a5ef55abd6ef1ec9b296e2cef88e4604bbd29 > Signed-off-by: Konstantin Belousov > > diff --git a/sys/dev/mlx5/mlx5_en/en.h b/sys/dev/mlx5/mlx5_en/en.h > index b5000c32eaf..ff5ef446e34 100644 > --- a/sys/dev/mlx5/mlx5_en/en.h > +++ b/sys/dev/mlx5/mlx5_en/en.h > @@ -80,8 +80,19 @@ > #define MLX5E_PARAMS_DEFAULT_LOG_RQ_SIZE 0xa > #define MLX5E_PARAMS_MAXIMUM_LOG_RQ_SIZE 0xe > > -/* freeBSD HW LRO is limited by 16KB - the size of max mbuf */ Though it is tempting to fix spelling errors and stylish things while working on code it usually helps to sepeate those into either a pre or post commit to reduce the lines of code change that need to be looked at when reviewing a diff. This also helps diff(1) to chunk things a bit better. > +#define MLX5E_MAX_RX_SEGS 7 > + > +#ifndef MLX5E_MAX_RX_BYTES > +#define MLX5E_MAX_RX_BYTES MCLBYTES > +#endif > + > +#if (MLX5E_MAX_RX_SEGS == 1) > +/* FreeBSD HW LRO is limited by 16KB - the size of max mbuf */ > #define MLX5E_PARAMS_DEFAULT_LRO_WQE_SZ MJUM16BYTES > +#else > +#define MLX5E_PARAMS_DEFAULT_LRO_WQE_SZ \ > + MIN(65535, MLX5E_MAX_RX_SEGS * MLX5E_MAX_RX_BYTES) > +#endif > #define MLX5E_PARAMS_DEFAULT_RX_CQ_MODERATION_USEC 0x10 > #define MLX5E_PARAMS_DEFAULT_RX_CQ_MODERATION_USEC_FROM_CQE 0x3 > #define MLX5E_PARAMS_DEFAULT_RX_CQ_MODERATION_PKTS 0x20 > @@ -530,6 +541,7 @@ struct mlx5e_rq { > struct mtx mtx; > bus_dma_tag_t dma_tag; > u32 wqe_sz; > + u32 nsegs; Alphabetic order, and should just use modify the current u32 line: u32 nsegs, wqe_sz; > struct mlx5e_rq_mbuf *mbuf; > struct ifnet *ifp; > struct mlx5e_rq_stats stats; > @@ -795,9 +807,12 @@ struct mlx5e_tx_wqe { > > struct mlx5e_rx_wqe { > struct mlx5_wqe_srq_next_seg next; > - struct mlx5_wqe_data_seg data; > + struct mlx5_wqe_data_seg data[]; None standard way to declare a pointer? > }; > > +/* the size of the structure above must be power of two */ the should be The. > +CTASSERT(powerof2(sizeof(struct mlx5e_rx_wqe))); > + > struct mlx5e_eeprom { > int lock_bit; > int i2c_addr; > diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c > index c2c4b0d7744..25544e561e3 100644 > --- a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c > +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c > @@ -33,9 +33,12 @@ > #ifndef ETH_DRIVER_VERSION > #define ETH_DRIVER_VERSION "3.4.1" > #endif > + Adding white space is a style change, and actually unwanted right here as the next line is tightly coupled to the previous line. > char mlx5e_version[] = "Mellanox Ethernet driver" > " (" ETH_DRIVER_VERSION ")"; > > +static int mlx5e_get_wqe_sz(struct mlx5e_priv *priv, u32 *wqe_sz, u32 *nsegs); > + Could you move the function before use and eliminate the need for the forward declare? > struct mlx5e_channel_param { > struct mlx5e_rq_param rq; > struct mlx5e_sq_param sq; > @@ -810,6 +813,11 @@ mlx5e_create_rq(struct mlx5e_channel *c, > int wq_sz; > int err; > int i; > + u32 nsegs, wqe_sz; > + > + err = mlx5e_get_wqe_sz(priv, &wqe_sz, &nsegs); > + if (err != 0) > + goto done; > > /* Create DMA descriptor TAG */ > if ((err = -bus_dma_tag_create( > @@ -819,9 +827,9 @@ mlx5e_create_rq(struct mlx5e_channel *c, > BUS_SPACE_MAXADDR, /* lowaddr */ > BUS_SPACE_MAXADDR, /* highaddr */ > NULL, NULL, /* filter, filterarg */ > - MJUM16BYTES, /* maxsize */ > - 1, /* nsegments */ > - MJUM16BYTES, /* maxsegsize */ > + nsegs * MLX5E_MAX_RX_BYTES, /* maxsize */ > + nsegs, /* nsegments */ > + nsegs * MLX5E_MAX_RX_BYTES, /* maxsegsize */ > 0, /* flags */ > NULL, NULL, /* lockfunc, lockfuncarg */ > &rq->dma_tag))) > @@ -834,23 +842,9 @@ mlx5e_create_rq(struct mlx5e_channel *c, > > rq->wq.db = &rq->wq.db[MLX5_RCV_DBR]; > > - if (priv->params.hw_lro_en) { > - rq->wqe_sz = priv->params.lro_wqe_sz; > - } else { > - rq->wqe_sz = MLX5E_SW2MB_MTU(priv->ifp->if_mtu); > - } > - if (rq->wqe_sz > MJUM16BYTES) { > - err = -ENOMEM; > + err = mlx5e_get_wqe_sz(priv, &rq->wqe_sz, &rq->nsegs); > + if (err != 0) > goto err_rq_wq_destroy; > - } else if (rq->wqe_sz > MJUM9BYTES) { > - rq->wqe_sz = MJUM16BYTES; > - } else if (rq->wqe_sz > MJUMPAGESIZE) { > - rq->wqe_sz = MJUM9BYTES; > - } else if (rq->wqe_sz > MCLBYTES) { > - rq->wqe_sz = MJUMPAGESIZE; > - } else { > - rq->wqe_sz = MCLBYTES; > - } > > wq_sz = mlx5_wq_ll_get_size(&rq->wq); > > @@ -861,7 +855,11 @@ mlx5e_create_rq(struct mlx5e_channel *c, > rq->mbuf = malloc(wq_sz * sizeof(rq->mbuf[0]), M_MLX5EN, M_WAITOK | M_ZERO); > for (i = 0; i != wq_sz; i++) { > struct mlx5e_rx_wqe *wqe = mlx5_wq_ll_get_wqe(&rq->wq, i); > +#if (MLX5E_MAX_RX_SEGS == 1) > uint32_t byte_count = rq->wqe_sz - MLX5E_NET_IP_ALIGN; > +#else > + int j; > +#endif > > err = -bus_dmamap_create(rq->dma_tag, 0, &rq->mbuf[i].dma_map); > if (err != 0) { > @@ -869,8 +867,15 @@ mlx5e_create_rq(struct mlx5e_channel *c, > bus_dmamap_destroy(rq->dma_tag, rq->mbuf[i].dma_map); > goto err_rq_mbuf_free; > } > - wqe->data.lkey = c->mkey_be; > - wqe->data.byte_count = cpu_to_be32(byte_count | MLX5_HW_START_PADDING); > + > + /* set value for constant fields */ > +#if (MLX5E_MAX_RX_SEGS == 1) > + wqe->data[0].lkey = c->mkey_be; > + wqe->data[0].byte_count = cpu_to_be32(byte_count | MLX5_HW_START_PADDING); > +#else > + for (j = 0; j < rq->nsegs; j++) > + wqe->data[j].lkey = c->mkey_be; > +#endif > } > > rq->ifp = c->ifp; > @@ -1809,16 +1814,51 @@ mlx5e_close_channel_wait(struct mlx5e_channel *volatile *pp) > free(c, M_MLX5EN); > } > > +static int > +mlx5e_get_wqe_sz(struct mlx5e_priv *priv, u32 *wqe_sz, u32 *nsegs) > +{ > + u32 r, n; Alpah order on variables: u32 n, r; > + > + r = priv->params.hw_lro_en ? priv->params.lro_wqe_sz : > + MLX5E_SW2MB_MTU(priv->ifp->if_mtu); > + if (r > MJUM16BYTES) > + return (-ENOMEM); > + > + if (r > MJUM9BYTES) > + r = MJUM16BYTES; > + else if (r > MJUMPAGESIZE) > + r = MJUM9BYTES; > + else if (r > MCLBYTES) > + r = MJUMPAGESIZE; > + else > + r = MCLBYTES; > + > + /* > + * n + 1 must be a power of two, because stride size must be. > + * Stride size is 16 * (n + 1), as the first segment is > + * control. > + */ > + for (n = howmany(r, MLX5E_MAX_RX_BYTES); !powerof2(n + 1); n++) > + ; > + > + *wqe_sz = r; > + *nsegs = n; > + return (0); > +} > + > static void > mlx5e_build_rq_param(struct mlx5e_priv *priv, > struct mlx5e_rq_param *param) > { > void *rqc = param->rqc; > void *wq = MLX5_ADDR_OF(rqc, rqc, wq); > + u32 wqe_sz, nsegs; Variable order: nsegs, wqe_sz; > > + mlx5e_get_wqe_sz(priv, &wqe_sz, &nsegs); > MLX5_SET(wq, wq, wq_type, MLX5_WQ_TYPE_LINKED_LIST); > MLX5_SET(wq, wq, end_padding_mode, MLX5_WQ_END_PAD_MODE_ALIGN); > - MLX5_SET(wq, wq, log_wq_stride, ilog2(sizeof(struct mlx5e_rx_wqe))); > + MLX5_SET(wq, wq, log_wq_stride, ilog2(sizeof(struct mlx5e_rx_wqe) + > + nsegs * sizeof(struct mlx5_wqe_data_seg))); > MLX5_SET(wq, wq, log_wq_sz, priv->params.log_rq_size); > MLX5_SET(wq, wq, pd, priv->pdn); > > diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c b/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c > index fb14be43b32..5868b0a2f55 100644 > --- a/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c > +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c > @@ -32,21 +32,47 @@ static inline int > mlx5e_alloc_rx_wqe(struct mlx5e_rq *rq, > struct mlx5e_rx_wqe *wqe, u16 ix) > { > - bus_dma_segment_t segs[1]; > + bus_dma_segment_t segs[rq->nsegs]; This code line compiles? Or another odd pointer declare? > struct mbuf *mb; > int nsegs; > int err; > - > +#if (MLX5E_MAX_RX_SEGS != 1) > + struct mbuf *mb_head; > + int i; > +#endif > if (rq->mbuf[ix].mbuf != NULL) > return (0); > > +#if (MLX5E_MAX_RX_SEGS == 1) > mb = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, rq->wqe_sz); > if (unlikely(!mb)) > return (-ENOMEM); > > - /* set initial mbuf length */ > mb->m_pkthdr.len = mb->m_len = rq->wqe_sz; > +#else > + mb_head = mb = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, > + MLX5E_MAX_RX_BYTES); > + if (unlikely(mb == NULL)) > + return (-ENOMEM); > + > + mb->m_len = MLX5E_MAX_RX_BYTES; > + mb->m_pkthdr.len = MLX5E_MAX_RX_BYTES; > > + for (i = 1; i < rq->nsegs; i++) { > + if (mb_head->m_pkthdr.len >= rq->wqe_sz) > + break; > + mb = mb->m_next = m_getjcl(M_NOWAIT, MT_DATA, 0, > + MLX5E_MAX_RX_BYTES); > + if (unlikely(mb == NULL)) { > + m_freem(mb_head); > + return (-ENOMEM); > + } > + mb->m_len = MLX5E_MAX_RX_BYTES; > + mb_head->m_pkthdr.len += MLX5E_MAX_RX_BYTES; > + } > + /* rewind to first mbuf in chain */ > + mb = mb_head; > +#endif > /* get IP header aligned */ > m_adj(mb, MLX5E_NET_IP_ALIGN); > > @@ -54,12 +80,26 @@ mlx5e_alloc_rx_wqe(struct mlx5e_rq *rq, > mb, segs, &nsegs, BUS_DMA_NOWAIT); > if (err != 0) > goto err_free_mbuf; > - if (unlikely(nsegs != 1)) { > + if (unlikely(nsegs == 0)) { > bus_dmamap_unload(rq->dma_tag, rq->mbuf[ix].dma_map); > err = -ENOMEM; > goto err_free_mbuf; > } > - wqe->data.addr = cpu_to_be64(segs[0].ds_addr); > +#if (MLX5E_MAX_RX_SEGS == 1) > + wqe->data[0].addr = cpu_to_be64(segs[0].ds_addr); > +#else > + wqe->data[0].addr = cpu_to_be64(segs[0].ds_addr); > + wqe->data[0].byte_count = cpu_to_be32(segs[0].ds_len | > + MLX5_HW_START_PADDING); > + for (i = 1; i != nsegs; i++) { > + wqe->data[i].addr = cpu_to_be64(segs[i].ds_addr); > + wqe->data[i].byte_count = cpu_to_be32(segs[i].ds_len); > + } > + for (; i < rq->nsegs; i++) { > + wqe->data[i].addr = 0; > + wqe->data[i].byte_count = 0; > + } > +#endif > > rq->mbuf[ix].mbuf = mb; > rq->mbuf[ix].data = mb->m_data; > @@ -214,6 +254,9 @@ mlx5e_build_rx_mbuf(struct mlx5_cqe64 *cqe, > { > struct ifnet *ifp = rq->ifp; > struct mlx5e_channel *c; > +#if (MLX5E_MAX_RX_SEGS != 1) > + struct mbuf *mb_head; > +#endif > int lro_num_seg; /* HW LRO session aggregated packets counter */ > uint64_t tstmp; > > @@ -224,7 +267,26 @@ mlx5e_build_rx_mbuf(struct mlx5_cqe64 *cqe, > rq->stats.lro_bytes += cqe_bcnt; > } > > +#if (MLX5E_MAX_RX_SEGS == 1) > mb->m_pkthdr.len = mb->m_len = cqe_bcnt; > +#else > + mb->m_pkthdr.len = cqe_bcnt; > + for (mb_head = mb; mb != NULL; mb = mb->m_next) { > + if (mb->m_len > cqe_bcnt) > + mb->m_len = cqe_bcnt; > + cqe_bcnt -= mb->m_len; > + if (likely(cqe_bcnt == 0)) { > + if (likely(mb->m_next != NULL)) { > + /* trim off empty mbufs */ Trim > + m_freem(mb->m_next); > + mb->m_next = NULL; > + } > + break; > + } > + } > + /* rewind to first mbuf in chain */ Rewind > + mb = mb_head; > +#endif > /* check if a Toeplitz hash was computed */ > if (cqe->rss_hash_type != 0) { > mb->m_pkthdr.flowid = be32_to_cpu(cqe->rss_hash_result); > @@ -447,7 +509,10 @@ mlx5e_rx_cq_comp(struct mlx5_core_cq *mcq) > int i = 0; > > #ifdef HAVE_PER_CQ_EVENT_PACKET > - struct mbuf *mb = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, rq->wqe_sz); > +#if (MHLEN < 15) > +#error "MHLEN is too small" > +#endif > + struct mbuf *mb = m_gethdr(M_NOWAIT, MT_DATA); > > if (mb != NULL) { > /* this code is used for debugging purpose only */ > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-current@freebsd.org Tue Apr 24 14:00:19 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E9487FA5B84 for ; Tue, 24 Apr 2018 14:00:18 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 70E906A584 for ; Tue, 24 Apr 2018 14:00:18 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 32F18FA5B7C; Tue, 24 Apr 2018 14:00:18 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB74FFA5B7A; Tue, 24 Apr 2018 14:00:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1F7546A581; Tue, 24 Apr 2018 14:00:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w3ODxx0t054487 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 24 Apr 2018 17:00:03 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w3ODxx0t054487 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w3ODxxS0054486; Tue, 24 Apr 2018 16:59:59 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 24 Apr 2018 16:59:59 +0300 From: Konstantin Belousov To: "Rodney W. Grimes" Cc: current@freebsd.org, menyy@mellanox.com, net@freebsd.org Subject: Re: mlx5(4) jumbo receive Message-ID: <20180424135959.GC6887@kib.kiev.ua> References: <20180424085553.GA6887@kib.kiev.ua> <201804241344.w3ODiOdA090977@pdx.rh.CN85.dnsmgr.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201804241344.w3ODiOdA090977@pdx.rh.CN85.dnsmgr.net> User-Agent: Mutt/1.9.5 (2018-04-13) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2018 14:00:19 -0000 On Tue, Apr 24, 2018 at 06:44:24AM -0700, Rodney W. Grimes wrote: > > Hello, > > the patch below is of interest for people who use Mellanox Connect-X/4 > > and 5 ethernet adapters and configure it for jumbo frames. The patch > > should improve the system behavior on low or fragmented memory situations. > > See the commit message for detailed description. Also, the patch should > > not cause performance degradation for the normal 1500 MTU by keeping the > > same configuration as currently unpatched driver. > > > > I am interested in hearing about regressions (or not) caused by the > > change. > > I'll hopefully get to test this early next week, but my schedule > is very full until then. > > I do have a fist full of sytle(9) and other comments though. > > You can probably ignore my nits on capitalize start of comments, > it seems that this may be the style of this code.... > > > commit f25c0c624177a1ca06ca051b0adb842acb66ec11 > > Author: Konstantin Belousov > > Date: Wed Nov 22 20:14:53 2017 +0200 > > > > mlx5en: Handle jumbo frames without requiring big clusters. > > > > The scatter list is formed by the chunks of MCLBYTES each, and larger > > than default packets are returned to the stack as the mbuf chain. > > > > This is an improvements over the original patch by hanss@mellanox.com > > which solves some busdma issues and sizes WQEs scatter list to only > > have the minimal sufficient number of elements. In particular, for > > the default MTU 1500 bytes the receive queue format is not changed > > comparing to the unpatched driver, avoiding a reported performance > > regression. > > > > Issue: 1065432 > > Issue: 1184316 > > Change-Id: Ib63a5ef55abd6ef1ec9b296e2cef88e4604bbd29 > > Signed-off-by: Konstantin Belousov > > > > diff --git a/sys/dev/mlx5/mlx5_en/en.h b/sys/dev/mlx5/mlx5_en/en.h > > index b5000c32eaf..ff5ef446e34 100644 > > --- a/sys/dev/mlx5/mlx5_en/en.h > > +++ b/sys/dev/mlx5/mlx5_en/en.h > > @@ -80,8 +80,19 @@ > > #define MLX5E_PARAMS_DEFAULT_LOG_RQ_SIZE 0xa > > #define MLX5E_PARAMS_MAXIMUM_LOG_RQ_SIZE 0xe > > > > -/* freeBSD HW LRO is limited by 16KB - the size of max mbuf */ > Though it is tempting to fix spelling errors and stylish things > while working on code it usually helps to sepeate those into > either a pre or post commit to reduce the lines of code change > that need to be looked at when reviewing a diff. This also helps > diff(1) to chunk things a bit better. > > > +#define MLX5E_MAX_RX_SEGS 7 > > + > > +#ifndef MLX5E_MAX_RX_BYTES > > +#define MLX5E_MAX_RX_BYTES MCLBYTES > > +#endif > > + > > +#if (MLX5E_MAX_RX_SEGS == 1) > > +/* FreeBSD HW LRO is limited by 16KB - the size of max mbuf */ > > #define MLX5E_PARAMS_DEFAULT_LRO_WQE_SZ MJUM16BYTES > > +#else > > +#define MLX5E_PARAMS_DEFAULT_LRO_WQE_SZ \ > > + MIN(65535, MLX5E_MAX_RX_SEGS * MLX5E_MAX_RX_BYTES) > > +#endif > > #define MLX5E_PARAMS_DEFAULT_RX_CQ_MODERATION_USEC 0x10 > > #define MLX5E_PARAMS_DEFAULT_RX_CQ_MODERATION_USEC_FROM_CQE 0x3 > > #define MLX5E_PARAMS_DEFAULT_RX_CQ_MODERATION_PKTS 0x20 > > @@ -530,6 +541,7 @@ struct mlx5e_rq { > > struct mtx mtx; > > bus_dma_tag_t dma_tag; > > u32 wqe_sz; > > + u32 nsegs; > > Alphabetic order, and should just use modify the current u32 line: > u32 nsegs, wqe_sz; > > > struct mlx5e_rq_mbuf *mbuf; > > struct ifnet *ifp; > > struct mlx5e_rq_stats stats; > > @@ -795,9 +807,12 @@ struct mlx5e_tx_wqe { > > > > struct mlx5e_rx_wqe { > > struct mlx5_wqe_srq_next_seg next; > > - struct mlx5_wqe_data_seg data; > > + struct mlx5_wqe_data_seg data[]; > > None standard way to declare a pointer? This is not a pointer declaration. It is a flexible array member. For the style issues, I will take look later. > > > }; > > > > +/* the size of the structure above must be power of two */ > the should be The. > > > +CTASSERT(powerof2(sizeof(struct mlx5e_rx_wqe))); > > + > > struct mlx5e_eeprom { > > int lock_bit; > > int i2c_addr; > > diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c > > index c2c4b0d7744..25544e561e3 100644 > > --- a/sys/dev/mlx5/mlx5_en/mlx5_en_main.c > > +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_main.c > > @@ -33,9 +33,12 @@ > > #ifndef ETH_DRIVER_VERSION > > #define ETH_DRIVER_VERSION "3.4.1" > > #endif > > + > Adding white space is a style change, and actually unwanted right here > as the next line is tightly coupled to the previous line. > > > char mlx5e_version[] = "Mellanox Ethernet driver" > > " (" ETH_DRIVER_VERSION ")"; > > > > +static int mlx5e_get_wqe_sz(struct mlx5e_priv *priv, u32 *wqe_sz, u32 *nsegs); > > + > > Could you move the function before use and eliminate the need > for the forward declare? > > > struct mlx5e_channel_param { > > struct mlx5e_rq_param rq; > > struct mlx5e_sq_param sq; > > @@ -810,6 +813,11 @@ mlx5e_create_rq(struct mlx5e_channel *c, > > int wq_sz; > > int err; > > int i; > > + u32 nsegs, wqe_sz; > > + > > + err = mlx5e_get_wqe_sz(priv, &wqe_sz, &nsegs); > > + if (err != 0) > > + goto done; > > > > /* Create DMA descriptor TAG */ > > if ((err = -bus_dma_tag_create( > > @@ -819,9 +827,9 @@ mlx5e_create_rq(struct mlx5e_channel *c, > > BUS_SPACE_MAXADDR, /* lowaddr */ > > BUS_SPACE_MAXADDR, /* highaddr */ > > NULL, NULL, /* filter, filterarg */ > > - MJUM16BYTES, /* maxsize */ > > - 1, /* nsegments */ > > - MJUM16BYTES, /* maxsegsize */ > > + nsegs * MLX5E_MAX_RX_BYTES, /* maxsize */ > > + nsegs, /* nsegments */ > > + nsegs * MLX5E_MAX_RX_BYTES, /* maxsegsize */ > > 0, /* flags */ > > NULL, NULL, /* lockfunc, lockfuncarg */ > > &rq->dma_tag))) > > @@ -834,23 +842,9 @@ mlx5e_create_rq(struct mlx5e_channel *c, > > > > rq->wq.db = &rq->wq.db[MLX5_RCV_DBR]; > > > > - if (priv->params.hw_lro_en) { > > - rq->wqe_sz = priv->params.lro_wqe_sz; > > - } else { > > - rq->wqe_sz = MLX5E_SW2MB_MTU(priv->ifp->if_mtu); > > - } > > - if (rq->wqe_sz > MJUM16BYTES) { > > - err = -ENOMEM; > > + err = mlx5e_get_wqe_sz(priv, &rq->wqe_sz, &rq->nsegs); > > + if (err != 0) > > goto err_rq_wq_destroy; > > - } else if (rq->wqe_sz > MJUM9BYTES) { > > - rq->wqe_sz = MJUM16BYTES; > > - } else if (rq->wqe_sz > MJUMPAGESIZE) { > > - rq->wqe_sz = MJUM9BYTES; > > - } else if (rq->wqe_sz > MCLBYTES) { > > - rq->wqe_sz = MJUMPAGESIZE; > > - } else { > > - rq->wqe_sz = MCLBYTES; > > - } > > > > wq_sz = mlx5_wq_ll_get_size(&rq->wq); > > > > @@ -861,7 +855,11 @@ mlx5e_create_rq(struct mlx5e_channel *c, > > rq->mbuf = malloc(wq_sz * sizeof(rq->mbuf[0]), M_MLX5EN, M_WAITOK | M_ZERO); > > for (i = 0; i != wq_sz; i++) { > > struct mlx5e_rx_wqe *wqe = mlx5_wq_ll_get_wqe(&rq->wq, i); > > +#if (MLX5E_MAX_RX_SEGS == 1) > > uint32_t byte_count = rq->wqe_sz - MLX5E_NET_IP_ALIGN; > > +#else > > + int j; > > +#endif > > > > err = -bus_dmamap_create(rq->dma_tag, 0, &rq->mbuf[i].dma_map); > > if (err != 0) { > > @@ -869,8 +867,15 @@ mlx5e_create_rq(struct mlx5e_channel *c, > > bus_dmamap_destroy(rq->dma_tag, rq->mbuf[i].dma_map); > > goto err_rq_mbuf_free; > > } > > - wqe->data.lkey = c->mkey_be; > > - wqe->data.byte_count = cpu_to_be32(byte_count | MLX5_HW_START_PADDING); > > + > > + /* set value for constant fields */ > > +#if (MLX5E_MAX_RX_SEGS == 1) > > + wqe->data[0].lkey = c->mkey_be; > > + wqe->data[0].byte_count = cpu_to_be32(byte_count | MLX5_HW_START_PADDING); > > +#else > > + for (j = 0; j < rq->nsegs; j++) > > + wqe->data[j].lkey = c->mkey_be; > > +#endif > > } > > > > rq->ifp = c->ifp; > > @@ -1809,16 +1814,51 @@ mlx5e_close_channel_wait(struct mlx5e_channel *volatile *pp) > > free(c, M_MLX5EN); > > } > > > > +static int > > +mlx5e_get_wqe_sz(struct mlx5e_priv *priv, u32 *wqe_sz, u32 *nsegs) > > +{ > > + u32 r, n; > Alpah order on variables: > u32 n, r; > > > + > > + r = priv->params.hw_lro_en ? priv->params.lro_wqe_sz : > > + MLX5E_SW2MB_MTU(priv->ifp->if_mtu); > > + if (r > MJUM16BYTES) > > + return (-ENOMEM); > > + > > + if (r > MJUM9BYTES) > > + r = MJUM16BYTES; > > + else if (r > MJUMPAGESIZE) > > + r = MJUM9BYTES; > > + else if (r > MCLBYTES) > > + r = MJUMPAGESIZE; > > + else > > + r = MCLBYTES; > > + > > + /* > > + * n + 1 must be a power of two, because stride size must be. > > + * Stride size is 16 * (n + 1), as the first segment is > > + * control. > > + */ > > + for (n = howmany(r, MLX5E_MAX_RX_BYTES); !powerof2(n + 1); n++) > > + ; > > + > > + *wqe_sz = r; > > + *nsegs = n; > > + return (0); > > +} > > + > > static void > > mlx5e_build_rq_param(struct mlx5e_priv *priv, > > struct mlx5e_rq_param *param) > > { > > void *rqc = param->rqc; > > void *wq = MLX5_ADDR_OF(rqc, rqc, wq); > > + u32 wqe_sz, nsegs; > Variable order: nsegs, wqe_sz; > > > > > + mlx5e_get_wqe_sz(priv, &wqe_sz, &nsegs); > > MLX5_SET(wq, wq, wq_type, MLX5_WQ_TYPE_LINKED_LIST); > > MLX5_SET(wq, wq, end_padding_mode, MLX5_WQ_END_PAD_MODE_ALIGN); > > - MLX5_SET(wq, wq, log_wq_stride, ilog2(sizeof(struct mlx5e_rx_wqe))); > > + MLX5_SET(wq, wq, log_wq_stride, ilog2(sizeof(struct mlx5e_rx_wqe) + > > + nsegs * sizeof(struct mlx5_wqe_data_seg))); > > MLX5_SET(wq, wq, log_wq_sz, priv->params.log_rq_size); > > MLX5_SET(wq, wq, pd, priv->pdn); > > > > diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c b/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c > > index fb14be43b32..5868b0a2f55 100644 > > --- a/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c > > +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_rx.c > > @@ -32,21 +32,47 @@ static inline int > > mlx5e_alloc_rx_wqe(struct mlx5e_rq *rq, > > struct mlx5e_rx_wqe *wqe, u16 ix) > > { > > - bus_dma_segment_t segs[1]; > > + bus_dma_segment_t segs[rq->nsegs]; > This code line compiles? Or another odd pointer declare? > > > struct mbuf *mb; > > int nsegs; > > int err; > > - > > +#if (MLX5E_MAX_RX_SEGS != 1) > > + struct mbuf *mb_head; > > + int i; > > +#endif > > if (rq->mbuf[ix].mbuf != NULL) > > return (0); > > > > +#if (MLX5E_MAX_RX_SEGS == 1) > > mb = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, rq->wqe_sz); > > if (unlikely(!mb)) > > return (-ENOMEM); > > > > - /* set initial mbuf length */ > > mb->m_pkthdr.len = mb->m_len = rq->wqe_sz; > > +#else > > + mb_head = mb = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, > > + MLX5E_MAX_RX_BYTES); > > + if (unlikely(mb == NULL)) > > + return (-ENOMEM); > > + > > + mb->m_len = MLX5E_MAX_RX_BYTES; > > + mb->m_pkthdr.len = MLX5E_MAX_RX_BYTES; > > > > + for (i = 1; i < rq->nsegs; i++) { > > + if (mb_head->m_pkthdr.len >= rq->wqe_sz) > > + break; > > + mb = mb->m_next = m_getjcl(M_NOWAIT, MT_DATA, 0, > > + MLX5E_MAX_RX_BYTES); > > + if (unlikely(mb == NULL)) { > > + m_freem(mb_head); > > + return (-ENOMEM); > > + } > > + mb->m_len = MLX5E_MAX_RX_BYTES; > > + mb_head->m_pkthdr.len += MLX5E_MAX_RX_BYTES; > > + } > > + /* rewind to first mbuf in chain */ > > + mb = mb_head; > > +#endif > > /* get IP header aligned */ > > m_adj(mb, MLX5E_NET_IP_ALIGN); > > > > @@ -54,12 +80,26 @@ mlx5e_alloc_rx_wqe(struct mlx5e_rq *rq, > > mb, segs, &nsegs, BUS_DMA_NOWAIT); > > if (err != 0) > > goto err_free_mbuf; > > - if (unlikely(nsegs != 1)) { > > + if (unlikely(nsegs == 0)) { > > bus_dmamap_unload(rq->dma_tag, rq->mbuf[ix].dma_map); > > err = -ENOMEM; > > goto err_free_mbuf; > > } > > - wqe->data.addr = cpu_to_be64(segs[0].ds_addr); > > +#if (MLX5E_MAX_RX_SEGS == 1) > > + wqe->data[0].addr = cpu_to_be64(segs[0].ds_addr); > > +#else > > + wqe->data[0].addr = cpu_to_be64(segs[0].ds_addr); > > + wqe->data[0].byte_count = cpu_to_be32(segs[0].ds_len | > > + MLX5_HW_START_PADDING); > > + for (i = 1; i != nsegs; i++) { > > + wqe->data[i].addr = cpu_to_be64(segs[i].ds_addr); > > + wqe->data[i].byte_count = cpu_to_be32(segs[i].ds_len); > > + } > > + for (; i < rq->nsegs; i++) { > > + wqe->data[i].addr = 0; > > + wqe->data[i].byte_count = 0; > > + } > > +#endif > > > > rq->mbuf[ix].mbuf = mb; > > rq->mbuf[ix].data = mb->m_data; > > @@ -214,6 +254,9 @@ mlx5e_build_rx_mbuf(struct mlx5_cqe64 *cqe, > > { > > struct ifnet *ifp = rq->ifp; > > struct mlx5e_channel *c; > > +#if (MLX5E_MAX_RX_SEGS != 1) > > + struct mbuf *mb_head; > > +#endif > > int lro_num_seg; /* HW LRO session aggregated packets counter */ > > uint64_t tstmp; > > > > @@ -224,7 +267,26 @@ mlx5e_build_rx_mbuf(struct mlx5_cqe64 *cqe, > > rq->stats.lro_bytes += cqe_bcnt; > > } > > > > +#if (MLX5E_MAX_RX_SEGS == 1) > > mb->m_pkthdr.len = mb->m_len = cqe_bcnt; > > +#else > > + mb->m_pkthdr.len = cqe_bcnt; > > + for (mb_head = mb; mb != NULL; mb = mb->m_next) { > > + if (mb->m_len > cqe_bcnt) > > + mb->m_len = cqe_bcnt; > > + cqe_bcnt -= mb->m_len; > > + if (likely(cqe_bcnt == 0)) { > > + if (likely(mb->m_next != NULL)) { > > + /* trim off empty mbufs */ > Trim > > > + m_freem(mb->m_next); > > + mb->m_next = NULL; > > + } > > + break; > > + } > > + } > > + /* rewind to first mbuf in chain */ > Rewind > > > + mb = mb_head; > > +#endif > > /* check if a Toeplitz hash was computed */ > > if (cqe->rss_hash_type != 0) { > > mb->m_pkthdr.flowid = be32_to_cpu(cqe->rss_hash_result); > > @@ -447,7 +509,10 @@ mlx5e_rx_cq_comp(struct mlx5_core_cq *mcq) > > int i = 0; > > > > #ifdef HAVE_PER_CQ_EVENT_PACKET > > - struct mbuf *mb = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, rq->wqe_sz); > > +#if (MHLEN < 15) > > +#error "MHLEN is too small" > > +#endif > > + struct mbuf *mb = m_gethdr(M_NOWAIT, MT_DATA); > > > > if (mb != NULL) { > > /* this code is used for debugging purpose only */ > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > -- > Rod Grimes rgrimes@freebsd.org From owner-freebsd-current@freebsd.org Tue Apr 24 15:27:25 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45354FA9106; Tue, 24 Apr 2018 15:27:25 +0000 (UTC) (envelope-from emaste@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E38DC7F9A3; Tue, 24 Apr 2018 15:27:24 +0000 (UTC) (envelope-from emaste@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1079) id DAC4613325; Tue, 24 Apr 2018 15:27:24 +0000 (UTC) Date: Tue, 24 Apr 2018 15:27:24 +0000 From: Ed Maste To: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Deprecating the lmc(4) driver Message-ID: <20180424152724.GA30980@freebsd.org> Mail-Followup-To: Ed Maste , freebsd-current@freebsd.org, freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.9.2 (2017-12-15) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2018 15:27:25 -0000 The lmc(4) driver supports hardware for a number of legacy interfaces, including EIA612/613, T1/E1, T3, etc. The driver's license is ambiguous and attempts to contact the author failed. I would like to remove this driver from 12.0, and have a deprecation notice in review D15182 [1]; I would appreciate hearing from anyone who is currently using lmc(4) in case it is still providing some value. [1] https://reviews.freebsd.org/D15182 From owner-freebsd-current@freebsd.org Tue Apr 24 23:30:36 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60FCCFB6E65 for ; Tue, 24 Apr 2018 23:30:36 +0000 (UTC) (envelope-from swills@FreeBSD.org) Received: from mouf.net (mouf.net [IPv6:2607:fc50:0:4400:216:3eff:fe69:33b3]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mouf.net", Issuer "mouf.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E1287794B6 for ; Tue, 24 Apr 2018 23:30:35 +0000 (UTC) (envelope-from swills@FreeBSD.org) Received: from lrrr.mouf.net (cpe-2606-A000-4102-2500-6245-CBFF-FE5F-6C74.dyn6.twc.com [IPv6:2606:a000:4102:2500:6245:cbff:fe5f:6c74]) (authenticated bits=0) by mouf.net (8.14.9/8.14.9) with ESMTP id w3ONUSkE063221 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Tue, 24 Apr 2018 23:30:34 GMT (envelope-from swills@FreeBSD.org) To: FreeBSD Current From: Steve Wills Subject: zfskern{txg_thread_enter} thread using 100% or more CPU Message-ID: Date: Tue, 24 Apr 2018 19:30:22 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (mouf.net [IPv6:2607:fc50:0:4400:216:3eff:fe69:33b3]); Tue, 24 Apr 2018 23:30:34 +0000 (UTC) X-Spam-Status: No, score=0.0 required=4.5 tests=none autolearn=unavailable autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mouf.net X-Virus-Scanned: clamav-milter 0.99.2 at mouf.net X-Virus-Status: Clean X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2018 23:30:36 -0000 Hi, Recently on multiple systems running CURRENT, I've been seeing the system become unresponsive. Leaving top(1) running has lead me to notice that when this happens, the system is still responding to ping and top over ssh is still working, but no new processes can start and switching to other tasks doesn't work. In top, I do see pid 17, [zfskern{txg_thread_enter}] monopolizing both CPU usage and disk IO. Any ideas how to troubleshoot this? It doesn't appear to be a hardware issue. Steve From owner-freebsd-current@freebsd.org Wed Apr 25 00:22:23 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1900CFB86B7 for ; Wed, 25 Apr 2018 00:22:23 +0000 (UTC) (envelope-from greg@unrelenting.technology) Received: from hraggstad.unrelenting.technology (hraggstad.unrelenting.technology [71.19.146.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hraggstad.unrelenting.technology", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8619587715; Wed, 25 Apr 2018 00:22:22 +0000 (UTC) (envelope-from greg@unrelenting.technology) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=unrelenting.technology; h=date:from:subject:to:message-id; s=default; bh=NWZv8IUlNj0DKWUMH7nxS1UFhbgtZPqR8z2Pk992F6I=; b=AkxJ+NOjDm1HNdBdPkQpbjtYGni6qmwWH1Ml89fSooyBh2fz8r5UO9Qz6Z8diJXEAdUg4nsNX086zjErSsWBmtylFwzHi1F6/P0SH1QDMFIfwLslc6qfW7plPS9TZp+wzsQuCJmO3QEK5d5iFRzATzCuqKkhW0EdjCLY/H9lpyM= Received: by hraggstad.unrelenting.technology (OpenSMTPD) with ESMTPSA id 60892eb4 TLS version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO; Wed, 25 Apr 2018 00:15:38 +0000 (UTC) Date: Wed, 25 Apr 2018 03:15:33 +0300 From: Greg V Subject: Re: zfskern{txg_thread_enter} thread using 100% or more CPU To: Steve Wills Cc: FreeBSD Current Message-Id: <1524615333.3550.1@hraggstad.unrelenting.technology> In-Reply-To: References: X-Mailer: geary/0.12.1 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2018 00:22:23 -0000 On Wed, Apr 25, 2018 at 2:30 AM, Steve Wills wrote: > Hi, > > Recently on multiple systems running CURRENT, I've been seeing the > system become unresponsive. Leaving top(1) running has lead me to > notice that when this happens, the system is still responding to ping > and top over ssh is still working, but no new processes can start and > switching to other tasks doesn't work. In top, I do see pid 17, > [zfskern{txg_thread_enter}] monopolizing both CPU usage and disk IO. > Any ideas how to troubleshoot this? It doesn't appear to be a > hardware issue. Hi, Do you have something writing to a gzip compressed dataset? You can use the vfssnoop DTrace script from https://forums.freebsd.org/threads/sharing-of-dtrace-scripts.32855/#post-181816 to see who's writing what. I don't remember if it was exactly txg_thread_enter or whatever, but both CPU and disk sounds a lot like heavily compressed writes. In my case, the Epiphany browser was downloading a large malware database to ~/.config/epiphany/gsb-threats.db :D From owner-freebsd-current@freebsd.org Wed Apr 25 02:19:58 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A686FBAA84; Wed, 25 Apr 2018 02:19:58 +0000 (UTC) (envelope-from kiri@kx.openedu.org) Received: from kx.openedu.org (flets-sg1027.kamome.or.jp [202.216.24.27]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 536E77FF95; Wed, 25 Apr 2018 02:19:56 +0000 (UTC) (envelope-from kiri@kx.openedu.org) Received: from kx.openedu.org (kx.openedu.org [202.216.24.27]) by kx.openedu.org (8.14.5/8.14.5) with ESMTP id w3P2JkPA082841; Wed, 25 Apr 2018 11:19:47 +0900 (JST) (envelope-from kiri@kx.openedu.org) Message-Id: <201804250219.w3P2JkPA082841@kx.openedu.org> Date: Wed, 25 Apr 2018 11:19:46 +0900 From: KIRIYAMA Kazuhiko To: freebsd-current , ports@freebsd.org Cc: kiri@kx.openedu.org User-Agent: Wanderlust/2.14.0 (Africa) SEMI/1.14.6 (Maruoka) FLIM/1.14.9 (=?ISO-8859-4?Q?Goj=F2?=) APEL/10.6 MULE XEmacs/21.4 (patch 22) (Instant Classic) (amd64--freebsd) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2018 02:19:58 -0000 Hi, Today, suddenly pkg or bsdtar failed to core dumped. 'pkg info -aI' crashed with: root@vms:~ # pkg info -aI Segmentation fault (core dumped) root@vms:~ # ll *.core -rw------- 1 root wheel 9555968 Apr 25 10:48 pkg.core root@vms:~ # /usr/libexec/gdb /usr/local/sbin/pkg GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)... (gdb) core pkg.core Core was generated by `pkg info -aI'. Program terminated with signal 11, Segmentation fault. #0 0x0000000800306878 in ?? () (gdb) bt #0 0x0000000800306878 in ?? () #1 0x000000080021307a in ?? () #2 0x0000000800af8980 in ?? () #3 0x0000000800af7c7c in ?? () #4 0x00007fffffffdab8 in ?? () #5 0x0000000800234400 in ?? () #6 0x0000000000000003 in ?? () #7 0x00007fffffffdb60 in ?? () #8 0x00007fffffffdb40 in ?? () #9 0x00000008002155ad in ?? () #10 0x00007fffffffdb60 in ?? () #11 0x0000000000000000 in ?? () (gdb) And 'make extract' in a port: root@vms:/var/ports/msrvms/sysutils/vm-bhyve-devel # make clean ===> Cleaning for vm-bhyve-devel-1.2b root@vms:/var/ports/msrvms/sysutils/vm-bhyve-devel # make deinstall ===> Deinstalling for vm-bhyve-devel ===> vm-bhyve-devel not installed, skipping root@vms:/var/ports/msrvms/sysutils/vm-bhyve-devel # make -de extract ===> License BSD2CLAUSE accepted by the user ===> vm-bhyve-devel-1.2b depends on file: /usr/local/sbin/pkg - found ===> Fetching all distfiles required by vm-bhyve-devel-1.2b for building ===> Extracting for vm-bhyve-devel-1.2b => SHA256 Checksum OK for churchers-vm-bhyve-v1.2b-c9ec4d05f61efeb98b6cd2dc663f859881a12431_GH0.tar.gz. Segmentation fault (core dumped) *** Failed target: do-extract *** Failed command: for file in churchers-vm-bhyve-v1.2b-c9ec4d05f61efeb98b6cd2dc663f859881a12431_GH0.tar.gz; do if ! (cd /var/ports/work/var/ports/msrvms/sysutils/vm-bhyve-devel/work && /usr/bin/tar -xf /var/ports/distfiles//$file --no-same-owner --no-same-permissions); then exit 1; fi; done *** Error code 1 Stop. make: stopped in /var/ports/msrvms/sysutils/vm-bhyve-devel root@vms:/var/ports/msrvms/sysutils/vm-bhyve-devel # cd /var/ports/work/var/ports/msrvms/sysutils/vm-bhyve-devel/work root@vms:/var/ports/work/var/ports/msrvms/sysutils/vm-bhyve-devel/work # ll *.core -rw------- 1 root wheel 9465856 Apr 25 10:53 bsdtar.core root@vms:/var/ports/work/var/ports/msrvms/sysutils/vm-bhyve-devel/work # /usr/libexec/gdb /usr/bin/tar GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... (gdb) core bsdtar.core warning: core file may not match specified executable file. Core was generated by `/usr/bin/tar -xf /var/ports/distfiles//churchers-vm-bhyve-v1.2b-c9ec4d05f61efeb9'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/lib/libarchive.so.7...Reading symbols from /usr/lib/debug//usr/lib/libarchive.so.7.debug...done. done. Loaded symbols for /usr/lib/libarchive.so.7 Reading symbols from /lib/libc.so.7...Reading symbols from /usr/lib/debug//lib/libc.so.7.debug...done. done. Loaded symbols for /lib/libc.so.7 Reading symbols from /lib/libz.so.6...Reading symbols from /usr/lib/debug//lib/libz.so.6.debug...done. done. Loaded symbols for /lib/libz.so.6 Reading symbols from /usr/lib/libbz2.so.4...Reading symbols from /usr/lib/debug//usr/lib/libbz2.so.4.debug...done. done. Loaded symbols for /usr/lib/libbz2.so.4 Reading symbols from /usr/lib/liblzma.so.5...Reading symbols from /usr/lib/debug//usr/lib/liblzma.so.5.debug...done. done. Loaded symbols for /usr/lib/liblzma.so.5 Reading symbols from /lib/libbsdxml.so.4...Reading symbols from /usr/lib/debug//lib/libbsdxml.so.4.debug...done. done. Loaded symbols for /lib/libbsdxml.so.4 Reading symbols from /lib/libcrypto.so.8...Reading symbols from /usr/lib/debug//lib/libcrypto.so.8.debug...done. done. Loaded symbols for /lib/libcrypto.so.8 Reading symbols from /lib/libthr.so.3...Reading symbols from /usr/lib/debug//lib/libthr.so.3.debug...done. done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /libexec/ld-elf.so.1...Reading symbols from /usr/lib/debug//libexec/ld-elf.so.1.debug...done. done. Loaded symbols for /libexec/ld-elf.so.1 #0 _init () at /ds/src/current/12.0/r331153/lib/csu/amd64/crti.S:34 34 subq $8,%rsp [New Thread 801075000 (LWP 100316/)] Current language: auto; currently asm (gdb) bt #0 _init () at /ds/src/current/12.0/r331153/lib/csu/amd64/crti.S:34 #1 0x000000080021a07a in objlist_call_init (list=, lockstate=) at /ds/src/current/12.0/r331153/libexec/rtld-elf/rtld.c:2678 #2 0x0000000800218fcc in _rtld (sp=, exit_proc=0x7fffffffe6d0, objp=0x7fffffffe6d8) at /ds/src/current/12.0/r331153/libexec/rtld-elf/rtld.c:767 #3 0x0000000800217019 in .rtld_start () at /ds/src/current/12.0/r331153/libexec/rtld-elf/amd64/rtld_start.S:39 #4 0x0000000000000000 in ?? () (gdb) root@vms:~ # uname -a FreeBSD vms.pis 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r331153: Tue Mar 20 10:13:56 JST 2018 admin@t.pis:/ds/obj/current/12.0/r331153/ds/src/current/12.0/r331153/amd64.amd64/sys/GENERIC amd64 root@vms:~ # Ports revision is r465515. Waht's happend ? Best regards --- KIRIYAMA Kazuhiko From owner-freebsd-current@freebsd.org Wed Apr 25 19:37:59 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC023FADB36; Wed, 25 Apr 2018 19:37:58 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5DBE473D72; Wed, 25 Apr 2018 19:37:58 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (unknown [127.0.1.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id 3FF111E7D3; Wed, 25 Apr 2018 19:37:58 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id 2C50AD02A; Wed, 25 Apr 2018 19:37:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id JeM4gY8bUbko; Wed, 25 Apr 2018 19:37:54 +0000 (UTC) Subject: Re: Failed buildworld buildkernel: /usr/obj/. . ./arm.armv7/tmp/usr/bin/ld: cannot find -lgcc_s for all_subdir_lib/libdl (a build race?) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.xzibition.com 01AD1D025 To: Mark Millard , freebsd-arm@freebsd.org, freebsd-toolchain@freebsd.org, FreeBSD Current References: From: Bryan Drewery Openpgp: preference=signencrypt Autocrypt: addr=bdrewery@FreeBSD.org; prefer-encrypt=mutual; keydata= xsBNBFJphmsBCADiFgmS4bIzwZijrS31SjEMzg+n5zNellgM+HkShwehpqCiyhXdWrvH6dTZ a6u50pbUIX7doTR7W7PQHCjCTqtpwvcj0eulZva+iHFp+XrbgSFHn+VVXgkYP2MFySyZRFab D2qqzJBEJofhpv4HvY6uQI5K99pMqKr1Z/lHqsijYYu4RH2OfwB5PinId7xeldzWEonVoCr+ rfxzO/UrgA6v/3layGZcKNHFjmc3NqoN1DXtdaEHqtjIozzbndVkH6lkFvIpIrI6i5ox8pwp VxsxLCr/4Musd5CWgHiet5kSw2SzNeA8FbxdLYCpXNVu+uBACEbCUP+CSNy3NVfEUxsBABEB AAHNJEJyeWFuIERyZXdlcnkgPGJkcmV3ZXJ5QEZyZWVCU0Qub3JnPsLAgAQTAQoAKgIbAwUJ CWYBgAULCQgHAwUVCgkICwUWAwIBAAIeAQIXgAUCUmmLqAIZAQAKCRA113G7bkaXz1woB/9j vZ2l1BMa8KR5zv3dk95RzVa4y94ZVHv59/smemCuZdBdb1Z/Lit3NNzhEzEfTv++5gZNh07z 9/G95rpDh9gCUAY3I4m4Joz4khitoCWz608bZ/tHHbS7dmzZ3iE3kl8gRTb9khFAwe8kwlDd jcdlqm1FDoxidRrK+tuFjuIkrOU6nSLk/BWNrEQNYRxoqrqRHrCb9ddwIh8Th6CeBjYMYgbK umFQhxN7cd3mfNuHueiZ7o7m9rnfllVxaPukHjNtcBbc51tmL4bTDsakoBx40LQAhcQ6++1T yE7u9JLgDuztu/EktwvrbSkV10KBPC4lIGm+pxsbfwM9CXXdz66kzsBNBFJphmsBCACiVFPf kNfaFtUSuY0395ueo/rMyHPGPQ2iwvERFCpeFGSQSgagpenNHLpFQKTg/dl6FOoST5tqyxMq fyHGHDzzU51bvA/IfaGoNi/BIhTe/toZNMRvpcI3PLjiGcnJnuwCCbAVOAGdb+t5cZtpNdOI cKYmrYG3u9RiBpe6dTF+qLrD/8Bs1wjhduQ8fcNNgnkXu8xDH4ZxY0lIc3QgvYWp9vimlQe6 iKjUd2/DX28ETZcD5h6pYV331KMPTrEI0p0yvFijUZce8c1XHFyL1j9sBAha5qpszJl6Uq5i LolhKRcGfcdmtD72vHQjUYglUyudSJUVyo2gMYjdbiFKzJulABEBAAHCwGUEGAEKAA8FAlJp hmsCGwwFCQlmAYAACgkQNddxu25Gl89UPggA2mGQp28yCUKsJ6KHFVy/lpHfoQrKF+s7HfKT U2ObVeVNX4I8ZdW1UO48mRqxEOwY8r5YSH6X06OmiqCX2aSMXg3N06/l+ztlB0+UGGlkXBjv l9/nii+bC6b8XWuu0X7Qpb9oYBK9YtoaoyuVplAmjdj/cPou65meKIaS1yDTjHh450DrW8Qg he6l0bFX4BHKTSm99U90ML7EY19B6iI2BZSqWutVsyD71oAREY6NGgDpCOIO6FS41+WeYCDR j8vsa/BiaoX2d2SBDsCwsEwe9fg5PYMi2uVIhvL6OrxnwOdB+TkgvOy5zZSNO29UG/JilZKo Ndz2wpEaUzChGGqLvQ== Organization: FreeBSD Message-ID: <3a09ff03-a214-c8b2-1042-c8d638db1b41@FreeBSD.org> Date: Wed, 25 Apr 2018 12:37:52 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="CqYb8pZ1uYMQbA0A2ji4CQkYiv2sdpdwi" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2018 19:37:59 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --CqYb8pZ1uYMQbA0A2ji4CQkYiv2sdpdwi Content-Type: multipart/mixed; boundary="UR7ptklLQDoJmiQ63PSX4xA7lOWKAzIWG"; protected-headers="v1" From: Bryan Drewery To: Mark Millard , freebsd-arm@freebsd.org, freebsd-toolchain@freebsd.org, FreeBSD Current Message-ID: <3a09ff03-a214-c8b2-1042-c8d638db1b41@FreeBSD.org> Subject: Re: Failed buildworld buildkernel: /usr/obj/. . ./arm.armv7/tmp/usr/bin/ld: cannot find -lgcc_s for all_subdir_lib/libdl (a build race?) References: In-Reply-To: --UR7ptklLQDoJmiQ63PSX4xA7lOWKAzIWG Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 4/21/18 5:58 PM, Mark Millard wrote: > /usr/obj/armv7_clang/arm.armv7/usr/src/arm.armv7/tmp/usr/bin/ld: cannot= find -lgcc_s This is just an annoying build race due to tools/install.sh not respecting -S, as far as I remember. --=20 Regards, Bryan Drewery --UR7ptklLQDoJmiQ63PSX4xA7lOWKAzIWG-- --CqYb8pZ1uYMQbA0A2ji4CQkYiv2sdpdwi Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEE+Rc8ssOq6npcih8JNddxu25Gl88FAlrg2RAACgkQNddxu25G l88bJwgAg6+E1EDgD0mmk/xdQW/B8ToTK0SoKE6QgFwRX4bq4yb9lgC3b2OlHwSA TRUBq9o1dvRFiJpqk8JYvXrXdaR8i+dEWe08qMmBxsNoEgqP5MBVkzQwXG8xnh5c B6YfOCa6LUI7b3enxg2a5cssPal/p3AFryKO5L9eyC5Rd6uYRK4o0GF94uTHlegl k6kLZp+HQ4t8YqflegZfc5hq7iyYGWshApQgNosYnzW9g5/bniPiyZQfaMqamJnS goEm/NC03VBbLpMiA8ocZMsKRMWnUG7MjW7XxBJVEfbZpdVcR564IwTCKrs3+gAu TWUtwm7qTipt3QS+mGdHs5RkuZUHkg== =eTYl -----END PGP SIGNATURE----- --CqYb8pZ1uYMQbA0A2ji4CQkYiv2sdpdwi-- From owner-freebsd-current@freebsd.org Wed Apr 25 19:39:52 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1C75BFADBCB for ; Wed, 25 Apr 2018 19:39:52 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AF08E74EE9 for ; Wed, 25 Apr 2018 19:39:51 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (unknown [127.0.1.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id 7B09A1A7 for ; Wed, 25 Apr 2018 19:39:51 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id B533ED03C for ; Wed, 25 Apr 2018 19:39:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id 95EpoY3eP989 for ; Wed, 25 Apr 2018 19:39:48 +0000 (UTC) Subject: Re: crash on process exit.. current at about r332467 DKIM-Filter: OpenDKIM Filter v2.10.3 mail.xzibition.com 444E1D037 To: freebsd-current References: <9479e941-39be-e6e2-869e-aac475c5e33a@freebsd.org> From: Bryan Drewery Openpgp: preference=signencrypt Autocrypt: addr=bdrewery@FreeBSD.org; prefer-encrypt=mutual; keydata= xsBNBFJphmsBCADiFgmS4bIzwZijrS31SjEMzg+n5zNellgM+HkShwehpqCiyhXdWrvH6dTZ a6u50pbUIX7doTR7W7PQHCjCTqtpwvcj0eulZva+iHFp+XrbgSFHn+VVXgkYP2MFySyZRFab D2qqzJBEJofhpv4HvY6uQI5K99pMqKr1Z/lHqsijYYu4RH2OfwB5PinId7xeldzWEonVoCr+ rfxzO/UrgA6v/3layGZcKNHFjmc3NqoN1DXtdaEHqtjIozzbndVkH6lkFvIpIrI6i5ox8pwp VxsxLCr/4Musd5CWgHiet5kSw2SzNeA8FbxdLYCpXNVu+uBACEbCUP+CSNy3NVfEUxsBABEB AAHNJEJyeWFuIERyZXdlcnkgPGJkcmV3ZXJ5QEZyZWVCU0Qub3JnPsLAgAQTAQoAKgIbAwUJ CWYBgAULCQgHAwUVCgkICwUWAwIBAAIeAQIXgAUCUmmLqAIZAQAKCRA113G7bkaXz1woB/9j vZ2l1BMa8KR5zv3dk95RzVa4y94ZVHv59/smemCuZdBdb1Z/Lit3NNzhEzEfTv++5gZNh07z 9/G95rpDh9gCUAY3I4m4Joz4khitoCWz608bZ/tHHbS7dmzZ3iE3kl8gRTb9khFAwe8kwlDd jcdlqm1FDoxidRrK+tuFjuIkrOU6nSLk/BWNrEQNYRxoqrqRHrCb9ddwIh8Th6CeBjYMYgbK umFQhxN7cd3mfNuHueiZ7o7m9rnfllVxaPukHjNtcBbc51tmL4bTDsakoBx40LQAhcQ6++1T yE7u9JLgDuztu/EktwvrbSkV10KBPC4lIGm+pxsbfwM9CXXdz66kzsBNBFJphmsBCACiVFPf kNfaFtUSuY0395ueo/rMyHPGPQ2iwvERFCpeFGSQSgagpenNHLpFQKTg/dl6FOoST5tqyxMq fyHGHDzzU51bvA/IfaGoNi/BIhTe/toZNMRvpcI3PLjiGcnJnuwCCbAVOAGdb+t5cZtpNdOI cKYmrYG3u9RiBpe6dTF+qLrD/8Bs1wjhduQ8fcNNgnkXu8xDH4ZxY0lIc3QgvYWp9vimlQe6 iKjUd2/DX28ETZcD5h6pYV331KMPTrEI0p0yvFijUZce8c1XHFyL1j9sBAha5qpszJl6Uq5i LolhKRcGfcdmtD72vHQjUYglUyudSJUVyo2gMYjdbiFKzJulABEBAAHCwGUEGAEKAA8FAlJp hmsCGwwFCQlmAYAACgkQNddxu25Gl89UPggA2mGQp28yCUKsJ6KHFVy/lpHfoQrKF+s7HfKT U2ObVeVNX4I8ZdW1UO48mRqxEOwY8r5YSH6X06OmiqCX2aSMXg3N06/l+ztlB0+UGGlkXBjv l9/nii+bC6b8XWuu0X7Qpb9oYBK9YtoaoyuVplAmjdj/cPou65meKIaS1yDTjHh450DrW8Qg he6l0bFX4BHKTSm99U90ML7EY19B6iI2BZSqWutVsyD71oAREY6NGgDpCOIO6FS41+WeYCDR j8vsa/BiaoX2d2SBDsCwsEwe9fg5PYMi2uVIhvL6OrxnwOdB+TkgvOy5zZSNO29UG/JilZKo Ndz2wpEaUzChGGqLvQ== Organization: FreeBSD Message-ID: <8644151e-7dce-1283-e804-93a4b1b7bb0b@FreeBSD.org> Date: Wed, 25 Apr 2018 12:39:47 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <9479e941-39be-e6e2-869e-aac475c5e33a@freebsd.org> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="abooaK8PsUHNk1KuIS7jFSXxx0uoCAv22" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2018 19:39:52 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --abooaK8PsUHNk1KuIS7jFSXxx0uoCAv22 Content-Type: multipart/mixed; boundary="t08czleOs9UmJhmQ2VqEJlCSzQghVkGjy"; protected-headers="v1" From: Bryan Drewery To: freebsd-current Message-ID: <8644151e-7dce-1283-e804-93a4b1b7bb0b@FreeBSD.org> Subject: Re: crash on process exit.. current at about r332467 References: <9479e941-39be-e6e2-869e-aac475c5e33a@freebsd.org> In-Reply-To: <9479e941-39be-e6e2-869e-aac475c5e33a@freebsd.org> --t08czleOs9UmJhmQ2VqEJlCSzQghVkGjy Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 4/23/18 7:50 AM, Julian Elischer wrote: > back trace at:=C2=A0 http://www.freebsd.org/~julian/bob-crash.png >=20 > If anyone wants to take a look.. >=20 > In the exit syscall, while deallocating a vm object. >=20 > I haven't see references to a similar crash in the last 10 days or so..= > But if it rings any bells... >=20 I just hit this on r332455 and have a dump. > panic: Bad link elm 0xfffff811cd920e60 prev->next !=3D elm > cpuid =3D 10 > time =3D 1524682939 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe23f= 450c3b0 > vpanic() at vpanic+0x18d/frame 0xfffffe23f450c410 > panic() at panic+0x43/frame 0xfffffe23f450c470 > vm_object_destroy() at vm_object_destroy/frame 0xfffffe23f450c4d0 > vm_object_deallocate() at vm_object_deallocate+0x45c/frame 0xfffffe23f4= 50c530 > vm_map_process_deferred() at vm_map_process_deferred+0x99/frame 0xfffff= e23f450c560 > vm_map_remove() at vm_map_remove+0xc6/frame 0xfffffe23f450c590 > exec_new_vmspace() at exec_new_vmspace+0x185/frame 0xfffffe23f450c5f0 > exec_elf64_imgact() at exec_elf64_imgact+0x8fb/frame 0xfffffe23f450c6e0= > kern_execve() at kern_execve+0x82c/frame 0xfffffe23f450ca40 > sys_execve() at sys_execve+0x4c/frame 0xfffffe23f450cac0 > amd64_syscall() at amd64_syscall+0x786/frame 0xfffffe23f450cbf0 > fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe23f450= cbf0 > --- syscall (59, FreeBSD ELF64, sys_execve), rip =3D 0x800d7af7a, rsp =3D= 0x7fffffffbd28, rbp =3D 0x7fffffffbe70 --- --=20 Regards, Bryan Drewery --t08czleOs9UmJhmQ2VqEJlCSzQghVkGjy-- --abooaK8PsUHNk1KuIS7jFSXxx0uoCAv22 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEE+Rc8ssOq6npcih8JNddxu25Gl88FAlrg2YMACgkQNddxu25G l88MaAf/XHd6hagj0Hi1XxiBgA+rBE+EXtoJvn8ChwtMyWMJ+raOQ9Cz2A2v4QyT 8MGPm4YNSXdshil0PnFRsP7bFkDL9sxvigJ4AI1nb4tlcQXC264jhWAddSftGBmr 2VsxjZtlvwwh43KYWri0rCp603aCJm31FHbncjM8bqKRCh5/K1xAsiMLxMFMWl4r xPiPKY2n/ttvD5Sx/7hQ4QLO8remjkF9yG3uJ+TNNdZ/t8NvSgOa8C3WMhsBR8Yn z9YkH/zR6QDQDLyZyVGT2KvJ4pXaI1yjRjZgfHjHISWztFx7gQv+OFhF5idyBpsU Wx5lDDi9iP9FDio2US+FgUr0JRX64Q== =1bAM -----END PGP SIGNATURE----- --abooaK8PsUHNk1KuIS7jFSXxx0uoCAv22-- From owner-freebsd-current@freebsd.org Wed Apr 25 19:41:42 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8AFB7FADCB3 for ; Wed, 25 Apr 2018 19:41:42 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2057A751B2 for ; Wed, 25 Apr 2018 19:41:42 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (unknown [127.0.1.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id DEC6B3EE for ; Wed, 25 Apr 2018 19:41:41 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id 1CB65D042 for ; Wed, 25 Apr 2018 19:41:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id 52CIJRwE7AJ6 for ; Wed, 25 Apr 2018 19:41:38 +0000 (UTC) Subject: Re: crash on process exit.. current at about r332467 DKIM-Filter: OpenDKIM Filter v2.10.3 mail.xzibition.com 5E31CD03D From: Bryan Drewery To: freebsd-current References: <9479e941-39be-e6e2-869e-aac475c5e33a@freebsd.org> <8644151e-7dce-1283-e804-93a4b1b7bb0b@FreeBSD.org> Openpgp: preference=signencrypt Autocrypt: addr=bdrewery@FreeBSD.org; prefer-encrypt=mutual; keydata= xsBNBFJphmsBCADiFgmS4bIzwZijrS31SjEMzg+n5zNellgM+HkShwehpqCiyhXdWrvH6dTZ a6u50pbUIX7doTR7W7PQHCjCTqtpwvcj0eulZva+iHFp+XrbgSFHn+VVXgkYP2MFySyZRFab D2qqzJBEJofhpv4HvY6uQI5K99pMqKr1Z/lHqsijYYu4RH2OfwB5PinId7xeldzWEonVoCr+ rfxzO/UrgA6v/3layGZcKNHFjmc3NqoN1DXtdaEHqtjIozzbndVkH6lkFvIpIrI6i5ox8pwp VxsxLCr/4Musd5CWgHiet5kSw2SzNeA8FbxdLYCpXNVu+uBACEbCUP+CSNy3NVfEUxsBABEB AAHNJEJyeWFuIERyZXdlcnkgPGJkcmV3ZXJ5QEZyZWVCU0Qub3JnPsLAgAQTAQoAKgIbAwUJ CWYBgAULCQgHAwUVCgkICwUWAwIBAAIeAQIXgAUCUmmLqAIZAQAKCRA113G7bkaXz1woB/9j vZ2l1BMa8KR5zv3dk95RzVa4y94ZVHv59/smemCuZdBdb1Z/Lit3NNzhEzEfTv++5gZNh07z 9/G95rpDh9gCUAY3I4m4Joz4khitoCWz608bZ/tHHbS7dmzZ3iE3kl8gRTb9khFAwe8kwlDd jcdlqm1FDoxidRrK+tuFjuIkrOU6nSLk/BWNrEQNYRxoqrqRHrCb9ddwIh8Th6CeBjYMYgbK umFQhxN7cd3mfNuHueiZ7o7m9rnfllVxaPukHjNtcBbc51tmL4bTDsakoBx40LQAhcQ6++1T yE7u9JLgDuztu/EktwvrbSkV10KBPC4lIGm+pxsbfwM9CXXdz66kzsBNBFJphmsBCACiVFPf kNfaFtUSuY0395ueo/rMyHPGPQ2iwvERFCpeFGSQSgagpenNHLpFQKTg/dl6FOoST5tqyxMq fyHGHDzzU51bvA/IfaGoNi/BIhTe/toZNMRvpcI3PLjiGcnJnuwCCbAVOAGdb+t5cZtpNdOI cKYmrYG3u9RiBpe6dTF+qLrD/8Bs1wjhduQ8fcNNgnkXu8xDH4ZxY0lIc3QgvYWp9vimlQe6 iKjUd2/DX28ETZcD5h6pYV331KMPTrEI0p0yvFijUZce8c1XHFyL1j9sBAha5qpszJl6Uq5i LolhKRcGfcdmtD72vHQjUYglUyudSJUVyo2gMYjdbiFKzJulABEBAAHCwGUEGAEKAA8FAlJp hmsCGwwFCQlmAYAACgkQNddxu25Gl89UPggA2mGQp28yCUKsJ6KHFVy/lpHfoQrKF+s7HfKT U2ObVeVNX4I8ZdW1UO48mRqxEOwY8r5YSH6X06OmiqCX2aSMXg3N06/l+ztlB0+UGGlkXBjv l9/nii+bC6b8XWuu0X7Qpb9oYBK9YtoaoyuVplAmjdj/cPou65meKIaS1yDTjHh450DrW8Qg he6l0bFX4BHKTSm99U90ML7EY19B6iI2BZSqWutVsyD71oAREY6NGgDpCOIO6FS41+WeYCDR j8vsa/BiaoX2d2SBDsCwsEwe9fg5PYMi2uVIhvL6OrxnwOdB+TkgvOy5zZSNO29UG/JilZKo Ndz2wpEaUzChGGqLvQ== Organization: FreeBSD Message-ID: Date: Wed, 25 Apr 2018 12:41:37 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <8644151e-7dce-1283-e804-93a4b1b7bb0b@FreeBSD.org> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="XrXTZQPqT7qonL4KyhQAbkofj9ZnNHGLl" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2018 19:41:42 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --XrXTZQPqT7qonL4KyhQAbkofj9ZnNHGLl Content-Type: multipart/mixed; boundary="q07P0RfWC0x9DqdfKOtoNi9gPYJFoA94D"; protected-headers="v1" From: Bryan Drewery To: freebsd-current Message-ID: Subject: Re: crash on process exit.. current at about r332467 References: <9479e941-39be-e6e2-869e-aac475c5e33a@freebsd.org> <8644151e-7dce-1283-e804-93a4b1b7bb0b@FreeBSD.org> In-Reply-To: <8644151e-7dce-1283-e804-93a4b1b7bb0b@FreeBSD.org> --q07P0RfWC0x9DqdfKOtoNi9gPYJFoA94D Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 4/25/18 12:39 PM, Bryan Drewery wrote: > On 4/23/18 7:50 AM, Julian Elischer wrote: >> back trace at:=C2=A0 http://www.freebsd.org/~julian/bob-crash.png >> >> If anyone wants to take a look.. >> >> In the exit syscall, while deallocating a vm object. >> >> I haven't see references to a similar crash in the last 10 days or so.= =2E >> But if it rings any bells... >> >=20 > I just hit this on r332455 and have a dump. >=20 >> panic: Bad link elm 0xfffff811cd920e60 prev->next !=3D elm >> cpuid =3D 10 >> time =3D 1524682939 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe23= f450c3b0 >> vpanic() at vpanic+0x18d/frame 0xfffffe23f450c410 >> panic() at panic+0x43/frame 0xfffffe23f450c470 >> vm_object_destroy() at vm_object_destroy/frame 0xfffffe23f450c4d0 >> vm_object_deallocate() at vm_object_deallocate+0x45c/frame 0xfffffe23f= 450c530 >> vm_map_process_deferred() at vm_map_process_deferred+0x99/frame 0xffff= fe23f450c560 >> vm_map_remove() at vm_map_remove+0xc6/frame 0xfffffe23f450c590 >> exec_new_vmspace() at exec_new_vmspace+0x185/frame 0xfffffe23f450c5f0 >> exec_elf64_imgact() at exec_elf64_imgact+0x8fb/frame 0xfffffe23f450c6e= 0 >> kern_execve() at kern_execve+0x82c/frame 0xfffffe23f450ca40 >> sys_execve() at sys_execve+0x4c/frame 0xfffffe23f450cac0 >> amd64_syscall() at amd64_syscall+0x786/frame 0xfffffe23f450cbf0 >> fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe23f45= 0cbf0 >> --- syscall (59, FreeBSD ELF64, sys_execve), rip =3D 0x800d7af7a, rsp = =3D 0x7fffffffbd28, rbp =3D 0x7fffffffbe70 --- >=20 >=20 >=20 It's a different stack than Julian's but it seems like the same issue to = me. --=20 Regards, Bryan Drewery --q07P0RfWC0x9DqdfKOtoNi9gPYJFoA94D-- --XrXTZQPqT7qonL4KyhQAbkofj9ZnNHGLl Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEE+Rc8ssOq6npcih8JNddxu25Gl88FAlrg2fEACgkQNddxu25G l89EZgf8Ca/haSUNmxU+Xq6cYacl19+4AFizBi6FcNw+QYWQ4EOEhNl0zSig5W4V cCUwkXcx18ULMBWcax4rFMBDi8TPXYPNPFtB8Pr+DNUWycN4f68jpLqnRxo5htlt QeKbS/uq2tPkon7ALBRRlUFRFoUZGtk/ptOCSocSMr6SCT6jkgF/FVcf7lR6GSSn 0q/mt9s8rZ9x4xN78/Atczvi01uLmDxSbLpJy/vUkGrrcYp+E6XdJPgKfne07BcF d56AApE7KiSgL1ipP6YhmylL1jEmScG0jnfr43HGHYP+NcDyvKo2Ye8URs8WqaAo RAOs658bQhF7EE4/WNt0KrLEbwB73Q== =/3P0 -----END PGP SIGNATURE----- --XrXTZQPqT7qonL4KyhQAbkofj9ZnNHGLl-- From owner-freebsd-current@freebsd.org Wed Apr 25 20:04:16 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABADBFAEC04 for ; Wed, 25 Apr 2018 20:04:16 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 36A967AAC4 for ; Wed, 25 Apr 2018 20:04:16 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id EF0E7FAEC02; Wed, 25 Apr 2018 20:04:15 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DCCF6FAEC01; Wed, 25 Apr 2018 20:04:15 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-lf0-x230.google.com (mail-lf0-x230.google.com [IPv6:2a00:1450:4010:c07::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 549D77AAA3; Wed, 25 Apr 2018 20:04:15 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-lf0-x230.google.com with SMTP id q5-v6so27248595lff.12; Wed, 25 Apr 2018 13:04:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=3sCM6E5N4KvBh7rCULFbBykZcPC8ujEmMoOZncpve1E=; b=rvNgNFRINHHCMF8YX2TNllAXuL/L1C6r1j6txuUCIA6ZNCozh+OJoxTxlcPgjKN51E r1/j0D5zRuFn/+mmR7geapji+uoglv+rcGmtZKgDRuafy1VIC4xU/zIOhCDKALBlKIiB O7bvAB+VPip9wuO8YnEM+w1KVfpRw3mnOTjYT70AKdyL1yY5Qxk7pbKwNPOJF9kRD/kp Dw4cDFP25au9cMQZHXOjDsgmn6ItYenXxNrektPnFhf0gSOYGB8oCjlWf0cYA4X1t65i IwW9eKi7j/r8UInANnwXc0s671pjrjxRpHDugqr25of9cvMfv6h4by52HlGLKZr7yxPX M5Xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=3sCM6E5N4KvBh7rCULFbBykZcPC8ujEmMoOZncpve1E=; b=Ei5Pfc1zEtZJxLb0+w3e0VDL3+xENDirtz/D6nek+evGRkCB1oigbl1SRZg3vMPSAq w7gmqcXzRxgCrgd1Y9GJfRtbUL47iv1GH5fw1/s+932d96KKuOri1zCo08fCoW6dXpIT 0a2jCWajSc2hUo0prTCPMrfNQexr3sZaPmCc5bbk35KXekf/nNBwmeFfnkhA2RTCZ+dg 2putlkz3H/WPTRbbwbKs5am4I1LiTt9rvZ4YxoQ13sQPGWnEhgmnK/HfyKoWjKTQcHc8 gSHlMBndAdoRraJKlXjh0PKc6cHvrTAFW1o5pFX/jtSiMvORXtQrCfiV2SC++TXgxTAO QJSw== X-Gm-Message-State: ALQs6tCiqikWIl63lII6qPihFJkfjvg9ouheQYjgkqlvPk1RVAUZ8FfA Zh7RM5+tvitXFvZ0vnDJKfWL4QaT/DOflTyafiqgQw== X-Google-Smtp-Source: AIpwx49b9udlSrZuQjIneuDnTsSUZMEzf05LCvbETc1cvEISt2IYhTS+VEJoPVnXsauvuVZ9rSo/p8uJRuT/Hfczwi4= X-Received: by 10.46.150.85 with SMTP id z21mr14873442ljh.127.1524686653917; Wed, 25 Apr 2018 13:04:13 -0700 (PDT) MIME-Version: 1.0 Received: by 10.46.135.75 with HTTP; Wed, 25 Apr 2018 13:04:13 -0700 (PDT) In-Reply-To: <20180424085553.GA6887@kib.kiev.ua> References: <20180424085553.GA6887@kib.kiev.ua> From: Ryan Stone Date: Wed, 25 Apr 2018 16:04:13 -0400 Message-ID: Subject: Re: mlx5(4) jumbo receive To: Konstantin Belousov Cc: FreeBSD Current , menny@mellanox.com, FreeBSD Net Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2018 20:04:16 -0000 On Tue, Apr 24, 2018 at 4:55 AM, Konstantin Belousov wrote: > +#ifndef MLX5E_MAX_RX_BYTES > +#define MLX5E_MAX_RX_BYTES MCLBYTES > +#endif Why do you use a 2KB buffer rather than a PAGE_SIZE'd buffer? MJUMPAGESIZE should offer significantly better performance for jumbo frames without increasing the risk of memory fragmentation. From owner-freebsd-current@freebsd.org Wed Apr 25 20:35:27 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 24815FAF8CC for ; Wed, 25 Apr 2018 20:35:27 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B93318273D for ; Wed, 25 Apr 2018 20:35:26 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (unknown [127.0.1.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id 7EB9DFD77 for ; Wed, 25 Apr 2018 20:35:26 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id 75A5BD10B for ; Wed, 25 Apr 2018 20:35:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id 131zBYt6PNsb for ; Wed, 25 Apr 2018 20:35:22 +0000 (UTC) Subject: Re: crash on process exit.. current at about r332467 DKIM-Filter: OpenDKIM Filter v2.10.3 mail.xzibition.com 42435D106 From: Bryan Drewery To: freebsd-current References: <9479e941-39be-e6e2-869e-aac475c5e33a@freebsd.org> <8644151e-7dce-1283-e804-93a4b1b7bb0b@FreeBSD.org> Openpgp: preference=signencrypt Autocrypt: addr=bdrewery@FreeBSD.org; prefer-encrypt=mutual; keydata= xsBNBFJphmsBCADiFgmS4bIzwZijrS31SjEMzg+n5zNellgM+HkShwehpqCiyhXdWrvH6dTZ a6u50pbUIX7doTR7W7PQHCjCTqtpwvcj0eulZva+iHFp+XrbgSFHn+VVXgkYP2MFySyZRFab D2qqzJBEJofhpv4HvY6uQI5K99pMqKr1Z/lHqsijYYu4RH2OfwB5PinId7xeldzWEonVoCr+ rfxzO/UrgA6v/3layGZcKNHFjmc3NqoN1DXtdaEHqtjIozzbndVkH6lkFvIpIrI6i5ox8pwp VxsxLCr/4Musd5CWgHiet5kSw2SzNeA8FbxdLYCpXNVu+uBACEbCUP+CSNy3NVfEUxsBABEB AAHNJEJyeWFuIERyZXdlcnkgPGJkcmV3ZXJ5QEZyZWVCU0Qub3JnPsLAgAQTAQoAKgIbAwUJ CWYBgAULCQgHAwUVCgkICwUWAwIBAAIeAQIXgAUCUmmLqAIZAQAKCRA113G7bkaXz1woB/9j vZ2l1BMa8KR5zv3dk95RzVa4y94ZVHv59/smemCuZdBdb1Z/Lit3NNzhEzEfTv++5gZNh07z 9/G95rpDh9gCUAY3I4m4Joz4khitoCWz608bZ/tHHbS7dmzZ3iE3kl8gRTb9khFAwe8kwlDd jcdlqm1FDoxidRrK+tuFjuIkrOU6nSLk/BWNrEQNYRxoqrqRHrCb9ddwIh8Th6CeBjYMYgbK umFQhxN7cd3mfNuHueiZ7o7m9rnfllVxaPukHjNtcBbc51tmL4bTDsakoBx40LQAhcQ6++1T yE7u9JLgDuztu/EktwvrbSkV10KBPC4lIGm+pxsbfwM9CXXdz66kzsBNBFJphmsBCACiVFPf kNfaFtUSuY0395ueo/rMyHPGPQ2iwvERFCpeFGSQSgagpenNHLpFQKTg/dl6FOoST5tqyxMq fyHGHDzzU51bvA/IfaGoNi/BIhTe/toZNMRvpcI3PLjiGcnJnuwCCbAVOAGdb+t5cZtpNdOI cKYmrYG3u9RiBpe6dTF+qLrD/8Bs1wjhduQ8fcNNgnkXu8xDH4ZxY0lIc3QgvYWp9vimlQe6 iKjUd2/DX28ETZcD5h6pYV331KMPTrEI0p0yvFijUZce8c1XHFyL1j9sBAha5qpszJl6Uq5i LolhKRcGfcdmtD72vHQjUYglUyudSJUVyo2gMYjdbiFKzJulABEBAAHCwGUEGAEKAA8FAlJp hmsCGwwFCQlmAYAACgkQNddxu25Gl89UPggA2mGQp28yCUKsJ6KHFVy/lpHfoQrKF+s7HfKT U2ObVeVNX4I8ZdW1UO48mRqxEOwY8r5YSH6X06OmiqCX2aSMXg3N06/l+ztlB0+UGGlkXBjv l9/nii+bC6b8XWuu0X7Qpb9oYBK9YtoaoyuVplAmjdj/cPou65meKIaS1yDTjHh450DrW8Qg he6l0bFX4BHKTSm99U90ML7EY19B6iI2BZSqWutVsyD71oAREY6NGgDpCOIO6FS41+WeYCDR j8vsa/BiaoX2d2SBDsCwsEwe9fg5PYMi2uVIhvL6OrxnwOdB+TkgvOy5zZSNO29UG/JilZKo Ndz2wpEaUzChGGqLvQ== Organization: FreeBSD Message-ID: <7de7c555-26f0-6b7d-f72f-37a17adea52c@FreeBSD.org> Date: Wed, 25 Apr 2018 13:35:20 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="PF4398TqiUJzExLaa4PAdBmjvZtivQfPi" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2018 20:35:27 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --PF4398TqiUJzExLaa4PAdBmjvZtivQfPi Content-Type: multipart/mixed; boundary="GUR4RNdWSuAvgiP6LeJNzkRnpezODOuET"; protected-headers="v1" From: Bryan Drewery To: freebsd-current Message-ID: <7de7c555-26f0-6b7d-f72f-37a17adea52c@FreeBSD.org> Subject: Re: crash on process exit.. current at about r332467 References: <9479e941-39be-e6e2-869e-aac475c5e33a@freebsd.org> <8644151e-7dce-1283-e804-93a4b1b7bb0b@FreeBSD.org> In-Reply-To: --GUR4RNdWSuAvgiP6LeJNzkRnpezODOuET Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 4/25/18 12:41 PM, Bryan Drewery wrote: > On 4/25/18 12:39 PM, Bryan Drewery wrote: >> On 4/23/18 7:50 AM, Julian Elischer wrote: >>> back trace at:=C2=A0 http://www.freebsd.org/~julian/bob-crash.png >>> >>> If anyone wants to take a look.. >>> >>> In the exit syscall, while deallocating a vm object. >>> >>> I haven't see references to a similar crash in the last 10 days or so= =2E. >>> But if it rings any bells... >>> >> >> I just hit this on r332455 and have a dump. >> >>> panic: Bad link elm 0xfffff811cd920e60 prev->next !=3D elm >>> cpuid =3D 10 >>> time =3D 1524682939 >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe2= 3f450c3b0 >>> vpanic() at vpanic+0x18d/frame 0xfffffe23f450c410 >>> panic() at panic+0x43/frame 0xfffffe23f450c470 >>> vm_object_destroy() at vm_object_destroy/frame 0xfffffe23f450c4d0 >>> vm_object_deallocate() at vm_object_deallocate+0x45c/frame 0xfffffe23= f450c530 >>> vm_map_process_deferred() at vm_map_process_deferred+0x99/frame 0xfff= ffe23f450c560 >>> vm_map_remove() at vm_map_remove+0xc6/frame 0xfffffe23f450c590 >>> exec_new_vmspace() at exec_new_vmspace+0x185/frame 0xfffffe23f450c5f0= >>> exec_elf64_imgact() at exec_elf64_imgact+0x8fb/frame 0xfffffe23f450c6= e0 >>> kern_execve() at kern_execve+0x82c/frame 0xfffffe23f450ca40 >>> sys_execve() at sys_execve+0x4c/frame 0xfffffe23f450cac0 >>> amd64_syscall() at amd64_syscall+0x786/frame 0xfffffe23f450cbf0 >>> fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe23f4= 50cbf0 >>> --- syscall (59, FreeBSD ELF64, sys_execve), rip =3D 0x800d7af7a, rsp= =3D 0x7fffffffbd28, rbp =3D 0x7fffffffbe70 --- >> >> >> >=20 > It's a different stack than Julian's but it seems like the same issue t= o me. >=20 Here's the real stack: > #12 0xffffffff80b6d1c3 in panic (fmt=3D0xffffffff81dee958 = "\322\237'\201\377\377\377\377") at /usr/src/sys/kern/kern_shutdown.c:764= > #13 0xffffffff80eac060 in vm_object_terminate (object=3D0xfffff80587c4d= e00) at /usr/src/sys/vm/vm_object.c:868 > #14 0xffffffff80eaaf2c in vm_object_deallocate (object=3D0xfffff80587c4= de00) at /usr/src/sys/vm/vm_object.c:684 > #15 0xffffffff80ea0089 in vm_map_entry_deallocate (system_map=3D, entry=3D) at /usr/src/sys/vm/vm_map.c:2997 > #16 vm_map_process_deferred () at /usr/src/sys/vm/vm_map.c:541 > #17 0xffffffff80ea5186 in _vm_map_unlock (map=3D, file=3D= , line=3D3189) at /usr/src/sys/vm/vm_map.c:554 > #18 vm_map_remove (map=3D, start=3D4096, end=3D140737488= 355328) at /usr/src/sys/vm/vm_map.c:3189 > #19 0xffffffff80b24c35 in exec_new_vmspace (imgp=3D0xfffffe23f450c8b0, = sv=3D) at /usr/src/sys/kern/kern_exec.c:1099 > #20 0xffffffff80afaf1b in exec_elf64_imgact (imgp=3D) at= /usr/src/sys/kern/imgact_elf.c:922 > #21 0xffffffff80b2380c in do_execve (td=3D, args=3D, mac_p=3D) at /usr/src/sys/kern/kern_exec.c:606= > #22 kern_execve (td=3D, args=3D, mac_p=3D= ) at /usr/src/sys/kern/kern_exec.c:351 > #23 0xffffffff80b22c9c in sys_execve (td=3D0xfffff801493c6000, uap=3D0x= fffff801493c63c0) at /usr/src/sys/kern/kern_exec.c:225 > (kgdb) frame 13 > #13 0xffffffff80eac060 in vm_object_terminate (object=3D0xfffff80587c4d= e00) at /usr/src/sys/vm/vm_object.c:868 > 868 KASSERT(object->cred =3D=3D NULL || object->type =3D=3D= OBJT_DEFAULT || > (kgdb) p *object > $2 =3D {lock =3D {lock_object =3D {lo_name =3D 0xffffffff81214cff "vm o= bject", lo_flags =3D 627245056, lo_data =3D 0, lo_witness =3D 0xfffff8123= fd6a700}, rw_lock =3D 18446735283140190208}, object_list =3D { > tqe_next =3D 0xfffff80587c67000, tqe_prev =3D 0xfffff80587c4dd20}, = shadow_head =3D {lh_first =3D 0x0}, shadow_list =3D {le_next =3D 0xfffff8= 0a9606c500, le_prev =3D 0xfffff8054f0c2c30}, memq =3D { > tqh_first =3D 0xfffff811d00c9980, tqh_last =3D 0xfffff811d850b8a0},= rtree =3D {rt_root =3D 18446735322516082688}, size =3D 2048, domain =3D = {dr_policy =3D 0x0, dr_iterator =3D 0}, generation =3D 1, ref_count =3D 0= , > shadow_count =3D 0, memattr =3D 6 '\006', type =3D 0 '\000', flags =3D= 12296, pg_color =3D 1024, paging_in_progress =3D 0, resident_page_count = =3D 5, backing_object =3D 0x0, backing_object_offset =3D 0, > pager_object_list =3D {tqe_next =3D 0x0, tqe_prev =3D 0x0}, rvq =3D {= lh_first =3D 0xfffff811cbac2240}, handle =3D 0x0, un_pager =3D {vnp =3D {= vnp_size =3D 0, writemappings =3D 0}, devp =3D {devp_pglist =3D { > tqh_first =3D 0x0, tqh_last =3D 0x0}, ops =3D 0x0, dev =3D 0x0}= , sgp =3D {sgp_pglist =3D {tqh_first =3D 0x0, tqh_last =3D 0x0}}, swp =3D= {swp_tmpfs =3D 0x0, swp_blks =3D {pt_root =3D 0}}}, cred =3D 0xfffff807e= bd91500, > charge =3D 8388608, umtx_data =3D 0x0} object->type =3D OBJT_DEFAULT. Er I'm not sure what's going on here as line 868 is a totally different assert than the queue(3) one in the msgbuf... > 868 KASSERT(object->cred =3D=3D NULL || object->type =3D=3D OBJ= T_DEFAULT || > 869 object->type =3D=3D OBJT_SWAP, > 870 ("%s: non-swap obj %p has cred", __func__, object)); --=20 Regards, Bryan Drewery --GUR4RNdWSuAvgiP6LeJNzkRnpezODOuET-- --PF4398TqiUJzExLaa4PAdBmjvZtivQfPi Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEE+Rc8ssOq6npcih8JNddxu25Gl88FAlrg5okACgkQNddxu25G l8/lnAgAvp4Hc6dL1GIb2F94XQQLZ8rDkGeuAbC6sH2QFhXKFDX/TCxYWkhTY2Lu a/W8WnntrfdrCxnc78jWDA/ssPyeExK/Vz0i0VBka8BOJBX/wx3q9cw2zK+2dnOf zA3QaZBzEKJNoXRAiXkNUQnZL4fZjtF8gf+9ENzEC41G1L6EALkyCfqxBD65Z+Hc e0T3fwMStfCKmeEB0JAc3v54/b/+2ZC8Fr+E9E4SgS6wIL9HK+xFgVKMYA8qQOU+ gXwZ3Nogt0Su7Aac27WSvsHNNQpBZ8NLH7Am9nlXX1Y6gc4iMjMX/e/+/LTpBRZO oT0Ypr4t4aa6ZG7sW1Rs6tFvzuJzMw== =U1+M -----END PGP SIGNATURE----- --PF4398TqiUJzExLaa4PAdBmjvZtivQfPi-- From owner-freebsd-current@freebsd.org Wed Apr 25 22:21:47 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 44435FB1CC3 for ; Wed, 25 Apr 2018 22:21:47 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1641F78967 for ; Wed, 25 Apr 2018 22:21:46 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: by mailman.ysv.freebsd.org (Postfix) id C8FF9FB1CBC; Wed, 25 Apr 2018 22:21:45 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8A8CAFB1CBB; Wed, 25 Apr 2018 22:21:45 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660054.outbound.protection.outlook.com [40.107.66.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1D12D78960; Wed, 25 Apr 2018 22:21:44 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM (52.132.66.153) by YQBPR0101MB0897.CANPRD01.PROD.OUTLOOK.COM (52.132.66.12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.715.18; Wed, 25 Apr 2018 22:21:43 +0000 Received: from YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::893c:efc2:d71f:945a]) by YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM ([fe80::893c:efc2:d71f:945a%13]) with mapi id 15.20.0715.018; Wed, 25 Apr 2018 22:21:43 +0000 From: Rick Macklem To: Ryan Stone , Konstantin Belousov CC: FreeBSD Current , "menny@mellanox.com" , FreeBSD Net Subject: Re: mlx5(4) jumbo receive Thread-Topic: mlx5(4) jumbo receive Thread-Index: AQHT3NCdClq8g0H1WkiFu85ReYNX16QSDLEp Date: Wed, 25 Apr 2018 22:21:43 +0000 Message-ID: References: <20180424085553.GA6887@kib.kiev.ua>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YQBPR0101MB0897; 7:CyvMySIgonstN1+bHIQ37R8cTLaibTVBFHfSoJASehLM87egVEc01UMhzzKKlcaIcmmzN8YInNn1wWS7Cp7lVTGySfslbuuM+kZtKxujN8oRk3yjOmxl8vEhvaYStbixezmDJBEgRRZm8FvtDCfQrEPYCT116LDgwwEq0IimpQQOhpjyf25UURndOD72/9usKxOtczZYRko3uUh76Nd6GL/bPWDLNOc65iMUcgl8llg2Hr8Vshb6aJlZ2xLDvYId x-ms-exchange-antispam-srfa-diagnostics: SOS; x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(8989080)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(8990040)(2017052603328)(7153060)(7193020); SRVR:YQBPR0101MB0897; x-ms-traffictypediagnostic: YQBPR0101MB0897: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(85827821059158); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(3231232)(944501410)(52105095)(3002001)(6041310)(20161123562045)(20161123558120)(20161123560045)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(6072148)(201708071742011); SRVR:YQBPR0101MB0897; BCL:0; PCL:0; RULEID:; SRVR:YQBPR0101MB0897; x-forefront-prvs: 06530126A4 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(346002)(396003)(376002)(39860400002)(39380400002)(199004)(189003)(305945005)(229853002)(2906002)(2900100001)(59450400001)(3280700002)(9686003)(53936002)(55016002)(3660700001)(786003)(86362001)(99286004)(26005)(486006)(11346002)(316002)(102836004)(7696005)(6506007)(186003)(76176011)(476003)(68736007)(54906003)(446003)(110136005)(8936002)(81166006)(5250100002)(81156014)(8676002)(97736004)(106356001)(33656002)(74482002)(39060400002)(4326008)(105586002)(25786009)(6246003)(6436002)(5660300001)(74316002)(14454004)(478600001); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB0897; H:YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: fVqqgwQGcL4uONKMbnDsW3NdInPUopTg2DUHQ2GaZp1odsc5NmvYZFq3ZXxiBK/OVJnQyXHjTC3LUSUMbkR0VlfWhIWf5yXoRfAAnhMKPZN4l1yjlasBRDZrrJu9Rv6ridB4L8eisIHYeO8LXyRl+VWr4W9GUu0yEkiqn33BaWdSNPIWXGtLnyJibf0HNGCu spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 64862c4f-1c94-4c4b-c3f4-08d5aafaecc6 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 64862c4f-1c94-4c4b-c3f4-08d5aafaecc6 X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Apr 2018 22:21:43.3007 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB0897 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2018 22:21:47 -0000 Ryan Stone wrote: >On Tue, Apr 24, 2018 at 4:55 AM, Konstantin Belousov >>wrote: >> +#ifndef MLX5E_MAX_RX_BYTES >> +#define MLX5E_MAX_RX_BYTES MCLBYTES >> +#endif > >Why do you use a 2KB buffer rather than a PAGE_SIZE'd buffer? >MJUMPAGESIZE should offer significantly better performance for jumbo >frames without increasing the risk of memory fragmentation. Actually, when I was playing with using jumbo mbuf clusters for NFS, I was = able to get it to fragment to the point where allocations failed when mixing 2K = and 4K mbuf clusters. Admittedly I was using a 256Mbyte i386 and it wasn't easily reproduced, but it was possible. --> Using a mix of 2K and 4K mbuf clusters can result in fragmentation, alt= hough I suspect that it isn't nearly as serious as what can happen when usi= ng 9K mbuf clusters. rick= From owner-freebsd-current@freebsd.org Thu Apr 26 08:03:11 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BB5B6FA5295 for ; Thu, 26 Apr 2018 08:03:11 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 513D67C6E6 for ; Thu, 26 Apr 2018 08:03:11 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 12E5AFA5292; Thu, 26 Apr 2018 08:03:11 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F30B6FA5291; Thu, 26 Apr 2018 08:03:10 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 65A757C6C5; Thu, 26 Apr 2018 08:03:10 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w3Q82xWi023845 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 26 Apr 2018 11:03:02 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w3Q82xWi023845 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w3Q82w9l023844; Thu, 26 Apr 2018 11:02:58 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 26 Apr 2018 11:02:58 +0300 From: Konstantin Belousov To: Ryan Stone Cc: FreeBSD Current , menyy@mellanox.com, FreeBSD Net Subject: Re: mlx5(4) jumbo receive Message-ID: <20180426080258.GI6887@kib.kiev.ua> References: <20180424085553.GA6887@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.5 (2018-04-13) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2018 08:03:11 -0000 On Wed, Apr 25, 2018 at 04:04:13PM -0400, Ryan Stone wrote: > On Tue, Apr 24, 2018 at 4:55 AM, Konstantin Belousov > wrote: > > +#ifndef MLX5E_MAX_RX_BYTES > > +#define MLX5E_MAX_RX_BYTES MCLBYTES > > +#endif > > Why do you use a 2KB buffer rather than a PAGE_SIZE'd buffer? > MJUMPAGESIZE should offer significantly better performance for jumbo > frames without increasing the risk of memory fragmentation. Part of the answer is that the patch was not written in one go (even not by one person), but evolved, and this is how it shaped. Another part is that indeed, as Rick stated, I am not sure about mixing the different sizes for mbuf allocator. This might be more FUD than factual-based considerations, but still. I believe that the patch as is provides the important improvements. If developing mlx4(4) change of the same nature, I will probably take this into the exp stage from the beginning. For mlx5(4), I think that the patch should be applied as is, then I might experiment with PAGE_SIZE as the later step. From owner-freebsd-current@freebsd.org Thu Apr 26 09:54:25 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D2A17FA857B for ; Thu, 26 Apr 2018 09:54:24 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f42.google.com (mail-lf0-f42.google.com [209.85.215.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 31DAD7975B for ; Thu, 26 Apr 2018 09:54:24 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f42.google.com with SMTP id r125-v6so29792172lfe.2 for ; Thu, 26 Apr 2018 02:54:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=BxZ7DxrLEXkxGcCn3e/ohmqqsEuNlvZv6TjiSK7iK0I=; b=Cu/SohLUZKwMVjVLt9F03guzkE2Ohuma7sPtdBUXkmGgAgW+q7+Z9nsp5sgpUhzm5e IEnXguKfBmZcuraSOZp8XKfa+ZfqvIUsD/mWevkYK1/S1En0RX2pKHXEdDA2POhpPuSh VzGdBKcRHhK7cftk2/2eG+V0o398+egch4NsJwGIcSHRLJXz2x7b+IFUnwkyRVCwqnuW QmULj8Cj75MOY7IF9tvYjZc++PieA5RnF1H/mQYGj7Vu27Xk7g3cvqKp5E+zqOujClW0 vWchdvM6H1b9fWuv261jZUKkLqdwfEjktH/3BqTyzH5tTbyzVtOdj6CZb/WcDIc/eWsK MJvA== X-Gm-Message-State: ALQs6tApx/GCdnKsDixiyG6UvpBq31f8/n3w1i+V0gCy7sctc3rFDq0r XfnSQFZkfImcXxxLQHNx295otYK2 X-Google-Smtp-Source: AB8JxZqjMWturTiufZghl+0uKTe3o0ypgOWE1VcMuvdWJHyfcWcToMoMV9mgn6bwancz33gJ4PQ28w== X-Received: by 2002:a19:9685:: with SMTP id y127-v6mr16956475lfd.77.1524736456344; Thu, 26 Apr 2018 02:54:16 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id u2sm1204335lji.4.2018.04.26.02.54.15 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Apr 2018 02:54:15 -0700 (PDT) Subject: Re: suspend/resume does not work in qemu (qemu-devel) From: Andriy Gapon To: FreeBSD Current References: <09a48756-2dda-e830-5943-c7f1b9fa053a@FreeBSD.org> Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <39728d5a-0a40-b888-f91b-a9cef66c40c2@FreeBSD.org> Date: Thu, 26 Apr 2018 12:54:14 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <09a48756-2dda-e830-5943-c7f1b9fa053a@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2018 09:54:25 -0000 On 19/04/2018 11:54, Andriy Gapon wrote: > > Not sure if this really matters and on what end the problem might be, but > resuming from S3 in qemu is stuck forever here: It seems that the same issue _might_ affect real hardware if sc (syscons) console is used. What I see is that a system suspends and resumes fine with vt. If I switch to sc then I see the following message during suspend (never seen with vt): vga0: saving 1024 bytes of video state And the system fails to resume. Of course, it is much harder to tell with the real hardware where the resume fails. So, I am not sure. > #3 x86emu_exec_one_byte (emu=0xffffffff81eac618 ) at > /usr/devel/git/cpu-pause/sys/contrib/x86emu/x86emu.c:4655 > #4 0xffffffff8106c398 in x86emu_exec (emu=0xffffffff81eac618 ) at > /usr/devel/git/cpu-pause/sys/contrib/x86emu/x86emu.c:246 > #5 0xffffffff8106b8d4 in x86bios_intr (regs=0xfffffe002e1933f0, intno=16) at > /usr/devel/git/cpu-pause/sys/compat/x86bios/x86bios.c:634 > #6 0xffffffff80fb7cd2 in vesa_bios_save_restore (code=2, p=0xfffff8000365e004) > at /usr/devel/git/cpu-pause/sys/dev/fb/vesa.c:551 > #7 vesa_load_state (adp=, p=) at > /usr/devel/git/cpu-pause/sys/dev/fb/vesa.c:1535 > #8 0xffffffff810632e0 in vga_resume (dev=0xfffff800039eb100) at > /usr/devel/git/cpu-pause/sys/isa/vga_isa.c:112 > #9 0xffffffff8106311e in isavga_resume (dev=0xfffff800039eb100) at > /usr/devel/git/cpu-pause/sys/isa/vga_isa.c:243 > #10 0xffffffff80ac2de4 in DEVICE_RESUME (dev=) at ./device_if.h:305 > ... > > Using -vga none works around the problem, of course. > Maybe it's buggy video BIOS in the emulation. > -- Andriy Gapon From owner-freebsd-current@freebsd.org Thu Apr 26 06:43:01 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D0701FA291D for ; Thu, 26 Apr 2018 06:43:01 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 61EE46C509 for ; Thu, 26 Apr 2018 06:43:01 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 16374FA291B; Thu, 26 Apr 2018 06:43:01 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CB89BFA2919; Thu, 26 Apr 2018 06:43:00 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr0-x22d.google.com (mail-wr0-x22d.google.com [IPv6:2a00:1450:400c:c0c::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4AD066C507; Thu, 26 Apr 2018 06:43:00 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr0-x22d.google.com with SMTP id p5-v6so27749938wre.12; Wed, 25 Apr 2018 23:43:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=kVgABVOp/EFIAlV+m0iQ2340GYLALiZEpjF1ulJ4rqc=; b=cEJ50ODjAxvmk/XoxjV7ZHNfeyedEzbt0j1RbWMDjrlw5O3D9zqaEFP9pEz+59+WcV iMwLxFsVucCO7YrUe68Mxs9rBVO1gEZbm0/j5mMD2VGzm8EbkooxVUmOwfYHW0VlwU5m EgowbjoJ5Z1xgWWUU0aa+OTdGt0C6geKWaiW0nZ+GJGzi2J4omeBB+F+gVQFxfHHO/no 9VCkaNEPYEaU4zh4EdKi9/egvvvdEo8GrilTWnNtjwaiyDp6RD9l8rR977fSwTW9cN/T Xtplf3nPYMvrkS7IwIrPDf6EO+3KbliPzhGzGyUr0hb52E4BEsFbfjSd6aFpE9jI228r 2nSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=kVgABVOp/EFIAlV+m0iQ2340GYLALiZEpjF1ulJ4rqc=; b=lPIhglHuSrYANv0Mspsvi4qxWDTyzrkHOWI12p4EWv97Hn3Vbvjg3x/3Q7VVTcBFYB mJrL2k1Y5UFh+H+q0ioPccciaqEz3JalN0KiAPp6WksPBZ+j6tolTnhu5KaQ6QPzY5PK zAj8J1meX76MrJXT8GLKa1NWh9VAs8PxHk9L4MiPpTC4UP5eZ6+7JImnUFMaOQVeXtAR zjVRPpDUZIVJ7N5HGFNlVsQe/KhmfzHyQdD+Yxf5BN8vckrwW0kH4j2ESJHQiyE8ec+3 dHNbgOUy2lMwtczI7fCySUf32TAnkzfgDuXlgbBMyOQJLhhjQbOc1Ksms5jCVQhAkXSx 9mwA== X-Gm-Message-State: ALQs6tBGVkezKfaw5muXatzsQO5FNYr537IPWagT1hGciafnOizqDq/A MXOW1XZ9ScZS5fbWvIFgOE8= X-Google-Smtp-Source: AIpwx4/4jnNhpZQaT4KzNiDX3tUofDS9TZcleDxGW3oeDjsz8WKtjT6Nptdb2ScpxJ7cBzDwhvoDsA== X-Received: by 2002:adf:b741:: with SMTP id n1-v6mr26629144wre.203.1524724979326; Wed, 25 Apr 2018 23:42:59 -0700 (PDT) Received: from bens-mac.home (LFbn-1-7077-100.w90-116.abo.wanadoo.fr. [90.116.246.100]) by smtp.gmail.com with ESMTPSA id q34-v6sm18526218wrb.27.2018.04.25.23.42.58 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 25 Apr 2018 23:42:58 -0700 (PDT) Content-Type: text/plain; charset=us-ascii; delsp=yes; format=flowed Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: mlx5(4) jumbo receive From: Ben RUBSON In-Reply-To: Date: Thu, 26 Apr 2018 08:43:01 +0200 Cc: Ryan Stone , Konstantin Belousov , FreeBSD Current , FreeBSD Net , Meny Yossefi Content-Transfer-Encoding: 7bit Message-Id: References: <20180424085553.GA6887@kib.kiev.ua> To: Rick Macklem X-Mailer: Apple Mail (2.3124) X-Mailman-Approved-At: Thu, 26 Apr 2018 10:41:33 +0000 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2018 06:43:02 -0000 On 26 Apr 2018, Rick Macklem wrote: > Ryan Stone wrote: >> On Tue, Apr 24, 2018 at 4:55 AM, Konstantin Belousov >> >>wrote: >>> +#ifndef MLX5E_MAX_RX_BYTES >>> +#define MLX5E_MAX_RX_BYTES MCLBYTES >>> +#endif >> >> Why do you use a 2KB buffer rather than a PAGE_SIZE'd buffer? >> MJUMPAGESIZE should offer significantly better performance for jumbo >> frames without increasing the risk of memory fragmentation. > Actually, when I was playing with using jumbo mbuf clusters for NFS, I > was able > to get it to fragment to the point where allocations failed when mixing > 2K and > 4K mbuf clusters. > Admittedly I was using a 256Mbyte i386 and it wasn't easily reproduced, but > it was possible. > --> Using a mix of 2K and 4K mbuf clusters can result in fragmentation, > although > I suspect that it isn't nearly as serious as what can happen when using 9K > mbuf clusters. I used to face the fragmentation issue easily with MTU set to 9000 (x86_64 / 64GB / Connect-X/3). I then decreased the MTU until 9K mbufs are not more used, to 4072 bytes then. The other used interface of this 2-ports card is set with a 1500 MTU. Until now (about a year later), no more issue. # vmstat -z | grep mbuf ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 26080155, 16400, 9652, 999757417, 0, 0 mbuf: 256, 26080155, 16419, 11349, 85322301444, 0, 0 mbuf_cluster: 2048, 4075022, 26052, 550, 1059817, 0, 0 mbuf_jumbo_page: 4096, 2037511, 16400, 9522, 45863599682, 0, 0 mbuf_jumbo_9k: 9216, 603707, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 339585, 0, 0, 0, 0, 0 Here's my experience :) Ben