From owner-freebsd-scsi@FreeBSD.ORG Mon Aug 12 11:06:52 2013 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id CC8C1AFE for ; Mon, 12 Aug 2013 11:06:52 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B9EDE214E for ; Mon, 12 Aug 2013 11:06:52 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r7CB6qsj085168 for ; Mon, 12 Aug 2013 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r7CB6qZn085166 for freebsd-scsi@FreeBSD.org; Mon, 12 Aug 2013 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 12 Aug 2013 11:06:52 GMT Message-Id: <201308121106.r7CB6qZn085166@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Aug 2013 11:06:52 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/179932 scsi [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP o kern/178795 scsi [mps] MSI for mps driver doesn't work under vmware o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free f kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c o kern/148083 scsi [aac] Strange device reporting o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 f kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus f kern/123674 scsi [ahc] ahc driver dumping o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc 14 problems total. From owner-freebsd-scsi@FreeBSD.ORG Wed Aug 14 02:47:06 2013 Return-Path: Delivered-To: FreeBSD-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A4D2A77C for ; Wed, 14 Aug 2013 02:47:06 +0000 (UTC) (envelope-from sean_bruno@yahoo.com) Received: from nm8-vm5.bullet.mail.ne1.yahoo.com (nm8-vm5.bullet.mail.ne1.yahoo.com [98.138.91.230]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6293F27E6 for ; Wed, 14 Aug 2013 02:47:05 +0000 (UTC) Received: from [98.138.101.129] by nm8.bullet.mail.ne1.yahoo.com with NNFMP; 14 Aug 2013 02:46:58 -0000 Received: from [98.138.226.131] by tm17.bullet.mail.ne1.yahoo.com with NNFMP; 14 Aug 2013 02:46:58 -0000 Received: from [127.0.0.1] by smtp218.mail.ne1.yahoo.com with NNFMP; 14 Aug 2013 02:46:58 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1376448418; bh=YBfxKs/NiCqx/OWPUUDmNtXXMT4UswnhDi7+4ISRSDk=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:Subject:From:Reply-To:To:Cc:In-Reply-To:References:Content-Type:Date:Message-ID:Mime-Version:X-Mailer; b=zgbQBCKnqUXMNS9OvhDTbCsOp0sFsUdwYez9jZF10NLIyLnuJDNWqbUaeVlsu3x5unNklrmDucvwbDx2wsXUdp7y6bl3AzKq09FA6+3MQldoFzWsGcelRbAT0YkSRYjqFIN8pmaQqWb0vWc+JiBn+aQVFpDXf2kgeycWlAMNMAY= X-Yahoo-Newman-Id: 457099.72250.bm@smtp218.mail.ne1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: 7nTCKPEVM1nlNBLUKKWhZkyrxYmi7SrueBFeMzYpPLesplv 6OeJrgXg0eshlaIyrtO7NbuBxEeOYQ6bEifuesuEAGuDA79eYkLhi7YvgYJr s.oO7yGt3gm4bZfQ.pDEFTW2BnbpFIfhwROd6FLs264ZkYZYWZ91OJtN7LbR lkD.gaKrn6aDs12RQ9solsPk6x3BcwEsgKcxHjBPggpbq90.M1ErKVMxjYo0 MTtYPrMECtLT8J2WWXLsg4KUm_uAvvjGG8hZ5r8b.uiEk72P9UwtDwlD4HeZ ijFyBgGwpyOfNTDKDZsqzJvahalgA1ehoWmF6KnVna5McUdYIePhYzjg2Vf3 WCNSyBOjiTfTbT07yM0jDjGojcHqK5WtYQ5eR8yvYeXmyI9U7oyz25_kd1WH TCnY7KAKVilZul5JdcOQLJPHynE9bEZGvRjgbTizxFOOAmqleoMm82oHF75C QZU2dY8uHQu026kXgj1.t_hbzK3qHq8WwBDR6BLKczdJv1HjPy3TP1KerwaU DJrop6WEEHg7qXS8yOhR7IpPlnTG.W5pChjvwI3NGbJplJ4KWKhUo0ULU X-Yahoo-SMTP: u5BKR6OswBC_iZJVfGRoMkTIpc8pEA4- X-Rocket-Received: from [192.168.1.209] (sean_bruno@71.202.40.63 with ) by smtp218.mail.ne1.yahoo.com with SMTP; 13 Aug 2013 19:46:58 -0700 PDT Subject: Re: Dell H310, JBOD mode "hard error" From: Sean Bruno To: sbruno@freebsd.org In-Reply-To: <1373822621.1431.5.camel@localhost> References: <1373822621.1431.5.camel@localhost> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-F13hOyrzBOZjHaUmC312" Date: Tue, 13 Aug 2013 19:46:56 -0700 Message-ID: <1376448416.1439.7.camel@localhost> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Cc: "FreeBSD-scsi@freebsd.org" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: sbruno@freebsd.org List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Aug 2013 02:47:06 -0000 --=-F13hOyrzBOZjHaUmC312 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable On Sun, 2013-07-14 at 10:23 -0700, Sean Bruno wrote: > Not sure what to make of this. I've tested a lot of svn revisions of > the thunderbolt code, but nothing looks obvious. =20 >=20 > When I use a single drive in "SYSPD" mode on a Dell H310 (falcon or > skinny drake) I get a /dev/mfisyspd0 device. The JBOD mode *seems* to > work just fine as long as I don't do multiple things at once to it, e.g. > single user fsck works, but multiuser things die. >=20 > I get a failure case that emits errors such as: >=20 > g_vfs_done():error 27 in callback > mfisyspd0p2[READ(offset=3D7176192, length=3D425984)]mfisyspd0: hard error > error =3D 5 > cmd=3Dread 15360-16383 > error 27 in callback > g_vfs_done():mfisyspd0: hard error mfisyspd0p2[READ(offset=3D7602176, > length=3D524288)]cmd=3Dread error =3D 5 > 16384-17407 > error 27 in callback > g_vfs_done():mfisyspd0: hard error mfisyspd0p2[READ(offset=3D8126464, > length=3D524288)]cmd=3Dread error =3D 5 > 14560-15359 > error 27 in callback > g_vfs_done():mfisyspd0: hard error mfisyspd0p2[READ(offset=3D7192576, > length=3D409600)]cmd=3Dread error =3D 5 > 15360-16383 >=20 >=20 > Sean Ah, I see something that Yahoo! does that FreeBSD does not finally. We tune MAXPHYS *up* to (512 * 1024) because of performance and available memory. mfi(4) set's its own (MFI_MAXPHYS) to (128 * 1024) instead of using the value from sys/param.h (btw, I don't quite get why, but whatever). Without a min() check in mfi_syspd.c that mimics the one in mfi_disk.c, Yahoo code falls over in "SYSPD" mode (mfi(4) real jbod mode). Patch: Index: mfi_syspd.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- mfi_syspd.c (revision 254313) +++ mfi_syspd.c (working copy) @@ -128,7 +128,9 @@ sc->pd_disk->d_drv1 =3D sc; sc->pd_disk->d_maxsize =3D sc->pd_controller->mfi_max_io * secsize; sc->pd_disk->d_name =3D "mfisyspd"; - sc->pd_disk->d_open =3D mfi_syspd_open; + sc->pd_disk->d_maxsize =3D min(sc->pd_controller->mfi_max_io * secsize, + (sc->pd_controller->mfi_max_sge - 1) * PAGE_SIZE); + sc->pd_disk->d_close =3D mfi_syspd_close; sc->pd_disk->d_strategy =3D mfi_syspd_strategy; sc->pd_disk->d_dump =3D mfi_syspd_dump; --=-F13hOyrzBOZjHaUmC312 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (FreeBSD) iQEcBAABAgAGBQJSCu+YAAoJEBkJRdwI6BaHFRUH/RD/SQ97OjATNreOBaoRoYoT hmIEbtI1fxTP/vaFn2qOMupKSNdfR2MSD+znECkDcHYHtuLt6oL6Mey1WqP0KbLE QTndpxa8o8BZd8+6FuhcL7wnUQqtdH6MZM2arCanF0CVHBK4mj2jdCqY02HRkcoi UpL1VZklj8+ruk+BSpA2VaKKRuSnxMVsh1OZWDtOhSx6BSqWGnfDAxY5E/n9DnAJ vJ+gjLAVaZni0qgb85+rrBTxYfwMfoUrhCe5KU6Pw6gE3LPYpQ8E3n3YuEjdglst pHnaDQWLqqvm9b2jSsy0nEf0J8yfYjy2Laat31yuLEoAgJnl3vFVTYcPE0FwvOc= =TPi7 -----END PGP SIGNATURE----- --=-F13hOyrzBOZjHaUmC312-- From owner-freebsd-scsi@FreeBSD.ORG Wed Aug 14 15:03:37 2013 Return-Path: Delivered-To: FreeBSD-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6EB28558; Wed, 14 Aug 2013 15:03:37 +0000 (UTC) (envelope-from ambrisko@ambrisko.com) Received: from mail.ambrisko.com (mail.ambrisko.com [70.91.206.90]) by mx1.freebsd.org (Postfix) with ESMTP id 4923420FB; Wed, 14 Aug 2013 15:03:37 +0000 (UTC) X-Ambrisko-Me: Yes Received: from server2.ambrisko.com (HELO internal.ambrisko.com) ([192.168.1.2]) by ironport.ambrisko.com with ESMTP; 14 Aug 2013 08:06:57 -0700 Received: from ambrisko.com (localhost [127.0.0.1]) by internal.ambrisko.com (8.14.4/8.14.4) with ESMTP id r7EF3VS6041728; Wed, 14 Aug 2013 08:03:31 -0700 (PDT) (envelope-from ambrisko@ambrisko.com) Received: (from ambrisko@localhost) by ambrisko.com (8.14.4/8.14.4/Submit) id r7EF3VAB041727; Wed, 14 Aug 2013 08:03:31 -0700 (PDT) (envelope-from ambrisko) Date: Wed, 14 Aug 2013 08:03:31 -0700 From: Doug Ambrisko To: sbruno@freebsd.org Subject: Re: Dell H310, JBOD mode "hard error" Message-ID: <20130814150331.GB34825@ambrisko.com> References: <1373822621.1431.5.camel@localhost> <1376448416.1439.7.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1376448416.1439.7.camel@localhost> User-Agent: Mutt/1.4.2.3i Cc: "FreeBSD-scsi@freebsd.org" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Aug 2013 15:03:37 -0000 On Tue, Aug 13, 2013 at 07:46:56PM -0700, Sean Bruno wrote: | On Sun, 2013-07-14 at 10:23 -0700, Sean Bruno wrote: | > Not sure what to make of this. I've tested a lot of svn revisions of | > the thunderbolt code, but nothing looks obvious. | > | > When I use a single drive in "SYSPD" mode on a Dell H310 (falcon or | > skinny drake) I get a /dev/mfisyspd0 device. The JBOD mode *seems* to | > work just fine as long as I don't do multiple things at once to it, e.g. | > single user fsck works, but multiuser things die. | > | > I get a failure case that emits errors such as: | > | > g_vfs_done():error 27 in callback | > mfisyspd0p2[READ(offset=7176192, length=425984)]mfisyspd0: hard error | > error = 5 | > cmd=read 15360-16383 | > error 27 in callback | > g_vfs_done():mfisyspd0: hard error mfisyspd0p2[READ(offset=7602176, | > length=524288)]cmd=read error = 5 | > 16384-17407 | > error 27 in callback | > g_vfs_done():mfisyspd0: hard error mfisyspd0p2[READ(offset=8126464, | > length=524288)]cmd=read error = 5 | > 14560-15359 | > error 27 in callback | > g_vfs_done():mfisyspd0: hard error mfisyspd0p2[READ(offset=7192576, | > length=409600)]cmd=read error = 5 | > 15360-16383 | > | > | > Sean | | Ah, I see something that Yahoo! does that FreeBSD does not finally. We | tune MAXPHYS *up* to (512 * 1024) because of performance and available | memory. | | mfi(4) set's its own (MFI_MAXPHYS) to (128 * 1024) instead of using the | value from sys/param.h (btw, I don't quite get why, but whatever). | | Without a min() check in mfi_syspd.c that mimics the one in mfi_disk.c, | Yahoo code falls over in "SYSPD" mode (mfi(4) real jbod mode). | | Patch: | | Index: mfi_syspd.c | =================================================================== | --- mfi_syspd.c (revision 254313) | +++ mfi_syspd.c (working copy) | @@ -128,7 +128,9 @@ | sc->pd_disk->d_drv1 = sc; | sc->pd_disk->d_maxsize = sc->pd_controller->mfi_max_io * secsize; | sc->pd_disk->d_name = "mfisyspd"; | - sc->pd_disk->d_open = mfi_syspd_open; | + sc->pd_disk->d_maxsize = min(sc->pd_controller->mfi_max_io * secsize, | + (sc->pd_controller->mfi_max_sge - 1) * PAGE_SIZE); | + | sc->pd_disk->d_close = mfi_syspd_close; | sc->pd_disk->d_strategy = mfi_syspd_strategy; | sc->pd_disk->d_dump = mfi_syspd_dump; That change for d_maxsize looks okay but do you really want to get rid of d_open? I assume this is a cut-n-paste type error and the patch (hand editted) should be: Index: mfi_syspd.c =================================================================== --- mfi_syspd.c (revision 254313) +++ mfi_syspd.c (working copy) @@ -128,7 +128,8 @@ sc->pd_disk->d_drv1 = sc; - sc->pd_disk->d_maxsize = sc->pd_controller->mfi_max_io * secsize; + sc->pd_disk->d_maxsize = min(sc->pd_controller->mfi_max_io * secsize, + (sc->pd_controller->mfi_max_sge - 1) * PAGE_SIZE); sc->pd_disk->d_name = "mfisyspd"; sc->pd_disk->d_open = mfi_syspd_open; sc->pd_disk->d_close = mfi_syspd_close; sc->pd_disk->d_strategy = mfi_syspd_strategy; sc->pd_disk->d_dump = mfi_syspd_dump; BTW, has mfiutil been updated to create real JBODs versus the RAID per drive? I know someone was talking about doing that. A note on implementing it, it also requires JBOD to be enabled in the controller. I'm not sure if all controllers support it. I forget when I was playing with it. I've always wondering if we should change the name for the syspd disk node but I left it for compatibility with LSI. We could do an alias. It is good in away that it doesn't create /dev/mfid* nodes so that it is easier to track bugs. Thanks, Doug A. From owner-freebsd-scsi@FreeBSD.ORG Wed Aug 14 15:16:08 2013 Return-Path: Delivered-To: FreeBSD-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E4B2A756; Wed, 14 Aug 2013 15:16:08 +0000 (UTC) (envelope-from prvs=19381d5a45=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5FC162185; Wed, 14 Aug 2013 15:16:08 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50005492799.msg; Wed, 14 Aug 2013 16:16:04 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 14 Aug 2013 16:16:04 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=19381d5a45=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <46DE157BA89B4C17B95BE5E738D0B2AF@multiplay.co.uk> From: "Steven Hartland" To: "Doug Ambrisko" , References: <1373822621.1431.5.camel@localhost> <1376448416.1439.7.camel@localhost> <20130814150331.GB34825@ambrisko.com> Subject: Re: Dell H310, JBOD mode "hard error" Date: Wed, 14 Aug 2013 16:16:23 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: FreeBSD-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Aug 2013 15:16:09 -0000 ----- Original Message ----- From: "Doug Ambrisko" To: Cc: Sent: Wednesday, August 14, 2013 4:03 PM Subject: Re: Dell H310, JBOD mode "hard error" > On Tue, Aug 13, 2013 at 07:46:56PM -0700, Sean Bruno wrote: > | On Sun, 2013-07-14 at 10:23 -0700, Sean Bruno wrote: > | > Not sure what to make of this. I've tested a lot of svn revisions of > | > the thunderbolt code, but nothing looks obvious. > | > > | > When I use a single drive in "SYSPD" mode on a Dell H310 (falcon or > | > skinny drake) I get a /dev/mfisyspd0 device. The JBOD mode *seems* to > | > work just fine as long as I don't do multiple things at once to it, e.g. > | > single user fsck works, but multiuser things die. > | > > | > I get a failure case that emits errors such as: > | > > | > g_vfs_done():error 27 in callback > | > mfisyspd0p2[READ(offset=7176192, length=425984)]mfisyspd0: hard error > | > error = 5 > | > cmd=read 15360-16383 > | > error 27 in callback > | > g_vfs_done():mfisyspd0: hard error mfisyspd0p2[READ(offset=7602176, > | > length=524288)]cmd=read error = 5 > | > 16384-17407 > | > error 27 in callback > | > g_vfs_done():mfisyspd0: hard error mfisyspd0p2[READ(offset=8126464, > | > length=524288)]cmd=read error = 5 > | > 14560-15359 > | > error 27 in callback > | > g_vfs_done():mfisyspd0: hard error mfisyspd0p2[READ(offset=7192576, > | > length=409600)]cmd=read error = 5 > | > 15360-16383 > | > > | > > | > Sean > | > | Ah, I see something that Yahoo! does that FreeBSD does not finally. We > | tune MAXPHYS *up* to (512 * 1024) because of performance and available > | memory. > | > | mfi(4) set's its own (MFI_MAXPHYS) to (128 * 1024) instead of using the > | value from sys/param.h (btw, I don't quite get why, but whatever). > | > | Without a min() check in mfi_syspd.c that mimics the one in mfi_disk.c, > | Yahoo code falls over in "SYSPD" mode (mfi(4) real jbod mode). > | > | Patch: > | > | Index: mfi_syspd.c > | =================================================================== > | --- mfi_syspd.c (revision 254313) > | +++ mfi_syspd.c (working copy) > | @@ -128,7 +128,9 @@ > | sc->pd_disk->d_drv1 = sc; > | sc->pd_disk->d_maxsize = sc->pd_controller->mfi_max_io * secsize; > | sc->pd_disk->d_name = "mfisyspd"; > | - sc->pd_disk->d_open = mfi_syspd_open; > | + sc->pd_disk->d_maxsize = min(sc->pd_controller->mfi_max_io * secsize, > | + (sc->pd_controller->mfi_max_sge - 1) * PAGE_SIZE); > | + > | sc->pd_disk->d_close = mfi_syspd_close; > | sc->pd_disk->d_strategy = mfi_syspd_strategy; > | sc->pd_disk->d_dump = mfi_syspd_dump; > > > That change for d_maxsize looks okay but do you really want to get > rid of d_open? I assume this is a cut-n-paste type error and the patch > (hand editted) should be: > > Index: mfi_syspd.c > =================================================================== > --- mfi_syspd.c (revision 254313) > +++ mfi_syspd.c (working copy) > @@ -128,7 +128,8 @@ > sc->pd_disk->d_drv1 = sc; > - sc->pd_disk->d_maxsize = sc->pd_controller->mfi_max_io * secsize; > + sc->pd_disk->d_maxsize = min(sc->pd_controller->mfi_max_io * secsize, > + (sc->pd_controller->mfi_max_sge - 1) * PAGE_SIZE); > sc->pd_disk->d_name = "mfisyspd"; > sc->pd_disk->d_open = mfi_syspd_open; > sc->pd_disk->d_close = mfi_syspd_close; > sc->pd_disk->d_strategy = mfi_syspd_strategy; > sc->pd_disk->d_dump = mfi_syspd_dump; > > BTW, has mfiutil been updated to create real JBODs versus the RAID per > drive? I know someone was talking about doing that. A note on implementing > it, it also requires JBOD to be enabled in the controller. I'm not sure > if all controllers support it. I forget when I was playing with it. I've > always wondering if we should change the name for the syspd disk node > but I left it for compatibility with LSI. We could do an alias. It is > good in away that it doesn't create /dev/mfid* nodes so that it is easier > to track bugs. Some do support native JBOD, some require an additional controller level setting, and some simply don't support it and I'm not aware of an easy way to determine which is the case I'm afraid. Last time I looked mfiutil used RAID not native JBOD for JBOD's Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-scsi@FreeBSD.ORG Wed Aug 14 15:19:55 2013 Return-Path: Delivered-To: FreeBSD-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id C10FD973 for ; Wed, 14 Aug 2013 15:19:55 +0000 (UTC) (envelope-from sean_bruno@yahoo.com) Received: from nm46-vm9.bullet.mail.gq1.yahoo.com (nm46-vm9.bullet.mail.gq1.yahoo.com [67.195.87.177]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8227321AE for ; Wed, 14 Aug 2013 15:19:55 +0000 (UTC) Received: from [98.137.12.191] by nm46.bullet.mail.gq1.yahoo.com with NNFMP; 14 Aug 2013 15:17:47 -0000 Received: from [208.71.42.206] by tm12.bullet.mail.gq1.yahoo.com with NNFMP; 14 Aug 2013 15:17:47 -0000 Received: from [127.0.0.1] by smtp217.mail.gq1.yahoo.com with NNFMP; 14 Aug 2013 15:17:47 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1376493467; bh=8E7yJwWn5gAs2rn37Vghm0JCULizkvTk5Efh4UI0Hu8=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:Subject:From:Reply-To:To:Cc:In-Reply-To:References:Content-Type:Date:Message-ID:Mime-Version:X-Mailer; b=ci/FQ6DgysgLLAp7mjL9Ej+ZXa93UKIqzWWVnG68l2aLdUJo0bXzIlWUm/RrJ7Q9XbtjIOo5Mcqg6o85xv+G1YwKKIHtROmshmF1K99sFcUEiozVyv2kqgPe3SR4vjXkNClWz1ezYrSlaTeL5f+VqAioObZm7RC9olb41xQXBp8= X-Yahoo-Newman-Id: 616298.34403.bm@smtp217.mail.gq1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: OW0hOF0VM1k9Aw_RmQIT5NdeC_Yyn51H5VyK5MkeW.sjwuN aaNxkwqcN2.OWbhQk7ZFFwabRz3uaIg4VO20cqzrrt01Q4b1jDHw5ghAv2f6 fp1Cp3eqmoopqssr5W_is2C_Filvd6cmH4LhTxnf58xhGTCVwMwREaQOp1E9 zoMw1Eo4C79nIOaaCGjacxBL7AKzFt1pZv7pd48HsWG7g2Yw7w9k_A1ucpqW 8lqcZsdRfXBq_KatR7DRLHhQBtya2LFAbgRY0HIyV0H2kosSMZcJeOx9dAc7 BzaMPO.fNq2xPq8.kfV19eYwSCU1TFjV_khyGC7ET.2nlHPj_Z3wG1.0jvEO WRNeQzD2K8PtgWyagx_RfAlXWpUwbvdONJBzIqhWfsq.IhuxF0doRTVnv7tO 4S8wDIPwa8AZN3hxuRlKUsSUSaIvYUL2s3FpW6a9M_A.f_0YoQHfGLwpHZOM DpLkBJ_GlQVzeHsa_hsPmOF4rdkSfMLvrqFojzWyyMV.EC3yO0Umo2wwSMf6 yWfSQUUo19fqEpVzFKp3n3KH_Oyvr_jvXSxs3qPzkqUE7fnU9fEW77TOa4nS 8wR82YH4- X-Yahoo-SMTP: u5BKR6OswBC_iZJVfGRoMkTIpc8pEA4- X-Rocket-Received: from [192.168.43.121] (sean_bruno@70.197.0.149 with ) by smtp217.mail.gq1.yahoo.com with SMTP; 14 Aug 2013 15:17:47 +0000 UTC Subject: Re: Dell H310, JBOD mode "hard error" From: Sean Bruno To: Doug Ambrisko In-Reply-To: <20130814150331.GB34825@ambrisko.com> References: <1373822621.1431.5.camel@localhost> <1376448416.1439.7.camel@localhost> <20130814150331.GB34825@ambrisko.com> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-0SZy4tGwEQjKTRv7VbEY" Date: Wed, 14 Aug 2013 08:17:44 -0700 Message-ID: <1376493464.1377.3.camel@localhost> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Cc: "FreeBSD-scsi@freebsd.org" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: sbruno@freebsd.org List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Aug 2013 15:19:55 -0000 --=-0SZy4tGwEQjKTRv7VbEY Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable > That change for d_maxsize looks okay but do you really want to get > rid of d_open? I assume this is a cut-n-paste type error and the patch > (hand editted) should be: >=20 > Index: mfi_syspd.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- mfi_syspd.c (revision 254313) > +++ mfi_syspd.c (working copy) > @@ -128,7 +128,8 @@ > sc->pd_disk->d_drv1 =3D sc; > - sc->pd_disk->d_maxsize =3D sc->pd_controller->mfi_max_io * secsize; > + sc->pd_disk->d_maxsize =3D min(sc->pd_controller->mfi_max_io * secsize, > + (sc->pd_controller->mfi_max_sge - 1) * PAGE_SIZE); > sc->pd_disk->d_name =3D "mfisyspd"; > sc->pd_disk->d_open =3D mfi_syspd_open; > sc->pd_disk->d_close =3D mfi_syspd_close; > sc->pd_disk->d_strategy =3D mfi_syspd_strategy; > sc->pd_disk->d_dump =3D mfi_syspd_dump; >=20 Bah, that's why I always send patches for review. Yes, the d_open deletion was a typo. Thanks for the update. > BTW, has mfiutil been updated to create real JBODs versus the RAID per > drive? I know someone was talking about doing that. A note on implement= ing > it, it also requires JBOD to be enabled in the controller. I'm not sure > if all controllers support it. I forget when I was playing with it. I'v= e > always wondering if we should change the name for the syspd disk node > but I left it for compatibility with LSI. We could do an alias. It is > good in away that it doesn't create /dev/mfid* nodes so that it is easier > to track bugs. mfiutil doesn't seem to know how to deal with SYSPD (real JBOD) at this time. It only allows you to see the physical drives, nothing shows up on the "show volumes" command. =46rom what I see the Dell H310 doing, there's not much there in the system to do with the real JBOD devices. I mean, there's no creation/deletion or other RAID things to do here. Sean --=-0SZy4tGwEQjKTRv7VbEY Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (FreeBSD) iQEcBAABAgAGBQJSC5+QAAoJEBkJRdwI6BaHiPAH/0OpKcwAw/OBkg3OVSh4ZGWN p+GKqDrhCevhzgDCaVneMXz51rf1yDom2T72qP8X83LZrMjzJMjnrMu6eeKhN4zx Dr7CPSrvNHf+amHPAlv5gBZpMjczbnVaoODFvsj1mJRxEqW/sWIxtqZsA51bYtgF ceC5So4I/U/ZUAqmx26KBpNTSN0vUuOAqra6yD+15tHy24qMaqQSXY8fzRNboRV4 sRP0z92v9YlIfBlYU14etgWuK0W5RWbC+Cp+tpw8fdBetc591CIFIoB+Mv/SKzid TMSeaDCEBHdf0qoyVRk1QW8A5iuvWnh2KqkQU8u1/l0E0sw9Kb8GVFCilxBmtG8= =v1x/ -----END PGP SIGNATURE----- --=-0SZy4tGwEQjKTRv7VbEY-- From owner-freebsd-scsi@FreeBSD.ORG Wed Aug 14 15:33:37 2013 Return-Path: Delivered-To: FreeBSD-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 4A44EECF; Wed, 14 Aug 2013 15:33:37 +0000 (UTC) (envelope-from ambrisko@ambrisko.com) Received: from mail.ambrisko.com (mail.ambrisko.com [70.91.206.90]) by mx1.freebsd.org (Postfix) with ESMTP id 2481622AC; Wed, 14 Aug 2013 15:33:36 +0000 (UTC) X-Ambrisko-Me: Yes Received: from server2.ambrisko.com (HELO internal.ambrisko.com) ([192.168.1.2]) by ironport.ambrisko.com with ESMTP; 14 Aug 2013 08:37:02 -0700 Received: from ambrisko.com (localhost [127.0.0.1]) by internal.ambrisko.com (8.14.4/8.14.4) with ESMTP id r7EFXaiV051957; Wed, 14 Aug 2013 08:33:36 -0700 (PDT) (envelope-from ambrisko@ambrisko.com) Received: (from ambrisko@localhost) by ambrisko.com (8.14.4/8.14.4/Submit) id r7EFXaut051956; Wed, 14 Aug 2013 08:33:36 -0700 (PDT) (envelope-from ambrisko) Date: Wed, 14 Aug 2013 08:33:36 -0700 From: Doug Ambrisko To: Steven Hartland Subject: Re: Dell H310, JBOD mode "hard error" Message-ID: <20130814153336.GB47170@ambrisko.com> References: <1373822621.1431.5.camel@localhost> <1376448416.1439.7.camel@localhost> <20130814150331.GB34825@ambrisko.com> <46DE157BA89B4C17B95BE5E738D0B2AF@multiplay.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <46DE157BA89B4C17B95BE5E738D0B2AF@multiplay.co.uk> User-Agent: Mutt/1.4.2.3i Cc: FreeBSD-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Aug 2013 15:33:37 -0000 On Wed, Aug 14, 2013 at 04:16:23PM +0100, Steven Hartland wrote: | | ----- Original Message ----- | From: "Doug Ambrisko" | To: | Cc: | Sent: Wednesday, August 14, 2013 4:03 PM | Subject: Re: Dell H310, JBOD mode "hard error" | | | >On Tue, Aug 13, 2013 at 07:46:56PM -0700, Sean Bruno wrote: | >| On Sun, 2013-07-14 at 10:23 -0700, Sean Bruno wrote: | >| > Not sure what to make of this. I've tested a lot of svn revisions of | >| > the thunderbolt code, but nothing looks obvious. | >| > | >| > When I use a single drive in "SYSPD" mode on a Dell H310 (falcon or | >| > skinny drake) I get a /dev/mfisyspd0 device. The JBOD mode *seems* to | >| > work just fine as long as I don't do multiple things at once to it, | >e.g. | >| > single user fsck works, but multiuser things die. | >| > | >| > I get a failure case that emits errors such as: | >| > | >| > g_vfs_done():error 27 in callback | >| > mfisyspd0p2[READ(offset=7176192, length=425984)]mfisyspd0: hard error | >| > error = 5 | >| > cmd=read 15360-16383 | >| > error 27 in callback | >| > g_vfs_done():mfisyspd0: hard error mfisyspd0p2[READ(offset=7602176, | >| > length=524288)]cmd=read error = 5 | >| > 16384-17407 | >| > error 27 in callback | >| > g_vfs_done():mfisyspd0: hard error mfisyspd0p2[READ(offset=8126464, | >| > length=524288)]cmd=read error = 5 | >| > 14560-15359 | >| > error 27 in callback | >| > g_vfs_done():mfisyspd0: hard error mfisyspd0p2[READ(offset=7192576, | >| > length=409600)]cmd=read error = 5 | >| > 15360-16383 | >| > | >| > | >| > Sean | >| | >| Ah, I see something that Yahoo! does that FreeBSD does not finally. We | >| tune MAXPHYS *up* to (512 * 1024) because of performance and available | >| memory. | >| | >| mfi(4) set's its own (MFI_MAXPHYS) to (128 * 1024) instead of using the | >| value from sys/param.h (btw, I don't quite get why, but whatever). | >| | >| Without a min() check in mfi_syspd.c that mimics the one in mfi_disk.c, | >| Yahoo code falls over in "SYSPD" mode (mfi(4) real jbod mode). | >| | >| Patch: | >| | >| Index: mfi_syspd.c | >| =================================================================== | >| --- mfi_syspd.c (revision 254313) | >| +++ mfi_syspd.c (working copy) | >| @@ -128,7 +128,9 @@ | >| sc->pd_disk->d_drv1 = sc; | >| sc->pd_disk->d_maxsize = sc->pd_controller->mfi_max_io * secsize; | >| sc->pd_disk->d_name = "mfisyspd"; | >| - sc->pd_disk->d_open = mfi_syspd_open; | >| + sc->pd_disk->d_maxsize = min(sc->pd_controller->mfi_max_io * secsize, | >| + (sc->pd_controller->mfi_max_sge - 1) * PAGE_SIZE); | >| + | >| sc->pd_disk->d_close = mfi_syspd_close; | >| sc->pd_disk->d_strategy = mfi_syspd_strategy; | >| sc->pd_disk->d_dump = mfi_syspd_dump; | > | > | >That change for d_maxsize looks okay but do you really want to get | >rid of d_open? I assume this is a cut-n-paste type error and the patch | >(hand editted) should be: | > | >Index: mfi_syspd.c | >=================================================================== | >--- mfi_syspd.c (revision 254313) | >+++ mfi_syspd.c (working copy) | >@@ -128,7 +128,8 @@ | > sc->pd_disk->d_drv1 = sc; | >- sc->pd_disk->d_maxsize = sc->pd_controller->mfi_max_io * secsize; | >+ sc->pd_disk->d_maxsize = min(sc->pd_controller->mfi_max_io * secsize, | >+ (sc->pd_controller->mfi_max_sge - 1) * PAGE_SIZE); | > sc->pd_disk->d_name = "mfisyspd"; | >sc->pd_disk->d_open = mfi_syspd_open; | > sc->pd_disk->d_close = mfi_syspd_close; | > sc->pd_disk->d_strategy = mfi_syspd_strategy; | > sc->pd_disk->d_dump = mfi_syspd_dump; | > | >BTW, has mfiutil been updated to create real JBODs versus the RAID per | >drive? I know someone was talking about doing that. A note on | >implementing | >it, it also requires JBOD to be enabled in the controller. I'm not sure | >if all controllers support it. I forget when I was playing with it. I've | >always wondering if we should change the name for the syspd disk node | >but I left it for compatibility with LSI. We could do an alias. It is | >good in away that it doesn't create /dev/mfid* nodes so that it is easier | >to track bugs. | | Some do support native JBOD, some require an additional controller level | setting, and some simply don't support it and I'm not aware of an easy way | to determine which is the case I'm afraid. It can be figured out via MegaCli. I remember MegaaCli can set it. I think it can find out if the controller supports it. MegaCli was more powerful then the POST tool since with I was able to have both JBODs and RAID volumes on the same controller. I recall some of the POST tools prevented that. I also need to play around with CacheCade more and how to set that up. Again, I was able to do via MegaCli except for the license activation. However, at the time with the version of CacheCade that was in the firmware I was using it slowed things down versus speeding it up. That should be fixed now but I haven't had time to test it out. I should finally have time to work on more FreeBSD things since some other issues I've been working on have been resolved. | Last time I looked mfiutil used RAID not native JBOD for JBOD's That's probably the case. Thanks, Doug A. From owner-freebsd-scsi@FreeBSD.ORG Thu Aug 15 21:40:49 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 82F16BCE; Thu, 15 Aug 2013 21:40:49 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ea0-x22a.google.com (mail-ea0-x22a.google.com [IPv6:2a00:1450:4013:c01::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2C3882BA0; Thu, 15 Aug 2013 21:40:48 +0000 (UTC) Received: by mail-ea0-f170.google.com with SMTP id h14so634130eak.29 for ; Thu, 15 Aug 2013 14:40:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=LjxekYHvC+iIKiI94sZqU16MN2ruOFa5AWPUBnQ+vtM=; b=js3WYVdRQFzSNF1xwd4hhesO53tCk9qjFYpYjKhwb1goSQcXysQkN3N9J4smm1lKrQ HkO4bXrlDb9QHLPBFhBU3ZJ5yoyOdrFipGVbG19IsNvy5hLZQxEbBwEJ1YmacLHNlJ87 2kDneupclCz583g3VVxSiuzg9q+UgNQSUBwJPeRPnBgJlSMjBYGFyVbnwlpNw+WzTqGe a0uAwObk3zBDRI5IWNV7yBTZD3tnY+B6BUj+u4WrVk9ZfjPJyMRmr+lSQhB8rP68Th9j z1U8iFDFEmRJCtaZpiFXvkFXQRPcAAuQqB3M96gvGsDG+dsY1r4dtQ/NVxe3WvZbQIpP xb+w== X-Received: by 10.15.90.132 with SMTP id q4mr180193eez.98.1376602846545; Thu, 15 Aug 2013 14:40:46 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([37.229.21.195]) by mx.google.com with ESMTPSA id k7sm1995775eeg.13.2013.08.15.14.40.44 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 15 Aug 2013 14:40:45 -0700 (PDT) Sender: Alexander Motin Message-ID: <520D4ADB.50209@FreeBSD.org> Date: Fri, 16 Aug 2013 00:40:43 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: FreeBSD SCSI , freebsd-hackers@FreeBSD.org, "Justin T. Gibbs" , Scott Long , ken , Jeff Roberson , Steven Hartland Subject: New CAM locking preview Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Aug 2013 21:40:49 -0000 Hi. Last weeks I've made substantial progress on my CAM locking work. In fact, at this moment I think I've tied all loose ends good enough to consider the new design viable and implementation worth further testing and bug fixing. So I would like to ask for review of my work from everybody who interested in CAM internals. In short, my idea was to split single per-SIM lock, that creates huge congestion under high IOPS, into several smaller ones. So design I've finally chosen includes such locks: 1) New per-device (per-LUN) locks to protect state of the devices and respective periphs. In most cases peripheral drivers just use that lock instead of SIM lock used before, so code modification is minimal and straightforward. 2) New per-target lock to protect list of LUNs fetched from the device. 3) Old single per-SIM lock to protect SIM driver internals, but only that. No parts of CAM itself use that lock. Keeping it for SIMs allows to keep API and hopefully ABI compatibility. Reducing its scope allows to reduce congestion. 4) New per-SIM lock to protect SIM and device command queues. That allows execute queued commands from any context unrelated to other locks. Also this lock serializes accesses to sim_action() method for the most of commands, this allows to mostly avoid busy spilling on SIM lock collision. 5) New per-bus locks to protect target, device and periphs reference counters. It allows to create and destroy paths unrelated to other locks in any possible context. Numbers above also define supposed lock ordering: while holding per-device lock 1) is allowed to request SIM lock 3), but not backward. Cases where opposite is required (command completions and async events) are handled via queuing events via several completion threads. The rest of locks are self-contained and does not really suppose cascading. All these changes combined with GEOM direct dispatch (it will be next separate project) allow to double system performance in disk I/O microbenchmarks, comparing to present head, same as it was announced on 2013-05 DevSummit: http://people.freebsd.org/~mav/camlock.pdf . Tests without GEOM changes also show performance improvement, but limited by heavy bottleneck at the GEOM g_up/g_down threads at the level of 5-20%. Project sources could be found at SVN projects/camlock branch: http://svnweb.freebsd.org/base/projects/camlock/ . Many early changes from that branch are already integrated to head, so to simplify review the rest patches for changes before r254059 were manually remade and could be found here: http://people.freebsd.org/~mav/camlock_patches/ . These changes do not require controller driver modifications, keeping KPIs and hopefully KBIs intact, but create base for later work to use multiqueue capabilities of new controllers. This work is sponsored by iXsystems, Inc. -- Alexander Motin From owner-freebsd-scsi@FreeBSD.ORG Fri Aug 16 00:37:07 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 17A909B8; Fri, 16 Aug 2013 00:37:07 +0000 (UTC) (envelope-from hps@bitfrost.no) Received: from mta.bitpro.no (mta.bitpro.no [92.42.64.202]) by mx1.freebsd.org (Postfix) with ESMTP id 98AE022F0; Fri, 16 Aug 2013 00:37:06 +0000 (UTC) Received: from mail.lockless.no (mail.lockless.no [46.29.221.38]) by mta.bitpro.no (Postfix) with ESMTP id 0F59D7A28D; Fri, 16 Aug 2013 02:36:59 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mail.lockless.no (Postfix) with ESMTP id 978818EE897; Fri, 16 Aug 2013 02:37:08 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.6.4 (20090625) (Debian) at lockless.no Received: from mail.lockless.no ([127.0.0.1]) by localhost (mail.lockless.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OCgvGLR2GiZr; Fri, 16 Aug 2013 02:37:07 +0200 (CEST) Received: from laptop015.hselasky.homeunix.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) by mail.lockless.no (Postfix) with ESMTPSA id 750858EE895; Fri, 16 Aug 2013 02:37:07 +0200 (CEST) Message-ID: <520D7479.3060505@bitfrost.no> Date: Fri, 16 Aug 2013 02:38:17 +0200 From: Hans Petter Selasky Organization: Bitfrost A/S User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130522 Thunderbird/17.0.6 MIME-Version: 1.0 To: Alexander Motin Subject: Re: New CAM locking preview References: <520D4ADB.50209@FreeBSD.org> In-Reply-To: <520D4ADB.50209@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Scott Long , Jeff Roberson , ken , freebsd-hackers@FreeBSD.org, FreeBSD SCSI , Steven Hartland , "Justin T. Gibbs" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Aug 2013 00:37:07 -0000 On 08/15/13 23:40, Alexander Motin wrote: > Hi. > > Last weeks I've made substantial progress on my CAM locking work. In > fact, at this moment I think I've tied all loose ends good enough to > consider the new design viable and implementation worth further testing > and bug fixing. So I would like to ask for review of my work from > everybody who interested in CAM internals. > > In short, my idea was to split single per-SIM lock, that creates huge > congestion under high IOPS, into several smaller ones. So design I've > finally chosen includes such locks: > 1) New per-device (per-LUN) locks to protect state of the devices and > respective periphs. In most cases peripheral drivers just use that lock > instead of SIM lock used before, so code modification is minimal and > straightforward. > 2) New per-target lock to protect list of LUNs fetched from the device. > 3) Old single per-SIM lock to protect SIM driver internals, but only > that. No parts of CAM itself use that lock. Keeping it for SIMs allows > to keep API and hopefully ABI compatibility. Reducing its scope allows > to reduce congestion. > 4) New per-SIM lock to protect SIM and device command queues. That > allows execute queued commands from any context unrelated to other > locks. Also this lock serializes accesses to sim_action() method for the > most of commands, this allows to mostly avoid busy spilling on SIM lock > collision. > 5) New per-bus locks to protect target, device and periphs reference > counters. It allows to create and destroy paths unrelated to other locks > in any possible context. > Sounds very good! I assume you have tested USB mass storage device unplug during various file system operations? --HPS From owner-freebsd-scsi@FreeBSD.ORG Fri Aug 16 00:45:44 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5408DB84; Fri, 16 Aug 2013 00:45:44 +0000 (UTC) (envelope-from prvs=1940958969=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 04DCB2350; Fri, 16 Aug 2013 00:45:42 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50005518017.msg; Fri, 16 Aug 2013 01:45:40 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Fri, 16 Aug 2013 01:45:40 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=1940958969=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <86A1FDF5CDCD42DD8CE438576DBB1839@multiplay.co.uk> From: "Steven Hartland" To: "Hans Petter Selasky" , "Alexander Motin" References: <520D4ADB.50209@FreeBSD.org> <520D7479.3060505@bitfrost.no> Subject: Re: New CAM locking preview Date: Fri, 16 Aug 2013 01:45:58 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: Scott Long , Jeff Roberson , ken , freebsd-hackers@FreeBSD.org, FreeBSD SCSI , "Justin T. Gibbs" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Aug 2013 00:45:44 -0000 ----- Original Message ----- From: "Hans Petter Selasky" To: "Alexander Motin" Cc: "Scott Long" ; "Jeff Roberson" ; "ken" ; ; "FreeBSD SCSI" ; "Steven Hartland" ; "Justin T. Gibbs" Sent: Friday, August 16, 2013 1:38 AM Subject: Re: New CAM locking preview > On 08/15/13 23:40, Alexander Motin wrote: >> Hi. >> >> Last weeks I've made substantial progress on my CAM locking work. In >> fact, at this moment I think I've tied all loose ends good enough to >> consider the new design viable and implementation worth further testing >> and bug fixing. So I would like to ask for review of my work from >> everybody who interested in CAM internals. >> >> In short, my idea was to split single per-SIM lock, that creates huge >> congestion under high IOPS, into several smaller ones. So design I've >> finally chosen includes such locks: >> 1) New per-device (per-LUN) locks to protect state of the devices and >> respective periphs. In most cases peripheral drivers just use that lock >> instead of SIM lock used before, so code modification is minimal and >> straightforward. >> 2) New per-target lock to protect list of LUNs fetched from the device. >> 3) Old single per-SIM lock to protect SIM driver internals, but only >> that. No parts of CAM itself use that lock. Keeping it for SIMs allows >> to keep API and hopefully ABI compatibility. Reducing its scope allows >> to reduce congestion. >> 4) New per-SIM lock to protect SIM and device command queues. That >> allows execute queued commands from any context unrelated to other >> locks. Also this lock serializes accesses to sim_action() method for the >> most of commands, this allows to mostly avoid busy spilling on SIM lock >> collision. >> 5) New per-bus locks to protect target, device and periphs reference >> counters. It allows to create and destroy paths unrelated to other locks >> in any possible context. >> > > Sounds very good! I assume you have tested USB mass storage device unplug during various file system operations? It does indeed sound like some very good work, thanks Alexander! @Hans if USB mass storage device unplug is something important to you then might I suggest its a good idea to grab the patches, run your own tests and report any issues you might find as I'm sure this would be most appreciated :) Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-scsi@FreeBSD.ORG Fri Aug 16 01:12:09 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 39F69F86; Fri, 16 Aug 2013 01:12:09 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-x22e.google.com (mail-wg0-x22e.google.com [IPv6:2a00:1450:400c:c00::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AF1E02472; Fri, 16 Aug 2013 01:12:07 +0000 (UTC) Received: by mail-wg0-f46.google.com with SMTP id k13so1073334wgh.13 for ; Thu, 15 Aug 2013 18:12:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=PHq+4h2tXFQIradobpqSO0JKQm6cWJqoq9eumez+nrI=; b=Pt/Gkz6gbsbEVj37cdNsoGvKIs60uRHqhlS4IciYXcT8OavkzuWUs2yHGjz5xvOXut ZmIp13BkFSx29IVlKd/PqW92zmiuQm2gc+V+LnfNvTYBuYOzsj1Tkbadb5171WYlx4IX kO5Y7cMFuLTp9rpfsjZcABwngj+qJ2kOdMEIGjT+bKgW+qH1k0AOKM2adLua+p7x8bTP IGZV8WNfca+rYwG7b2SqC4zWTgi4GokBQhvusgiyQWhT+G5Li/4To7ezDgyIfqDXGTGN gAp8GwNcBebY4VUxHY6ai7TlaGrLNSog1g2NqgsPYfaKtY937mauZjsyW0ZXDf7QRoiM D/UA== MIME-Version: 1.0 X-Received: by 10.180.37.16 with SMTP id u16mr226772wij.46.1376615525882; Thu, 15 Aug 2013 18:12:05 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.116.136 with HTTP; Thu, 15 Aug 2013 18:12:05 -0700 (PDT) In-Reply-To: <520D4ADB.50209@FreeBSD.org> References: <520D4ADB.50209@FreeBSD.org> Date: Thu, 15 Aug 2013 18:12:05 -0700 X-Google-Sender-Auth: D6pqo5evTjWdHMuJBBeiW4ha7JA Message-ID: Subject: Re: New CAM locking preview From: Adrian Chadd To: Alexander Motin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Scott Long , Jeff Roberson , ken , "freebsd-hackers@freebsd.org" , FreeBSD SCSI , Steven Hartland , "Justin T. Gibbs" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Aug 2013 01:12:09 -0000 Cool! I assume you've run this with full witness debugging enabled, to catch lock ordering issues? This is great. I look forward to per-CPU, pinned, completion threads that I can do interesting things with (like schedule aio-sendfile completions..) -adrian On 15 August 2013 14:40, Alexander Motin wrote: > Hi. > > Last weeks I've made substantial progress on my CAM locking work. In fact, > at this moment I think I've tied all loose ends good enough to consider the > new design viable and implementation worth further testing and bug fixing. > So I would like to ask for review of my work from everybody who interested > in CAM internals. > > In short, my idea was to split single per-SIM lock, that creates huge > congestion under high IOPS, into several smaller ones. So design I've > finally chosen includes such locks: > 1) New per-device (per-LUN) locks to protect state of the devices and > respective periphs. In most cases peripheral drivers just use that lock > instead of SIM lock used before, so code modification is minimal and > straightforward. > 2) New per-target lock to protect list of LUNs fetched from the device. > 3) Old single per-SIM lock to protect SIM driver internals, but only > that. No parts of CAM itself use that lock. Keeping it for SIMs allows to > keep API and hopefully ABI compatibility. Reducing its scope allows to > reduce congestion. > 4) New per-SIM lock to protect SIM and device command queues. That allows > execute queued commands from any context unrelated to other locks. Also > this lock serializes accesses to sim_action() method for the most of > commands, this allows to mostly avoid busy spilling on SIM lock collision. > 5) New per-bus locks to protect target, device and periphs reference > counters. It allows to create and destroy paths unrelated to other locks in > any possible context. > > Numbers above also define supposed lock ordering: while holding per-device > lock 1) is allowed to request SIM lock 3), but not backward. Cases where > opposite is required (command completions and async events) are handled via > queuing events via several completion threads. The rest of locks are > self-contained and does not really suppose cascading. > > All these changes combined with GEOM direct dispatch (it will be next > separate project) allow to double system performance in disk I/O > microbenchmarks, comparing to present head, same as it was announced on > 2013-05 DevSummit: http://people.freebsd.org/~**mav/camlock.pdf. Tests without GEOM changes also show performance improvement, but limited > by heavy bottleneck at the GEOM g_up/g_down threads at the level of 5-20%. > > Project sources could be found at SVN projects/camlock branch: > http://svnweb.freebsd.org/**base/projects/camlock/. Many early changes from that branch are already integrated to head, so to > simplify review the rest patches for changes before r254059 were manually > remade and could be found here: http://people.freebsd.org/~** > mav/camlock_patches/ . > > These changes do not require controller driver modifications, keeping KPIs > and hopefully KBIs intact, but create base for later work to use multiqueue > capabilities of new controllers. > > This work is sponsored by iXsystems, Inc. > > -- > Alexander Motin > ______________________________**_________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-**hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@** > freebsd.org " > From owner-freebsd-scsi@FreeBSD.ORG Fri Aug 16 06:51:27 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2930AD5; Fri, 16 Aug 2013 06:51:27 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-x232.google.com (mail-ee0-x232.google.com [IPv6:2a00:1450:4013:c00::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8910F23BE; Fri, 16 Aug 2013 06:51:25 +0000 (UTC) Received: by mail-ee0-f50.google.com with SMTP id d51so768042eek.37 for ; Thu, 15 Aug 2013 23:51:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=KnwRBAjKnIuR3HOQY/riLwATfVN+0cHyD8Je4BbaQb0=; b=tEL4dRivnVqSd9ABdiJRI1703tNB6meeYFHwaCdRD29PhK9s1paMWLyOj212wIHCXf 150+zMDUtfu8kshqajvT8CDAVXTvNsWbKKeoBgA+3X9N2jFoMj2IILyFpDAvk/Bxp9g5 NzpXVJgQUFTRtsMNDxb0qtW/6Rk8g0VdPgyggszmDYIUaLwRdR/aQbWWqrx7AP4D6njM cE5RG9EYIl+0i+T4rEIlWycQLd/BgJlrL2l7dcZ5fb0jPLE6vjaw7VMy5gK5WDyseoiw /X4AJGraTCpRujGgcdO/fAJzxMRbr0S0G87iqoGnRDARCvEvkYjA3QzmLnvtyYdMooxr 9GLQ== X-Received: by 10.15.76.71 with SMTP id m47mr26109eey.85.1376635883909; Thu, 15 Aug 2013 23:51:23 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([37.229.21.195]) by mx.google.com with ESMTPSA id bq1sm210635eeb.9.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 15 Aug 2013 23:51:22 -0700 (PDT) Sender: Alexander Motin Message-ID: <520DCBE8.9050703@FreeBSD.org> Date: Fri, 16 Aug 2013 09:51:20 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: Adrian Chadd Subject: Re: New CAM locking preview References: <520D4ADB.50209@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Scott Long , Jeff Roberson , ken , "freebsd-hackers@freebsd.org" , FreeBSD SCSI , Steven Hartland , "Justin T. Gibbs" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Aug 2013 06:51:27 -0000 On 16.08.2013 04:12, Adrian Chadd wrote: > Cool! > > I assume you've run this with full witness debugging enabled, to catch > lock ordering issues? Of course! I've endless times switched between debug and normal builds to test both correctness and performance after each change. But more external tests are welcome. > This is great. I look forward to per-CPU, pinned, completion threads > that I can do interesting things with (like schedule aio-sendfile > completions..) > > On 15 August 2013 14:40, Alexander Motin > wrote: > > Hi. > > Last weeks I've made substantial progress on my CAM locking work. In > fact, at this moment I think I've tied all loose ends good enough to > consider the new design viable and implementation worth further > testing and bug fixing. So I would like to ask for review of my work > from everybody who interested in CAM internals. > > In short, my idea was to split single per-SIM lock, that creates > huge congestion under high IOPS, into several smaller ones. So > design I've finally chosen includes such locks: > 1) New per-device (per-LUN) locks to protect state of the devices > and respective periphs. In most cases peripheral drivers just use > that lock instead of SIM lock used before, so code modification is > minimal and straightforward. > 2) New per-target lock to protect list of LUNs fetched from the > device. > 3) Old single per-SIM lock to protect SIM driver internals, but > only that. No parts of CAM itself use that lock. Keeping it for SIMs > allows to keep API and hopefully ABI compatibility. Reducing its > scope allows to reduce congestion. > 4) New per-SIM lock to protect SIM and device command queues. That > allows execute queued commands from any context unrelated to other > locks. Also this lock serializes accesses to sim_action() method for > the most of commands, this allows to mostly avoid busy spilling on > SIM lock collision. > 5) New per-bus locks to protect target, device and periphs > reference counters. It allows to create and destroy paths unrelated > to other locks in any possible context. > > Numbers above also define supposed lock ordering: while holding > per-device lock 1) is allowed to request SIM lock 3), but not > backward. Cases where opposite is required (command completions and > async events) are handled via queuing events via several completion > threads. The rest of locks are self-contained and does not really > suppose cascading. > > All these changes combined with GEOM direct dispatch (it will be > next separate project) allow to double system performance in disk > I/O microbenchmarks, comparing to present head, same as it was > announced on 2013-05 DevSummit: > http://people.freebsd.org/~__mav/camlock.pdf > . Tests without GEOM > changes also show performance improvement, but limited by heavy > bottleneck at the GEOM g_up/g_down threads at the level of 5-20%. > > Project sources could be found at SVN projects/camlock branch: > http://svnweb.freebsd.org/__base/projects/camlock/ > . Many early > changes from that branch are already integrated to head, so to > simplify review the rest patches for changes before r254059 were > manually remade and could be found here: > http://people.freebsd.org/~__mav/camlock_patches/ > . > > These changes do not require controller driver modifications, > keeping KPIs and hopefully KBIs intact, but create base for later > work to use multiqueue capabilities of new controllers. > > This work is sponsored by iXsystems, Inc. -- Alexander Motin From owner-freebsd-scsi@FreeBSD.ORG Fri Aug 16 08:16:11 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 22AC7F0F; Fri, 16 Aug 2013 08:16:11 +0000 (UTC) (envelope-from jlh@FreeBSD.org) Received: from caravan.chchile.org (caravan.chchile.org [178.32.125.136]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A7B752743; Fri, 16 Aug 2013 08:16:10 +0000 (UTC) Received: by caravan.chchile.org (Postfix, from userid 1000) id 504B7B70D2; Fri, 16 Aug 2013 08:16:03 +0000 (UTC) Date: Fri, 16 Aug 2013 10:16:03 +0200 From: Jeremie Le Hen To: Alexander Motin Subject: Re: New CAM locking preview Message-ID: <20130816081603.GA4984@caravan.chchile.org> Mail-Followup-To: Alexander Motin , FreeBSD SCSI , freebsd-hackers@FreeBSD.org, "Justin T. Gibbs" , Scott Long , ken , Jeff Roberson , Steven Hartland References: <520D4ADB.50209@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <520D4ADB.50209@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Scott Long , Jeff Roberson , ken , freebsd-hackers@FreeBSD.org, FreeBSD SCSI , Steven Hartland , "Justin T. Gibbs" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Aug 2013 08:16:11 -0000 On Fri, Aug 16, 2013 at 12:40:43AM +0300, Alexander Motin wrote: > Hi. > > Last weeks I've made substantial progress on my CAM locking work. In > fact, at this moment I think I've tied all loose ends good enough to > consider the new design viable and implementation worth further testing > and bug fixing. So I would like to ask for review of my work from > everybody who interested in CAM internals. > > In short, my idea was to split single per-SIM lock, that creates huge > congestion under high IOPS, into several smaller ones. So design I've > finally chosen includes such locks: > 1) New per-device (per-LUN) locks to protect state of the devices and > respective periphs. In most cases peripheral drivers just use that lock > instead of SIM lock used before, so code modification is minimal and > straightforward. > 2) New per-target lock to protect list of LUNs fetched from the device. > 3) Old single per-SIM lock to protect SIM driver internals, but only > that. No parts of CAM itself use that lock. Keeping it for SIMs allows > to keep API and hopefully ABI compatibility. Reducing its scope allows > to reduce congestion. > 4) New per-SIM lock to protect SIM and device command queues. That > allows execute queued commands from any context unrelated to other > locks. Also this lock serializes accesses to sim_action() method for the > most of commands, this allows to mostly avoid busy spilling on SIM lock > collision. > 5) New per-bus locks to protect target, device and periphs reference > counters. It allows to create and destroy paths unrelated to other locks > in any possible context. > > Numbers above also define supposed lock ordering: while holding > per-device lock 1) is allowed to request SIM lock 3), but not backward. > Cases where opposite is required (command completions and async events) > are handled via queuing events via several completion threads. The rest > of locks are self-contained and does not really suppose cascading. > > All these changes combined with GEOM direct dispatch (it will be next > separate project) allow to double system performance in disk I/O > microbenchmarks, comparing to present head, same as it was announced on > 2013-05 DevSummit: http://people.freebsd.org/~mav/camlock.pdf . Tests > without GEOM changes also show performance improvement, but limited by > heavy bottleneck at the GEOM g_up/g_down threads at the level of 5-20%. > > Project sources could be found at SVN projects/camlock branch: > http://svnweb.freebsd.org/base/projects/camlock/ . Many early changes > from that branch are already integrated to head, so to simplify review > the rest patches for changes before r254059 were manually remade and > could be found here: http://people.freebsd.org/~mav/camlock_patches/ . > > These changes do not require controller driver modifications, keeping > KPIs and hopefully KBIs intact, but create base for later work to use > multiqueue capabilities of new controllers. > > This work is sponsored by iXsystems, Inc. Excellent, thanks to both you and iXsystems. I'm eager to see everything merged to -CURRENT before the code slush ;). -- Jeremie Le Hen Scientists say the world is made up of Protons, Neutrons and Electrons. They forgot to mention Morons.