Date: Thu, 3 Sep 2009 21:45:07 +0200 From: Ivan Voras <ivoras@freebsd.org> To: Scott Long <scottl@samsco.org> Cc: svn-src-head@freebsd.org, Alexander Motin <mav@freebsd.org>, src-committers@freebsd.org, svn-src-all@freebsd.org Subject: Re: svn commit: r196777 - head/sys/dev/ahci Message-ID: <9bbcef730909031245o7c380925sd29b2cc976c4d7dd@mail.gmail.com> In-Reply-To: <20090903114121.C20031@pooker.samsco.org> References: <200909031237.n83CbIgk032551@svn.freebsd.org> <1872D962-9297-4C45-9F73-4BB823C49D74@samsco.org> <4A9FD8B4.2080605@FreeBSD.org> <20090903095224.N20031@pooker.samsco.org> <9bbcef730909031037y4aecd692t4812718b1fd7e78e@mail.gmail.com> <20090903114121.C20031@pooker.samsco.org>
next in thread | previous in thread | raw e-mail | index | archive | help
2009/9/3 Scott Long <scottl@samsco.org>: > The problem is lack of kernel address space, not lack of RAM, but that's Yes. > just semantics in this discussion. =C2=A0I've tested with increasing MAXP= HYS in > increments to 1M. =C2=A0Performance increases logrithmically, and effecti= vely > hits a max at 512K for the variety of controllers that I tested. =C2=A0Th= e gain > from 64K to 128K is huge, the gain from 128K to 256K is ok, the gain from > 256k to 512k is measurable but less significant, and the gain from 512k t= o > 1m is almost not measurable. >From what I've seen with iostat, all machines I can currently test (SATA, various SCSI-like RAID, QLogic FC HBA), 64 k is the maximum transfer size, which I assume is DFLTPHYS. From your tests it's apparent it is suboptimal... > I have simple patches to increase MAXPHYS. =C2=A0The introduction of the = the > maxio paramter in the CAM SIM interface is there in preparation for this. > However, a _LOT_LOT_LOT_ of drivers in the tree falsely assume that MAXPH= YS > and DEFLTPHYS are 128k and 64k respectively, and size their data structur= es > accordingly. =C2=A0Changing these values will cause the drivers to fail i= n bad > ways. =C2=A0So an audit needs to be done. =C2=A0Also, MAXPHYS is abused b= y the swapper > in the struct-buf, so that needs to be reviewed as well. Since GEOM is ok with it, couldn't a migration path be to introduce a new tunable, something like LARGE_MAXPHYS (or to rename it completely to MAX_IO) or whatever and use it (after reviewing cases of abuses) in new and better maintained drivers? If the new size is larger than the old one, maybe the maximum damage would be a little bit of wasted space now and then? I can test mostly ciss and mfi. I see mfi uses it in one place that looks kind of logical: 334 /* 335 * Get information needed for sizing the contiguous memory for= the 336 * frame pool. Size down the sgl parameter since we know that 337 * we will never need more than what's required for MAXPHYS. 338 * It would be nice if these constants were available at runti= me 339 * instead of compile time. 340 */ 341 status =3D sc->mfi_read_fw_status(sc); 342 sc->mfi_max_fw_cmds =3D status & MFI_FWSTATE_MAXCMD_MASK; 343 max_fw_sge =3D (status & MFI_FWSTATE_MAXSGL_MASK) >> 16; 344 sc->mfi_max_sge =3D min(max_fw_sge, ((MAXPHYS / PAGE_SIZE) + 1= )); But ciss doesn't reference it at all so either it deviously assumes it or is independent of it. > Even though kernel address space is less restricted on 64bit platforms, i= t's > still not free and limitless. =C2=A0Large I/O's requires more work in the= VM to > assign address space, and in turn causes more lock contention. =C2=A0I ha= ven't > done any practical measurements of this on common workloads, but I can > anecdotally say that I see increased lock contention from it in locking > profiles. =C2=A0If FreeBSD wants to seriously increase MAXPHYS, this need= s to be > looked at and either proven to not be important, or fixed. I think the major benefits would be for the server-side crowd using RAID modes with some kind of striping. --=20 f+rEnSIBITAhITAhLR1nM9F4cIs5KJrhbcsVtUIt7K1MhWJy1A=3D=3D
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9bbcef730909031245o7c380925sd29b2cc976c4d7dd>