From owner-freebsd-amd64@FreeBSD.ORG Fri Mar 17 13:04:45 2006 Return-Path: X-Original-To: amd64@freebsd.org Delivered-To: freebsd-amd64@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5644216A401 for ; Fri, 17 Mar 2006 13:04:45 +0000 (UTC) (envelope-from jc@oxado.com) Received: from mars.interactivemediafactory.net (ci-77.custnet-1.n-3.pari1.eu.psigh.com [62.50.134.77]) by mx1.FreeBSD.org (Postfix) with ESMTP id D345B43D45 for ; Fri, 17 Mar 2006 13:04:44 +0000 (GMT) (envelope-from jc@oxado.com) Received: from JC-8600.oxado.com (localhost [127.0.0.1]) by mars.interactivemediafactory.net (8.12.11/8.12.11) with ESMTP id k2HD4a2o051735; Fri, 17 Mar 2006 14:04:41 +0100 (CET) (envelope-from jc@oxado.com) Message-Id: <7.0.0.16.0.20060317133742.03ceb0c8@imfeurope.com> X-Mailer: QUALCOMM Windows Eudora Version 7.0.0.16 Date: Fri, 17 Mar 2006 14:02:53 +0100 To: Robert Leftwich From: Jacques Caron In-Reply-To: <441946A7.7090302@rtl.fmailbox.com> References: <4411D6D8.5030101@wmptl.com> <44122C1C.6030301@samsco.org> <441946A7.7090302@rtl.fmailbox.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Cc: amd64@freebsd.org Subject: Re: 6.0-RELEASE/AMD64 Ram Capacity? X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Mar 2006 13:04:45 -0000 Hi, At 12:06 16/03/2006, Robert Leftwich wrote: >>Scott Long writes: >> >>>The ATA driver is still a bit flakey with more than 4GB of RAM, so >>>make sure that you are using either a modern SCSI controller or a >>>RAID controller. > >Would you mind clarifying if this applies to *only* > 4GB of ram or >if it includes exactly 4GB? Also does PAE impact this flakey-ness? >Are there any particular chipsets that are flaky/non-flaky? Are all >FreeBSD 6.x versions affected? > >I guess what it boils down to is that I'm running an asus a8n-sli >premium with 4GB ram and 4*160GB SATA2 drives in raid 0+1 (via the >onboard nVidia (pseudo) raid) and currently using 6.1beta3 (as 6.0x >does not recognise all 4GB of ram) - should I be worried that the >drives are going to be flaky? It will affect any machine with more than 2 (S)ATA drives where some of the memory is above the 32-bit limit, since it requires the busdma code to allocate bounce buffers under the 32-bit limit, and the busdma code has a hard limit on the amount of bounce buffers it allocates for a given type in some conditions (in 5.4 at least, haven't checked what the status is in 6.0 and later), and the ATA driver wants more. To check if things work, simply do (on a box that you're ready to see panic): dd if=/dev/adX of=/dev/null bs=512k & dd if=/dev/adY of=/dev/null bs=512k & dd if=/dev/adZ of=/dev/null bs=512k & dd if=/dev/adT of=/dev/null bs=512k & where X,Y,Z,T are the device numbers of your different drives. If your box is still alive after your have entered the 4th line you should be safe (on 5.4 you actually won't even get to the 4th line, a panic will interfere with your day). You may also want to check sysctl -a | grep busdma before/while you run the test: hw.busdma.zone1.total_bpages: 320 hw.busdma.zone1.free_bpages: 320 hw.busdma.zone1.reserved_bpages: 0 hw.busdma.zone1.active_bpages: 0 hw.busdma.zone1.total_bounced: 763930088 hw.busdma.zone1.total_deferred: 0 hw.busdma.zone1.lowaddr: 0xffffffff hw.busdma.zone1.alignment: 2 hw.busdma.zone1.boundary: 65536 On a stock 5.4 total_bpages will be much lower than 320, and active_bpages will reach the limit pretty soon (once the 2nd dd is running) and kaboom (once your launch the 3rd dd) since the driver expects the pages to be there (busdma tells it they are). I run my 5.4 boxes with the following patch: diff -c sys/dev/ata/ata-dma.c.orig sys/dev/ata/ata-dma.c *** sys/dev/ata/ata-dma.c.orig Fri Sep 2 16:14:19 2005 --- sys/dev/ata/ata-dma.c Thu Oct 27 12:40:12 2005 *************** *** 111,117 **** BUS_SPACE_MAXADDR_32BIT, BUS_SPACE_MAXADDR, NULL, NULL, 256 * DEV_BSIZE, ATA_DMA_ENTRIES, ch->dma->max_iosize, ! BUS_DMA_ALLOCNOW, NULL, NULL, &ch->dma->ddmatag)) goto error; if (bus_dmamem_alloc(ch->dma->cdmatag, (void **)&ch->dma->dmatab, 0, --- 111,117 ---- BUS_SPACE_MAXADDR_32BIT, BUS_SPACE_MAXADDR, NULL, NULL, 256 * DEV_BSIZE, ATA_DMA_ENTRIES, ch->dma->max_iosize, ! 0, NULL, NULL, &ch->dma->ddmatag)) goto error; if (bus_dmamem_alloc(ch->dma->cdmatag, (void **)&ch->dma->dmatab, 0, It is not the "right" fix (the problem is in busdma, not ata, or at least it used to be in 5.4), but it's a workaround that works pretty well: busdma will gladly allocate the bounce buffers and ata will happily use them. Note again that I really haven't checked if things have changed in post-5.4 versions, so maybe it works now. I believe Scott made at least one change somewhere in there, but I haven't had the opportunity to look at it or test it. And obviously I can't tell you if the above patch works with anything other than 5.4. Final note: bounce buffer allocation is *very* slow, so with the above patch it takes a few seconds for each ata driver to initialize. Hope that helps, Jacques.