From owner-freebsd-performance@FreeBSD.ORG Wed Dec 26 00:47:26 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F63116A421 for ; Wed, 26 Dec 2007 00:47:26 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.183]) by mx1.freebsd.org (Postfix) with ESMTP id ADF5713C469 for ; Wed, 26 Dec 2007 00:47:25 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so4110961pyb.3 for ; Tue, 25 Dec 2007 16:47:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=VdJ/Dj63At+wYijzZ0lovvtqcKm3C4vv+6CGubJL87Q=; b=UZ4b4kFcF2/PKPN0aVMWVFewoT0ghX17MlGwzOcLPYRFYHWC7iIGbOxg4GHslQX9fiZYZ6YAa2a5Z7sQ3M6nU+Qas4y7kW0t9kOtU8eiGmUpVqx+ab5+oFgYiyc/oUAyZnLuA8oq/5idHvtIsu+2tAjk4vbat71cJn5mJmHBxBk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=djeny/r2mRrvhhhP+Ujrf6wJ3NvXqA2+qhWmmN1XGQSeKdR5MPAQJbsFxQSvrWhW3Ke6kDgIUcoin+f4NMxxaWlJE2VQzv7BFh1OBsXnx5nQ9OkpcSeZXNdDyG66FzD5uiQzrQjgOBpvwznhSdlJxPpU6m685pWwCvijvqCrTQU= Received: by 10.65.252.13 with SMTP id e13mr11386973qbs.84.1198630044965; Tue, 25 Dec 2007 16:47:24 -0800 (PST) Received: by 10.64.184.9 with HTTP; Tue, 25 Dec 2007 16:47:24 -0800 (PST) Message-ID: <8e10486b0712251647uddf80f9sa6cacb42dfa94ff5@mail.gmail.com> Date: Tue, 25 Dec 2007 21:47:24 -0300 From: "Alexandre Biancalana" To: "Alfred Perlstein" In-Reply-To: <20071222002535.GL16982@elvis.mu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> <20071221201625.GZ16982@elvis.mu.org> <8e10486b0712211249v4c5571ddud21b277f686992b2@mail.gmail.com> <20071221212808.GE16982@elvis.mu.org> <8e10486b0712211555n3efe8729qff14387be128cf10@mail.gmail.com> <20071222002535.GL16982@elvis.mu.org> Cc: freebsd-performance@freebsd.org Subject: Re: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Dec 2007 00:47:26 -0000 On 12/21/07, Alfred Perlstein wrote: > Have you tried the 'noatime' mount option? That should help. Yeah, the zfs set atime=off was already done.... > > Can you provide a histogram of the count of files per directory? Excuse-me, but I don't understand.... From owner-freebsd-performance@FreeBSD.ORG Wed Dec 26 18:35:11 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F52416A418; Wed, 26 Dec 2007 18:35:11 +0000 (UTC) (envelope-from jcmichot@flash.usenet-fr.net) Received: from flash.usenet-fr.net (flash.usenet-fr.net [88.174.64.5]) by mx1.freebsd.org (Postfix) with ESMTP id BE7E013C474; Wed, 26 Dec 2007 18:35:10 +0000 (UTC) (envelope-from jcmichot@flash.usenet-fr.net) Received: by flash.usenet-fr.net (Postfix, from userid 200) id 9C79195AE3; Wed, 26 Dec 2007 19:03:27 +0100 (CET) Date: Wed, 26 Dec 2007 19:03:27 +0100 From: Jean-Claude MICHOT To: freebsd-performance@freebsd.org, freebsd-net@freebsd.org Message-ID: <20071226180327.GA39735@flash.usenet-fr.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.2i Cc: Subject: DELL PowerEdge 860 and Broadcom Gigabit Ethernet poor performance. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Dec 2007 18:35:11 -0000 The server is a DELL PowerEdge 860 freshly installed with FreeBSD 7.0-BETA4 (GENERIC Kernel). pciconf and part of boot information: bge0@pci0:4:0:0: class=0x020000 card=0x01e61028 chip=0x165914e4 rev=0x11 hdr=0x00 vendor = 'Broadcom Corporation' device = 'BCM5721 NetXtreme Gigabit Ethernet PCI Express' class = network subclass = ethernet bge1@pci0:5:0:0: class=0x020000 card=0x01e61028 chip=0x165914e4 rev=0x11 hdr=0x00 vendor = 'Broadcom Corporation' device = 'BCM5721 NetXtreme Gigabit Ethernet PCI Express' class = network subclass = ethernet bge0: mem 0xfe5f0000-0xfe5fffff irq 16 at device 0.0 on pci4 miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge0: Ethernet address: 00:1c:23:e1:78:7e bge0: [ITHREAD] bge1: mem 0xfe3f0000-0xfe3fffff irq 17 at device 0.0 on pci5 miibus1: on bge1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge1: Ethernet address: 00:1c:23:e1:78:7f bge1: [ITHREAD] There's no problem with input throughput (upto 980 Mbits) but output throughput never go upper to 540 Mbits :( Just for test, i have add to this server an Intel Gigabit Ethernet board (em) and there's no problem to output data up to around 980 Mbits with this addon board. If i boot the server with Linux Ubuntu, there's no output throughput problem with Broadcom, so it seem to be FreeBSD bge driver problem. I'm not the only one to have poor output performance with bge on DELL PowerEdge 860 http://lists.freebsd.org/pipermail/freebsd-net/2007-June/014373.html I have also try various patch or setup driver default values http://lists.freebsd.org/pipermail/freebsd-net/2007-November/015951.html http://lists.freebsd.org/pipermail/freebsd-net/2007-November/015956.html But all theses attempts to get better ouput performance was unsuccessful :( Any idea are welcome. PS: If it's usefull to debug and try to fix the problem, i can provide root access to a DELL PE860 test server with bge. -- UNIX is user-friendly. it's just selective about who its friends are. From owner-freebsd-performance@FreeBSD.ORG Wed Dec 26 19:49:15 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1A3016A46C for ; Wed, 26 Dec 2007 19:49:15 +0000 (UTC) (envelope-from netslists@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.243]) by mx1.freebsd.org (Postfix) with ESMTP id 90F9C13C4F9 for ; Wed, 26 Dec 2007 19:49:15 +0000 (UTC) (envelope-from netslists@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so531858anc.13 for ; Wed, 26 Dec 2007 11:49:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding; bh=wSuie2gWE8WpbKCkvIWWJxklWfv+MIevEV0YUTyLceE=; b=w5BSizMm7b9heSWMHD49ojencmMvKmrgFgpI72A+pZsEj8h+G+L5jg7MFb8uESJpmhHuLqZAUwnL0G09kvn6GyttxW0ArRxre3FZMc6E+r4M/xdrj8RSHm9kSN7F4x710YiB1qVd+dlf9SXE8qc6xnRk3GTlHHXpIhBLjRo9yQU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding; b=aKtKby+FDlXQh9dMas5D50q1tHtlBIBU1H8v/lHgBNTO1w+WqlyIsX6QUsdXZm4d6g/TqenFfjMTl1gfOO+B51B6Ktffvs1FHeWP2K+ammsKw8m3sdECbbE4o3pgKSj3Y48G9H8eSlgpzvUllMDxG2zNSSkzBnDxW+jmSOA0oNo= Received: by 10.100.110.15 with SMTP id i15mr14674910anc.76.1198697087677; Wed, 26 Dec 2007 11:24:47 -0800 (PST) Received: from ?192.168.12.8? ( [97.101.40.241]) by mx.google.com with ESMTPS id c40sm7839142anc.16.2007.12.26.11.24.46 (version=SSLv3 cipher=RC4-MD5); Wed, 26 Dec 2007 11:24:46 -0800 (PST) Message-ID: <4772AA7C.1020206@gmail.com> Date: Wed, 26 Dec 2007 14:24:44 -0500 From: Sten Daniel Soersdal User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Jean-Claude MICHOT References: <20071226180327.GA39735@flash.usenet-fr.net> In-Reply-To: <20071226180327.GA39735@flash.usenet-fr.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, freebsd-performance@freebsd.org Subject: Re: DELL PowerEdge 860 and Broadcom Gigabit Ethernet poor performance. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Dec 2007 19:49:15 -0000 Jean-Claude MICHOT wrote: > The server is a DELL PowerEdge 860 freshly installed with > FreeBSD 7.0-BETA4 (GENERIC Kernel). > > pciconf and part of boot information: > > bge0@pci0:4:0:0: class=0x020000 card=0x01e61028 chip=0x165914e4 rev=0x11 hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'BCM5721 NetXtreme Gigabit Ethernet PCI Express' > class = network > subclass = ethernet > bge1@pci0:5:0:0: class=0x020000 card=0x01e61028 chip=0x165914e4 rev=0x11 hdr=0x00 > vendor = 'Broadcom Corporation' > device = 'BCM5721 NetXtreme Gigabit Ethernet PCI Express' > class = network > subclass = ethernet > > bge0: mem 0xfe5f0000-0xfe5fffff irq 16 at device 0.0 on pci4 > miibus0: on bge0 > brgphy0: PHY 1 on miibus0 > brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto > bge0: Ethernet address: 00:1c:23:e1:78:7e > bge0: [ITHREAD] > bge1: mem 0xfe3f0000-0xfe3fffff irq 17 at device 0.0 on pci5 > miibus1: on bge1 > brgphy1: PHY 1 on miibus1 > brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto > bge1: Ethernet address: 00:1c:23:e1:78:7f > bge1: [ITHREAD] > > There's no problem with input throughput (upto 980 Mbits) but output > throughput never go upper to 540 Mbits :( > > Just for test, i have add to this server an Intel Gigabit Ethernet board > (em) and there's no problem to output data up to around 980 Mbits with > this addon board. > > If i boot the server with Linux Ubuntu, there's no output throughput > problem with Broadcom, so it seem to be FreeBSD bge driver problem. > > I'm not the only one to have poor output performance with bge on > DELL PowerEdge 860 > http://lists.freebsd.org/pipermail/freebsd-net/2007-June/014373.html > > I have also try various patch or setup driver default values > http://lists.freebsd.org/pipermail/freebsd-net/2007-November/015951.html > http://lists.freebsd.org/pipermail/freebsd-net/2007-November/015956.html > > But all theses attempts to get better ouput performance was unsuccessful :( > > Any idea are welcome. > > PS: If it's usefull to debug and try to fix the problem, i can provide > root access to a DELL PE860 test server with bge. > Have you tried setting the tcp send and receive windows? The defaults are: net.inet.tcp.sendspace: 32768 net.inet.tcp.recvspace: 65536 Also you might want to try to lower: net.inet.tcp.delacktime: 100 -- Sten Daniel Soersdal From owner-freebsd-performance@FreeBSD.ORG Thu Dec 27 01:18:52 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A48B316A419; Thu, 27 Dec 2007 01:18:52 +0000 (UTC) (envelope-from jcmichot@flash.usenet-fr.net) Received: from flash.usenet-fr.net (flash.usenet-fr.net [88.174.64.5]) by mx1.freebsd.org (Postfix) with ESMTP id 5B2A313C469; Thu, 27 Dec 2007 01:18:52 +0000 (UTC) (envelope-from jcmichot@flash.usenet-fr.net) Received: by flash.usenet-fr.net (Postfix, from userid 200) id D1E6895BD3; Thu, 27 Dec 2007 02:18:50 +0100 (CET) Date: Thu, 27 Dec 2007 02:18:50 +0100 From: Jean-Claude MICHOT To: Sten Daniel Soersdal Message-ID: <20071227011850.GA43415@flash.usenet-fr.net> References: <20071226180327.GA39735@flash.usenet-fr.net> <4772AA7C.1020206@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4772AA7C.1020206@gmail.com> User-Agent: Mutt/1.4.2.2i Cc: freebsd-net@freebsd.org, freebsd-performance@freebsd.org, Jean-Claude MICHOT Subject: Re: DELL PowerEdge 860 and Broadcom Gigabit Ethernet poor performance. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2007 01:18:52 -0000 On Wed, Dec 26, 2007 at 02:24:44PM -0500, Sten Daniel Soersdal said: > Jean-Claude MICHOT wrote: > >The server is a DELL PowerEdge 860 freshly installed with > >FreeBSD 7.0-BETA4 (GENERIC Kernel). > > > >There's no problem with input throughput (upto 980 Mbits) but output > >throughput never go upper to 540 Mbits :( > > > >Just for test, i have add to this server an Intel Gigabit Ethernet board > >(em) and there's no problem to output data up to around 980 Mbits with > >this addon board. > > > >If i boot the server with Linux Ubuntu, there's no output throughput > >problem with Broadcom, so it seem to be FreeBSD bge driver problem. > > > >I'm not the only one to have poor output performance with bge on > >DELL PowerEdge 860 > >http://lists.freebsd.org/pipermail/freebsd-net/2007-June/014373.html > > > >I have also try various patch or setup driver default values > >http://lists.freebsd.org/pipermail/freebsd-net/2007-November/015951.html > >http://lists.freebsd.org/pipermail/freebsd-net/2007-November/015956.html > > > >But all theses attempts to get better ouput performance was unsuccessful :( > > > >Any idea are welcome. > > > >PS: If it's usefull to debug and try to fix the problem, i can provide > > root access to a DELL PE860 test server with bge. > > Have you tried setting the tcp send and receive windows? > The defaults are: > > net.inet.tcp.sendspace: 32768 > net.inet.tcp.recvspace: 65536 > > Also you might want to try to lower: > > net.inet.tcp.delacktime: 100 Yes, no difference. But as i said, same freebsd "config" with Intel em board instead of Broadcom bge, and there's no output throuhgput problem. JC -- "Those people who think they know everything are a great annoyance to those of us who do." Isaac Asimov From owner-freebsd-performance@FreeBSD.ORG Fri Dec 28 17:03:58 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E929A16A420 for ; Fri, 28 Dec 2007 17:03:58 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.179]) by mx1.freebsd.org (Postfix) with ESMTP id 77B0013C506 for ; Fri, 28 Dec 2007 17:03:58 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so6573091pyb.3 for ; Fri, 28 Dec 2007 09:03:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=JKBuR5af+Ztj2dxsLD1BzBb5C7Hd2ImHBubhEdiRgaM=; b=aU+THgSG+PUX13v6sHuKpv+km8OXCiUEubyVvG3v63d0Am4SOprDwRfPuJRAc5OendGJvjqdjgrMmAMEmxJ+gxScnk25cBP1MQCmYBmlcfhmJWErac3PYeS3MKQF1VYPThgVSKZGdAd6d099lmyXW5WIiFZab02On7Vwo/Nm/ck= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=OBCVNfP3ccKXr80kTZ0Phd6YFswxIYRuOULyuhMotfwA2YQJVoVefWE3MjwHjmky6479GrPQVm+3V1RnLY+S4Gyfcj9vdjfYSq6q/0HQE4D0OASnAk6YBuBqaf2ZZqIUt+20dTlW6MILVg9clDT6gv3aL19ScTUAnxND1eHc/y0= Received: by 10.64.150.18 with SMTP id x18mr18364553qbd.0.1198861431128; Fri, 28 Dec 2007 09:03:51 -0800 (PST) Received: by 10.64.184.9 with HTTP; Fri, 28 Dec 2007 09:03:50 -0800 (PST) Message-ID: <8e10486b0712280903n431dd608nfb9e2fdf5f79d623@mail.gmail.com> Date: Fri, 28 Dec 2007 15:03:50 -0200 From: "Alexandre Biancalana" To: "Alfred Perlstein" In-Reply-To: <8e10486b0712251647uddf80f9sa6cacb42dfa94ff5@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> <20071221201625.GZ16982@elvis.mu.org> <8e10486b0712211249v4c5571ddud21b277f686992b2@mail.gmail.com> <20071221212808.GE16982@elvis.mu.org> <8e10486b0712211555n3efe8729qff14387be128cf10@mail.gmail.com> <20071222002535.GL16982@elvis.mu.org> <8e10486b0712251647uddf80f9sa6cacb42dfa94ff5@mail.gmail.com> Cc: freebsd-performance@freebsd.org Subject: Re: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2007 17:03:59 -0000 On 12/25/07, Alexandre Biancalana wrote: > On 12/21/07, Alfred Perlstein wrote: > > > Have you tried the 'noatime' mount option? That should help. > > Yeah, the zfs set atime=off was already done.... > > > > > Can you provide a histogram of the count of files per directory? > > Excuse-me, but I don't understand.... > Can you explain better ? From owner-freebsd-performance@FreeBSD.ORG Fri Dec 28 21:40:50 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D728A16A468 for ; Fri, 28 Dec 2007 21:40:50 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id C7F0F13C4E8 for ; Fri, 28 Dec 2007 21:40:50 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id BF9371A4D80; Fri, 28 Dec 2007 13:38:51 -0800 (PST) Date: Fri, 28 Dec 2007 13:38:51 -0800 From: Alfred Perlstein To: Alexandre Biancalana Message-ID: <20071228213851.GE76698@elvis.mu.org> References: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> <20071221201625.GZ16982@elvis.mu.org> <8e10486b0712211249v4c5571ddud21b277f686992b2@mail.gmail.com> <20071221212808.GE16982@elvis.mu.org> <8e10486b0712211555n3efe8729qff14387be128cf10@mail.gmail.com> <20071222002535.GL16982@elvis.mu.org> <8e10486b0712251647uddf80f9sa6cacb42dfa94ff5@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8e10486b0712251647uddf80f9sa6cacb42dfa94ff5@mail.gmail.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-performance@freebsd.org Subject: Re: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2007 21:40:50 -0000 * Alexandre Biancalana [071225 16:45] wrote: > On 12/21/07, Alfred Perlstein wrote: > > > Have you tried the 'noatime' mount option? That should help. > > Yeah, the zfs set atime=off was already done.... > > > > > Can you provide a histogram of the count of files per directory? > > Excuse-me, but I don't understand.... Distribution of files per directory, example: 1000 files in 9 dirs 1001 files in 12 dirs find /path_to_root/ -type f | sed 's/\/[^/]*$//' | uniq -c | \ awk '{print $1}' | sort -n | uniq -c That will print out the number of files per dir. 2 98 3 102 1 103 2 105 Or something like that. -- - Alfred Perlstein From owner-freebsd-performance@FreeBSD.ORG Sat Dec 29 15:50:30 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4C07916A418 for ; Sat, 29 Dec 2007 15:50:30 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.176]) by mx1.freebsd.org (Postfix) with ESMTP id F061C13C448 for ; Sat, 29 Dec 2007 15:50:29 +0000 (UTC) (envelope-from biancalana@gmail.com) Received: by py-out-1112.google.com with SMTP id u52so8235699pyb.10 for ; Sat, 29 Dec 2007 07:50:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=OgUldBwYAcfMrUwqmClBXJaWocgM8DRkk0XRNVtEsPM=; b=fbvPbOdhWZZpx2nlLd+7vyvmWUQn+Bn0R/LCGiHpPCk2YBjpTFZgEifzS/Y3nSyp6RD0SN7xoN2sjUuVFtSRdiSN0eUm2gIYsHvHabrAerrGtUjTLjVNrH1DoCPvI0L2a2v+dR25nDX6wS41eYWsZiNZB1wKci6kHMCAFuExAVg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=TGeEGDi1gTOewY4cV+z0LqSxkKZMI2H8BT6bIHtkSwSN61VpDf1ITLzCpiD695tkw6ff0TQ+atFbLTwDLI9+u47w9udm74wrCYl+DxwnBZUFbfFgx2eg+GChhoIoc5MRtMjgDg9aZMN1PcMPfKmLULkuZKo9CIyIuQIRjrngRLs= Received: by 10.64.241.3 with SMTP id o3mr9613180qbh.16.1198943429060; Sat, 29 Dec 2007 07:50:29 -0800 (PST) Received: by 10.64.184.9 with HTTP; Sat, 29 Dec 2007 07:50:29 -0800 (PST) Message-ID: <8e10486b0712290750i41ca2354q7baba9217a1fd9aa@mail.gmail.com> Date: Sat, 29 Dec 2007 13:50:29 -0200 From: "Alexandre Biancalana" To: "Alfred Perlstein" In-Reply-To: <20071228213851.GE76698@elvis.mu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8e10486b0712191109n3d21b02cyf5183ee0cd01d8ce@mail.gmail.com> <20071221201625.GZ16982@elvis.mu.org> <8e10486b0712211249v4c5571ddud21b277f686992b2@mail.gmail.com> <20071221212808.GE16982@elvis.mu.org> <8e10486b0712211555n3efe8729qff14387be128cf10@mail.gmail.com> <20071222002535.GL16982@elvis.mu.org> <8e10486b0712251647uddf80f9sa6cacb42dfa94ff5@mail.gmail.com> <20071228213851.GE76698@elvis.mu.org> Cc: freebsd-performance@freebsd.org Subject: Re: Bad performance when accessing a lot of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Dec 2007 15:50:30 -0000 On Dec 28, 2007 7:38 PM, Alfred Perlstein wrote: > > * Alexandre Biancalana [071225 16:45] wrote: > > On 12/21/07, Alfred Perlstein wrote: > > > > > Have you tried the 'noatime' mount option? That should help. > > > > Yeah, the zfs set atime=off was already done.... > > > > > > > > Can you provide a histogram of the count of files per directory? > > > > Excuse-me, but I don't understand.... > > Distribution of files per directory, example: > > 1000 files in 9 dirs > 1001 files in 12 dirs > > find /path_to_root/ -type f | sed 's/\/[^/]*$//' | uniq -c | \ > awk '{print $1}' | sort -n | uniq -c > > That will print out the number of files per dir. > > 2 98 > 3 102 > 1 103 > 2 105 Ohh yes !! Now I understand.... The problem is that last night the server stop responding and now is panicing with the message: ZFS(panic): zfs: allocating allocated segment(offset=2781261201408 size=131072)