From owner-freebsd-fs Mon Nov 24 02:19:06 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id CAA22171 for fs-outgoing; Mon, 24 Nov 1997 02:19:06 -0800 (PST) (envelope-from owner-freebsd-fs) Received: from godzilla.zeta.org.au (godzilla.zeta.org.au [203.2.228.19]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id CAA22146 for ; Mon, 24 Nov 1997 02:18:57 -0800 (PST) (envelope-from bde@zeta.org.au) Received: (from bde@localhost) by godzilla.zeta.org.au (8.8.7/8.6.9) id VAA09419 for fs@freebsd.org; Mon, 24 Nov 1997 21:14:21 +1100 Date: Mon, 24 Nov 1997 21:14:21 +1100 From: Bruce Evans Message-Id: <199711241014.VAA09419@godzilla.zeta.org.au> To: fs@freebsd.org Subject: ufs slowness Sender: owner-freebsd-fs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk ffs (with a block size of 8K and a frag size of 1K) seems to be about twice as slow as ext2fs (with a block size of 4K and a frag size of 4K) for reading small files. This seems to be due to ffs doing lots more seeks. E.g., for `tar cf /dev/null linux-2.1.63' where there are about 50MB of files in the directory, ffs takes about 40 seconds and ext2fs takes about 20 seconds. The extfs partition is on faster tracks but is fuller. Typical output from systat: ext2fs ffs seeks 372 144 xfers 372 145 blks 2751 1395 msps 0.5 4.1 This was on an IDE so `msps' may actually be right (though it may be broken for dma mode). All file systems were mounted async,noatime. 2.5MB/sec exploits about half the disk bandwidth. Bruce From owner-freebsd-fs Mon Nov 24 10:40:08 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id KAA22843 for fs-outgoing; Mon, 24 Nov 1997 10:40:08 -0800 (PST) (envelope-from owner-freebsd-fs) Received: from usr01.primenet.com (tlambert@usr01.primenet.com [206.165.6.201]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id KAA22833 for ; Mon, 24 Nov 1997 10:40:05 -0800 (PST) (envelope-from tlambert@usr01.primenet.com) Received: (from tlambert@localhost) by usr01.primenet.com (8.8.5/8.8.5) id LAA12388; Mon, 24 Nov 1997 11:39:43 -0700 (MST) From: Terry Lambert Message-Id: <199711241839.LAA12388@usr01.primenet.com> Subject: Re: ufs slowness To: bde@zeta.org.au (Bruce Evans) Date: Mon, 24 Nov 1997 18:39:42 +0000 (GMT) Cc: fs@FreeBSD.ORG In-Reply-To: <199711241014.VAA09419@godzilla.zeta.org.au> from "Bruce Evans" at Nov 24, 97 09:14:21 pm X-Mailer: ELM [version 2.4 PL23] Content-Type: text Sender: owner-freebsd-fs@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk > ffs (with a block size of 8K and a frag size of 1K) seems to be about > twice as slow as ext2fs (with a block size of 4K and a frag size of 4K) > for reading small files. This seems to be due to ffs doing lots more > seeks. E.g., for `tar cf /dev/null linux-2.1.63' where there are about > 50MB of files in the directory, ffs takes about 40 seconds and ext2fs > takes about 20 seconds. The extfs partition is on faster tracks but > is fuller. Typical output from systat: > > ext2fs ffs > seeks 372 144 > xfers 372 145 > blks 2751 1395 > msps 0.5 4.1 > > This was on an IDE so `msps' may actually be right (though it may be broken > for dma mode). All file systems were mounted async,noatime. 2.5MB/sec > exploits about half the disk bandwidth. Well, there's a lot of reasons that this would be true: o As you say, the ext2fs is on faster tracks. This is probably not hugely significant. o Is this a ZBR disk? If not, are you using FreeBSD's default settings, which pessimize geometry optimizations for these disks? o Was the FFS optimizing for space or time when writing? o Did you set a reasonable reserve so the hash-to-disk was efficient on the FFS writes, or did you take FreeBSD's politically motivated defaults (Hey! I'm "wasting" almost 1G of my 9G disk!). o ext2fs is extent based, so it's probably not dealing with indirect blocks. o You are engaging in an atypical usage pattern by doing a "tar" as your test. First, there is zero locality of reference, and second, the way tar traverses means that on a tree that large, you've effectively disable the name cache for FFS (you've damaged it for ext2fs as well, but not to the same degree of fairness, given the relative costs of directory operations and ext2fs's use of extent based files for storing directory data). I'd say the *vast* majority of time spent is in directory operations, rather than actual file data reading (ie: I think the hit from going to indirect blocks in FFS is small). I'm also betting that you created the ext2fs by tarring up the FFS and untarring it onto the ext2fs. Do the same to recreate an FFS on the same fast tracks, and I'll bet you see the performance gap narrow significantly (ie: creation order is important for files vs. directories in preterbing the tar traversal). Now I'm not arguing that an FS shouldn't be using a btree for its directory structure or anything like that 8-). But I think the particular test (a "tar race") isn't quite a fair comparison. There are a lot of ways to "put in the fix" on such a race without intending to do so. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-fs Mon Nov 24 11:29:55 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id LAA26634 for fs-outgoing; Mon, 24 Nov 1997 11:29:55 -0800 (PST) (envelope-from owner-freebsd-fs) Received: from godzilla.zeta.org.au (godzilla.zeta.org.au [203.2.228.19]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id LAA26625 for ; Mon, 24 Nov 1997 11:29:51 -0800 (PST) (envelope-from bde@zeta.org.au) Received: (from bde@localhost) by godzilla.zeta.org.au (8.8.7/8.6.9) id GAA27999; Tue, 25 Nov 1997 06:24:35 +1100 Date: Tue, 25 Nov 1997 06:24:35 +1100 From: Bruce Evans Message-Id: <199711241924.GAA27999@godzilla.zeta.org.au> To: bde@zeta.org.au, tlambert@primenet.com Subject: Re: ufs slowness Cc: fs@FreeBSD.ORG Sender: owner-freebsd-fs@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk >> ext2fs ffs >> seeks 372 144 >> xfers 372 145 >> blks 2751 1395 >> msps 0.5 4.1 >o Is this a ZBR disk? If not, are you using FreeBSD's > default settings, which pessimize geometry optimizations > for these disks? Of course it's ZBR. FreeBSD's default settings haven't done any significant geometry optimizations for several years. >o Was the FFS optimizing for space or time when writing? Time of course. The ufs disk wasn't very full (53% actually). Ther ext2fs disk was 93% full. >o Did you set a reasonable reserve so the hash-to-disk was > efficient on the FFS writes, or did you take FreeBSD's > politically motivated defaults (Hey! I'm "wasting" almost > 1G of my 9G disk!). Irrelevant, since it wasn't very full. >o ext2fs is extent based, so it's probably not dealing with > indirect blocks. Not important for small files. The speed is about the same for large (> memory size) sequential files. >o You are engaging in an atypical usage pattern by doing > a "tar" as your test. First, there is zero locality of This is not atypical for me. > reference, and second, the way tar traverses means that > on a tree that large, you've effectively disable the name > cache for FFS (you've damaged it for ext2fs as well, but > not to the same degree of fairness, given the relative > costs of directory operations and ext2fs's use of extent > based files for storing directory data). I used a large enough directory to damage the (data) cache on purpose. There are 3739 files. This is apparently enough to also damage directory caches >I'd say the *vast* majority of time spent is in directory operations, >rather than actual file data reading (ie: I think the hit from going >to indirect blocks in FFS is small). I agree. Perhaps it's just ext2fs hanging on to directory blocks better. >I'm also betting that you created the ext2fs by tarring up the >FFS and untarring it onto the ext2fs. Do the same to recreate an I actually used `cp -pR' from ext2fs to ufs. Bruce From owner-freebsd-fs Mon Nov 24 11:46:57 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id LAA28207 for fs-outgoing; Mon, 24 Nov 1997 11:46:57 -0800 (PST) (envelope-from owner-freebsd-fs) Received: from usr01.primenet.com (tlambert@usr01.primenet.com [206.165.6.201]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id LAA28202 for ; Mon, 24 Nov 1997 11:46:54 -0800 (PST) (envelope-from tlambert@usr01.primenet.com) Received: (from tlambert@localhost) by usr01.primenet.com (8.8.5/8.8.5) id MAA18929; Mon, 24 Nov 1997 12:46:11 -0700 (MST) From: Terry Lambert Message-Id: <199711241946.MAA18929@usr01.primenet.com> Subject: Re: ufs slowness To: bde@zeta.org.au (Bruce Evans) Date: Mon, 24 Nov 1997 19:46:10 +0000 (GMT) Cc: bde@zeta.org.au, tlambert@primenet.com, fs@FreeBSD.ORG In-Reply-To: <199711241924.GAA27999@godzilla.zeta.org.au> from "Bruce Evans" at Nov 25, 97 06:24:35 am X-Mailer: ELM [version 2.4 PL23] Content-Type: text Sender: owner-freebsd-fs@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk > >o Is this a ZBR disk? If not, are you using FreeBSD's > > default settings, which pessimize geometry optimizations > > for these disks? > > Of course it's ZBR. FreeBSD's default settings haven't done any > significant geometry optimizations for several years. Sorry; the IDE-ness didn't make that obvious. [ ... ] > >I'd say the *vast* majority of time spent is in directory operations, > >rather than actual file data reading (ie: I think the hit from going > >to indirect blocks in FFS is small). > > I agree. Perhaps it's just ext2fs hanging on to directory blocks better. Maybe. I don't see how, though. > >I'm also betting that you created the ext2fs by tarring up the > >FFS and untarring it onto the ext2fs. Do the same to recreate an > > I actually used `cp -pR' from ext2fs to ufs. I think you'll find that this is the culprit. You should do an "ls -fF" in a couple of equivalent directories and see the ordering of directories vs. files for thiose directories which contain both. You should find that they aren't identical. I think you are seeing a collision between "depth first" and "breadth first" when you tar up the FFS directory. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-fs Mon Nov 24 23:36:24 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id XAA21137 for fs-outgoing; Mon, 24 Nov 1997 23:36:24 -0800 (PST) (envelope-from owner-freebsd-fs) Received: from godzilla.zeta.org.au (godzilla.zeta.org.au [203.2.228.19]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id XAA21131 for ; Mon, 24 Nov 1997 23:36:19 -0800 (PST) (envelope-from bde@zeta.org.au) Received: (from bde@localhost) by godzilla.zeta.org.au (8.8.7/8.6.9) id SAA25843; Tue, 25 Nov 1997 18:33:34 +1100 Date: Tue, 25 Nov 1997 18:33:34 +1100 From: Bruce Evans Message-Id: <199711250733.SAA25843@godzilla.zeta.org.au> To: bde@zeta.org.au, tlambert@primenet.com Subject: Re: ufs slowness Cc: fs@FreeBSD.ORG Sender: owner-freebsd-fs@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk >> >I'd say the *vast* majority of time spent is in directory operations, >> >rather than actual file data reading (ie: I think the hit from going >> >to indirect blocks in FFS is small). >> >> I agree. Perhaps it's just ext2fs hanging on to directory blocks better. > >Maybe. I don't see how, though. It has its own little cache for bitmap and inode blocks. I would have thought that this was usually a pessimization, but perhaps it prevents thrashing in this particular test. >> >I'm also betting that you created the ext2fs by tarring up the >> >FFS and untarring it onto the ext2fs. Do the same to recreate an >> >> I actually used `cp -pR' from ext2fs to ufs. > >I think you'll find that this is the culprit. Don't think so. The directories are fairly small and should stay in a cache. >You should do an "ls -fF" in a couple of equivalent directories and >see the ordering of directories vs. files for thiose directories which >contain both. You should find that they aren't identical. cp -pR doesn't quite preserve the order. I would have thought it uses the order reaterned by readdir(). After copying from ext2fs to ffs using two tars in a pipe, ffs is exactly as slow as before, although I've disturbed the ffs partition a little by building a world in it (it grew from 53% full to 66% full). Bruce From owner-freebsd-fs Tue Nov 25 09:39:25 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id JAA01220 for fs-outgoing; Tue, 25 Nov 1997 09:39:25 -0800 (PST) (envelope-from owner-freebsd-fs) Received: from smtp02.primenet.com (smtp02.primenet.com [206.165.6.132]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id JAA01214 for ; Tue, 25 Nov 1997 09:39:19 -0800 (PST) (envelope-from tlambert@usr08.primenet.com) Received: (from daemon@localhost) by smtp02.primenet.com (8.8.8/8.8.8) id KAA18621; Tue, 25 Nov 1997 10:42:54 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp02.primenet.com, id smtpd018612; Tue Nov 25 10:42:45 1997 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id KAA00604; Tue, 25 Nov 1997 10:38:36 -0700 (MST) From: Terry Lambert Message-Id: <199711251738.KAA00604@usr08.primenet.com> Subject: Re: ufs slowness To: bde@zeta.org.au (Bruce Evans) Date: Tue, 25 Nov 1997 17:38:36 +0000 (GMT) Cc: bde@zeta.org.au, tlambert@primenet.com, fs@FreeBSD.ORG In-Reply-To: <199711250733.SAA25843@godzilla.zeta.org.au> from "Bruce Evans" at Nov 25, 97 06:33:34 pm X-Mailer: ELM [version 2.4 PL23] Content-Type: text Sender: owner-freebsd-fs@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk > >You should do an "ls -fF" in a couple of equivalent directories and > >see the ordering of directories vs. files for thiose directories which > >contain both. You should find that they aren't identical. > > cp -pR doesn't quite preserve the order. I would have thought it uses > the order reaterned by readdir(). > > After copying from ext2fs to ffs using two tars in a pipe, ffs is exactly > as slow as before, although I've disturbed the ffs partition a little > by building a world in it (it grew from 53% full to 66% full). How about copying back to ext2fs? Then at least the same algorithm will have populated both. I think "optimal" might be "in such a way that a file may not displace a directory from cache". This would imply: 1) Breadth first 2) Create all files before subdirectories for any directory Of course, I could be wrong. If it's depth first, with no way to adjust it, you'd want the directories first. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-fs Tue Nov 25 10:31:22 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id KAA05582 for fs-outgoing; Tue, 25 Nov 1997 10:31:22 -0800 (PST) (envelope-from owner-freebsd-fs) Received: from godzilla.zeta.org.au (godzilla.zeta.org.au [203.2.228.19]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id KAA05565 for ; Tue, 25 Nov 1997 10:31:17 -0800 (PST) (envelope-from bde@zeta.org.au) Received: (from bde@localhost) by godzilla.zeta.org.au (8.8.7/8.6.9) id FAA15404; Wed, 26 Nov 1997 05:26:57 +1100 Date: Wed, 26 Nov 1997 05:26:57 +1100 From: Bruce Evans Message-Id: <199711251826.FAA15404@godzilla.zeta.org.au> To: bde@zeta.org.au, tlambert@primenet.com Subject: Re: ufs slowness Cc: fs@FreeBSD.ORG Sender: owner-freebsd-fs@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk >> After copying from ext2fs to ffs using two tars in a pipe, ffs is exactly >> as slow as before, although I've disturbed the ffs partition a little >> by building a world in it (it grew from 53% full to 66% full). > >How about copying back to ext2fs? Then at least the same algorithm >will have populated both. See later mail. It's slower, but I think that is because the FreeBSD ext2fs is worse that the Linux one :-). >I think "optimal" might be "in such a way that a file may not displace >a directory from cache". > >This would imply: > >1) Breadth first >2) Create all files before subdirectories for any directory > >Of course, I could be wrong. If it's depth first, with no way to >adjust it, you'd want the directories first. It's othing to do with that - see later mail. ffs wants to seek a lot to switch cylinder groups (for almost every new directory?), but I think the main problem is that it wants to seek lot to handle fragments. I'm now making space to repartition. I'll try a 4K/4K ffs to match the ext2fs block size exactly. Another thing: it didn't help to move all the files into one big directory (with ~2500 files and 38MB data). This was 20% slower, presumably due to the directory being too big to cache properly. Bruce From owner-freebsd-fs Fri Nov 28 10:57:43 1997 Return-Path: Received: (from root@localhost) by hub.freebsd.org (8.8.7/8.8.7) id KAA00997 for fs-outgoing; Fri, 28 Nov 1997 10:57:43 -0800 (PST) (envelope-from owner-freebsd-fs) Received: from godzilla.zeta.org.au (godzilla.zeta.org.au [203.2.228.19]) by hub.freebsd.org (8.8.7/8.8.7) with ESMTP id KAA00991 for ; Fri, 28 Nov 1997 10:57:35 -0800 (PST) (envelope-from bde@zeta.org.au) Received: (from bde@localhost) by godzilla.zeta.org.au (8.8.7/8.6.9) id FAA10339; Sat, 29 Nov 1997 05:56:17 +1100 Date: Sat, 29 Nov 1997 05:56:17 +1100 From: Bruce Evans Message-Id: <199711281856.FAA10339@godzilla.zeta.org.au> To: bde@zeta.org.au, tlambert@primenet.com Subject: Re: ufs slowness Cc: fs@FreeBSD.ORG Sender: owner-freebsd-fs@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk I wrote: >ffs wants to seek a lot to switch cylinder groups (for almost every new >directory?), but I think the main problem is that it wants to seek lot >to handle fragments. I'm now making space to repartition. I'll try a >4K/4K ffs to match the ext2fs block size exactly. This was the problem. The 4K/4K ffs was about the same speed as a 4K/4K ext2fs, and a 4K/512 ffs was about as slow as an 8K/1K ffs (20+ seconds vs 40+ seconds). Some analysis of the i/o: ffs-4K-4K ffs-4K-512 contiguous blocks 7093 3481 forward seeks 450 2857 backwards seeks 258 1647 forward seeks > 64K 407 1037 backward seeks > 64K 211 797 average block size 7013 6055 The drive has an average access time 10.5 ms. Bruce