From owner-freebsd-fs@FreeBSD.ORG Mon Jan 3 13:25:46 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 577D916A4CE for ; Mon, 3 Jan 2005 13:25:46 +0000 (GMT) Received: from naboo.binarysolutions.dk (port554.ds1-kd.adsl.cybercity.dk [212.242.185.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5F99043D3F for ; Mon, 3 Jan 2005 13:25:45 +0000 (GMT) (envelope-from kvs@binarysolutions.dk) Received: by naboo.binarysolutions.dk (Postfix, from userid 1000) id A972443B54; Mon, 3 Jan 2005 14:25:40 +0100 (CET) Date: Mon, 3 Jan 2005 14:25:40 +0100 From: Kenneth Vestergaard Schmidt To: freebsd-fs@freebsd.org Message-ID: <20050103132540.GB21037@binarysolutions.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-GPG-Fingerprint: A11F BB5D BD79 7228 A198 CF53 D508 53A9 2213 E772 X-GPG-Key: http://www.binarysolutions.dk/~kvs/key.asc User-Agent: Mutt/1.5.6+20040722i Subject: Extending di_nlink and its ilk X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jan 2005 13:25:46 -0000 Hello. I've run into a wee problem trying to create a nice backup-machine. We made it using rsync, hardlinks, and a modified link-by-hash patch for rsync. link-by-hash creates an md4 checksum of the file's contents. It then stores the file in /dana/hashes/abcdef/1234567890 and hardlinks it to the correct place. This way, identical files only get stored once. At this point, we ran into the problem with di_nlink and related fields only being 16-bit, since we were creating more than 32765 sub-directories. I fixed this by only creating 256 directories, each containing a lot of files. However, we soon ran into yet another problem, that of more than 32767 links to one file - when we link by contents, this limit comes up real quick. My initial idea was to patch the file-system to use one of the spare values at the end of various inode-structs to provide a 32-bit or 64-bit value to the link count. Of course, some backward-compatible scheme must be employed were the original di_nlink is read first, but I wanted to hear if this is a totally hare-brained scheme before I start doing it, or if it would actually be useful to others? The only other choice I have is a couple of extremely ugly hacks to rsync, which I'd rather not do. -- Best Regards Kenneth Vestergaard Schmidt From owner-freebsd-fs@FreeBSD.ORG Mon Jan 3 21:43:46 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2AEFD16A4CE for ; Mon, 3 Jan 2005 21:43:46 +0000 (GMT) Received: from lancia.kaluga.ru (lancia.kaluga.ru [62.148.128.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 37B5943D1D for ; Mon, 3 Jan 2005 21:43:45 +0000 (GMT) (envelope-from freebsd-fs@merdin.com) Received: from localhost (242.net-144.kaluga.ru [62.148.144.242] (may be forged)) by lancia.kaluga.ru (8.13.1/8.13.1) with ESMTP id j03Lhgaj080687 for ; Tue, 4 Jan 2005 00:43:43 +0300 (MSK) Received: from localhost ([127.0.0.1]) by [127.0.0.1] with ESMTP (SpamPal v1.583) sender ; 04 Jan 2005 00:43:42 +0400 Date: Tue, 4 Jan 2005 00:43:42 +0300 From: Pavel Merdine X-Mailer: The Bat! (v3.0.1.33) UNREG / CD5BF9353B3B7091 X-Priority: 3 (Normal) Message-ID: <505295882.20050104004342@merdin.com> To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Antivirus: Dr.Web (R) for Mail Servers on lancia.kaluga.ru host X-Antivirus-Code: 100000 Subject: dup alloc, please advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jan 2005 21:43:46 -0000 Hello, /kernel: mode = 040700, inum = 12, fs = /somemount /kernel: panic: ffs_valloc: dup alloc /kernel: /kernel: syncing disks... 49 22 8 7 7 7 7 7 7 7 7 7 7 7 It happened just in 10 minutes after fsck. There are lots of writes to that disk by many nfs clients at the same times. What should I do in such situations? FreeBSD 4.10-STABLE #0: Mon Jul 26 05:46:38 EDT 2004 By the way, sometimes I also get /kernel: ufs_rename: fvp == tvp (can't happen) in logs. -- / Pavel Merdine From owner-freebsd-fs@FreeBSD.ORG Mon Jan 3 21:53:34 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0F0EE16A4CE for ; Mon, 3 Jan 2005 21:53:34 +0000 (GMT) Received: from web41212.mail.yahoo.com (web41212.mail.yahoo.com [66.218.93.45]) by mx1.FreeBSD.org (Postfix) with SMTP id D166E43D49 for ; Mon, 3 Jan 2005 21:53:33 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 32557 invoked by uid 60001); 3 Jan 2005 21:53:33 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=BtB2NZv/ECw/pg6ZGNQJxY/OpW7W3kEIrm4BPDY5WWL83DBKKB6hmkL6x6SpvtRPAXexFNuO/10oIOEDcoQeFhoLctM5ahDUNmthMmB/KecwANriZQb+fCcWZZlm4mu5i81M3VdQoqRw7wRoqz8h0J6Xq9bcJi0pH0rE+s7Mmlg= ; Message-ID: <20050103215333.32555.qmail@web41212.mail.yahoo.com> Received: from [213.54.133.144] by web41212.mail.yahoo.com via HTTP; Mon, 03 Jan 2005 13:53:33 PST Date: Mon, 3 Jan 2005 13:53:33 -0800 (PST) From: Arne "Wörner" To: Pavel Merdine , freebsd-fs@freebsd.org In-Reply-To: <505295882.20050104004342@merdin.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: dup alloc, please advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jan 2005 21:53:34 -0000 > /kernel: mode = 040700, inum = 12, fs = /somemount > /kernel: panic: ffs_valloc: dup alloc > /kernel: > /kernel: syncing disks... 49 22 8 7 7 7 7 7 7 7 7 7 7 7 > > It happened just in 10 minutes after fsck. > Was fsck completed? If yes, what were the log messages/results? > There are lots of writes to that disk by many nfs clients at the > same times. > Not exactly the same time, I hope... :-)) > What should I do in such situations? > Have you tried: 1. Booting to single user mode and doing a "fsck /somemount" manually (I would like to see the results). 2. Halting the system gracefully (with halt or so) 3. Mounting just clean file systems (no -f or so; no soft updates (just for the beginning)) > FreeBSD 4.10-STABLE #0: Mon Jul 26 05:46:38 EDT 2004 > > By the way, sometimes I also get > /kernel: ufs_rename: fvp == tvp (can't happen) > in logs. > Might be an old inconsistency, too (if I remember right, you were the one, who liked to mount unclean fs :-)) ). -Arne __________________________________ Do you Yahoo!? Send a seasonal email greeting and help others. Do good. http://celebrity.mail.yahoo.com From owner-freebsd-fs@FreeBSD.ORG Mon Jan 3 22:17:30 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CA53716A4CE for ; Mon, 3 Jan 2005 22:17:30 +0000 (GMT) Received: from lancia.kaluga.ru (lancia.kaluga.ru [62.148.128.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id F22E143D2D for ; Mon, 3 Jan 2005 22:17:29 +0000 (GMT) (envelope-from freebsd-fs@merdin.com) Received: from localhost (242.net-144.kaluga.ru [62.148.144.242] (may be forged)) by lancia.kaluga.ru (8.13.1/8.13.1) with ESMTP id j03MHSZm088477 for ; Tue, 4 Jan 2005 01:17:28 +0300 (MSK) Received: from localhost ([127.0.0.1]) by [127.0.0.1] with ESMTP (SpamPal v1.583) sender ; 04 Jan 2005 01:17:28 +0400 Date: Tue, 4 Jan 2005 01:17:28 +0300 From: Pavel Merdine X-Mailer: The Bat! (v3.0.1.33) UNREG / CD5BF9353B3B7091 X-Priority: 3 (Normal) Message-ID: <1964212095.20050104011728@merdin.com> To: =?Windows-1251?Q?Arne_W=F6rner?= In-Reply-To: <20050103215333.32555.qmail@web41212.mail.yahoo.com> References: <505295882.20050104004342@merdin.com> <20050103215333.32555.qmail@web41212.mail.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=Windows-1251 Content-Transfer-Encoding: 8bit X-Antivirus: Dr.Web (R) for Mail Servers on lancia.kaluga.ru host X-Antivirus-Code: 100000 Subject: Re[2]: dup alloc, please advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jan 2005 22:17:30 -0000 Hello , Tuesday, January 4, 2005, 12:53:33 AM, you wrote: >> /kernel: mode = 040700, inum = 12, fs = /somemount >> /kernel: panic: ffs_valloc: dup alloc >> /kernel: >> /kernel: syncing disks... 49 22 8 7 7 7 7 7 7 7 7 7 7 7 >> >> It happened just in 10 minutes after fsck. >> > Was fsck completed? If yes, what were the log messages/results? I think, yes. I cannot provide the result, because I dont write logs. I'll make sure they are recorded next time. I use a script like: /sbin/mount $2 || (/sbin/fsck -y $2 && /sbin/mount $2) to mount data disks. >> What should I do in such situations? >> > Have you tried: > 1. Booting to single user mode and doing a "fsck /somemount" > manually (I would like to see the results). > 2. Halting the system gracefully (with halt or so) > 3. Mounting just clean file systems (no -f or so; no soft updates > (just for the beginning)) That is a production machine, so I cant experiment with it. It mounts all disks automatically thru the script. The machine just halted when I tried to do umount -f on disk that was using 100% according to systat. After that the machine gone thru usual boot and checked most of data disks. In 10 minutes after that it panicked due to error on another disk (not the one I tried to umount). "Failty" disk was unmounted. >> FreeBSD 4.10-STABLE #0: Mon Jul 26 05:46:38 EDT 2004 >> >> By the way, sometimes I also get >> /kernel: ufs_rename: fvp == tvp (can't happen) >> in logs. >> > Might be an old inconsistency, too (if I remember right, you were > the one, who liked to mount unclean fs :-)) ). Maybe you write. But now I dont mount with -f. Dont say I cant to umount -f :) It's not about me liking -f. There should be some documentation saying about proper use of softupdates and ide write cache. It's not always clear even from the sources. > -Arne > __________________________________ > Do you Yahoo!? > Send a seasonal email greeting and help others. Do good. > http://celebrity.mail.yahoo.com -- / Pavel Merdine From owner-freebsd-fs@FreeBSD.ORG Mon Jan 3 23:37:29 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B699916A4CE for ; Mon, 3 Jan 2005 23:37:29 +0000 (GMT) Received: from web41215.mail.yahoo.com (web41215.mail.yahoo.com [66.218.93.48]) by mx1.FreeBSD.org (Postfix) with SMTP id 9602943D49 for ; Mon, 3 Jan 2005 23:37:29 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 8006 invoked by uid 60001); 3 Jan 2005 23:37:29 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=z6Xw+f9gKiVLJnJykTRRgSoKe7EO+HQlLsQPgAf+We9aiDaDHcDLdxcX3JseegsUOPygy9O7SR81nXMUKxplUo2Gp4hAEcqTgh1NoTynKJwBkCfd3bCXjoFEvdL9VpD38KlRyHg+F8GyJxw59Wfook7ZrpP2Z86WkZa1mxgUyqo= ; Message-ID: <20050103233729.8004.qmail@web41215.mail.yahoo.com> Received: from [213.54.133.144] by web41215.mail.yahoo.com via HTTP; Mon, 03 Jan 2005 15:37:29 PST Date: Mon, 3 Jan 2005 15:37:29 -0800 (PST) From: Arne "Wörner" To: Pavel Merdine , freebsd-fs@freebsd.org In-Reply-To: <1964212095.20050104011728@merdin.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: Re[2]: dup alloc, please advice X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jan 2005 23:37:29 -0000 Hiho! --- Pavel Merdine wrote: > Hello , > Tuesday, January 4, 2005, 12:53:33 AM, you wrote: > >> /kernel: mode = 040700, inum = 12, fs = /somemount > >> /kernel: panic: ffs_valloc: dup alloc > >> /kernel: > >> /kernel: syncing disks... 49 22 8 7 7 7 7 7 7 7 7 7 7 7 > >> It happened just in 10 minutes after fsck. > > Was fsck completed? If yes, what were the log > > messages/results? > I think, yes. > I cannot provide the result, because I dont write logs. I'll > make sure > they are recorded next time. > I use a script like: > /sbin/mount $2 || (/sbin/fsck -y $2 && /sbin/mount $2) > to mount data disks. > Ok. I do not see any error... Maybe you should try /sbin/fsck -f -y $2 && /sbin/mount $2 during your next boot (that should solve many (maybe all?) inconsistencies...). > The machine just halted when I tried to do umount -f on disk > that was using 100% according to systat. > umount -f is funny (shoots away all open files, I think)... :-)) I do not know, why that does not work gracefully... Maybe somebody can tell you, what one can do, to find out, why a disc is 100% busy... > > Maybe you write. But now I dont mount with -f. Dont say I cant > to umount -f :) > Maybe there are/were still legacies? > It's not about me liking -f. There should be some documentation > saying about proper use of softupdates and ide write cache. It's > not always clear even from the sources. > I have the problem that atacontrol says, that write cache is enabled, although hw.ata.wc says, that write cache is disabled... I think, it is in any case safer (soft updates or not), to disable write cache, because you never know, which data is discarded in case of a sudden reset or a power failure... Funnily after a dd if=/dev/ad1 bs=128k count=2000 of=/dev/null a dd of=/opt/a bs=64 count=1000 if=/dev/zero ; fsync /opt/a is about 5 times faster (I have the theory, that the disc write cache can be substituted by FBSD's cache). Bye Arne __________________________________ Do you Yahoo!? Yahoo! Mail - Easier than ever with enhanced search. Learn more. http://info.mail.yahoo.com/mail_250 From owner-freebsd-fs@FreeBSD.ORG Tue Jan 4 00:08:50 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F052F16A4CE for ; Tue, 4 Jan 2005 00:08:49 +0000 (GMT) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6A8B743D1D for ; Tue, 4 Jan 2005 00:08:49 +0000 (GMT) (envelope-from scottl@freebsd.org) Received: from [192.168.254.11] (junior-wifi.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.12.11/8.12.10) with ESMTP id j040BTLd010842; Mon, 3 Jan 2005 17:11:31 -0700 (MST) (envelope-from scottl@freebsd.org) Message-ID: <41D9DE30.1070209@freebsd.org> Date: Mon, 03 Jan 2005 17:07:12 -0700 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.2) Gecko/20040929 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Kenneth Vestergaard Schmidt References: <20050103132540.GB21037@binarysolutions.dk> In-Reply-To: <20050103132540.GB21037@binarysolutions.dk> X-Enigmail-Version: 0.86.1.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, hits=0.0 required=3.8 tests=none autolearn=no version=2.63 X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on pooker.samsco.org cc: freebsd-fs@freebsd.org Subject: Re: Extending di_nlink and its ilk X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jan 2005 00:08:50 -0000 Kenneth Vestergaard Schmidt wrote: > Hello. > > I've run into a wee problem trying to create a nice backup-machine. We made > it using rsync, hardlinks, and a modified link-by-hash patch for rsync. > > link-by-hash creates an md4 checksum of the file's contents. It then stores > the file in /dana/hashes/abcdef/1234567890 and hardlinks it to the correct > place. This way, identical files only get stored once. > > At this point, we ran into the problem with di_nlink and related fields > only being 16-bit, since we were creating more than 32765 sub-directories. > > I fixed this by only creating 256 directories, each containing a lot of > files. However, we soon ran into yet another problem, that of more than > 32767 links to one file - when we link by contents, this limit comes up > real quick. > > My initial idea was to patch the file-system to use one of the spare > values at the end of various inode-structs to provide a 32-bit or 64-bit > value to the link count. Of course, some backward-compatible scheme must > be employed were the original di_nlink is read first, but I wanted to > hear if this is a totally hare-brained scheme before I start doing it, > or if it would actually be useful to others? > > The only other choice I have is a couple of extremely ugly hacks to rsync, > which I'd rather not do. > > The downside to having really large directories is that the lookup and readdir operations are linear in UFS. The DIRHASH code helps this quite a bit, but it's still not very optimal. The performance scalability problem is likely why there has been little pressure to increase the size of di_nlink. Assuming that performance is not an important consideration, the next problem is how to specify an alternate link counter (let's call it di_nlink2 here) in a way that is as backwards compatible as possible. How will tools like fsck and dump, let alone the kernel FS code, know to use the new field as opposed to the old one? Do we assign special meaning to a specific value of di_nlink? i.e. if di_nlink is set to 0xffff, then everything should assume to use di_nlink2? What pretects us from a version of fsck that doesn't understand the magic value from completely trashing the FS? There are no magic values for di_nlink right now, so anything that you choose has the possibility of colliding with a valid value. It's quite common to share a disk between different versions of BSD, as well as different BSDs, so you really cannot have too many seatbelts here. We could bump the magic in the superblock and use that to instruct the tools on how to treat di_nlink, but that's a pretty dramatic change and will make it much harder to share disks. It would basically amount to creating 'UFS3', and at that point it would be more prudent to review better directory layout policies at the same time. Scott From owner-freebsd-fs@FreeBSD.ORG Tue Jan 4 07:31:20 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 793DB16A4CE for ; Tue, 4 Jan 2005 07:31:20 +0000 (GMT) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.FreeBSD.org (Postfix) with ESMTP id 189AB43D46 for ; Tue, 4 Jan 2005 07:31:19 +0000 (GMT) (envelope-from avg@icyb.net.ua) Received: from [212.40.38.87] (oddity.topspin.kiev.ua [212.40.38.87]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id JAA20927 for ; Tue, 04 Jan 2005 09:31:17 +0200 (EET) (envelope-from avg@icyb.net.ua) Message-ID: <41DA4645.20502@icyb.net.ua> Date: Tue, 04 Jan 2005 09:31:17 +0200 From: Andriy Gapon User-Agent: Mozilla Thunderbird 1.0 (X11/20041230) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=KOI8-U Content-Transfer-Encoding: 7bit Subject: FYI: userland udf read-write support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jan 2005 07:31:20 -0000 http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/75781 -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Jan 5 23:33:16 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B609A16A4D0; Wed, 5 Jan 2005 23:33:16 +0000 (GMT) Received: from mail-svr1.cs.utah.edu (mail-svr1.cs.utah.edu [155.98.64.200]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8BEB143D2D; Wed, 5 Jan 2005 23:33:16 +0000 (GMT) (envelope-from saggarwa@cs.utah.edu) Received: from localhost (localhost [127.0.0.1]) by mail-svr1.cs.utah.edu (Postfix) with ESMTP id 39BC8346ED; Wed, 5 Jan 2005 16:33:16 -0700 (MST) Received: from mail-svr1.cs.utah.edu ([127.0.0.1]) by localhost (mail-svr1.cs.utah.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 01793-10; Wed, 5 Jan 2005 16:33:16 -0700 (MST) Received: from faith.cs.utah.edu (faith.cs.utah.edu [155.98.65.40]) by mail-svr1.cs.utah.edu (Postfix) with ESMTP id D8034346D9; Wed, 5 Jan 2005 16:33:15 -0700 (MST) Received: by faith.cs.utah.edu (Postfix, from userid 4973) id 7F3452EC21; Wed, 5 Jan 2005 16:33:13 -0700 (MST) Received: from localhost (localhost [127.0.0.1]) by faith.cs.utah.edu (Postfix) with ESMTP id D069D34406; Wed, 5 Jan 2005 23:33:13 +0000 (UTC) Date: Wed, 5 Jan 2005 16:33:13 -0700 (MST) From: Siddharth Aggarwal To: freebsd-fs@freebsd.org, "" Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Virus-Scanned: amavisd-new at cs.utah.edu Subject: write retry on filesystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 23:33:16 -0000 Hi, I have a pseudo disk driver that does a copy on write to a log device. I want to disable further retries to write if the disk space on the log device is full. I have inserted the following code into the strategy routine For this I have set the error code and also set the resid to zero, thinking that the filesystem above will not retry the operation because it assumes that there are no more bytes to be written (since I set resid to 0). However, control keeps coming back to this code, suggesting that the FS (buffer cache) is constantly retrying the operation. Is there any other way to do this (maybe I should try a different error code. I tried ENOSPC too). The reason I want to disable retry is to allow the user to manually select and delete logs to reclaim space and then do the file operation again if he chooses to. The problem arising is because the buffer cache is asynchronously flushed to disk (which could be much after a vfs file write operation). So the application has no way of knowing that a write to disk failed because COW failed due to log device space shortage. So the syncer periodically keeps trying to flush the buffer cache to disk because every time the strategy routine (in my driver which is between the buffer cache and the disk) cannot complete the operation. Any suggestions on how I can accomplish this? Any suggested hack in vfs_bio? Thanks, Sid. strategy () { ..... if (failed < 0) /* no more free space on log device */ { printf ("No more space on disk! Copy on write failed\n"); bp->b_error = EACCES; bp->b_flags |= B_ERROR; bp->b_resid = 0; devstat_end_transaction_buf(&ss->device_stats, bp); biodone(bp); return; } .... } From owner-freebsd-fs@FreeBSD.ORG Thu Jan 6 00:06:38 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B640E16A4CE for ; Thu, 6 Jan 2005 00:06:38 +0000 (GMT) Received: from ccshst09.cs.uoguelph.ca (ccshst09.cs.uoguelph.ca [131.104.96.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3254343D4C for ; Thu, 6 Jan 2005 00:06:38 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from snowhite.cis.uoguelph.ca (snowhite.cis.uoguelph.ca [131.104.48.1])j0606YDf006351 for ; Wed, 5 Jan 2005 19:06:34 -0500 Received: (from rick@localhost) by snowhite.cis.uoguelph.ca (8.9.3/8.9.3) id TAA13854 for fs@freebsd.org; Wed, 5 Jan 2005 19:06:55 -0500 (EST) Date: Wed, 5 Jan 2005 19:06:55 -0500 (EST) From: rick@snowhite.cis.uoguelph.ca Message-Id: <200501060006.TAA13854@snowhite.cis.uoguelph.ca> To: fs@freebsd.org X-Scanned-By: MIMEDefang 2.44 Subject: yet another NFSv4 beta server X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jan 2005 00:06:38 -0000 In case anyone is interested, I've put another Beta release of my NFSv2,3,4 server up anonymous ftp from ftp.cis.uoguelph.ca in pub/nfsv4. I haven't switched to FreeBSD5.3 yet, but plan to soon. I'll announce when a FreeBSD5.3 port is available. Have fun, rick From owner-freebsd-fs@FreeBSD.ORG Fri Jan 7 09:30:30 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3459D16A4CE for ; Fri, 7 Jan 2005 09:30:30 +0000 (GMT) Received: from web30306.mail.mud.yahoo.com (web30306.mail.mud.yahoo.com [68.142.200.99]) by mx1.FreeBSD.org (Postfix) with SMTP id D27ED43D1D for ; Fri, 7 Jan 2005 09:30:29 +0000 (GMT) (envelope-from barath_yes@yahoo.com.sg) Received: (qmail 43308 invoked by uid 60001); 7 Jan 2005 09:30:29 -0000 Message-ID: <20050107093029.43306.qmail@web30306.mail.mud.yahoo.com> Received: from [203.118.46.137] by web30306.mail.mud.yahoo.com via HTTP; Fri, 07 Jan 2005 17:30:29 CST Date: Fri, 7 Jan 2005 17:30:29 +0800 (CST) From: Barath S To: FreeBsd-MailGrp MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Subject: using mfs of size > 64Mb and system stability X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Jan 2005 09:30:30 -0000 My intention is create a file system in memory of size > 128Mb and use that for my application. But, I couldn't do that as my system hangs whenever I create a mfs of size > 64Mb. I did the following... a) Configured memory file system of size 128Mb using 'mdmfs' command with 'malloc' option. When I tried to copy files to this mfs, m/c hanged after copying some files. b) Then I rebooted by m/c and configured memory file system of size 64Mb and this time the copying operation was successful c) I observed that the system hanged whenever I configure a mfs of size > 64Mb and used that mfs System details: i386 system with 256Mb RAM running FreeBSD 5.2.1 a) Can someone provide me hints on what could be the problem ? Thanks __________________________________________________ Do You Yahoo!? Download the latest ringtones, games, and more! http://sg.mobile.yahoo.com From owner-freebsd-fs@FreeBSD.ORG Fri Jan 7 09:43:48 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F0EA516A4CE for ; Fri, 7 Jan 2005 09:43:48 +0000 (GMT) Received: from web41201.mail.yahoo.com (web41201.mail.yahoo.com [66.218.93.34]) by mx1.FreeBSD.org (Postfix) with SMTP id BCA0843D39 for ; Fri, 7 Jan 2005 09:43:48 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 42612 invoked by uid 60001); 7 Jan 2005 09:43:48 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=QP4PxZvmtkvgsVs2lnSt/F/c74rXbTg01mKzIIw4wcNyQv1W1w+YVDF4SZcS68Bc8QGYgtRC3mMZzjd/La/amEROVJ+Z16OXo12qLZ+xdofKjaHIU9DhfepGp9AYV/GsRS5AeSBAarVvYMTJxqac/jwK8EsA8T+D/tShupJX4Lc= ; Message-ID: <20050107094348.42610.qmail@web41201.mail.yahoo.com> Received: from [213.54.149.179] by web41201.mail.yahoo.com via HTTP; Fri, 07 Jan 2005 01:43:48 PST Date: Fri, 7 Jan 2005 01:43:48 -0800 (PST) From: Arne "Wörner" To: Barath S , FreeBsd-MailGrp In-Reply-To: <20050107093029.43306.qmail@web30306.mail.mud.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: using mfs of size > 64Mb and system stability X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Jan 2005 09:43:49 -0000 --- Barath S wrote: > My intention is create a file system in memory of size > > 128Mb and use that for my application. But, I > couldn't do that as my system hangs whenever I create > a mfs of size > 64Mb. I did the following... > > a) Configured memory file system of size 128Mb using > 'mdmfs' command with 'malloc' option. When I tried to > copy files to this mfs, m/c hanged after copying some > files. > b) Then I rebooted by m/c and configured memory file > system of size 64Mb and this time the copying > operation was successful > c) I observed that the system hanged whenever I > configure a mfs of size > 64Mb and used that mfs > > a) Can someone provide me hints on what could be the > problem ? > I use R5.3. According to the man page mdconfig(8) a md of type malloc should use "-o reserve", if it is large and to be "filled" (I think the man page means "filled quickly"): "If the -o reserve option is not set, creating and filling a large malloc-backed memory disk is a very easy way to panic a system." Maybe that helps? Did you try type "swap"? -Arne __________________________________ Do you Yahoo!? The all-new My Yahoo! - Get yours free! http://my.yahoo.com From owner-freebsd-fs@FreeBSD.ORG Fri Jan 7 09:50:06 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 81DC716A4CE for ; Fri, 7 Jan 2005 09:50:06 +0000 (GMT) Received: from web30308.mail.mud.yahoo.com (web30308.mail.mud.yahoo.com [68.142.200.101]) by mx1.FreeBSD.org (Postfix) with SMTP id 2580F43D5A for ; Fri, 7 Jan 2005 09:50:06 +0000 (GMT) (envelope-from barath_yes@yahoo.com.sg) Received: (qmail 4799 invoked by uid 60001); 7 Jan 2005 09:50:05 -0000 Message-ID: <20050107095005.4797.qmail@web30308.mail.mud.yahoo.com> Received: from [203.118.46.137] by web30308.mail.mud.yahoo.com via HTTP; Fri, 07 Jan 2005 17:50:05 CST Date: Fri, 7 Jan 2005 17:50:05 +0800 (CST) From: Barath S To: =?iso-8859-1?q?Arne=20W=F6rner?= , FreeBsd-MailGrp In-Reply-To: <20050107094348.42610.qmail@web41201.mail.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Subject: Re: using mfs of size > 64Mb and system stability X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Jan 2005 09:50:06 -0000 Thank you Arne. I didn't try with 'swap' option as my requirement is to have the files in memory. I will test with 5.3 and see it that works. regards --- Arne Wörner wrote: > --- Barath S wrote: > > My intention is create a file system in memory of > size > > > 128Mb and use that for my application. But, I > > couldn't do that as my system hangs whenever I > create > > a mfs of size > 64Mb. I did the following... > > > > a) Configured memory file system of size 128Mb > using > > 'mdmfs' command with 'malloc' option. When I tried > to > > copy files to this mfs, m/c hanged after copying > some > > files. > > b) Then I rebooted by m/c and configured memory > file > > system of size 64Mb and this time the copying > > operation was successful > > c) I observed that the system hanged whenever I > > configure a mfs of size > 64Mb and used that mfs > > > > a) Can someone provide me hints on what could be > the > > problem ? > > > I use R5.3. According to the man page mdconfig(8) a > md of type > malloc should use "-o reserve", if it is large and > to be "filled" > (I think the man page means "filled quickly"): > "If the -o reserve option is not set, creating and > filling > a large malloc-backed memory disk is a very easy > way to > panic a system." > > Maybe that helps? > > Did you try type "swap"? > > -Arne > > > > > __________________________________ > Do you Yahoo!? > The all-new My Yahoo! - Get yours free! > http://my.yahoo.com > > > __________________________________________________ Do You Yahoo!? Download the latest ringtones, games, and more! http://sg.mobile.yahoo.com From owner-freebsd-fs@FreeBSD.ORG Sat Jan 8 17:33:28 2005 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2ECDB16A4CE for ; Sat, 8 Jan 2005 17:33:28 +0000 (GMT) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 911FD43D31 for ; Sat, 8 Jan 2005 17:33:27 +0000 (GMT) (envelope-from scottl@freebsd.org) Received: from [192.168.254.11] (junior-wifi.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.12.11/8.12.10) with ESMTP id j08Haeo7031654; Sat, 8 Jan 2005 10:36:41 -0700 (MST) (envelope-from scottl@freebsd.org) Message-ID: <41E01905.3040200@freebsd.org> Date: Sat, 08 Jan 2005 10:31:49 -0700 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.2) Gecko/20040929 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Barath S References: <20050107095005.4797.qmail@web30308.mail.mud.yahoo.com> In-Reply-To: <20050107095005.4797.qmail@web30308.mail.mud.yahoo.com> X-Enigmail-Version: 0.86.1.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, hits=0.0 required=3.8 tests=none autolearn=no version=2.63 X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on pooker.samsco.org cc: FreeBsd-MailGrp Subject: Re: using mfs of size > 64Mb and system stability X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Jan 2005 17:33:28 -0000 Using the swap option really is the preferred method. The available kernel malloc pool is limited and is scaled on the amount of system RAM that you have; on my 256MB laptop, it's reporting only 85MB available for kernel malloc. So this would explain why trying to malloc 128MB for md could fail =-). With the swap option, pages are allocated from the buffer/cache and held there. They are only swapped to disk when the system is under memory pressure, otherwise they stay in RAM. Scott Barath S wrote: > Thank you Arne. > I didn't try with 'swap' option as my requirement is > to have the files in memory. I will test with 5.3 and > see it that works. > regards > --- Arne Wörner wrote: > >>--- Barath S wrote: >> >>>My intention is create a file system in memory of >> >>size >> >>>>128Mb and use that for my application. But, I >>> >>>couldn't do that as my system hangs whenever I >> >>create >> >>>a mfs of size > 64Mb. I did the following... >>> >>>a) Configured memory file system of size 128Mb >> >>using >> >>>'mdmfs' command with 'malloc' option. When I tried >> >>to >> >>>copy files to this mfs, m/c hanged after copying >> >>some >> >>>files. >>>b) Then I rebooted by m/c and configured memory >> >>file >> >>>system of size 64Mb and this time the copying >>>operation was successful >>>c) I observed that the system hanged whenever I >>>configure a mfs of size > 64Mb and used that mfs >>> >>>a) Can someone provide me hints on what could be >> >>the >> >>>problem ? >>> >> >>I use R5.3. According to the man page mdconfig(8) a >>md of type >>malloc should use "-o reserve", if it is large and >>to be "filled" >>(I think the man page means "filled quickly"): >> "If the -o reserve option is not set, creating and >>filling >> a large malloc-backed memory disk is a very easy >>way to >> panic a system." >> >>Maybe that helps? >> >>Did you try type "swap"? >> >>-Arne >> >> >> >> >>__________________________________ >>Do you Yahoo!? >>The all-new My Yahoo! - Get yours free! >>http://my.yahoo.com >> >> >> > > > __________________________________________________ > Do You Yahoo!? > Download the latest ringtones, games, and more! > http://sg.mobile.yahoo.com > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"