From owner-freebsd-fs@FreeBSD.ORG Sun Feb 3 20:31:14 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9D9B916A41A for ; Sun, 3 Feb 2008 20:31:14 +0000 (UTC) (envelope-from Artis.Caune@latnet.lv) Received: from esbens.latnet.lv (esbens.latnet.lv [159.148.19.115]) by mx1.freebsd.org (Postfix) with ESMTP id 6537B13C478 for ; Sun, 3 Feb 2008 20:31:13 +0000 (UTC) (envelope-from Artis.Caune@latnet.lv) Received: from localhost (localhost.localdomain [127.0.0.1]) by esbens.latnet.lv (Postfix) with ESMTP id BD85B260CC9; Sun, 3 Feb 2008 22:02:08 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at esbens.latnet.lv Received: from esbens.latnet.lv ([127.0.0.1]) by localhost (esbens.latnet.lv [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tD9XB8Ae03bz; Sun, 3 Feb 2008 22:02:06 +0200 (EET) Received: from [10.0.0.1] (unknown [87.110.135.65]) by esbens.latnet.lv (Postfix) with ESMTP id 1F6F525DCA8; Sun, 3 Feb 2008 22:02:06 +0200 (EET) Message-ID: <47A61DB7.1060506@latnet.lv> Date: Sun, 03 Feb 2008 22:01:59 +0200 From: Artis Caune User-Agent: Thunderbird 2.0.0.6 (Windows/20070728) MIME-Version: 1.0 To: Niki Denev References: <2e77fc10802020152k2f5385c5w5938d91b1183f8e0@mail.gmail.com> In-Reply-To: <2e77fc10802020152k2f5385c5w5938d91b1183f8e0@mail.gmail.com> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS panics X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2008 20:31:14 -0000 Niki Denev wrote: > I also have this in loader.conf : > > vm.kmem_size="1G" > vm.kmem_size_max="1G" Try with: vm.kmem_size="1500M" vfs.zfs.arc_max="512M" vfs.zfs.prefetch_disable="1" From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 08:40:17 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 27B5016A417 for ; Mon, 4 Feb 2008 08:40:17 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from redbull.bpaserver.net (redbullneu.bpaserver.net [213.198.78.217]) by mx1.freebsd.org (Postfix) with ESMTP id D529E13C465 for ; Mon, 4 Feb 2008 08:40:16 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from outgoing.leidinger.net (p54A55FEA.dip.t-dialin.net [84.165.95.234]) by redbull.bpaserver.net (Postfix) with ESMTP id 102B32E0D3; Mon, 4 Feb 2008 09:22:02 +0100 (CET) Received: from webmail.leidinger.net (webmail.leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id A6F492BF5A; Mon, 4 Feb 2008 09:21:51 +0100 (CET) Received: (from www@localhost) by webmail.leidinger.net (8.14.2/8.13.8/Submit) id m148LotL087018; Mon, 4 Feb 2008 09:21:50 +0100 (CET) (envelope-from Alexander@Leidinger.net) Received: from proxy.leidinger.net (proxy.leidinger.net [192.168.1.103]) by webmail.leidinger.net (Horde MIME library) with HTTP; Mon, 04 Feb 2008 09:21:50 +0100 Message-ID: <20080204092150.vprsphymqoog8cw4@webmail.leidinger.net> X-Priority: 3 (Normal) Date: Mon, 04 Feb 2008 09:21:50 +0100 From: Alexander Leidinger To: freebsd-fs@freebsd.org, pjd@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Internet Messaging Program (IMP) H3 (4.1.5) / FreeBSD-8.0 X-BPAnet-MailScanner-Information: Please contact the ISP for more information X-BPAnet-MailScanner: Found to be clean X-BPAnet-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-14.823, required 6, BAYES_00 -15.00, RDNS_DYNAMIC 0.10, TW_ZF 0.08) X-BPAnet-MailScanner-From: alexander@leidinger.net X-Spam-Status: No Cc: Subject: ZFS: invalid label -- what is expected? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 08:40:17 -0000 Hi, I'm in the uncomfortable situation that I have a ZFS which emits the =20 message "label missing or invalid" for one of the 3 disks. As this is =20 neither a mirror nor a raidz, the whole pool is not accessible. This =20 pool just served as a backup for some data which had to be transferred =20 from a graid3 to a raidz. The graid3 does not exist anymore, and the =20 raidz is not populated yet... As I want to have the data back from the interrim-backup pool (several =20 jails, the FreeBSD CVS, ... so more or less nothing important, but =20 time consuming to get back to the previous state by =20 reinstalling/downloading/...), I would like to know where this label =20 zfs is complaining about is supposed to reside, and how is it suppsoed =20 to look like? I would like to give a repair a try (giving zfs the =20 label it expects and hoping it is able to recover at least most of the =20 data (on this pool I had the property copies=3D2)). So anybody out there who can give me some information I need to do =20 this, or at least knows some places in the zfs source I should have a =20 look at? Please CC me, as I'm not subscribed to freebsd-fs@. Bye, Alexander. --=20 Automobile, n.: =09A four-wheeled vehicle that runs up hills and down =09pedestrians. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID =3D B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID =3D 72077137 From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 11:06:58 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C652016A419 for ; Mon, 4 Feb 2008 11:06:58 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B159613C442 for ; Mon, 4 Feb 2008 11:06:58 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m14B6wlx017238 for ; Mon, 4 Feb 2008 11:06:58 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m14B6whd017234 for freebsd-fs@FreeBSD.org; Mon, 4 Feb 2008 11:06:58 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 4 Feb 2008 11:06:58 GMT Message-Id: <200802041106.m14B6whd017234@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 11:06:58 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o kern/116170 fs [panic] Kernel panic when mounting /tmp 3 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o bin/118249 fs mv(1): moving a directory changes its mtime 2 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 06:26:17 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 93C0616A46E for ; Mon, 4 Feb 2008 06:26:17 +0000 (UTC) (envelope-from kyle@moffetthome.net) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.178]) by mx1.freebsd.org (Postfix) with ESMTP id 3E0C413C458 for ; Mon, 4 Feb 2008 06:26:17 +0000 (UTC) (envelope-from kyle@moffetthome.net) Received: by py-out-1112.google.com with SMTP id u52so3124813pyb.10 for ; Sun, 03 Feb 2008 22:26:16 -0800 (PST) Received: by 10.35.83.20 with SMTP id k20mr7496969pyl.7.1202104709219; Sun, 03 Feb 2008 21:58:29 -0800 (PST) Received: by 10.35.115.9 with HTTP; Sun, 3 Feb 2008 21:58:29 -0800 (PST) Message-ID: Date: Mon, 4 Feb 2008 00:58:29 -0500 From: "Kyle Moffett" To: "Jeffrey Hutzelman" In-Reply-To: <876FB8E38251C27B14CCCA29@atlantis.pc.cs.cmu.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <18CC5A4A2AC36D7FF57615EE@ganymede.hub.org> <478AF6BC.8050604@highperformance.net> <20080114142124.Y55696@fledge.watson.org> <876FB8E38251C27B14CCCA29@atlantis.pc.cs.cmu.edu> X-Mailman-Approved-At: Mon, 04 Feb 2008 12:15:24 +0000 Cc: rra@stanford.edu, freebsd-fs@freebsd.org, Robert Watson , matt@linuxbox.com, freebsd-afs@freebsd.org, "Jason C. Wells" , port-freebsd@openafs.org, openafs-devel@openafs.org, freebsd-questions@freebsd.org Subject: Re: [OpenAFS-devel] Re: AFS ... or equivalent ... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 06:26:17 -0000 On Jan 16, 2008 1:48 PM, Jeffrey Hutzelman wrote: > The "let's just slurp everything into the main distribution so we don't > have to worry about stable interfaces" approach is really poor. It > encourages bad engineering practice among people maintaining the main > distribution, discourages innovation and extension by others, and generally > doesn't scale. It's far better to either attempt to maintain stable > external interfaces to the VFS and VM subsystems, or else admit that you > don't have the resources to do so given the relatively small number of > external users, in which case you almost certainly also don't have the > resources to keep on top of updates to something like OpenAFS. The Linux Kernel presents a very strong counter-argument-by-example. The amount of patches merged per released version has been linearly increasing over the last several years; the 2.6.23 => 2.6.24 patch was 49MB uncompressed, with a 5.7MB changelog. Of that, a significant portion were VFS changes which touched most filesystems. The various filesystem-related changes alone between 2.6.23 and 2.6.24 were 2.9MB. For reference, the *entire* OpenAFS diff between 2.4.6 and 2.5.30 is all of 8.2MB. The Linux Kernel changes include partial support for having per-process views of a single filesystem (Specifically /proc, so /proc/net can have differing contents between network namespaces). Other features which Linux supports that virtually no other OS does is multiple filesystem namespaces, where the mount-tree is selectively independent or shared between namespaces. I realize that some people are probably already aware of most of that, but I thought it should be mentioned that "slurp everything into the main distribution" actually scales very well with respect to the Linux kernel. It means that the people who are making changes (to the VFS, for example) have to go around and fix *all* the filesystems, and in addition when a bug gets fixed in one filesystem then most of the others get checked for that same bug. OpenAFS also does not benchmark very well under Linux against most of the other networked filesystems (even ones using encryption), as it does not support the fine-grained locking that Linux does. Unfortunately it isn't practical for Linux to reuse existing OpenAFS code as the licenses are partially incompatible. > In the long run, I'm guessing that the OpenAFS cache manager evolves more > quickly than FreeBSD's VFS interface, which makes pulling the CM into the > kernel tree a losing battle. If you disagree, by all means fork that part > of AFS (or get someone else to do so) and see what happens (AFS's > user/kernel and RPC interfaces are both fairly stable, so forking just the > kernel parts should be mostly feasible). As it so happens this is exactly what is happening right now in the Linux Kernel. David Howells (original author of the Linux "keyring" subsystem) has been writing a generic userspace+kernelspace FS-Cache system which can use either a block device or a mounted filesystem as storage. It presently supports NFS and the minimal in-kernel AFS client and is planned to be mostly merged into 2.6.25. The benefits of being able to share innovations in caching between AFS, NFS, and other networked filesystems is quite significant. My apologies for anything in this email that may be construed as offensive; the intent is as an honest technical discussion of development methods and practices. Cheers, Kyle Moffett From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 15:47:46 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40ED016A417; Mon, 4 Feb 2008 15:47:46 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id 8544213C469; Mon, 4 Feb 2008 15:47:45 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id 2EFD243DEC6; Mon, 4 Feb 2008 17:47:44 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Sdig71qR3LVa; Mon, 4 Feb 2008 17:47:44 +0200 (EET) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 3466343DDA5; Mon, 4 Feb 2008 17:47:43 +0200 (EET) Message-ID: <47A7339E.4070708@icyb.net.ua> Date: Mon, 04 Feb 2008 17:47:42 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20080123) MIME-Version: 1.0 To: Remko Lodder , scottl@FreeBSD.org, freebsd-fs@freebsd.org References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> In-Reply-To: <47A2F404.7010208@icyb.net.ua> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Pav Lucistnik , Bruce Evans Subject: fs/udf: vm pages "overlap" while reading large dir X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 15:47:46 -0000 More on the problem with reading big directories on UDF. First, some sleuthing. I came to believe that the problem is caused by some larger change in vfs/vm/buf area. It seems that now VMIO is applied to more vnode types than before. In particular it seems that now vnodes for devices have non-NULL v_object (or maybe this is about directory vnodes, I am not sure). Also it seems that the problem should happen for any directory with size larger than four 2048-bytes sectors (I think that any directory with > 300 files would definitely qualify). After some code reading and run-time debugging, here are some facts about udf directory reading: 1. bread-ing is done via device vnode (as opposed to directory vnodes), as a consequence udf_strategy is not involved in directory reading 2. the device vnode has a non-NULL v_object, this means that VMIO is used 3. it seems that the code assumed that non-VM buffers are used 4. bread is done on as much data as possible, up to MAXBSIZE [and even slightly more in some cases] (i.e. code determines directory data size and attempts to read in everything in one go) 5. physical sector number adjusted to DEV_BSIZE (512 bytes) sectors is passed to bread - this is because of #1 above and peculiarities of buf system 6. directory reading and file reading are quite different, because the latter does reading via file vnode Let's a consider a simple case of "directory reading" 5 * PAGE_SIZE (4K) bytes starting from a physical sector N with N%2 = 0 from media with sector size of 2K (most common CD/DVD case): 1. we want to read 12 KB = 3 pages = 6 sectors starting from sector N, N%2 = 0 2. 4*N is passed as a sector number to bread 3. bo_bsize of the corresponding bufobj is a media sector size, i.e. 2K 4. actual read in bread will happen from b_iooffset of 4*N*DEV_BSIZE = N*2K, i.e. correct offset within the device 5. for a fresh read getblk will be called 6. getblk will set b_offset to blkno*bo_bsize=4*N*2K, i.e. 4 times the correct byte offset within the device 7. allocbuf will allocate pages using B_VMIO case 8. allocbuf calculates base page index as follows: pi = b_offset/PAGE_SIZE this means the following: sectors N and N+1 will be bound to a page with index 4*N*2K/4K = 2*N sectors N+2 and N+3 will be bound to the next page 2*N +1 sectors N+4 and N+5 is tied to the next page 2*N + 2 Now let's consider a "directory read" of 2 sectors (1 page) starting at physical sector N+1. Using the same calculations as above, we see that the sector will be tied to a page 2*(N+1) = 2*N + 2. And this page already contains valid cached data from the previous read, *but* it contains data from sectors N+4 and N+5 instead of N+1 and N+2. So, we see, that because of b_offset being 4 times the actual byte offset, we get incorrect page indexing. Namely, sector X gets associated with different pages depending on what sector is used as a starting sector for bread. If bread starts at sector N, then data of sector N+1 is associated with (second half of) page with index 2*N; but if bread starts at sector N+1 then it gets associated with (the first half of) page with index 2*(N+1). Previously, before VMIO change, data for all reads was put into separate buffers that did not share anything between them. So the problem was limited only to wasting memory with duplicate data, but no actual over-runs did happen. Now we have the over-runs because the VM pages are shared between the buffers of the same vnode. One obvious solution is to limit bread size to 2*PAGE_SIZE = 4 * sector_size. In this case, as before, we would waste some memory on duplicate data but we would avoid page overruns. If we limit bread size even more, to 1 sector, then we would not have any duplicate data at all. But there would still be some resource waste - each page would correspond to one sector, so 4K page would have only 2K of valid data and the other half in each page is unused. Another solution, which to me seems to be better, is to do "usual" reading - pass a directory vnode and 2048-byte sector offset to bread. In this case udf_strategy will get called for bklno translation, all offsets and indexes will be correct and everything will work perfectly as it is the case for all other filesystems. The only overhead in this case comes from the need to handle UDF_INVALID_BMAP case (where data is embedded into file entry). So it means that we have to call bmap_internal to see if we have that special case and hanlde it, and if the case is not special we would call bread on a directory vnode which means that bmap_internal would be called again in udf_strategy. One optimization that can be done in this case is to create a ligher version of bmap_internal that would merely check for the special case and wouldn't do anything else. I am attaching a patch based on the ideas above. It fixes the problem for me and doesn't seem to create any new ones :-) About the patch: hunk #1 fixes a copy/paste hunk #2 should fixes strategy to not set junk b_blkno in case of udf_bmap_internal failure and also replaces divisions/multiplications with bit-shifts hunk #3 really limits max size in bread to MAXBSIZE and also makes bread to be done a directory vnode BTW, maybe it's a good idea to further limit size of bread() anyway. Like always reading 1 sector. Anyway, this should make any difference only in minority of cases and I am not sure about performance effects (I do not expect difference in anything else other than performance). on 01/02/2008 12:27 Andriy Gapon said the following: > on 01/02/2008 12:00 Andriy Gapon said the following: >> ---- a different, under-debugged problem ----- >> BTW, on some smaller directories (but still large ones) I get some very >> strange problems with reading a directory too. It seems like some bad >> interaction between udf and buffer cache system. I added a lot of >> debugging prints and the problems looks like the following: >> >> read starting at physical sector (2048-byte one) N, size is ~20K, N%4=0 >> bread(4 * N, some_big_size) >> correct data is read >> repeat the above couple dozen times >> read starting at physical sector (2048-byte one) N+1, size is ~20K >> bread(4 * (N+1), some_big_size) >> data is read from physical sector N+4 (instead of N+1) >> >> I remember that Bruce Evance warned me that something like this could >> happen but I couldn't understand him, because I don't understand >> VM/buffer subsystem. I'll try to dig up the email. >> > > Sorry for the flood - additional info: > if I limit max read size in udf_readatoffset() to 2048 (instead of > MAXBSIZE), then large directories can be read OK. Seems like something > with overlapping buffers, maybe? > > BTW, here's how I created test environment for the described issues (in > tcsh): > > mkdir /tmp/bigdir > > cd /tmp/bigdir > > set i=1 > > while ($i < NNNNN) > touch file.$i > set i=`expr $i + 1` > end > > cd /tmp > > mkisofs -udf -o test.iso bigdir > > mdconfig -a -t vnode -f test.iso -S 2048 -u 0 > > mount_udf /dev/md0 /mnt > > ls -l /mnt > -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 15:55:26 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 52F8616A41B; Mon, 4 Feb 2008 15:55:26 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id E835A13C4CC; Mon, 4 Feb 2008 15:55:25 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from phobos.samsco.home (phobos.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.8/8.13.8) with ESMTP id m14FtEvB054627; Mon, 4 Feb 2008 08:55:15 -0700 (MST) (envelope-from scottl@samsco.org) Message-ID: <47A73562.4010309@samsco.org> Date: Mon, 04 Feb 2008 08:55:14 -0700 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.11) Gecko/20071128 SeaMonkey/1.1.7 MIME-Version: 1.0 To: Andriy Gapon References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A7339E.4070708@icyb.net.ua> In-Reply-To: <47A7339E.4070708@icyb.net.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=5.4 tests=ALL_TRUSTED autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on pooker.samsco.org Cc: freebsd-fs@FreeBSD.org, freebsd-hackers@FreeBSD.org, Remko Lodder , Pav Lucistnik , Bruce Evans Subject: Re: fs/udf: vm pages "overlap" while reading large dir X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 15:55:26 -0000 Andriy Gapon wrote: > More on the problem with reading big directories on UDF. > > First, some sleuthing. I came to believe that the problem is caused by > some larger change in vfs/vm/buf area. It seems that now VMIO is applied > to more vnode types than before. In particular it seems that now vnodes > for devices have non-NULL v_object (or maybe this is about directory > vnodes, I am not sure). > > Also it seems that the problem should happen for any directory with size > larger than four 2048-bytes sectors (I think that any directory with > > 300 files would definitely qualify). > > After some code reading and run-time debugging, here are some facts > about udf directory reading: > 1. bread-ing is done via device vnode (as opposed to directory vnodes), > as a consequence udf_strategy is not involved in directory reading > 2. the device vnode has a non-NULL v_object, this means that VMIO is used > 3. it seems that the code assumed that non-VM buffers are used > 4. bread is done on as much data as possible, up to MAXBSIZE [and even > slightly more in some cases] (i.e. code determines directory data size > and attempts to read in everything in one go) > 5. physical sector number adjusted to DEV_BSIZE (512 bytes) sectors is > passed to bread - this is because of #1 above and peculiarities of buf > system > 6. directory reading and file reading are quite different, because the > latter does reading via file vnode > > Let's a consider a simple case of "directory reading" 5 * PAGE_SIZE (4K) > bytes starting from a physical sector N with N%2 = 0 from media with > sector size of 2K (most common CD/DVD case): > 1. we want to read 12 KB = 3 pages = 6 sectors starting from sector N, > N%2 = 0 > 2. 4*N is passed as a sector number to bread > 3. bo_bsize of the corresponding bufobj is a media sector size, i.e. 2K > 4. actual read in bread will happen from b_iooffset of 4*N*DEV_BSIZE = > N*2K, i.e. correct offset within the device > 5. for a fresh read getblk will be called > 6. getblk will set b_offset to blkno*bo_bsize=4*N*2K, i.e. 4 times the > correct byte offset within the device > 7. allocbuf will allocate pages using B_VMIO case > 8. allocbuf calculates base page index as follows: pi = b_offset/PAGE_SIZE > this means the following: > sectors N and N+1 will be bound to a page with index 4*N*2K/4K = 2*N > sectors N+2 and N+3 will be bound to the next page 2*N +1 > sectors N+4 and N+5 is tied to the next page 2*N + 2 > > Now let's consider a "directory read" of 2 sectors (1 page) starting at > physical sector N+1. > Using the same calculations as above, we see that the sector will be > tied to a page 2*(N+1) = 2*N + 2. > And this page already contains valid cached data from the previous read, > *but* it contains data from sectors N+4 and N+5 instead of N+1 and N+2. > > So, we see, that because of b_offset being 4 times the actual byte > offset, we get incorrect page indexing. Namely, sector X gets associated > with different pages depending on what sector is used as a starting > sector for bread. If bread starts at sector N, then data of sector N+1 > is associated with (second half of) page with index 2*N; but if bread > starts at sector N+1 then it gets associated with (the first half of) > page with index 2*(N+1). > > Previously, before VMIO change, data for all reads was put into separate > buffers that did not share anything between them. So the problem was > limited only to wasting memory with duplicate data, but no actual > over-runs did happen. Now we have the over-runs because the VM pages are > shared between the buffers of the same vnode. > > One obvious solution is to limit bread size to 2*PAGE_SIZE = 4 * > sector_size. In this case, as before, we would waste some memory on > duplicate data but we would avoid page overruns. If we limit bread size > even more, to 1 sector, then we would not have any duplicate data at > all. But there would still be some resource waste - each page would > correspond to one sector, so 4K page would have only 2K of valid data > and the other half in each page is unused. > > Another solution, which to me seems to be better, is to do "usual" > reading - pass a directory vnode and 2048-byte sector offset to bread. > In this case udf_strategy will get called for bklno translation, all > offsets and indexes will be correct and everything will work perfectly > as it is the case for all other filesystems. > The only overhead in this case comes from the need to handle > UDF_INVALID_BMAP case (where data is embedded into file entry). So it > means that we have to call bmap_internal to see if we have that special > case and hanlde it, and if the case is not special we would call bread > on a directory vnode which means that bmap_internal would be called > again in udf_strategy. > One optimization that can be done in this case is to create a ligher > version of bmap_internal that would merely check for the special case > and wouldn't do anything else. > > I am attaching a patch based on the ideas above. It fixes the problem > for me and doesn't seem to create any new ones :-) > About the patch: > hunk #1 fixes a copy/paste > hunk #2 should fixes strategy to not set junk b_blkno in case of > udf_bmap_internal failure and also replaces divisions/multiplications > with bit-shifts > hunk #3 really limits max size in bread to MAXBSIZE and also makes bread > to be done a directory vnode > > BTW, maybe it's a good idea to further limit size of bread() anyway. > Like always reading 1 sector. Anyway, this should make any difference > only in minority of cases and I am not sure about performance effects (I > do not expect difference in anything else other than performance). > I think you forgot to attach the patch =-) This is an excellent write-up, and I think you're on the right track. It's been nearly 8 years since I wrote most of this code, so my memory is a bit fuzzy, but I think the reason that I used the device vnode is because I was having trouble with overlapping buffers when using the directory vnode, so this was the easiest way to avoid that at the time. Since it sounds like this is no longer a viable solution, I definitely support your ideas. Scott From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 15:56:24 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 685B316A468; Mon, 4 Feb 2008 15:56:24 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id AA68913C44B; Mon, 4 Feb 2008 15:56:23 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id 9178F43D7AD; Mon, 4 Feb 2008 17:56:22 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id whg4EB8ETErA; Mon, 4 Feb 2008 17:56:22 +0200 (EET) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 9ABA143C8F2; Mon, 4 Feb 2008 17:56:21 +0200 (EET) Message-ID: <47A735A4.3060506@icyb.net.ua> Date: Mon, 04 Feb 2008 17:56:20 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20080123) MIME-Version: 1.0 To: Remko Lodder , scottl@FreeBSD.org, freebsd-fs@freebsd.org References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> In-Reply-To: <47A2F404.7010208@icyb.net.ua> Content-Type: multipart/mixed; boundary="------------070300020503030807040707" Cc: freebsd-hackers@freebsd.org, Pav Lucistnik , Bruce Evans Subject: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 15:56:24 -0000 This is a multi-part message in MIME format. --------------070300020503030807040707 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit More on the problem with reading big directories on UDF. First, some sleuthing. I came to believe that the problem is caused by some larger change in vfs/vm/buf area. It seems that now VMIO is applied to more vnode types than before. In particular it seems that now vnodes for devices have non-NULL v_object (or maybe this is about directory vnodes, I am not sure). Also it seems that the problem should happen for any directory with size larger than four 2048-bytes sectors (I think that any directory with > 300 files would definitely qualify). After some code reading and run-time debugging, here are some facts about udf directory reading: 1. bread-ing is done via device vnode (as opposed to directory vnodes), as a consequence udf_strategy is not involved in directory reading 2. the device vnode has a non-NULL v_object, this means that VMIO is used 3. it seems that the code assumed that non-VM buffers are used 4. bread is done on as much data as possible, up to MAXBSIZE [and even slightly more in some cases] (i.e. code determines directory data size and attempts to read in everything in one go) 5. physical sector number adjusted to DEV_BSIZE (512 bytes) sectors is passed to bread - this is because of #1 above and peculiarities of buf system 6. directory reading and file reading are quite different, because the latter does reading via file vnode Let's a consider a simple case of "directory reading" 5 * PAGE_SIZE (4K) bytes starting from a physical sector N with N%2 = 0 from media with sector size of 2K (most common CD/DVD case): 1. we want to read 12 KB = 3 pages = 6 sectors starting from sector N, N%2 = 0 2. 4*N is passed as a sector number to bread 3. bo_bsize of the corresponding bufobj is a media sector size, i.e. 2K 4. actual read in bread will happen from b_iooffset of 4*N*DEV_BSIZE = N*2K, i.e. correct offset within the device 5. for a fresh read getblk will be called 6. getblk will set b_offset to blkno*bo_bsize=4*N*2K, i.e. 4 times the correct byte offset within the device 7. allocbuf will allocate pages using B_VMIO case 8. allocbuf calculates base page index as follows: pi = b_offset/PAGE_SIZE this means the following: sectors N and N+1 will be bound to a page with index 4*N*2K/4K = 2*N sectors N+2 and N+3 will be bound to the next page 2*N +1 sectors N+4 and N+5 is tied to the next page 2*N + 2 Now let's consider a "directory read" of 2 sectors (1 page) starting at physical sector N+1. Using the same calculations as above, we see that the sector will be tied to a page 2*(N+1) = 2*N + 2. And this page already contains valid cached data from the previous read, *but* it contains data from sectors N+4 and N+5 instead of N+1 and N+2. So, we see, that because of b_offset being 4 times the actual byte offset, we get incorrect page indexing. Namely, sector X gets associated with different pages depending on what sector is used as a starting sector for bread. If bread starts at sector N, then data of sector N+1 is associated with (second half of) page with index 2*N; but if bread starts at sector N+1 then it gets associated with (the first half of) page with index 2*(N+1). Previously, before VMIO change, data for all reads was put into separate buffers that did not share anything between them. So the problem was limited only to wasting memory with duplicate data, but no actual over-runs did happen. Now we have the over-runs because the VM pages are shared between the buffers of the same vnode. One obvious solution is to limit bread size to 2*PAGE_SIZE = 4 * sector_size. In this case, as before, we would waste some memory on duplicate data but we would avoid page overruns. If we limit bread size even more, to 1 sector, then we would not have any duplicate data at all. But there would still be some resource waste - each page would correspond to one sector, so 4K page would have only 2K of valid data and the other half in each page is unused. Another solution, which to me seems to be better, is to do "usual" reading - pass a directory vnode and 2048-byte sector offset to bread. In this case udf_strategy will get called for bklno translation, all offsets and indexes will be correct and everything will work perfectly as it is the case for all other filesystems. The only overhead in this case comes from the need to handle UDF_INVALID_BMAP case (where data is embedded into file entry). So it means that we have to call bmap_internal to see if we have that special case and hanlde it, and if the case is not special we would call bread on a directory vnode which means that bmap_internal would be called again in udf_strategy. One optimization that can be done in this case is to create a ligher version of bmap_internal that would merely check for the special case and wouldn't do anything else. I am attaching a patch based on the ideas above. It fixes the problem for me and doesn't seem to create any new ones :-) About the patch: hunk #1 fixes a copy/paste hunk #2 should fixes strategy to not set junk b_blkno in case of udf_bmap_internal failure and also replaces divisions/multiplications with bit-shifts hunk #3 really limits max size in bread to MAXBSIZE and also makes bread to be done a directory vnode BTW, maybe it's a good idea to further limit size of bread() anyway. Like always reading 1 sector. Anyway, this should make any difference only in minority of cases and I am not sure about performance effects (I do not expect difference in anything else other than performance). on 01/02/2008 12:27 Andriy Gapon said the following: > on 01/02/2008 12:00 Andriy Gapon said the following: >> ---- a different, under-debugged problem ----- >> BTW, on some smaller directories (but still large ones) I get some very >> strange problems with reading a directory too. It seems like some bad >> interaction between udf and buffer cache system. I added a lot of >> debugging prints and the problems looks like the following: >> >> read starting at physical sector (2048-byte one) N, size is ~20K, N%4=0 >> bread(4 * N, some_big_size) >> correct data is read >> repeat the above couple dozen times >> read starting at physical sector (2048-byte one) N+1, size is ~20K >> bread(4 * (N+1), some_big_size) >> data is read from physical sector N+4 (instead of N+1) >> >> I remember that Bruce Evance warned me that something like this could >> happen but I couldn't understand him, because I don't understand >> VM/buffer subsystem. I'll try to dig up the email. >> > > Sorry for the flood - additional info: > if I limit max read size in udf_readatoffset() to 2048 (instead of > MAXBSIZE), then large directories can be read OK. Seems like something > with overlapping buffers, maybe? > > BTW, here's how I created test environment for the described issues (in > tcsh): > > mkdir /tmp/bigdir > > cd /tmp/bigdir > > set i=1 > > while ($i < NNNNN) > touch file.$i > set i=`expr $i + 1` > end > > cd /tmp > > mkisofs -udf -o test.iso bigdir > > mdconfig -a -t vnode -f test.iso -S 2048 -u 0 > > mount_udf /dev/md0 /mnt > > ls -l /mnt > -- Andriy Gapon --------------070300020503030807040707 Content-Type: text/x-patch; name="udf_vnops.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="udf_vnops.diff" --- udf_vnops.c.orig 2008-01-29 23:50:49.000000000 +0200 +++ udf_vnops.c 2008-02-03 23:36:15.000000000 +0200 @@ -340,7 +340,7 @@ #define lblkno(udfmp, loc) ((loc) >> (udfmp)->bshift) #define blkoff(udfmp, loc) ((loc) & (udfmp)->bmask) -#define lblktosize(imp, blk) ((blk) << (udfmp)->bshift) +#define lblktosize(udfmp, blk) ((blk) << (udfmp)->bshift) static int udf_read(struct vop_read_args *ap) @@ -802,7 +802,6 @@ int maxsize; daddr_t sector; struct bufobj *bo; - int multiplier; bp = a->a_bp; vp = a->a_vp; @@ -813,21 +812,17 @@ * Files that are embedded in the fentry don't translate well * to a block number. Reject. */ - if (udf_bmap_internal(node, bp->b_lblkno * node->udfmp->bsize, - §or, &maxsize)) { + if (udf_bmap_internal(node, lblktosize(node->udfmp, bp->b_lblkno), §or, &maxsize)) { clrbuf(bp); bp->b_blkno = -1; + bufdone(bp); + return (0); } + /* udf_bmap_internal gives sector numbers, bio works with device blocks */ + bp->b_blkno = sector << (node->udfmp->bshift - DEV_BSHIFT); + } - /* bmap gives sector numbers, bio works with device blocks */ - multiplier = node->udfmp->bsize / DEV_BSIZE; - bp->b_blkno = sector * multiplier; - } - if ((long)bp->b_blkno == -1) { - bufdone(bp); - return (0); - } bo = node->udfmp->im_bo; bp->b_iooffset = dbtob(bp->b_blkno); BO_STRATEGY(bo, bp); @@ -1061,6 +1056,7 @@ udfmp = node->udfmp; *bp = NULL; + /* this call is only to handle UDF_INVALID_BMAP case, its results are not used for common case */ error = udf_bmap_internal(node, offset, §or, &max_size); if (error == UDF_INVALID_BMAP) { /* @@ -1078,9 +1074,8 @@ /* Adjust the size so that it is within range */ if (*size == 0 || *size > max_size) *size = max_size; - *size = min(*size, MAXBSIZE); - - if ((error = udf_readlblks(udfmp, sector, *size + (offset & udfmp->bmask), bp))) { + *size = min(*size, MAXBSIZE - blkoff(udfmp, offset)); + if ((error = bread(node->i_vnode, lblkno(udfmp, offset), (*size + blkoff(udfmp, offset) + udfmp->bmask) & ~udfmp->bmask, NOCRED, bp))) { printf("warning: udf_readlblks returned error %d\n", error); /* note: *bp may be non-NULL */ return (error); --------------070300020503030807040707-- From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 16:27:07 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 27F4F16A4D8 for ; Mon, 4 Feb 2008 16:27:07 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from shadow.wildlava.net (shadow.wildlava.net [67.40.138.81]) by mx1.freebsd.org (Postfix) with ESMTP id F20EA13C4E8 for ; Mon, 4 Feb 2008 16:27:06 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from [129.162.240.70] (unknown [129.162.240.70]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by shadow.wildlava.net (Postfix) with ESMTP id 0731B8F429 for ; Mon, 4 Feb 2008 09:27:05 -0700 (MST) Message-ID: <47A73C8D.3000107@skyrush.com> Date: Mon, 04 Feb 2008 09:25:49 -0700 From: Joe Peterson User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 16:27:07 -0000 I have a ZFS pool of one partition (i.e. no mirror/RAID) with a reported checksum error (on one file). Scrubs find it, and when I try to read the file, I get an I/O error ("bad address" on fbsd). When I try to copy the file, I get only 655360 bytes copied, and then the copy stops. I assume this is because the next block is where the error is. However, I'd like to do some forensics on this, especially since I am not convinced the disk is bad. Is there a way to force ZFS to give me the whole file so I can compare it to my backup? Is there a way to get ZFS to give more specifics on the checksum issue (i.e. which block, where in file, etc.)? Thanks, Joe From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 16:33:11 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5F29416A420 for ; Mon, 4 Feb 2008 16:33:11 +0000 (UTC) (envelope-from cracauer@koef.zs64.net) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by mx1.freebsd.org (Postfix) with ESMTP id EAC1013C47E for ; Mon, 4 Feb 2008 16:33:10 +0000 (UTC) (envelope-from cracauer@koef.zs64.net) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by koef.zs64.net (8.14.2/8.14.2) with ESMTP id m14GX8oa096377; Mon, 4 Feb 2008 17:33:08 +0100 (CET) (envelope-from cracauer@koef.zs64.net) Received: (from cracauer@localhost) by koef.zs64.net (8.14.2/8.14.2/Submit) id m14GX8BE096376; Mon, 4 Feb 2008 11:33:08 -0500 (EST) (envelope-from cracauer) Date: Mon, 4 Feb 2008 11:33:08 -0500 From: Martin Cracauer To: "Julian H. Stacey" Message-ID: <20080204163308.GA96092@cons.org> References: <20080201172214.GA55957@cons.org> <200802021916.m12JGUjN049706@fire.js.berklix.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200802021916.m12JGUjN049706@fire.js.berklix.net> User-Agent: Mutt/1.4.2.3i Cc: freebsd-fs@freebsd.org, Martin Cracauer Subject: Re: fsck and mount disagree on whether superblocks are usable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 16:33:11 -0000 Julian H. Stacey wrote on Sat, Feb 02, 2008 at 08:16:30PM +0100: > Martin Cracauer wrote: > > This is not an emergency but I find it odd. Mount and fsck agree on > > whether superblocks are usable. Mount can mount readonly, but fsck > > can use neither the primary superblock nor the alternatives. > > > > 32 is not a file system superblock > > Just in case, You know secondary block on newer FSs moved from 32 ? > Ref man fsck_ufs > -b Use the block specified immediately after the flag as the super > block for the file system. An alternate super block is usually > located at block 32 for UFS1, and block 160 for UFS2. Thanks, Julian. I'm honestly don't know how to tell whether I have ufs1 or ufs2. Anyone? The source machines runs 6-stable, the receiver runs 7-stable, but the filesystems have been created long in the past. I also think I might have a disk geometry problem here, that blocks aren't where they are supposed to be. I ran fsck by disabling the check to the second superblock, just using the first one. I lost some files but not enough to have an outright block mapping mixup. The whole thing still looks strange. Martin -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer http://www.cons.org/cracauer/ FreeBSD - where you want to go, today. http://www.freebsd.org/ From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 16:02:06 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D334316A41B; Mon, 4 Feb 2008 16:02:06 +0000 (UTC) (envelope-from jerrymc@gizmo.acns.msu.edu) Received: from gizmo.acns.msu.edu (gizmo.acns.msu.edu [35.8.1.43]) by mx1.freebsd.org (Postfix) with ESMTP id 701FC13C45D; Mon, 4 Feb 2008 16:02:06 +0000 (UTC) (envelope-from jerrymc@gizmo.acns.msu.edu) Received: from gizmo.acns.msu.edu (localhost [127.0.0.1]) by gizmo.acns.msu.edu (8.13.6/8.13.6) with ESMTP id m14FwiQg007718; Mon, 4 Feb 2008 10:58:44 -0500 (EST) (envelope-from jerrymc@gizmo.acns.msu.edu) Received: (from jerrymc@localhost) by gizmo.acns.msu.edu (8.13.6/8.13.6/Submit) id m14Fwh3o007717; Mon, 4 Feb 2008 10:58:43 -0500 (EST) (envelope-from jerrymc) Date: Mon, 4 Feb 2008 10:58:43 -0500 From: Jerry McAllister To: Kyle Moffett Message-ID: <20080204155842.GA7685@gizmo.acns.msu.edu> References: <18CC5A4A2AC36D7FF57615EE@ganymede.hub.org> <478AF6BC.8050604@highperformance.net> <20080114142124.Y55696@fledge.watson.org> <876FB8E38251C27B14CCCA29@atlantis.pc.cs.cmu.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.2i X-Mailman-Approved-At: Mon, 04 Feb 2008 17:11:32 +0000 Cc: rra@stanford.edu, freebsd-fs@freebsd.org, Robert Watson , matt@linuxbox.com, freebsd-afs@freebsd.org, "Jason C. Wells" , port-freebsd@openafs.org, openafs-devel@openafs.org, freebsd-questions@freebsd.org, Jeffrey Hutzelman Subject: Re: [OpenAFS-devel] Re: AFS ... or equivalent ... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 16:02:06 -0000 On Mon, Feb 04, 2008 at 12:58:29AM -0500, Kyle Moffett wrote: > On Jan 16, 2008 1:48 PM, Jeffrey Hutzelman wrote: > > The "let's just slurp everything into the main distribution so we don't > > have to worry about stable interfaces" approach is really poor. It > > encourages bad engineering practice among people maintaining the main > > distribution, discourages innovation and extension by others, and generally > > doesn't scale. It's far better to either attempt to maintain stable > > external interfaces to the VFS and VM subsystems, or else admit that you > > don't have the resources to do so given the relatively small number of > > external users, in which case you almost certainly also don't have the > > resources to keep on top of updates to something like OpenAFS. > > The Linux Kernel presents a very strong counter-argument-by-example. > The amount of patches merged per released version has been linearly > increasing over the last several years; the 2.6.23 => 2.6.24 patch was > 49MB uncompressed, with a 5.7MB changelog. Of that, a significant > portion were VFS changes which touched most filesystems. The various > filesystem-related changes alone between 2.6.23 and 2.6.24 were > 2.9MB. So, there are reasons why many of us prefer FreeBSD to Linux. ////jerry ........ For reference, the *entire* OpenAFS diff between 2.4.6 and > 2.5.30 is all of 8.2MB. The Linux Kernel changes include partial > support for having per-process views of a single filesystem > (Specifically /proc, so /proc/net can have differing contents between > network namespaces). Other features which Linux supports that > virtually no other OS does is multiple filesystem namespaces, where > the mount-tree is selectively independent or shared between > namespaces. > From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 17:25:35 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2E44416A41A for ; Mon, 4 Feb 2008 17:25:35 +0000 (UTC) (envelope-from jhs@berklix.org) Received: from tower.berklix.org (tower.berklix.org [83.236.223.114]) by mx1.freebsd.org (Postfix) with ESMTP id A650513C458 for ; Mon, 4 Feb 2008 17:25:34 +0000 (UTC) (envelope-from jhs@berklix.org) Received: from js.berklix.net (p549A5BC6.dip.t-dialin.net [84.154.91.198]) (authenticated bits=0) by tower.berklix.org (8.13.6/8.13.6) with ESMTP id m14HPVl7090420; Mon, 4 Feb 2008 17:25:32 GMT (envelope-from jhs@berklix.org) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by js.berklix.net (8.13.8/8.13.8) with ESMTP id m14HRP9t024266; Mon, 4 Feb 2008 18:27:25 +0100 (CET) (envelope-from jhs@berklix.org) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.13.8/8.13.8) with ESMTP id m14HREuN049123; Mon, 4 Feb 2008 18:27:19 +0100 (CET) (envelope-from jhs@fire.js.berklix.net) Message-Id: <200802041727.m14HREuN049123@fire.js.berklix.net> To: Martin Cracauer In-reply-to: <20080204163308.GA96092@cons.org> References: <20080201172214.GA55957@cons.org> <200802021916.m12JGUjN049706@fire.js.berklix.net> <20080204163308.GA96092@cons.org> Comments: In-reply-to Martin Cracauer message dated "Mon, 04 Feb 2008 11:33:08 -0500." Date: Mon, 04 Feb 2008 18:27:14 +0100 From: "Julian H. Stacey" Cc: freebsd-fs@freebsd.org Subject: Re: fsck and mount disagree on whether superblocks are usable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 17:25:35 -0000 Martin Cracauer wrote: > Julian H. Stacey wrote on Sat, Feb 02, 2008 at 08:16:30PM +0100: > > Martin Cracauer wrote: > > > This is not an emergency but I find it odd. Mount and fsck agree on > > > whether superblocks are usable. Mount can mount readonly, but fsck > > > can use neither the primary superblock nor the alternatives. > > > > > > 32 is not a file system superblock > > > > Just in case, You know secondary block on newer FSs moved from 32 ? > > Ref man fsck_ufs > > -b Use the block specified immediately after the flag as the super > > block for the file system. An alternate super block is usually > > located at block 32 for UFS1, and block 160 for UFS2. > > Thanks, Julian. > > I'm honestly don't know how to tell whether I have ufs1 or ufs2. I didnt either, but wanted to know & just found one way: dumpfs /dev/____ | grep -i ufs -- Julian Stacey. BSD Unix Linux Net Consultant, Munich. http://berklix.com From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 18:49:56 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 02A1416A417; Mon, 4 Feb 2008 18:49:56 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from redbull.bpaserver.net (redbullneu.bpaserver.net [213.198.78.217]) by mx1.freebsd.org (Postfix) with ESMTP id B066513C4F5; Mon, 4 Feb 2008 18:49:55 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from outgoing.leidinger.net (p54A55BED.dip.t-dialin.net [84.165.91.237]) by redbull.bpaserver.net (Postfix) with ESMTP id 551C92E273; Mon, 4 Feb 2008 19:49:45 +0100 (CET) Received: from webmail.leidinger.net (webmail.leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id E1DF52B994; Mon, 4 Feb 2008 19:49:36 +0100 (CET) Received: (from www@localhost) by webmail.leidinger.net (8.14.2/8.13.8/Submit) id m14InaDm091273; Mon, 4 Feb 2008 19:49:36 +0100 (CET) (envelope-from Alexander@Leidinger.net) Received: from proxy.leidinger.net (proxy.leidinger.net [192.168.1.103]) by webmail.leidinger.net (Horde MIME library) with HTTP; Mon, 04 Feb 2008 19:49:36 +0100 Message-ID: <20080204194936.thzddva9a8s4cwsg@webmail.leidinger.net> X-Priority: 3 (Normal) Date: Mon, 04 Feb 2008 19:49:36 +0100 From: Alexander Leidinger To: Alexander Leidinger References: <20080204092150.vprsphymqoog8cw4@webmail.leidinger.net> In-Reply-To: <20080204092150.vprsphymqoog8cw4@webmail.leidinger.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.1.5) / FreeBSD-8.0 X-BPAnet-MailScanner-Information: Please contact the ISP for more information X-BPAnet-MailScanner: Found to be clean X-BPAnet-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-14.823, required 6, BAYES_00 -15.00, RDNS_DYNAMIC 0.10, TW_ZF 0.08) X-BPAnet-MailScanner-From: alexander@leidinger.net X-Spam-Status: No Cc: freebsd-fs@freebsd.org, pjd@freebsd.org Subject: Re: ZFS: invalid label -- what is expected? (solved) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 18:49:56 -0000 Quoting Alexander Leidinger (from Mon, 04 Feb 2008 09:21:50 +0100): > So anybody out there who can give me some information I need to do > this, or at least knows some places in the zfs source I should have a > look at? Please CC me, as I'm not subscribed to freebsd-fs@. After digging around in the ZFS sources and investigating some on-disk structures (with some confusing results, as the data didn't seem corrupt to me), I exported the pool. After that all disk where again in the online state. All are connected via firewire and came back in a different order (daX) after a reboot (panic). It seems our implementation wasn't able to handle this without the reimport help. Bye, Alexander. -- You will be run over by a bus. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 18:51:21 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 25EEE16A420 for ; Mon, 4 Feb 2008 18:51:21 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outJ.internet-mail-service.net (outJ.internet-mail-service.net [216.240.47.233]) by mx1.freebsd.org (Postfix) with ESMTP id 0450913C47E for ; Mon, 4 Feb 2008 18:51:20 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Mon, 04 Feb 2008 10:36:55 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id 315561270B2; Mon, 4 Feb 2008 10:36:55 -0800 (PST) Message-ID: <47A75B47.2040604@elischer.org> Date: Mon, 04 Feb 2008 10:36:55 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Andriy Gapon References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> In-Reply-To: <47A735A4.3060506@icyb.net.ua> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Bruce Evans , freebsd-hackers@freebsd.org, scottl@FreeBSD.org, freebsd-fs@freebsd.org, Pav Lucistnik , Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 18:51:21 -0000 Andriy Gapon wrote: > More on the problem with reading big directories on UDF. You do realise that you have now made yourself the official maintainer of the UDF file system by submitting a competent and insightful analysis of the problem? > > First, some sleuthing. I came to believe that the problem is caused by > some larger change in vfs/vm/buf area. It seems that now VMIO is applied > to more vnode types than before. In particular it seems that now vnodes > for devices have non-NULL v_object (or maybe this is about directory > vnodes, I am not sure). > > Also it seems that the problem should happen for any directory with size > larger than four 2048-bytes sectors (I think that any directory with > > 300 files would definitely qualify). > > After some code reading and run-time debugging, here are some facts > about udf directory reading: > 1. bread-ing is done via device vnode (as opposed to directory vnodes), > as a consequence udf_strategy is not involved in directory reading > 2. the device vnode has a non-NULL v_object, this means that VMIO is used > 3. it seems that the code assumed that non-VM buffers are used > 4. bread is done on as much data as possible, up to MAXBSIZE [and even > slightly more in some cases] (i.e. code determines directory data size > and attempts to read in everything in one go) > 5. physical sector number adjusted to DEV_BSIZE (512 bytes) sectors is > passed to bread - this is because of #1 above and peculiarities of buf > system > 6. directory reading and file reading are quite different, because the > latter does reading via file vnode > > Let's a consider a simple case of "directory reading" 5 * PAGE_SIZE (4K) > bytes starting from a physical sector N with N%2 = 0 from media with > sector size of 2K (most common CD/DVD case): > 1. we want to read 12 KB = 3 pages = 6 sectors starting from sector N, > N%2 = 0 > 2. 4*N is passed as a sector number to bread > 3. bo_bsize of the corresponding bufobj is a media sector size, i.e. 2K > 4. actual read in bread will happen from b_iooffset of 4*N*DEV_BSIZE = > N*2K, i.e. correct offset within the device > 5. for a fresh read getblk will be called > 6. getblk will set b_offset to blkno*bo_bsize=4*N*2K, i.e. 4 times the > correct byte offset within the device > 7. allocbuf will allocate pages using B_VMIO case > 8. allocbuf calculates base page index as follows: pi = b_offset/PAGE_SIZE > this means the following: > sectors N and N+1 will be bound to a page with index 4*N*2K/4K = 2*N > sectors N+2 and N+3 will be bound to the next page 2*N +1 > sectors N+4 and N+5 is tied to the next page 2*N + 2 > > Now let's consider a "directory read" of 2 sectors (1 page) starting at > physical sector N+1. > Using the same calculations as above, we see that the sector will be > tied to a page 2*(N+1) = 2*N + 2. > And this page already contains valid cached data from the previous read, > *but* it contains data from sectors N+4 and N+5 instead of N+1 and N+2. > > So, we see, that because of b_offset being 4 times the actual byte > offset, we get incorrect page indexing. Namely, sector X gets associated > with different pages depending on what sector is used as a starting > sector for bread. If bread starts at sector N, then data of sector N+1 > is associated with (second half of) page with index 2*N; but if bread > starts at sector N+1 then it gets associated with (the first half of) > page with index 2*(N+1). > > Previously, before VMIO change, data for all reads was put into separate > buffers that did not share anything between them. So the problem was > limited only to wasting memory with duplicate data, but no actual > over-runs did happen. Now we have the over-runs because the VM pages are > shared between the buffers of the same vnode. > > One obvious solution is to limit bread size to 2*PAGE_SIZE = 4 * > sector_size. In this case, as before, we would waste some memory on > duplicate data but we would avoid page overruns. If we limit bread size > even more, to 1 sector, then we would not have any duplicate data at > all. But there would still be some resource waste - each page would > correspond to one sector, so 4K page would have only 2K of valid data > and the other half in each page is unused. > > Another solution, which to me seems to be better, is to do "usual" > reading - pass a directory vnode and 2048-byte sector offset to bread. > In this case udf_strategy will get called for bklno translation, all > offsets and indexes will be correct and everything will work perfectly > as it is the case for all other filesystems. > The only overhead in this case comes from the need to handle > UDF_INVALID_BMAP case (where data is embedded into file entry). So it > means that we have to call bmap_internal to see if we have that special > case and hanlde it, and if the case is not special we would call bread > on a directory vnode which means that bmap_internal would be called > again in udf_strategy. > One optimization that can be done in this case is to create a ligher > version of bmap_internal that would merely check for the special case > and wouldn't do anything else. > > I am attaching a patch based on the ideas above. It fixes the problem > for me and doesn't seem to create any new ones :-) > About the patch: > hunk #1 fixes a copy/paste > hunk #2 should fixes strategy to not set junk b_blkno in case of > udf_bmap_internal failure and also replaces divisions/multiplications > with bit-shifts > hunk #3 really limits max size in bread to MAXBSIZE and also makes bread > to be done a directory vnode > > BTW, maybe it's a good idea to further limit size of bread() anyway. > Like always reading 1 sector. Anyway, this should make any difference > only in minority of cases and I am not sure about performance effects (I > do not expect difference in anything else other than performance). > > on 01/02/2008 12:27 Andriy Gapon said the following: >> on 01/02/2008 12:00 Andriy Gapon said the following: >>> ---- a different, under-debugged problem ----- >>> BTW, on some smaller directories (but still large ones) I get some very >>> strange problems with reading a directory too. It seems like some bad >>> interaction between udf and buffer cache system. I added a lot of >>> debugging prints and the problems looks like the following: >>> >>> read starting at physical sector (2048-byte one) N, size is ~20K, N%4=0 >>> bread(4 * N, some_big_size) >>> correct data is read >>> repeat the above couple dozen times >>> read starting at physical sector (2048-byte one) N+1, size is ~20K >>> bread(4 * (N+1), some_big_size) >>> data is read from physical sector N+4 (instead of N+1) >>> >>> I remember that Bruce Evance warned me that something like this could >>> happen but I couldn't understand him, because I don't understand >>> VM/buffer subsystem. I'll try to dig up the email. >>> >> Sorry for the flood - additional info: >> if I limit max read size in udf_readatoffset() to 2048 (instead of >> MAXBSIZE), then large directories can be read OK. Seems like something >> with overlapping buffers, maybe? >> >> BTW, here's how I created test environment for the described issues (in >> tcsh): >> >> mkdir /tmp/bigdir >> >> cd /tmp/bigdir >> >> set i=1 >> >> while ($i < NNNNN) >> touch file.$i >> set i=`expr $i + 1` >> end >> >> cd /tmp >> >> mkisofs -udf -o test.iso bigdir >> >> mdconfig -a -t vnode -f test.iso -S 2048 -u 0 >> >> mount_udf /dev/md0 /mnt >> >> ls -l /mnt >> > > > > ------------------------------------------------------------------------ > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 20:35:16 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1671516A46E for ; Mon, 4 Feb 2008 20:35:16 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from raven.customer.vol.cz (raven.customer.vol.cz [195.250.144.108]) by mx1.freebsd.org (Postfix) with ESMTP id 9D8CD13C4E9 for ; Mon, 4 Feb 2008 20:35:14 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from [192.168.0.23] (rb5dg130.net.upc.cz [89.176.238.130]) (authenticated bits=0) by raven.customer.vol.cz (8.14.1/8.14.1) with ESMTP id m14K7hQE000556; Mon, 4 Feb 2008 21:07:46 +0100 (CET) (envelope-from pav@FreeBSD.org) From: Pav Lucistnik To: Julian Elischer In-Reply-To: <47A75B47.2040604@elischer.org> References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> Content-Type: text/plain; charset=ISO-8859-2 Date: Mon, 04 Feb 2008 21:07:43 +0100 Message-Id: <1202155663.62432.0.camel@ikaros.oook.cz> Mime-Version: 1.0 X-Mailer: Evolution 2.12.3 FreeBSD GNOME Team Port Content-Transfer-Encoding: 8bit X-Spam-Score: -2.548 () AWL,BAYES_00 X-Scanned-By: MIMEDefang 2.63 on 195.250.144.108 X-Milter: Spamilter (Reciever: raven.customer.vol.cz; Sender-ip: 89.176.238.130; Sender-helo: [192.168.0.23]; ) Cc: Bruce Evans , freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, Andriy Gapon , freebsd-fs@FreeBSD.org, Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pav@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 20:35:16 -0000 Julian Elischer pí¹e v po 04. 02. 2008 v 10:36 -0800: > Andriy Gapon wrote: > > More on the problem with reading big directories on UDF. > > You do realise that you have now made yourself the official > maintainer of the UDF file system by submitting a competent > and insightful analysis of the problem? Yay, and can you fix the sequential read performance while you're at it? Kthx! -- Pav Lucistnik Squish. Larger than the normal icky things, and twice as icky. From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 20:58:04 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 19E5916A418 for ; Mon, 4 Feb 2008 20:58:04 +0000 (UTC) (envelope-from cracauer@koef.zs64.net) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by mx1.freebsd.org (Postfix) with ESMTP id A6EB213C448 for ; Mon, 4 Feb 2008 20:58:03 +0000 (UTC) (envelope-from cracauer@koef.zs64.net) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by koef.zs64.net (8.14.2/8.14.2) with ESMTP id m14Kw1Qx007620; Mon, 4 Feb 2008 21:58:01 +0100 (CET) (envelope-from cracauer@koef.zs64.net) Received: (from cracauer@localhost) by koef.zs64.net (8.14.2/8.14.2/Submit) id m14Kw1Z2007619; Mon, 4 Feb 2008 15:58:01 -0500 (EST) (envelope-from cracauer) Date: Mon, 4 Feb 2008 15:58:01 -0500 From: Martin Cracauer To: "Julian H. Stacey" Message-ID: <20080204205801.GA7398@cons.org> References: <20080201172214.GA55957@cons.org> <200802021916.m12JGUjN049706@fire.js.berklix.net> <20080204163308.GA96092@cons.org> <200802041727.m14HREuN049123@fire.js.berklix.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200802041727.m14HREuN049123@fire.js.berklix.net> User-Agent: Mutt/1.4.2.3i Cc: freebsd-fs@freebsd.org, Martin Cracauer Subject: Re: fsck and mount disagree on whether superblocks are usable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 20:58:04 -0000 Julian H. Stacey wrote on Mon, Feb 04, 2008 at 06:27:14PM +0100: > Martin Cracauer wrote: > > Julian H. Stacey wrote on Sat, Feb 02, 2008 at 08:16:30PM +0100: > > > Martin Cracauer wrote: > > > > This is not an emergency but I find it odd. Mount and fsck agree on > > > > whether superblocks are usable. Mount can mount readonly, but fsck > > > > can use neither the primary superblock nor the alternatives. > > > > > > > > 32 is not a file system superblock > > > > > > Just in case, You know secondary block on newer FSs moved from 32 ? > > > Ref man fsck_ufs > > > -b Use the block specified immediately after the flag as the super > > > block for the file system. An alternate super block is usually > > > located at block 32 for UFS1, and block 160 for UFS2. > > > > Thanks, Julian. > > > > I'm honestly don't know how to tell whether I have ufs1 or ufs2. > > I didnt either, but wanted to know & just found one way: > > dumpfs /dev/____ | grep -i ufs Yupp, there you go. The reason why it failed for me is that it was looking for the superblocks in the wrong place. This works: fsck_ffs -b 160 /dev/ad0s1a I now need to debug why the target machine's fsck seemed to think it's ufs1 or why else it looked at 32 when the source machine didn't. Martin -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer http://www.cons.org/cracauer/ FreeBSD - where you want to go, today. http://www.freebsd.org/ From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 21:03:06 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9EFFD16A420 for ; Mon, 4 Feb 2008 21:03:06 +0000 (UTC) (envelope-from jhs@berklix.org) Received: from tower.berklix.org (tower.berklix.org [83.236.223.114]) by mx1.freebsd.org (Postfix) with ESMTP id 0D8CA13C465 for ; Mon, 4 Feb 2008 21:03:05 +0000 (UTC) (envelope-from jhs@berklix.org) Received: from js.berklix.net (p549A7E7E.dip.t-dialin.net [84.154.126.126]) (authenticated bits=0) by tower.berklix.org (8.13.6/8.13.6) with ESMTP id m14L33gQ097040; Mon, 4 Feb 2008 21:03:03 GMT (envelope-from jhs@berklix.org) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by js.berklix.net (8.13.8/8.13.8) with ESMTP id m14L4vu9025875; Mon, 4 Feb 2008 22:04:57 +0100 (CET) (envelope-from jhs@berklix.org) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.13.8/8.13.8) with ESMTP id m14L4lep052239; Mon, 4 Feb 2008 22:04:52 +0100 (CET) (envelope-from jhs@fire.js.berklix.net) Message-Id: <200802042104.m14L4lep052239@fire.js.berklix.net> To: Martin Cracauer In-reply-to: <20080204205801.GA7398@cons.org> References: <20080201172214.GA55957@cons.org> <200802021916.m12JGUjN049706@fire.js.berklix.net> <20080204163308.GA96092@cons.org> <200802041727.m14HREuN049123@fire.js.berklix.net> <20080204205801.GA7398@cons.org> Comments: In-reply-to Martin Cracauer message dated "Mon, 04 Feb 2008 15:58:01 -0500." Date: Mon, 04 Feb 2008 22:04:47 +0100 From: "Julian H. Stacey" Cc: freebsd-fs@freebsd.org Subject: Re: fsck and mount disagree on whether superblocks are usable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 21:03:06 -0000 Martin Cracauer wrote: > Julian H. Stacey wrote on Mon, Feb 04, 2008 at 06:27:14PM +0100: > > Martin Cracauer wrote: > > > Julian H. Stacey wrote on Sat, Feb 02, 2008 at 08:16:30PM +0100: > > > > Martin Cracauer wrote: > > > > > This is not an emergency but I find it odd. Mount and fsck agree on > > > > > whether superblocks are usable. Mount can mount readonly, but fsck > > > > > can use neither the primary superblock nor the alternatives. > > > > > > > > > > 32 is not a file system superblock > > > > > > > > Just in case, You know secondary block on newer FSs moved from 32 ? > > > > Ref man fsck_ufs > > > > -b Use the block specified immediately after the flag as the super > > > > block for the file system. An alternate super block is usually > > > > located at block 32 for UFS1, and block 160 for UFS2. > > > > > > Thanks, Julian. > > > > > > I'm honestly don't know how to tell whether I have ufs1 or ufs2. > > > > I didnt either, but wanted to know & just found one way: > > > > dumpfs /dev/____ | grep -i ufs > > Yupp, there you go. > The reason why it failed for me is that it was looking for the > superblocks in the wrong place. > > This works: > fsck_ffs -b 160 /dev/ad0s1a > > I now need to debug why the target machine's fsck seemed to think it's > ufs1 or why else it looked at 32 when the source machine didn't. Yup, always nice to understand whats going on/went on, but at some stage in your shoes, I'd copy all data to another place & then newfs & copy back, for peace of mind :-) -- Julian Stacey. BSD Unix Linux Net Consultant, Munich. http://berklix.com From owner-freebsd-fs@FreeBSD.ORG Mon Feb 4 22:41:05 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CBC0916A417 for ; Mon, 4 Feb 2008 22:41:05 +0000 (UTC) (envelope-from cracauer@koef.zs64.net) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by mx1.freebsd.org (Postfix) with ESMTP id 5FA8813C458 for ; Mon, 4 Feb 2008 22:41:04 +0000 (UTC) (envelope-from cracauer@koef.zs64.net) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by koef.zs64.net (8.14.2/8.14.2) with ESMTP id m14Mf2KL011501; Mon, 4 Feb 2008 23:41:02 +0100 (CET) (envelope-from cracauer@koef.zs64.net) Received: (from cracauer@localhost) by koef.zs64.net (8.14.2/8.14.2/Submit) id m14Mf29J011500; Mon, 4 Feb 2008 17:41:02 -0500 (EST) (envelope-from cracauer) Date: Mon, 4 Feb 2008 17:41:02 -0500 From: Martin Cracauer To: "Julian H. Stacey" Message-ID: <20080204224102.GA11286@cons.org> References: <20080201172214.GA55957@cons.org> <200802021916.m12JGUjN049706@fire.js.berklix.net> <20080204163308.GA96092@cons.org> <200802041727.m14HREuN049123@fire.js.berklix.net> <20080204205801.GA7398@cons.org> <200802042104.m14L4lep052239@fire.js.berklix.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200802042104.m14L4lep052239@fire.js.berklix.net> User-Agent: Mutt/1.4.2.3i Cc: freebsd-fs@freebsd.org, Martin Cracauer Subject: Re: fsck and mount disagree on whether superblocks are usable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2008 22:41:05 -0000 Julian H. Stacey wrote on Mon, Feb 04, 2008 at 10:04:47PM +0100: > Martin Cracauer wrote: > > Julian H. Stacey wrote on Mon, Feb 04, 2008 at 06:27:14PM +0100: > > > Martin Cracauer wrote: > > > > Julian H. Stacey wrote on Sat, Feb 02, 2008 at 08:16:30PM +0100: > > > > > Martin Cracauer wrote: > > > > > > This is not an emergency but I find it odd. Mount and fsck agree on > > > > > > whether superblocks are usable. Mount can mount readonly, but fsck > > > > > > can use neither the primary superblock nor the alternatives. > > > > > > > > > > > > 32 is not a file system superblock > > > > > > > > > > Just in case, You know secondary block on newer FSs moved from 32 ? > > > > > Ref man fsck_ufs > > > > > -b Use the block specified immediately after the flag as the super > > > > > block for the file system. An alternate super block is usually > > > > > located at block 32 for UFS1, and block 160 for UFS2. > > > > > > > > Thanks, Julian. > > > > > > > > I'm honestly don't know how to tell whether I have ufs1 or ufs2. > > > > > > I didnt either, but wanted to know & just found one way: > > > > > > dumpfs /dev/____ | grep -i ufs > > > > Yupp, there you go. > > > The reason why it failed for me is that it was looking for the > > superblocks in the wrong place. > > > > This works: > > fsck_ffs -b 160 /dev/ad0s1a > > > > I now need to debug why the target machine's fsck seemed to think it's > > ufs1 or why else it looked at 32 when the source machine didn't. > > Yup, always nice to understand whats going on/went on, but at some > stage in your shoes, I'd copy all data to another place & then newfs > & copy back, for peace of mind :-) Why do you say that? `df -i` shows I only lost about 50,000 files :-) ~(grisu)1% df -i / Filesystem 1024-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad0s1a 137150084 91580772 34597306 73% 976369 16758285 6% / ~(wings)11# df -i /mnt/tmp Filesystem 1024-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad0s1a 137150084 88153916 38024162 70% 932371 16802283 5% /mnt/tmp I still want to find out what's going on here. The disk geometry as reported by both fdisk and disklabel is identical, so that's not it. I don't see the reason why fsck should get confused here and I think that people might get bitten by this phaenomen after doing something less insane than I did. Basically, a 7-stable fsck destroyed a 6-stable filesystem here in a situation where it might or might not be justified. Martin -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer http://www.cons.org/cracauer/ FreeBSD - where you want to go, today. http://www.freebsd.org/ From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 12:00:24 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 397F416A47F for ; Tue, 5 Feb 2008 12:00:24 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id EE0C513C4E1 for ; Tue, 5 Feb 2008 12:00:23 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 48F682087; Tue, 5 Feb 2008 13:00:15 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.3/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 0CDEF2085; Tue, 5 Feb 2008 13:00:15 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id DFD88844B3; Tue, 5 Feb 2008 13:00:14 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Joe Peterson References: <47A73C8D.3000107@skyrush.com> Date: Tue, 05 Feb 2008 13:00:14 +0100 In-Reply-To: <47A73C8D.3000107@skyrush.com> (Joe Peterson's message of "Mon\, 04 Feb 2008 09\:25\:49 -0700") Message-ID: <86prvby5o1.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 12:00:24 -0000 Joe Peterson writes: > When I try to copy the file, I get only 655360 bytes copied, and then the= copy > stops. I assume this is because the next block is where the error is. Try to lseek past it. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 12:08:44 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3753E16A419; Tue, 5 Feb 2008 12:08:44 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id EAFB413C45A; Tue, 5 Feb 2008 12:08:43 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 864D42084; Tue, 5 Feb 2008 13:08:35 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.3/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 7587F2082; Tue, 5 Feb 2008 13:08:35 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 5A572844A6; Tue, 5 Feb 2008 13:08:35 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Alexander Leidinger References: <20080204092150.vprsphymqoog8cw4@webmail.leidinger.net> <20080204194936.thzddva9a8s4cwsg@webmail.leidinger.net> Date: Tue, 05 Feb 2008 13:08:35 +0100 In-Reply-To: <20080204194936.thzddva9a8s4cwsg@webmail.leidinger.net> (Alexander Leidinger's message of "Mon\, 04 Feb 2008 19\:49\:36 +0100") Message-ID: <86lk5zy5a4.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, pjd@freebsd.org Subject: Re: ZFS: invalid label -- what is expected? (solved) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 12:08:44 -0000 Alexander Leidinger writes: > After digging around in the ZFS sources and investigating some on-disk > structures (with some confusing results, as the data didn't seem > corrupt to me), I exported the pool. After that all disk where again > in the online state. All are connected via firewire and came back in a > different order (daX) after a reboot (panic). It seems our > implementation wasn't able to handle this without the reimport help. IIRC, it works fine for ATA devices but not for CAM devices (which include SCSI, SAS, USB, Firewire). I'm not sure whether pjd@ is working on that, but when creating a new pool, you can save yourself some trouble down the road by labeling the disks. You can also fix an existing pool by replacing each disk one by one with larger labeled disks - they must be larger, since the label will consume some space and ZFS will refuse to replace a disk in a raidz pool with one that is smaller than the smallest pre-existing disk. As a bonus, you will end up with a larger pool than you started out with :) I've been toying with the idea of writing a gnop-like geom that allows a disk to be referenced by its serial number if the underlying driver is able to supply it. That would bypass glabel's disk-shrinking issue when working on whole disks. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 13:11:26 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9AF6B16A418 for ; Tue, 5 Feb 2008 13:11:26 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id 41D9B13C46B for ; Tue, 5 Feb 2008 13:11:26 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 19DEE45E94; Tue, 5 Feb 2008 13:46:53 +0100 (CET) Received: from localhost (pjd.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 2708045E91; Tue, 5 Feb 2008 13:46:48 +0100 (CET) Date: Tue, 5 Feb 2008 13:46:26 +0100 From: Pawel Jakub Dawidek To: Dag-Erling Sm??rgrav Message-ID: <20080205124626.GB95316@garage.freebsd.pl> References: <20080204092150.vprsphymqoog8cw4@webmail.leidinger.net> <20080204194936.thzddva9a8s4cwsg@webmail.leidinger.net> <86lk5zy5a4.fsf@ds4.des.no> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MfFXiAuoTsnnDAfZ" Content-Disposition: inline In-Reply-To: <86lk5zy5a4.fsf@ds4.des.no> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 8.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=3.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org, Alexander Leidinger Subject: Re: ZFS: invalid label -- what is expected? (solved) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 13:11:26 -0000 --MfFXiAuoTsnnDAfZ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Feb 05, 2008 at 01:08:35PM +0100, Dag-Erling Sm??rgrav wrote: > Alexander Leidinger writes: > > After digging around in the ZFS sources and investigating some on-disk > > structures (with some confusing results, as the data didn't seem > > corrupt to me), I exported the pool. After that all disk where again > > in the online state. All are connected via firewire and came back in a > > different order (daX) after a reboot (panic). It seems our > > implementation wasn't able to handle this without the reimport help. >=20 > IIRC, it works fine for ATA devices but not for CAM devices (which > include SCSI, SAS, USB, Firewire). I'm not sure whether pjd@ is working > on that, but when creating a new pool, you can save yourself some > trouble down the road by labeling the disks. It is properly done in perforce already. If component can't be found, ZFS tries to look it up by checking all GEOM providers one by one. This should solve all those issues and doesn't rely on serial numbers which are not always there. > You can also fix an existing pool by replacing each disk one by one with > larger labeled disks - they must be larger, since the label will consume > some space and ZFS will refuse to replace a disk in a raidz pool with > one that is smaller than the smallest pre-existing disk. As a bonus, > you will end up with a larger pool than you started out with :) >=20 > I've been toying with the idea of writing a gnop-like geom that allows a > disk to be referenced by its serial number if the underlying driver is > able to supply it. That would bypass glabel's disk-shrinking issue when > working on whole disks. Been there, done that. You can probably find discussion about this in the archives. The consensus (well, not mine) was that it is not good idea. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --MfFXiAuoTsnnDAfZ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHqFqgForvXbEpPzQRAgjWAKDWyvIR5dvFwgwpa89l3HJmHaBTZACgsffH K5GYr7PsVO5+SPUWM8kwZWc= =qlgN -----END PGP SIGNATURE----- --MfFXiAuoTsnnDAfZ-- From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 13:31:21 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0070316A418 for ; Tue, 5 Feb 2008 13:31:20 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from shadow.wildlava.net (shadow.wildlava.net [67.40.138.81]) by mx1.freebsd.org (Postfix) with ESMTP id C8F2413C4CE for ; Tue, 5 Feb 2008 13:31:20 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from [64.134.205.78] (dhcp64-134-205-78.lman.aus.wayport.net [64.134.205.78]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by shadow.wildlava.net (Postfix) with ESMTP id A12198F41F; Tue, 5 Feb 2008 06:31:19 -0700 (MST) Message-ID: <47A864D9.4060504@skyrush.com> Date: Tue, 05 Feb 2008 06:30:01 -0700 From: Joe Peterson User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: =?UTF-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= References: <47A73C8D.3000107@skyrush.com> <86prvby5o1.fsf@ds4.des.no> In-Reply-To: <86prvby5o1.fsf@ds4.des.no> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 13:31:21 -0000 Dag-Erling Smørgrav wrote: > Joe Peterson writes: >> When I try to copy the file, I get only 655360 bytes copied, and then the copy >> stops. I assume this is because the next block is where the error is. > > Try to lseek past it. Well, I'd like to actually read the "bad" data too, so I can see if it is really bad or if there is a metadata issue. Basically, I'd like to recover all the file's bytes this once without having ZFS stop me due to the checksum failure, just for debugging purposes. Is this impossible in ZFS? -Thanks, Joe From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 13:38:28 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 919E316A421 for ; Tue, 5 Feb 2008 13:38:28 +0000 (UTC) (envelope-from freebsd-stable@tychl.net) Received: from mail.tychl.net (masq.tychl.net [IPv6:2001:470:1f01:716::1]) by mx1.freebsd.org (Postfix) with ESMTP id 687D213C4CC for ; Tue, 5 Feb 2008 13:38:28 +0000 (UTC) (envelope-from freebsd-stable@tychl.net) Received: from localhost (localhost [127.0.0.1]) by mail.tychl.net (Postfix) with ESMTP id BC3531CB4C; Tue, 5 Feb 2008 08:38:27 -0500 (EST) X-Virus-Scanned: amavisd-new at tychl.net Received: from mail.tychl.net ([192.168.0.2]) by localhost (masq.tychl.net [127.0.0.1]) (amavisd-new, port 10024) with SMTP id lESaR6pb77Qf; Tue, 5 Feb 2008 08:38:25 -0500 (EST) Received: from [127.0.0.1] (unknown [172.16.1.94]) by mail.tychl.net (Postfix) with ESMTP id 9662A1CB48; Tue, 5 Feb 2008 08:38:25 -0500 (EST) Message-ID: <47A866D1.3040703@tychl.net> Date: Tue, 05 Feb 2008 08:38:25 -0500 From: Nick Gustas User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Joe Peterson References: <47A73C8D.3000107@skyrush.com> <86prvby5o1.fsf@ds4.des.no> <47A864D9.4060504@skyrush.com> In-Reply-To: <47A864D9.4060504@skyrush.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 13:38:28 -0000 Joe Peterson wrote: > Dag-Erling Sm=C3=B8rgrav wrote: > =20 >> Joe Peterson writes: >> =20 >>> When I try to copy the file, I get only 655360 bytes copied, and then= the copy >>> stops. I assume this is because the next block is where the error is= . >>> =20 >> Try to lseek past it. >> =20 > > Well, I'd like to actually read the "bad" data too, so I can see if it = is > really bad or if there is a metadata issue. Basically, I'd like to rec= over > all the file's bytes this once without having ZFS stop me due to the ch= ecksum > failure, just for debugging purposes. Is this impossible in ZFS? > > -Thanks, Joe > =20 This may do what you want, but I'm not sure if this only disables=20 creation of checksums, or it disables use of preexisting checksums=20 entirely. The man page entry would suggest it disables them for reads=20 too. If this is the case, it should do what you want. http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Chec= ksums Verify the type of checksum used: zfs get checksum Tuning is achieved dynamically by using: zfs set checksum=3Doff And reverted: zfs set checksum=3D'on | fletcher2 | fletcher4 | sha256' Fletcher2 checksum (the default) has been observed to consume roughly=20 1Ghz of a CPU when checksumming 500 MByte per second. Man page: checksum=3Don | off | fletcher2, | fletcher4 | sha256 Controls the checksum used to verify data integrity. The=20 default value is "on", which automatically selects an appropriate=20 algorithm (currently, fletcher2, but this may change in future=20 releases). The value "off" disables integrity checking on user data. =20 Disabling checksums is NOT a recommended practice. From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 14:10:01 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE8E516A417; Tue, 5 Feb 2008 14:10:01 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 69EA313C4D1; Tue, 5 Feb 2008 14:10:01 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 2BC7A2087; Tue, 5 Feb 2008 15:09:53 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.3/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 1C2F62084; Tue, 5 Feb 2008 15:09:53 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 01F03844A6; Tue, 5 Feb 2008 15:09:52 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Pawel Jakub Dawidek References: <20080204092150.vprsphymqoog8cw4@webmail.leidinger.net> <20080204194936.thzddva9a8s4cwsg@webmail.leidinger.net> <86lk5zy5a4.fsf@ds4.des.no> <20080205124626.GB95316@garage.freebsd.pl> Date: Tue, 05 Feb 2008 15:09:52 +0100 In-Reply-To: <20080205124626.GB95316@garage.freebsd.pl> (Pawel Jakub Dawidek's message of "Tue\, 5 Feb 2008 13\:46\:26 +0100") Message-ID: <86d4rbxznz.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, Alexander Leidinger Subject: Re: ZFS: invalid label -- what is expected? (solved) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 14:10:01 -0000 Pawel Jakub Dawidek writes: > Dag-Erling Sm=C3=B8rgrav writes: > > I've been toying with the idea of writing a gnop-like geom that allows a > > disk to be referenced by its serial number if the underlying driver is > > able to supply it. That would bypass glabel's disk-shrinking issue when > > working on whole disks. > Been there, done that. You can probably find discussion about this in > the archives. The consensus (well, not mine) was that it is not good > idea. Those who think it's a bad idea are free to not use it, while The Rest Of Us [tm] reap its benefits :) DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 14:17:37 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9D29B16A421 for ; Tue, 5 Feb 2008 14:17:37 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from shadow.wildlava.net (shadow.wildlava.net [67.40.138.81]) by mx1.freebsd.org (Postfix) with ESMTP id 6F09E13C4F2 for ; Tue, 5 Feb 2008 14:17:37 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from [64.134.205.78] (dhcp64-134-205-78.lman.aus.wayport.net [64.134.205.78]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by shadow.wildlava.net (Postfix) with ESMTP id 4B9998F3EB; Tue, 5 Feb 2008 07:17:36 -0700 (MST) Message-ID: <47A86FB3.2060204@skyrush.com> Date: Tue, 05 Feb 2008 07:16:19 -0700 From: Joe Peterson User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Nick Gustas References: <47A73C8D.3000107@skyrush.com> <86prvby5o1.fsf@ds4.des.no> <47A864D9.4060504@skyrush.com> <47A866D1.3040703@tychl.net> In-Reply-To: <47A866D1.3040703@tychl.net> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 14:17:37 -0000 Nick Gustas wrote: >> Well, I'd like to actually read the "bad" data too, so I can see if it is >> really bad or if there is a metadata issue. Basically, I'd like to recover >> all the file's bytes this once without having ZFS stop me due to the checksum >> failure, just for debugging purposes. Is this impossible in ZFS? > This may do what you want, but I'm not sure if this only disables > creation of checksums, or it disables use of preexisting checksums > entirely. The man page entry would suggest it disables them for reads > too. If this is the case, it should do what you want. > > http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Checksums Unfortunately, no - I tried this already, and it has no effect. I assume it only determines whether the checksums are created, not whether checked if they already exist. Thanks, Joe From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 14:19:21 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9347C16A41A for ; Tue, 5 Feb 2008 14:19:21 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 4FFFC13C4EA for ; Tue, 5 Feb 2008 14:19:21 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 92816208E; Tue, 5 Feb 2008 15:19:12 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.3/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 782D62088; Tue, 5 Feb 2008 15:19:12 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 5F18D844A6; Tue, 5 Feb 2008 15:19:12 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Joe Peterson References: <47A73C8D.3000107@skyrush.com> <86prvby5o1.fsf@ds4.des.no> <47A864D9.4060504@skyrush.com> Date: Tue, 05 Feb 2008 15:19:12 +0100 In-Reply-To: <47A864D9.4060504@skyrush.com> (Joe Peterson's message of "Tue\, 05 Feb 2008 06\:30\:01 -0700") Message-ID: <864pcnxz8f.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 14:19:21 -0000 Joe Peterson writes: > Dag-Erling Sm=C3=B8rgrav writes: > > Joe Peterson writes: > > > When I try to copy the file, I get only 655360 bytes copied, and > > > then the copy stops. I assume this is because the next block is > > > where the error is. > > Try to lseek past it. > Well, I'd like to actually read the "bad" data too, so I can see if it is > really bad or if there is a metadata issue. Basically, I'd like to recov= er > all the file's bytes this once without having ZFS stop me due to the chec= ksum > failure, just for debugging purposes. Is this impossible in ZFS? There is now way to "read the bad data" since an unrecoverable checksum error means that ZFS has no idea which of the multiple version of the affected block is the right one. (I assume this was a raidz pool; if not, imagine Nelson Muntz from the Simpsons yelling "ha ha!" at you) My advice is to use 'dd conv=3Dnoerror' with a sufficiently small block size to recover what parts you can. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 14:23:59 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A64716A420 for ; Tue, 5 Feb 2008 14:23:59 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id AFA1513C4E3 for ; Tue, 5 Feb 2008 14:23:58 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1JMOiE-0000XV-Dw for freebsd-fs@freebsd.org; Tue, 05 Feb 2008 14:23:50 +0000 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 05 Feb 2008 14:23:50 +0000 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 05 Feb 2008 14:23:50 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Tue, 05 Feb 2008 15:24:47 +0100 Lines: 36 Message-ID: References: <20080204092150.vprsphymqoog8cw4@webmail.leidinger.net> <20080204194936.thzddva9a8s4cwsg@webmail.leidinger.net> <86lk5zy5a4.fsf@ds4.des.no> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig7ACCADB04BF6C38D3FC989C0" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.6 (X11/20071022) In-Reply-To: <86lk5zy5a4.fsf@ds4.des.no> X-Enigmail-Version: 0.95.0 Sender: news Subject: Re: ZFS: invalid label -- what is expected? (solved) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 14:23:59 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig7ACCADB04BF6C38D3FC989C0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Dag-Erling Sm=C3=B8rgrav wrote: > I've been toying with the idea of writing a gnop-like geom that allows = a > disk to be referenced by its serial number if the underlying driver is > able to supply it. That would bypass glabel's disk-shrinking issue whe= n > working on whole disks. I agree with the idea, but why not extend glabel instead? It's already in GENERIC and I would (as an end-user) expect to find the support for this mode of usage in it :) This could lead to "interesting" situations - where a drive/slice has a UFS label, a disk ID label and (if pushed...) a "native glabel" label. --------------enig7ACCADB04BF6C38D3FC989C0 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHqHGvldnAQVacBcgRAmRVAKDKtK+7eKsloX5YWaNtQqSTbSNeBQCgg/Ll sUSYBlRHIuHqIChLUwIwObo= =jRWD -----END PGP SIGNATURE----- --------------enig7ACCADB04BF6C38D3FC989C0-- From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 14:40:17 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 43E4F16A498; Tue, 5 Feb 2008 14:40:17 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id A74CC13C4E8; Tue, 5 Feb 2008 14:40:16 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id C63B443DC32; Tue, 5 Feb 2008 16:40:15 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id F77-1TpguA4B; Tue, 5 Feb 2008 16:40:15 +0200 (EET) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 46AF743D33F; Tue, 5 Feb 2008 16:40:14 +0200 (EET) Message-ID: <47A8754C.5010607@icyb.net.ua> Date: Tue, 05 Feb 2008 16:40:12 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20080123) MIME-Version: 1.0 To: pav@FreeBSD.org References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> <1202155663.62432.0.camel@ikaros.oook.cz> In-Reply-To: <1202155663.62432.0.camel@ikaros.oook.cz> Content-Type: multipart/mixed; boundary="------------060803080903090508090302" Cc: Bruce Evans , freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, freebsd-fs@FreeBSD.org, Julian Elischer , Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 14:40:17 -0000 This is a multi-part message in MIME format. --------------060803080903090508090302 Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: 8bit on 04/02/2008 22:07 Pav Lucistnik said the following: > Julian Elischer pí¹e v po 04. 02. 2008 v 10:36 -0800: >> Andriy Gapon wrote: >>> More on the problem with reading big directories on UDF. >> You do realise that you have now made yourself the official >> maintainer of the UDF file system by submitting a competent >> and insightful analysis of the problem? > > Yay, and can you fix the sequential read performance while you're at it? > Kthx! > Pav, this was almost trivial :-) See the attached patch, first hunk is just for consistency. The code was borrowed from cd9660, only field/variable names are adjusted. But there is another issue that I also mentioned in the email about directory reading. It is UDF_INVALID_BMAP case of udf_bmap_internal, i.e. the case when file data is embedded into a file entry. This is a special case that needs to be handled differently. udf_readatoffset() handles it, but the latest udf_read code doesn't. I have a real UDF filesystem where this type of allocation is used for small files and those files can not be read. This is described in Part 4, section 14.6.8 of ECMA-167. -- Andriy Gapon --------------060803080903090508090302 Content-Type: text/x-patch; name="udf_ra.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="udf_ra.patch" --- udf_vnops.c.orig 2008-01-29 23:50:49.000000000 +0200 +++ udf_vnops.c 2008-02-05 01:30:23.000000000 +0200 @@ -851,7 +846,7 @@ udf_bmap(struct vop_bmap_args *a) if (a->a_runb) *a->a_runb = 0; - error = udf_bmap_internal(node, a->a_bn * node->udfmp->bsize, &lsector, + error = udf_bmap_internal(node, a->a_bn << node->udfmp->bshift, &lsector, &max_size); if (error) return (error); @@ -859,9 +854,27 @@ udf_bmap(struct vop_bmap_args *a) /* Translate logical to physical sector number */ *a->a_bnp = lsector << (node->udfmp->bshift - DEV_BSHIFT); - /* Punt on read-ahead for now */ - if (a->a_runp) - *a->a_runp = 0; + /* + * Determine maximum number of readahead blocks following the + * requested block. + */ + if (a->a_runp) { + off_t fsize; + int nblk; + + fsize = le64toh(node->fentry->inf_len); + nblk = (fsize >> node->udfmp->bshift) - (a->a_bn + 1); + if (nblk <= 0) + *a->a_runp = 0; + else if (nblk >= (MAXBSIZE >> node->udfmp->bshift)) + *a->a_runp = (MAXBSIZE >> node->udfmp->bshift) - 1; + else + *a->a_runp = nblk; + } + + if (a->a_runb) { + *a->a_runb = 0; + } return (0); } --- udf_vfsops.c.orig 2007-03-13 03:50:24.000000000 +0200 +++ udf_vfsops.c 2008-02-05 01:29:10.000000000 +0200 @@ -330,6 +330,11 @@ udf_mountfs(struct vnode *devvp, struct bo = &devvp->v_bufobj; + if (devvp->v_rdev->si_iosize_max != 0) + mp->mnt_iosize_max = devvp->v_rdev->si_iosize_max; + if (mp->mnt_iosize_max > MAXPHYS) + mp->mnt_iosize_max = MAXPHYS; + /* XXX: should be M_WAITOK */ MALLOC(udfmp, struct udf_mnt *, sizeof(struct udf_mnt), M_UDFMOUNT, M_NOWAIT | M_ZERO); --------------060803080903090508090302-- From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 14:52:45 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 021AE16A4CC; Tue, 5 Feb 2008 14:52:45 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id 9DEF213C459; Tue, 5 Feb 2008 14:52:44 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id 7FA8243F66F; Tue, 5 Feb 2008 16:52:43 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5ZvYvl+ATS2F; Tue, 5 Feb 2008 16:52:43 +0200 (EET) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 7A32C43F569; Tue, 5 Feb 2008 16:52:42 +0200 (EET) Message-ID: <47A87838.6060300@icyb.net.ua> Date: Tue, 05 Feb 2008 16:52:40 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20080123) MIME-Version: 1.0 To: Julian Elischer References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> In-Reply-To: <47A75B47.2040604@elischer.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Bruce Evans , freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, freebsd-fs@FreeBSD.org, Pav Lucistnik , Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 14:52:45 -0000 on 04/02/2008 20:36 Julian Elischer said the following: > Andriy Gapon wrote: >> More on the problem with reading big directories on UDF. > > You do realise that you have now made yourself the official > maintainer of the UDF file system by submitting a competent > and insightful analysis of the problem? I feel like I have to reply to this :-) First of all, thank you. But unfortunately my knowledge of core kernel is quite limited, I pick up some bits here and there by reading the code, but this is too little and sometimes my deductions are wrong. Another thing is very limited time. I do like fixing code, but I rarely have time and when I have it must be something very interesting and important to me. This time it is interesting :-) Because I actually use UDF quite a lot, I use DVD-RAM media for various archival and data transfer purposes, and sometimes there is some unusual case. I also try to keep up with FreeBSD development as much as possible, so sometimes there are new issues after upgrades. So, thank you again, but please think of me as of occasional patch submitter. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 16:13:33 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8811916A417 for ; Tue, 5 Feb 2008 16:13:33 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from shadow.wildlava.net (shadow.wildlava.net [67.40.138.81]) by mx1.freebsd.org (Postfix) with ESMTP id 5465413C447 for ; Tue, 5 Feb 2008 16:13:33 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from [129.162.240.95] (unknown [129.162.240.95]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by shadow.wildlava.net (Postfix) with ESMTP id DB8C68F424; Tue, 5 Feb 2008 09:13:31 -0700 (MST) Message-ID: <47A88ADE.7050503@skyrush.com> Date: Tue, 05 Feb 2008 09:12:14 -0700 From: Joe Peterson User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: =?UTF-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= References: <47A73C8D.3000107@skyrush.com> <86prvby5o1.fsf@ds4.des.no> <47A864D9.4060504@skyrush.com> <864pcnxz8f.fsf@ds4.des.no> In-Reply-To: <864pcnxz8f.fsf@ds4.des.no> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 16:13:33 -0000 Dag-Erling Smørgrav wrote: > There is now way to "read the bad data" since an unrecoverable checksum > error means that ZFS has no idea which of the multiple version of the > affected block is the right one. Nope, no mirror, no RAIDZ - just one partition. But as far as I know, there were no read errors, just a checksum error. I've also done a couple of surface scans of the drive, and no problems. So all I can imagine is that either data got "changed" on the disk (due to who know what), or the metadata got messed up (either hardware or some SW bug). I'd like to figure out what ZFS thinks the bytes in the file really are and why they are showing as a checksum error. So, since I only have one copy (i.e. no RAID/mirror), then I should be able to tell ZFS to "go ahead and read the bytes, not stopping when it hits the checksum mismatch). Then I could do an analysis of the data, compared to what the file should contain. I assume I can hack the ZFS source to "disable" stopping on the checksum problem, but I figured there might be some debug mode that would let me do this without delving into the code. > (I assume this was a raidz pool; if not, imagine Nelson Muntz from the > Simpsons yelling "ha ha!" at you) Ah, don't worry, I have backups (I'm just playing around with ZFS at the moment... :) > My advice is to use 'dd conv=noerror' with a sufficiently small block > size to recover what parts you can. I haven't lost anything, so no need to do that. I just want to see what's up with this particular ZFS issue. If it's a bug, at least I could submit it to Sun. Thanks, Joe From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 17:22:39 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 95B1A16A41A for ; Tue, 5 Feb 2008 17:22:39 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 4C8C013C461 for ; Tue, 5 Feb 2008 17:22:39 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id D38C02083; Tue, 5 Feb 2008 18:22:30 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.3/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id C61502082; Tue, 5 Feb 2008 18:22:30 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id AC4B4844B3; Tue, 5 Feb 2008 18:22:30 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Joe Peterson References: <47A73C8D.3000107@skyrush.com> <86prvby5o1.fsf@ds4.des.no> <47A864D9.4060504@skyrush.com> <864pcnxz8f.fsf@ds4.des.no> <47A88ADE.7050503@skyrush.com> Date: Tue, 05 Feb 2008 18:22:30 +0100 In-Reply-To: <47A88ADE.7050503@skyrush.com> (Joe Peterson's message of "Tue\, 05 Feb 2008 09\:12\:14 -0700") Message-ID: <86abmfwc6h.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 17:22:39 -0000 Joe Peterson writes: > Dag-Erling Sm=C3=B8rgrav writes: > > There is now way to "read the bad data" since an unrecoverable > > checksum error means that ZFS has no idea which of the multiple > > version of the affected block is the right one. > Nope, no mirror, no RAIDZ - just one partition. But as far as I know, th= ere > were no read errors, just a checksum error. A checksum error results from a read error. Check your drive's SMART error log if it has one. It might not be detectable in a surface scan, as the damaged sector will be automatically reassigned if it's written to (which ZFS may very well have done) DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 17:31:10 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D05AA16A41B for ; Tue, 5 Feb 2008 17:31:10 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (cl-162.ewr-01.us.sixxs.net [IPv6:2001:4830:1200:a1::2]) by mx1.freebsd.org (Postfix) with ESMTP id 38F5B13C455 for ; Tue, 5 Feb 2008 17:31:09 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.1/8.13.8) with ESMTP id m15HV2a8085869; Tue, 5 Feb 2008 11:31:03 -0600 (CST) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.1/8.13.8/Submit) id m15HV2dU085868; Tue, 5 Feb 2008 11:31:02 -0600 (CST) (envelope-from brooks) Date: Tue, 5 Feb 2008 11:31:02 -0600 From: Brooks Davis To: Dag-Erling Sm??rgrav Message-ID: <20080205173102.GA85735@lor.one-eyed-alien.net> References: <47A73C8D.3000107@skyrush.com> <86prvby5o1.fsf@ds4.des.no> <47A864D9.4060504@skyrush.com> <864pcnxz8f.fsf@ds4.des.no> <47A88ADE.7050503@skyrush.com> <86abmfwc6h.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6TrnltStXW4iwmi0" Content-Disposition: inline In-Reply-To: <86abmfwc6h.fsf@ds4.des.no> User-Agent: Mutt/1.5.16 (2007-06-09) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (lor.one-eyed-alien.net [127.0.0.1]); Tue, 05 Feb 2008 11:31:03 -0600 (CST) Cc: freebsd-fs@freebsd.org Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 17:31:10 -0000 --6TrnltStXW4iwmi0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Feb 05, 2008 at 06:22:30PM +0100, Dag-Erling Sm??rgrav wrote: > Joe Peterson writes: > > Dag-Erling Sm??rgrav writes: > > > There is now way to "read the bad data" since an unrecoverable > > > checksum error means that ZFS has no idea which of the multiple > > > version of the affected block is the right one. > > Nope, no mirror, no RAIDZ - just one partition. But as far as I know, = there > > were no read errors, just a checksum error. >=20 > A checksum error results from a read error. Check your drive's SMART > error log if it has one. It might not be detectable in a surface scan, > as the damaged sector will be automatically reassigned if it's written > to (which ZFS may very well have done) We've also experienced several situations were zfs was detecting corruption caused by bad cabling or bad controller firmware so SMART had nothing to report. -- Brooks --6TrnltStXW4iwmi0 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFHqJ1VXY6L6fI4GtQRAs8DAKCaaSkIJhcqe3WENQxt1qJsCjLdQwCfVNVq XsfLeRC8Mga3F+v3+vaaZrk= =29O6 -----END PGP SIGNATURE----- --6TrnltStXW4iwmi0-- From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 17:39:42 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8DE8816A417 for ; Tue, 5 Feb 2008 17:39:42 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from shadow.wildlava.net (shadow.wildlava.net [67.40.138.81]) by mx1.freebsd.org (Postfix) with ESMTP id 5865213C442 for ; Tue, 5 Feb 2008 17:39:42 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from [129.162.240.95] (unknown [129.162.240.95]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by shadow.wildlava.net (Postfix) with ESMTP id 28BD78F424; Tue, 5 Feb 2008 10:39:41 -0700 (MST) Message-ID: <47A89F0F.1030505@skyrush.com> Date: Tue, 05 Feb 2008 10:38:23 -0700 From: Joe Peterson User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: =?UTF-8?B?RGFnLUVybGluZyBTbcO4cmdyYXY=?= References: <47A73C8D.3000107@skyrush.com> <86prvby5o1.fsf@ds4.des.no> <47A864D9.4060504@skyrush.com> <864pcnxz8f.fsf@ds4.des.no> <47A88ADE.7050503@skyrush.com> <86abmfwc6h.fsf@ds4.des.no> In-Reply-To: <86abmfwc6h.fsf@ds4.des.no> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: freebsd-fs@freebsd.org Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 17:39:42 -0000 Dag-Erling Smørgrav wrote: > A checksum error results from a read error. Check your drive's SMART > error log if it has one. It might not be detectable in a surface scan, > as the damaged sector will be automatically reassigned if it's written > to (which ZFS may very well have done) I've checked SMART - no [unrecoverable] errors and no additional sector reallocations, and I've done a SeaTools long test - no problems found. But I do not understand: in zpool status, there are stats on read errors in addition to checksum errors. If I understand correctly, a read error would be the system/HW reporting an error on read, whereas the whole idea of the checksums in ZFS is to catch errors that are *not* reported as read errors (i.e. silent bit changes that normal filesystems would never catch). What I seem to be seeing is a case in which ZFS says the checksum is wrong. There are only counts in the CKSUM col, not the other cols in the status, so I do not think this is a "read error" - it is ZFS's last line of defense (the checksum) reporting a mismatch. In other words, I assume the read would complete if ZFS did not catch the checksum mismatch, and what I'd like to do is let it complete so I can see for myself where these bit errors are by comparing the read file to a known good copy (that I have). If there are no mismatches, it would mean there is a metadata error of ZFS bug. -Joe From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 18:16:23 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 126A516A419 for ; Tue, 5 Feb 2008 18:16:23 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from raven.customer.vol.cz (raven.customer.vol.cz [195.250.144.108]) by mx1.freebsd.org (Postfix) with ESMTP id 7059913C459 for ; Tue, 5 Feb 2008 18:16:22 +0000 (UTC) (envelope-from pav@FreeBSD.org) Received: from [192.168.0.23] (rb5dg130.net.upc.cz [89.176.238.130]) (authenticated bits=0) by raven.customer.vol.cz (8.14.1/8.14.1) with ESMTP id m15IG9Qh021467; Tue, 5 Feb 2008 19:16:11 +0100 (CET) (envelope-from pav@FreeBSD.org) From: Pav Lucistnik To: Andriy Gapon In-Reply-To: <47A8754C.5010607@icyb.net.ua> References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> <1202155663.62432.0.camel@ikaros.oook.cz> <47A8754C.5010607@icyb.net.ua> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-a0as8OlXkhwaCBWRWkrr" Date: Tue, 05 Feb 2008 19:16:08 +0100 Message-Id: <1202235368.68281.12.camel@ikaros.oook.cz> Mime-Version: 1.0 X-Mailer: Evolution 2.12.3 FreeBSD GNOME Team Port X-Spam-Score: -2.57 () AWL,BAYES_00 X-Scanned-By: MIMEDefang 2.63 on 195.250.144.108 X-Milter: Spamilter (Reciever: raven.customer.vol.cz; Sender-ip: 89.176.238.130; Sender-helo: [192.168.0.23]; ) Cc: Bruce Evans , freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, freebsd-fs@FreeBSD.org, Julian Elischer , Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pav@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 18:16:23 -0000 --=-a0as8OlXkhwaCBWRWkrr Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: quoted-printable Andriy Gapon p=ED=B9e v =FAt 05. 02. 2008 v 16:40 +0200: > > Yay, and can you fix the sequential read performance while you're at it= ? > > Kthx! > this was almost trivial :-) > See the attached patch, first hunk is just for consistency. > The code was borrowed from cd9660, only field/variable names are adjusted= . Omg looks good. Can't wait for it to bubble throught to 6-STABLE :) Thanks million for working on this. --=20 Pav Lucistnik What is the airspeed velocity of an unladen swallow? --=-a0as8OlXkhwaCBWRWkrr Content-Type: application/pgp-signature; name=signature.asc Content-Description: Toto je =?UTF-8?Q?digit=C3=A1ln=C4=9B?= =?ISO-8859-1?Q?_podepsan=E1?= =?UTF-8?Q?_=C4=8D=C3=A1st?= =?ISO-8859-1?Q?_zpr=E1vy?= -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEABECAAYFAkeop+UACgkQntdYP8FOsoJTEwCgnFMMuMEb3JCazBXQyIJvILD+ uJoAn0tWakjK81PqRferUhg2Z7fgOtsb =ysEA -----END PGP SIGNATURE----- --=-a0as8OlXkhwaCBWRWkrr-- From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 19:01:48 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45FDA16A46C for ; Tue, 5 Feb 2008 19:01:48 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from shadow.wildlava.net (shadow.wildlava.net [67.40.138.81]) by mx1.freebsd.org (Postfix) with ESMTP id 060C713C4E8 for ; Tue, 5 Feb 2008 19:01:47 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from [129.162.240.95] (unknown [129.162.240.95]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by shadow.wildlava.net (Postfix) with ESMTP id 7D73E8F429; Tue, 5 Feb 2008 12:01:46 -0700 (MST) Message-ID: <47A8B24D.9050904@skyrush.com> Date: Tue, 05 Feb 2008 12:00:29 -0700 From: Joe Peterson User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Brooks Davis References: <47A73C8D.3000107@skyrush.com> <86prvby5o1.fsf@ds4.des.no> <47A864D9.4060504@skyrush.com> <864pcnxz8f.fsf@ds4.des.no> <47A88ADE.7050503@skyrush.com> <86abmfwc6h.fsf@ds4.des.no> <20080205173102.GA85735@lor.one-eyed-alien.net> In-Reply-To: <20080205173102.GA85735@lor.one-eyed-alien.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Dag-Erling Sm??rgrav Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 19:01:48 -0000 Brooks Davis wrote: > We've also experienced several situations were zfs was detecting > corruption caused by bad cabling or bad controller firmware so SMART had > nothing to report. Were the errors sporadic/intermittent? I'd think that if it was cabling, the checksum error could happen some times and not others. Mine seems to be predictable and always there... Also, no error was reported when the file was written, and I think ZFS checks after writing. Strange. This is one reason why I'm very curious which bits are bad. -Joe From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 19:06:37 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B0C8716A41B; Tue, 5 Feb 2008 19:06:37 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (cl-162.ewr-01.us.sixxs.net [IPv6:2001:4830:1200:a1::2]) by mx1.freebsd.org (Postfix) with ESMTP id 166F613C4E8; Tue, 5 Feb 2008 19:06:36 +0000 (UTC) (envelope-from brooks@lor.one-eyed-alien.net) Received: from lor.one-eyed-alien.net (localhost [127.0.0.1]) by lor.one-eyed-alien.net (8.14.1/8.13.8) with ESMTP id m15J6S5N086766; Tue, 5 Feb 2008 13:06:28 -0600 (CST) (envelope-from brooks@lor.one-eyed-alien.net) Received: (from brooks@localhost) by lor.one-eyed-alien.net (8.14.1/8.13.8/Submit) id m15J6SQV086765; Tue, 5 Feb 2008 13:06:28 -0600 (CST) (envelope-from brooks) Date: Tue, 5 Feb 2008 13:06:28 -0600 From: Brooks Davis To: Joe Peterson Message-ID: <20080205190628.GA86728@lor.one-eyed-alien.net> References: <47A73C8D.3000107@skyrush.com> <86prvby5o1.fsf@ds4.des.no> <47A864D9.4060504@skyrush.com> <864pcnxz8f.fsf@ds4.des.no> <47A88ADE.7050503@skyrush.com> <86abmfwc6h.fsf@ds4.des.no> <20080205173102.GA85735@lor.one-eyed-alien.net> <47A8B24D.9050904@skyrush.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="7JfCtLOvnd9MIVvH" Content-Disposition: inline In-Reply-To: <47A8B24D.9050904@skyrush.com> User-Agent: Mutt/1.5.16 (2007-06-09) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (lor.one-eyed-alien.net [127.0.0.1]); Tue, 05 Feb 2008 13:06:28 -0600 (CST) Cc: freebsd-fs@freebsd.org, Dag-Erling Sm??rgrav , Brooks Davis Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 19:06:37 -0000 --7JfCtLOvnd9MIVvH Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Feb 05, 2008 at 12:00:29PM -0700, Joe Peterson wrote: > Brooks Davis wrote: > > We've also experienced several situations were zfs was detecting > > corruption caused by bad cabling or bad controller firmware so SMART had > > nothing to report. >=20 > Were the errors sporadic/intermittent? I'd think that if it was cabling,= the > checksum error could happen some times and not others. Mine seems to be > predictable and always there... Also, no error was reported when the fil= e was > written, and I think ZFS checks after writing. Strange. This is one rea= son > why I'm very curious which bits are bad. I'm not sure if they were consistent. We didn't really care since either t= he data one the disk was platter wasn't the data we put there or we couldn't r= ead reliably. We've not seen any checksum errors (in our somewhat limited experience) that weren't a symptom of a real, underlying problem. -- Brooks --7JfCtLOvnd9MIVvH Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFHqLOzXY6L6fI4GtQRAnNnAJoCZqqBdXS6DF3H4N4+QxXgF42UxACg3anr W9puMMUQ98Rq9hDD57Ay8u0= =o34K -----END PGP SIGNATURE----- --7JfCtLOvnd9MIVvH-- From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 19:31:35 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 36F2D16A418 for ; Tue, 5 Feb 2008 19:31:35 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (ns1.bitblocks.com [64.142.15.60]) by mx1.freebsd.org (Postfix) with ESMTP id 136F413C467 for ; Tue, 5 Feb 2008 19:31:34 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id 3D69C5B59; Tue, 5 Feb 2008 11:09:46 -0800 (PST) To: Joe Peterson In-reply-to: Your message of "Tue, 05 Feb 2008 10:38:23 MST." <47A89F0F.1030505@skyrush.com> Date: Tue, 05 Feb 2008 11:09:45 -0800 From: Bakul Shah Message-Id: <20080205190946.3D69C5B59@mail.bitblocks.com> Cc: freebsd-fs@freebsd.org Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 19:31:35 -0000 > I've checked SMART - no [unrecoverable] errors and no additional sector > reallocations, and I've done a SeaTools long test - no problems found. > > But I do not understand: in zpool status, there are stats on read errors in > addition to checksum errors. If I understand correctly, a read error would be > the system/HW reporting an error on read, whereas the whole idea of the > checksums in ZFS is to catch errors that are *not* reported as read errors > (i.e. silent bit changes that normal filesystems would never catch). What I > seem to be seeing is a case in which ZFS says the checksum is wrong. There > are only counts in the CKSUM col, not the other cols in the status, so I do > not think this is a "read error" - it is ZFS's last line of defense (the > checksum) reporting a mismatch. > > In other words, I assume the read would complete if ZFS did not catch the > checksum mismatch, and what I'd like to do is let it complete so I can see for > myself where these bit errors are by comparing the read file to a known good > copy (that I have). If there are no mismatches, it would mean there is a > metadata error of ZFS bug. It could also be a memory error of some sort. Does your system haev ECC memory? Also note that standalone tests do not seem to catch all sorts of errors that heavy use of Unix can sometimes trigger on a marginal system. But I agree with you that it would be useful to have a debug mode where you can get at the data even if it is bad (and a test mode where you can write bad data on purpose:-). [A long rant on writing testable code deleted] You have access to the zfs sources! At the very least you can add code to report the bad checksum & offset and see if matches with checksum of the same block(s) in your known good copy. From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 19:39:24 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE2AE16A418 for ; Tue, 5 Feb 2008 19:39:24 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 9396413C474 for ; Tue, 5 Feb 2008 19:39:24 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 56F781CC038; Tue, 5 Feb 2008 11:21:37 -0800 (PST) Date: Tue, 5 Feb 2008 11:21:37 -0800 From: Jeremy Chadwick To: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= Message-ID: <20080205192137.GA66362@eos.sc1.parodius.com> References: <47A73C8D.3000107@skyrush.com> <86prvby5o1.fsf@ds4.des.no> <47A864D9.4060504@skyrush.com> <864pcnxz8f.fsf@ds4.des.no> <47A88ADE.7050503@skyrush.com> <86abmfwc6h.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <86abmfwc6h.fsf@ds4.des.no> User-Agent: Mutt/1.5.16 (2007-06-09) Cc: freebsd-fs@freebsd.org Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 19:39:24 -0000 On Tue, Feb 05, 2008 at 06:22:30PM +0100, Dag-Erling Smørgrav wrote: > Joe Peterson writes: > > Dag-Erling Smørgrav writes: > > > There is now way to "read the bad data" since an unrecoverable > > > checksum error means that ZFS has no idea which of the multiple > > > version of the affected block is the right one. > > Nope, no mirror, no RAIDZ - just one partition. But as far as I know, there > > were no read errors, just a checksum error. > > A checksum error results from a read error. Check your drive's SMART > error log if it has one. It might not be detectable in a surface scan, > as the damaged sector will be automatically reassigned if it's written > to (which ZFS may very well have done) Joe's drive is in decent shape, and SMART shows absolutely no sign of temporary or even transient error. The thread regarding how all of tha was done, and the results, is over on -stable: http://lists.freebsd.org/pipermail/freebsd-stable/2008-January/039983.html Worth noting is that Joe's situation is somewhat similar to one which happened to me during the course of his and I's conversation (totally by chance!). Joe was able to get ZFS to increment CKSUM counts, while in my case ZFS scrubbing has never once incremented R/W/CK counters -- even after the incident. In my case, SMART showed no problems, the cabling was absolutely fine, and the controller stable and reliable (ICH7). I'm pretty familiar with diagnosing disk-level problems (I deal with it at work on a near-daily basis, and that's with SCSI), so I feel confident stating Joe and I's problems were *not* caused by flaky hardware. Joe and I have different hardware, though we both saw the issue with Seagate disks. The timing of the discussion became even more interesting when someone a day or so later someone posted about reproducable deadlock when copying data from a ZFS pool to a UFS filesystem -- which is exactly what I was doing when my crash happened. We don't know if that individual saw any DMA errors or timeouts in the ATA subsystem before the panic, though: http://lists.freebsd.org/pipermail/freebsd-stable/2008-January/040047.html I realise I classify pretty much everything as "priority 0" (of utmost importance), but there doesn't appear to be any indication of anyone caring about this problem. I want to do more to help, but I don't know what I *can* do, besides offer to buy people identical hardware for attempted reproduction, or put up a test box with serial console access for developers to beat up on. If we need some central point or archival of all the issues of this type seen with ZFS, I would be more than happy to take on that task. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Tue Feb 5 20:43:55 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2643016A419; Tue, 5 Feb 2008 20:43:55 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id AF62813C447; Tue, 5 Feb 2008 20:43:54 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from phobos.samsco.home (phobos.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.8/8.13.8) with ESMTP id m15KhlRQ067518; Tue, 5 Feb 2008 13:43:48 -0700 (MST) (envelope-from scottl@samsco.org) Message-ID: <47A8CA83.1000405@samsco.org> Date: Tue, 05 Feb 2008 13:43:47 -0700 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.11) Gecko/20071128 SeaMonkey/1.1.7 MIME-Version: 1.0 To: Andriy Gapon References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> <1202155663.62432.0.camel@ikaros.oook.cz> <47A8754C.5010607@icyb.net.ua> In-Reply-To: <47A8754C.5010607@icyb.net.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.4 required=5.4 tests=ALL_TRUSTED autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on pooker.samsco.org Cc: Bruce Evans , freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, freebsd-fs@FreeBSD.org, pav@FreeBSD.org, Julian Elischer , Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Feb 2008 20:43:55 -0000 Andriy Gapon wrote: > on 04/02/2008 22:07 Pav Lucistnik said the following: >> Julian Elischer píše v po 04. 02. 2008 v 10:36 -0800: >>> Andriy Gapon wrote: >>>> More on the problem with reading big directories on UDF. >>> You do realise that you have now made yourself the official >>> maintainer of the UDF file system by submitting a competent >>> and insightful analysis of the problem? >> Yay, and can you fix the sequential read performance while you're at it? >> Kthx! >> > > Pav, > > this was almost trivial :-) > See the attached patch, first hunk is just for consistency. > The code was borrowed from cd9660, only field/variable names are adjusted. > Your patch looks reasonable. Btw, for the same reason that read-ahead makes file reading much faster, I would not change directory reading to be 1 sector at a time (unless you also do read-ahead for it). > But there is another issue that I also mentioned in the email about > directory reading. It is UDF_INVALID_BMAP case of udf_bmap_internal, > i.e. the case when file data is embedded into a file entry. > This is a special case that needs to be handled differently. > udf_readatoffset() handles it, but the latest udf_read code doesn't. > I have a real UDF filesystem where this type of allocation is used for > small files and those files can not be read. Oh, so directory data can also follow this convention? Blah. Feel free to fix that too if you want =-) Scott From owner-freebsd-fs@FreeBSD.ORG Wed Feb 6 06:15:01 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3704816A46D for ; Wed, 6 Feb 2008 06:15:01 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outE.internet-mail-service.net (outE.internet-mail-service.net [216.240.47.228]) by mx1.freebsd.org (Postfix) with ESMTP id 2EDA713C469 for ; Wed, 6 Feb 2008 06:15:00 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Tue, 05 Feb 2008 22:15:00 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id 48E721270F8; Tue, 5 Feb 2008 22:14:59 -0800 (PST) Message-ID: <47A95065.3060703@elischer.org> Date: Tue, 05 Feb 2008 22:15:01 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Andriy Gapon References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> <1202155663.62432.0.camel@ikaros.oook.cz> <47A8754C.5010607@icyb.net.ua> In-Reply-To: <47A8754C.5010607@icyb.net.ua> Content-Type: text/plain; charset=ISO-8859-2; format=flowed Content-Transfer-Encoding: 8bit Cc: Bruce Evans , freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, freebsd-fs@FreeBSD.org, pav@FreeBSD.org, Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 06:15:01 -0000 Andriy Gapon wrote: > on 04/02/2008 22:07 Pav Lucistnik said the following: >> Julian Elischer pí¹e v po 04. 02. 2008 v 10:36 -0800: >>> Andriy Gapon wrote: >>>> More on the problem with reading big directories on UDF. >>> You do realise that you have now made yourself the official >>> maintainer of the UDF file system by submitting a competent >>> and insightful analysis of the problem? >> Yay, and can you fix the sequential read performance while you're at it? >> Kthx! >> > > Pav, > > this was almost trivial :-) Ok that settles it.. it's yours. > See the attached patch, first hunk is just for consistency. > The code was borrowed from cd9660, only field/variable names are adjusted. > > But there is another issue that I also mentioned in the email about > directory reading. It is UDF_INVALID_BMAP case of udf_bmap_internal, > i.e. the case when file data is embedded into a file entry. > This is a special case that needs to be handled differently. > udf_readatoffset() handles it, but the latest udf_read code doesn't. > I have a real UDF filesystem where this type of allocation is used for > small files and those files can not be read. > > This is described in Part 4, section 14.6.8 of ECMA-167. > > From owner-freebsd-fs@FreeBSD.ORG Wed Feb 6 07:22:11 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26F0A16A420 for ; Wed, 6 Feb 2008 07:22:11 +0000 (UTC) (envelope-from bsd@fluffles.net) Received: from mail.fluffles.net (fluffles.net [80.69.95.190]) by mx1.freebsd.org (Postfix) with ESMTP id 04D0013C465 for ; Wed, 6 Feb 2008 07:22:10 +0000 (UTC) (envelope-from bsd@fluffles.net) Received: from [10.0.0.18] (cust.95.160.adsl.cistron.nl [195.64.95.160]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: info@fluffles.net) by mail.fluffles.net (Postfix) with ESMTP id 9937AB29D5D for ; Wed, 6 Feb 2008 08:01:43 +0100 (CET) Message-ID: <47A95BC0.1060307@fluffles.net> Date: Wed, 06 Feb 2008 08:03:28 +0100 From: "fluffles.net" User-Agent: Thunderbird 2.0.0.6 (X11/20071022) MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit Cc: Subject: delayed write buffer X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 07:22:11 -0000 Hello kind list, I was wondering how to tune FreeBSD's VFS write buffer. I would like a large amount of RAM (say 500MB out of 1GB) to be reserved or allocated to be used as write buffer for my backup NAS system. If i understand the mechanism correctly, a "dd if=/dev/zero of=/mnt/fs bs=1m count=500" would act as if i used a malloc-backed md device. This leads to very nice performance gains, especially in my case because i'm using encryption causing throughput to be limited to 22MB/s. But if the first 500MB is free, i can mask this limitation and experience a fast drive, when writing that is. Anyone can point me to the right directions? I tried playing with some sysctl vfs (notably the vfs.maxmallocbufspace tunable) but did not achieve the desired effect. And yes, i do know a lot of dirty buffers is dangerous but my storage setup is redundant enough. Besides i'm just curious in this topic. :) Could a regular BIO-FLUSH caused by UFS metadata sync be the curlpit? Thanks for any assistance! Kind greetings, Veronica From owner-freebsd-fs@FreeBSD.ORG Wed Feb 6 10:00:19 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5470816A417; Wed, 6 Feb 2008 10:00:19 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 1002613C45A; Wed, 6 Feb 2008 10:00:19 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id 198F0208C; Wed, 6 Feb 2008 11:00:11 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.3/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 74AC5208A; Wed, 6 Feb 2008 11:00:10 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 4DD69844A2; Wed, 6 Feb 2008 11:00:10 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Brooks Davis References: <47A73C8D.3000107@skyrush.com> <86prvby5o1.fsf@ds4.des.no> <47A864D9.4060504@skyrush.com> <864pcnxz8f.fsf@ds4.des.no> <47A88ADE.7050503@skyrush.com> <86abmfwc6h.fsf@ds4.des.no> <20080205173102.GA85735@lor.one-eyed-alien.net> Date: Wed, 06 Feb 2008 11:00:10 +0100 In-Reply-To: <20080205173102.GA85735@lor.one-eyed-alien.net> (Brooks Davis's message of "Tue\, 5 Feb 2008 11\:31\:02 -0600") Message-ID: <86tzkmv1zp.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 10:00:19 -0000 Brooks Davis writes: > We've also experienced several situations were zfs was detecting > corruption caused by bad cabling or bad controller firmware so SMART > had nothing to report. I once worked on a FreeBSD-based turn-key web hosting solution. Our first (and, eventually, only) customer had a four-server rig where two of the servers shared a split rack-mount SCSI cabinet. Either the cables or the cabinet (most likely the latter) were defective, and at some point, all data written to disk became silently corrupted. To add insult to injury, the backups were unusable. I don't recall why; either the customer hadn't put tapes in the backup server, or the backups had been corrupted as well, or maybe the backup solution I'd written simply didn't work and hadn't been properly tested. It was around that time I discovered that FreeBSD's version of GNU tar, when run with certain command-line options (such as those typically used by *shiver* Amanda), would create archives that no tar implementation - itself included - could extract. That rig was a disaster in so many ways... When I first ordered it, I also ordered spares for everything - spare PSUs, spare cooling fans, spare disks etc. Of course, the spares we actually got didn't fit. I can understand delivering the wrong spare fan when the original is factory-installed, but when you ship a SCSI cabinet full of disks along with spare disks that *don't match*, you know you're about to lose a customer. Oh, and the SCSI cables we used between the servers and the cabinet didn't fit or lock properly, so we kept losing touch with the disks until we replaced the cables. Strangely enough, that supplier later lost their exclusive contract with the University of Oslo, and went out of business. Somebody apparently figured that Dell could more easily handle larger orders, was less likely to ship the wrong parts, and more likely to admit it and quickly send a replacement if it did happen (although I've heard of a case where someone complained to Dell about six missing screws, and shortly thereafter received six separate cardboard boxes, each containing one screw) (oh shit, I'm turning into a rambling old fart!) DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Wed Feb 6 10:06:52 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3EB3E16A468; Wed, 6 Feb 2008 10:06:52 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id EED0513C458; Wed, 6 Feb 2008 10:06:51 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id D69972082; Wed, 6 Feb 2008 11:06:42 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.3/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id C7D8A2049; Wed, 6 Feb 2008 11:06:42 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id ADB9384493; Wed, 6 Feb 2008 11:06:42 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Jeremy Chadwick References: <47A73C8D.3000107@skyrush.com> <86prvby5o1.fsf@ds4.des.no> <47A864D9.4060504@skyrush.com> <864pcnxz8f.fsf@ds4.des.no> <47A88ADE.7050503@skyrush.com> <86abmfwc6h.fsf@ds4.des.no> <20080205192137.GA66362@eos.sc1.parodius.com> Date: Wed, 06 Feb 2008 11:06:42 +0100 In-Reply-To: <20080205192137.GA66362@eos.sc1.parodius.com> (Jeremy Chadwick's message of "Tue\, 5 Feb 2008 11\:21\:37 -0800") Message-ID: <86prvav1ot.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 10:06:52 -0000 Jeremy Chadwick writes: > I realise I classify pretty much everything as "priority 0" (of utmost > importance), but there doesn't appear to be any indication of anyone > caring about this problem. Most likely, nobody with the ability to find and fix the cause of your problem has been able to reproduce it. It's really hard to fix a bug you can't reproduce; you're basically flying blind, unless the person who reported the issue is very experienced and can cooperate in diagnosing the problem (which is very rare). > I want to do more to help, but I don't know what I *can* do, besides > offer to buy people identical hardware for attempted reproduction, or > put up a test box with serial console access for developers to beat up > on. Several developers already *have* identical hardware, or nearly so. (speaking for myself, all my Seagate disks are dead, and I use Western Digital exclusively these days, as they tend to live longer in my experience; sometimes they don't even fail until *after* the warranty runs out!) > | Making life hard for others since 1977. PGP: | 4BD6C0C= B | Ooh, a kindred spirit! :) DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Wed Feb 6 10:55:52 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1DFCC16A420 for ; Wed, 6 Feb 2008 10:55:52 +0000 (UTC) (envelope-from antik@bsd.ee) Received: from zzz.ee (kalah.zzz.ee [194.204.30.253]) by mx1.freebsd.org (Postfix) with ESMTP id C3A1C13C4DD for ; Wed, 6 Feb 2008 10:55:51 +0000 (UTC) (envelope-from antik@bsd.ee) Received: by zzz.ee (Postfix, from userid 3019) id 050C282822D; Wed, 6 Feb 2008 12:37:44 +0200 (EET) X-Spam-Checker-Version: SpamAssassin on spamassassin.zzz.ee X-Spam-Level: X-Spam-Guessed-Language: en X-Spam-Status: No, score=-3.0 required=5.0 tests=ALL_TRUSTED,BAYES_55 X-Spam-Checker-URL: http://info.zzz.ee Received: from andrei.demo (adsl215.uninet.ee [194.204.62.215]) by zzz.ee (Postfix) with ESMTP id 6FAED827FE4 for ; Wed, 6 Feb 2008 12:37:42 +0200 (EET) From: Andrei Kolu To: freebsd-fs@freebsd.org Date: Wed, 6 Feb 2008 12:37:41 +0200 User-Agent: KMail/1.9.7 References: <47A73C8D.3000107@skyrush.com> <20080205192137.GA66362@eos.sc1.parodius.com> <86prvav1ot.fsf@ds4.des.no> In-Reply-To: <86prvav1ot.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200802061237.41660.antik@bsd.ee> Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 10:55:52 -0000 On Wednesday 06 February 2008 12:06:42 Dag-Erling Sm=C3=B8rgrav wrote: > Jeremy Chadwick writes: > > I realise I classify pretty much everything as "priority 0" (of utmost > > importance), but there doesn't appear to be any indication of anyone > > caring about this problem. > > Most likely, nobody with the ability to find and fix the cause of your > problem has been able to reproduce it. It's really hard to fix a bug > you can't reproduce; you're basically flying blind, unless the person > who reported the issue is very experienced and can cooperate in > diagnosing the problem (which is very rare). > > > I want to do more to help, but I don't know what I *can* do, besides > > offer to buy people identical hardware for attempted reproduction, or > > put up a test box with serial console access for developers to beat up > > on. > > Several developers already *have* identical hardware, or nearly so. > > (speaking for myself, all my Seagate disks are dead, and I use Western > Digital exclusively these days, as they tend to live longer in my > experience; sometimes they don't even fail until *after* the warranty > runs out!) I have exactly opposite experience- all my WD drives are dead and only Seag= ate=20 survived. Actually I'd recommend to use as many different manufacturers as= =20 possible. For example in NAS with 4-8 hard drives I got at least two=20 different drives- Seagate and Hitachi- in case of some bad model (or bad da= y=20 in manufacturer plant) you'll not loose all drives in short period of time.= =2E.=20 There is no such a thing like bad brand- there is bad models or bad days... From owner-freebsd-fs@FreeBSD.ORG Wed Feb 6 16:29:27 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89F5616A418; Wed, 6 Feb 2008 16:29:27 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id D8D9013C44B; Wed, 6 Feb 2008 16:29:26 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id AA36843C906; Wed, 6 Feb 2008 18:29:24 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pJDvGNN+uRPR; Wed, 6 Feb 2008 18:29:24 +0200 (EET) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id BA5EB43C20E; Wed, 6 Feb 2008 18:29:23 +0200 (EET) Message-ID: <47A9E062.3040409@icyb.net.ua> Date: Wed, 06 Feb 2008 18:29:22 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20080123) MIME-Version: 1.0 To: Scott Long , Poul-Henning Kamp References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A7339E.4070708@icyb.net.ua> <47A73562.4010309@samsco.org> In-Reply-To: <47A73562.4010309@samsco.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, freebsd-hackers@FreeBSD.org, Remko Lodder , Pav Lucistnik , Bruce Evans Subject: Re: fs/udf: vm pages "overlap" while reading large dir X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 16:29:27 -0000 on 04/02/2008 17:55 Scott Long said the following: > Andriy Gapon wrote: [snip] >> After some code reading and run-time debugging, here are some facts >> about udf directory reading: >> 1. bread-ing is done via device vnode (as opposed to directory vnodes), >> as a consequence udf_strategy is not involved in directory reading >> 2. the device vnode has a non-NULL v_object, this means that VMIO is used [strike out incorrect guess] >> 4. bread is done on as much data as possible, up to MAXBSIZE [and even >> slightly more in some cases] (i.e. code determines directory data size >> and attempts to read in everything in one go) >> 5. physical sector number adjusted to DEV_BSIZE (512 bytes) sectors is >> passed to bread - this is because of #1 above and peculiarities of buf >> system >> 6. directory reading and file reading are quite different, because the >> latter does reading via file vnode >> >> Let's a consider a simple case of "directory reading" 5 * PAGE_SIZE (4K) >> bytes starting from a physical sector N with N%2 = 0 from media with >> sector size of 2K (most common CD/DVD case): >> 1. we want to read 12 KB = 3 pages = 6 sectors starting from sector N, >> N%2 = 0 >> 2. 4*N is passed as a sector number to bread >> 3. bo_bsize of the corresponding bufobj is a media sector size, i.e. 2K >> 4. actual read in bread will happen from b_iooffset of 4*N*DEV_BSIZE = >> N*2K, i.e. correct offset within the device >> 5. for a fresh read getblk will be called >> 6. getblk will set b_offset to blkno*bo_bsize=4*N*2K, i.e. 4 times the >> correct byte offset within the device >> 7. allocbuf will allocate pages using B_VMIO case >> 8. allocbuf calculates base page index as follows: pi = b_offset/PAGE_SIZE >> this means the following: >> sectors N and N+1 will be bound to a page with index 4*N*2K/4K = 2*N >> sectors N+2 and N+3 will be bound to the next page 2*N +1 >> sectors N+4 and N+5 is tied to the next page 2*N + 2 [insert] note that bstrategy will get correct params as b_iooffset is set to blkno*DEV_BSIZE, i.e. correct byte offset. >> >> Now let's consider a "directory read" of 2 sectors (1 page) starting at >> physical sector N+1. >> Using the same calculations as above, we see that the sector will be >> tied to a page 2*(N+1) = 2*N + 2. >> And this page already contains valid cached data from the previous read, >> *but* it contains data from sectors N+4 and N+5 instead of N+1 and N+2. >> >> So, we see, that because of b_offset being 4 times the actual byte >> offset, we get incorrect page indexing. Namely, sector X gets associated >> with different pages depending on what sector is used as a starting >> sector for bread. If bread starts at sector N, then data of sector N+1 >> is associated with (second half of) page with index 2*N; but if bread >> starts at sector N+1 then it gets associated with (the first half of) >> page with index 2*(N+1). [snip] > I think you forgot to attach the patch =-) This is an excellent > write-up, and I think you're on the right track. It's been nearly 8 > years since I wrote most of this code, so my memory is a bit fuzzy, but > I think the reason that I used the device vnode is because I was having > trouble with overlapping buffers when using the directory vnode, so this > was the easiest way to avoid that at the time. Since it sounds like > this is no longer a viable solution, I definitely support your ideas. [additional analysis of the problem, with new facts that might give rise to a different solution] Small summary of the above long description. For directory reading fs/udf performs bread() on a (underlying) device vnode. It passes block number as if block size was 512 bytes (i.e. byte_offset_within_dev/512). On the other hand vm page index calculation code uses the following formula: (blkno*bo_bsize)/PAGE_SIZE. For CD/DVD devices bo_bsize is set to 2048. This results in page index overlap when reading sufficiently many blocks from adjacent starting blocks. That is, it seems that in "normal" cases page index gets calculated as byte_offset/PAGE_SIZE, but in this case page index gets to be 4*byte_offset/PAGE_SIZE. More details are in the quoted above text. With a lot of help from Bruce Evance I found this commit which seems to have made a big difference: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/vfs_bio.c.diff?r1=1.458;r2=1.459;f=h Before this change page index for device vnodes was always (blkno*512)/PAGE_SIZE. This is because of the vn_isdisk() case. udf_mountfs device vnode is passed to g_vfs_open() (in geom_vfs.c) and the latter has the following line of code: bo->bo_bsize = pp->sectorsize; Where pp is geom provider for the device in question. I am not sure if the mentioned above vfs_bio.c change broke something else in reading from device vnodes or if it fixed something for that. I am also not sure what would be the consequences of setting bo_bsize to 512 for vnodes of "disk" devices. It would most probably fix current code in UDF, but I am not sure if it might break anything else. Just wanted to let the more knowledgeable people know and make a decision. P.S. bo_bsize seems to be referenced only in a handful of files and in most of them it seems that it is related to "file" vnodes (as opposed to "device" vnodes). -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Feb 6 16:34:30 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42BB916A419; Wed, 6 Feb 2008 16:34:30 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id CE1AE13C4E1; Wed, 6 Feb 2008 16:34:29 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id F2A4043F8F2; Wed, 6 Feb 2008 18:34:28 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pVY5er24lJp3; Wed, 6 Feb 2008 18:34:28 +0200 (EET) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id CC0CE43CA85; Wed, 6 Feb 2008 18:34:26 +0200 (EET) Message-ID: <47A9E191.8000607@icyb.net.ua> Date: Wed, 06 Feb 2008 18:34:25 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20080123) MIME-Version: 1.0 To: pav@FreeBSD.org References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> <1202155663.62432.0.camel@ikaros.oook.cz> <47A8754C.5010607@icyb.net.ua> <1202235368.68281.12.camel@ikaros.oook.cz> In-Reply-To: <1202235368.68281.12.camel@ikaros.oook.cz> Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: 8bit Cc: Bruce Evans , freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, freebsd-fs@FreeBSD.org, Julian Elischer , Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 16:34:30 -0000 on 05/02/2008 20:16 Pav Lucistnik said the following: > Andriy Gapon pí¹e v út 05. 02. 2008 v 16:40 +0200: > >>> Yay, and can you fix the sequential read performance while you're at it? >>> Kthx! > >> this was almost trivial :-) >> See the attached patch, first hunk is just for consistency. >> The code was borrowed from cd9660, only field/variable names are adjusted. > > Omg looks good. Can't wait for it to bubble throught to 6-STABLE :) > > Thanks million for working on this. > Actually the patch is not entirely correct. max_size returned from udf_bmap_internal should be used to calculate number of continuous sectors for read-ahead (as opposed to file size in the patch). -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Feb 6 16:36:02 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ABC7D16A477; Wed, 6 Feb 2008 16:36:02 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id 4366D13C442; Wed, 6 Feb 2008 16:36:02 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id 6E34643E003; Wed, 6 Feb 2008 18:36:01 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2uP3KJXq51GZ; Wed, 6 Feb 2008 18:36:01 +0200 (EET) Received: from [10.2.1.87] (gateway.cybervisiontech.com.ua [88.81.251.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 9BACA43C20E; Wed, 6 Feb 2008 18:36:00 +0200 (EET) Message-ID: <47A9E1EF.5020507@icyb.net.ua> Date: Wed, 06 Feb 2008 18:35:59 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20080123) MIME-Version: 1.0 To: Scott Long References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> <1202155663.62432.0.camel@ikaros.oook.cz> <47A8754C.5010607@icyb.net.ua> <47A8CA83.1000405@samsco.org> In-Reply-To: <47A8CA83.1000405@samsco.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@FreeBSD.org, scottl@FreeBSD.org, freebsd-fs@FreeBSD.org, pav@FreeBSD.org, Julian Elischer , Remko Lodder Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 16:36:02 -0000 on 05/02/2008 22:43 Scott Long said the following: > Andriy Gapon wrote: > >> But there is another issue that I also mentioned in the email about >> directory reading. It is UDF_INVALID_BMAP case of udf_bmap_internal, >> i.e. the case when file data is embedded into a file entry. >> This is a special case that needs to be handled differently. >> udf_readatoffset() handles it, but the latest udf_read code doesn't. >> I have a real UDF filesystem where this type of allocation is used for >> small files and those files can not be read. > > Oh, so directory data can also follow this convention? Blah. Feel free > to fix that too if you want =-) Yes, this allocation type can be applied to both files and directories. Directories are already handled because of readatoffset() but udf_read needs to be made aware of this special case. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Feb 6 16:42:53 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6EE4B16A41A for ; Wed, 6 Feb 2008 16:42:53 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from shadow.wildlava.net (shadow.wildlava.net [67.40.138.81]) by mx1.freebsd.org (Postfix) with ESMTP id 2E71013C478 for ; Wed, 6 Feb 2008 16:42:53 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from [67.40.138.82] (crater.wildlava.net [67.40.138.82]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by shadow.wildlava.net (Postfix) with ESMTP id 79A628F394; Wed, 6 Feb 2008 09:42:52 -0700 (MST) Message-ID: <47A9E38B.6040100@skyrush.com> Date: Wed, 06 Feb 2008 09:42:51 -0700 From: Joe Peterson User-Agent: Thunderbird 2.0.0.9 (X11/20071127) MIME-Version: 1.0 To: Bakul Shah References: <20080205190946.3D69C5B59@mail.bitblocks.com> In-Reply-To: <20080205190946.3D69C5B59@mail.bitblocks.com> X-Enigmail-Version: 0.95.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Forcing full file read in ZFS even when checksum error encountered X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 16:42:53 -0000 Bakul Shah wrote: > It could also be a memory error of some sort. Does your > system haev ECC memory? Yes, I always insist on ECC. > Also note that standalone tests do > not seem to catch all sorts of errors that heavy use of Unix > can sometimes trigger on a marginal system. I do plan to do a few more HW checks (cables, etc.), just to make sure. I had been avoiding touching my HW config to preserve the current state of this issue. However, given the coincidental experience Jeremy talked about and the fact that the DMA errors I have seen using ZFS on FreeBSD that I do not see using ZFS-Fuse on the same disk/pool in Linux, I have a gut feeling something funny is going on. > But I agree with you that it would be useful to have a debug > mode where you can get at the data even if it is bad (and a > test mode where you can write bad data on purpose:-). [A > long rant on writing testable code deleted] Yes, the danger of course is if someone forget's that the debug mode is engaged, but I think care could be taken to make sure this cannot easily be done accidentally or massive warnings can be issues to make sure the user knows. > You have access to the zfs sources! At the very least you can > add code to report the bad checksum & offset and see if > matches with checksum of the same block(s) in your known good > copy. Yep, this is my next planned step. Thanks, Joe From owner-freebsd-fs@FreeBSD.ORG Wed Feb 6 23:33:28 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1550B16A476; Wed, 6 Feb 2008 23:33:28 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from falcon.cybervisiontech.com (falcon.cybervisiontech.com [217.20.163.9]) by mx1.freebsd.org (Postfix) with ESMTP id 5EA6C13C459; Wed, 6 Feb 2008 23:33:27 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from localhost (localhost [127.0.0.1]) by falcon.cybervisiontech.com (Postfix) with ESMTP id 66E62744013; Thu, 7 Feb 2008 01:33:25 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at falcon.cybervisiontech.com Received: from falcon.cybervisiontech.com ([127.0.0.1]) by localhost (falcon.cybervisiontech.com [127.0.0.1]) (amavisd-new, port 10027) with ESMTP id gHzvoAmT8B76; Thu, 7 Feb 2008 01:33:25 +0200 (EET) Received: from [10.74.70.239] (unknown [193.138.145.53]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by falcon.cybervisiontech.com (Postfix) with ESMTP id 801A6744012; Thu, 7 Feb 2008 01:33:23 +0200 (EET) Message-ID: <47AA43B9.1040608@icyb.net.ua> Date: Thu, 07 Feb 2008 01:33:13 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.9 (X11/20071208) MIME-Version: 1.0 To: pav@FreeBSD.org, scottl@FreeBSD.org References: <200612221824.kBMIOhfM049471@freefall.freebsd.org> <47A2EDB0.8000801@icyb.net.ua> <47A2F404.7010208@icyb.net.ua> <47A735A4.3060506@icyb.net.ua> <47A75B47.2040604@elischer.org> <1202155663.62432.0.camel@ikaros.oook.cz> <47A8754C.5010607@icyb.net.ua> <1202235368.68281.12.camel@ikaros.oook.cz> <47A9E191.8000607@icyb.net.ua> In-Reply-To: <47A9E191.8000607@icyb.net.ua> Content-Type: multipart/mixed; boundary="------------090706050706020808090509" Cc: freebsd-fs@FreeBSD.org, freebsd-hackers@FreeBSD.org Subject: Re: fs/udf: vm pages "overlap" while reading large dir [patch] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 23:33:28 -0000 This is a multi-part message in MIME format. --------------090706050706020808090509 Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: 7bit on 06/02/2008 18:34 Andriy Gapon said the following: > > Actually the patch is not entirely correct. max_size returned from > udf_bmap_internal should be used to calculate number of continuous > sectors for read-ahead (as opposed to file size in the patch). > Attached is an updated patch. The most prominent changes from the previous version: 1. udf_read can handle files with data embedded into fentry (sysutils/udfclient can produce such things for small files). 2. above mentioned files can now be mmap-ed, so that you can not only cat them, but also cp them; btw, I believe that cp(1) should have logic to try to fallback to read if mmap() fails - remember kern/92040 ? :-) 3. udf_bmap uses max_size[*] returned by udf_bmap_internal to calculate a number of contiguous blocks for read-ahead; [*] - this is a number of contiguous bytes from a given offset within a file. Things that stay the same: 1. I still use bread() via directory vnode; I think it's better even if reading via device vnode would work correctly. 2. I still try to read as much data as possible in directory bread(), I think this is a sort of an ad-hoc read-ahead without all the heuristics and logic that cluster reading does; I think that this should be ok because we do not have random reading of directory, it's always sequential - either to read-in the whole directory or linearly search through it until wanted entry is found. Detailed description of the patch. udf_vnops.c: Hunk1 - this is "just in case" patch, reject files with unsupported strategy right away instead of deferring failure to udf_bmap_internal. Hunk2 - fix typo in existing macro, add new macro to check if a file has data embedded into fentry ("unusual file"). Hunk3 - for an unusual file just uiomove data from fentry. Hunk4 - cosmetic changes plus a fix for udf_bmap_internal errors not actually being honored; also added an additional printf, because udf_strategy should never really be called for unusual files. Hunk5 - return EOPNOTSUPP for unusual files, this is correct because we can not correctly bmap them and also this enables correct handling of this files in vm code (vnode_pager); also code for read-ahead calculation borrowed from cd9660. Hunk6- explain function of udf_bmap_internal call in this place. Hunk7 - some cosmetics; prevent size passed to bread from being greater than MAXBSIZE; do bread via directory vnode rather than device vnode (udf_readlblks was a macro-wrapper around bread). udf_vfsops.c: Hunk1 - this is borrowed from cd9660, apparently this data is needed for correct cluster reading. Couple of words about testing. udf in 7.0-RC1 plus this patch correctly handled everything I could throw at it - huge directories, unusual files, reading, mmaping. Using udfclientfs I wrote a directory on DVD-RAM UDF disk that contained 2G of files from ports distfiles. The files varied in size from about 100 of bytes to hundreds of megabytes. I watched systat -vmstat while copying this directory. With unpatched code some files would not be copied (small "unusual files"), KB/t value stayed at 2K (meaning reads always were sector sized), MB/s was about 1. With the patch all files were copied successfully and correctly (md5 verified), KB/t varied from 2K to 64K (apparently depending on size of currently copied file), MB/s varied from 1 to 4 which is not bad for these DVD-RAM disk and drive. If somebody asks I can produce a UDF image that would contain various test cases: large directory, unusual files, etc. Or you can give a try to udfclient/udfclientfs from sysutils/udfclient port. I hope this patch will be useful. -- Andriy Gapon --------------090706050706020808090509 Content-Type: text/x-patch; name="udf_latest2.diff" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="udf_latest2.diff" --- udf_vnops.c.orig 2008-01-29 23:50:49.000000000 +0200 +++ udf_vnops.c 2008-02-07 00:12:34.000000000 +0200 @@ -168,6 +168,16 @@ udf_open(struct vop_open_args *ap) { struct udf_node *np = VTON(ap->a_vp); off_t fsize; + /* + * We currently do not support any other strategy type, so + * udf_bmap_internal, udf_bmap, udf_strategy would fail. + * I.e. we can not access content of this file anyway. + */ + if (le16toh(np->fentry->icbtag.strat_type) != 4) { + printf("Unsupported strategy type %d\n", le16toh(np->fentry->icbtag.strat_type)); + return ENODEV; + } + fsize = le64toh(np->fentry->inf_len); vnode_create_vobject(ap->a_vp, fsize, ap->a_td); return 0; @@ -340,7 +350,9 @@ udf_pathconf(struct vop_pathconf_args *a #define lblkno(udfmp, loc) ((loc) >> (udfmp)->bshift) #define blkoff(udfmp, loc) ((loc) & (udfmp)->bmask) -#define lblktosize(imp, blk) ((blk) << (udfmp)->bshift) +#define lblktosize(udfmp, blk) ((blk) << (udfmp)->bshift) + +#define HAS_EMBEDDED_DATA(node) ((le16toh((node)->fentry->icbtag.flags) & 0x7) == 3) static int udf_read(struct vop_read_args *ap) @@ -359,6 +371,26 @@ udf_read(struct vop_read_args *ap) return (0); if (uio->uio_offset < 0) return (EINVAL); + + if (HAS_EMBEDDED_DATA(node)) { + struct file_entry *fentry; + uint8_t *data; + + fentry = node->fentry; + data = &fentry->data[le32toh(fentry->l_ea)]; + fsize = le32toh(fentry->l_ad); + + n = uio->uio_resid; + diff = fsize - uio->uio_offset; + if (diff <= 0) + return (0); + if (diff < n) + n = diff; + + error = uiomove(data + uio->uio_offset, (int)n, uio); + return (error); + } + fsize = le64toh(node->fentry->inf_len); udfmp = node->udfmp; do { @@ -799,10 +831,10 @@ udf_strategy(struct vop_strategy_args *a struct buf *bp; struct vnode *vp; struct udf_node *node; + struct bufobj *bo; int maxsize; daddr_t sector; - struct bufobj *bo; - int multiplier; + int error; bp = a->a_bp; vp = a->a_vp; @@ -813,24 +845,23 @@ udf_strategy(struct vop_strategy_args *a * Files that are embedded in the fentry don't translate well * to a block number. Reject. */ - if (udf_bmap_internal(node, bp->b_lblkno * node->udfmp->bsize, - §or, &maxsize)) { + error = udf_bmap_internal(node, lblktosize(node->udfmp, bp->b_lblkno), §or, &maxsize); + if (error) { + if (error == UDF_INVALID_BMAP) + printf("udf_strategy called for file with data embedded in fentry\n"); clrbuf(bp); bp->b_blkno = -1; + bufdone(bp); + return (0); } - - /* bmap gives sector numbers, bio works with device blocks */ - multiplier = node->udfmp->bsize / DEV_BSIZE; - bp->b_blkno = sector * multiplier; - - } - if ((long)bp->b_blkno == -1) { - bufdone(bp); - return (0); + /* udf_bmap_internal gives sector numbers, bio works with device blocks */ + bp->b_blkno = sector << (node->udfmp->bshift - DEV_BSHIFT); } + bo = node->udfmp->im_bo; bp->b_iooffset = dbtob(bp->b_blkno); BO_STRATEGY(bo, bp); + return (0); } @@ -851,17 +882,43 @@ udf_bmap(struct vop_bmap_args *a) if (a->a_runb) *a->a_runb = 0; - error = udf_bmap_internal(node, a->a_bn * node->udfmp->bsize, &lsector, - &max_size); + /* + * UDF_INVALID_BMAP means data embedded into fentry, this is an internal + * error that should not be propagated to calling code. + * Most obvious mapping for this error is EOPNOTSUPP as we can not truly + * translate block numbers in this case. + * Incidentally, this return code will make vnode pager to use VOP_READ + * to get data for mmap-ed pages and udf_read knows how to do the right + * thing for this kind of files. + */ + error = udf_bmap_internal(node, a->a_bn << node->udfmp->bshift, &lsector, &max_size); + if (error == UDF_INVALID_BMAP) + return EOPNOTSUPP; if (error) return (error); /* Translate logical to physical sector number */ *a->a_bnp = lsector << (node->udfmp->bshift - DEV_BSHIFT); - /* Punt on read-ahead for now */ - if (a->a_runp) - *a->a_runp = 0; + /* + * Determine maximum number of readahead blocks following the + * requested block. + */ + if (a->a_runp) { + int nblk; + + nblk = (max_size >> node->udfmp->bshift) - 1; + if (nblk <= 0) + *a->a_runp = 0; + else if (nblk >= (MAXBSIZE >> node->udfmp->bshift)) + *a->a_runp = (MAXBSIZE >> node->udfmp->bshift) - 1; + else + *a->a_runp = nblk; + } + + if (a->a_runb) { + *a->a_runb = 0; + } return (0); } @@ -1060,7 +1117,10 @@ udf_readatoffset(struct udf_node *node, udfmp = node->udfmp; - *bp = NULL; + /* + * This call is made not only to detect UDF_INVALID_BMAP case, + * max_size is used as an ad-hoc read-ahead hint for "normal" case. + */ error = udf_bmap_internal(node, offset, §or, &max_size); if (error == UDF_INVALID_BMAP) { /* @@ -1078,9 +1138,10 @@ udf_readatoffset(struct udf_node *node, /* Adjust the size so that it is within range */ if (*size == 0 || *size > max_size) *size = max_size; - *size = min(*size, MAXBSIZE); + *size = min(*size, MAXBSIZE - blkoff(udfmp, offset)); - if ((error = udf_readlblks(udfmp, sector, *size + (offset & udfmp->bmask), bp))) { + *bp = NULL; + if ((error = bread(node->i_vnode, lblkno(udfmp, offset), (*size + blkoff(udfmp, offset) + udfmp->bmask) & ~udfmp->bmask, NOCRED, bp))) { printf("warning: udf_readlblks returned error %d\n", error); /* note: *bp may be non-NULL */ return (error); --- udf_vfsops.c.orig 2007-03-13 03:50:24.000000000 +0200 +++ udf_vfsops.c 2008-02-05 01:29:10.000000000 +0200 @@ -330,6 +330,11 @@ udf_mountfs(struct vnode *devvp, struct bo = &devvp->v_bufobj; + if (devvp->v_rdev->si_iosize_max != 0) + mp->mnt_iosize_max = devvp->v_rdev->si_iosize_max; + if (mp->mnt_iosize_max > MAXPHYS) + mp->mnt_iosize_max = MAXPHYS; + /* XXX: should be M_WAITOK */ MALLOC(udfmp, struct udf_mnt *, sizeof(struct udf_mnt), M_UDFMOUNT, M_NOWAIT | M_ZERO); --------------090706050706020808090509-- From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 01:28:35 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7DABE16A417 for ; Thu, 7 Feb 2008 01:28:35 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.170]) by mx1.freebsd.org (Postfix) with ESMTP id 01E9713C447 for ; Thu, 7 Feb 2008 01:28:34 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by ug-out-1314.google.com with SMTP id y2so836731uge.37 for ; Wed, 06 Feb 2008 17:28:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:mime-version:content-type:content-transfer-encoding:content-disposition:x-google-sender-auth; bh=A47Uzlr5w+3geNFR1JT+VmEw8eh24ejzk93ik7cQqcs=; b=XPZGm3pen+bzBFkBB0infwkNZXh1IwznrKfpsTqAL47ZTcBUgUIa7Ot+UPWcFRIoqdjBkXXe0Lm4IwZ1ml2kHsBPdggp1yYK9wpvHUL3WadOefCoVYpgGEVd2LqITWyihY5D1QXzjS24JixlRjfH/MkqrRpENx3lsxeCHBRgOI8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:mime-version:content-type:content-transfer-encoding:content-disposition:x-google-sender-auth; b=uVzsTBmN6hZ+divyO3Yg7SeGVc/wBKMFtkersXhGALgqj+KqUf+jl2urysk3T1yUmN5RkvLfOGTFq1YW8BxSyOK+SKICMfFhMIlOLneQfzSsU+OlOVIO07n+4C/1RgTBFN4OjvAVdl/VpV+vZyQZYZm7TdtcO/iBjQR38AKY8rQ= Received: by 10.86.100.7 with SMTP id x7mr9799786fgb.10.1202346041497; Wed, 06 Feb 2008 17:00:41 -0800 (PST) Received: by 10.86.28.19 with HTTP; Wed, 6 Feb 2008 17:00:41 -0800 (PST) Message-ID: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> Date: Thu, 7 Feb 2008 02:00:41 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: freebsd-arch@freebsd.org, freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Google-Sender-Auth: f21f19ae39864046 Cc: Scot Hetzel , Jeff Roberson , Doug Barton , Yar Tikhiy Subject: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 01:28:35 -0000 As exposed by several users, NTFS seems to be broken even before first VFS commits happeing around the end of December. Those commits exposed some problems about NTFS which are currently under investigation. Ultimately, This filesystem is also unmaintained at the moment. Speaking with jeff, we agreed on what can be a possible compromise: remove the kernel support for NTFS and maybe take care of the FUSE implementation. What I now propose is a small survey which can shade a light on us about what do you think about this idea and its implications: - Do you use NTFS? - Are you interested in maintaining it? - Do you know a good reason to not use FUSE ntfs implementation? What the kernel counter part adds? - Do you think axing the kernel support a good idea? Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 01:32:04 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 41AA616A41B; Thu, 7 Feb 2008 01:32:04 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.delphij.net (delphij-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:2c9::2]) by mx1.freebsd.org (Postfix) with ESMTP id CEA1313C448; Thu, 7 Feb 2008 01:32:03 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.geekcn.org (unknown [202.108.54.204]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tarsier.delphij.net (Postfix) with ESMTP id EA90D28448; Thu, 7 Feb 2008 09:32:02 +0800 (CST) Received: from localhost (unknown [202.108.54.204]) by tarsier.geekcn.org (Postfix) with ESMTP id 91757EC3651; Thu, 7 Feb 2008 09:32:02 +0800 (CST) X-Virus-Scanned: amavisd-new at geekcn.org Received: from tarsier.geekcn.org ([202.108.54.204]) by localhost (mail.geekcn.org [202.108.54.204]) (amavisd-new, port 10024) with ESMTP id SW30hY5-j3HC; Thu, 7 Feb 2008 09:31:57 +0800 (CST) Received: from charlie.delphij.net (71.5.7.139.ptr.us.xo.net [71.5.7.139]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTP id AC641EC3609; Thu, 7 Feb 2008 09:31:55 +0800 (CST) DomainKey-Signature: a=rsa-sha1; s=default; d=delphij.net; c=nofws; q=dns; h=message-id:date:from:reply-to:organization:user-agent: mime-version:to:cc:subject:references:in-reply-to: x-enigmail-version:openpgp:content-type:content-transfer-encoding; b=mAbi3wCRgKz9I3tp9XRZvxY0jiy+k0MHkIpJmzS2vuSVN1qBgFXEzLxm3QiYGW7e4 YK+grpS8VWNDo4qYlRHOw== Message-ID: <47AA5F87.6040905@delphij.net> Date: Wed, 06 Feb 2008 17:31:51 -0800 From: Xin LI Organization: The FreeBSD Project User-Agent: Thunderbird 2.0.0.9 (X11/20080122) MIME-Version: 1.0 To: Attilio Rao References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> In-Reply-To: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> X-Enigmail-Version: 0.95.5 OpenPGP: id=18EDEBA0; url=http://www.delphij.net/delphij.asc Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Yar Tikhiy , Doug Barton , Jeff Roberson , freebsd-fs@freebsd.org, Scot Hetzel , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: d@delphij.net List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 01:32:04 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Attilio Rao wrote: > As exposed by several users, NTFS seems to be broken even before first > VFS commits happeing around the end of December. Those commits exposed > some problems about NTFS which are currently under investigation. > Ultimately, This filesystem is also unmaintained at the moment. > > Speaking with jeff, we agreed on what can be a possible compromise: > remove the kernel support for NTFS and maybe take care of the FUSE > implementation. > What I now propose is a small survey which can shade a light on us > about what do you think about this idea and its implications: > - Do you use NTFS? I use it occasionally when I need to copy files from the Windows partition. > - Are you interested in maintaining it? No. > - Do you know a good reason to not use FUSE ntfs implementation? What > the kernel counter part adds? No, the only reason come to mind is that one don't need to install separately, and thus might be good if you run -CURRENT. > - Do you think axing the kernel support a good idea? If nobody would maintain it, as long as the FUSE NTFS implementation is usable I think it's a good idea to axe it in 8.0. Cheers, - -- Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHql+Hi+vbBBjt66ARAg0bAJ4lsJRyQrRGayvskC41dMCwDuwCfACffhca BcYVAWhFBciKHIeEYWmDSi0= =YhRg -----END PGP SIGNATURE----- From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 02:32:12 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8477216A41B; Thu, 7 Feb 2008 02:32:12 +0000 (UTC) (envelope-from ganbold@micom.mng.net) Received: from publicd.ub.mng.net (publicd.ub.mng.net [202.179.0.88]) by mx1.freebsd.org (Postfix) with ESMTP id 2702413C457; Thu, 7 Feb 2008 02:32:11 +0000 (UTC) (envelope-from ganbold@micom.mng.net) Received: from [202.179.0.164] (helo=daemon.micom.mng.net) by publicd.ub.mng.net with esmtpa (Exim 4.67 (FreeBSD)) (envelope-from ) id 1JMvq2-000KBi-2A; Thu, 07 Feb 2008 09:46:06 +0800 Message-ID: <47AA62DD.5050704@micom.mng.net> Date: Thu, 07 Feb 2008 09:46:05 +0800 From: Ganbold User-Agent: Thunderbird 2.0.0.9 (X11/20071225) MIME-Version: 1.0 To: Attilio Rao References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> In-Reply-To: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Yar Tikhiy , Doug Barton , Jeff Roberson , freebsd-fs@freebsd.org, Scot Hetzel , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 02:32:12 -0000 Attilio Rao wrote: > As exposed by several users, NTFS seems to be broken even before first > VFS commits happeing around the end of December. Those commits exposed > some problems about NTFS which are currently under investigation. > Ultimately, This filesystem is also unmaintained at the moment. > > Speaking with jeff, we agreed on what can be a possible compromise: > remove the kernel support for NTFS and maybe take care of the FUSE > implementation. > What I now propose is a small survey which can shade a light on us > about what do you think about this idea and its implications: > - Do you use NTFS? > Yes, I have external NTFS USB hard drive and from time to time I need to copy some files, delete etc. > - Are you interested in maintaining it? > - Do you know a good reason to not use FUSE ntfs implementation? FUSE implementation allows to write to NTFS partition, so it is really useful. However either fusefs-ntfs related stuff or kernel itself seems like buggy, I'm having fatal trap/crash when I try to mount NTFS partition in FreeBSD-7.0-PRERELEASE. > What > the kernel counter part adds? > - Do you think axing the kernel support a good idea? > Maybe, since there is already a port fusefs-ntfs. Ganbold > Thanks, > Attilio > > > -- Alex Haley was adopted! From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 03:10:06 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23F6216A418 for ; Thu, 7 Feb 2008 03:10:06 +0000 (UTC) (envelope-from matt@gsicomp.on.ca) Received: from daisy2.compar.com (mail1.compar.com [216.208.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id CCB8313C43E for ; Thu, 7 Feb 2008 03:10:05 +0000 (UTC) (envelope-from matt@gsicomp.on.ca) Received: from localhost (localhost.compar.com [127.0.0.1]) by daisy2.compar.com (Postfix) with ESMTP id 844BE13C48A; Wed, 6 Feb 2008 22:10:05 -0500 (EST) X-Virus-Scanned: amavisd-new at compar.com Received: from unknown by localhost (amavisd-new, unix socket) id qyheHgjwNJDC; Wed, 6 Feb 2008 22:10:02 -0500 (EST) Received: from hermes (CPE00062566c7bb-CM001ac3584898.cpe.net.cable.rogers.com [99.236.43.116]) by daisy2.compar.com (Postfix) with SMTP id 5CB1413C488; Wed, 6 Feb 2008 22:10:01 -0500 (EST) Message-ID: <002901c86936$eeb3cb50$1200a8c0@hermes> From: "Matt Emmerton" To: "Attilio Rao" , , References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> Date: Wed, 6 Feb 2008 22:10:02 -0500 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 Cc: Jeff Roberson , Scot Hetzel , Yar Tikhiy , Doug Barton Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 03:10:06 -0000 > Speaking with jeff, we agreed on what can be a possible compromise: > remove the kernel support for NTFS and maybe take care of the FUSE > implementation. > What I now propose is a small survey which can shade a light on us > about what do you think about this idea and its implications: > - Do you use NTFS? Sometimes. > - Are you interested in maintaining it? Nope. > - Do you know a good reason to not use FUSE ntfs implementation? What > the kernel counter part adds? FUSE would be fine by me. I didn't realize FUSE was available on FreeBSD -- in the past I typically booted with a Knoppix livecd and then tar over netcat to get files back to my BSD machines. I'll use FUSE on FreeBSD from now on. > - Do you think axing the kernel support a good idea? Yep. -- Matt Emmerton From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 07:00:16 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 19EB216A421 for ; Thu, 7 Feb 2008 07:00:16 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from mail2.fluidhosting.com (mx24.fluidhosting.com [204.14.89.7]) by mx1.freebsd.org (Postfix) with SMTP id 7FAF013C46E for ; Thu, 7 Feb 2008 07:00:15 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: (qmail 5473 invoked by uid 399); 7 Feb 2008 06:33:35 -0000 Received: from localhost (HELO ?192.168.0.4?) (dougb@dougbarton.us@127.0.0.1) by localhost with ESMTP; 7 Feb 2008 06:33:35 -0000 X-Originating-IP: 127.0.0.1 Message-ID: <47AAA63C.4040504@FreeBSD.org> Date: Wed, 06 Feb 2008 22:33:32 -0800 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Attilio Rao References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> In-Reply-To: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Jeff Roberson , Scot Hetzel , Yar Tikhiy , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 07:00:16 -0000 Attilio Rao wrote: > As exposed by several users, NTFS seems to be broken even before first > VFS commits happeing around the end of December. Those commits exposed > some problems about NTFS which are currently under investigation. > Ultimately, This filesystem is also unmaintained at the moment. > > Speaking with jeff, we agreed on what can be a possible compromise: > remove the kernel support for NTFS and maybe take care of the FUSE > implementation. > What I now propose is a small survey which can shade a light on us > about what do you think about this idea and its implications: > - Do you use NTFS? When it works, yes, I use it often. > - Are you interested in maintaining it? EOVERMYHEAD > - Do you know a good reason to not use FUSE ntfs implementation? What > the kernel counter part adds? Two, first I prefer to eat our own dog food, and I rarely need to write to the NTFS partition (although it does sometimes come in handy). And second the last time I tried to use it in -current it panicked too, but that was some time ago. > - Do you think axing the kernel support a good idea? I think we should do it right, or not do it in the base. We should have _one_ working implementation, if that is FUSE then that's fine with me. Doug -- This .signature sanitized for your protection From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 07:26:25 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3132216A418; Thu, 7 Feb 2008 07:26:25 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 07C4413C4DD; Thu, 7 Feb 2008 07:26:24 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id 6408B1A4D83; Wed, 6 Feb 2008 23:13:14 -0800 (PST) Date: Wed, 6 Feb 2008 23:13:14 -0800 From: Alfred Perlstein To: Attilio Rao Message-ID: <20080207071314.GO99258@elvis.mu.org> References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> User-Agent: Mutt/1.4.2.3i Cc: Yar Tikhiy , Doug Barton , Jeff Roberson , freebsd-fs@freebsd.org, Scot Hetzel , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 07:26:25 -0000 * Attilio Rao [080206 17:00] wrote: > As exposed by several users, NTFS seems to be broken even before first > VFS commits happeing around the end of December. Those commits exposed > some problems about NTFS which are currently under investigation. > Ultimately, This filesystem is also unmaintained at the moment. > > Speaking with jeff, we agreed on what can be a possible compromise: > remove the kernel support for NTFS and maybe take care of the FUSE > implementation. > What I now propose is a small survey which can shade a light on us > about what do you think about this idea and its implications: > - Do you use NTFS? > - Are you interested in maintaining it? > - Do you know a good reason to not use FUSE ntfs implementation? What > the kernel counter part adds? > - Do you think axing the kernel support a good idea? My pragmatic view on this is that I think it's odd that something that is sort-of working for a few people is going to be axed by people that don't use it, while promising to replace it with something better. Maybe a nicer way of saying/asking would be to ask: Is the FUSE replacement going to be tested to the point where it's better than then current NTFS code? thanks, -- - Alfred Perlstein From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 08:19:01 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3F2EB16A46E for ; Thu, 7 Feb 2008 08:19:01 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from mail2.fluidhosting.com (mx24.fluidhosting.com [204.14.89.7]) by mx1.freebsd.org (Postfix) with SMTP id BA1C113C4D3 for ; Thu, 7 Feb 2008 08:19:00 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: (qmail 7941 invoked by uid 399); 7 Feb 2008 08:18:59 -0000 Received: from localhost (HELO ?192.168.0.4?) (dougb@dougbarton.us@127.0.0.1) by localhost with ESMTP; 7 Feb 2008 08:18:59 -0000 X-Originating-IP: 127.0.0.1 Message-ID: <47AABEEF.7020807@FreeBSD.org> Date: Thu, 07 Feb 2008 00:18:55 -0800 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Alfred Perlstein References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <20080207071314.GO99258@elvis.mu.org> In-Reply-To: <20080207071314.GO99258@elvis.mu.org> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Yar Tikhiy , Scot Hetzel , Jeff Roberson , Attilio Rao , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 08:19:01 -0000 Alfred Perlstein wrote: > Maybe a nicer way of saying/asking would be to ask: > > Is the FUSE replacement going to be tested to the point where it's > better than then current NTFS code? Given that the current NTFS code in the base panics within minutes of any kind of serious access, and has the ability to take the other filesystems down with it (including UFS2) that won't be hard. Doug -- This .signature sanitized for your protection From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 10:04:48 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 139C316A420 for ; Thu, 7 Feb 2008 10:04:48 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.190]) by mx1.freebsd.org (Postfix) with ESMTP id 83E1E13C468 for ; Thu, 7 Feb 2008 10:04:47 +0000 (UTC) (envelope-from uspoerlein@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so974184nfb.33 for ; Thu, 07 Feb 2008 02:04:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=P+7NVKuyU+V5cHtPwPm/xKOJXFlCNysV2DUt9RaHey4=; b=WEIN3woWnebbZ7pgLN7Cgew49vqHhjiY/CCqRXxCu4I1XWNYjbjzWKPTIh3u9R6ZIROm4ifgD6VHHQVk4uR5arenN3Y2eCQlX2DkdaeP8LwTwFV1nbKibM7NdDQCIp/egin6Lj8xk+ngsrBeMypltkwvsPtQYxe+q6QoCNJdtEE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=ESbzGUYNXq+AouEOCMRWKn9gu/Zy1adtQ5O2Bab0YvxUxcIt5M4oNTpR83J6kCXJuFJ/hUwnniuR1aKYxab+QJX92zlX/d/uUtQoOI/G+aTzeJnXcBpU1S9YIN60jN5lS81pGDtjg9MudvC8Zcd2tJYVupTTdT6tTduGTkzQnf0= Received: by 10.78.204.7 with SMTP id b7mr19658321hug.74.1202376934099; Thu, 07 Feb 2008 01:35:34 -0800 (PST) Received: by 10.78.173.4 with HTTP; Thu, 7 Feb 2008 01:35:34 -0800 (PST) Message-ID: <7ad7ddd90802070135y6eeaded6lcffc6b5751ee5c40@mail.gmail.com> Date: Thu, 7 Feb 2008 10:35:34 +0100 From: "Ulrich Spoerlein" To: "Doug Barton" In-Reply-To: <47AABEEF.7020807@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <20080207071314.GO99258@elvis.mu.org> <47AABEEF.7020807@FreeBSD.org> Cc: Attilio Rao , Yar Tikhiy , Scot Hetzel , Jeff Roberson , Alfred Perlstein , freebsd-fs@freebsd.org, freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 10:04:48 -0000 On Feb 7, 2008 9:18 AM, Doug Barton wrote: > Alfred Perlstein wrote: > > > Maybe a nicer way of saying/asking would be to ask: > > > > Is the FUSE replacement going to be tested to the point where it's > > better than then current NTFS code? > > Given that the current NTFS code in the base panics within minutes of > any kind of serious access, and has the ability to take the other > filesystems down with it (including UFS2) that won't be hard. Please keep in mind, that the current FUSE port to FreeBSD is lagging behind and I can reproducibly panic -CURRENT using fuse-ntfs, too. Nevertheless, I think FUSE is the way to go. Perhaps someone could port the NetBSD interface for it to FreeBSD, so we can ship a kernel with FUSE preinstalled (sans GPL). Cheers, Uli From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 10:41:53 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A0E2B16A468 for ; Thu, 7 Feb 2008 10:41:53 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.153]) by mx1.freebsd.org (Postfix) with ESMTP id 0780F13C465 for ; Thu, 7 Feb 2008 10:41:52 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by fg-out-1718.google.com with SMTP id 16so2789787fgg.35 for ; Thu, 07 Feb 2008 02:41:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=C94noNvwVnXJtnnhMt2sdcAc6/kOXcq58DrlyTom694=; b=NUgyHORW3gt3bVUOeHAeAJ959DOt8SmxOxlLwjo1BQbw4+VB6CDrXtN1wTPzj38rUddpM6fVe2dqE4l4Ls5CMGUV6jJxmw+hDj8PvCbJivTZqoiFzqpIoh3Owvy0FBmsWnxdIp6coJEta8rdkk/NfQz9/+VSrM2yph8ipJ53DwM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=g2quMsB2JY/Lc1Mp7CcK0qx4aKoKowZRlC3k2LubqOWC/PTmF0U+f+7SU66GJM1XgelyRnDcJ0ZolpLJsttiWXNcLsvIfckSfxyc/4fsH0zP3xznKHSNjQqasWLuk/2fpNdVM7sWjb17s+TMSqL3HiOcrfz9z6lPKs2BP/QzYsQ= Received: by 10.86.77.5 with SMTP id z5mr10245974fga.41.1202380911665; Thu, 07 Feb 2008 02:41:51 -0800 (PST) Received: by 10.86.28.19 with HTTP; Thu, 7 Feb 2008 02:41:51 -0800 (PST) Message-ID: <3bbf2fe10802070241m6026ad63w78788d83acb48a8@mail.gmail.com> Date: Thu, 7 Feb 2008 11:41:51 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Alfred Perlstein" In-Reply-To: <20080207071314.GO99258@elvis.mu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <20080207071314.GO99258@elvis.mu.org> X-Google-Sender-Auth: 2ad8ee6dd56e829c Cc: Yar Tikhiy , Doug Barton , Jeff Roberson , freebsd-fs@freebsd.org, Scot Hetzel , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 10:41:53 -0000 2008/2/7, Alfred Perlstein : > * Attilio Rao [080206 17:00] wrote: > > As exposed by several users, NTFS seems to be broken even before first > > VFS commits happeing around the end of December. Those commits exposed > > some problems about NTFS which are currently under investigation. > > Ultimately, This filesystem is also unmaintained at the moment. > > > > Speaking with jeff, we agreed on what can be a possible compromise: > > remove the kernel support for NTFS and maybe take care of the FUSE > > implementation. > > What I now propose is a small survey which can shade a light on us > > about what do you think about this idea and its implications: > > - Do you use NTFS? > > - Are you interested in maintaining it? > > - Do you know a good reason to not use FUSE ntfs implementation? What > > the kernel counter part adds? > > - Do you think axing the kernel support a good idea? > > > My pragmatic view on this is that I think it's odd that something > that is sort-of working for a few people is going to be axed by > people that don't use it, while promising to replace it with something > better. > > Maybe a nicer way of saying/asking would be to ask: > > Is the FUSE replacement going to be tested to the point where it's > better than then current NTFS code? It is possible for you to reply without a flammable answer to any question? In particular in this case where it is just a polling... Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 11:49:38 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 02ACF16A417 for ; Thu, 7 Feb 2008 11:49:38 +0000 (UTC) (envelope-from joao.barros@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.189]) by mx1.freebsd.org (Postfix) with ESMTP id 7761013C459 for ; Thu, 7 Feb 2008 11:49:37 +0000 (UTC) (envelope-from joao.barros@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so991526nfb.33 for ; Thu, 07 Feb 2008 03:49:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=OtxQQ+/UWwyOnNk8u0zbPULQr+jma0JIdwIkhAFc4u8=; b=Pvf3LXW1qWzbytvD42zmzviF8MT8NVWW5jFCNtArAvs2cOSQrRbrHv+ysxP01bZSq6ZMo+yC3MK5ysSqEfMY9jUJ+hd1pXe5qizMf4zdt5E8H1EfMhrcSXbROH81NNhgdcMqUtKYYwOc6d0ZJmB215rtt/hVvcnGnIkODHVvbs8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=bbwMgCIKEETjZ0WNygdtUsUOFHEXx53lIkU90B+wMz3IDFj8hkyfTFWyYYQMP0MCtvAEvlayZnpKAYNblylXowL0akAVpZTXdrre40tnyCdjibGSqGr3D+k2OZQCuJ4YQvOuJjOzywuoPICavE+in8mVdcZZl+WI1UcSFIqKVPI= Received: by 10.78.172.20 with SMTP id u20mr19923628hue.13.1202383294236; Thu, 07 Feb 2008 03:21:34 -0800 (PST) Received: by 10.78.189.6 with HTTP; Thu, 7 Feb 2008 03:21:34 -0800 (PST) Message-ID: <70e8236f0802070321n9097d3fy1b39f637b3c2a06@mail.gmail.com> Date: Thu, 7 Feb 2008 11:21:34 +0000 From: "Joao Barros" To: "Attilio Rao" In-Reply-To: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> Cc: Yar Tikhiy , Doug Barton , Jeff Roberson , freebsd-fs@freebsd.org, Scot Hetzel , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 11:49:38 -0000 On Feb 7, 2008 1:00 AM, Attilio Rao wrote: > As exposed by several users, NTFS seems to be broken even before first > VFS commits happeing around the end of December. Those commits exposed > some problems about NTFS which are currently under investigation. > Ultimately, This filesystem is also unmaintained at the moment. > > Speaking with jeff, we agreed on what can be a possible compromise: > remove the kernel support for NTFS and maybe take care of the FUSE > implementation. > What I now propose is a small survey which can shade a light on us > about what do you think about this idea and its implications: > - Do you use NTFS? Yes. Important in a dual booting enviroment. > - Are you interested in maintaining it? I would If I had the needed knowledge in FS and Kernel. I only have availability to offer. > - Do you know a good reason to not use FUSE ntfs implementation? What > the kernel counter part adds? Yes: Speed. A year ago when building my zfs box I had to migrate 500GB of data off NTFS. FUSE ntfs is WAY slow. I didn't do a proper benchmark then but I could setup something now if interest arises. I didn't have any problems like those being reported with CURRENT from April 2007 if I recall the date correctly when I copied all that data. > - Do you think axing the kernel support a good idea? Yes if... Yes if FUSE ntfs can have performance on par with the current ntfs support. Yes if FUSE ntfs license model doesn't become an issue. Yes because FUSE ntfs write support is neat =) -- Joao Barros From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 12:47:44 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F99416A41A for ; Thu, 7 Feb 2008 12:47:44 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from ns.trinitel.com (186.161.36.72.static.reverse.ltdomains.com [72.36.161.186]) by mx1.freebsd.org (Postfix) with ESMTP id C6A6913C45B for ; Thu, 7 Feb 2008 12:47:43 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from proton.storspeed.com (209-163-168-124.static.tenantsolutions.com [209.163.168.124] (may be forged)) (authenticated bits=0) by ns.trinitel.com (8.14.1/8.14.1) with ESMTP id m17Clfm9082473; Thu, 7 Feb 2008 06:47:42 -0600 (CST) (envelope-from anderson@freebsd.org) Message-ID: <47AAFDED.9030301@freebsd.org> Date: Thu, 07 Feb 2008 06:47:41 -0600 From: Eric Anderson User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Attilio Rao References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> In-Reply-To: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.1 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on ns.trinitel.com Cc: Yar Tikhiy , Doug Barton , Jeff Roberson , freebsd-fs@freebsd.org, Scot Hetzel , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 12:47:44 -0000 Attilio Rao wrote: > As exposed by several users, NTFS seems to be broken even before first > VFS commits happeing around the end of December. Those commits exposed > some problems about NTFS which are currently under investigation. > Ultimately, This filesystem is also unmaintained at the moment. > > Speaking with jeff, we agreed on what can be a possible compromise: > remove the kernel support for NTFS and maybe take care of the FUSE > implementation. > What I now propose is a small survey which can shade a light on us > about what do you think about this idea and its implications: > - Do you use NTFS? Yes, however not often, but when I do use it, I need it in the base OS really. > - Are you interested in maintaining it? Possibly. I would really need to look into it a bit more. > - Do you know a good reason to not use FUSE ntfs implementation? What > the kernel counter part adds? FUSE is slow, requires a port (unless PUFFS is ported, which I've probed about before). > - Do you think axing the kernel support a good idea? I think Alfred's point is really interesting. How many people that don't use it that say 'axe it' does it take to override 1 person saying 'keep it!'? Personally, I much prefer having it in the base. Eric From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 13:47:05 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 381A916A474 for ; Thu, 7 Feb 2008 13:47:05 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 9263D13C46A for ; Thu, 7 Feb 2008 13:47:04 +0000 (UTC) (envelope-from andre@freebsd.org) Received: (qmail 88078 invoked from network); 7 Feb 2008 12:39:01 -0000 Received: from localhost (HELO [127.0.0.1]) ([127.0.0.1]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 7 Feb 2008 12:39:01 -0000 Message-ID: <47AB05A1.7010803@freebsd.org> Date: Thu, 07 Feb 2008 14:20:33 +0100 From: Andre Oppermann User-Agent: Thunderbird 1.5.0.14 (Windows/20071210) MIME-Version: 1.0 To: Eric Anderson References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <47AAFDED.9030301@freebsd.org> In-Reply-To: <47AAFDED.9030301@freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Yar Tikhiy , Doug Barton , Jeff Roberson , Attilio Rao , Scot Hetzel , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 13:47:05 -0000 Eric Anderson wrote: > I think Alfred's point is really interesting. How many people that > don't use it that say 'axe it' does it take to override 1 person saying > 'keep it!'? The real question is how many people does it take to say 'I'll maintain it'? Just one. Without it, it will only bitrot as evidenced by Attilios question. NTFS is currently broken, just not as obvious because WITNESS didn't track and enforce lockmgr locks. -- Andre From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 14:13:50 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E0D2E16A475 for ; Thu, 7 Feb 2008 14:13:50 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.188]) by mx1.freebsd.org (Postfix) with ESMTP id 639D713C474 for ; Thu, 7 Feb 2008 14:13:49 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so1018598nfb.33 for ; Thu, 07 Feb 2008 06:13:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=3UN7cZ/LHrPUmp1cnztgkuaME35GdS2Z2TYVkFtdCDw=; b=dPUe6xdXnUO6kU89IrceUOnLgtbtuJL9Mg02CSYK7IbQOOHvKzOXZRVcHyabbLPqpFlbRVI/w8K33E32FOe6WD0iacnM8RHiqAWEZVwwVaZd45WYkGA5Cgjp1kFudJbzWkol4oyJ+9aABZbGqNb7jDBcQvgbdQ8oJaLG4Q3bM8k= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=ELdGt3Ke23Vi9NvEosCNT7O1zZfhL8lVqKEXKjYNrjlBMQ2mVV/ggQceFWeJJkY3NcZDnCT8AKHyuktuy686cjbwNYdoUIYxE31q7IQKbqr1GtFZ0ExTc20WSEngZ0IFarwY/zTJC3JkSimKwBK2QC5vriHj7zBL2+mZDuseesE= Received: by 10.86.77.5 with SMTP id z5mr10450631fga.41.1202393628793; Thu, 07 Feb 2008 06:13:48 -0800 (PST) Received: by 10.86.28.19 with HTTP; Thu, 7 Feb 2008 06:13:48 -0800 (PST) Message-ID: <3bbf2fe10802070613mf2bf3feg5dcb480501fcfbbc@mail.gmail.com> Date: Thu, 7 Feb 2008 15:13:48 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Andre Oppermann" In-Reply-To: <47AB05A1.7010803@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <47AAFDED.9030301@freebsd.org> <47AB05A1.7010803@freebsd.org> X-Google-Sender-Auth: 9197a749fbe5dc62 Cc: Yar Tikhiy , Scot Hetzel , Jeff Roberson , freebsd-fs@freebsd.org, Doug Barton , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 14:13:51 -0000 2008/2/7, Andre Oppermann : > Eric Anderson wrote: > > I think Alfred's point is really interesting. How many people that > > don't use it that say 'axe it' does it take to override 1 person saying > > 'keep it!'? > > The real question is how many people does it take to say 'I'll maintain > it'? Just one. Without it, it will only bitrot as evidenced by Attilios > question. NTFS is currently broken, just not as obvious because WITNESS > didn't track and enforce lockmgr locks. Andre catched exactly my point. The big problem is that we have a list of several unmaintained fs. NTFS is in this list. The support is not reliable, it is only available in read mode and eventually bugged. I'm not sure I want to keep this if nobody wants to maintain it. Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 14:18:21 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3DB5616A41B; Thu, 7 Feb 2008 14:18:21 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 140E513C465; Thu, 7 Feb 2008 14:18:21 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id 063421A4D7E; Thu, 7 Feb 2008 06:18:21 -0800 (PST) Date: Thu, 7 Feb 2008 06:18:20 -0800 From: Alfred Perlstein To: Attilio Rao Message-ID: <20080207141820.GR99258@elvis.mu.org> References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <47AAFDED.9030301@freebsd.org> <47AB05A1.7010803@freebsd.org> <3bbf2fe10802070613mf2bf3feg5dcb480501fcfbbc@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3bbf2fe10802070613mf2bf3feg5dcb480501fcfbbc@mail.gmail.com> User-Agent: Mutt/1.4.2.3i Cc: Yar Tikhiy , Scot Hetzel , Andre Oppermann , Jeff Roberson , freebsd-fs@freebsd.org, Doug Barton , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 14:18:21 -0000 * Attilio Rao [080207 06:13] wrote: > 2008/2/7, Andre Oppermann : > > Eric Anderson wrote: > > > I think Alfred's point is really interesting. How many people that > > > don't use it that say 'axe it' does it take to override 1 person saying > > > 'keep it!'? > > > > The real question is how many people does it take to say 'I'll maintain > > it'? Just one. Without it, it will only bitrot as evidenced by Attilios > > question. NTFS is currently broken, just not as obvious because WITNESS > > didn't track and enforce lockmgr locks. > > Andre catched exactly my point. > The big problem is that we have a list of several unmaintained fs. > NTFS is in this list. The support is not reliable, it is only > available in read mode and eventually bugged. > I'm not sure I want to keep this if nobody wants to maintain it. All I'm saying is that I think this is a bit premature considering the users. Within less than 24hrs we've had a few users reporting in as users, I'm sure the fixes (now that we have some good assertions) are going to be trivial. Why not let it ferment/rot for a release cycle and then see what the story is? thanks, -- - Alfred Perlstein From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 14:21:06 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B6F2A16A41B for ; Thu, 7 Feb 2008 14:21:06 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mu-out-0910.google.com (mu-out-0910.google.com [209.85.134.184]) by mx1.freebsd.org (Postfix) with ESMTP id 1D3A313C502 for ; Thu, 7 Feb 2008 14:21:05 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by mu-out-0910.google.com with SMTP id w9so2558683mue.6 for ; Thu, 07 Feb 2008 06:21:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=FK33ihQ08EWI7RTYsGilBvE8HTNip5xxyypZ/4YWWys=; b=EENBQwb7Q1L4uKqCQXvhqsQ3I29R49T3Msf8zOJHyFFdxFYYsb0FrxveDj3G968bPzEPQglWwhoYMs/W7bCmRzVHL/2N9lMfPiguhGgfux9exa9Fao1xeeqD08jkIjTClySQddyf3zByZgxYNo1Y1/5Z/aPKO5vQ8UOdSMn4EWo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=rE1r151PzV6+qr/TKRe5EXaqbWH6MV5nOtV+ADLLoT48ZCbv+HtjJ8HIRDxHddI3pFXHXHz38gJ4KL1MQfequCSAZwQSipX4wiuMdGyXyZsIL1+FNntVGZbTpHu4e8wQoM8TX/2NGvJvd0MiW7ZKqygYKs8/uBYUFh+XW82B2wk= Received: by 10.82.118.2 with SMTP id q2mr20602538buc.23.1202394064508; Thu, 07 Feb 2008 06:21:04 -0800 (PST) Received: by 10.86.28.19 with HTTP; Thu, 7 Feb 2008 06:21:04 -0800 (PST) Message-ID: <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> Date: Thu, 7 Feb 2008 15:21:04 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Alfred Perlstein" In-Reply-To: <20080207141820.GR99258@elvis.mu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <47AAFDED.9030301@freebsd.org> <47AB05A1.7010803@freebsd.org> <3bbf2fe10802070613mf2bf3feg5dcb480501fcfbbc@mail.gmail.com> <20080207141820.GR99258@elvis.mu.org> X-Google-Sender-Auth: 5062320f97659437 Cc: Yar Tikhiy , Scot Hetzel , Andre Oppermann , Jeff Roberson , freebsd-fs@freebsd.org, Doug Barton , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 14:21:06 -0000 2008/2/7, Alfred Perlstein : > * Attilio Rao [080207 06:13] wrote: > > 2008/2/7, Andre Oppermann : > > > Eric Anderson wrote: > > > > I think Alfred's point is really interesting. How many people that > > > > don't use it that say 'axe it' does it take to override 1 person saying > > > > 'keep it!'? > > > > > > The real question is how many people does it take to say 'I'll maintain > > > it'? Just one. Without it, it will only bitrot as evidenced by Attilios > > > question. NTFS is currently broken, just not as obvious because WITNESS > > > didn't track and enforce lockmgr locks. > > > > Andre catched exactly my point. > > The big problem is that we have a list of several unmaintained fs. > > NTFS is in this list. The support is not reliable, it is only > > available in read mode and eventually bugged. > > I'm not sure I want to keep this if nobody wants to maintain it. > > All I'm saying is that I think this is a bit premature considering > the users. Within less than 24hrs we've had a few users reporting > in as users, I'm sure the fixes (now that we have some good assertions) > are going to be trivial. > > Why not let it ferment/rot for a release cycle and then see what > the story is? Obviously if we can fix it is better, but axing is an opportunity I don't want to leave out and this is why I wanted to poll users about this issue. Eventually, if an axing is decided, it won't happen in short times but only once all situations for "migration" will be probed and finished. Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 14:36:27 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 46A2216A421 for ; Thu, 7 Feb 2008 14:36:27 +0000 (UTC) (envelope-from joao.barros@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.186]) by mx1.freebsd.org (Postfix) with ESMTP id A923D13C461 for ; Thu, 7 Feb 2008 14:36:26 +0000 (UTC) (envelope-from joao.barros@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so1023276nfb.33 for ; Thu, 07 Feb 2008 06:36:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=Z+jmF5ddDFls1a4sUKcu/UX5JmSgII2Q2ZzKpt2rIDA=; b=D1UPK+MkSLUSlT8/BJoMnhFmmna9Ngn5Qop6PtXLZemiTqWuoR4WNJzcVZzOcgtV9HyGu1SYCRSA2i4mWIXjMAl5JfP7SK/1AWSRCAwV7gtukqGkfk+lZTSmWbgyTpHzfP/UXTJXcm6kmpEHN6oYCS7zrDz9Vq2DGHMXnA1n7Pk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=a5awSB7MmSE0jsgInUup9ozJy7ONNeyvFmbj+n8mJ3Ojl5U7e3j2EpATXN8NxFqQvJz4u05ByO7zSeC/5nDgY+rexT5Xim54bTgOVQwDgX7oS8+JTATFD3DObRS5RUqKUrdbw09eLUYvRKjo070gETCt90cLfhzRAiZZLNMLptU= Received: by 10.78.149.15 with SMTP id w15mr20277100hud.72.1202394983656; Thu, 07 Feb 2008 06:36:23 -0800 (PST) Received: by 10.78.189.6 with HTTP; Thu, 7 Feb 2008 06:36:23 -0800 (PST) Message-ID: <70e8236f0802070636p1af7495fi8b7e6b07129843c3@mail.gmail.com> Date: Thu, 7 Feb 2008 14:36:23 +0000 From: "Joao Barros" To: "Attilio Rao" In-Reply-To: <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <47AAFDED.9030301@freebsd.org> <47AB05A1.7010803@freebsd.org> <3bbf2fe10802070613mf2bf3feg5dcb480501fcfbbc@mail.gmail.com> <20080207141820.GR99258@elvis.mu.org> <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> Cc: Yar Tikhiy , Scot Hetzel , Andre Oppermann , Jeff Roberson , Alfred Perlstein , freebsd-fs@freebsd.org, Doug Barton , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 14:36:27 -0000 On Feb 7, 2008 2:21 PM, Attilio Rao wrote: > 2008/2/7, Alfred Perlstein : > > * Attilio Rao [080207 06:13] wrote: > > > 2008/2/7, Andre Oppermann : > > > > Eric Anderson wrote: > > > > > I think Alfred's point is really interesting. How many people that > > > > > don't use it that say 'axe it' does it take to override 1 person saying > > > > > 'keep it!'? > > > > > > > > The real question is how many people does it take to say 'I'll maintain > > > > it'? Just one. Without it, it will only bitrot as evidenced by Attilios > > > > question. NTFS is currently broken, just not as obvious because WITNESS > > > > didn't track and enforce lockmgr locks. > > > > > > Andre catched exactly my point. > > > The big problem is that we have a list of several unmaintained fs. > > > NTFS is in this list. The support is not reliable, it is only > > > available in read mode and eventually bugged. > > > I'm not sure I want to keep this if nobody wants to maintain it. > > > > All I'm saying is that I think this is a bit premature considering > > the users. Within less than 24hrs we've had a few users reporting > > in as users, I'm sure the fixes (now that we have some good assertions) > > are going to be trivial. > > > > Why not let it ferment/rot for a release cycle and then see what > > the story is? > > Obviously if we can fix it is better, but axing is an opportunity I > don't want to leave out and this is why I wanted to poll users about > this issue. Eventually, if an axing is decided, it won't happen in > short times but only once all situations for "migration" will be > probed and finished. > > Attilio I read (I think on ntfs-3g website) that one of the developers was working for Apple specifically to fully support NTFS on OSX. If this is to be true, Apple willing to give this code to FreeBSD, would make it another interesting solution IMHO. -- Joao Barros From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 14:51:54 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3814616A418; Thu, 7 Feb 2008 14:51:54 +0000 (UTC) (envelope-from yar@comp.chem.msu.su) Received: from comp.chem.msu.su (comp.chem.msu.su [158.250.32.97]) by mx1.freebsd.org (Postfix) with ESMTP id 7F72D13C455; Thu, 7 Feb 2008 14:51:53 +0000 (UTC) (envelope-from yar@comp.chem.msu.su) Received: from comp.chem.msu.su (localhost [127.0.0.1]) by comp.chem.msu.su (8.13.4/8.13.4) with ESMTP id m17EpnFp016500; Thu, 7 Feb 2008 17:51:49 +0300 (MSK) (envelope-from yar@comp.chem.msu.su) Received: (from yar@localhost) by comp.chem.msu.su (8.13.4/8.13.4/Submit) id m17Epnp7016499; Thu, 7 Feb 2008 17:51:49 +0300 (MSK) (envelope-from yar) Date: Thu, 7 Feb 2008 17:51:48 +0300 From: Yar Tikhiy To: Joao Barros Message-ID: <20080207145148.GL7592@comp.chem.msu.su> References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <47AAFDED.9030301@freebsd.org> <47AB05A1.7010803@freebsd.org> <3bbf2fe10802070613mf2bf3feg5dcb480501fcfbbc@mail.gmail.com> <20080207141820.GR99258@elvis.mu.org> <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> <70e8236f0802070636p1af7495fi8b7e6b07129843c3@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <70e8236f0802070636p1af7495fi8b7e6b07129843c3@mail.gmail.com> User-Agent: Mutt/1.5.9i Cc: freebsd-fs@freebsd.org, Scot Hetzel , Andre Oppermann , Jeff Roberson , Alfred Perlstein , Attilio Rao , Doug Barton , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 14:51:54 -0000 On Thu, Feb 07, 2008 at 02:36:23PM +0000, Joao Barros wrote: > > I read (I think on ntfs-3g website) that one of the developers was > working for Apple specifically to fully support NTFS on OSX. If this > is to be true, Apple willing to give this code to FreeBSD, would make > it another interesting solution IMHO. But NTFS-3G is FUSE-based. OTOH, Mac OS X has a native read-only NTFS driver apparently inherited from FreeBSD. Unfortunately there isn't much for the two projects to share in this area: Just reading from NTFS is almost trivial, and the rest is system-specific VFS stuff. -- Yar From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 14:56:37 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42A1D16A417; Thu, 7 Feb 2008 14:56:37 +0000 (UTC) (envelope-from yar@comp.chem.msu.su) Received: from comp.chem.msu.su (comp.chem.msu.su [158.250.32.97]) by mx1.freebsd.org (Postfix) with ESMTP id 53D1913C45B; Thu, 7 Feb 2008 14:56:36 +0000 (UTC) (envelope-from yar@comp.chem.msu.su) Received: from comp.chem.msu.su (localhost [127.0.0.1]) by comp.chem.msu.su (8.13.4/8.13.4) with ESMTP id m17ESUee015906; Thu, 7 Feb 2008 17:28:30 +0300 (MSK) (envelope-from yar@comp.chem.msu.su) Received: (from yar@localhost) by comp.chem.msu.su (8.13.4/8.13.4/Submit) id m17ESUlS015899; Thu, 7 Feb 2008 17:28:30 +0300 (MSK) (envelope-from yar) Date: Thu, 7 Feb 2008 17:28:29 +0300 From: Yar Tikhiy To: Andre Oppermann Message-ID: <20080207142829.GK7592@comp.chem.msu.su> References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <47AAFDED.9030301@freebsd.org> <47AB05A1.7010803@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47AB05A1.7010803@freebsd.org> User-Agent: Mutt/1.5.9i Cc: freebsd-fs@freebsd.org, Doug Barton , Jeff Roberson , Attilio Rao , Scot Hetzel , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 14:56:37 -0000 On Thu, Feb 07, 2008 at 02:20:33PM +0100, Andre Oppermann wrote: > Eric Anderson wrote: > >I think Alfred's point is really interesting. How many people that > >don't use it that say 'axe it' does it take to override 1 person saying > >'keep it!'? > > The real question is how many people does it take to say 'I'll maintain > it'? Just one. Without it, it will only bitrot as evidenced by Attilios > question. NTFS is currently broken, just not as obvious because WITNESS > didn't track and enforce lockmgr locks. Yeah, uncared-for code dies, rots, and stinks, but I doubt whether the NTFS driver has died, or it is just ill [1]. :-) Being a rather simple, read-only FS driver, it should have fairly good separation between its lower layer, dealing with NTFS internals, and its upper layer, hooking the FS up to our VFS framework. Now the problem seems to be concentrated in the latter, so the layer should just be brought in accord with today's VFS locking semantics. This is a job I hope I can carry out if provided with brief details on what's wrong with NTFS now -- Attilio said he had been investigating the NTFS issues already. [1] I can't help recalling Scene 2, "Bring out your dead," from "Monty Python and the Holy Grail" here. -- Yar From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 15:10:52 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD04816A417 for ; Thu, 7 Feb 2008 15:10:52 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.171]) by mx1.freebsd.org (Postfix) with ESMTP id 4779E13C442 for ; Thu, 7 Feb 2008 15:10:51 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: by ug-out-1314.google.com with SMTP id y2so1013485uge.37 for ; Thu, 07 Feb 2008 07:10:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:received:received:subject:from:to:cc:in-reply-to:references:content-type:date:message-id:mime-version:x-mailer; bh=JIO/5gn+BpAVpXYc0h7Qa9aerl7vX+5FIjw7oiRUR4I=; b=E+sqAD/sg6ltDafDM53Kw0e9BeyLqZmJPLhZj29dkcob7UecgM0/mA3tQsC155mIwfSM0jV6kp2PKdBUQojPvc3xjExZv3eAFO+w/VFz4euXvin0OB+AHsYcoPUTyzI+rs++dZcRy4QikFAjS8qmC5ye8gIN/dtl7ZfSRK4yc+M= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=subject:from:to:cc:in-reply-to:references:content-type:date:message-id:mime-version:x-mailer; b=AaQqxZwYEJ0GEkRnMlvqmVnlyP4cfHsTDj1rSv7LNTLj14ArrQyf3FeFRMQeqP2VYkypOZlzckW2BfAkYVXwNfd/p9yFBIcoioLEC5/iD5sW9rQ41bRfr0Vtz71coVlOJu6ZvcwRhtd2uDpFukp18o4yEyRRFvPWsSje92RR07A= Received: by 10.78.140.17 with SMTP id n17mr20331791hud.47.1202395353231; Thu, 07 Feb 2008 06:42:33 -0800 (PST) Received: from ?127.0.0.1? ( [217.206.187.79]) by mx.google.com with ESMTPS id t2sm13843826gve.3.2008.02.07.06.42.31 (version=SSLv3 cipher=RC4-MD5); Thu, 07 Feb 2008 06:42:32 -0800 (PST) From: Tom Evans To: Attilio Rao In-Reply-To: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-JZcY1Bk03LPL/MIaQ2VE" Date: Thu, 07 Feb 2008 14:42:30 +0000 Message-Id: <1202395350.2126.14.camel@localhost> Mime-Version: 1.0 X-Mailer: Evolution 2.10.2 FreeBSD GNOME Team Port Cc: Yar Tikhiy , Doug Barton , Jeff Roberson , freebsd-fs@freebsd.org, Scot Hetzel , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 15:10:52 -0000 --=-JZcY1Bk03LPL/MIaQ2VE Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On Thu, 2008-02-07 at 02:00 +0100, Attilio Rao wrote: > As exposed by several users, NTFS seems to be broken even before first > VFS commits happeing around the end of December. Those commits exposed > some problems about NTFS which are currently under investigation. > Ultimately, This filesystem is also unmaintained at the moment. >=20 > Speaking with jeff, we agreed on what can be a possible compromise: > remove the kernel support for NTFS and maybe take care of the FUSE > implementation. > What I now propose is a small survey which can shade a light on us > about what do you think about this idea and its implications: > - Do you use NTFS? Yes > - Are you interested in maintaining it? No, wouldn't know how > - Do you know a good reason to not use FUSE ntfs implementation? What > the kernel counter part adds? Didn't work last time I tried, running RELENG_7 and the NTFS support in base does everything I need, don't experience any problems. I don't like the idea of=20 > - Do you think axing the kernel support a good idea? >=20 Truthfully, no not really. If no-one wishes to maintain it/fix it now, then I'd prefer it to be disconnected from the build until it annoys someone enough. Apparently its already broken and unusable, but WFM. /dev/ad4s1 on /mnt/windows (ntfs, local, read-only) If the base NTFS support was axed, would the fuse-kmod, or equivalent, find its way into the base? > Thanks, > Attilio >=20 Tom --=-JZcY1Bk03LPL/MIaQ2VE Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (FreeBSD) iD8DBQBHqxjTlcRvFfyds/cRAqXLAJ9dtvANtiO9qEmVhM/VgZsbFdyYDgCffbWu PSyGQUCq8Hl6+zzcJC6YTEI= =tGRz -----END PGP SIGNATURE----- --=-JZcY1Bk03LPL/MIaQ2VE-- From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 15:21:53 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B97B216A417 for ; Thu, 7 Feb 2008 15:21:53 +0000 (UTC) (envelope-from jhs@berklix.org) Received: from tower.berklix.org (tower.berklix.org [83.236.223.114]) by mx1.freebsd.org (Postfix) with ESMTP id 319B813C45B for ; Thu, 7 Feb 2008 15:21:52 +0000 (UTC) (envelope-from jhs@berklix.org) Received: from js.berklix.net (p549A69A0.dip.t-dialin.net [84.154.105.160]) (authenticated bits=0) by tower.berklix.org (8.13.6/8.13.6) with ESMTP id m17FLhIV035503; Thu, 7 Feb 2008 15:21:44 GMT (envelope-from jhs@berklix.org) Received: from fire.js.berklix.net (fire.js.berklix.net [192.168.91.41]) by js.berklix.net (8.13.8/8.13.8) with ESMTP id m17FN3aw013911; Thu, 7 Feb 2008 16:23:03 +0100 (CET) (envelope-from jhs@berklix.org) Received: from fire.js.berklix.net (localhost [127.0.0.1]) by fire.js.berklix.net (8.13.8/8.13.8) with ESMTP id m17FMrt9047084; Thu, 7 Feb 2008 16:22:58 +0100 (CET) (envelope-from jhs@fire.js.berklix.net) Message-Id: <200802071522.m17FMrt9047084@fire.js.berklix.net> To: Yar Tikhiy From: "Julian Stacey" Organization: http://berklix.com BSD Unix Linux Consultancy, Munich Germany User-agent: EXMH on FreeBSD http://berklix.com/free/ X-URL: http://berklix.com In-reply-to: Your message "Thu, 07 Feb 2008 17:28:29 +0300." <20080207142829.GK7592@comp.chem.msu.su> Date: Thu, 07 Feb 2008 16:22:53 +0100 Sender: jhs@berklix.org Cc: freebsd-fs@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 15:21:53 -0000 > [1] I can't help recalling Scene 2, "Bring out your dead," from > "Monty Python and the Holy Grail" here. Thanks for bringing the laugh of the day :-) From memory, Approx: I'm not dead ! Yes he is, Take him away! No I'm not dead, Honest! He's dead now!, Take him away! PS I too have occasionaly tried to read NTFS FS of someone else's USB external NTFS, Even wanted to write ideally. I too didn't know about FUSE port(s) ( sysutils/fusefs* ) No skills or time to offer though. > This is > a job I hope I can carry out if provided with brief details on > what's wrong with NTFS now -- Attilio said he had been investigating > the NTFS issues already. Well done ! -- Julian Stacey. BSD Unix Linux Net Consultant, Munich. http://berklix.com From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 15:53:31 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2851616A418 for ; Thu, 7 Feb 2008 15:53:31 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from wr-out-0506.google.com (wr-out-0506.google.com [64.233.184.235]) by mx1.freebsd.org (Postfix) with ESMTP id 9FA4813C45A for ; Thu, 7 Feb 2008 15:53:30 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: by wr-out-0506.google.com with SMTP id 68so3076869wri.3 for ; Thu, 07 Feb 2008 07:53:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=bq68j90h68TIK9fv096vRTksM3tVfjmwDFt7m8mZav8=; b=mPzrwM36W9SB1d/qg0Ggcts8S+yKUDTmT/HHXGxCvWuzGa4i60GsPxslk1e4sQtQPwbiSTlN9CYBjqeMyLR3v/BtyqUp04W8z31FqWIx9sTP87nBV4dbfU4ZmfXIprlY7SoQwg63Flb1AbAlbQ+C0WmT9BmgHBUFum0qP0JVDWk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=YqooXH94aInR36D020iaKjKx3XvkTbQP5BcRkF7grH61Q5aDXd8SAs1BwjvczbDteYvYVIdjkgKy6bWAT0PR8kl78nkDWmzePqRMTZbnJ6p8UoP6yiS5vsraLDXyMOShwV4PsDE538nqVGYbCj5NNP5GRrC8oeI8lK0qpEeEqS0= Received: by 10.141.171.6 with SMTP id y6mr7671183rvo.174.1202399608950; Thu, 07 Feb 2008 07:53:28 -0800 (PST) Received: by 10.141.170.18 with HTTP; Thu, 7 Feb 2008 07:53:28 -0800 (PST) Message-ID: <2e77fc10802070753n2f6fc9cbw5b2039e6b79e4323@mail.gmail.com> Date: Thu, 7 Feb 2008 17:53:28 +0200 From: "Niki Denev" Sender: ndenev@gmail.com To: "Artis Caune" In-Reply-To: <47A61DB7.1060506@latnet.lv> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <2e77fc10802020152k2f5385c5w5938d91b1183f8e0@mail.gmail.com> <47A61DB7.1060506@latnet.lv> X-Google-Sender-Auth: e19b9d17b7d0adf2 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS panics X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 15:53:31 -0000 On Feb 3, 2008 10:01 PM, Artis Caune wrote: > Niki Denev wrote: > > I also have this in loader.conf : > > > > vm.kmem_size="1G" > > vm.kmem_size_max="1G" > > Try with: > vm.kmem_size="1500M" > vfs.zfs.arc_max="512M" > vfs.zfs.prefetch_disable="1" > > > > With these : vfs.zfs.arc_max="512M" vfs.zfs.prefetch_disable="1" I got exactly the same panic after about 2 hours of 32 bonnie++ processes in parallel. From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 16:19:45 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DED9316A418 for ; Thu, 7 Feb 2008 16:19:45 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from ns.trinitel.com (186.161.36.72.static.reverse.ltdomains.com [72.36.161.186]) by mx1.freebsd.org (Postfix) with ESMTP id A9E2813C457 for ; Thu, 7 Feb 2008 16:19:45 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from proton.storspeed.com (209-163-168-124.static.tenantsolutions.com [209.163.168.124] (may be forged)) (authenticated bits=0) by ns.trinitel.com (8.14.1/8.14.1) with ESMTP id m17GJcnr061777; Thu, 7 Feb 2008 10:19:40 -0600 (CST) (envelope-from anderson@freebsd.org) Message-ID: <47AB2F9A.8070909@freebsd.org> Date: Thu, 07 Feb 2008 10:19:38 -0600 From: Eric Anderson User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: "fluffles.net" References: <47A95BC0.1060307@fluffles.net> In-Reply-To: <47A95BC0.1060307@fluffles.net> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.2 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on ns.trinitel.com Cc: freebsd-fs@freebsd.org Subject: Re: delayed write buffer X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 16:19:46 -0000 fluffles.net wrote: > Hello kind list, > > I was wondering how to tune FreeBSD's VFS write buffer. I would like a > large amount of RAM (say 500MB out of 1GB) to be reserved or allocated > to be used as write buffer for my backup NAS system. > > If i understand the mechanism correctly, a "dd if=/dev/zero of=/mnt/fs > bs=1m count=500" would act as if i used a malloc-backed md device. This > leads to very nice performance gains, especially in my case because i'm > using encryption causing throughput to be limited to 22MB/s. But if the > first 500MB is free, i can mask this limitation and experience a fast > drive, when writing that is. > > Anyone can point me to the right directions? I tried playing with some > sysctl vfs (notably the vfs.maxmallocbufspace tunable) but did not > achieve the desired effect. And yes, i do know a lot of dirty buffers is > dangerous but my storage setup is redundant enough. Besides i'm just > curious in this topic. :) Could a regular BIO-FLUSH caused by UFS > metadata sync be the curlpit? > > Thanks for any assistance! > Kind greetings, GEOM cache maybe? I can't recall what the status is on it, or what it's real benefits were. I recall it being used for some RAID5 stuff somehow though. Or, how about making a gjournal, with the journal being a MD-backed 'disk'? Eric From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 17:20:53 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5DB9916A46D; Thu, 7 Feb 2008 17:20:53 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 1EDD113C45E; Thu, 7 Feb 2008 17:20:53 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id C14074785A; Thu, 7 Feb 2008 12:20:52 -0500 (EST) Date: Thu, 7 Feb 2008 17:20:52 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Attilio Rao In-Reply-To: <3bbf2fe10802070613mf2bf3feg5dcb480501fcfbbc@mail.gmail.com> Message-ID: <20080207171913.M96200@fledge.watson.org> References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <47AAFDED.9030301@freebsd.org> <47AB05A1.7010803@freebsd.org> <3bbf2fe10802070613mf2bf3feg5dcb480501fcfbbc@mail.gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Yar Tikhiy , Scot Hetzel , Andre Oppermann , Jeff Roberson , freebsd-fs@freebsd.org, Doug Barton , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 17:20:53 -0000 On Thu, 7 Feb 2008, Attilio Rao wrote: > 2008/2/7, Andre Oppermann : > >> Eric Anderson wrote: >>> I think Alfred's point is really interesting. How many people that don't >>> use it that say 'axe it' does it take to override 1 person saying 'keep >>> it!'? >> >> The real question is how many people does it take to say 'I'll maintain >> it'? Just one. Without it, it will only bitrot as evidenced by Attilios >> question. NTFS is currently broken, just not as obvious because WITNESS >> didn't track and enforce lockmgr locks. > > Andre catched exactly my point. The big problem is that we have a list of > several unmaintained fs. NTFS is in this list. The support is not reliable, > it is only available in read mode and eventually bugged. I'm not sure I want > to keep this if nobody wants to maintain it. If you axe write support, does the maintainability of the kernel ntfs get easier? As I understand it, the write support is rather limited, and debugging and fixing read support is generally a lot easier for a variety of reasons. There's also a lot less risk to data. :-) I think it's reasonable to surmise that, given our rather limited write support currently, the kernel ntfs code is used for data migration and limited sharing to FreeBSD in various forms, but that msdofs remains the general data transport of choice... Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 17:40:58 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E70216A46E for ; Thu, 7 Feb 2008 17:40:58 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outF.internet-mail-service.net (outF.internet-mail-service.net [216.240.47.229]) by mx1.freebsd.org (Postfix) with ESMTP id 366BF13C459 for ; Thu, 7 Feb 2008 17:40:58 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Thu, 07 Feb 2008 09:40:57 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id D81591270FD; Thu, 7 Feb 2008 09:40:55 -0800 (PST) Message-ID: <47AB42AB.6020107@elischer.org> Date: Thu, 07 Feb 2008 09:40:59 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Attilio Rao References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <47AAFDED.9030301@freebsd.org> <47AB05A1.7010803@freebsd.org> <3bbf2fe10802070613mf2bf3feg5dcb480501fcfbbc@mail.gmail.com> <20080207141820.GR99258@elvis.mu.org> <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> In-Reply-To: <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Yar Tikhiy , Scot Hetzel , Andre Oppermann , Jeff Roberson , Alfred Perlstein , freebsd-fs@freebsd.org, Doug Barton , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 17:40:58 -0000 Attilio Rao wrote: > 2008/2/7, Alfred Perlstein : >> * Attilio Rao [080207 06:13] wrote: >>> 2008/2/7, Andre Oppermann : >>>> Eric Anderson wrote: >>>>> I think Alfred's point is really interesting. How many people that >>>>> don't use it that say 'axe it' does it take to override 1 person saying >>>>> 'keep it!'? >>>> The real question is how many people does it take to say 'I'll maintain >>>> it'? Just one. Without it, it will only bitrot as evidenced by Attilios >>>> question. NTFS is currently broken, just not as obvious because WITNESS >>>> didn't track and enforce lockmgr locks. >>> Andre catched exactly my point. >>> The big problem is that we have a list of several unmaintained fs. >>> NTFS is in this list. The support is not reliable, it is only >>> available in read mode and eventually bugged. >>> I'm not sure I want to keep this if nobody wants to maintain it. >> All I'm saying is that I think this is a bit premature considering >> the users. Within less than 24hrs we've had a few users reporting >> in as users, I'm sure the fixes (now that we have some good assertions) >> are going to be trivial. >> >> Why not let it ferment/rot for a release cycle and then see what >> the story is? > > Obviously if we can fix it is better, but axing is an opportunity I > don't want to leave out and this is why I wanted to poll users about > this issue. Eventually, if an axing is decided, it won't happen in > short times but only once all situations for "migration" will be > probed and finished. I think axing it would be a mistake. > > Attilio > > From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 18:05:24 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1376C16A46B for ; Thu, 7 Feb 2008 18:05:24 +0000 (UTC) (envelope-from qpadla@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.174]) by mx1.freebsd.org (Postfix) with ESMTP id 83DA313C4E8 for ; Thu, 7 Feb 2008 18:05:23 +0000 (UTC) (envelope-from qpadla@gmail.com) Received: by ug-out-1314.google.com with SMTP id y2so1074798uge.37 for ; Thu, 07 Feb 2008 10:05:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:reply-to:to:subject:date:user-agent:cc:references:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:message-id; bh=z/MklbfQrp57C3iLkRTr3HlI+9bzPk5nmM+4JI2GIn8=; b=Mk3xt473S6yCfm65pb4tm9gNKua4AlYJV3HSuf5IqTxV5hTH9NDdt9+6k4XFaJQMd9FT+79sPwh09pGtHEid9WOMyR0ojcBBZBHBz4UkQWrwBdxeHspAWzIG+XpdEHh/1jroUDy9AnIZVX+8ViQ9lV0EdEwch/D+SMxV5q9plo0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:reply-to:to:subject:date:user-agent:cc:references:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:message-id; b=ZO4nz0Cc22Qr31sjBcMWJsPyf46WUq8nce8W8WVqDfYUVukW0B3KkHMiHeRMQP48AFn3q+U1rBvcPUKl21bu2DfVV1/uR0NgUAgGHZzvlfaq2MdTzK6AdqOpfOPM/jAr/gc+T1njP2PjrfnLji3SXx54P2EieYFBYrxPA7wEw0s= Received: by 10.66.251.20 with SMTP id y20mr4288809ugh.67.1202405985490; Thu, 07 Feb 2008 09:39:45 -0800 (PST) Received: from atlas.local ( [89.162.141.1]) by mx.google.com with ESMTPS id k28sm10179747ugd.77.2008.02.07.09.39.43 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 07 Feb 2008 09:39:44 -0800 (PST) From: Nikolay Pavlov To: freebsd-arch@freebsd.org Date: Thu, 7 Feb 2008 19:39:47 +0200 User-Agent: KMail/1.9.6 (enterprise 0.20070907.709405) References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <70e8236f0802070321n9097d3fy1b39f637b3c2a06@mail.gmail.com> In-Reply-To: <70e8236f0802070321n9097d3fy1b39f637b3c2a06@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200802071939.48039.qpadla@gmail.com> Cc: freebsd-fs@freebsd.org, Doug Barton , Jeff Roberson , Yar Tikhiy , Attilio Rao , Scot Hetzel Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: qpadla@gmail.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 18:05:24 -0000 On Thursday 07 February 2008 13:21:34 Joao Barros wrote: > On Feb 7, 2008 1:00 AM, Attilio Rao wrote: > > As exposed by several users, NTFS seems to be broken even before first > > VFS commits happeing around the end of December. Those commits exposed > > some problems about NTFS which are currently under investigation. > > Ultimately, This filesystem is also unmaintained at the moment. > > > > Speaking with jeff, we agreed on what can be a possible compromise: > > remove the kernel support for NTFS and maybe take care of the FUSE > > implementation. > > What I now propose is a small survey which can shade a light on us > > about what do you think about this idea and its implications: > > - Do you use NTFS? > > Yes. Important in a dual booting enviroment. > > > - Are you interested in maintaining it? > > I would If I had the needed knowledge in FS and Kernel. I only have > availability to offer. > > > - Do you know a good reason to not use FUSE ntfs implementation? What > > the kernel counter part adds? > > Yes: Speed. I think this is related only for FreeBSD: http://www.ntfs-3g.org/performance.html I've used it on linux before and the performance was in pair to kernel implementation. > A year ago when building my zfs box I had to migrate 500GB of data off > NTFS. FUSE ntfs is WAY slow. I didn't do a proper benchmark then but I > could setup something now if interest arises. > I didn't have any problems like those being reported with CURRENT from > April 2007 if I recall the date correctly when I copied all that data. > > > - Do you think axing the kernel support a good idea? > > Yes if... > > Yes if FUSE ntfs can have performance on par with the current ntfs > support. Yes if FUSE ntfs license model doesn't become an issue. > Yes because FUSE ntfs write support is neat =) -- ====================================================================== - Best regards, Nikolay Pavlov. <<<----------------------------------- ====================================================================== From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 18:05:29 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DFEDC16A4AC for ; Thu, 7 Feb 2008 18:05:29 +0000 (UTC) (envelope-from qpadla@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.174]) by mx1.freebsd.org (Postfix) with ESMTP id 6455F13C448 for ; Thu, 7 Feb 2008 18:05:29 +0000 (UTC) (envelope-from qpadla@gmail.com) Received: by ug-out-1314.google.com with SMTP id y2so1074798uge.37 for ; Thu, 07 Feb 2008 10:05:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:reply-to:to:subject:date:user-agent:cc:references:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:message-id; bh=KcEvHuyiKyOYj0PRxSH2SkmL41i3VQ3mi32svLSPjv4=; b=oaSFcIGF+cKkkauFG0wvAVdT7GKSIg1WwuLTL9/cLM/7hZUKaiW8Tv0u0Z1/S6Zq5GDNugq2E7eoqo18shUOXpkb3IvIMU/pwAcHedMsGft7fgv4/yzcyyUCvgAG1Yd2cDng4s6v29tRMdQMVnuE3HgVFTBVU9lffR/RxIK4Gck= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:reply-to:to:subject:date:user-agent:cc:references:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:message-id; b=WAVy2jdMB88QCBet3h2F6HOMpBmDN6m7DU9Qc3JnCBcRSgCkuEZ1wgQq9s/W1lGP+weTE1/SlMGDNpuplonHGh+mKPxcKMvM+I72nv8DK3H8PcnG5C5wfLtrqXVE8Fr7p97WOhuGmUrdm6kC4DZugYqhfuxjMIR4yqsI9kCWBs4= Received: by 10.67.115.9 with SMTP id s9mr4278392ugm.81.1202406080482; Thu, 07 Feb 2008 09:41:20 -0800 (PST) Received: from atlas.local ( [89.162.141.1]) by mx.google.com with ESMTPS id u7sm3062198uge.35.2008.02.07.09.41.17 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 07 Feb 2008 09:41:19 -0800 (PST) From: Nikolay Pavlov To: freebsd-arch@freebsd.org Date: Thu, 7 Feb 2008 19:41:22 +0200 User-Agent: KMail/1.9.6 (enterprise 0.20070907.709405) References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <47AAFDED.9030301@freebsd.org> In-Reply-To: <47AAFDED.9030301@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200802071941.23199.qpadla@gmail.com> Cc: freebsd-fs@freebsd.org, Yar Tikhiy , Doug Barton , Jeff Roberson , Attilio Rao , Scot Hetzel Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: qpadla@gmail.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 18:05:30 -0000 On Thursday 07 February 2008 14:47:41 Eric Anderson wrote: > FUSE is slow, requires a port (unless PUFFS is ported, which I've probed > about before). I think this is not an argument: http://www.ntfs-3g.org/performance.html -- ====================================================================== - Best regards, Nikolay Pavlov. <<<----------------------------------- ====================================================================== From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 18:05:42 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 35B0816A420 for ; Thu, 7 Feb 2008 18:05:42 +0000 (UTC) (envelope-from joao.barros@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.187]) by mx1.freebsd.org (Postfix) with ESMTP id 93F5413C467 for ; Thu, 7 Feb 2008 18:05:41 +0000 (UTC) (envelope-from joao.barros@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so1073902nfb.33 for ; Thu, 07 Feb 2008 10:05:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=ezjvcAe+bLr0s4F0itMGCQZ/FTjWB2MJavwMsBxRWAM=; b=lAwtOQjNyMhwicXMF/m8PVT8vclcOIezaxfLjq2Yme8X9Do2fMJGmkg+hndW7dtbYjfO6mzCPrHlbCTHf6GG0KT0Ni6lWTAbYOWIAcxdd9Xq+wetRc/W/xMgS3x8MOSgi6SAQZFJs9CoMDMtKI/RfAKvAQAmaFg3lzG0X9j98ZQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=EzGIbqlcTGgWLcfPAVcqRkHFVmfHDEHVh71u9Nps9QF/xl7HM4zRzWoeohQ6uGxvkWX+a7sjdo3LNHgXHBdiVpfNcpapq9daUKB1zuOWe+6skDW5sZr78PhI7MNwdzCcKUQbugwE5AGa8O+gqENO0//tpqAUa7qv9iaA6smvr88= Received: by 10.78.122.16 with SMTP id u16mr20818341huc.21.1202407539265; Thu, 07 Feb 2008 10:05:39 -0800 (PST) Received: by 10.78.189.6 with HTTP; Thu, 7 Feb 2008 10:05:39 -0800 (PST) Message-ID: <70e8236f0802071005w7a52923w94be1f35917055d5@mail.gmail.com> Date: Thu, 7 Feb 2008 18:05:39 +0000 From: "Joao Barros" To: "Robert Watson" In-Reply-To: <20080207171913.M96200@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <47AAFDED.9030301@freebsd.org> <47AB05A1.7010803@freebsd.org> <3bbf2fe10802070613mf2bf3feg5dcb480501fcfbbc@mail.gmail.com> <20080207171913.M96200@fledge.watson.org> Cc: freebsd-fs@freebsd.org, Yar Tikhiy , Scot Hetzel , Andre Oppermann , Jeff Roberson , Attilio Rao , Doug Barton , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 18:05:42 -0000 On Feb 7, 2008 5:20 PM, Robert Watson wrote: > On Thu, 7 Feb 2008, Attilio Rao wrote: > > > 2008/2/7, Andre Oppermann : > > > >> Eric Anderson wrote: > >>> I think Alfred's point is really interesting. How many people that don't > >>> use it that say 'axe it' does it take to override 1 person saying 'keep > >>> it!'? > >> > >> The real question is how many people does it take to say 'I'll maintain > >> it'? Just one. Without it, it will only bitrot as evidenced by Attilios > >> question. NTFS is currently broken, just not as obvious because WITNESS > >> didn't track and enforce lockmgr locks. > > > > Andre catched exactly my point. The big problem is that we have a list of > > several unmaintained fs. NTFS is in this list. The support is not reliable, > > it is only available in read mode and eventually bugged. I'm not sure I want > > to keep this if nobody wants to maintain it. > > If you axe write support, does the maintainability of the kernel ntfs get > easier? As I understand it, the write support is rather limited, and If I recall correctly ntfs volumes are mounted readonly by default (I'm unable to verify now). > debugging and fixing read support is generally a lot easier for a variety of > reasons. There's also a lot less risk to data. :-) I think it's reasonable > to surmise that, given our rather limited write support currently, the kernel > ntfs code is used for data migration and limited sharing to FreeBSD in various > forms, but that msdofs remains the general data transport of choice... With this in mind, I used FAT32, but occasional Windows crashes lead to some filesystem corruption and time consuming fsck. I converted the fs to ntfs and had no more issues. General data transport of choice for usb pens, external disks, iPods, and when you need the ability to read/write it everywhere, but for running Windows it's not the best choice when compared to ntfs. If you think of it, FAT32 is the best supported (r/w) fs (on disk) by all platforms: Windows, FreeBSD, OS X, Linux. Having read the news today about the corporate support OpenID got, I dream of a *good* universally supported fs. But I digress =) > > Robert N M Watson > Computer Laboratory > University of Cambridge -- Joao Barros From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 18:18:45 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0805416A41A for ; Thu, 7 Feb 2008 18:18:45 +0000 (UTC) (envelope-from joao.barros@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.186]) by mx1.freebsd.org (Postfix) with ESMTP id 5AE4F13C45E for ; Thu, 7 Feb 2008 18:18:44 +0000 (UTC) (envelope-from joao.barros@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so1077287nfb.33 for ; Thu, 07 Feb 2008 10:18:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=kr+ZYDhgJY5kqdXY9h+KjlofOHezPv/7JH9ed8T7qj4=; b=HaFiiLi0xjmvmqMXO9W+J0EqxD2OTMxntxWBPTFeNkeWWLQop7ipLpuchDXUMb+Qz2ARtg4YvsnfMdWg9U4bnY5GK5jcr8Z1c2moOxhl4ZZjbd2omxxC8Mv26aWfImDHv1UcVZXLjoWXJOuOPCxJgGbDD40mfXM9hNuILMjnQVo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=ggpOkG9jowuSALZgLEBOks7fa0O1IhTOA8C/6Q/ILRnMaZODRuz0GKplyI3Gqpd/6oZfoSnENzp6nRNveWz8hhDUiCL6bx097XoQyPvHATgUMI/M2D5w0ZDiyu7Yal/VDJccjFFGJbeeqRCi5C69WCY9IqV109wVPBbwyVw8YPw= Received: by 10.78.193.5 with SMTP id q5mr20877493huf.4.1202408322717; Thu, 07 Feb 2008 10:18:42 -0800 (PST) Received: by 10.78.189.6 with HTTP; Thu, 7 Feb 2008 10:18:42 -0800 (PST) Message-ID: <70e8236f0802071018n389afa3bu161eaa5c6563cbc0@mail.gmail.com> Date: Thu, 7 Feb 2008 18:18:42 +0000 From: "Joao Barros" To: qpadla@gmail.com In-Reply-To: <200802071941.23199.qpadla@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <47AAFDED.9030301@freebsd.org> <200802071941.23199.qpadla@gmail.com> Cc: Attilio Rao , Yar Tikhiy , Doug Barton , Jeff Roberson , freebsd-fs@freebsd.org, Scot Hetzel , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 18:18:45 -0000 On Feb 7, 2008 5:41 PM, Nikolay Pavlov wrote: > On Thursday 07 February 2008 14:47:41 Eric Anderson wrote: > > FUSE is slow, requires a port (unless PUFFS is ported, which I've probed > > about before). > > I think this is not an argument: > http://www.ntfs-3g.org/performance.html > Eric has valid points. How relevant is a benchmark on Linux to your argument? -- Joao Barros From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 18:52:57 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ABA7916A46C for ; Thu, 7 Feb 2008 18:52:57 +0000 (UTC) (envelope-from qpadla@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.175]) by mx1.freebsd.org (Postfix) with ESMTP id 8B1D113C4E5 for ; Thu, 7 Feb 2008 18:52:56 +0000 (UTC) (envelope-from qpadla@gmail.com) Received: by ug-out-1314.google.com with SMTP id y2so1092625uge.37 for ; Thu, 07 Feb 2008 10:52:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:reply-to:to:subject:date:user-agent:cc:references:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:message-id; bh=SNf+RTbTcpqVnF+bFdNgH1Xarr3DJch7+PEnLPEMnPM=; b=EBuzShm6HZj8E3/QprX0jsfRVIYtbfdqcuSt2X2lfZ40YM9tVTJb67a3kz+aLzHbd85R+A1+bTOlXeVm+YTm5nc9oZk7uIDVZY9YpRiqu0vwZtBg8OIW1D+Z+2DV13aTbggGxILuK6S1pzuhy5v9iJLP4dFLZn2Z5kbOlnRtsLs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:reply-to:to:subject:date:user-agent:cc:references:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:message-id; b=CJoG6nwB8RckEB+59q9vccQ86D4QkA130QdIt1VZZsigHxOIXcdNG/1rCbknQQ5a+LmdYZ0E0J0TPgdfzBVrWhHLqdsPhUly1bigoq6pVCywQrdnf8Vm00EpoLCEXV+2UzpQQHGEHaIgtmNA21iiXPz9CVceipTpCnf+LEwe7aY= Received: by 10.67.116.7 with SMTP id t7mr4377062ugm.38.1202410374581; Thu, 07 Feb 2008 10:52:54 -0800 (PST) Received: from atlas.local ( [89.162.141.1]) by mx.google.com with ESMTPS id w28sm3100136uge.79.2008.02.07.10.52.52 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 07 Feb 2008 10:52:53 -0800 (PST) From: Nikolay Pavlov To: "Joao Barros" Date: Thu, 7 Feb 2008 20:52:55 +0200 User-Agent: KMail/1.9.6 (enterprise 0.20070907.709405) References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <200802071941.23199.qpadla@gmail.com> <70e8236f0802071018n389afa3bu161eaa5c6563cbc0@mail.gmail.com> In-Reply-To: <70e8236f0802071018n389afa3bu161eaa5c6563cbc0@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200802072052.56918.qpadla@gmail.com> Cc: Attilio Rao , Yar Tikhiy , Doug Barton , Jeff Roberson , freebsd-fs@freebsd.org, Scot Hetzel , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: qpadla@gmail.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 18:52:57 -0000 On Thursday 07 February 2008 20:18:42 Joao Barros wrote: > On Feb 7, 2008 5:41 PM, Nikolay Pavlov wrote: > > On Thursday 07 February 2008 14:47:41 Eric Anderson wrote: > > > FUSE is slow, requires a port (unless PUFFS is ported, which I've > > > probed about before). > > > > I think this is not an argument: > > http://www.ntfs-3g.org/performance.html > > Eric has valid points. > How relevant is a benchmark on Linux to your argument? But it's a userland application. This page is demonstration of it's potential performance that could be achieved, but were is the FreeBSD NTFS implementation stats? Let me ask you: compered to what FUSE is slow? -- ====================================================================== - Best regards, Nikolay Pavlov. <<<----------------------------------- ====================================================================== From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 23:03:10 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C02BF16A46E for ; Thu, 7 Feb 2008 23:03:10 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from mail2.fluidhosting.com (mx24.fluidhosting.com [204.14.89.7]) by mx1.freebsd.org (Postfix) with SMTP id 34F4B13C45E for ; Thu, 7 Feb 2008 23:03:09 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: (qmail 12676 invoked by uid 399); 7 Feb 2008 23:03:08 -0000 Received: from localhost (HELO ?192.168.0.4?) (dougb@dougbarton.us@127.0.0.1) by localhost with ESMTP; 7 Feb 2008 23:03:08 -0000 X-Originating-IP: 127.0.0.1 Message-ID: <47AB8E28.4050502@FreeBSD.org> Date: Thu, 07 Feb 2008 15:03:04 -0800 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Robert Watson References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <47AAFDED.9030301@freebsd.org> <47AB05A1.7010803@freebsd.org> <3bbf2fe10802070613mf2bf3feg5dcb480501fcfbbc@mail.gmail.com> <20080207171913.M96200@fledge.watson.org> In-Reply-To: <20080207171913.M96200@fledge.watson.org> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Yar Tikhiy , Scot Hetzel , Andre Oppermann , Jeff Roberson , Attilio Rao , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 23:03:10 -0000 Robert Watson wrote: > If you axe write support, does the maintainability of the kernel ntfs > get easier? As I understand it, the write support is rather > limited, and debugging and fixing read support is generally a lot > easier for a variety of reasons. The few serious attempts I've made to use the NTFS in the base to write data were unsuccessful, and given the instability of the code I didn't push it very hard since I didn't want to scramble my data. > There's also a lot less risk to data. :-) Right. :) > I think it's reasonable to surmise that, given our rather limited > write support currently, the kernel ntfs code is used for data > migration and limited sharing to FreeBSD in various forms, but that > msdofs remains the general data transport of choice... Read support that works is better than R/W support that doesn't, yes. So if someone wants to step up to maintain this and axing write support gets us something that actually works, who am I to argue? I think keeping the goal(s) of R/W support in the base, and/or working FUSE in mind is a Good Thing, but getting a working solution to at least one of these problems would be a big step forward. Doug -- This .signature sanitized for your protection From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 23:34:24 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D266516A41A; Thu, 7 Feb 2008 23:34:24 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 67F3D13C4CC; Thu, 7 Feb 2008 23:34:24 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.2/8.14.1) with ESMTP id m17NSP7O093272; Thu, 7 Feb 2008 16:28:25 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Thu, 07 Feb 2008 16:31:09 -0700 (MST) Message-Id: <20080207.163109.-1384053103.imp@bsdimp.com> To: attilio@freebsd.org From: "M. Warner Losh" In-Reply-To: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> X-Mailer: Mew version 5.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: yar@freebsd.org, dougb@freebsd.org, jeff@freebsd.org, freebsd-fs@freebsd.org, swhetzel@gmail.com, freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 23:34:24 -0000 In message: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> "Attilio Rao" writes: : - Do you use NTFS? : - Are you interested in maintaining it? : - Do you know a good reason to not use FUSE ntfs implementation? What : the kernel counter part adds? : - Do you think axing the kernel support a good idea? I use NTFS extensively and it has worked as recently as early in the 7.0 release branch. I think axing it is a horrible idea. Warner From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 23:34:29 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 15C6816A46C; Thu, 7 Feb 2008 23:34:29 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id B2C1013C45B; Thu, 7 Feb 2008 23:34:28 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.2/8.14.1) with ESMTP id m17NU7PS093277; Thu, 7 Feb 2008 16:30:07 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Thu, 07 Feb 2008 16:32:51 -0700 (MST) Message-Id: <20080207.163251.179960372.imp@bsdimp.com> To: dougb@freebsd.org From: "M. Warner Losh" In-Reply-To: <47AABEEF.7020807@FreeBSD.org> References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <20080207071314.GO99258@elvis.mu.org> <47AABEEF.7020807@FreeBSD.org> X-Mailer: Mew version 5.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: attilio@freebsd.org, yar@freebsd.org, swhetzel@gmail.com, jeff@freebsd.org, alfred@freebsd.org, freebsd-fs@freebsd.org, freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 23:34:29 -0000 In message: <47AABEEF.7020807@FreeBSD.org> Doug Barton writes: : Alfred Perlstein wrote: : : > Maybe a nicer way of saying/asking would be to ask: : > : > Is the FUSE replacement going to be tested to the point where it's : > better than then current NTFS code? : : Given that the current NTFS code in the base panics within minutes of : any kind of serious access, and has the ability to take the other : filesystems down with it (including UFS2) that won't be hard. This change in behavior is very recent. It used to be the one file system you could count on to get data off a disk that was throwing disk errors back at the OS. Warner From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 23:39:05 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8CFAB16A418; Thu, 7 Feb 2008 23:39:05 +0000 (UTC) (envelope-from obrien@NUXI.org) Received: from dragon.nuxi.org (trang.nuxi.org [74.95.12.85]) by mx1.freebsd.org (Postfix) with ESMTP id 63DF913C4CC; Thu, 7 Feb 2008 23:39:05 +0000 (UTC) (envelope-from obrien@NUXI.org) Received: from dragon.nuxi.org (obrien@localhost [127.0.0.1]) by dragon.nuxi.org (8.14.1/8.14.1) with ESMTP id m17NQ57g003728; Thu, 7 Feb 2008 15:26:05 -0800 (PST) (envelope-from obrien@dragon.nuxi.org) Received: (from obrien@localhost) by dragon.nuxi.org (8.14.1/8.14.1/Submit) id m17NQ50x003727; Thu, 7 Feb 2008 15:26:05 -0800 (PST) (envelope-from obrien) Date: Thu, 7 Feb 2008 15:26:05 -0800 From: "David O'Brien" To: Alfred Perlstein Message-ID: <20080207232605.GA3673@dragon.NUXI.org> Mail-Followup-To: obrien@freebsd.org, Alfred Perlstein , Attilio Rao , Yar Tikhiy , Doug Barton , Jeff Roberson , freebsd-fs@freebsd.org, Scot Hetzel , freebsd-arch@freebsd.org References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <20080207071314.GO99258@elvis.mu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080207071314.GO99258@elvis.mu.org> X-Operating-System: FreeBSD 8.0-CURRENT User-Agent: Mutt/1.5.16 (2007-06-09) Cc: freebsd-fs@freebsd.org, Yar Tikhiy , Doug Barton , Jeff Roberson , Attilio Rao , Scot Hetzel , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: obrien@freebsd.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 23:39:05 -0000 On Wed, Feb 06, 2008 at 11:13:14PM -0800, Alfred Perlstein wrote: > * Attilio Rao [080206 17:00] wrote: > > remove the kernel support for NTFS and maybe take care of the FUSE > > implementation. .. > > My pragmatic view on this is that I think it's odd that something > that is sort-of working for a few people is going to be axed by > people that don't use it, while promising to replace it with something > better. > > Maybe a nicer way of saying/asking would be to ask: > > Is the FUSE replacement going to be tested to the point where it's > better than then current NTFS code? Hear, Hear! From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 23:37:19 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B55BA16A418; Thu, 7 Feb 2008 23:37:19 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 5959F13C447; Thu, 7 Feb 2008 23:37:19 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.2/8.14.1) with ESMTP id m17NWAlC093322; Thu, 7 Feb 2008 16:32:10 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Thu, 07 Feb 2008 16:34:54 -0700 (MST) Message-Id: <20080207.163454.-1471235838.imp@bsdimp.com> To: attilio@FreeBSD.org From: "M. Warner Losh" In-Reply-To: <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> References: <3bbf2fe10802070613mf2bf3feg5dcb480501fcfbbc@mail.gmail.com> <20080207141820.GR99258@elvis.mu.org> <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> X-Mailer: Mew version 5.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Thu, 07 Feb 2008 23:43:00 +0000 Cc: yar@FreeBSD.org, swhetzel@gmail.com, andre@FreeBSD.org, jeff@FreeBSD.org, alfred@FreeBSD.org, anderson@FreeBSD.org, freebsd-fs@FreeBSD.org, dougb@FreeBSD.org, freebsd-arch@FreeBSD.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 23:37:19 -0000 In message: <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> "Attilio Rao" writes: : 2008/2/7, Alfred Perlstein : : > * Attilio Rao [080207 06:13] wrote: : > > 2008/2/7, Andre Oppermann : : > > > Eric Anderson wrote: : > > > > I think Alfred's point is really interesting. How many people that : > > > > don't use it that say 'axe it' does it take to override 1 person saying : > > > > 'keep it!'? : > > > : > > > The real question is how many people does it take to say 'I'll maintain : > > > it'? Just one. Without it, it will only bitrot as evidenced by Attilios : > > > question. NTFS is currently broken, just not as obvious because WITNESS : > > > didn't track and enforce lockmgr locks. : > > : > > Andre catched exactly my point. : > > The big problem is that we have a list of several unmaintained fs. : > > NTFS is in this list. The support is not reliable, it is only : > > available in read mode and eventually bugged. : > > I'm not sure I want to keep this if nobody wants to maintain it. : > : > All I'm saying is that I think this is a bit premature considering : > the users. Within less than 24hrs we've had a few users reporting : > in as users, I'm sure the fixes (now that we have some good assertions) : > are going to be trivial. : > : > Why not let it ferment/rot for a release cycle and then see what : > the story is? : : Obviously if we can fix it is better, but axing is an opportunity I : don't want to leave out and this is why I wanted to poll users about : this issue. Eventually, if an axing is decided, it won't happen in : short times but only once all situations for "migration" will be : probed and finished. WE SHOULD NOT AXE IT. IT IS TOO USEFUL. VERY RECENTLY IT WORKED VERY WELL. There's a lot of other systems in the tree that aren't nearly as useful that nobody is complaining about that are actually in much worse shape. Warner From owner-freebsd-fs@FreeBSD.ORG Thu Feb 7 23:54:42 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 543EC16A41A; Thu, 7 Feb 2008 23:54:42 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id EA7AF13C465; Thu, 7 Feb 2008 23:54:41 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.2/8.14.1) with ESMTP id m17NoWFd093666; Thu, 7 Feb 2008 16:50:32 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Thu, 07 Feb 2008 16:53:16 -0700 (MST) Message-Id: <20080207.165316.1678770676.imp@bsdimp.com> To: attilio@freebsd.org From: "M. Warner Losh" In-Reply-To: <20080207.163454.-1471235838.imp@bsdimp.com> References: <20080207141820.GR99258@elvis.mu.org> <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> <20080207.163454.-1471235838.imp@bsdimp.com> X-Mailer: Mew version 5.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Fri, 08 Feb 2008 00:02:10 +0000 Cc: yar@freebsd.org, swhetzel@gmail.com, andre@freebsd.org, jeff@freebsd.org, alfred@freebsd.org, freebsd-fs@freebsd.org, dougb@freebsd.org, freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2008 23:54:42 -0000 In message: <20080207.163454.-1471235838.imp@bsdimp.com> "M. Warner Losh" writes: : In message: <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> : "Attilio Rao" writes: : : 2008/2/7, Alfred Perlstein : : : > * Attilio Rao [080207 06:13] wrote: : : > > 2008/2/7, Andre Oppermann : : : > > > Eric Anderson wrote: : : > > > > I think Alfred's point is really interesting. How many people that : : > > > > don't use it that say 'axe it' does it take to override 1 person saying : : > > > > 'keep it!'? : : > > > : : > > > The real question is how many people does it take to say 'I'll maintain : : > > > it'? Just one. Without it, it will only bitrot as evidenced by Attilios : : > > > question. NTFS is currently broken, just not as obvious because WITNESS : : > > > didn't track and enforce lockmgr locks. : : > > : : > > Andre catched exactly my point. : : > > The big problem is that we have a list of several unmaintained fs. : : > > NTFS is in this list. The support is not reliable, it is only : : > > available in read mode and eventually bugged. : : > > I'm not sure I want to keep this if nobody wants to maintain it. : : > : : > All I'm saying is that I think this is a bit premature considering : : > the users. Within less than 24hrs we've had a few users reporting : : > in as users, I'm sure the fixes (now that we have some good assertions) : : > are going to be trivial. : : > : : > Why not let it ferment/rot for a release cycle and then see what : : > the story is? : : : : Obviously if we can fix it is better, but axing is an opportunity I : : don't want to leave out and this is why I wanted to poll users about : : this issue. Eventually, if an axing is decided, it won't happen in : : short times but only once all situations for "migration" will be : : probed and finished. : : WE SHOULD NOT AXE IT. IT IS TOO USEFUL. VERY RECENTLY IT WORKED VERY : WELL. : : There's a lot of other systems in the tree that aren't nearly as : useful that nobody is complaining about that are actually in much : worse shape. OK. I shouldn't have shouted. My basic point is that ntfs worked very recently, and therefore we owe it to ourselves to give it some time to get fixed. fuse is unknown, not even in head and the performance characteristics between the two aren't known. Also, I use ntfs to recover data from "crashed" disks because it copes well with bad spots on the disk. None of the other filesystems in the tree does this, and that makes it a very powerful tool for dealing with crashed disks that others say are unrecoverable. Warner From owner-freebsd-fs@FreeBSD.ORG Fri Feb 8 02:06:03 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7473A16A41A; Fri, 8 Feb 2008 02:06:03 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 347B013C4E7; Fri, 8 Feb 2008 02:06:03 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.107] (cpe-24-94-75-93.hawaii.res.rr.com [24.94.75.93]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m1825pFv091255; Thu, 7 Feb 2008 21:05:54 -0500 (EST) (envelope-from jroberson@chesapeake.net) Date: Thu, 7 Feb 2008 16:06:42 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: "M. Warner Losh" In-Reply-To: <20080207.165316.1678770676.imp@bsdimp.com> Message-ID: <20080207155338.Q15691@desktop> References: <20080207141820.GR99258@elvis.mu.org> <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> <20080207.163454.-1471235838.imp@bsdimp.com> <20080207.165316.1678770676.imp@bsdimp.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Mailman-Approved-At: Fri, 08 Feb 2008 02:48:16 +0000 Cc: freebsd-fs@freebsd.org, yar@freebsd.org, swhetzel@gmail.com, andre@freebsd.org, jeff@freebsd.org, alfred@freebsd.org, attilio@freebsd.org, dougb@freebsd.org, freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 02:06:03 -0000 On Thu, 7 Feb 2008, M. Warner Losh wrote: > In message: <20080207.163454.-1471235838.imp@bsdimp.com> > "M. Warner Losh" writes: > : In message: <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> > : "Attilio Rao" writes: > : : 2008/2/7, Alfred Perlstein : > : : > * Attilio Rao [080207 06:13] wrote: > : : > > 2008/2/7, Andre Oppermann : > : : > > > Eric Anderson wrote: > : : > > > > I think Alfred's point is really interesting. How many people that > : : > > > > don't use it that say 'axe it' does it take to override 1 person saying > : : > > > > 'keep it!'? > : : > > > > : : > > > The real question is how many people does it take to say 'I'll maintain > : : > > > it'? Just one. Without it, it will only bitrot as evidenced by Attilios > : : > > > question. NTFS is currently broken, just not as obvious because WITNESS > : : > > > didn't track and enforce lockmgr locks. > : : > > > : : > > Andre catched exactly my point. > : : > > The big problem is that we have a list of several unmaintained fs. > : : > > NTFS is in this list. The support is not reliable, it is only > : : > > available in read mode and eventually bugged. > : : > > I'm not sure I want to keep this if nobody wants to maintain it. > : : > > : : > All I'm saying is that I think this is a bit premature considering > : : > the users. Within less than 24hrs we've had a few users reporting > : : > in as users, I'm sure the fixes (now that we have some good assertions) > : : > are going to be trivial. > : : > > : : > Why not let it ferment/rot for a release cycle and then see what > : : > the story is? > : : > : : Obviously if we can fix it is better, but axing is an opportunity I > : : don't want to leave out and this is why I wanted to poll users about > : : this issue. Eventually, if an axing is decided, it won't happen in > : : short times but only once all situations for "migration" will be > : : probed and finished. > : > : WE SHOULD NOT AXE IT. IT IS TOO USEFUL. VERY RECENTLY IT WORKED VERY > : WELL. > : > : There's a lot of other systems in the tree that aren't nearly as > : useful that nobody is complaining about that are actually in much > : worse shape. > > OK. I shouldn't have shouted. My basic point is that ntfs worked > very recently, and therefore we owe it to ourselves to give it some > time to get fixed. fuse is unknown, not even in head and the > performance characteristics between the two aren't known. Also, I use > ntfs to recover data from "crashed" disks because it copes well with > bad spots on the disk. None of the other filesystems in the tree does > this, and that makes it a very powerful tool for dealing with crashed > disks that others say are unrecoverable. Not picking on anyone in particular, but let's keep in mind that this was an enquiry not a real proposal to axe it right away. I suggested Attilio find out if there were users and clearly there are. So there is value in keeping this thing working and fuse isn't a sure bet. We just wanted to understand the situation before acting. However, this is open source. Some one needs to step up to the plate and fix these bugs. It's only 4,700 lines of code. It shouldn't be insurmountable for someone who has a passing understanding of VFS. Some of the bugs were exposed by better asserts and witness support by Attilio. I don't think his effort to fix lockmgr should be hung up trying to understand ntfs however unless he directly broke it. It's going to have to continue firing asserts until someone fixes it. Also, ntfs is a strange bird compared to other filesystems. Briefly looking at it, there may be some subtle architectural problems with it. For example, it creates 'ntnode' inodes that aren't associated with vnodes and so have their own lifecycle management. It is likely that this is the source of the panics that I have heard of. An eager volunteer might also consider making it MPSAFE to further reduce the number of filesystems which require Giant so we can eventually drop the hideous giant wrappers. Cheers, Jeff > > Warner > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Fri Feb 8 03:30:14 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 03F9916A419; Fri, 8 Feb 2008 03:30:14 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from ns.trinitel.com (186.161.36.72.static.reverse.ltdomains.com [72.36.161.186]) by mx1.freebsd.org (Postfix) with ESMTP id CE4B013C458; Fri, 8 Feb 2008 03:30:13 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from proton.local (r74-193-81-203.pfvlcmta01.grtntx.tl.dh.suddenlink.net [74.193.81.203]) (authenticated bits=0) by ns.trinitel.com (8.14.1/8.14.1) with ESMTP id m183U0Rq016971; Thu, 7 Feb 2008 21:30:03 -0600 (CST) (envelope-from anderson@freebsd.org) Message-ID: <47ABCCB3.70009@freebsd.org> Date: Thu, 07 Feb 2008 21:29:55 -0600 From: Eric Anderson User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Jeff Roberson References: <20080207141820.GR99258@elvis.mu.org> <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> <20080207.163454.-1471235838.imp@bsdimp.com> <20080207.165316.1678770676.imp@bsdimp.com> <20080207155338.Q15691@desktop> In-Reply-To: <20080207155338.Q15691@desktop> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.1 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on ns.trinitel.com X-Mailman-Approved-At: Fri, 08 Feb 2008 03:34:37 +0000 Cc: freebsd-fs@freebsd.org, yar@freebsd.org, swhetzel@gmail.com, andre@freebsd.org, jeff@freebsd.org, alfred@freebsd.org, attilio@freebsd.org, dougb@freebsd.org, freebsd-arch@freebsd.org, "M. Warner Losh" Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 03:30:14 -0000 Jeff Roberson wrote: > On Thu, 7 Feb 2008, M. Warner Losh wrote: > >> In message: <20080207.163454.-1471235838.imp@bsdimp.com> >> "M. Warner Losh" writes: >> : In message: <3bbf2fe10802070621h574f5d3kb4fbd86adbab11c@mail.gmail.com> >> : "Attilio Rao" writes: >> : : 2008/2/7, Alfred Perlstein : >> : : > * Attilio Rao [080207 06:13] wrote: >> : : > > 2008/2/7, Andre Oppermann : >> : : > > > Eric Anderson wrote: >> : : > > > > I think Alfred's point is really interesting. How many >> people that >> : : > > > > don't use it that say 'axe it' does it take to override 1 >> person saying >> : : > > > > 'keep it!'? >> : : > > > >> : : > > > The real question is how many people does it take to say >> 'I'll maintain >> : : > > > it'? Just one. Without it, it will only bitrot as >> evidenced by Attilios >> : : > > > question. NTFS is currently broken, just not as obvious >> because WITNESS >> : : > > > didn't track and enforce lockmgr locks. >> : : > > >> : : > > Andre catched exactly my point. >> : : > > The big problem is that we have a list of several unmaintained >> fs. >> : : > > NTFS is in this list. The support is not reliable, it is only >> : : > > available in read mode and eventually bugged. >> : : > > I'm not sure I want to keep this if nobody wants to maintain it. >> : : > >> : : > All I'm saying is that I think this is a bit premature considering >> : : > the users. Within less than 24hrs we've had a few users reporting >> : : > in as users, I'm sure the fixes (now that we have some good >> assertions) >> : : > are going to be trivial. >> : : > >> : : > Why not let it ferment/rot for a release cycle and then see what >> : : > the story is? >> : : >> : : Obviously if we can fix it is better, but axing is an opportunity I >> : : don't want to leave out and this is why I wanted to poll users about >> : : this issue. Eventually, if an axing is decided, it won't happen in >> : : short times but only once all situations for "migration" will be >> : : probed and finished. >> : >> : WE SHOULD NOT AXE IT. IT IS TOO USEFUL. VERY RECENTLY IT WORKED VERY >> : WELL. >> : >> : There's a lot of other systems in the tree that aren't nearly as >> : useful that nobody is complaining about that are actually in much >> : worse shape. >> >> OK. I shouldn't have shouted. My basic point is that ntfs worked >> very recently, and therefore we owe it to ourselves to give it some >> time to get fixed. fuse is unknown, not even in head and the >> performance characteristics between the two aren't known. Also, I use >> ntfs to recover data from "crashed" disks because it copes well with >> bad spots on the disk. None of the other filesystems in the tree does >> this, and that makes it a very powerful tool for dealing with crashed >> disks that others say are unrecoverable. > > Not picking on anyone in particular, but let's keep in mind that this > was an enquiry not a real proposal to axe it right away. I suggested > Attilio find out if there were users and clearly there are. So there is > value in keeping this thing working and fuse isn't a sure bet. We just > wanted to understand the situation before acting. > > However, this is open source. Some one needs to step up to the plate > and fix these bugs. It's only 4,700 lines of code. It shouldn't be > insurmountable for someone who has a passing understanding of VFS. > > Some of the bugs were exposed by better asserts and witness support by > Attilio. I don't think his effort to fix lockmgr should be hung up > trying to understand ntfs however unless he directly broke it. It's > going to have to continue firing asserts until someone fixes it. > > Also, ntfs is a strange bird compared to other filesystems. Briefly > looking at it, there may be some subtle architectural problems with it. > For example, it creates 'ntnode' inodes that aren't associated with > vnodes and so have their own lifecycle management. It is likely that > this is the source of the panics that I have heard of. > > An eager volunteer might also consider making it MPSAFE to further > reduce the number of filesystems which require Giant so we can > eventually drop the hideous giant wrappers. Are all the known bugs entered into gnats already? Eric From owner-freebsd-fs@FreeBSD.ORG Fri Feb 8 11:42:02 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D8EFD16A417; Fri, 8 Feb 2008 11:42:02 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) Received: from 0.mx.codelabs.ru (0.mx.codelabs.ru [144.206.177.45]) by mx1.freebsd.org (Postfix) with ESMTP id 8377A13C461; Fri, 8 Feb 2008 11:42:02 +0000 (UTC) (envelope-from rea-fbsd@codelabs.ru) DomainKey-Signature: a=rsa-sha1; q=dns; c=simple; s=one; d=codelabs.ru; h=Received:Date:From:To:Cc:Message-ID:References:MIME-Version:Content-Type:Content-Disposition:In-Reply-To:Sender:X-Spam-Status:Subject; b=ciJjB86naW0YoOgEaRDpV3h0H6Xxoh8/UgffukYx9/oBIU7xExXXOpUvLoBcoubCtD6pO2lKbRMawsnroORCsBZ0aTgHnk1b7b5KXeL6I/+XdOxTa7brvU8M2IIcpgu9TjKfxahmVdFt1nGQgHAv3cLNB/n51sqJQTDEO3gLZ2A=; Received: from void.codelabs.ru (void.codelabs.ru [144.206.177.25]) by 0.mx.codelabs.ru with esmtpsa (TLSv1:AES256-SHA:256) id 1JNRQX-000HBR-Ni; Fri, 08 Feb 2008 14:29:53 +0300 Date: Fri, 8 Feb 2008 14:29:51 +0300 From: Eygene Ryabinkin To: Attilio Rao Message-ID: References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> Sender: rea-fbsd@codelabs.ru X-Spam-Status: No, score=-1.9 required=4.0 tests=ALL_TRUSTED,AWL,BAYES_40 Cc: Yar Tikhiy , Doug Barton , Jeff Roberson , freebsd-fs@freebsd.org, Scot Hetzel , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 11:42:03 -0000 Attilio, good day. Thu, Feb 07, 2008 at 02:00:41AM +0100, Attilio Rao wrote: > As exposed by several users, NTFS seems to be broken even before first > VFS commits happeing around the end of December. Those commits exposed > some problems about NTFS which are currently under investigation. > Ultimately, This filesystem is also unmaintained at the moment. > > Speaking with jeff, we agreed on what can be a possible compromise: > remove the kernel support for NTFS and maybe take care of the FUSE > implementation. > What I now propose is a small survey which can shade a light on us > about what do you think about this idea and its implications: > - Do you use NTFS? Yes, especially on the multi-homed notebook systems. In read-only mode it rocks. > - Are you interested in maintaining it? Yes. If you can throw the buglist for NTFS on me, I will be very grateful. > - Do you know a good reason to not use FUSE ntfs implementation? What > the kernel counter part adds? Don't know, newer tried FUSE. > - Do you think axing the kernel support a good idea? IMO, not a good one. -- Eygene From owner-freebsd-fs@FreeBSD.ORG Fri Feb 8 16:23:47 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 24E5216A41A; Fri, 8 Feb 2008 16:23:47 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id E2A0813C442; Fri, 8 Feb 2008 16:23:46 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id F0DE247EE5; Fri, 8 Feb 2008 11:23:45 -0500 (EST) Date: Fri, 8 Feb 2008 16:23:45 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Eygene Ryabinkin In-Reply-To: Message-ID: <20080208162231.R17387@fledge.watson.org> References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, Yar Tikhiy , Doug Barton , Jeff Roberson , Attilio Rao , Scot Hetzel , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 16:23:47 -0000 On Fri, 8 Feb 2008, Eygene Ryabinkin wrote: >> - Are you interested in maintaining it? > > Yes. If you can throw the buglist for NTFS on me, I will be very grateful. Sign the man up :-). FWIW, I think it would be really good if we could get active maintainers for more of the edge case file systems (i.e., not UFS, NFS) so that as the VFS work gets more active in HEAD, those file systems not only keep up, but also improve... Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-fs@FreeBSD.ORG Fri Feb 8 22:18:27 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CFBF116A475 for ; Fri, 8 Feb 2008 22:18:27 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from mail2.fluidhosting.com (mx24.fluidhosting.com [204.14.89.7]) by mx1.freebsd.org (Postfix) with SMTP id 06D1613C478 for ; Fri, 8 Feb 2008 22:18:26 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: (qmail 10691 invoked by uid 399); 8 Feb 2008 22:18:26 -0000 Received: from localhost (HELO ?192.168.0.8?) (dougb@dougbarton.us@127.0.0.1) by localhost with ESMTP; 8 Feb 2008 22:18:26 -0000 X-Originating-IP: 127.0.0.1 Message-ID: <47ACD532.2090206@FreeBSD.org> Date: Fri, 08 Feb 2008 14:18:26 -0800 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Robert Watson References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <20080208162231.R17387@fledge.watson.org> In-Reply-To: <20080208162231.R17387@fledge.watson.org> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Yar Tikhiy , Scot Hetzel , Jeff Roberson , Attilio Rao , freebsd-arch@freebsd.org, Eygene Ryabinkin Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 22:18:27 -0000 Robert Watson wrote: > > On Fri, 8 Feb 2008, Eygene Ryabinkin wrote: > >>> - Are you interested in maintaining it? >> >> Yes. If you can throw the buglist for NTFS on me, I will be very >> grateful. > > Sign the man up :-). Sounds good to me. I've posted on -current several times since December with crash dumps, etc. I think the problem at this point is that we don't know what the bugs are, or else I suspect at least some of them would be fixed already. Doug -- This .signature sanitized for your protection From owner-freebsd-fs@FreeBSD.ORG Fri Feb 8 22:29:44 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D675416A417; Fri, 8 Feb 2008 22:29:44 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from shadow.wildlava.net (shadow.wildlava.net [67.40.138.81]) by mx1.freebsd.org (Postfix) with ESMTP id 75BD913C442; Fri, 8 Feb 2008 22:29:44 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from [10.0.3.98] (mail.boulder.swri.edu [65.241.78.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by shadow.wildlava.net (Postfix) with ESMTP id 301BB8F394; Fri, 8 Feb 2008 15:29:43 -0700 (MST) Message-ID: <47ACD7D4.5050905@skyrush.com> Date: Fri, 08 Feb 2008 15:29:40 -0700 From: Joe Peterson User-Agent: Thunderbird 2.0.0.9 (X11/20071119) MIME-Version: 1.0 To: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Analysis of disk file block with ZFS checksum error X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 22:29:44 -0000 In my experimentation with the ZFS filesystem, I encountered one case of a file block with a checksum mismatch. Doing a "zpool scrub" revealed it, and trying to read the file yielded an error - only the part of the file before the bad block was read (ZFS aborts reading at this point, which makes sense), resulting in a short file. The reason the CKSUM error is not fixable is because my ZFS pool contains only one device (no mirror or RAIDZ), but I do have the original/good version of the file affected. Here's the output of zpool status (new scrub in process): pool: tank state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: scrub in progress, 64.36% done, 0h18m to go config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 2 hda6 ONLINE 0 0 2 errors: Permanent errors have been detected in the following files: /mnt/tank/fbsd/home/joe/music/jukebox/christmas/Esquivel/ Merry_XMas_from_the_SpaceAge_Bachelor_Pad/07-Snowfall.mp3 I was curious about what actually happened: was this a ZFS bug, trouble with its metadata, or truly a bad block? In order to determine this, I modified ZFS's source code temporarily to ignore the checksum mismatch and let the file read fully. What I then got was the full-length file and no errors, showing that there were no disk read errors associated with the read (I already had assumed this from the fact that zpool status showed only a non-zero CKSUM count), however, I may have seen other error counts previously (ZFS resets them to zero on, e.g., reboot). I received no errors when originally copying this file *to* the ZFS pool - only on subsequent reads/scrubs. (Note that I have posted before about DMA errors in my log for the disk I am using, but I have had nothing but successful SeaTools tests (surface scans) of the drive. Jeremy Chadwick had similar issues, as did others, so I think it is worth investigating if there is some OS/software cause rather than real HW issues. This is one reason I wanted to investigate my ZFS checksum issue more deeply.) I also have a good backup of the file in question, so I now have two copies of the file: one good, and one with a bad block. The file is 3575936 bytes long, and recordsize (in ZFS) is 128K, making the file about 27 blocks long. Curiously, the bad section of the file is exactly 65536 bytes long (1/2 a block). The bad block starts at exactly the 5th 128K block (byte 65536 or hex a0000). I wanted to see the characteristics of the bad data. Was just one bit flipped randomly? No. It is just one bit or set of bits in the bytes that are affected? It doesn't seem so. Were there any other stange patterns here? Well, yes, and maybe someout out there with more knowledge/experience in disk modes of failure will recognize something (I have included some data below). For one thing (as I mentioned), only 65536 bytes are bad (and it's exactly this many, with a few "good" bytes thrown in, but not far from what matches random chance would produce. Also, all bad bytes have a zero in the high bit - interesting? Also, near the end of the block, the bad bytes all go to zero, strangely coincident with the first "good" zero in that bad block - not sure if that's coincidence or not. Also, I calculated the number of "Bits same" (matching bits) in the good vs. bad bytes, and it appears fairly random, so it appears that the bad bytes are very random in nature and not correlated much at all with the good bytes. So except for the fact that the 2nd half (65536 bytes) of the ZFS block are good, the bad block seems to consist of random data, except for the string of zero bytes near the end and the zero high-bit. It's not as if one bit on the disk flipped - it affects the whole (1/2) block. Does this seem like a disk error, controller error/bug, cable problem (I recently put a new cable on, so I doubt this). It seems to me something more systemic rather than a random bit error - opinions are more than welcome. Here is some info from a python program I wrote to look at the data (I've left out spans of essentially uninteresting portions showing similar stuff, but I can get you the whole thing if interested): File pos Good Bad Match Good (bin) Bad (bin) Bits same 0009fff0 d9 d9 Yes 11011001 11011001 8 0009fff1 05 05 Yes 00000101 00000101 8 0009fff2 c1 c1 Yes 11000001 11000001 8 0009fff3 81 81 Yes 10000001 10000001 8 0009fff4 5f 5f Yes 01011111 01011111 8 0009fff5 66 66 Yes 01100110 01100110 8 0009fff6 5e 5e Yes 01011110 01011110 8 0009fff7 a1 a1 Yes 10100001 10100001 8 0009fff8 ca ca Yes 11001010 11001010 8 0009fff9 9d 9d Yes 10011101 10011101 8 0009fffa 00 00 Yes 00000000 00000000 8 0009fffb 90 90 Yes 10010000 10010000 8 0009fffc 32 32 Yes 00110010 00110010 8 0009fffd 62 62 Yes 01100010 01100010 8 0009fffe a8 a8 Yes 10101000 10101000 8 0009ffff b2 b2 Yes 10110010 10110010 8 --- Start of bad block --- 000a0000 d1 24 No 11010001 00100100 2 000a0001 6b 7b No 01101011 01111011 7 000a0002 d1 31 No 11010001 00110001 5 000a0003 56 33 No 01010110 00110011 4 000a0004 44 38 No 01000100 00111000 3 000a0005 c3 41 No 11000011 01000001 6 000a0006 df 46 No 11011111 01000110 4 000a0007 07 45 No 00000111 01000101 6 000a0008 4c 7b No 01001100 01111011 3 000a0009 a0 40 No 10100000 01000000 5 000a000a 54 0a No 01010100 00001010 3 000a000b 35 40 No 00110101 01000000 3 000a000c 88 24 No 10001000 00100100 4 000a000d 38 24 No 00111000 00100100 5 000a000e f5 7d No 11110101 01111101 6 000a000f 28 31 No 00101000 00110001 5 . . . 000af6c1 d3 33 No 11010011 00110011 5 000af6c2 97 39 No 10010111 00111001 3 000af6c3 a5 32 No 10100101 00110010 3 000af6c4 6a 41 No 01101010 01000001 4 000af6c5 16 39 No 00010110 00111001 3 000af6c6 f2 7d No 11110010 01111101 3 000af6c7 21 40 No 00100001 01000000 5 000af6c8 52 0a No 01010010 00001010 5 000af6c9 00 00 Yes 00000000 00000000 8 000af6ca 2c 00 No 00101100 00000000 5 000af6cb 42 00 No 01000010 00000000 6 000af6cc 31 00 No 00110001 00000000 5 000af6cd a1 00 No 10100001 00000000 5 000af6ce d1 00 No 11010001 00000000 4 000af6cf 90 00 No 10010000 00000000 6 000af6d0 9c 00 No 10011100 00000000 4 . . . 000afff8 26 00 No 00100110 00000000 5 000afff9 8c 00 No 10001100 00000000 5 000afffa a8 00 No 10101000 00000000 5 000afffb 0c 00 No 00001100 00000000 6 000afffc f1 00 No 11110001 00000000 3 000afffd 93 00 No 10010011 00000000 4 000afffe 2c 00 No 00101100 00000000 5 000affff 2e 00 No 00101110 00000000 4 --- End of bad block --- 000b0000 62 62 Yes 01100010 01100010 8 000b0001 56 56 Yes 01010110 01010110 8 000b0002 91 91 Yes 10010001 10010001 8 000b0003 04 04 Yes 00000100 00000100 8 000b0004 01 01 Yes 00000001 00000001 8 000b0005 2d 2d Yes 00101101 00101101 8 000b0006 0e 0e Yes 00001110 00001110 8 000b0007 89 89 Yes 10001001 10001001 8 000b0008 8a 8a Yes 10001010 10001010 8 000b0009 ad ad Yes 10101101 10101101 8 000b000a 4e 4e Yes 01001110 01001110 8 000b000b a3 a3 Yes 10100011 10100011 8 000b000c 13 13 Yes 00010011 00010011 8 000b000d 4d 4d Yes 01001101 01001101 8 000b000e 07 07 Yes 00000111 00000111 8 000b000f 66 66 Yes 01100110 01100110 8 . . . -Joe From owner-freebsd-fs@FreeBSD.ORG Fri Feb 8 22:58:14 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E3D7416A418; Fri, 8 Feb 2008 22:58:14 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from shadow.wildlava.net (shadow.wildlava.net [67.40.138.81]) by mx1.freebsd.org (Postfix) with ESMTP id B090F13C459; Fri, 8 Feb 2008 22:58:14 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from [10.0.3.98] (mail.boulder.swri.edu [65.241.78.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by shadow.wildlava.net (Postfix) with ESMTP id 9A00D8F394; Fri, 8 Feb 2008 15:58:12 -0700 (MST) Message-ID: <47ACDE82.1050100@skyrush.com> Date: Fri, 08 Feb 2008 15:58:10 -0700 From: Joe Peterson User-Agent: Thunderbird 2.0.0.9 (X11/20071119) MIME-Version: 1.0 To: Mark Day References: <47ACD7D4.5050905@skyrush.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Analysis of disk file block with ZFS checksum error X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 22:58:15 -0000 Mark Day wrote: > Based on the subset of data you posted, the bad data looks like ASCII > text. > The bad data from offset a0000 to a000f is: > > ${138AFE{@ > @$$}1 > > The bad data from offset af6c1 to af6c8 is: > > 392A9}@ > > I don't recognize the content beyond that, but I'd guess that somehow > the > contents of some other file managed to overwrite that portion of the bad > file. As for how that happened, I don't know. But if someone > recognizes > where the bad content came from, that might be a clue. Gary/Mark, Good eye! Yes, it indeed does appear to be ASCII. I *thought* something in the repetition when I originally did an od -a looked interesting. I dumped the whole bad section as a string, and here's (partly) what I get: ${138AFE{@ @$$}138AFE}@ @$${138AFF{@ [A3:^80(^91^2146F)] @$$}138AFF}@ @$${138B00{@ @$$}138B00}@ @$${138B01{@ [181:^80(^91^2146F)] @$$}138B01}@ @$${138B02{@ @$$}138B02}@ @$${138B03{@ [2C:^80(^91^2146F)] @$$}138B03}@ @$${138B04{@ @$$}138B04}@ . . . @$${138B8B{@ <(21470=Thu Jan 24 23:20:58 2008)> [117:^80(^91^21470)] @$$}138B8B}@ . . . @$${138C18{@ <(21472=1201242069)>[-2:^80(^82^85)(^83^1B5)(^84=b)(^85=1)(^86=0)(^87=0) (^88=0)(^89^2146C)(^8A=)(^8B=40)(^8C=2e)(^8D^84)(^8E=0)(^90^21472) (^91^21460)] @$$}138C18}@ @$${138C19{@ <(21473=a72f78)>[2:^80(^89^21473)] @$$}138C19}@ @$${138C1A{@ @$$}138C1A}@ . . . and more of the same. Note the date string. There are several like that. Anyone recognize this text format? -Joe From owner-freebsd-fs@FreeBSD.ORG Fri Feb 8 23:02:08 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A0B5916A468; Fri, 8 Feb 2008 23:02:08 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 8A1F213C461; Fri, 8 Feb 2008 23:02:08 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id 3680B1A4D82; Fri, 8 Feb 2008 15:02:08 -0800 (PST) Date: Fri, 8 Feb 2008 15:02:08 -0800 From: Alfred Perlstein To: Joe Peterson Message-ID: <20080208230208.GM99258@elvis.mu.org> References: <47ACD7D4.5050905@skyrush.com> <47ACDE82.1050100@skyrush.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47ACDE82.1050100@skyrush.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Analysis of disk file block with ZFS checksum error X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 23:02:08 -0000 * Joe Peterson [080208 14:58] wrote: > Mark Day wrote: > > Based on the subset of data you posted, the bad data looks like ASCII > > text. > > The bad data from offset a0000 to a000f is: > > > > ${138AFE{@ > > @$$}1 > > > > The bad data from offset af6c1 to af6c8 is: > > > > 392A9}@ > > > > I don't recognize the content beyond that, but I'd guess that somehow > > the > > contents of some other file managed to overwrite that portion of the bad > > file. As for how that happened, I don't know. But if someone > > recognizes > > where the bad content came from, that might be a clue. > > > Gary/Mark, > > Good eye! Yes, it indeed does appear to be ASCII. I *thought* > something in the repetition when I originally did an od -a looked > interesting. > > I dumped the whole bad section as a string, and here's (partly) what I get: > > ${138AFE{@ > @$$}138AFE}@ > > @$${138AFF{@ > [A3:^80(^91^2146F)] > @$$}138AFF}@ > > @$${138B00{@ Looks like terminal output/codes that have been stripped... -Alfred From owner-freebsd-fs@FreeBSD.ORG Fri Feb 8 23:07:22 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9FB6316A418; Fri, 8 Feb 2008 23:07:22 +0000 (UTC) (envelope-from mday@apple.com) Received: from mail-out4.apple.com (mail-out4.apple.com [17.254.13.23]) by mx1.freebsd.org (Postfix) with ESMTP id 81DCF13C465; Fri, 8 Feb 2008 23:07:22 +0000 (UTC) (envelope-from mday@apple.com) Received: from relay12.apple.com (relay12.apple.com [17.128.113.53]) by mail-out4.apple.com (Postfix) with ESMTP id 181D2218F490; Fri, 8 Feb 2008 14:49:55 -0800 (PST) Received: from relay12.apple.com (unknown [127.0.0.1]) by relay12.apple.com (Symantec Mail Security) with ESMTP id 00CE0464002; Fri, 8 Feb 2008 14:49:55 -0800 (PST) X-AuditID: 11807135-a9033bb00000423e-ab-47acdc921fbe Received: from doomsday.apple.com (doomsday.apple.com [17.202.43.217]) by relay12.apple.com (Apple SCV relay) with ESMTP id D0FE9420005; Fri, 8 Feb 2008 14:49:54 -0800 (PST) Message-Id: From: Mark Day To: Joe Peterson In-Reply-To: <47ACD7D4.5050905@skyrush.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v919.2) Date: Fri, 8 Feb 2008 14:49:54 -0800 References: <47ACD7D4.5050905@skyrush.com> X-Mailer: Apple Mail (2.919.2) X-Brightmail-Tracker: AAAAAA== Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Analysis of disk file block with ZFS checksum error X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 23:07:22 -0000 On Feb 8, 2008, at 2:29 PM, Joe Peterson wrote: > For one thing (as I mentioned), only 65536 bytes are bad (and it's > exactly this many, with a few "good" bytes thrown in, but not far from > what matches random chance would produce. Also, all bad bytes have a > zero in the high bit - interesting? Also, near the end of the block, > the bad bytes all go to zero, strangely coincident with the first > "good" > zero in that bad block - not sure if that's coincidence or not. > Also, I > calculated the number of "Bits same" (matching bits) in the good vs. > bad > bytes, and it appears fairly random, so it appears that the bad bytes > are very random in nature and not correlated much at all with the good > bytes. > > So except for the fact that the 2nd half (65536 bytes) of the ZFS > block > are good, the bad block seems to consist of random data, except for > the > string of zero bytes near the end and the zero high-bit. It's not > as if > one bit on the disk flipped - it affects the whole (1/2) block. Does > this seem like a disk error, controller error/bug, cable problem (I > recently put a new cable on, so I doubt this). It seems to me > something > more systemic rather than a random bit error - opinions are more than > welcome. Based on the subset of data you posted, the bad data looks like ASCII text. The bad data from offset a0000 to a000f is: ${138AFE{@ @$$}1 The bad data from offset af6c1 to af6c8 is: 392A9}@ I don't recognize the content beyond that, but I'd guess that somehow the contents of some other file managed to overwrite that portion of the bad file. As for how that happened, I don't know. But if someone recognizes where the bad content came from, that might be a clue. -Mark From owner-freebsd-fs@FreeBSD.ORG Fri Feb 8 23:15:57 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C5FFF16A420; Fri, 8 Feb 2008 23:15:56 +0000 (UTC) (envelope-from gcorcoran@rcn.com) Received: from smtp02.lnh.mail.rcn.net (smtp02.lnh.mail.rcn.net [207.172.157.102]) by mx1.freebsd.org (Postfix) with ESMTP id A2A9B13C45B; Fri, 8 Feb 2008 23:15:55 +0000 (UTC) (envelope-from gcorcoran@rcn.com) Received: from mr02.lnh.mail.rcn.net ([207.172.157.22]) by smtp02.lnh.mail.rcn.net with ESMTP; 08 Feb 2008 17:46:28 -0500 Received: from smtp01.lnh.mail.rcn.net (smtp01.lnh.mail.rcn.net [207.172.4.11]) by mr02.lnh.mail.rcn.net (MOS 3.8.6-GA) with ESMTP id OKA67028; Fri, 8 Feb 2008 17:46:23 -0500 (EST) Received: from 207-172-55-230.c3-0.tlg-ubr5.atw-tlg.pa.cable.rcn.com (HELO [10.56.78.161]) ([207.172.55.230]) by smtp01.lnh.mail.rcn.net with ESMTP; 08 Feb 2008 17:45:19 -0500 Message-ID: <47ACDBB1.40604@rcn.com> Date: Fri, 08 Feb 2008 17:46:09 -0500 From: Gary Corcoran User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: Joe Peterson References: <47ACD7D4.5050905@skyrush.com> In-Reply-To: <47ACD7D4.5050905@skyrush.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Junkmail-Whitelist: YES (by domain whitelist at mr02.lnh.mail.rcn.net) Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Analysis of disk file block with ZFS checksum error X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 23:15:57 -0000 Joe Peterson wrote: > In my experimentation with the ZFS filesystem, I encountered one case of > a file block with a checksum mismatch. Doing a "zpool scrub" revealed > it, and trying to read the file yielded an error - only the part of the > file before the bad block was read (ZFS aborts reading at this point, > which makes sense), resulting in a short file. The reason the CKSUM > error is not fixable is because my ZFS pool contains only one device (no > mirror or RAIDZ), but I do have the original/good version of the file > affected. Here's the output of zpool status (new scrub in process): > > pool: tank > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: http://www.sun.com/msg/ZFS-8000-8A > scrub: scrub in progress, 64.36% done, 0h18m to go > config: > > NAME STATE READ WRITE CKSUM > tank ONLINE 0 0 2 > hda6 ONLINE 0 0 2 > > errors: Permanent errors have been detected in the following files: > > /mnt/tank/fbsd/home/joe/music/jukebox/christmas/Esquivel/ > Merry_XMas_from_the_SpaceAge_Bachelor_Pad/07-Snowfall.mp3 > > > I was curious about what actually happened: was this a ZFS bug, trouble > with its metadata, or truly a bad block? In order to determine this, I > modified ZFS's source code temporarily to ignore the checksum mismatch > and let the file read fully. What I then got was the full-length file > and no errors, showing that there were no disk read errors associated > with the read (I already had assumed this from the fact that zpool > status showed only a non-zero CKSUM count), however, I may have seen > other error counts previously (ZFS resets them to zero on, e.g., > reboot). I received no errors when originally copying this file *to* > the ZFS pool - only on subsequent reads/scrubs. > > (Note that I have posted before about DMA errors in my log for the disk > I am using, but I have had nothing but successful SeaTools tests > (surface scans) of the drive. Jeremy Chadwick had similar issues, as > did others, so I think it is worth investigating if there is some > OS/software cause rather than real HW issues. This is one reason I > wanted to investigate my ZFS checksum issue more deeply.) > > I also have a good backup of the file in question, so I now have two > copies of the file: one good, and one with a bad block. The file is > 3575936 bytes long, and recordsize (in ZFS) is 128K, making the file > about 27 blocks long. Curiously, the bad section of the file is exactly > 65536 bytes long (1/2 a block). The bad block starts at exactly the 5th > 128K block (byte 65536 or hex a0000). > > I wanted to see the characteristics of the bad data. Was just one bit > flipped randomly? No. It is just one bit or set of bits in the bytes > that are affected? It doesn't seem so. Were there any other stange > patterns here? Well, yes, and maybe someout out there with more > knowledge/experience in disk modes of failure will recognize something > (I have included some data below). > > For one thing (as I mentioned), only 65536 bytes are bad (and it's > exactly this many, with a few "good" bytes thrown in, but not far from > what matches random chance would produce. Also, all bad bytes have a > zero in the high bit - interesting? Also, near the end of the block, > the bad bytes all go to zero, strangely coincident with the first "good" > zero in that bad block - not sure if that's coincidence or not. Also, I > calculated the number of "Bits same" (matching bits) in the good vs. bad > bytes, and it appears fairly random, so it appears that the bad bytes > are very random in nature and not correlated much at all with the good > bytes. > > So except for the fact that the 2nd half (65536 bytes) of the ZFS block > are good, the bad block seems to consist of random data, except for the > string of zero bytes near the end and the zero high-bit. It's not as if > one bit on the disk flipped - it affects the whole (1/2) block. Does > this seem like a disk error, controller error/bug, cable problem (I > recently put a new cable on, so I doubt this). It seems to me something > more systemic rather than a random bit error - opinions are more than > welcome. > > Here is some info from a python program I wrote to look at the data > (I've left out spans of essentially uninteresting portions showing > similar stuff, but I can get you the whole thing if interested): > > File pos Good Bad Match Good (bin) Bad (bin) Bits same > 0009fff0 d9 d9 Yes 11011001 11011001 8 > 0009fff1 05 05 Yes 00000101 00000101 8 > 0009fff2 c1 c1 Yes 11000001 11000001 8 > 0009fff3 81 81 Yes 10000001 10000001 8 > 0009fff4 5f 5f Yes 01011111 01011111 8 > 0009fff5 66 66 Yes 01100110 01100110 8 > 0009fff6 5e 5e Yes 01011110 01011110 8 > 0009fff7 a1 a1 Yes 10100001 10100001 8 > 0009fff8 ca ca Yes 11001010 11001010 8 > 0009fff9 9d 9d Yes 10011101 10011101 8 > 0009fffa 00 00 Yes 00000000 00000000 8 > 0009fffb 90 90 Yes 10010000 10010000 8 > 0009fffc 32 32 Yes 00110010 00110010 8 > 0009fffd 62 62 Yes 01100010 01100010 8 > 0009fffe a8 a8 Yes 10101000 10101000 8 > 0009ffff b2 b2 Yes 10110010 10110010 8 > --- Start of bad block --- > 000a0000 d1 24 No 11010001 00100100 2 > 000a0001 6b 7b No 01101011 01111011 7 > 000a0002 d1 31 No 11010001 00110001 5 > 000a0003 56 33 No 01010110 00110011 4 > 000a0004 44 38 No 01000100 00111000 3 > 000a0005 c3 41 No 11000011 01000001 6 > 000a0006 df 46 No 11011111 01000110 4 > 000a0007 07 45 No 00000111 01000101 6 > 000a0008 4c 7b No 01001100 01111011 3 > 000a0009 a0 40 No 10100000 01000000 5 > 000a000a 54 0a No 01010100 00001010 3 > 000a000b 35 40 No 00110101 01000000 3 > 000a000c 88 24 No 10001000 00100100 4 > 000a000d 38 24 No 00111000 00100100 5 > 000a000e f5 7d No 11110101 01111101 6 > 000a000f 28 31 No 00101000 00110001 5 > . > . > . > 000af6c1 d3 33 No 11010011 00110011 5 > 000af6c2 97 39 No 10010111 00111001 3 > 000af6c3 a5 32 No 10100101 00110010 3 > 000af6c4 6a 41 No 01101010 01000001 4 > 000af6c5 16 39 No 00010110 00111001 3 > 000af6c6 f2 7d No 11110010 01111101 3 > 000af6c7 21 40 No 00100001 01000000 5 > 000af6c8 52 0a No 01010010 00001010 5 > 000af6c9 00 00 Yes 00000000 00000000 8 > 000af6ca 2c 00 No 00101100 00000000 5 > 000af6cb 42 00 No 01000010 00000000 6 > 000af6cc 31 00 No 00110001 00000000 5 > 000af6cd a1 00 No 10100001 00000000 5 > 000af6ce d1 00 No 11010001 00000000 4 > 000af6cf 90 00 No 10010000 00000000 6 > 000af6d0 9c 00 No 10011100 00000000 4 > . > . > . > 000afff8 26 00 No 00100110 00000000 5 > 000afff9 8c 00 No 10001100 00000000 5 > 000afffa a8 00 No 10101000 00000000 5 > 000afffb 0c 00 No 00001100 00000000 6 > 000afffc f1 00 No 11110001 00000000 3 > 000afffd 93 00 No 10010011 00000000 4 > 000afffe 2c 00 No 00101100 00000000 5 > 000affff 2e 00 No 00101110 00000000 4 > --- End of bad block --- > 000b0000 62 62 Yes 01100010 01100010 8 > 000b0001 56 56 Yes 01010110 01010110 8 > 000b0002 91 91 Yes 10010001 10010001 8 > 000b0003 04 04 Yes 00000100 00000100 8 > 000b0004 01 01 Yes 00000001 00000001 8 > 000b0005 2d 2d Yes 00101101 00101101 8 > 000b0006 0e 0e Yes 00001110 00001110 8 > 000b0007 89 89 Yes 10001001 10001001 8 > 000b0008 8a 8a Yes 10001010 10001010 8 > 000b0009 ad ad Yes 10101101 10101101 8 > 000b000a 4e 4e Yes 01001110 01001110 8 > 000b000b a3 a3 Yes 10100011 10100011 8 > 000b000c 13 13 Yes 00010011 00010011 8 > 000b000d 4d 4d Yes 01001101 01001101 8 > 000b000e 07 07 Yes 00000111 00000111 8 > 000b000f 66 66 Yes 01100110 01100110 8 > . > . > . A wild guess: bytes all with the top bit zero, plus I see occasional 0x0a bytes in there - it looks like that 1/2 block might have gotten overwritten with some ASCII text. Don't ask me how though... (but if it did, I am afraid I would suspect a bug in ZFS :-( ) Gary From owner-freebsd-fs@FreeBSD.ORG Fri Feb 8 23:36:12 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D646916A418; Fri, 8 Feb 2008 23:36:12 +0000 (UTC) (envelope-from freebsd@chillt.de) Received: from dd15624.kasserver.com (dd15624.kasserver.com [85.13.136.215]) by mx1.freebsd.org (Postfix) with ESMTP id 9556213C448; Fri, 8 Feb 2008 23:36:12 +0000 (UTC) (envelope-from freebsd@chillt.de) Received: from hundertwasser.cs.tcd.ie (dslb-084-060-126-218.pools.arcor-ip.net [84.60.126.218]) by dd15624.kasserver.com (Postfix) with ESMTP id 8BED9181DEC20; Sat, 9 Feb 2008 00:09:05 +0100 (CET) Message-ID: <47ACE12A.6010205@chillt.de> Date: Fri, 08 Feb 2008 23:09:30 +0000 From: Bartosz Fabianowski User-Agent: Thunderbird 2.0.0.9 (X11/20080124) MIME-Version: 1.0 To: Joe Peterson References: <47ACD7D4.5050905@skyrush.com> <47ACDE82.1050100@skyrush.com> In-Reply-To: <47ACDE82.1050100@skyrush.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Analysis of disk file block with ZFS checksum error X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 23:36:12 -0000 I'd say that's the mork database format [1,2], as used by Mozilla products, for example in the Firefox history.dat file. - Bartosz [1] http://www.mozilla.org/mailnews/arch/mork/primer.txt [2] http://www.jwz.org/hacks/mork.pl From owner-freebsd-fs@FreeBSD.ORG Fri Feb 8 23:47:58 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3133916A418 for ; Fri, 8 Feb 2008 23:47:58 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by mx1.freebsd.org (Postfix) with ESMTP id E00F213C46A for ; Fri, 8 Feb 2008 23:47:57 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.2/8.14.2) id m18NTPfx095860; Fri, 8 Feb 2008 17:29:25 -0600 (CST) (envelope-from dan) Date: Fri, 8 Feb 2008 17:29:24 -0600 From: Dan Nelson To: Joe Peterson Message-ID: <20080208232923.GD85696@dan.emsphone.com> References: <47ACD7D4.5050905@skyrush.com> <47ACDE82.1050100@skyrush.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47ACDE82.1050100@skyrush.com> X-OS: FreeBSD 7.0-PRERELEASE User-Agent: Mutt/1.5.17 (2007-11-01) Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Analysis of disk file block with ZFS checksum error X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 23:47:58 -0000 In the last episode (Feb 08), Joe Peterson said: > Mark Day wrote: > > Based on the subset of data you posted, the bad data looks like > > ASCII text. The bad data from offset a0000 to a000f is: > > > > ${138AFE{@ > > @$$}1 > > > > The bad data from offset af6c1 to af6c8 is: > > > > 392A9}@ > > > > I don't recognize the content beyond that, but I'd guess that > > somehow the contents of some other file managed to overwrite that > > portion of the bad file. As for how that happened, I don't know. > > But if someone recognizes where the bad content came from, that > > might be a clue. > > Good eye! Yes, it indeed does appear to be ASCII. I *thought* > something in the repetition when I originally did an od -a looked > interesting. > > I dumped the whole bad section as a string, and here's (partly) what I get: > > @$${138B8B{@ > <(21470=Thu Jan 24 23:20:58 2008)> > [117:^80(^91^21470)] > @$$}138B8B}@ ... > @$${138C18{@ > <(21472=1201242069)>[-2:^80(^82^85)(^83^1B5)(^84=b)(^85=1)(^86=0)(^87=0) > (^88=0)(^89^2146C)(^8A=)(^8B=40)(^8C=2e)(^8D^84)(^8E=0)(^90^21472) > (^91^21460)] > @$$}138C18}@ > > and more of the same. Note the date string. There are several like > that. Anyone recognize this text format? It's a Mork database from the Mozilla project: http://developer.mozilla.org/en/docs/Mork_Structure#Rows -- Dan Nelson dnelson@allantgroup.com From owner-freebsd-fs@FreeBSD.ORG Fri Feb 8 23:54:34 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 772D616A418; Fri, 8 Feb 2008 23:54:34 +0000 (UTC) (envelope-from cdillon@wolves.k12.mo.us) Received: from mail.wolves.k12.mo.us (mail.wolves.k12.mo.us [207.160.214.1]) by mx1.freebsd.org (Postfix) with ESMTP id 4828E13C467; Fri, 8 Feb 2008 23:54:34 +0000 (UTC) (envelope-from cdillon@wolves.k12.mo.us) Received: from localhost (localhost [127.0.0.1]) by mail.wolves.k12.mo.us (Postfix) with ESMTP id 1558FB841; Fri, 8 Feb 2008 17:35:19 -0600 (CST) X-Virus-Scanned: amavisd-new at wolves.k12.mo.us Received: from mail.wolves.k12.mo.us ([127.0.0.1]) by localhost (mail.wolves.k12.mo.us [127.0.0.1]) (amavisd-new, port 10024) with LMTP id n-2JjJY+zU0K; Fri, 8 Feb 2008 17:35:18 -0600 (CST) Received: from wolves.k12.mo.us (localhost [127.0.0.1]) by mail.wolves.k12.mo.us (Postfix) with ESMTP id E20D8B820; Fri, 8 Feb 2008 17:35:17 -0600 (CST) Received: from rstech21.int.wolves.k12.mo.us (rstech21.int.wolves.k12.mo.us [10.1.3.201]) by www.wolves.k12.mo.us (Horde MIME library) with HTTP; Fri, 08 Feb 2008 17:35:17 -0600 Message-ID: <20080208173517.rdtobnxqg4g004c4@www.wolves.k12.mo.us> Date: Fri, 08 Feb 2008 17:35:17 -0600 From: Chris Dillon To: Joe Peterson References: <47ACD7D4.5050905@skyrush.com> <47ACDE82.1050100@skyrush.com> In-Reply-To: <47ACDE82.1050100@skyrush.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.1.5) / FreeBSD-6.2 Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Analysis of disk file block with ZFS checksum error X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2008 23:54:34 -0000 Quoting Joe Peterson : > I dumped the whole bad section as a string, and here's (partly) what I get: [...edited for brevity...] > @$${138B8B{@ > <(21470=Thu Jan 24 23:20:58 2008)> > [117:^80(^91^21470)] > @$$}138B8B}@ > > @$${138C18{@ > <(21472=1201242069)>[-2:^80(^82^85)(^83^1B5)(^84=b)(^85=1)(^86=0)(^87=0) > (^88=0)(^89^2146C)(^8A=)(^8B=40)(^8C=2e)(^8D^84)(^8E=0)(^90^21472) > (^91^21460)] > @$$}138C18}@ > > @$${138C19{@ > <(21473=a72f78)>[2:^80(^89^21473)] > @$$}138C19}@ > > @$${138C1A{@ > @$$}138C1A}@ > > and more of the same. Note the date string. There are several like > that. Anyone recognize this text format? That is a chunk of a Mozilla Mork-format database. Perhaps the Firefox URL history or address book from Thunderbird. -- Chris Dillon - NetEng/SysAdm Reeds Spring R-IV School District Technology Department 175 Elementary Rd. Reeds Spring, MO 65737 Voice: 417-272-8266 Fax: 417-272-0015 From owner-freebsd-fs@FreeBSD.ORG Sat Feb 9 00:15:46 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9387716A418; Sat, 9 Feb 2008 00:15:46 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from shadow.wildlava.net (shadow.wildlava.net [67.40.138.81]) by mx1.freebsd.org (Postfix) with ESMTP id 61BFB13C467; Sat, 9 Feb 2008 00:15:46 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from [10.0.3.98] (mail.boulder.swri.edu [65.241.78.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by shadow.wildlava.net (Postfix) with ESMTP id 8FB708F42A; Fri, 8 Feb 2008 17:15:45 -0700 (MST) Message-ID: <47ACF0AE.3040802@skyrush.com> Date: Fri, 08 Feb 2008 17:15:42 -0700 From: Joe Peterson User-Agent: Thunderbird 2.0.0.9 (X11/20071119) MIME-Version: 1.0 To: Chris Dillon References: <47ACD7D4.5050905@skyrush.com> <47ACDE82.1050100@skyrush.com> <20080208173517.rdtobnxqg4g004c4@www.wolves.k12.mo.us> In-Reply-To: <20080208173517.rdtobnxqg4g004c4@www.wolves.k12.mo.us> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Analysis of disk file block with ZFS checksum error X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 00:15:46 -0000 Chris Dillon wrote: > That is a chunk of a Mozilla Mork-format database. Perhaps the > Firefox URL history or address book from Thunderbird. Interesting (thanks to all who recognized Mork). I do use Firefox and Thunderbird, so it's feasible, but how the heck would a piece of one of those files find its way into 1/2 of a ZFS block in one of my mp3 files? I wonder if it could have been done on write when the file was copied to the ZFS pool (maybe some write-caching issue?), but I thought ZFS would have verified the block after write. It seems unlikely that it would get changed later - I never rewrote that file after the original copy... -Joe From owner-freebsd-fs@FreeBSD.ORG Sat Feb 9 00:26:28 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 463F716A417 for ; Sat, 9 Feb 2008 00:26:28 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outN.internet-mail-service.net (outN.internet-mail-service.net [216.240.47.237]) by mx1.freebsd.org (Postfix) with ESMTP id 29B5D13C46B for ; Sat, 9 Feb 2008 00:26:28 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Fri, 08 Feb 2008 16:26:27 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id DAC1E127152; Fri, 8 Feb 2008 16:26:26 -0800 (PST) Message-ID: <47ACF338.3020802@elischer.org> Date: Fri, 08 Feb 2008 16:26:32 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Joe Peterson References: <47ACD7D4.5050905@skyrush.com> <47ACDE82.1050100@skyrush.com> <20080208173517.rdtobnxqg4g004c4@www.wolves.k12.mo.us> <47ACF0AE.3040802@skyrush.com> In-Reply-To: <47ACF0AE.3040802@skyrush.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Analysis of disk file block with ZFS checksum error X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 00:26:28 -0000 Joe Peterson wrote: > Chris Dillon wrote: >> That is a chunk of a Mozilla Mork-format database. Perhaps the >> Firefox URL history or address book from Thunderbird. > > Interesting (thanks to all who recognized Mork). I do use Firefox and > Thunderbird, so it's feasible, but how the heck would a piece of one of > those files find its way into 1/2 of a ZFS block in one of my mp3 files? > I wonder if it could have been done on write when the file was copied > to the ZFS pool (maybe some write-caching issue?), but I thought ZFS > would have verified the block after write. It seems unlikely that it > would g it could be an old file.. what kind of disks? I had a scenario where 3ware controllers were just failing to write to a drive in the array, so old data showed through. it was possible by looking to see where the boundary between good and bad was, to identify the culprit.. the filesystem and the partitions and the raids all were on different alignments so teh only part of the system that had a boundary that aligned with the bad data was the physical stripes laid down by the controller. It was 64k stripes and 64k data missing, exactly on stripe boundaries. Due to the fact that FreeBSD had partitioned the drive staring at 63 blocks in, nothing else aligned with the problem. > -Joe > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Sat Feb 9 03:09:48 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3E18516A417; Sat, 9 Feb 2008 03:09:48 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from shadow.wildlava.net (shadow.wildlava.net [67.40.138.81]) by mx1.freebsd.org (Postfix) with ESMTP id 1151913C47E; Sat, 9 Feb 2008 03:09:47 +0000 (UTC) (envelope-from joe@skyrush.com) Received: from [67.40.138.82] (crater.wildlava.net [67.40.138.82]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by shadow.wildlava.net (Postfix) with ESMTP id 033158F42A; Fri, 8 Feb 2008 20:09:47 -0700 (MST) Message-ID: <47AD1979.8020704@skyrush.com> Date: Fri, 08 Feb 2008 20:09:45 -0700 From: Joe Peterson User-Agent: Thunderbird 2.0.0.9 (X11/20071127) MIME-Version: 1.0 To: Julian Elischer References: <47ACD7D4.5050905@skyrush.com> <47ACDE82.1050100@skyrush.com> <20080208173517.rdtobnxqg4g004c4@www.wolves.k12.mo.us> <47ACF0AE.3040802@skyrush.com> <47ACF338.3020802@elischer.org> In-Reply-To: <47ACF338.3020802@elischer.org> X-Enigmail-Version: 0.95.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Analysis of disk file block with ZFS checksum error X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 03:09:48 -0000 Julian Elischer wrote: > it could be an old file.. > what kind of disks? It's a Seagate ST3500630A parallel ATA drive. > I had a scenario where 3ware controllers were just failing to write to > a drive in the array, so old data showed through. I have an Intel ICH4 controller - nothing unusual. > the filesystem and the partitions and the raids all were on different > alignments so teh only part of the system that had a boundary that > aligned with the bad data was the physical stripes laid down by the > controller. It was 64k stripes and 64k data missing, exactly on > stripe boundaries. Due to the fact that FreeBSD had partitioned the > drive staring at 63 blocks in, nothing else aligned with the problem. Hmm, well this is a straight-forward disk situation - never used RAID on this drive. Give what is happening, I wonder the changes of it being HW, OS, or a filesystem issue. -Joe From owner-freebsd-fs@FreeBSD.ORG Sat Feb 9 04:22:09 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5017E16A418 for ; Sat, 9 Feb 2008 04:22:09 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outU.internet-mail-service.net (outU.internet-mail-service.net [216.240.47.244]) by mx1.freebsd.org (Postfix) with ESMTP id 3A03F13C457 for ; Sat, 9 Feb 2008 04:22:09 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.40) with ESMTP; Fri, 08 Feb 2008 20:22:08 -0800 Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id B145A12714B; Fri, 8 Feb 2008 20:22:07 -0800 (PST) Message-ID: <47AD2A75.8010908@elischer.org> Date: Fri, 08 Feb 2008 20:22:13 -0800 From: Julian Elischer User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Joe Peterson References: <47ACD7D4.5050905@skyrush.com> <47ACDE82.1050100@skyrush.com> <20080208173517.rdtobnxqg4g004c4@www.wolves.k12.mo.us> <47ACF0AE.3040802@skyrush.com> <47ACF338.3020802@elischer.org> <47AD1979.8020704@skyrush.com> In-Reply-To: <47AD1979.8020704@skyrush.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Analysis of disk file block with ZFS checksum error X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 04:22:09 -0000 Joe Peterson wrote: > Julian Elischer wrote: >> it could be an old file.. >> what kind of disks? > > It's a Seagate ST3500630A parallel ATA drive. > >> I had a scenario where 3ware controllers were just failing to write to >> a drive in the array, so old data showed through. > > I have an Intel ICH4 controller - nothing unusual. > >> the filesystem and the partitions and the raids all were on different >> alignments so teh only part of the system that had a boundary that >> aligned with the bad data was the physical stripes laid down by the >> controller. It was 64k stripes and 64k data missing, exactly on >> stripe boundaries. Due to the fact that FreeBSD had partitioned the >> drive staring at 63 blocks in, nothing else aligned with the problem. > > Hmm, well this is a straight-forward disk situation - never used RAID on > this drive. Give what is happening, I wonder the changes of it being > HW, OS, or a filesystem issue. > > -Joe still, see whether the 64k lines up with the drive or with the filesystem (if the filesystem is not on an exact 64k boundary of the drive). From owner-freebsd-fs@FreeBSD.ORG Sat Feb 9 17:40:04 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6180416A49A for ; Sat, 9 Feb 2008 17:40:04 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 1B45A13C45B for ; Sat, 9 Feb 2008 17:40:03 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from root by ciao.gmane.org with local (Exim 4.43) id 1JNtgI-000893-CD for freebsd-fs@freebsd.org; Sat, 09 Feb 2008 17:40:02 +0000 Received: from www.creo.hu ([217.113.62.14]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 09 Feb 2008 17:40:02 +0000 Received: from csaba-ml by www.creo.hu with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 09 Feb 2008 17:40:02 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Csaba Henk Date: Sat, 9 Feb 2008 17:36:00 +0000 (UTC) Lines: 24 Message-ID: References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <20080207071314.GO99258@elvis.mu.org> <47AABEEF.7020807@FreeBSD.org> <7ad7ddd90802070135y6eeaded6lcffc6b5751ee5c40@mail.gmail.com> X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: www.creo.hu User-Agent: slrn/0.9.8.1 (FreeBSD) Sender: news Cc: freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 17:40:04 -0000 On 2008-02-07, Ulrich Spoerlein wrote: > On Feb 7, 2008 9:18 AM, Doug Barton wrote: >> Alfred Perlstein wrote: >> >> > Maybe a nicer way of saying/asking would be to ask: >> > >> > Is the FUSE replacement going to be tested to the point where it's >> > better than then current NTFS code? >> >> Given that the current NTFS code in the base panics within minutes of >> any kind of serious access, and has the ability to take the other >> filesystems down with it (including UFS2) that won't be hard. > > Please keep in mind, that the current FUSE port to FreeBSD is lagging > behind and I can reproducibly panic -CURRENT using fuse-ntfs, too. Hey, I'm catching up with it. Regarding your panic: we have discussed it: I could not reproduce it and you did not succeed to give me enough information (no offense here, it's not always easy...). I suggest you to file a PR, maybe others can reproduce it. Csaba From owner-freebsd-fs@FreeBSD.ORG Sat Feb 9 17:50:04 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4310616A4C8 for ; Sat, 9 Feb 2008 17:50:04 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id F301313C44B for ; Sat, 9 Feb 2008 17:50:03 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from root by ciao.gmane.org with local (Exim 4.43) id 1JNtpy-00007M-6k for freebsd-fs@freebsd.org; Sat, 09 Feb 2008 17:50:02 +0000 Received: from www.creo.hu ([217.113.62.14]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 09 Feb 2008 17:50:02 +0000 Received: from csaba-ml by www.creo.hu with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 09 Feb 2008 17:50:02 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Csaba Henk Date: Sat, 9 Feb 2008 17:39:06 +0000 (UTC) Lines: 7 Message-ID: References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <70e8236f0802070321n9097d3fy1b39f637b3c2a06@mail.gmail.com> X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: www.creo.hu User-Agent: slrn/0.9.8.1 (FreeBSD) Sender: news Cc: freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 17:50:04 -0000 On 2008-02-07, Joao Barros wrote: > Yes if FUSE ntfs can have performance on par with the current ntfs support. Userspace components are LGPL / GPL licensed, kernel components are BSD licensed. Csaba From owner-freebsd-fs@FreeBSD.ORG Sat Feb 9 17:55:07 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C602916A419 for ; Sat, 9 Feb 2008 17:55:07 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 8130213C4D1 for ; Sat, 9 Feb 2008 17:55:07 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from root by ciao.gmane.org with local (Exim 4.43) id 1JNtuo-0000Mv-IP for freebsd-fs@freebsd.org; Sat, 09 Feb 2008 17:55:02 +0000 Received: from www.creo.hu ([217.113.62.14]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 09 Feb 2008 17:55:02 +0000 Received: from csaba-ml by www.creo.hu with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 09 Feb 2008 17:55:02 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Csaba Henk Date: Sat, 9 Feb 2008 17:51:36 +0000 (UTC) Lines: 13 Message-ID: References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <1202395350.2126.14.camel@localhost> X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: www.creo.hu User-Agent: slrn/0.9.8.1 (FreeBSD) Sender: news Cc: freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 17:55:07 -0000 On 2008-02-07, martinko wrote: >> If the base NTFS support was axed, would the fuse-kmod, or equivalent, >> find its way into the base? It's now not far from the point where it could be included. > And what about puffs from NetBSD ? > Wouldn't it be better to import puffs and ReFUSE ? A PUFFS port probably would need more work than sort out the not too several issues with fuse4bsd. Csaba From owner-freebsd-fs@FreeBSD.ORG Sat Feb 9 19:59:57 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 86F8E16A469; Sat, 9 Feb 2008 19:59:57 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id 472AB13C45B; Sat, 9 Feb 2008 19:59:57 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (localhost [127.0.0.1]) by spam.des.no (Postfix) with ESMTP id C10F12082; Sat, 9 Feb 2008 20:59:48 +0100 (CET) X-Spam-Tests: AWL X-Spam-Learn: disabled X-Spam-Score: -0.3/3.0 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on tim.des.no Received: from ds4.des.no (des.no [80.203.243.180]) by smtp.des.no (Postfix) with ESMTP id 3ED4B207F; Sat, 9 Feb 2008 20:59:48 +0100 (CET) Received: by ds4.des.no (Postfix, from userid 1001) id 120D78449D; Sat, 9 Feb 2008 20:59:48 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Csaba Henk References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <70e8236f0802070321n9097d3fy1b39f637b3c2a06@mail.gmail.com> Date: Sat, 09 Feb 2008 20:59:47 +0100 In-Reply-To: (Csaba Henk's message of "Sat\, 9 Feb 2008 17\:39\:06 +0000 \(UTC\)") Message-ID: <867ihdc34c.fsf@ds4.des.no> User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/22.1 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org, freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 19:59:57 -0000 Csaba Henk writes: > Userspace components are LGPL / GPL licensed, kernel components are > BSD licensed. Are you planning to have both the kernel part and the userland part committed to base? How much work would you guess it would take to reimplement the userland part under a BSD license? DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Sat Feb 9 21:42:29 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 843BD16A46C for ; Sat, 9 Feb 2008 21:42:29 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.159]) by mx1.freebsd.org (Postfix) with ESMTP id A178E13C4EA for ; Sat, 9 Feb 2008 21:42:28 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by fg-out-1718.google.com with SMTP id 16so3480801fgg.35 for ; Sat, 09 Feb 2008 13:42:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=dIo9OfBGPa0bD7CaoSePC9lM2FPZ72znfQnuYdGcY0s=; b=trJyEftHZ8a0yJtOIh0mOpmGassVNZb6uLLrN0MUEULc11McScBa5Qr1h1RLPmzcpAS/mrt80FLKHAprIIk6VHDrI+0qdMHlJwNaLIPiwp16AQPQncVbgcjK+KU7BF6b2mjsDvJ7pB8HHpVM4+DvUMcmrybE5u8VT6gVsN9JIm4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=pLEJcl4DbU7LMex4WS5msHYbN/B78xlz1aOBOVfRtvazPt7DLZx61FAIYJFdfVw290qxFehxiZ/2jwbm7LE3BGltiHg+mJbRAmndAcUi2iKc43ok4/VR/vj+ICyyyZ9Qrbb4Ujpz4FfR0UvHTJiqdOdSzYqtwlVw6FeUEBeHF9M= Received: by 10.86.4.2 with SMTP id 2mr2202809fgd.7.1202593347129; Sat, 09 Feb 2008 13:42:27 -0800 (PST) Received: by 10.86.28.19 with HTTP; Sat, 9 Feb 2008 13:42:27 -0800 (PST) Message-ID: <3bbf2fe10802091342u189097cna04cbad8d0177efe@mail.gmail.com> Date: Sat, 9 Feb 2008 22:42:27 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Scot Hetzel" In-Reply-To: <790a9fff0802091338h4d2698d7v74fb029694bcf9bf@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <20080208162231.R17387@fledge.watson.org> <47ACD532.2090206@FreeBSD.org> <790a9fff0802091338h4d2698d7v74fb029694bcf9bf@mail.gmail.com> X-Google-Sender-Auth: 49245fc8f2919dd9 Cc: Yar Tikhiy , Doug Barton , Jeff Roberson , freebsd-fs@freebsd.org, Robert Watson , freebsd-arch@freebsd.org, Eygene Ryabinkin Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 21:42:29 -0000 2008/2/9, Scot Hetzel : > On 2/8/08, Doug Barton wrote: > > Robert Watson wrote: > > > > > > On Fri, 8 Feb 2008, Eygene Ryabinkin wrote: > > > > > >>> - Are you interested in maintaining it? > > >> > > >> Yes. If you can throw the buglist for NTFS on me, I will be very > > >> grateful. > > > > > > Sign the man up :-). > > > > > > Sounds good to me. I've posted on -current several times since > > December with crash dumps, etc. I think the problem at this point is > > that we don't know what the bugs are, or else I suspect at least some > > of them would be fixed already. > > > > I believe I have solved the panics with the NTFS filesystem by porting > the NetBSD lockmgr -> mutex locking changes to our NTFS filesystem > implementation. > > The locking changes are available in PR 120483 > > http://www.freebsd.org/cgi/query-pr.cgi?pr=120483 > > I have also submitted a style change PR which will allow us to easily > compare the NetBSD implementation to our NTFS implementation in PR > 120482: > > http://www.freebsd.org/cgi/query-pr.cgi?pr=120482 Scot, why didn't you modify the patch in a way it reduces diffs against our HEAD as I suggested privately? :( Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-fs@FreeBSD.ORG Sat Feb 9 22:02:54 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5FEF816A41A for ; Sat, 9 Feb 2008 22:02:54 +0000 (UTC) (envelope-from swhetzel@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.158]) by mx1.freebsd.org (Postfix) with ESMTP id 7B23913C474 for ; Sat, 9 Feb 2008 22:02:53 +0000 (UTC) (envelope-from swhetzel@gmail.com) Received: by fg-out-1718.google.com with SMTP id 16so3484890fgg.35 for ; Sat, 09 Feb 2008 14:02:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=H6gAETpq1rVv7MLHK6yFdMVA37TaJQAVarMT2cWLD3w=; b=qTPSz6/YIz8SThkjhhNwRLNlgzg29Z61Am3Rt9EMFIrzU8wg2vNmQdNfgDtWZ+XLmySuBOapQ7F5MzxrOQyk9ul2UZmFc5GixBXJPjwkiCOb0SD/qvb+bP6bdQbS/sag6k1c6rsMSPuCZOsMZ43YxIRpIQPlJU+aXeR+BJEl3w8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=XMKZ2tQxZkBuitlSX33FGidVU8Uq9OumIDgs5zb2QQYkyvjZ4/Tf1YuoE4xXQA3/CcuUluGmPLWARISb+YFGakvFu8BCR2ib+2r56MEAJDMY/1iLbf6NcQTBow6YJdPpTb61ytnvQ/YdhxhzrlVDBdd5+f9uZQx5SddtST/vsBU= Received: by 10.86.70.8 with SMTP id s8mr13277536fga.29.1202593132153; Sat, 09 Feb 2008 13:38:52 -0800 (PST) Received: by 10.86.99.17 with HTTP; Sat, 9 Feb 2008 13:38:52 -0800 (PST) Message-ID: <790a9fff0802091338h4d2698d7v74fb029694bcf9bf@mail.gmail.com> Date: Sat, 9 Feb 2008 15:38:52 -0600 From: "Scot Hetzel" To: "Eygene Ryabinkin" , "Doug Barton" In-Reply-To: <47ACD532.2090206@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> <20080208162231.R17387@fledge.watson.org> <47ACD532.2090206@FreeBSD.org> Cc: freebsd-fs@freebsd.org, Yar Tikhiy , Jeff Roberson , Attilio Rao , Robert Watson , freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 22:02:54 -0000 On 2/8/08, Doug Barton wrote: > Robert Watson wrote: > > > > On Fri, 8 Feb 2008, Eygene Ryabinkin wrote: > > > >>> - Are you interested in maintaining it? > >> > >> Yes. If you can throw the buglist for NTFS on me, I will be very > >> grateful. > > > > Sign the man up :-). > > > Sounds good to me. I've posted on -current several times since > December with crash dumps, etc. I think the problem at this point is > that we don't know what the bugs are, or else I suspect at least some > of them would be fixed already. > I believe I have solved the panics with the NTFS filesystem by porting the NetBSD lockmgr -> mutex locking changes to our NTFS filesystem implementation. The locking changes are available in PR 120483 http://www.freebsd.org/cgi/query-pr.cgi?pr=120483 I have also submitted a style change PR which will allow us to easily compare the NetBSD implementation to our NTFS implementation in PR 120482: http://www.freebsd.org/cgi/query-pr.cgi?pr=120482 After the locking patch and style patch have been applied, all that is left is to review the remaining changes between the FreeBSD and NetBSD NTFS code to see if we should port additional fixes from NetBSD. Scot From owner-freebsd-fs@FreeBSD.ORG Sat Feb 9 23:11:51 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20CEF16A507 for ; Sat, 9 Feb 2008 23:11:49 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.156]) by mx1.freebsd.org (Postfix) with ESMTP id D797F13C46E for ; Sat, 9 Feb 2008 23:11:48 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by fg-out-1718.google.com with SMTP id 16so3498209fgg.35 for ; Sat, 09 Feb 2008 15:11:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=68CCEfZiOKGKAEMClAdnUv1hodZZhLRYoapwo9PhneM=; b=qxNymWPj47zoTQnmNyuTM7say4U905hHYoPTmE+9Rb0DYT/UO38QiJx6crVHXPkNA5/d+us/0MQ3i3llISmrSKLARuHvRoSKWZj3OqN1vTSkQX489MUX/8rNi2LrO+0FMps8T7L0PT8KfJ/W1QOuKQL0G8fRKO2X/JPT9FW8d8k= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=QNoTw0rEnQ2HjH1pfyXKNcNNHLZxCJbKfap7qQebbnCYWWciN1IQFeMlIne1HVtX9ZededxQYVnATmzlDsY30N4n5WtOJY+DR90fURk1m6wI5wy5wyVa1t/2Yhn7REwtsnqW97dKJzsJqM4EZ+5lfLUgSXEMxP8kpDJtjTtPjjA= Received: by 10.86.53.8 with SMTP id b8mr13323795fga.64.1202598707310; Sat, 09 Feb 2008 15:11:47 -0800 (PST) Received: by 10.86.28.19 with HTTP; Sat, 9 Feb 2008 15:11:47 -0800 (PST) Message-ID: <3bbf2fe10802091511p201ee404g76a835f49d9291ac@mail.gmail.com> Date: Sun, 10 Feb 2008 00:11:47 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Yar Tikhiy" In-Reply-To: <20080209203317.GB62234@comp.chem.msu.su> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <790a9fff0801312241s346068b6s40fcae71ebbf546@mail.gmail.com> <3bbf2fe10802011041t28e419c9n5f0f6f34d6450184@mail.gmail.com> <20080205162217.GA56373@comp.chem.msu.su> <3bbf2fe10802051156p1cc6ea67t7938a60e306323ce@mail.gmail.com> <20080206112930.GD7592@comp.chem.msu.su> <3bbf2fe10802060549u66b1067cy4bb9d4232ccef05d@mail.gmail.com> <20080206144802.GH7592@comp.chem.msu.su> <3bbf2fe10802060652y3782d023k9c5299168a81093d@mail.gmail.com> <3bbf2fe10802060657h1dbc93e2lf3fd1b0c6843f674@mail.gmail.com> <20080209203317.GB62234@comp.chem.msu.su> X-Google-Sender-Auth: 88db5c7db2d3e34b Cc: yar@freebsd.org, Scot Hetzel , Doug Barton , freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: panic: System call lstat returning with 1 locks held X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Feb 2008 23:11:51 -0000 2008/2/9, Yar Tikhiy : > On Wed, Feb 06, 2008 at 03:57:58PM +0100, Attilio Rao wrote: > > 2008/2/6, Attilio Rao : > > > 2008/2/6, Yar Tikhiy : > > > > > > > On Wed, Feb 06, 2008 at 02:49:49PM +0100, Attilio Rao wrote: > > > > [...] > > > > > > > > > Want to see if this bt has been helpful? :) > > > > > Can you try the attached patch and see if kernel rings a bell?: > > > > > http://www.freebsd.org/~attilio/ntfs_debug.diff > > > > > > > > > > > > The kernel just panics. :-) > > > > > > > > > This is the new I wanted to know! :) > > > With better checks in lockmgr code, we would have caught more > > > informations about it. > > > > > > Can you please now add DDB support and once it breaks in DDB do a > > > 'show alllocks' and maybe other small investigations? > > > This should shade a light for us. > > > > Could you please enable NTFS_DEBUG too and maybe see, when the kernel > > panics, what is the value of i_usecount for the specified ip? > > I want to exclude refcount leaking. > > > i_usecount is just zero for the faulty ip: > With the determinant yar's help, I think I found how the lock leak happens. Basically, in ntfs_ntput() the inode refcount (ip->i_usecount) is decreased and after checked. When check its i_usecount == 0, it means that initially i_usecount == 1 which also means the lockmgr() was held. For the i_usecount == 0 logic, however, no lockmgr release operation is previewed. This patch should fix the NTFS problems even with stricter assertions I plan to commit rather soon: http://www.freebsd.org/~attilio/ntfs.diff This patch was initially provided by yar as a workaround, but it moved me in analyzing refcount handling and finding this bug. Please test and report if it solves problems for you. Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein