From owner-freebsd-amd64@FreeBSD.ORG Mon Oct 10 04:59:16 2011 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 713D8106566B; Mon, 10 Oct 2011 04:59:16 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 491988FC14; Mon, 10 Oct 2011 04:59:16 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9A4xG9a053533; Mon, 10 Oct 2011 04:59:16 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9A4xG40053529; Mon, 10 Oct 2011 04:59:16 GMT (envelope-from linimon) Date: Mon, 10 Oct 2011 04:59:16 GMT Message-Id: <201110100459.p9A4xG40053529@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-amd64@FreeBSD.org, freebsd-bugs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/161212: [radeon] [panic] Radeon 4650 on amd64 crashes kernel on startx X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Oct 2011 04:59:16 -0000 Old Synopsis: Radeon 4650 on amd64 crashes kernel on startx New Synopsis: [radeon] [panic] Radeon 4650 on amd64 crashes kernel on startx Responsible-Changed-From-To: freebsd-amd64->freebsd-bugs Responsible-Changed-By: linimon Responsible-Changed-When: Mon Oct 10 04:58:36 UTC 2011 Responsible-Changed-Why: reclassify. http://www.freebsd.org/cgi/query-pr.cgi?pr=161212 From owner-freebsd-amd64@FreeBSD.ORG Mon Oct 10 05:08:59 2011 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E935106564A; Mon, 10 Oct 2011 05:08:59 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 367278FC08; Mon, 10 Oct 2011 05:08:59 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9A58xAR067900; Mon, 10 Oct 2011 05:08:59 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9A58xXf067896; Mon, 10 Oct 2011 05:08:59 GMT (envelope-from linimon) Date: Mon, 10 Oct 2011 05:08:59 GMT Message-Id: <201110100508.p9A58xXf067896@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-amd64@FreeBSD.org, freebsd-bugs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/161326: [build] [patch] cannot buildworld FreeBSD-9.0-BETA3 (RELENG_9) X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Oct 2011 05:08:59 -0000 Old Synopsis: [PATCH] cannot buildworld FreeBSD-9.0-BETA3 (RELENG_9) New Synopsis: [build] [patch] cannot buildworld FreeBSD-9.0-BETA3 (RELENG_9) Responsible-Changed-From-To: freebsd-amd64->freebsd-bugs Responsible-Changed-By: linimon Responsible-Changed-When: Mon Oct 10 05:08:37 UTC 2011 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=161326 From owner-freebsd-amd64@FreeBSD.ORG Mon Oct 10 11:07:02 2011 Return-Path: Delivered-To: freebsd-amd64@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22216106567C for ; Mon, 10 Oct 2011 11:07:02 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 088308FC1F for ; Mon, 10 Oct 2011 11:07:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9AB71Xo032318 for ; Mon, 10 Oct 2011 11:07:01 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9AB71IO032316 for freebsd-amd64@FreeBSD.org; Mon, 10 Oct 2011 11:07:01 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 10 Oct 2011 11:07:01 GMT Message-Id: <201110101107.p9AB71IO032316@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-amd64@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-amd64@FreeBSD.org X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Oct 2011 11:07:02 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o amd64/161419 amd64 [loader] FreeBSD 9.0beta3 under Virtualbox: page fault o amd64/161418 amd64 [panic] FreeBSD 9.0beta3 under Virtualbox: lost device o amd64/161205 amd64 Bug report freebsd 9.0 beta 3 [nfs] [pfsync] [regressi o kern/160833 amd64 Keyboard USB doesn't work o amd64/160561 amd64 no C-states on atom D525 o amd64/160419 amd64 [acpi_thermal] acpi_thermal kernel thread high CPU usa o amd64/159809 amd64 RELENG_8_1 /UPDATING wrong re: COMPAT_IA32 alias for C o amd64/157386 amd64 [powerd] Enabling powerd(8) with default settings on I o amd64/156464 amd64 fpsetprec does not work o amd64/156106 amd64 [boot] boot0 fails to start o amd64/156074 amd64 [hang] Removing CD-Rom from Lenovo T61p hangs system o amd64/155249 amd64 [build] 8.1 buildworld failure o amd64/155135 amd64 [boot] Does Not Boot On a Very Standard Hardware o amd64/154957 amd64 [boot] Install boot CD won't boot up - keeps rebooting o amd64/154629 amd64 [panic] Fatal trap 9: general protection fault while i o amd64/153935 amd64 [hang] system hangs while trying to do 'shutdown -h no o amd64/153831 amd64 [boot] CD bootloader won't on Tyan s2912G2nr o amd64/153496 amd64 [hyper-v] [install] Install on Hyper-V leaves corrupt o amd64/153372 amd64 [panic] kernel panic o amd64/153175 amd64 [amd64] Kernel Panic on only FreeBSD 8 amd64 o amd64/152874 amd64 [install] 8.1 install fails where 7.3 works due to lac o amd64/152430 amd64 [boot] HP ProLiant Microserver n36l cannot boot into i o amd64/151385 amd64 [boot] Installation hangs on MacBook o amd64/150170 amd64 [patch] [amd64] [headers] SIG_ATOMIC_MIN/SIG_ATOMIC_MA o amd64/145991 amd64 [NOTES] [patch] Add a requires line to /sys/amd64/conf o amd64/144405 amd64 [build] [patch] include /usr/obj/lib32 in cleanworld t s amd64/143173 amd64 [ata] Promise FastTrack TX4 + SATA DVD, installer can' f amd64/141413 amd64 [hang] Tyan 2881 m3289 SMDC freeze f amd64/141060 amd64 [install] Can't install 8.0-RELEASE on the server wher o amd64/140715 amd64 [boot] Dell M600 Blade fails to boot 7.2+ 64 bit o amd64/139998 amd64 [panic][net] 7.2 amd64 panic in rtrequest1_fib o amd64/139924 amd64 [boot] cd or dvd not load o amd64/137942 amd64 [pci] 8.0-BETA2 having problems with Asus M2N-SLI-delu o amd64/135265 amd64 [mpt] Boot from install cd hangs on HP DL160 G5 with L o amd64/135040 amd64 [ata] FreeBSD/amd64 does not (always) detect disk on S o amd64/133977 amd64 [panic] [ffs] "panic: ffs_blkfree: freeing free block" o amd64/133701 amd64 Recompiling the kernel with k8temp or smbios break GEO o amd64/132574 amd64 [boot] [hang] Freeze on bootstrap loader (CD) using AT o amd64/131456 amd64 [acpi] [ata] ACPI & ATA problems s amd64/131209 amd64 [panic] [bce] 7.1-STABLE amd64 crash - m0 NULL o amd64/130368 amd64 [hang] Switching from xorg to console locks up compute o amd64/129889 amd64 [boot] [hang] The booting process stops at the line mo o amd64/129426 amd64 [panic] FreeBSD 7.0 crash after subdiskXX: detached o amd64/129315 amd64 [em] amd64 motherboard: Intel DG965WH motherboard comp f amd64/128765 amd64 [install] Install CD loads to Install choices but stop o amd64/127640 amd64 [amd64] gcc(1) will not build shared libraries with -f o amd64/125002 amd64 [install] amd64, SATA hard disks not detected o amd64/124432 amd64 [panic] 7.0-STABLE panic: invalbuf: dirty bufs o amd64/122549 amd64 7.0-RELEASE-amd64-bootonly.iso doesn't work w/ serial o amd64/120202 amd64 [amd64] [patch] [panic] kernel panic at start_all_aps, f amd64/117296 amd64 [ata] I don`t see second SATA IDE on VIA VT8237A f amd64/116620 amd64 [hang] ifconfig spins when creating carp(4) device on s amd64/115815 amd64 [ata] [request] Gigabyte GA-M61P-S3 Motherboard unsupp o amd64/115194 amd64 LCD screen remains blank after Dell XPS M1210 lid is c o amd64/91405 amd64 [asr] [panic] Kernel panic caused by asr on 6.0-amd64 55 problems total. From owner-freebsd-amd64@FreeBSD.ORG Sun Oct 9 11:00:03 2011 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E0211065673 for ; Sun, 9 Oct 2011 11:00:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 3ECF58FC14 for ; Sun, 9 Oct 2011 11:00:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p99B03GT094659 for ; Sun, 9 Oct 2011 11:00:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p99B03BG094638; Sun, 9 Oct 2011 11:00:03 GMT (envelope-from gnats) Resent-Date: Sun, 9 Oct 2011 11:00:03 GMT Resent-Message-Id: <201110091100.p99B03BG094638@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-amd64@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Floris Bos Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B15B106564A for ; Sun, 9 Oct 2011 10:55:10 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 62EB28FC0C for ; Sun, 9 Oct 2011 10:55:10 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p99At9Cl027732 for ; Sun, 9 Oct 2011 10:55:09 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id p99At9Ih027727; Sun, 9 Oct 2011 10:55:09 GMT (envelope-from nobody) Message-Id: <201110091055.p99At9Ih027727@red.freebsd.org> Date: Sun, 9 Oct 2011 10:55:09 GMT From: Floris Bos To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 X-Mailman-Approved-At: Mon, 10 Oct 2011 16:11:41 +0000 Cc: Subject: amd64/161418: [panic] FreeBSD 9.0beta3 under Virtualbox: lost device X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Oct 2011 11:00:03 -0000 >Number: 161418 >Category: amd64 >Synopsis: [panic] FreeBSD 9.0beta3 under Virtualbox: lost device >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-amd64 >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sun Oct 09 11:00:02 UTC 2011 >Closed-Date: >Last-Modified: >Originator: Floris Bos >Release: 9.0 beta 3 >Organization: >Environment: >Description: Running FreeBSD 9.0 beta 3 under Virtualbox 4.1.4 r74291 (KUbuntu 11.04 host system). - While extracting the installation files to a slow hard disk you often get a "lost device" kernel panic. This doesn't happen always (sometimes installation does complete successfully) but more often than not. Screenshots of 2 seperate occasions: http://postimage.org/image/w98sqs04/ http://postimage.org/image/zft33ick/ (dmesg on the host system does not show disk errors) >How-To-Repeat: >Fix: >Release-Note: >Audit-Trail: >Unformatted: From owner-freebsd-amd64@FreeBSD.ORG Tue Oct 11 15:10:08 2011 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 618A21065679 for ; Tue, 11 Oct 2011 15:10:08 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 3EAC08FC0A for ; Tue, 11 Oct 2011 15:10:08 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9BFA7Us025729 for ; Tue, 11 Oct 2011 15:10:07 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9BFA7k7025728; Tue, 11 Oct 2011 15:10:07 GMT (envelope-from gnats) Resent-Date: Tue, 11 Oct 2011 15:10:07 GMT Resent-Message-Id: <201110111510.p9BFA7k7025728@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-amd64@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, George Breahna Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 052DE106564A for ; Tue, 11 Oct 2011 15:07:14 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id DE6348FC12 for ; Tue, 11 Oct 2011 15:07:13 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p9BF7Dts036763 for ; Tue, 11 Oct 2011 15:07:13 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id p9BF7Dqw036762; Tue, 11 Oct 2011 15:07:13 GMT (envelope-from nobody) Message-Id: <201110111507.p9BF7Dqw036762@red.freebsd.org> Date: Tue, 11 Oct 2011 15:07:13 GMT From: George Breahna To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 X-Mailman-Approved-At: Tue, 11 Oct 2011 16:35:56 +0000 Cc: Subject: amd64/161493: NFS v3 directory structure update slow X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Oct 2011 15:10:08 -0000 >Number: 161493 >Category: amd64 >Synopsis: NFS v3 directory structure update slow >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-amd64 >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Oct 11 15:10:07 UTC 2011 >Closed-Date: >Last-Modified: >Originator: George Breahna >Release: 9.0 Beta 2 >Organization: >Environment: FreeBSD store2 9.0-BETA2 FreeBSD 9.0-BETA2 #0: Sun Sep 18 22:02:45 EDT 2011 pulsar@store2.emailarray.com:/usr/obj/usr/src/sys/PULSAR amd64 >Description: We used to run a NFS server on FreeBSD 6.2 but we built a new box recently and installed 9.0 Beta 2 on it. The data was moved over as it serves as the back-end for a mail system. It runs NFS v3 over TCP only and all the NFS-related processes (rpcbind, mountd, lockd, etc ) run with the -h switch and bind to the local IP address. The NFS server exports the data to 7 NFS clients ranging from FreeBSD 6.1 to 8.2, the majority being 8.2 The mount on the NFS clients is done simply with -o tcp,rsize=32768,wsize=32768 Usual file operations, such as accessing files, creating directories, removing files, chmod, chown, etc work perfectly but we noticed there were issues in removing directories that contained data. We had a strange error: rm -rf nick/ rm: fts_read: Input/output error Using 'truss' on rm revealed this: open("..",O_RDONLY,00) ERR#5 'Input/output error' After much testing and debugging we realized the problem is in the NFS protocol. ( either server or client but we assume server since this used to work very well with FreeBSD 6.2 ). The problem appears to be that NFS does not show the '..' after modifying a directory structure. Take the following example executed on a FreeBSD 8.2 client accessing the NFS share from the 9.0B2 server: imap5# mkdir test1 imap5# cd test1 imap5# touch file1 imap5# touch file2 imap5# ls -la ls: ..: Input/output error total 4 drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 Notice the '..' is missing from the display. If we now try and remove the directory 'test1' it will throw the "rm: fts_read: Input/output error" error. If we wait in between 1 minute and 5 minutes, '..' will eventually appear by itself. During this whole time, '..' effectively exists on the NFS server but it's not displayed by any of the NFS clients. I can force the NFS client to show it faster by doing an ls -la from the parent level. For example: imap5# mkdir test1 imap5# touch test1/file1 imap5# touch test1/file2 imap5# touch test1/file3 imap5# ls -la test1 total 8 drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 imap5# cd test1 imap5# ls -la total 8 drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 but if we wait 5 seconds after that display and try again: ls -la ls: ..: Input/output error total 4 drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 Again, if we wait longer ( 1-5 minutes ), the '..' will properly appear in there. There are no error messages on the console or other log files. This is reproducible 100% of the time with any FreeBSD client. Have tried unmounting/remounting several times without any effect. Also tried different rsize/wsize, no effect. I think there is some delay in updating the directory structure and it's causing this bug. Here's also some output from nfsstat on the server: Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 114731225 20496896 254966151 133 11697392 19963641 0 9228861 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 4313471 1157651 39 1955 16511932 15479669 0 116927742 Mknod Fsstat Fsinfo PathConf Commit 0 4748487 48 0 14921747 Server Ret-Failed 0 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 0 0 0 613368147 Server Write Gathering: WriteOps WriteRPC Opsaved 19963641 19963641 0 >How-To-Repeat: imap5# mkdir test1 imap5# cd test1 imap5# touch file1 imap5# touch file2 imap5# ls -la ls: ..: Input/output error total 4 drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 >Fix: >Release-Note: >Audit-Trail: >Unformatted: From owner-freebsd-amd64@FreeBSD.ORG Wed Oct 12 13:29:57 2011 Return-Path: Delivered-To: freebsd-amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB60B106566C; Wed, 12 Oct 2011 13:29:57 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 7076E8FC19; Wed, 12 Oct 2011 13:29:57 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id C712F46B32; Wed, 12 Oct 2011 09:29:56 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 4B3818A02E; Wed, 12 Oct 2011 09:29:56 -0400 (EDT) From: John Baldwin To: freebsd-amd64@freebsd.org Date: Wed, 12 Oct 2011 09:29:39 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110617; KDE/4.5.5; amd64; ; ) References: <201110111507.p9BF7Dqw036762@red.freebsd.org> In-Reply-To: <201110111507.p9BF7Dqw036762@red.freebsd.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201110120929.39901.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Wed, 12 Oct 2011 09:29:56 -0400 (EDT) Cc: Rick Macklem , freebsd-gnats-submit@freebsd.org, George Breahna Subject: Re: amd64/161493: NFS v3 directory structure update slow X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2011 13:29:57 -0000 On Tuesday, October 11, 2011 11:07:13 am George Breahna wrote: > > >Number: 161493 > >Category: amd64 > >Synopsis: NFS v3 directory structure update slow > >Confidential: no > >Severity: critical > >Priority: high > >Responsible: freebsd-amd64 > >State: open > >Quarter: > >Keywords: > >Date-Required: > >Class: sw-bug > >Submitter-Id: current-users > >Arrival-Date: Tue Oct 11 15:10:07 UTC 2011 > >Closed-Date: > >Last-Modified: > >Originator: George Breahna > >Release: 9.0 Beta 2 > >Organization: > >Environment: > FreeBSD store2 9.0-BETA2 FreeBSD 9.0-BETA2 #0: Sun Sep 18 22:02:45 EDT 2011 pulsar@store2.emailarray.com:/usr/obj/usr/src/sys/PULSAR amd64 > >Description: > We used to run a NFS server on FreeBSD 6.2 but we built a new box recently and installed 9.0 Beta 2 on it. The data was moved over as it serves as the back-end for a mail system. It runs NFS v3 over TCP only and all the NFS- related processes (rpcbind, mountd, lockd, etc ) run with the -h switch and bind to the local IP address. > > The NFS server exports the data to 7 NFS clients ranging from FreeBSD 6.1 to 8.2, the majority being 8.2 The mount on the NFS clients is done simply with - o tcp,rsize=32768,wsize=32768 > > Usual file operations, such as accessing files, creating directories, removing files, chmod, chown, etc work perfectly but we noticed there were issues in removing directories that contained data. We had a strange error: > > rm -rf nick/ > rm: fts_read: Input/output error > > Using 'truss' on rm revealed this: > > open("..",O_RDONLY,00) ERR#5 'Input/output error' > > After much testing and debugging we realized the problem is in the NFS protocol. ( either server or client but we assume server since this used to work very well with FreeBSD 6.2 ). The problem appears to be that NFS does not show the '..' after modifying a directory structure. Take the following example executed on a FreeBSD 8.2 client accessing the NFS share from the 9.0B2 server: > > imap5# mkdir test1 > imap5# cd test1 > imap5# touch file1 > imap5# touch file2 > imap5# ls -la > ls: ..: Input/output error > total 4 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > > Notice the '..' is missing from the display. If we now try and remove the directory 'test1' it will throw the "rm: fts_read: Input/output error" error. > > If we wait in between 1 minute and 5 minutes, '..' will eventually appear by itself. During this whole time, '..' effectively exists on the NFS server but it's not displayed by any of the NFS clients. > > I can force the NFS client to show it faster by doing an ls -la from the parent level. For example: > > imap5# mkdir test1 > imap5# touch test1/file1 > imap5# touch test1/file2 > imap5# touch test1/file3 > imap5# ls -la test1 > total 8 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > imap5# cd test1 > imap5# ls -la > total 8 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > but if we wait 5 seconds after that display and try again: > > ls -la > ls: ..: Input/output error > total 4 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > Again, if we wait longer ( 1-5 minutes ), the '..' will properly appear in there. > > There are no error messages on the console or other log files. This is reproducible 100% of the time with any FreeBSD client. Have tried unmounting/remounting several times without any effect. Also tried different rsize/wsize, no effect. I think there is some delay in updating the directory structure and it's causing this bug. > > Here's also some output from nfsstat on the server: > > > Server Info: > Getattr Setattr Lookup Readlink Read Write Create Remove > 114731225 20496896 254966151 133 11697392 19963641 0 9228861 > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access > 4313471 1157651 39 1955 16511932 15479669 0 116927742 > Mknod Fsstat Fsinfo PathConf Commit > 0 4748487 48 0 14921747 > Server Ret-Failed > 0 > Server Faults > 0 > Server Cache Stats: > Inprog Idem Non-idem Misses > 0 0 0 613368147 > Server Write Gathering: > WriteOps WriteRPC Opsaved > 19963641 19963641 0 > > >How-To-Repeat: > imap5# mkdir test1 > imap5# cd test1 > imap5# touch file1 > imap5# touch file2 > imap5# ls -la > ls: ..: Input/output error > total 4 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > >Fix: Can you try using the "old" NFS server as a test? -- John Baldwin From owner-freebsd-amd64@FreeBSD.ORG Wed Oct 12 13:30:15 2011 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 927FE106566B for ; Wed, 12 Oct 2011 13:30:15 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 81CA68FC1C for ; Wed, 12 Oct 2011 13:30:15 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9CDUFkr003977 for ; Wed, 12 Oct 2011 13:30:15 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9CDUFb9003972; Wed, 12 Oct 2011 13:30:15 GMT (envelope-from gnats) Date: Wed, 12 Oct 2011 13:30:15 GMT Message-Id: <201110121330.p9CDUFb9003972@freefall.freebsd.org> To: freebsd-amd64@FreeBSD.org From: John Baldwin Cc: Subject: Re: amd64/161493: NFS v3 directory structure update slow X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John Baldwin List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2011 13:30:15 -0000 The following reply was made to PR amd64/161493; it has been noted by GNATS. From: John Baldwin To: freebsd-amd64@freebsd.org Cc: George Breahna , freebsd-gnats-submit@freebsd.org, Rick Macklem Subject: Re: amd64/161493: NFS v3 directory structure update slow Date: Wed, 12 Oct 2011 09:29:39 -0400 On Tuesday, October 11, 2011 11:07:13 am George Breahna wrote: > > >Number: 161493 > >Category: amd64 > >Synopsis: NFS v3 directory structure update slow > >Confidential: no > >Severity: critical > >Priority: high > >Responsible: freebsd-amd64 > >State: open > >Quarter: > >Keywords: > >Date-Required: > >Class: sw-bug > >Submitter-Id: current-users > >Arrival-Date: Tue Oct 11 15:10:07 UTC 2011 > >Closed-Date: > >Last-Modified: > >Originator: George Breahna > >Release: 9.0 Beta 2 > >Organization: > >Environment: > FreeBSD store2 9.0-BETA2 FreeBSD 9.0-BETA2 #0: Sun Sep 18 22:02:45 EDT 2011 pulsar@store2.emailarray.com:/usr/obj/usr/src/sys/PULSAR amd64 > >Description: > We used to run a NFS server on FreeBSD 6.2 but we built a new box recently and installed 9.0 Beta 2 on it. The data was moved over as it serves as the back-end for a mail system. It runs NFS v3 over TCP only and all the NFS- related processes (rpcbind, mountd, lockd, etc ) run with the -h switch and bind to the local IP address. > > The NFS server exports the data to 7 NFS clients ranging from FreeBSD 6.1 to 8.2, the majority being 8.2 The mount on the NFS clients is done simply with - o tcp,rsize=32768,wsize=32768 > > Usual file operations, such as accessing files, creating directories, removing files, chmod, chown, etc work perfectly but we noticed there were issues in removing directories that contained data. We had a strange error: > > rm -rf nick/ > rm: fts_read: Input/output error > > Using 'truss' on rm revealed this: > > open("..",O_RDONLY,00) ERR#5 'Input/output error' > > After much testing and debugging we realized the problem is in the NFS protocol. ( either server or client but we assume server since this used to work very well with FreeBSD 6.2 ). The problem appears to be that NFS does not show the '..' after modifying a directory structure. Take the following example executed on a FreeBSD 8.2 client accessing the NFS share from the 9.0B2 server: > > imap5# mkdir test1 > imap5# cd test1 > imap5# touch file1 > imap5# touch file2 > imap5# ls -la > ls: ..: Input/output error > total 4 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > > Notice the '..' is missing from the display. If we now try and remove the directory 'test1' it will throw the "rm: fts_read: Input/output error" error. > > If we wait in between 1 minute and 5 minutes, '..' will eventually appear by itself. During this whole time, '..' effectively exists on the NFS server but it's not displayed by any of the NFS clients. > > I can force the NFS client to show it faster by doing an ls -la from the parent level. For example: > > imap5# mkdir test1 > imap5# touch test1/file1 > imap5# touch test1/file2 > imap5# touch test1/file3 > imap5# ls -la test1 > total 8 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > imap5# cd test1 > imap5# ls -la > total 8 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > but if we wait 5 seconds after that display and try again: > > ls -la > ls: ..: Input/output error > total 4 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > Again, if we wait longer ( 1-5 minutes ), the '..' will properly appear in there. > > There are no error messages on the console or other log files. This is reproducible 100% of the time with any FreeBSD client. Have tried unmounting/remounting several times without any effect. Also tried different rsize/wsize, no effect. I think there is some delay in updating the directory structure and it's causing this bug. > > Here's also some output from nfsstat on the server: > > > Server Info: > Getattr Setattr Lookup Readlink Read Write Create Remove > 114731225 20496896 254966151 133 11697392 19963641 0 9228861 > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access > 4313471 1157651 39 1955 16511932 15479669 0 116927742 > Mknod Fsstat Fsinfo PathConf Commit > 0 4748487 48 0 14921747 > Server Ret-Failed > 0 > Server Faults > 0 > Server Cache Stats: > Inprog Idem Non-idem Misses > 0 0 0 613368147 > Server Write Gathering: > WriteOps WriteRPC Opsaved > 19963641 19963641 0 > > >How-To-Repeat: > imap5# mkdir test1 > imap5# cd test1 > imap5# touch file1 > imap5# touch file2 > imap5# ls -la > ls: ..: Input/output error > total 4 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > >Fix: Can you try using the "old" NFS server as a test? -- John Baldwin From owner-freebsd-amd64@FreeBSD.ORG Wed Oct 12 13:30:36 2011 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 48A91106564A; Wed, 12 Oct 2011 13:30:36 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 20FAE8FC14; Wed, 12 Oct 2011 13:30:36 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9CDUZMS006004; Wed, 12 Oct 2011 13:30:36 GMT (envelope-from jhb@freefall.freebsd.org) Received: (from jhb@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9CDUZUd005991; Wed, 12 Oct 2011 13:30:35 GMT (envelope-from jhb) Date: Wed, 12 Oct 2011 13:30:35 GMT Message-Id: <201110121330.p9CDUZUd005991@freefall.freebsd.org> To: jhb@FreeBSD.org, freebsd-amd64@FreeBSD.org, freebsd-fs@FreeBSD.org From: jhb@FreeBSD.org Cc: Subject: Re: kern/161493: NFS v3 directory structure update slow X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2011 13:30:36 -0000 Synopsis: NFS v3 directory structure update slow Responsible-Changed-From-To: freebsd-amd64->freebsd-fs Responsible-Changed-By: jhb Responsible-Changed-When: Wed Oct 12 13:30:14 UTC 2011 Responsible-Changed-Why: Move this over to fs@. http://www.freebsd.org/cgi/query-pr.cgi?pr=161493 From owner-freebsd-amd64@FreeBSD.ORG Thu Oct 13 00:54:13 2011 Return-Path: Delivered-To: freebsd-amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B1B8C1065679; Thu, 13 Oct 2011 00:54:13 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 2ADEC8FC0C; Thu, 13 Oct 2011 00:54:12 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AmkAAJAvlk6DaFvO/2dsb2JhbABDhHWULJAmgVMBAQUjVhsOBgQCAg0ZAlkGE4gGpl2Rf4EsgXSDJIEUBJN1kXA X-IronPort-AV: E=Sophos;i="4.69,337,1315195200"; d="scan'208";a="139627737" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 12 Oct 2011 20:25:11 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 1D1A4B3F24; Wed, 12 Oct 2011 20:25:11 -0400 (EDT) Date: Wed, 12 Oct 2011 20:25:11 -0400 (EDT) From: Rick Macklem To: John Baldwin Message-ID: <1214713861.3007306.1318465511106.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <201110120929.39901.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) X-Mailman-Approved-At: Thu, 13 Oct 2011 15:29:06 +0000 Cc: Rick Macklem , freebsd-gnats-submit@freebsd.org, George Breahna , freebsd-amd64@freebsd.org Subject: Re: amd64/161493: NFS v3 directory structure update slow X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Oct 2011 00:54:13 -0000 John Baldwin wrote: > On Tuesday, October 11, 2011 11:07:13 am George Breahna wrote: > > > > >Number: 161493 > > >Category: amd64 > > >Synopsis: NFS v3 directory structure update slow > > >Confidential: no > > >Severity: critical > > >Priority: high > > >Responsible: freebsd-amd64 > > >State: open > > >Quarter: > > >Keywords: > > >Date-Required: > > >Class: sw-bug > > >Submitter-Id: current-users > > >Arrival-Date: Tue Oct 11 15:10:07 UTC 2011 > > >Closed-Date: > > >Last-Modified: > > >Originator: George Breahna > > >Release: 9.0 Beta 2 > > >Organization: > > >Environment: > > FreeBSD store2 9.0-BETA2 FreeBSD 9.0-BETA2 #0: Sun Sep 18 22:02:45 > > EDT 2011 > pulsar@store2.emailarray.com:/usr/obj/usr/src/sys/PULSAR amd64 > > >Description: > > We used to run a NFS server on FreeBSD 6.2 but we built a new box > > recently > and installed 9.0 Beta 2 on it. The data was moved over as it serves > as the > back-end for a mail system. It runs NFS v3 over TCP only and all the > NFS- > related processes (rpcbind, mountd, lockd, etc ) run with the -h > switch and > bind to the local IP address. > > > > The NFS server exports the data to 7 NFS clients ranging from > > FreeBSD 6.1 to > 8.2, the majority being 8.2 The mount on the NFS clients is done > simply with - > o tcp,rsize=32768,wsize=32768 > > > > Usual file operations, such as accessing files, creating > > directories, > removing files, chmod, chown, etc work perfectly but we noticed there > were > issues in removing directories that contained data. We had a strange > error: > > > > rm -rf nick/ > > rm: fts_read: Input/output error > > > > Using 'truss' on rm revealed this: > > > > open("..",O_RDONLY,00) ERR#5 'Input/output error' > > > > After much testing and debugging we realized the problem is in the > > NFS > protocol. ( either server or client but we assume server since this > used to > work very well with FreeBSD 6.2 ). The problem appears to be that NFS > does not > show the '..' after modifying a directory structure. Take the > following > example executed on a FreeBSD 8.2 client accessing the NFS share from > the > 9.0B2 server: > > > > imap5# mkdir test1 > > imap5# cd test1 > > imap5# touch file1 > > imap5# touch file2 > > imap5# ls -la > > ls: ..: Input/output error > > total 4 > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > > > > Notice the '..' is missing from the display. If we now try and > > remove the > directory 'test1' it will throw the "rm: fts_read: Input/output error" > error. > > > > If we wait in between 1 minute and 5 minutes, '..' will eventually > > appear by > itself. During this whole time, '..' effectively exists on the NFS > server but > it's not displayed by any of the NFS clients. > > > > I can force the NFS client to show it faster by doing an ls -la from > > the > parent level. For example: > > > > imap5# mkdir test1 > > imap5# touch test1/file1 > > imap5# touch test1/file2 > > imap5# touch test1/file3 > > imap5# ls -la test1 > > total 8 > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > imap5# cd test1 > > imap5# ls -la > > total 8 > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > > > but if we wait 5 seconds after that display and try again: > > > > ls -la > > ls: ..: Input/output error > > total 4 > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > > > Again, if we wait longer ( 1-5 minutes ), the '..' will properly > > appear in > there. > > > > There are no error messages on the console or other log files. This > > is > reproducible 100% of the time with any FreeBSD client. Have tried > unmounting/remounting several times without any effect. Also tried > different > rsize/wsize, no effect. I think there is some delay in updating the > directory > structure and it's causing this bug. > > > > Here's also some output from nfsstat on the server: > > > > > > Server Info: > > Getattr Setattr Lookup Readlink Read Write Create > Remove > > 114731225 20496896 254966151 133 11697392 19963641 0 > 9228861 > > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus > Access > > 4313471 1157651 39 1955 16511932 15479669 0 > 116927742 > > Mknod Fsstat Fsinfo PathConf Commit > > 0 4748487 48 0 14921747 > > Server Ret-Failed > > 0 > > Server Faults > > 0 > > Server Cache Stats: > > Inprog Idem Non-idem Misses > > 0 0 0 613368147 > > Server Write Gathering: > > WriteOps WriteRPC Opsaved > > 19963641 19963641 0 > > > > >How-To-Repeat: > > imap5# mkdir test1 > > imap5# cd test1 > > imap5# touch file1 > > imap5# touch file2 > > imap5# ls -la > > ls: ..: Input/output error > > total 4 > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > > >Fix: > > Can you try using the "old" NFS server as a test? > Please make sure you have the patch in r225356 in your server's kernel sources (it went into head on Sep. 3, but I don't know if your Sep. 11 build would have it?). It fixed a problem that would cause lookup of ".." to fail intermittently, because a field in struct nameidata added on Aug. 13 wasn't initialized. You can find the one line patch here: http://svnweb.freebsd.org/base/head/sys/fs/nfsserver/nfs_nfsdport.c?r1=224911&r2=225356 Please let us know if you have this patch and, if not, apply it and see if the problem goes away. Thanks, rick From owner-freebsd-amd64@FreeBSD.ORG Thu Oct 13 01:41:41 2011 Return-Path: Delivered-To: freebsd-amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26D431065670 for ; Thu, 13 Oct 2011 01:41:41 +0000 (UTC) (envelope-from george@polarismail.com) Received: from smtp3.emailarray.com (smtp3.emailarray.com [65.39.216.17]) by mx1.freebsd.org (Postfix) with ESMTP id B673F8FC15 for ; Thu, 13 Oct 2011 01:41:40 +0000 (UTC) Received: (qmail 98010 invoked by uid 89); 13 Oct 2011 01:41:38 -0000 Received: from unknown (HELO GeorgePC) (sheken@top-consulting.net@50.100.137.136) (POLARISLOCAL) by smtp3.emailarray.com with SMTP; 13 Oct 2011 01:41:37 -0000 From: "George Breahna" To: "'John Baldwin'" , References: <201110111507.p9BF7Dqw036762@red.freebsd.org> <201110120929.39901.jhb@freebsd.org> In-Reply-To: <201110120929.39901.jhb@freebsd.org> Date: Wed, 12 Oct 2011 21:41:34 -0400 Message-ID: <015201cc8949$3e00f5d0$ba02e170$@com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: AcyI4wphQysqP/ehR3q9dxKM+wzXjgAZgksQ Content-Language: en-us X-DSPAM-Result: Innocent X-DSPAM-Processed: Wed Oct 12 21:41:38 2011 X-DSPAM-Confidence: 0.9938 X-DSPAM-Improbability: 1 in 16110 chance of being spam X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 1,4e9641d211842034310950 X-DSPAM-Factors: 27, freebsd+org, 0.00170, freebsd+org, 0.00170, Breahna, 0.00242, Breahna, 0.00242, 2011+9, 0.00283, freebsd, 0.00340, freebsd, 0.00340, UTC+2011, 0.00356, FreeBSD, 0.00359, FreeBSD, 0.00359, no+>, 0.00382, George+Breahna, 0.00678, George+Breahna, 0.00678, there+>, 0.00785, error+>, 0.00796, error+>, 0.00796, la+>, 0.00813, la+>, 0.00813, org+Cc, 0.00813, Tue+Oct, 0.00863, To+freebsd, 0.00867, >Description, 0.00887, wrote+>, 0.00982, idem, 0.01000, >Date+Required, 0.01000, Breahna+>, 0.01000, >State, 0.01000 X-PolarisMail-Flags: x X-Mailman-Approved-At: Thu, 13 Oct 2011 15:52:40 +0000 Cc: 'Rick Macklem' , freebsd-gnats-submit@freebsd.org Subject: RE: amd64/161493: NFS v3 directory structure update slow X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Oct 2011 01:41:41 -0000 I can also confirm that using -o option in nfsd makes the problem go away. George -----Original Message----- From: John Baldwin [mailto:jhb@freebsd.org] Sent: Wednesday, October 12, 2011 9:30 AM To: freebsd-amd64@freebsd.org Cc: George Breahna; freebsd-gnats-submit@freebsd.org; Rick Macklem Subject: Re: amd64/161493: NFS v3 directory structure update slow On Tuesday, October 11, 2011 11:07:13 am George Breahna wrote: > > >Number: 161493 > >Category: amd64 > >Synopsis: NFS v3 directory structure update slow > >Confidential: no > >Severity: critical > >Priority: high > >Responsible: freebsd-amd64 > >State: open > >Quarter: > >Keywords: > >Date-Required: > >Class: sw-bug > >Submitter-Id: current-users > >Arrival-Date: Tue Oct 11 15:10:07 UTC 2011 > >Closed-Date: > >Last-Modified: > >Originator: George Breahna > >Release: 9.0 Beta 2 > >Organization: > >Environment: > FreeBSD store2 9.0-BETA2 FreeBSD 9.0-BETA2 #0: Sun Sep 18 22:02:45 EDT 2011 pulsar@store2.emailarray.com:/usr/obj/usr/src/sys/PULSAR amd64 > >Description: > We used to run a NFS server on FreeBSD 6.2 but we built a new box recently and installed 9.0 Beta 2 on it. The data was moved over as it serves as the back-end for a mail system. It runs NFS v3 over TCP only and all the NFS- related processes (rpcbind, mountd, lockd, etc ) run with the -h switch and bind to the local IP address. > > The NFS server exports the data to 7 NFS clients ranging from FreeBSD 6.1 to 8.2, the majority being 8.2 The mount on the NFS clients is done simply with - o tcp,rsize=32768,wsize=32768 > > Usual file operations, such as accessing files, creating directories, removing files, chmod, chown, etc work perfectly but we noticed there were issues in removing directories that contained data. We had a strange error: > > rm -rf nick/ > rm: fts_read: Input/output error > > Using 'truss' on rm revealed this: > > open("..",O_RDONLY,00) ERR#5 'Input/output error' > > After much testing and debugging we realized the problem is in the NFS protocol. ( either server or client but we assume server since this used to work very well with FreeBSD 6.2 ). The problem appears to be that NFS does not show the '..' after modifying a directory structure. Take the following example executed on a FreeBSD 8.2 client accessing the NFS share from the 9.0B2 server: > > imap5# mkdir test1 > imap5# cd test1 > imap5# touch file1 > imap5# touch file2 > imap5# ls -la > ls: ..: Input/output error > total 4 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > > Notice the '..' is missing from the display. If we now try and remove the directory 'test1' it will throw the "rm: fts_read: Input/output error" error. > > If we wait in between 1 minute and 5 minutes, '..' will eventually appear by itself. During this whole time, '..' effectively exists on the NFS server but it's not displayed by any of the NFS clients. > > I can force the NFS client to show it faster by doing an ls -la from the parent level. For example: > > imap5# mkdir test1 > imap5# touch test1/file1 > imap5# touch test1/file2 > imap5# touch test1/file3 > imap5# ls -la test1 > total 8 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > imap5# cd test1 > imap5# ls -la > total 8 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > but if we wait 5 seconds after that display and try again: > > ls -la > ls: ..: Input/output error > total 4 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > Again, if we wait longer ( 1-5 minutes ), the '..' will properly appear in there. > > There are no error messages on the console or other log files. This is reproducible 100% of the time with any FreeBSD client. Have tried unmounting/remounting several times without any effect. Also tried different rsize/wsize, no effect. I think there is some delay in updating the directory structure and it's causing this bug. > > Here's also some output from nfsstat on the server: > > > Server Info: > Getattr Setattr Lookup Readlink Read Write Create Remove > 114731225 20496896 254966151 133 11697392 19963641 0 9228861 > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access > 4313471 1157651 39 1955 16511932 15479669 0 116927742 > Mknod Fsstat Fsinfo PathConf Commit > 0 4748487 48 0 14921747 > Server Ret-Failed > 0 > Server Faults > 0 > Server Cache Stats: > Inprog Idem Non-idem Misses > 0 0 0 613368147 > Server Write Gathering: > WriteOps WriteRPC Opsaved > 19963641 19963641 0 > > >How-To-Repeat: > imap5# mkdir test1 > imap5# cd test1 > imap5# touch file1 > imap5# touch file2 > imap5# ls -la > ls: ..: Input/output error > total 4 > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > >Fix: Can you try using the "old" NFS server as a test? -- John Baldwin From owner-freebsd-amd64@FreeBSD.ORG Thu Oct 13 01:42:31 2011 Return-Path: Delivered-To: freebsd-amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 43DE51065674 for ; Thu, 13 Oct 2011 01:42:31 +0000 (UTC) (envelope-from george@polarismail.com) Received: from smtp3.emailarray.com (smtp3.emailarray.com [65.39.216.17]) by mx1.freebsd.org (Postfix) with ESMTP id DC0BA8FC0C for ; Thu, 13 Oct 2011 01:42:30 +0000 (UTC) Received: (qmail 93093 invoked by uid 89); 13 Oct 2011 01:15:50 -0000 Received: from unknown (HELO GeorgePC) (sheken@top-consulting.net@50.100.137.136) (POLARISLOCAL) by smtp3.emailarray.com with SMTP; 13 Oct 2011 01:15:48 -0000 From: "George Breahna" To: "'Rick Macklem'" , "'John Baldwin'" References: <201110120929.39901.jhb@freebsd.org> <1214713861.3007306.1318465511106.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <1214713861.3007306.1318465511106.JavaMail.root@erie.cs.uoguelph.ca> Date: Wed, 12 Oct 2011 21:15:45 -0400 Message-ID: <014f01cc8945$a2f648e0$e8e2daa0$@com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: AcyJPpNZg2o+f8zUTIGdaGTjmdep/wABtocQ Content-Language: en-us X-DSPAM-Result: Innocent X-DSPAM-Processed: Wed Oct 12 21:15:50 2011 X-DSPAM-Confidence: 0.9952 X-DSPAM-Improbability: 1 in 20716 chance of being spam X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 1,4e963bc611849563212579 X-DSPAM-Factors: 27, Url*freebsd, 0.00170, Url*freebsd, 0.00170, freebsd+org, 0.00170, freebsd+org, 0.00170, Breahna, 0.00242, Breahna, 0.00242, freebsd, 0.00340, freebsd, 0.00340, To+John, 0.00351, UTC+2011, 0.00356, FreeBSD, 0.00359, FreeBSD, 0.00359, no+>, 0.00382, Ok+I, 0.00406, but+>, 0.00490, you+wrote, 0.00577, From+Rick, 0.00629, 2011+8, 0.00645, >+Can, 0.00650, George+Breahna, 0.00678, George+Breahna, 0.00678, there+>, 0.00785, there+>, 0.00785, error+>, 0.00796, error+>, 0.00796, la+>, 0.00813, la+>, 0.00813 X-PolarisMail-Flags: x X-Mailman-Approved-At: Thu, 13 Oct 2011 15:53:02 +0000 Cc: freebsd-amd64@freebsd.org Subject: RE: amd64/161493: NFS v3 directory structure update slow X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Oct 2011 01:42:31 -0000 Ok, I will try this. I noticed you wrote another patch, available here, called the dotdot = patch. It modifies another file on top of the one mentioned in the link = you gave me. Is that unnecessary now ? http://people.freebsd.org/~rmacklem/dotdot.patch George -----Original Message----- From: Rick Macklem [mailto:rmacklem@uoguelph.ca]=20 Sent: Wednesday, October 12, 2011 8:25 PM To: John Baldwin Cc: George Breahna; freebsd-gnats-submit@freebsd.org; Rick Macklem; = freebsd-amd64@freebsd.org Subject: Re: amd64/161493: NFS v3 directory structure update slow John Baldwin wrote: > On Tuesday, October 11, 2011 11:07:13 am George Breahna wrote: > > > > >Number: 161493 > > >Category: amd64 > > >Synopsis: NFS v3 directory structure update slow > > >Confidential: no > > >Severity: critical > > >Priority: high > > >Responsible: freebsd-amd64 > > >State: open > > >Quarter: > > >Keywords: > > >Date-Required: > > >Class: sw-bug > > >Submitter-Id: current-users > > >Arrival-Date: Tue Oct 11 15:10:07 UTC 2011 > > >Closed-Date: > > >Last-Modified: > > >Originator: George Breahna > > >Release: 9.0 Beta 2 > > >Organization: > > >Environment: > > FreeBSD store2 9.0-BETA2 FreeBSD 9.0-BETA2 #0: Sun Sep 18 22:02:45 > > EDT 2011 > pulsar@store2.emailarray.com:/usr/obj/usr/src/sys/PULSAR amd64 > > >Description: > > We used to run a NFS server on FreeBSD 6.2 but we built a new box > > recently > and installed 9.0 Beta 2 on it. The data was moved over as it serves > as the > back-end for a mail system. It runs NFS v3 over TCP only and all the > NFS- > related processes (rpcbind, mountd, lockd, etc ) run with the -h > switch and > bind to the local IP address. > > > > The NFS server exports the data to 7 NFS clients ranging from > > FreeBSD 6.1 to > 8.2, the majority being 8.2 The mount on the NFS clients is done > simply with - > o tcp,rsize=3D32768,wsize=3D32768 > > > > Usual file operations, such as accessing files, creating > > directories, > removing files, chmod, chown, etc work perfectly but we noticed there > were > issues in removing directories that contained data. We had a strange > error: > > > > rm -rf nick/ > > rm: fts_read: Input/output error > > > > Using 'truss' on rm revealed this: > > > > open("..",O_RDONLY,00) ERR#5 'Input/output error' > > > > After much testing and debugging we realized the problem is in the > > NFS > protocol. ( either server or client but we assume server since this > used to > work very well with FreeBSD 6.2 ). The problem appears to be that NFS > does not > show the '..' after modifying a directory structure. Take the > following > example executed on a FreeBSD 8.2 client accessing the NFS share from > the > 9.0B2 server: > > > > imap5# mkdir test1 > > imap5# cd test1 > > imap5# touch file1 > > imap5# touch file2 > > imap5# ls -la > > ls: ..: Input/output error > > total 4 > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > > > > Notice the '..' is missing from the display. If we now try and > > remove the > directory 'test1' it will throw the "rm: fts_read: Input/output error" > error. > > > > If we wait in between 1 minute and 5 minutes, '..' will eventually > > appear by > itself. During this whole time, '..' effectively exists on the NFS > server but > it's not displayed by any of the NFS clients. > > > > I can force the NFS client to show it faster by doing an ls -la from > > the > parent level. For example: > > > > imap5# mkdir test1 > > imap5# touch test1/file1 > > imap5# touch test1/file2 > > imap5# touch test1/file3 > > imap5# ls -la test1 > > total 8 > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > imap5# cd test1 > > imap5# ls -la > > total 8 > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > > > but if we wait 5 seconds after that display and try again: > > > > ls -la > > ls: ..: Input/output error > > total 4 > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > > > Again, if we wait longer ( 1-5 minutes ), the '..' will properly > > appear in > there. > > > > There are no error messages on the console or other log files. This > > is > reproducible 100% of the time with any FreeBSD client. Have tried > unmounting/remounting several times without any effect. Also tried > different > rsize/wsize, no effect. I think there is some delay in updating the > directory > structure and it's causing this bug. > > > > Here's also some output from nfsstat on the server: > > > > > > Server Info: > > Getattr Setattr Lookup Readlink Read Write Create > Remove > > 114731225 20496896 254966151 133 11697392 19963641 0 > 9228861 > > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus > Access > > 4313471 1157651 39 1955 16511932 15479669 0 > 116927742 > > Mknod Fsstat Fsinfo PathConf Commit > > 0 4748487 48 0 14921747 > > Server Ret-Failed > > 0 > > Server Faults > > 0 > > Server Cache Stats: > > Inprog Idem Non-idem Misses > > 0 0 0 613368147 > > Server Write Gathering: > > WriteOps WriteRPC Opsaved > > 19963641 19963641 0 > > > > >How-To-Repeat: > > imap5# mkdir test1 > > imap5# cd test1 > > imap5# touch file1 > > imap5# touch file2 > > imap5# ls -la > > ls: ..: Input/output error > > total 4 > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > > >Fix: >=20 > Can you try using the "old" NFS server as a test? >=20 Please make sure you have the patch in r225356 in your server's kernel sources (it went into head on Sep. 3, but I don't know if your Sep. 11 build would have it?). It fixed a problem that would cause lookup of ".." to fail intermittently, because a field in struct nameidata added on Aug. 13 wasn't initialized. You can find the one line patch here: = http://svnweb.freebsd.org/base/head/sys/fs/nfsserver/nfs_nfsdport.c?r1=3D= 224911&r2=3D225356 Please let us know if you have this patch and, if not, apply it and see if the problem goes away. Thanks, rick From owner-freebsd-amd64@FreeBSD.ORG Thu Oct 13 02:16:51 2011 Return-Path: Delivered-To: freebsd-amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D96D106568D; Thu, 13 Oct 2011 02:16:51 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 180EA8FC08; Thu, 13 Oct 2011 02:16:50 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AmkAAEdJlk6DaFvO/2dsb2JhbABDFoRflCyQJoFTAQEEASNWBQcPEQMBAQEBAgINFgMCSAkIBhOHfgimSZFzgSyBdIMkgRQEk3WRcA X-IronPort-AV: E=Sophos;i="4.69,337,1315195200"; d="scan'208";a="139636659" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 12 Oct 2011 22:16:50 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 091CBB3F6D; Wed, 12 Oct 2011 22:16:50 -0400 (EDT) Date: Wed, 12 Oct 2011 22:16:50 -0400 (EDT) From: Rick Macklem To: George Breahna Message-ID: <287177506.3014184.1318472210023.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <014f01cc8945$a2f648e0$e8e2daa0$@com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) X-Mailman-Approved-At: Thu, 13 Oct 2011 15:53:41 +0000 Cc: freebsd-amd64@freebsd.org Subject: Re: amd64/161493: NFS v3 directory structure update slow X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Oct 2011 02:16:51 -0000 George Breahna wrote: > Ok, I will try this. > > I noticed you wrote another patch, available here, called the dotdot > patch. It modifies another file on top of the one mentioned in the > link you gave me. Is that unnecessary now ? > > http://people.freebsd.org/~rmacklem/dotdot.patch > The above patch also fixed the old server. See below for the patch for the old server. (You will be running the old server if you start both mountd and nfsd with a "-o" option. This happens if you have oldnfs_server_enable="YES" in your /etc/rc.conf.) Since "nfsstat -s" shows non-zero counts, you are running the new/default server. ("nfsstat -o -s" reports stats for the old server, which should be all zeros if you are running the new/default one.) In summary, I don't think you are running the old server and only need to patch the old server if you choose to run it, as jhb@ suggested to help with isolating the problem. (I would suggest you do that, if the patch for the new/regular server doesn't fix the problem.) > George > > -----Original Message----- > From: Rick Macklem [mailto:rmacklem@uoguelph.ca] > Sent: Wednesday, October 12, 2011 8:25 PM > To: John Baldwin > Cc: George Breahna; freebsd-gnats-submit@freebsd.org; Rick Macklem; > freebsd-amd64@freebsd.org > Subject: Re: amd64/161493: NFS v3 directory structure update slow > > John Baldwin wrote: > > On Tuesday, October 11, 2011 11:07:13 am George Breahna wrote: > > > > > > >Number: 161493 > > > >Category: amd64 > > > >Synopsis: NFS v3 directory structure update slow > > > >Confidential: no > > > >Severity: critical > > > >Priority: high > > > >Responsible: freebsd-amd64 > > > >State: open > > > >Quarter: > > > >Keywords: > > > >Date-Required: > > > >Class: sw-bug > > > >Submitter-Id: current-users > > > >Arrival-Date: Tue Oct 11 15:10:07 UTC 2011 > > > >Closed-Date: > > > >Last-Modified: > > > >Originator: George Breahna > > > >Release: 9.0 Beta 2 > > > >Organization: > > > >Environment: > > > FreeBSD store2 9.0-BETA2 FreeBSD 9.0-BETA2 #0: Sun Sep 18 22:02:45 > > > EDT 2011 > > pulsar@store2.emailarray.com:/usr/obj/usr/src/sys/PULSAR amd64 > > > >Description: > > > We used to run a NFS server on FreeBSD 6.2 but we built a new box > > > recently > > and installed 9.0 Beta 2 on it. The data was moved over as it serves > > as the > > back-end for a mail system. It runs NFS v3 over TCP only and all the > > NFS- > > related processes (rpcbind, mountd, lockd, etc ) run with the -h > > switch and > > bind to the local IP address. > > > > > > The NFS server exports the data to 7 NFS clients ranging from > > > FreeBSD 6.1 to > > 8.2, the majority being 8.2 The mount on the NFS clients is done > > simply with - > > o tcp,rsize=32768,wsize=32768 > > > > > > Usual file operations, such as accessing files, creating > > > directories, > > removing files, chmod, chown, etc work perfectly but we noticed > > there > > were > > issues in removing directories that contained data. We had a strange > > error: > > > > > > rm -rf nick/ > > > rm: fts_read: Input/output error > > > > > > Using 'truss' on rm revealed this: > > > > > > open("..",O_RDONLY,00) ERR#5 'Input/output error' > > > > > > After much testing and debugging we realized the problem is in the > > > NFS > > protocol. ( either server or client but we assume server since this > > used to > > work very well with FreeBSD 6.2 ). The problem appears to be that > > NFS > > does not > > show the '..' after modifying a directory structure. Take the > > following > > example executed on a FreeBSD 8.2 client accessing the NFS share > > from > > the > > 9.0B2 server: > > > > > > imap5# mkdir test1 > > > imap5# cd test1 > > > imap5# touch file1 > > > imap5# touch file2 > > > imap5# ls -la > > > ls: ..: Input/output error > > > total 4 > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > > > > > > Notice the '..' is missing from the display. If we now try and > > > remove the > > directory 'test1' it will throw the "rm: fts_read: Input/output > > error" > > error. > > > > > > If we wait in between 1 minute and 5 minutes, '..' will eventually > > > appear by > > itself. During this whole time, '..' effectively exists on the NFS > > server but > > it's not displayed by any of the NFS clients. > > > > > > I can force the NFS client to show it faster by doing an ls -la > > > from > > > the > > parent level. For example: > > > > > > imap5# mkdir test1 > > > imap5# touch test1/file1 > > > imap5# touch test1/file2 > > > imap5# touch test1/file3 > > > imap5# ls -la test1 > > > total 8 > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > > imap5# cd test1 > > > imap5# ls -la > > > total 8 > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > > > > > but if we wait 5 seconds after that display and try again: > > > > > > ls -la > > > ls: ..: Input/output error > > > total 4 > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > > > > > Again, if we wait longer ( 1-5 minutes ), the '..' will properly > > > appear in > > there. > > > > > > There are no error messages on the console or other log files. > > > This > > > is > > reproducible 100% of the time with any FreeBSD client. Have tried > > unmounting/remounting several times without any effect. Also tried > > different > > rsize/wsize, no effect. I think there is some delay in updating the > > directory > > structure and it's causing this bug. > > > > > > Here's also some output from nfsstat on the server: > > > > > > > > > Server Info: > > > Getattr Setattr Lookup Readlink Read Write Create > > Remove > > > 114731225 20496896 254966151 133 11697392 19963641 0 > > 9228861 > > > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus > > Access > > > 4313471 1157651 39 1955 16511932 15479669 0 > > 116927742 > > > Mknod Fsstat Fsinfo PathConf Commit > > > 0 4748487 48 0 14921747 > > > Server Ret-Failed > > > 0 > > > Server Faults > > > 0 > > > Server Cache Stats: > > > Inprog Idem Non-idem Misses > > > 0 0 0 613368147 > > > Server Write Gathering: > > > WriteOps WriteRPC Opsaved > > > 19963641 19963641 0 > > > > > > >How-To-Repeat: > > > imap5# mkdir test1 > > > imap5# cd test1 > > > imap5# touch file1 > > > imap5# touch file2 > > > imap5# ls -la > > > ls: ..: Input/output error > > > total 4 > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > > > >Fix: > > > > Can you try using the "old" NFS server as a test? > > > Please make sure you have the patch in r225356 in your server's > kernel sources (it went into head on Sep. 3, but I don't know if > your Sep. 11 build would have it?). It fixed a problem that would > cause lookup of ".." to fail intermittently, because a field in > struct nameidata added on Aug. 13 wasn't initialized. > > You can find the one line patch here: > http://svnweb.freebsd.org/base/head/sys/fs/nfsserver/nfs_nfsdport.c?r1=224911&r2=225356 Clarification, this is the patch for the new/default server. There is a similar patch for the old server. For the old server, the patch is: http://svnweb.freebsd.org/base/head/sys/nfsserver/nfs_serv.c?r1=219028&r2=225356 rick > > Please let us know if you have this patch and, if not, apply it > and see if the problem goes away. > > Thanks, rick From owner-freebsd-amd64@FreeBSD.ORG Thu Oct 13 02:19:25 2011 Return-Path: Delivered-To: freebsd-amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 64A0A106568A for ; Thu, 13 Oct 2011 02:19:25 +0000 (UTC) (envelope-from george@polarismail.com) Received: from smtp3.emailarray.com (smtp3.emailarray.com [65.39.216.17]) by mx1.freebsd.org (Postfix) with ESMTP id EF8088FC12 for ; Thu, 13 Oct 2011 02:19:24 +0000 (UTC) Received: (qmail 4041 invoked by uid 89); 13 Oct 2011 02:19:23 -0000 Received: from unknown (HELO GeorgePC) (sheken@top-consulting.net@50.100.137.136) (POLARISLOCAL) by smtp3.emailarray.com with SMTP; 13 Oct 2011 02:19:22 -0000 From: "George Breahna" To: "'Rick Macklem'" References: <014f01cc8945$a2f648e0$e8e2daa0$@com> <287177506.3014184.1318472210023.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <287177506.3014184.1318472210023.JavaMail.root@erie.cs.uoguelph.ca> Date: Wed, 12 Oct 2011 22:19:19 -0400 Message-ID: <016901cc894e$83e5ed80$8bb1c880$@com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: AcyJTixDEczJvZLwSB2+lWOv1Ba4TgAACA9A Content-Language: en-us X-DSPAM-Result: Innocent X-DSPAM-Processed: Wed Oct 12 22:19:23 2011 X-DSPAM-Confidence: 0.9962 X-DSPAM-Improbability: 1 in 26150 chance of being spam X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 1,4e964aab11841046321393 X-DSPAM-Factors: 27, References*com>, 0.00148, Url*freebsd, 0.00169, Url*freebsd, 0.00169, freebsd+org, 0.00170, freebsd+org, 0.00170, server+is, 0.00211, Breahna, 0.00242, Breahna, 0.00242, ?+>, 0.00264, freebsd, 0.00340, freebsd, 0.00340, To+John, 0.00348, UTC+2011, 0.00354, FreeBSD, 0.00358, FreeBSD, 0.00358, no+>, 0.00380, 17+PM, 0.00396, Ok+I, 0.00402, but+>, 0.00487, >+Cc, 0.00501, To+George, 0.00549, you+wrote, 0.00569, From+Rick, 0.00624, From+Rick, 0.00624, >+Can, 0.00643, 2011+8, 0.00644, George+Breahna, 0.00673 X-PolarisMail-Flags: x X-Mailman-Approved-At: Thu, 13 Oct 2011 15:54:10 +0000 Cc: freebsd-amd64@freebsd.org Subject: RE: amd64/161493: NFS v3 directory structure update slow X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Oct 2011 02:19:25 -0000 I just about to write back to mention this. I "fixed" the problem by running with -o but the old server still has = that problem but very rarely. The new server is affected 100% of the = time for some reason. If I'm not using NFSv4 is there a performance reason why I should still = use the new nfs server over the old one ? Thank you very much for your help guys. George -----Original Message----- From: Rick Macklem [mailto:rmacklem@uoguelph.ca]=20 Sent: Wednesday, October 12, 2011 10:17 PM To: George Breahna Cc: freebsd-amd64@freebsd.org; John Baldwin Subject: Re: amd64/161493: NFS v3 directory structure update slow George Breahna wrote: > Ok, I will try this. >=20 > I noticed you wrote another patch, available here, called the dotdot > patch. It modifies another file on top of the one mentioned in the > link you gave me. Is that unnecessary now ? >=20 > http://people.freebsd.org/~rmacklem/dotdot.patch >=20 The above patch also fixed the old server. See below for the patch for the old server. (You will be running the old server if you start both mountd and nfsd with a "-o" option. This happens if you have oldnfs_server_enable=3D"YES" in your /etc/rc.conf.) Since "nfsstat -s" shows non-zero counts, you are running the new/default server. ("nfsstat -o -s" reports stats for the old server, which should be all zeros if you are running the new/default = one.) In summary, I don't think you are running the old server and only need to patch the old server if you choose to run it, as jhb@ suggested to help with isolating the problem. (I would suggest you do that, if the patch for the new/regular server doesn't fix the problem.) > George >=20 > -----Original Message----- > From: Rick Macklem [mailto:rmacklem@uoguelph.ca] > Sent: Wednesday, October 12, 2011 8:25 PM > To: John Baldwin > Cc: George Breahna; freebsd-gnats-submit@freebsd.org; Rick Macklem; > freebsd-amd64@freebsd.org > Subject: Re: amd64/161493: NFS v3 directory structure update slow >=20 > John Baldwin wrote: > > On Tuesday, October 11, 2011 11:07:13 am George Breahna wrote: > > > > > > >Number: 161493 > > > >Category: amd64 > > > >Synopsis: NFS v3 directory structure update slow > > > >Confidential: no > > > >Severity: critical > > > >Priority: high > > > >Responsible: freebsd-amd64 > > > >State: open > > > >Quarter: > > > >Keywords: > > > >Date-Required: > > > >Class: sw-bug > > > >Submitter-Id: current-users > > > >Arrival-Date: Tue Oct 11 15:10:07 UTC 2011 > > > >Closed-Date: > > > >Last-Modified: > > > >Originator: George Breahna > > > >Release: 9.0 Beta 2 > > > >Organization: > > > >Environment: > > > FreeBSD store2 9.0-BETA2 FreeBSD 9.0-BETA2 #0: Sun Sep 18 22:02:45 > > > EDT 2011 > > pulsar@store2.emailarray.com:/usr/obj/usr/src/sys/PULSAR amd64 > > > >Description: > > > We used to run a NFS server on FreeBSD 6.2 but we built a new box > > > recently > > and installed 9.0 Beta 2 on it. The data was moved over as it serves > > as the > > back-end for a mail system. It runs NFS v3 over TCP only and all the > > NFS- > > related processes (rpcbind, mountd, lockd, etc ) run with the -h > > switch and > > bind to the local IP address. > > > > > > The NFS server exports the data to 7 NFS clients ranging from > > > FreeBSD 6.1 to > > 8.2, the majority being 8.2 The mount on the NFS clients is done > > simply with - > > o tcp,rsize=3D32768,wsize=3D32768 > > > > > > Usual file operations, such as accessing files, creating > > > directories, > > removing files, chmod, chown, etc work perfectly but we noticed > > there > > were > > issues in removing directories that contained data. We had a strange > > error: > > > > > > rm -rf nick/ > > > rm: fts_read: Input/output error > > > > > > Using 'truss' on rm revealed this: > > > > > > open("..",O_RDONLY,00) ERR#5 'Input/output error' > > > > > > After much testing and debugging we realized the problem is in the > > > NFS > > protocol. ( either server or client but we assume server since this > > used to > > work very well with FreeBSD 6.2 ). The problem appears to be that > > NFS > > does not > > show the '..' after modifying a directory structure. Take the > > following > > example executed on a FreeBSD 8.2 client accessing the NFS share > > from > > the > > 9.0B2 server: > > > > > > imap5# mkdir test1 > > > imap5# cd test1 > > > imap5# touch file1 > > > imap5# touch file2 > > > imap5# ls -la > > > ls: ..: Input/output error > > > total 4 > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > > > > > > Notice the '..' is missing from the display. If we now try and > > > remove the > > directory 'test1' it will throw the "rm: fts_read: Input/output > > error" > > error. > > > > > > If we wait in between 1 minute and 5 minutes, '..' will eventually > > > appear by > > itself. During this whole time, '..' effectively exists on the NFS > > server but > > it's not displayed by any of the NFS clients. > > > > > > I can force the NFS client to show it faster by doing an ls -la > > > from > > > the > > parent level. For example: > > > > > > imap5# mkdir test1 > > > imap5# touch test1/file1 > > > imap5# touch test1/file2 > > > imap5# touch test1/file3 > > > imap5# ls -la test1 > > > total 8 > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > > imap5# cd test1 > > > imap5# ls -la > > > total 8 > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > > > > > but if we wait 5 seconds after that display and try again: > > > > > > ls -la > > > ls: ..: Input/output error > > > total 4 > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > > > > > Again, if we wait longer ( 1-5 minutes ), the '..' will properly > > > appear in > > there. > > > > > > There are no error messages on the console or other log files. > > > This > > > is > > reproducible 100% of the time with any FreeBSD client. Have tried > > unmounting/remounting several times without any effect. Also tried > > different > > rsize/wsize, no effect. I think there is some delay in updating the > > directory > > structure and it's causing this bug. > > > > > > Here's also some output from nfsstat on the server: > > > > > > > > > Server Info: > > > Getattr Setattr Lookup Readlink Read Write Create > > Remove > > > 114731225 20496896 254966151 133 11697392 19963641 0 > > 9228861 > > > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus > > Access > > > 4313471 1157651 39 1955 16511932 15479669 0 > > 116927742 > > > Mknod Fsstat Fsinfo PathConf Commit > > > 0 4748487 48 0 14921747 > > > Server Ret-Failed > > > 0 > > > Server Faults > > > 0 > > > Server Cache Stats: > > > Inprog Idem Non-idem Misses > > > 0 0 0 613368147 > > > Server Write Gathering: > > > WriteOps WriteRPC Opsaved > > > 19963641 19963641 0 > > > > > > >How-To-Repeat: > > > imap5# mkdir test1 > > > imap5# cd test1 > > > imap5# touch file1 > > > imap5# touch file2 > > > imap5# ls -la > > > ls: ..: Input/output error > > > total 4 > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > > > >Fix: > > > > Can you try using the "old" NFS server as a test? > > > Please make sure you have the patch in r225356 in your server's > kernel sources (it went into head on Sep. 3, but I don't know if > your Sep. 11 build would have it?). It fixed a problem that would > cause lookup of ".." to fail intermittently, because a field in > struct nameidata added on Aug. 13 wasn't initialized. >=20 > You can find the one line patch here: > = http://svnweb.freebsd.org/base/head/sys/fs/nfsserver/nfs_nfsdport.c?r1=3D= 224911&r2=3D225356 Clarification, this is the patch for the new/default server. There is a similar patch for the old server. For the old server, the patch is: = http://svnweb.freebsd.org/base/head/sys/nfsserver/nfs_serv.c?r1=3D219028&= r2=3D225356 rick >=20 > Please let us know if you have this patch and, if not, apply it > and see if the problem goes away. >=20 > Thanks, rick From owner-freebsd-amd64@FreeBSD.ORG Thu Oct 13 02:25:47 2011 Return-Path: Delivered-To: freebsd-amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 956ED10656A3; Thu, 13 Oct 2011 02:25:47 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id E430A8FC16; Thu, 13 Oct 2011 02:25:46 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AmkAAKhLlk6DaFvO/2dsb2JhbABDFoRflCyQJoFTAQEEASNWBQcPEQMBAQEBAgINFgMCSAkIBhOHfgimUJFzgSyBdIMkgRQEk3WRcA X-IronPort-AV: E=Sophos;i="4.69,337,1315195200"; d="scan'208";a="139637355" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 12 Oct 2011 22:25:46 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id ED33AB3F5F; Wed, 12 Oct 2011 22:25:45 -0400 (EDT) Date: Wed, 12 Oct 2011 22:25:45 -0400 (EDT) From: Rick Macklem To: George Breahna Message-ID: <100238587.3014568.1318472745957.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <016901cc894e$83e5ed80$8bb1c880$@com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) X-Mailman-Approved-At: Thu, 13 Oct 2011 15:54:33 +0000 Cc: freebsd-amd64@freebsd.org Subject: Re: amd64/161493: NFS v3 directory structure update slow X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Oct 2011 02:25:47 -0000 George Breahna wrote: > I just about to write back to mention this. > > I "fixed" the problem by running with -o but the old server still has > that problem but very rarely. The new server is affected 100% of the > time for some reason. > Well, the problem was that the new field in struct nameidata wasn't initialized by the NFS servers. As such, it just depends on what happens to be on the stack, complicated slightly by cases where the test of the variable is != 0 vs another case where it must be 1 for the problem to show up. > If I'm not using NFSv4 is there a performance reason why I should > still use the new nfs server over the old one ? > The new/default server uses a different duplicate request cache specifically designed for NFS, including over TCP. I hope that this cache will work more efficiently than the generic (knows nothing about NFS) one in the krpc, which is what the old server uses. Whether this has any real effect on performance for your case, I can't say. Try the patches. I think both will work ok after the patches are applied, since they specifically fixed a problem w.r.t. lookup of "..". rick > Thank you very much for your help guys. > > George > > -----Original Message----- > From: Rick Macklem [mailto:rmacklem@uoguelph.ca] > Sent: Wednesday, October 12, 2011 10:17 PM > To: George Breahna > Cc: freebsd-amd64@freebsd.org; John Baldwin > Subject: Re: amd64/161493: NFS v3 directory structure update slow > > George Breahna wrote: > > Ok, I will try this. > > > > I noticed you wrote another patch, available here, called the dotdot > > patch. It modifies another file on top of the one mentioned in the > > link you gave me. Is that unnecessary now ? > > > > http://people.freebsd.org/~rmacklem/dotdot.patch > > > The above patch also fixed the old server. See below for the patch > for the old server. (You will be running the old server if you start > both mountd and nfsd with a "-o" option. This happens if you have > oldnfs_server_enable="YES" > in your /etc/rc.conf.) > > Since "nfsstat -s" shows non-zero counts, you are running the > new/default server. ("nfsstat -o -s" reports stats for the old > server, which should be all zeros if you are running the new/default > one.) > > In summary, I don't think you are running the old server and only > need to patch the old server if you choose to run it, as jhb@ > suggested > to help with isolating the problem. (I would suggest you do that, if > the patch for the new/regular server doesn't fix the problem.) > > > George > > > > -----Original Message----- > > From: Rick Macklem [mailto:rmacklem@uoguelph.ca] > > Sent: Wednesday, October 12, 2011 8:25 PM > > To: John Baldwin > > Cc: George Breahna; freebsd-gnats-submit@freebsd.org; Rick Macklem; > > freebsd-amd64@freebsd.org > > Subject: Re: amd64/161493: NFS v3 directory structure update slow > > > > John Baldwin wrote: > > > On Tuesday, October 11, 2011 11:07:13 am George Breahna wrote: > > > > > > > > >Number: 161493 > > > > >Category: amd64 > > > > >Synopsis: NFS v3 directory structure update slow > > > > >Confidential: no > > > > >Severity: critical > > > > >Priority: high > > > > >Responsible: freebsd-amd64 > > > > >State: open > > > > >Quarter: > > > > >Keywords: > > > > >Date-Required: > > > > >Class: sw-bug > > > > >Submitter-Id: current-users > > > > >Arrival-Date: Tue Oct 11 15:10:07 UTC 2011 > > > > >Closed-Date: > > > > >Last-Modified: > > > > >Originator: George Breahna > > > > >Release: 9.0 Beta 2 > > > > >Organization: > > > > >Environment: > > > > FreeBSD store2 9.0-BETA2 FreeBSD 9.0-BETA2 #0: Sun Sep 18 > > > > 22:02:45 > > > > EDT 2011 > > > pulsar@store2.emailarray.com:/usr/obj/usr/src/sys/PULSAR amd64 > > > > >Description: > > > > We used to run a NFS server on FreeBSD 6.2 but we built a new > > > > box > > > > recently > > > and installed 9.0 Beta 2 on it. The data was moved over as it > > > serves > > > as the > > > back-end for a mail system. It runs NFS v3 over TCP only and all > > > the > > > NFS- > > > related processes (rpcbind, mountd, lockd, etc ) run with the -h > > > switch and > > > bind to the local IP address. > > > > > > > > The NFS server exports the data to 7 NFS clients ranging from > > > > FreeBSD 6.1 to > > > 8.2, the majority being 8.2 The mount on the NFS clients is done > > > simply with - > > > o tcp,rsize=32768,wsize=32768 > > > > > > > > Usual file operations, such as accessing files, creating > > > > directories, > > > removing files, chmod, chown, etc work perfectly but we noticed > > > there > > > were > > > issues in removing directories that contained data. We had a > > > strange > > > error: > > > > > > > > rm -rf nick/ > > > > rm: fts_read: Input/output error > > > > > > > > Using 'truss' on rm revealed this: > > > > > > > > open("..",O_RDONLY,00) ERR#5 'Input/output error' > > > > > > > > After much testing and debugging we realized the problem is in > > > > the > > > > NFS > > > protocol. ( either server or client but we assume server since > > > this > > > used to > > > work very well with FreeBSD 6.2 ). The problem appears to be that > > > NFS > > > does not > > > show the '..' after modifying a directory structure. Take the > > > following > > > example executed on a FreeBSD 8.2 client accessing the NFS share > > > from > > > the > > > 9.0B2 server: > > > > > > > > imap5# mkdir test1 > > > > imap5# cd test1 > > > > imap5# touch file1 > > > > imap5# touch file2 > > > > imap5# ls -la > > > > ls: ..: Input/output error > > > > total 4 > > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > > > > > > > > Notice the '..' is missing from the display. If we now try and > > > > remove the > > > directory 'test1' it will throw the "rm: fts_read: Input/output > > > error" > > > error. > > > > > > > > If we wait in between 1 minute and 5 minutes, '..' will > > > > eventually > > > > appear by > > > itself. During this whole time, '..' effectively exists on the NFS > > > server but > > > it's not displayed by any of the NFS clients. > > > > > > > > I can force the NFS client to show it faster by doing an ls -la > > > > from > > > > the > > > parent level. For example: > > > > > > > > imap5# mkdir test1 > > > > imap5# touch test1/file1 > > > > imap5# touch test1/file2 > > > > imap5# touch test1/file3 > > > > imap5# ls -la test1 > > > > total 8 > > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > > > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > > > imap5# cd test1 > > > > imap5# ls -la > > > > total 8 > > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > > > drwx------ 10 vpopmail vchkpw 1024 Oct 11 10:59 .. > > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > > > > > > > but if we wait 5 seconds after that display and try again: > > > > > > > > ls -la > > > > ls: ..: Input/output error > > > > total 4 > > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:59 . > > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file1 > > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file2 > > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:59 file3 > > > > > > > > Again, if we wait longer ( 1-5 minutes ), the '..' will properly > > > > appear in > > > there. > > > > > > > > There are no error messages on the console or other log files. > > > > This > > > > is > > > reproducible 100% of the time with any FreeBSD client. Have tried > > > unmounting/remounting several times without any effect. Also tried > > > different > > > rsize/wsize, no effect. I think there is some delay in updating > > > the > > > directory > > > structure and it's causing this bug. > > > > > > > > Here's also some output from nfsstat on the server: > > > > > > > > > > > > Server Info: > > > > Getattr Setattr Lookup Readlink Read Write Create > > > Remove > > > > 114731225 20496896 254966151 133 11697392 19963641 0 > > > 9228861 > > > > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus > > > Access > > > > 4313471 1157651 39 1955 16511932 15479669 0 > > > 116927742 > > > > Mknod Fsstat Fsinfo PathConf Commit > > > > 0 4748487 48 0 14921747 > > > > Server Ret-Failed > > > > 0 > > > > Server Faults > > > > 0 > > > > Server Cache Stats: > > > > Inprog Idem Non-idem Misses > > > > 0 0 0 613368147 > > > > Server Write Gathering: > > > > WriteOps WriteRPC Opsaved > > > > 19963641 19963641 0 > > > > > > > > >How-To-Repeat: > > > > imap5# mkdir test1 > > > > imap5# cd test1 > > > > imap5# touch file1 > > > > imap5# touch file2 > > > > imap5# ls -la > > > > ls: ..: Input/output error > > > > total 4 > > > > drwxr-xr-x 2 root vchkpw 512 Oct 11 10:55 . > > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file1 > > > > -rw-r--r-- 1 root vchkpw 0 Oct 11 10:55 file2 > > > > >Fix: > > > > > > Can you try using the "old" NFS server as a test? > > > > > Please make sure you have the patch in r225356 in your server's > > kernel sources (it went into head on Sep. 3, but I don't know if > > your Sep. 11 build would have it?). It fixed a problem that would > > cause lookup of ".." to fail intermittently, because a field in > > struct nameidata added on Aug. 13 wasn't initialized. > > > > You can find the one line patch here: > > http://svnweb.freebsd.org/base/head/sys/fs/nfsserver/nfs_nfsdport.c?r1=224911&r2=225356 > > Clarification, this is the patch for the new/default server. There is > a > similar patch for the old server. For the old server, the patch is: > http://svnweb.freebsd.org/base/head/sys/nfsserver/nfs_serv.c?r1=219028&r2=225356 > > rick > > > > Please let us know if you have this patch and, if not, apply it > > and see if the problem goes away. > > > > Thanks, rick From owner-freebsd-amd64@FreeBSD.ORG Fri Oct 14 21:40:22 2011 Return-Path: Delivered-To: freebsd-amd64@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1C101065670 for ; Fri, 14 Oct 2011 21:40:22 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-yx0-f177.google.com (mail-yx0-f177.google.com [209.85.213.177]) by mx1.freebsd.org (Postfix) with ESMTP id 7F6988FC0C for ; Fri, 14 Oct 2011 21:40:22 +0000 (UTC) Received: by yxk36 with SMTP id 36so1644342yxk.8 for ; Fri, 14 Oct 2011 14:40:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=1WFVI6R4unc3x++UK0GF5Jscf2/0HMPPKxMXXVO5I5c=; b=hLAet3mY4lv9T9tAMuWIuUO/QlhjY3PU2BxFlFkZPlwxv8/AebUcIn8S5phAHYhhpI caZ3BQMCgYLH+Qgipoq8Mhk2i4P+xooVSqNu3zGd7izsb7LUSisf5PTFGfDw9fRPPKSh geaTmvvj8FbK2eWGZE8FcEmUlAwJn71552GaY= MIME-Version: 1.0 Received: by 10.182.202.68 with SMTP id kg4mr6513792obc.21.1318628421832; Fri, 14 Oct 2011 14:40:21 -0700 (PDT) Received: by 10.182.220.1 with HTTP; Fri, 14 Oct 2011 14:40:21 -0700 (PDT) In-Reply-To: References: <714EF3C9-33B0-4EF5-B52C-1E95F7F432F9@fisglobal.com> Date: Sat, 15 Oct 2011 01:40:21 +0400 Message-ID: From: Sergey Kandaurov To: Devin Teske Content-Type: multipart/mixed; boundary=e89a8f646a1dac909804af491af9 Cc: Dave Robison , FreeBSD amd64 Subject: Re: 32-bit route(8) on amd64 host and jails X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2011 21:40:22 -0000 --e89a8f646a1dac909804af491af9 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 21 September 2011 16:47, Sergey Kandaurov wrote: > On 21 September 2011 06:40, Devin Teske wrote= : >> I'm noticing that a 32-bit route(8) shows strange behaviour while runnin= g under an amd64 kernel (regardless of whether in the base-host -- utilizin= g /usr/lib32/libc.so.7 -- or in a jail and/or vimage -- 32-bit in nature; r= esults are same). >> >> Executable runs fine, but you can't (a) set the default route or (b) vie= w the default route (after successfully setting it with the amd64 build, of= course). >> >> ASIDE: This is under 8.1-RELEASE. >> >> When attempting to set the default route, you get the following... >> >> root@kps0a / # route add -net default 10.10.125.99 >> route: writing to routing socket: Invalid argument >> add net default: gateway 10.10.125.99: Invalid argument >> >> Meanwhile, using the amd64 version, no issues. >> >> When attempting to view the default route, you get the following... >> >> root@kps0a / # route -n get default >> =A0 route to: default >> destination: default >> =A0 =A0 =A0 mask: default >> =A0 =A0gateway: default >> =A0 =A0 =A0flags: >> =A0recvpipe =A0sendpipe =A0ssthresh =A0rtt,msec =A0 =A0mtu =A0 =A0 =A0 = =A0weight =A0 =A0expire >> =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 = =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0-1316570637 >> >> It's the "gateway: default" that's out of place. >> > > Currently, FreeBSD has a poor shape of rtsocket freebsd32 compatibility, > AFAIK only sysctl layer (sysctl_rtsock) is aware of it. > That means when a 32bit app writes some data to a routing socket, > kernel expects to receive it in the native ABI, and it doesn't work. Can you please try this preliminary patch and report back? You only need to rebuild kernel and reboot with the attached patch to enable use of e.g. 32-bit route(8) command on amd64 kernel. No any userland utility needs to rebuild to enable the compatibility. Tested on 8.2 amd64 only with route get default and route delete/add. --=20 wbr, pluknet --e89a8f646a1dac909804af491af9 Content-Type: text/x-patch; charset=US-ASCII; name="rtmsg32.diff" Content-Disposition: attachment; filename="rtmsg32.diff" Content-Transfer-Encoding: base64 X-Attachment-Id: f_gtrp0t4t0 SW5kZXg6IHN5cy9uZXQvcnRzb2NrLmMKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gc3lzL25ldC9ydHNvY2suYwko cmV2aXNpb24gMjI1NjcxKQorKysgc3lzL25ldC9ydHNvY2suYwkod29ya2luZyBjb3B5KQpAQCAt NzUsNiArNzUsNyBAQCBleHRlcm4gdm9pZCBzY3RwX2FkZHJfY2hhbmdlKHN0cnVjdCBpZmFkZHIg KmlmYSwgaQogCiAjaWZkZWYgQ09NUEFUX0ZSRUVCU0QzMgogI2luY2x1ZGUgPHN5cy9tb3VudC5o PgorI2luY2x1ZGUgPHN5cy9zeXNlbnQuaD4KICNpbmNsdWRlIDxjb21wYXQvZnJlZWJzZDMyL2Zy ZWVic2QzMi5oPgogCiBzdHJ1Y3QgaWZfZGF0YTMyIHsKQEAgLTExNCw2ICsxMTUsNDIgQEAgc3Ry dWN0IGlmX21zZ2hkcjMyIHsKIAl1aW50MTZfdCBpZm1faW5kZXg7CiAJc3RydWN0CWlmX2RhdGEz MiBpZm1fZGF0YTsKIH07CisKK3N0cnVjdCBydF9tZXRyaWNzMzIgeworCXVpbnQzMl90CXJteF9s b2NrczsKKwl1aW50MzJfdAlybXhfbXR1OworCXVpbnQzMl90CXJteF9ob3Bjb3VudDsKKwl1aW50 MzJfdAlybXhfZXhwaXJlOworCXVpbnQzMl90CXJteF9yZWN2cGlwZTsKKwl1aW50MzJfdAlybXhf c2VuZHBpcGU7CisJdWludDMyX3QJcm14X3NzdGhyZXNoOworCXVpbnQzMl90CXJteF9ydHQ7CisJ dWludDMyX3QJcm14X3J0dHZhcjsKKwl1aW50MzJfdAlybXhfcGtzZW50OworCXVpbnQzMl90CXJt eF93ZWlnaHQ7CisJdWludDMyX3QJcm14X2ZpbGxlclszXTsKK307CisKK3N0cnVjdCBydF9tc2do ZHIzMiB7CisJdWludDE2X3QJcnRtX21zZ2xlbjsKKwl1aW50OF90CQlydG1fdmVyc2lvbjsKKwl1 aW50OF90CQlydG1fdHlwZTsKKwl1aW50MTZfdAlydG1faW5kZXg7CisJdWludDE2X3QJcnRtX2hv bGUxOworCXVpbnQzMl90CXJ0bV9mbGFnczsKKwl1aW50MzJfdAlydG1fYWRkcnM7CisJdWludDMy X3QJcnRtX3BpZDsKKwl1aW50MzJfdAlydG1fc2VxOworCXVpbnQzMl90CXJ0bV9lcnJubzsKKwl1 aW50MzJfdAlydG1fZm1hc2s7CisJdWludDMyX3QJcnRtX2luaXRzOworCXN0cnVjdCBydF9tZXRy aWNzMzIgcnRtX3JteDsKK307CisKKyNkZWZpbmUJU0FfU0laRTMyKHNhKQkJCQkJCVwKKyAgICAo ICAoIShzYSkgfHwgKChzdHJ1Y3Qgc29ja2FkZHIgKikoc2EpKS0+c2FfbGVuID09IDApID8JXAor CXNpemVvZihpbnQzMl90KQkJOgkJCQlcCisJMSArICggKCgoc3RydWN0IHNvY2thZGRyICopKHNh KSktPnNhX2xlbiAtIDEpIHwgKHNpemVvZihpbnQzMl90KSAtIDEpKSkKICNlbmRpZgogCiBNQUxM T0NfREVGSU5FKE1fUlRBQkxFLCAicm91dGV0YmwiLCAicm91dGluZyB0YWJsZXMiKTsKQEAgLTUw MSw2ICs1MzgsMTM4IEBAIHJ0bV9nZXRfamFpbGVkKHN0cnVjdCBydF9hZGRyaW5mbyAqaW5mbywg c3RydWN0IGlmCiAJcmV0dXJuICgwKTsKIH0KIAorI2lmZGVmIENPTVBBVF9GUkVFQlNEMzIKK3N0 YXRpYyB2b2lkCitmcmVlYnNkMzJfcnRfbWV0cmljc19pbihzdHJ1Y3QgcnRfbWV0cmljczMyICpz cmMsIHN0cnVjdCBydF9tZXRyaWNzICpkc3QpCit7CisKKwliemVybyhkc3QsIHNpemVvZigqZHN0 KSk7CisJQ1AoKnNyYywgKmRzdCwgcm14X210dSk7CisJQ1AoKnNyYywgKmRzdCwgcm14X2V4cGly ZSk7CisJQ1AoKnNyYywgKmRzdCwgcm14X3Brc2VudCk7CisJQ1AoKnNyYywgKmRzdCwgcm14X3dl aWdodCk7Cit9CisKK3N0YXRpYyB2b2lkCitmcmVlYnNkMzJfcnRfbWV0cmljc19vdXQoc3RydWN0 IHJ0X21ldHJpY3MgKnNyYywgc3RydWN0IHJ0X21ldHJpY3MzMiAqZHN0KQoreworCisJYnplcm8o ZHN0LCBzaXplb2YoKmRzdCkpOworCUNQKCpzcmMsICpkc3QsIHJteF9tdHUpOworCUNQKCpzcmMs ICpkc3QsIHJteF9leHBpcmUpOworCUNQKCpzcmMsICpkc3QsIHJteF9wa3NlbnQpOworCUNQKCpz cmMsICpkc3QsIHJteF93ZWlnaHQpOworfQorCitzdGF0aWMgdm9pZAorZnJlZWJzZDMyX3J0X21z Z2hkcl9pbihzdHJ1Y3QgcnRfbXNnaGRyMzIgKnNyYywgc3RydWN0IHJ0X21zZ2hkciAqZHN0KQor eworCisJYnplcm8oZHN0LCBzaXplb2YoKmRzdCkpOworCS8qIENQKCpzcmMsICpkc3QsIHJ0bV9t c2dsZW4pOyAqLwkvKiB1cGRhdGVkIHNlcGFyYXRlbHkgKi8KKwlDUCgqc3JjLCAqZHN0LCBydG1f dmVyc2lvbik7CisJQ1AoKnNyYywgKmRzdCwgcnRtX3R5cGUpOworCUNQKCpzcmMsICpkc3QsIHJ0 bV9pbmRleCk7CisJQ1AoKnNyYywgKmRzdCwgcnRtX2ZsYWdzKTsKKwlDUCgqc3JjLCAqZHN0LCBy dG1fYWRkcnMpOworCUNQKCpzcmMsICpkc3QsIHJ0bV9waWQpOworCUNQKCpzcmMsICpkc3QsIHJ0 bV9zZXEpOworCUNQKCpzcmMsICpkc3QsIHJ0bV9lcnJubyk7CisJQ1AoKnNyYywgKmRzdCwgcnRt X2ZtYXNrKTsKKwlDUCgqc3JjLCAqZHN0LCBydG1faW5pdHMpOworCWZyZWVic2QzMl9ydF9tZXRy aWNzX2luKCZzcmMtPnJ0bV9ybXgsICZkc3QtPnJ0bV9ybXgpOworfQorCitzdGF0aWMgdm9pZAor ZnJlZWJzZDMyX3J0X21zZ2hkcl9vdXQoc3RydWN0IHJ0X21zZ2hkciAqc3JjLCBzdHJ1Y3QgcnRf bXNnaGRyMzIgKmRzdCkKK3sKKworCWJ6ZXJvKGRzdCwgc2l6ZW9mKCpkc3QpKTsKKwkvKiBDUCgq c3JjLCAqZHN0LCBydG1fbXNnbGVuKTsgKi8JLyogdXBkYXRlZCBzZXBhcmF0ZWx5ICovCisJQ1Ao KnNyYywgKmRzdCwgcnRtX3ZlcnNpb24pOworCUNQKCpzcmMsICpkc3QsIHJ0bV90eXBlKTsKKwlD UCgqc3JjLCAqZHN0LCBydG1faW5kZXgpOworCUNQKCpzcmMsICpkc3QsIHJ0bV9mbGFncyk7CisJ Q1AoKnNyYywgKmRzdCwgcnRtX2FkZHJzKTsKKwlDUCgqc3JjLCAqZHN0LCBydG1fcGlkKTsKKwlD UCgqc3JjLCAqZHN0LCBydG1fc2VxKTsKKwlDUCgqc3JjLCAqZHN0LCBydG1fZXJybm8pOworCUNQ KCpzcmMsICpkc3QsIHJ0bV9mbWFzayk7CisJQ1AoKnNyYywgKmRzdCwgcnRtX2luaXRzKTsKKwlm cmVlYnNkMzJfcnRfbWV0cmljc19vdXQoJnNyYy0+cnRtX3JteCwgJmRzdC0+cnRtX3JteCk7Cit9 CisKK3N0YXRpYyBpbnQKK2ZyZWVic2QzMl9ydF9tc3BhY2VfbGVuX2luKGNhZGRyX3QgY3AsIGNh ZGRyX3QgY3BsaW0sIGludCAqYnVmbGVuKQoreworCXN0cnVjdCBzb2NrYWRkciAqc2E7CisJaW50 IGk7CisKKwlmb3IgKGkgPSAwLCAqYnVmbGVuID0gMDsgaSA8IFJUQVhfTUFYICYmIGNwIDwgY3Bs aW07IGkrKykgeworCQlzYSA9IChzdHJ1Y3Qgc29ja2FkZHIgKiljcDsKKworCQlpZiAoY3AgKyBz YS0+c2FfbGVuID4gY3BsaW0pCisJCQlyZXR1cm4gKEVJTlZBTCk7CisJCWNwICs9IFNBX1NJWkUz MihzYSk7CisJCSpidWZsZW4gKz0gU0FfU0laRShzYSk7CisJfQorCXJldHVybiAoMCk7Cit9CisK K3N0YXRpYyBpbnQKK2ZyZWVic2QzMl9ydF9tc3BhY2VfbGVuX291dChjYWRkcl90IGNwLCBjYWRk cl90IGNwbGltLCBpbnQgKmJ1ZmxlbikKK3sKKwlzdHJ1Y3Qgc29ja2FkZHIgKnNhOworCWludCBp OworCisJZm9yIChpID0gMCwgKmJ1ZmxlbiA9IDA7IGkgPCBSVEFYX01BWCAmJiBjcCA8IGNwbGlt OyBpKyspIHsKKwkJc2EgPSAoc3RydWN0IHNvY2thZGRyICopY3A7CisKKwkJaWYgKGNwICsgc2Et PnNhX2xlbiA+IGNwbGltKQorCQkJcmV0dXJuIChFSU5WQUwpOworCQljcCArPSBTQV9TSVpFKHNh KTsKKwkJKmJ1ZmxlbiArPSBTQV9TSVpFMzIoc2EpOworCX0KKwlyZXR1cm4gKDApOworfQorCitz dGF0aWMgaW50CitmcmVlYnNkMzJfcnRfbXNwYWNlX2luKGNhZGRyX3QgY3AsIGNhZGRyX3QgY3A2 NCwgY2FkZHJfdCBjcGxpbSkKK3sKKwlzdHJ1Y3Qgc29ja2FkZHIgKnNhOworCWludCBpOworCisJ Zm9yIChpID0gMDsgaSA8IFJUQVhfTUFYICYmIGNwIDwgY3BsaW07IGkrKykgeworCQlzYSA9IChz dHJ1Y3Qgc29ja2FkZHIgKiljcDsKKworCQlpZiAoY3AgKyBzYS0+c2FfbGVuID4gY3BsaW0pCisJ CQlyZXR1cm4gKEVJTlZBTCk7CisJCW1lbWNweShjcDY0LCBjcCwgU0FfU0laRTMyKHNhKSk7CisJ CWNwICs9IFNBX1NJWkUzMihzYSk7CisJCWNwNjQgKz0gU0FfU0laRShzYSk7CisJfQorCXJldHVy biAoMCk7Cit9CisKK3N0YXRpYyBpbnQKK2ZyZWVic2QzMl9ydF9tc3BhY2Vfb3V0KGNhZGRyX3Qg Y3AsIGNhZGRyX3QgY3AzMiwgY2FkZHJfdCBjcGxpbSkKK3sKKwlzdHJ1Y3Qgc29ja2FkZHIgKnNh OworCWludCBpOworCisJZm9yIChpID0gMDsgaSA8IFJUQVhfTUFYICYmIGNwIDwgY3BsaW07IGkr KykgeworCQlzYSA9IChzdHJ1Y3Qgc29ja2FkZHIgKiljcDsKKworCQlpZiAoY3AgKyBzYS0+c2Ff bGVuID4gY3BsaW0pCisJCQlyZXR1cm4gKEVJTlZBTCk7CisJCW1lbWNweShjcDMyLCBjcCwgU0Ff U0laRShzYSkpOworCQljcCArPSBTQV9TSVpFKHNhKTsKKwkJY3AzMiArPSBTQV9TSVpFMzIoc2Ep OworCX0KKwlyZXR1cm4gKDApOworfQorI2VuZGlmCisKIC8qQVJHU1VTRUQqLwogc3RhdGljIGlu dAogcm91dGVfb3V0cHV0KHN0cnVjdCBtYnVmICptLCBzdHJ1Y3Qgc29ja2V0ICpzbykKQEAgLTUx Myw3ICs2ODIsMTQgQEAgcm91dGVfb3V0cHV0KHN0cnVjdCBtYnVmICptLCBzdHJ1Y3Qgc29ja2V0 ICpzbykKIAlpbnQgbGVuLCBlcnJvciA9IDA7CiAJc3RydWN0IGlmbmV0ICppZnAgPSBOVUxMOwog CXVuaW9uIHNvY2thZGRyX3VuaW9uIHNhdW47CisJc2l6ZV90IHJ0bWxlbjsKKyNpZmRlZiBDT01Q QVRfRlJFRUJTRDMyCisJc3RydWN0IHJ0X21zZ2hkcjMyICpydG0zMiA9IE5VTEw7CisJaW50IGxl bjMyID0gMCwgcnRfbXNwYWNlX2xlbiA9IDAsIHdyYXAzMiA9IDA7CiAKKwlpZiAoU1ZfQ1VSUFJP Q19GTEFHKFNWX0lMUDMyKSkKKwkJd3JhcDMyID0gMTsKKyNlbmRpZgogI2RlZmluZSBzZW5kZXJy KGUpIHsgZXJyb3IgPSBlOyBnb3RvIGZsdXNoO30KIAlpZiAobSA9PSBOVUxMIHx8ICgobS0+bV9s ZW4gPCBzaXplb2YobG9uZykpICYmCiAJCSAgICAgICAobSA9IG1fcHVsbHVwKG0sIHNpemVvZihs b25nKSkpID09IE5VTEwpKQpAQCAtNTIxLDE3ICs2OTcsNTAgQEAgcm91dGVfb3V0cHV0KHN0cnVj dCBtYnVmICptLCBzdHJ1Y3Qgc29ja2V0ICpzbykKIAlpZiAoKG0tPm1fZmxhZ3MgJiBNX1BLVEhE UikgPT0gMCkKIAkJcGFuaWMoInJvdXRlX291dHB1dCIpOwogCWxlbiA9IG0tPm1fcGt0aGRyLmxl bjsKLQlpZiAobGVuIDwgc2l6ZW9mKCpydG0pIHx8CisKKyNpZmRlZiBDT01QQVRfRlJFRUJTRDMy CisJaWYgKHdyYXAzMikgeworCQlydG1sZW4gPSBzaXplb2YoKnJ0bTMyKTsKKwkJbGVuMzIgPSBs ZW47CisJfSBlbHNlCisjZW5kaWYKKwkJcnRtbGVuID0gc2l6ZW9mKCpydG0pOworCisJaWYgKGxl biA8IHJ0bWxlbiB8fAogCSAgICBsZW4gIT0gbXRvZChtLCBzdHJ1Y3QgcnRfbXNnaGRyICopLT5y dG1fbXNnbGVuKSB7CiAJCWluZm8ucnRpX2luZm9bUlRBWF9EU1RdID0gTlVMTDsKIAkJc2VuZGVy cihFSU5WQUwpOwogCX0KLQlSX01hbGxvYyhydG0sIHN0cnVjdCBydF9tc2doZHIgKiwgbGVuKTsK KworI2lmZGVmIENPTVBBVF9GUkVFQlNEMzIKKwlpZiAod3JhcDMyKSB7CisJCVJfTWFsbG9jKHJ0 bTMyLCBzdHJ1Y3QgcnRfbXNnaGRyMzIgKiwgbGVuMzIpOworCQlpZiAocnRtMzIgPT0gTlVMTCkg eworCQkJaW5mby5ydGlfaW5mb1tSVEFYX0RTVF0gPSBOVUxMOworCQkJc2VuZGVycihFTk9CVUZT KTsKKwkJfQorCQltX2NvcHlkYXRhKG0sIDAsIGxlbjMyLCAoY2FkZHJfdClydG0zMik7CisJCWZy ZWVic2QzMl9ydF9tc3BhY2VfbGVuX2luKChjYWRkcl90KShydG0zMiArIDEpLCBsZW4zMiArIChj YWRkcl90KXJ0bTMyLCAmcnRfbXNwYWNlX2xlbik7CisJCS8qIGZpeHVwIGxlbiB0byBhbGxvYyBy dG0gb2YgbmF0aXZlIHNpemUgKi8KKwkJbGVuID0gcnRfbXNwYWNlX2xlbiArIHNpemVvZigqcnRt KTsKKwl9CisjZW5kaWYKKwlSX1phbGxvYyhydG0sIHN0cnVjdCBydF9tc2doZHIgKiwgbGVuKTsK IAlpZiAocnRtID09IE5VTEwpIHsKIAkJaW5mby5ydGlfaW5mb1tSVEFYX0RTVF0gPSBOVUxMOwog CQlzZW5kZXJyKEVOT0JVRlMpOwogCX0KLQltX2NvcHlkYXRhKG0sIDAsIGxlbiwgKGNhZGRyX3Qp cnRtKTsKKworI2lmZGVmIENPTVBBVF9GUkVFQlNEMzIKKwlpZiAod3JhcDMyKSB7CisJCWZyZWVi c2QzMl9ydF9tc2doZHJfaW4ocnRtMzIsIHJ0bSk7CisJCWZyZWVic2QzMl9ydF9tc3BhY2VfaW4o KGNhZGRyX3QpKHJ0bTMyICsgMSksCisJCSAgICAoY2FkZHJfdCkocnRtICsgMSksIChjYWRkcl90 KXJ0bTMyICsgbGVuMzIpOworCQlydG0tPnJ0bV9tc2dsZW4gPSBsZW47CisJfSBlbHNlCisjZW5k aWYKKwkJbV9jb3B5ZGF0YShtLCAwLCBsZW4sIChjYWRkcl90KXJ0bSk7CisKIAlpZiAocnRtLT5y dG1fdmVyc2lvbiAhPSBSVE1fVkVSU0lPTikgewogCQlpbmZvLnJ0aV9pbmZvW1JUQVhfRFNUXSA9 IE5VTEw7CiAJCXNlbmRlcnIoRVBST1RPTk9TVVBQT1JUKTsKQEAgLTU0Myw2ICs3NTIsMTYgQEAg cm91dGVfb3V0cHV0KHN0cnVjdCBtYnVmICptLCBzdHJ1Y3Qgc29ja2V0ICpzbykKIAkJaW5mby5y dGlfaW5mb1tSVEFYX0RTVF0gPSBOVUxMOwogCQlzZW5kZXJyKEVJTlZBTCk7CiAJfQorLyoKKyAg ICAgICAgaW50IGk7IHVpbnQxNl90ICpqOworCisgICAgICAgIGZvciAoaSA9IDA7IGkgKiAoaW50 KXNpemVvZih1aW50MTZfdCkgPCBsZW47IGkrKykgeworICAgICAgICAgICAgICAgICAgaiA9ICh1 aW50MTZfdCAqKXJ0bTsKKyAgICAgICAgICAgICAgICAgIGlmIChpICUgOCA9PSAwKQorICAgICAg ICAgICAgICAgICAgICAgICAgIHByaW50ZigiXG4weCUwNGp4ICIsICh1aW50bWF4X3QpaSpzaXpl b2YodWludDE2X3QpKTsKKyAgICAgICAgICAgICAgICAgIHByaW50ZigiJTA0eCAiLCBic3dhcDE2 KGpbaV0pKTsKKyAgICAgICAgfTsgcHJpbnRmKCJcbiIpOworKi8KIAlpbmZvLnJ0aV9mbGFncyA9 IHJ0bS0+cnRtX2ZsYWdzOwogCWlmIChpbmZvLnJ0aV9pbmZvW1JUQVhfRFNUXSA9PSBOVUxMIHx8 CiAJICAgIGluZm8ucnRpX2luZm9bUlRBWF9EU1RdLT5zYV9mYW1pbHkgPj0gQUZfTUFYIHx8CkBA IC04NzcsMTIgKzEwOTYsNDEgQEAgZmx1c2g6CiAJCXJwID0gc290b3Jhd2NiKHNvKTsKIAl9CiAJ aWYgKHJ0bSkgewotCQltX2NvcHliYWNrKG0sIDAsIHJ0bS0+cnRtX21zZ2xlbiwgKGNhZGRyX3Qp cnRtKTsKLQkJaWYgKG0tPm1fcGt0aGRyLmxlbiA8IHJ0bS0+cnRtX21zZ2xlbikgewotCQkJbV9m cmVlbShtKTsKLQkJCW0gPSBOVUxMOwotCQl9IGVsc2UgaWYgKG0tPm1fcGt0aGRyLmxlbiA+IHJ0 bS0+cnRtX21zZ2xlbikKLQkJCW1fYWRqKG0sIHJ0bS0+cnRtX21zZ2xlbiAtIG0tPm1fcGt0aGRy Lmxlbik7CisjaWZkZWYgQ09NUEFUX0ZSRUVCU0QzMgorCQlpZiAod3JhcDMyKSB7CisJCQlpbnQg cnRfbXNwYWNlX2xlbjMyOworCisJCQlmcmVlYnNkMzJfcnRfbXNwYWNlX2xlbl9vdXQoKGNhZGRy X3QpKHJ0bSArIDEpLAorCQkJICAgIGxlbiArIChjYWRkcl90KXJ0bSwgJnJ0X21zcGFjZV9sZW4z Mik7CisJCQlsZW4zMiA9IHJ0X21zcGFjZV9sZW4zMiArIHNpemVvZigqcnRtMzIpOworCQkJaWYg KGxlbjMyID4gcnRtMzItPnJ0bV9tc2dsZW4pIHsKKwkJCQlzdHJ1Y3QgcnRfbXNnaGRyMzIgKm5l d19ydG0zMjsKKwkJCQlSX01hbGxvYyhuZXdfcnRtMzIsIHN0cnVjdCBydF9tc2doZHIzMiAqLCBs ZW4zMik7CisJCQkJaWYgKG5ld19ydG0zMiA9PSBOVUxMKQorCQkJCQlzZW5kZXJyKEVOT0JVRlMp OworCQkJCWJjb3B5KHJ0bTMyLCBuZXdfcnRtMzIsIHJ0bTMyLT5ydG1fbXNnbGVuKTsKKwkJCQlG cmVlKHJ0bTMyKTsgcnRtMzIgPSBuZXdfcnRtMzI7CisJCQl9CisJCQlmcmVlYnNkMzJfcnRfbXNn aGRyX291dChydG0sIHJ0bTMyKTsKKwkJCXJ0bTMyLT5ydG1fbXNnbGVuID0gbGVuMzI7CisJCQlm cmVlYnNkMzJfcnRfbXNwYWNlX291dCgoY2FkZHJfdCkocnRtICsgMSksCisJCQkgICAgKGNhZGRy X3QpKHJ0bTMyICsgMSksIChjYWRkcl90KXJ0bSArIGxlbik7CisKKwkJCW1fY29weWJhY2sobSwg MCwgcnRtMzItPnJ0bV9tc2dsZW4sIChjYWRkcl90KXJ0bTMyKTsKKwkJCWlmIChtLT5tX3BrdGhk ci5sZW4gPCBydG0zMi0+cnRtX21zZ2xlbikgeworCQkJCW1fZnJlZW0obSk7CisJCQkJbSA9IE5V TEw7CisJCQl9IGVsc2UgaWYgKG0tPm1fcGt0aGRyLmxlbiA+IHJ0bTMyLT5ydG1fbXNnbGVuKQor CQkJCW1fYWRqKG0sIHJ0bTMyLT5ydG1fbXNnbGVuIC0gbS0+bV9wa3RoZHIubGVuKTsKKwkJfSBl bHNlIHsKKyNlbmRpZgorCQkJbV9jb3B5YmFjayhtLCAwLCBydG0tPnJ0bV9tc2dsZW4sIChjYWRk cl90KXJ0bSk7CisJCQlpZiAobS0+bV9wa3RoZHIubGVuIDwgcnRtLT5ydG1fbXNnbGVuKSB7CisJ CQkJbV9mcmVlbShtKTsKKwkJCQltID0gTlVMTDsKKwkJCX0gZWxzZSBpZiAobS0+bV9wa3RoZHIu bGVuID4gcnRtLT5ydG1fbXNnbGVuKQorCQkJCW1fYWRqKG0sIHJ0bS0+cnRtX21zZ2xlbiAtIG0t Pm1fcGt0aGRyLmxlbik7CisJCX0KIAl9CiAJaWYgKG0pIHsKIAkJaWYgKHJwKSB7CkBAIC04OTgs OCArMTE0NiwxMyBAQCBmbHVzaDoKIAkJCXJ0X2Rpc3BhdGNoKG0sIGluZm8ucnRpX2luZm9bUlRB WF9EU1RdKTsKIAl9CiAJLyogaW5mby5ydGlfaW5mb1tSVEFYX0RTVF0gKHVzZWQgYWJvdmUpIGNh biBwb2ludCBpbnNpZGUgb2YgcnRtICovCi0JaWYgKHJ0bSkKKwlpZiAocnRtKSB7CiAJCUZyZWUo cnRtKTsKKyNpZmRlZiBDT01QQVRfRlJFRUJTRDMyCisJCWlmICh3cmFwMzIpCisJCQlGcmVlKHJ0 bTMyKTsKKyNlbmRpZgorCX0KICAgICB9CiAJcmV0dXJuIChlcnJvcik7CiAjdW5kZWYJc2FfZXF1 YWwK --e89a8f646a1dac909804af491af9--