Date: Tue, 15 Mar 2005 16:32:26 GMT From: Cyril Vechera <cv-c@fluid.ru> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/78884: nfs client cache negative lookups Message-ID: <200503151632.j2FGWQGA004944@www.freebsd.org> Resent-Message-ID: <200503151640.j2FGe2Ig051698@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 78884
>Category: kern
>Synopsis: nfs client cache negative lookups
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: change-request
>Submitter-Id: current-users
>Arrival-Date: Tue Mar 15 16:40:02 GMT 2005
>Closed-Date:
>Last-Modified:
>Originator: Cyril Vechera
>Release: RELENG_5
>Organization:
Net Ltd.
>Environment:
FreeBSD cv.ws.local 5.3-RELEASE-p5 FreeBSD 5.3-RELEASE-p5 #1: Tue Mar 15 20:54:40 MSK 2005 root@5.devel.incru.net:/usr/src/sys/i386/compile/OFFICIER i386
>Description:
Current NFS client does not cache negative lookups that results significant network overload with identical lookup requests when searching in the PATH, libraries, configs etc.
When the caching of negative lookups is active it reduces meta-data traffic close to 80% and increases applictions startup time.
>How-To-Repeat:
>Fix:
Implementation below does not fit strong caching policy requirements due lack of cached entries time-to-live support. So negative caching is off by default and can be turned on via sysctl or mount_nfs option -C (negcache). But it is reasonable enough to turn it on everywhere with read-only NFS root and mounts with single-client. For more effective negative caching I'd recommend to set sysctl debug.ncnegfactor less than default 16, maybe to 4 or 2.
Here are two patches - with sysctl control and with mount flag control
variant 1, sysctl:
====cut here
*** /usr/src/sys/nfs_vnops.c.orig Tue Mar 15 00:17:26 2005
--- /usr/src/sys/nfs_vnops.c Tue Mar 15 00:20:08 2005
***************
*** 238,243 ****
--- 238,247 ----
static int nfsv3_commit_on_close = 0;
SYSCTL_INT(_vfs_nfs, OID_AUTO, nfsv3_commit_on_close, CTLFLAG_RW,
&nfsv3_commit_on_close, 0, "write+commit on close, else only write");
+
+ static int nfs_cache_negative_lookup = 0;
+ SYSCTL_INT(_vfs_nfs, OID_AUTO, cache_negative_lookup, CTLFLAG_RW,
+ &nfs_cache_negative_lookup, 0, "cache negative lookup");
#if 0
SYSCTL_INT(_vfs_nfs, OID_AUTO, access_cache_hits, CTLFLAG_RD,
&nfsstats.accesscache_hits, 0, "NFS ACCESS cache hit count");
***************
*** 767,776 ****
wantparent = flags & (LOCKPARENT|WANTPARENT);
nmp = VFSTONFS(dvp->v_mount);
np = VTONFS(dvp);
! if ((error = cache_lookup(dvp, vpp, cnp)) && error != ENOENT) {
struct vattr vattr;
int vpid;
if ((error = VOP_ACCESS(dvp, VEXEC, cnp->cn_cred, td)) != 0) {
*vpp = NULLVP;
return (error);
--- 767,782 ----
wantparent = flags & (LOCKPARENT|WANTPARENT);
nmp = VFSTONFS(dvp->v_mount);
np = VTONFS(dvp);
! error = cache_lookup(dvp, vpp, cnp);
! if (error) {
struct vattr vattr;
int vpid;
+ if (error == ENOENT) {
+ nfsstats.lookupcache_hits++;
+ return (error);
+ }
+
if ((error = VOP_ACCESS(dvp, VEXEC, cnp->cn_cred, td)) != 0) {
*vpp = NULLVP;
return (error);
***************
*** 941,946 ****
--- 946,953 ----
}
if (cnp->cn_nameiop != LOOKUP && (flags & ISLASTCN))
cnp->cn_flags |= SAVENAME;
+ if (error == ENOENT && nfs_cache_negative_lookup && (cnp->cn_flags & MAKEENTRY))
+ cache_enter(dvp, NULL, cnp);
}
return (error);
}
====cut here
variant 2, mount_nfs option:
====cut here
*** sys/nfs_vnops.c.orig Tue Mar 15 00:17:26 2005
--- sys/nfs_vnops.c Tue Mar 15 00:22:02 2005
***************
*** 767,776 ****
wantparent = flags & (LOCKPARENT|WANTPARENT);
nmp = VFSTONFS(dvp->v_mount);
np = VTONFS(dvp);
! if ((error = cache_lookup(dvp, vpp, cnp)) && error != ENOENT) {
struct vattr vattr;
int vpid;
if ((error = VOP_ACCESS(dvp, VEXEC, cnp->cn_cred, td)) != 0) {
*vpp = NULLVP;
return (error);
--- 767,782 ----
wantparent = flags & (LOCKPARENT|WANTPARENT);
nmp = VFSTONFS(dvp->v_mount);
np = VTONFS(dvp);
! error = cache_lookup(dvp, vpp, cnp);
! if (error) {
struct vattr vattr;
int vpid;
+ if (error == ENOENT) {
+ nfsstats.lookupcache_hits++;
+ return (error);
+ }
+
if ((error = VOP_ACCESS(dvp, VEXEC, cnp->cn_cred, td)) != 0) {
*vpp = NULLVP;
return (error);
***************
*** 941,946 ****
--- 942,949 ----
}
if (cnp->cn_nameiop != LOOKUP && (flags & ISLASTCN))
cnp->cn_flags |= SAVENAME;
+ if (error == ENOENT && (nmp->nm_flag & NFSMNT_NEGCACHE) && (cnp->cn_flags & MAKEENTRY))
+ cache_enter(dvp, NULL, cnp);
}
return (error);
}
*** sys/nfsargs.h.orig Tue Mar 15 00:25:40 2005
--- sys/nfsargs.h Tue Mar 15 00:24:36 2005
***************
*** 78,83 ****
--- 78,84 ----
#define NFSMNT_NOCONN 0x00000080 /* Don't Connect the socket */
/* 0x100 free, was NFSMNT_NQNFS */
#define NFSMNT_NFSV3 0x00000200 /* Use NFS Version 3 protocol */
+ #define NFSMNT_NEGCACHE 0x00000400 /* Cache negative lookups */
/* 0x400 free, was NFSMNT_KERB */
#define NFSMNT_DUMBTIMR 0x00000800 /* Don't estimate rtt dynamically */
/* 0x1000 free, was NFSMNT_LEASETERM */
====cut here
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200503151632.j2FGWQGA004944>