From owner-freebsd-stable@FreeBSD.ORG Fri Mar 24 18:00:38 2006 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DD28016A400; Fri, 24 Mar 2006 18:00:38 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4B54643D45; Fri, 24 Mar 2006 18:00:38 +0000 (GMT) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.13.4.20060308/8.13.4) with ESMTP id k2OI0Opa005585; Fri, 24 Mar 2006 10:00:24 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.13.4.20060308/8.13.4/Submit) id k2OI0KF8005579; Fri, 24 Mar 2006 10:00:20 -0800 (PST) Date: Fri, 24 Mar 2006 10:00:20 -0800 (PST) From: Matthew Dillon Message-Id: <200603241800.k2OI0KF8005579@apollo.backplane.com> To: Peter Jeremy References: <200603211607.30372.mi+mx@aldan.algebra.com> <200603231403.36136.mi+mx@aldan.algebra.com> <200603232048.k2NKm4QL067644@apollo.backplane.com> <200603231626.19102.mi+mx@aldan.algebra.com> <200603232316.k2NNGBka068754@apollo.backplane.com> <20060324084940.GA703@turion.vk2pj.dyndns.org> Cc: alc@freebsd.org, Mikhail Teterin , stable@freebsd.org Subject: Re: Reading via mmap stinks (Re: weird bugs with mmap-ing via NFS) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Mar 2006 18:00:38 -0000 :On an amd64 system running about 6-week old -stable, both behave :pretty much identically. In both cases, systat reports that the disk :is about 96% busy whilst loading the cache. In the cache case, mmap :is significantly faster. : :... :turion% ls -l /6_i386/var/tmp/test :-rw-r--r-- 1 peter wheel 586333684 Mar 24 19:24 /6_i386/var/tmp/test :turion% /usr/bin/time -l grep dfhfhdsfhjdsfl /6_i386/var/tmp/test : 21.69 real 0.16 user 0.68 sys :[umount/remount /6_i386/var] :turion% /usr/bin/time -l grep --mmap dfhfhdsfhjdsfl /6_i386/var/tmp/test : 21.68 real 0.41 user 0.51 sys :The speed gain with mmap is clearly evident when the data is cached and :the CPU clock wound right down (99MHz ISO 2200MHz): :... :-- :Peter Jeremy That pretty much means that the read-ahead algorithm is working. If it weren't, the disk would not be running at near 100%. Ok. The next test is to NOT do umount/remount and then use a data set that is ~2x system memory (but can still be mmap'd by grep). Rerun the data set multiple times using grep and grep --mmap. If the times for the mmap case blow up relative to the non-mmap case, then the vm_page_alloc() calls and/or vm_page_count_severe() (and other tests) in the vm_fault case are causing the read-ahead to drop out. If this is the case the problem is not in the read-ahead path, but probably in the pageout code not maintaining a sufficient number of free and cache pages. The system would only be allocating ~60MB/s (or whatever your disk can do), so the pageout thread ought to be able to keep up. If the times for the mmap case do not blow up, we are back to square one and I would start investigating the disk driver that Mikhail is using. -Matt Matthew Dillon