Date: Wed, 15 Feb 2012 11:40:12 +0200 From: "Pavlo" <devgs@ukr.net> To: freebsd-fs@freebsd.org Subject: ZFS and mem management Message-ID: <15861.1329298812.1414986334451204096@ffe12.ukr.net>
next in thread | raw e-mail | index | archive | help
Hello. We have an issue with memory management on FreeBSD and i suspect it is related to FS. We are using ZFS, here quick stats: zpool status pool: disk1 state: ONLINE scan: resilvered 657G in 8h30m with 0 errors on Tue Feb 14 21:17:37 2012 config: NAME STATE READ WRITE CKSUM disk1 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 gpt/disk2 ONLINE 0 0 0 gpt/disk4 ONLINE 0 0 0 gpt/disk6 ONLINE 0 0 0 gpt/disk8 ONLINE 0 0 0 gpt/disk10 ONLINE 0 0 0 gpt/disk12 ONLINE 0 0 0 mirror-7 ONLINE 0 0 0 gpt/disk14 ONLINE 0 0 0 gpt/disk15 ONLINE 0 0 0 errors: No known data errors pool: zroot state: ONLINE scan: resilvered 34.9G in 0h11m with 0 errors on Tue Feb 14 12:57:52 2012 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/sys0 ONLINE 0 0 0 gpt/sys1 ONLINE 0 0 0 errors: No known data errors ------------------------------------------------------------------------ System Memory: 0.95% 75.61 MiB Active, 0.24% 19.02 MiB Inact 18.25% 1.41 GiB Wired, 0.01% 480.00 KiB Cache 80.54% 6.24 GiB Free, 0.01% 604.00 KiB Gap Real Installed: 8.00 GiB Real Available: 99.84% 7.99 GiB Real Managed: 96.96% 7.74 GiB Logical Total: 8.00 GiB Logical Used: 21.79% 1.74 GiB Logical Free: 78.21% 6.26 GiB Kernel Memory: 1.18 GiB Data: 99.05% 1.17 GiB Text: 0.95% 11.50 MiB Kernel Memory Map: 4.39 GiB Size: 23.32% 1.02 GiB Free: 76.68% 3.37 GiB ------------------------------------------------------------------------ ------------------------------------------------------------------------ ZFS Subsystem Report Wed Feb 15 10:53:03 2012 ------------------------------------------------------------------------ System Information: Kernel Version: 802516 (osreldate) Hardware Platform: amd64 Processor Architecture: amd64 ZFS Storage pool Version: 28 ZFS Filesystem Version: 5 FreeBSD 8.2-STABLE #12: Thu Feb 9 11:35:23 EET 2012 root 10:53AM up 56 mins, 6 users, load averages: 0.00, 0.00, 0.00 ------------------------------------------------------------------------ Background: we are using some tool that does indexing of some data and then pushes it into database (currently bdb-5.2). Instances of indexer are running continuously one after another. Time of indexing for one instance of indexer may vary between 2 seconds and 30 minutes. But mostly it is below one minute. There is nothing else running on the machine except system stuff and daemons. After several hours of indexing i can see a lot of active memory, it's ok. Then i check the number of vnodes. and it's really huge: 300k+ even tho nobody has so many opened files. Reading docs and googling i figured that's because of cached pages that reside in memory (unmounting of disk causes whole memory to be freed). Also I figured that happens only when I am accessing files via mmap(). Looks like pretty legit behaviour but the issue is: This spectacle continues (approximately for 12 hours) unlit indexers began to be killed out of swap. As I wrote above I observe a lot of used vnodes and like 7GB of active memory. I made a tool that allocates memory using malloc() to check what's the limit of available memory that can be allocated. It is several megabytes, sometimes more. Unless that tool gets killed out of swap as well. So how i can see the issue: for some reason after some process had exited normally all mapped pages don't get freed. I red about and I agree that this is reasonable behaviour if we have spare memory. But following this logic these pages can be flushed back to file at any time when system is under stress conditions. So when I ask for a piece of RAM, OS should do that trick and give me what I ask. But that's never happens. Those pages are like frozen. Until I unmount disk. Even after there is not a single instance of indexer running. I believe all this is caused by mmap() for sure : BDB uses mmap() for accessing databases and we tested indexing with out pushing data to DB. Worked shiny. You may suggest that that's something wrong with BDB. But we have some more tools of ours that using mmap() as well and the behaviour is exact. Thank you. Paul, Ukraine.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?15861.1329298812.1414986334451204096>