Date: Sat, 17 May 2014 08:45:29 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Damian Danielecki <danieleckid@gmail.com> Cc: freebsd-stable@freebsd.org Subject: Re: [9.3 PRE] filesystem full even if there are free descriptors and disk space Message-ID: <933053021.3570827.1400330729023.JavaMail.root@uoguelph.ca> In-Reply-To: <CANwN-8SGgWL-n13v0GPfMR%2BEeTtJ9-8qo4Ui26rcoJnFoKWnJA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Damian Danielecki wrote: > I am receiving errors like this on heavily loaded FreeBSD > 9.3-PRERELEASE nfs server: > May 17 05:01:02 nfsd kernel: pid 4173 (nfsd), uid 0 inumber 185391682 > on /exports: filesystem full > Do you have -maproot=root on the line for /exports in /etc/exports? If not, it might be running into a quota for "nobody", which is what "uid 0" is mapped to by default. rick > Filesystem is newly created and server newly installed with custom > minimalist kernel&world. I am experienced user. > There are many free inodes and much disk space: > > # df -i > Filesystem 1K-blocks Used Avail Capacity iused > ifree %iused Mounted on > /dev/mirror/gm0s1a 473048844 3449532 431755408 1% 306673 > 60848397 1% / > devfs 1 1 0 100% 0 > 0 100% /dev > /dev/da0p1 3783567612 1772595572 1708286632 51% 20964037 > 468035769 4% /exports > > # df -h > Filesystem Size Used Avail Capacity Mounted on > /dev/mirror/gm0s1a 451G 3.3G 411G 1% / > devfs 1.0k 1.0k 0B 100% /dev > /dev/da0p1 3.5T 1.7T 1.6T 51% /exports > > # uname -a > FreeBSD nfsd.xxx.pl 9.3-PRERELEASE FreeBSD 9.3-PRERELEASE #5: Fri May > 16 15:41:36 CEST 2014 > root@nfsd.xxx.pl:/usr/obj/usr/src/sys/FREEBSD9 amd64 > > Filesystem is clean and not fragmented. > For sure fsck has been done on unmounted filesystem. > I see these errors also after fsck. > > # fsck -t ufs -y /dev/da0p1 > ** /dev/da0p1 > ** Last Mounted on /exports > ** Phase 1 - Check Blocks and Sizes > ** Phase 2 - Check Pathnames > ** Phase 3 - Check Connectivity > ** Phase 4 - Check Reference Counts > ** Phase 5 - Check Cyl groups > 20964037 files, 443148893 used, 502743010 free (396842 frags, > 62793271 > blocks, 0.0% fragmentation) > > These are my filesystem params. > > # tunefs -p /exports > tunefs: POSIX.1e ACLs: (-a) disabled > tunefs: NFSv4 ACLs: (-N) disabled > tunefs: MAC multilabel: (-l) disabled > tunefs: soft updates: (-n) enabled > tunefs: soft update journaling: (-j) disabled > tunefs: gjournal: (-J) disabled > tunefs: trim: (-t) disabled > tunefs: maximum blocks per file in a cylinder group: (-e) 4096 > tunefs: average file size: (-f) 16384 > tunefs: average number of files in a directory: (-s) 64 > tunefs: minimum percentage of free space: (-m) 8% > tunefs: space to hold for metadata blocks: (-k) 6408 > tunefs: optimization preference: (-o) time > tunefs: volume label: (-L) > > My mouting flags are: > /dev/da0p1 on /exports (ufs, NFS exported, local, noatime, nosuid, > with quotas, soft-updates) > > I found file releated to sample inode, it's users's backup. > > File: "/exports/(...)/xxx.tar.gz" > Size: 53706752 FileType: Regular File > Mode: (0640/-rw-r-----) Uid: ( 6225/ (6225)) Gid: ( 1676/ > (1676)) > Device: 0,78 Inode: 185391682 Links: 1 > Access: Sat May 17 05:00:57 2014 > Modify: Sat May 17 05:01:02 2014 > Change: Sat May 17 05:01:48 2014 > > I am unable to decompress it, it's actually damaged: > # cp xxx.tar.gz /otherfilesystem/ > # cd /otherfilesystem && gzip -d xxx.tar.gz > gzip: xxx.tar.gz: unexpected end of file > gzip: xxx.tar.gz: uncompress failed > > I've tried to rerun this user's backup and I am able to reproduce > problem just now. The same backup file of the same user now has > different inode but problem is the same: > pid 4173 (nfsd), uid 0 inumber 185391652 on /exports: filesystem full > > # stat -x xxx.tar.gz > File: "xxx.tar.gz" > Size: 55902208 FileType: Regular File > Mode: (0640/-rw-r-----) Uid: ( 6225/ (6225)) Gid: ( 1676/ > (1676)) > Device: 0,78 Inode: 185391652 Links: 1 > Access: Sat May 17 12:51:47 2014 > Modify: Sat May 17 12:51:51 2014 > Change: Sat May 17 12:52:52 2014 > > Of course I am able to create any new big file from /dev/random, > there > is free space. I've easily created single 30GB file via nfs. > I've also tried to count md5 sum of newly generated 1GB file many > times via nfs and it's still the same. I checked this to be sure nfs > transmission is valid and da0 device is working properly. > > I've added some primitive debug to kernel sources but this is > production env so I will be able to reboot server only at night. Just > now it's impossible. > > /usr/src/sys/ufs # grep -R 'filesystem full' * > ffs/ffs_balloc.c: ffs_fserr(fs, > ip->i_number, "filesystem full line 320"); > ffs/ffs_balloc.c: ffs_fserr(fs, > ip->i_number, "filesystem full line 397"); > ffs/ffs_balloc.c: ffs_fserr(fs, > ip->i_number, "filesystem full line 882"); > ffs/ffs_balloc.c: ffs_fserr(fs, > ip->i_number, "filesystem full line 960"); > ffs/ffs_alloc.c: ffs_fserr(fs, ip->i_number, > "filesystem full line 227"); > ffs/ffs_alloc.c: ffs_fserr(fs, ip->i_number, > "filesystem full line 438"); > > Any help will be appreciated! Should be corrected before 9.3-RELEASE. > > DD > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?933053021.3570827.1400330729023.JavaMail.root>