Date: Thu, 26 Mar 2015 20:46:46 +0300 From: Artem Kuchin <artem@artem.ru> To: freebsd-questions@freebsd.org Subject: rm -rf stalls server Message-ID: <55144606.7000003@artem.ru>
next in thread | raw e-mail | index | archive | help
Hello! I have a server with 100+ sites running nginx, apache,mysql The disk system is 2 TOSHIBA 3TB disk in geom mirror FS is UFS SU+J tunefs -p / tunefs: POSIX.1e ACLs: (-a) disabled tunefs: NFSv4 ACLs: (-N) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: soft update journaling: (-j) enabled tunefs: gjournal: (-J) disabled tunefs: trim: (-t) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 4096 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: space to hold for metadata blocks: (-k) 6408 tunefs: optimization preference: (-o) time tunefs: volume label: (-L) root Ram is 32GB load averages: 1.83, 1.77, 1.83 usually about 500 processes are started Everything run nice and smooth. Periodically i need to run fs intensive tasks, like tar for backup. That kills server after 5 minutes of running. I worked it around using pv limiting bandwidth to 10M. But, sometimes i need to delete huge amount of small files in different dirs. Yes, i just need to do it. So, i do rm -rf tree_root After i do it in 5 minutes process count goes over 1000, mysql is flooded with request it cannot complete. I can barely type anything. Top shown tons of processes in 'ufs' status. So, I ctrl-C rm command and kill -9 apache, otherwise it will take like 20 minutes to resume normal operations. Then i do /usr/bin/nice -n 20 rm -rf tree_root And it seems like rm -rf now is playing nice :) But, if load for web sites rises even for about 30% during this command server goes to stall again, 1000 process, hundereds of stuck mysql requests, tons of processes in ufs state. As i understand, freebsd does not have ionice So, any idea how to make this rm -rf work reaaaaaaly slow, not creating any load on hdd or metadata locks (i think it is actually fs meta data locks fault). Artem
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55144606.7000003>