Date: Fri, 08 Jul 2011 09:01:03 +0200 From: Adam PAPAI <wooh@wooh.hu> To: freebsd-questions@freebsd.org Subject: RELEASE-8.1 with heavy network activity (nginx + php-fpm) Message-ID: <4E16AB2F.6050109@wooh.hu>
next in thread | raw e-mail | index | archive | help
I have a FreeBSD box with 8.1. It seems we hit the limits for the maximum network connections, because I always see this message in the nginx-error.log: 2011/07/08 08:48:40 [error] 40438#0: *30564045 kevent() reported that connect() failed (54: Connection reset by peer) while connecting to upstream, client: 188.36.171.27, server: netadclick.com, request: "GET /hirdetes/148 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.netadclick.com", referrer: "http://www.netadclick.com/hirdetes/148" 2011/07/08 08:48:41 [error] 40438#0: *30564021 writev() failed (54: Connection reset by peer) while sending request to upstream, client: 95.171.73.72, server: netadclick.com, request: "GET /hirdetes/1051/0505 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.netadclick.com", referrer: "http://www.freespirit.hu/szex/durva_ezert_ne_hordj_tangat" The Box has approximately: 35,000 - 65,000 connections at the same time: In the early morning it has 45,564 connections: netstat -n | grep '^tcp4' | wc -l 45564 The system runs correctly for a few minutes, and after it's reaching "some" limits, all connections are reset (Connection reset by peer, the error log is filled with thousands of Connection reset by peer message) and it starts working correctly again for another few minutes. I've tuned some variables like: kern.ipc.shmall=32768 kern.ipc.shmmax=134217728 kern.ipc.semmap=256 net.inet.ip.maxfragpackets=1600 kern.threads.max_threads_per_proc=4096 kern.maxfiles=204800 kern.maxfilesperproc=200000 kern.maxvnodes=200000 net.inet.tcp.maxtcptw=200000 kern.ipc.nmbjumbo9=24000 kern.ipc.nmbjumbo16=10240 kern.ipc.maxsockets=65536 net.inet.tcp.fast_finwait2_recycle=1 net.inet.tcp.recvbuf_auto=0 net.inet.tcp.recvspace=65535 net.inet.ip.portrange.randomized=0 net.inet.ip.intr_queue_maxlen=4096 net.inet.tcp.finwait2_timeout=30000 kern.ipc.somaxconn=2048 /boot/loader.conf kern.ipc.semmni=1024 kern.ipc.semmns=2048 kern.ipc.semmnu=1024 kern.ipc.maxsockets=65536 kern.maxproc=12328 kern.ipc.shmmax=2147483648 kern.ipc.somaxconn=4096 net.inet.tcp.hostcache.cachelimit=65532 What should I do? Any ideas? Any tuning tips? Thanks in advance, -- Adam PAPAI
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4E16AB2F.6050109>