Date: Fri, 26 Jul 2019 16:01:10 +0300 (MSK) From: "Igor V. Ruzanov" <igorr@pochta-mx.canmos.ru> To: MJ <mafsys1234@gmail.com> Cc: freebsd-questions@freebsd.org Subject: Re: Help:: Listen queue overflow killing servers Message-ID: <alpine.BSF.2.00.1907261556250.14674@pochta.canmos.ru> In-Reply-To: <907bd63c-7487-0b01-b202-7c67b8c0d07f@gmail.com> References: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> <alpine.BSF.2.00.1907261525460.14674@pochta.canmos.ru> <907bd63c-7487-0b01-b202-7c67b8c0d07f@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
truss is usefull tool, i agree! And also, look at memory usage with top (for example) command. In ZFS context you just NEED to perform system tuning because of huge RAM usage by ZFS. And one more thing - tell vmstat -z and see how many failures for different system structures in command output. I gues you didn't tune the system to use it in highload conditions. On Fri, 26 Jul 2019, MJ wrote: |I'd also look at why it's disk bound. I don't know the ops knowledge of dtrace |but running truss on one of the stuck processes might given him a clue as to |why these processes are bottlenecking the system. | |You could probably up kern.ipc.soacceptqueue for some temporary relief but I'd |suggest that would just mask the problem. | |Cheers | |Mark | |On 26/07/2019 10:28 pm, Igor V. Ruzanov wrote: |> You can increase TCP queue at system level: |> sysctl kern.ipc.somaxconn=65535 (its maximum value). But often its not |> enough because of application you use. So dig into application settings to |> increase the queue of TCP sockets to be handled in your system. |> |> |> On Fri, 26 Jul 2019, Paul Macdonald via freebsd-questions wrote: |> |> | |> |Hi, |> | |> |Over the past few months i've seen several boxes (4 or 5) become |> unresponsive |> |as a result of a Listen queue overflow state. |> | |> |Processes stack up, none are killable, all these are within jails and |> neither |> |the jail can be stopped nor the server rebooted (without a power cycle). |> | |> |All are on ZFS and are std apache/php/mysql servers with nothing too exotic. |> | |> |All on 12.0-RELEASE, i've only started seeing these issues recently, but it |> |feels like more and more. |> | |> |/var/log/messages shows tyically; |> | |> | kernel: sonewconn: pcb 0xfffff813395e3d58: Listen queue overflow: 193 |> |already in queue awaiting acceptance (83 occurrences) |> | |> |netstat -Lan shows |> | |> |tcp4 193/0/128 x.x.x.x.443 |> |tcp4 193/0/128 x.x.x.x.80 |> | |> |connections cannot be killed with tcpdrop ( except ssh which can!) |> | |> |All processes seem to be in Disk State ( many many apache processes but |> others |> |getting stuck too) |> | |> |www 60089 0.0 0.1 196588 78328 - DJ 21:07 1:19.54 |> |/usr/local/sbin/httpd -DNOHTTPACCEPT |> |..<snoip> |> | |> |www 93713 0.0 0.0 183576 33164 - DJ 23:57 0:00.01 |> |/usr/local/sbin/httpd -DNOHTTPACCEPT |> | |> |but no zombies.. |> | |> |last pid: 24773; load averages: 0.00, 0.00, 0.00 up |> |52+11:41:09 11:48:02 |> |918 processes: 1 running, 917 sleeping |> |CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle |> |Mem: 107M Active, 3729M Inact, 93G Wired, 27G Free |> |ARC: 79G Total, 54G MFU, 23G MRU, 243M Anon, 710M Header, 1615M Other |> | 73G Compressed, 191G Uncompressed, 2.60:1 Ratio |> |Swap: 4096M Total, 4096M Free |> | |> | |> |I'd appreciate any advice as at present it looks like my only option is to |> hard |> |power cycle these |> | |> |many thanks |> | |> |Paul. |> | |> | |> | |> |-- |> |------------------------- |> |Paul Macdonald |> |IFDNRG Ltd |> |Web and video hosting |> |------------------------- |> |t: 0131 5548070 |> |m: 07970339546 |> |e: paul@ifdnrg.com |> |w: http://www.ifdnrg.com |> |------------------------- |> |IFDNRG |> |40 Maritime Street |> |Edinburgh |> |EH6 6SA |> |---------------------------------------------------- |> | |> |Virtual Servers from £50.00pm |> |High specification Dedicated Servers from £150.00pm |> | |> |---------------------------------------------------- |> | |> |_______________________________________________ |> |freebsd-questions@freebsd.org mailing list |> |https://lists.freebsd.org/mailman/listinfo/freebsd-questions |> |To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" |> | |> _______________________________________________ |> freebsd-questions@freebsd.org mailing list |> https://lists.freebsd.org/mailman/listinfo/freebsd-questions |> To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" |_______________________________________________ |freebsd-questions@freebsd.org mailing list |https://lists.freebsd.org/mailman/listinfo/freebsd-questions |To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.1907261556250.14674>