Date: Fri, 26 Jul 2019 23:24:50 +1000 From: MJ <mafsys1234@gmail.com> To: freebsd-questions@freebsd.org Subject: Re: Help:: Listen queue overflow killing servers Message-ID: <22f7262b-eda8-f9d1-8836-61bcea8e1c5f@gmail.com> In-Reply-To: <a4675162-4241-2145-3380-c12253032da1@ifdnrg.com> References: <3a62375a-432c-3533-a7bc-e5573c26fa9c@ifdnrg.com> <92866b76-5f11-2523-cc8f-0d92cc91a50e@bytecamp.net> <a4675162-4241-2145-3380-c12253032da1@ifdnrg.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 26/07/2019 11:08 pm, Paul Macdonald via freebsd-questions wrote: > > On 26/07/2019 13:52, Robert Schulze wrote: >> Hi, >> >> Am 26.07.19 um 13:58 schrieb Paul Macdonald via freebsd-questions: >> I think, these processes waiting for disk i/o are actually your problem. >> Since they cannot answer further requests, they run into the listen >> queue overflow. >> >> You should check the processes with procstat: >> >> list kernel threads: >> # procstat -kk <PID> >> >> list open files: >> # procstat -f <PID> > > > One of the things we do (whihc may be bad) is to log to a single file > ( e.g all.sites.log, this doesn;t seem to cause problems in general , > but i can see how if there's X child processes then they may all need > write locks) > Unless it's hammering the log and there's locks on it. > Is that a really bad idea? ( Often handy to have one file for > differnet vhosts, but maybe that needs a rethink) Syslog works this way. > > In this case the drive is NVMe, and there's actually only a handful > of sites, (Other servers have several hundreds of sites, much busy but > don;t display the issue) > > In answer to some of the other suggestions, its not actually under > high load ( 5000 lines in the apache log for the whole day), and > system has 16C/32T, 128GB RAM > > ZFS is using a bunch of RAM as i've not limited the ARC, but there's > 27GB free currently. > > I guess i actually have 2 questions > > 1) why are the queues filling up (i'll revert to seperate logs to see > if that helps, although the issue is sporadic, and first time on this > box) > Most likely (given limited information) because the processes are disk bound doing something. They're not able to service the queue, thus leaving a growing queue for the system. > 2) Once the queues are over limit, is this actually unresolvable other > than a hard reboot? > No, you can keep running, you'll just get worse and worse... :-( You need to track down what the processes are doing, as per others' suggestions, and then work back from there. > > >> with kind regards, >> Robert Schulze >> _______________________________________________ >> freebsd-questions@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-questions >> To unsubscribe, send any mail to >> "freebsd-questions-unsubscribe@freebsd.org" >>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?22f7262b-eda8-f9d1-8836-61bcea8e1c5f>