Date: Fri, 26 Jul 2019 23:02:43 +0100 From: Paul Macdonald <paul@ifdnrg.com> To: freebsd-questions@freebsd.org Subject: Re: Help:: Listen queue overflow killing servers Message-ID: <e977271f-752d-6705-5d26-a4d1ae5667e8@ifdnrg.com> In-Reply-To: <6485e15869f8b205cf36811adaeed0e5.squirrel@webmail.harte-lyne.ca> References: <mailman.1039.1564172643.29249.freebsd-questions@freebsd.org> <6485e15869f8b205cf36811adaeed0e5.squirrel@webmail.harte-lyne.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
On 26/07/2019 22:18, James B. Byrne via freebsd-questions wrote: > >>>>>> On 7/26/19 4:58 AM, Paul Macdonald via freebsd-questions wrote: >>>>>>> Over the past few months i've seen several boxes (4 or 5) become >>>>>>> unresponsive as a result of a Listen queue overflow state. > Since upgrading our hosts to 12.0 we have experienced many 'lockouts' > of both bhyve vm guests and jails, all running on zfs. There are > known issues with Bhyve guests getting into a deadlock state waiting > on zio or encountering memory exhaustion in releases after > FreeBSD-11.1. This has multiple causes: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231117 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594 > > We have worked around this by restricting max ARC to the net of system > memory minus all vm allocations minus 4GB. We also noted that as the > capacity of zfs pools approached 80% the deadlock is encountered more > frequently. The ARC on the system that went down today was unbounded, and stupidly large (60GB) but we had 30GB free so i didn't see harm, although i've now restricted. (now: Max of 10GB, currently 4GB wired, 118GB free.) We're totally aware of keeping ZFS below 80% utilisation, we've seen major performance cliffs if thats reached in the past. In this case we're at 2% utilisation on a new (<1 month) 1TB NVMe on PCI-e No bhyve, just 1 jail. This was a high spec box under no significant load, but now the fifth box in a few months thats crumpled under listen queue overflows Paul -- ------------------------- Paul Macdonald IFDNRG Ltd Web and video hosting ------------------------- t: 0131 5548070 m: 07970339546 e: paul@ifdnrg.com w: http://www.ifdnrg.com ------------------------- IFDNRG 40 Maritime Street Edinburgh EH6 6SA ---------------------------------------------------- Virtual Servers from £50.00pm High specification Dedicated Servers from £150.00pm ----------------------------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e977271f-752d-6705-5d26-a4d1ae5667e8>