Date: Tue, 3 Dec 2019 18:03:00 +0100 From: Peter Eriksson <pen@lysator.liu.se> To: FreeBSD FS <freebsd-fs@freebsd.org> Cc: Andriy Gapon <avg@FreeBSD.org>, Bengt Ahlgren <bengt.ahlgren@ri.se> Subject: Re: Slow reboots due to ZFS cleanup in kern_shutdown() .. zio_fini() Message-ID: <DD98024A-4DA5-45C1-953E-0C515A9F7DFB@lysator.liu.se> In-Reply-To: <uh75zixfiqw.fsf@P142.sics.se> References: <AD17E454-6A51-436D-A853-07F04A406EC9@lysator.liu.se> <D2A11CE9-9B24-4E40-A51A-8D318E0288C9@lysator.liu.se> <20191202225424.GG43802@raichu> <3b71fe37-c29f-e3e5-ff96-5dce15cc7553@FreeBSD.org> <DAD21732-AB7C-4B8D-99BF-25C7DD238A31@lysator.liu.se> <uh75zixfiqw.fsf@P142.sics.se>
next in thread | previous in thread | raw e-mail | index | archive | help
We run without swap on our servers=E2=80=A6 I=E2=80=99m trying to instrument the uma_zdestroy() code now, should = provide some more details in an hour or two=E2=80=A6 :-) - Peter > On 3 Dec 2019, at 17:59, Bengt Ahlgren <bengt.ahlgren@ri.se> wrote: >=20 > Peter Eriksson <pen@lysator.liu.se <mailto:pen@lysator.liu.se>> = writes: >=20 >>>> uma_destroy() frees all of the memory cached in the zone back to = the >>>> page allocator. This operation takes time proportional to the = number of >>>> cached items. I would expect most of the time to be spent in >>>> zone_reclaim(), called by zone_dtor(). >>>=20 >>> But spending *minutes* there is really unexpected. >>> I have never seen anything like that. >>> I wonder if there is anything untypical about the system's hardware >>> (like a very >>> big number of processors) or configuration. >>>=20 >>> --=20 >>> Andriy Gapon >>=20 >> It=E2=80=99s a fairly standard Dell PowerEdge R730xd server with = Intel Xeon >> E5-2620v4 CPUs & 256GB of RAM=E2=80=A6 (and an LSI SAS3 HBA and Intel = 10GE >> ethernet) >>=20 >> CPU: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (2100.04-MHz K8-class = CPU) >> Origin=3D"GenuineIntel" Id=3D0x406f1 Family=3D0x6 Model=3D0x4f = Stepping=3D1 >> = Features=3D0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE= ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> >> = Features2=3D0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SS= SE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSC= DLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> >> AMD Features=3D0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> >> AMD Features2=3D0x121<LAHF,ABM,Prefetch> >> Structured Extended >> = Features=3D0x21cbfbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,= RTM,PQM,NFPUSG,PQE,RDSEED,ADX,SMAP,PROCTRACE> >> Structured Extended = Features3=3D0x9c000400<MD_CLEAR,IBPB,STIBP,L1DFL,SSBD> >> XSAVE Features=3D0x1<XSAVEOPT> >> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr >> TSC: P-state invariant, performance statistics >> real memory =3D 274869518336 (262136 MB) >> avail memory =3D 267244859392 (254864 MB) >> Event timer "LAPIC" quality 600 >> ACPI APIC Table: <DELL PE_SC3 > >> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs >> FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 hardware threads >>=20 >> - Peter >=20 > I also have a Dell server, but older (R515, 128GB RAM, AMD 2x8 cores, > ~140TB disk, ARC cap 64GB) that has experienced the same behaviour. I > get the impression the shutdown time is related to used swap, but I = have > not investigated further. I think the system disk (with swap - > mechanical!) activity light is on during this time. Now running = 11.3-R, > but I'm sure I saw this with late 10.x as well. >=20 > Bengt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?DD98024A-4DA5-45C1-953E-0C515A9F7DFB>