Date: Fri, 24 Sep 2010 13:22:12 +0200 From: Leroy van Logchem <leroy.vanlogchem@gmail.com> To: freebsd-bugs@freebsd.org Subject: 8.1-RELEASE: [zfs] [kmem] zfs destroy snapshot results in panic: kmem_map too small Message-ID: <AANLkTi=w0S1mEEwBVVtsY5UoxY0DYU_qxJnaL76VuTKP@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
-----------------------------------------------------------------------------
Problem : Kernel panic "kmem_malloc(114688): kmem_map too small
Trigger : Destroy ZFS snapshots (each bigger >80GB)
Version : FreeBSD 8.1-RELEASE (GENERIC AMD64 but with DDB)
Workaround : kmem / zfs arc tuning in /boot/loader.conf
System Info : Manufacturer: Supermicro
Mainboard : X8DTN
Cpu(s) Two : Intel(R) Xeon(R) E5520 @ 2.27GHz
Memory : 10240 MB (speed: 1066 MHz)
Storage : mpt0 Adapter:
Board Name: SAS3081E
Board Assembly: L3-00159-02D
Chip Name: C1068E
24 drives are connected to mpt0: /dev/da[0 up to 23]
Each drive :
512 # sectorsize
2000398934016 # mediasize in bytes (1.8T)
3907029168 # mediasize in sectors
243201 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
JK1130YAHUSJMT # Disk ident.
# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
tank 43.5T 31.0T 12.5T 71% ONLINE -
-----------------------------------------------------------------------------
This server has been running since dec 2009 without downtime.
We keep 14 daily zfs snapshots. The problem of our nightly kernel panic started
when the size of each snapshot exceeded about 80 GB. A cronjob first creates
new snapshots just fine and destroys expired (older >14 days) snapshots.
During the command "zfs destroy <snapname>" the memory free seen in top(1)
drops very quickly and soon after that this panic is generated:
panic: kmem_malloc(114688): kmem_map too small: 3307884544 total allocated
cpuid = 2
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
panic() at panic+0x182
kmem_malloc() at kmem_malloc+0x5b5
uma_large_malloc() at uma_large_malloc+0x4a
malloc() at malloc+0x14b
zio_compress_data() at zio_compress_data+0xa2
zio_write_bp_init() at zio_write_bp_init+0xc2
zio_exectute() at zio_execute+0x77
taskq_run_safe() at taskq_run_safe+0x13
taskqueue_run() at taskqueue_run+0x91
taskqueue_thread_loop() at taskqueue_thread_loop+0x3f
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff81261edd30, rbp = 0 ---
panic: kmem_malloc(118784): kmem_map too small: 3307884544 total allocated
cpuid = 2
During these panics no tuning or /boot/loader.conf values where present.
Only after hitting this behaviour yesterday i created boot/loader.conf:
vm.kmem_size=18G
vfs.zfs.arc_max=6G
After which the snapshots could be destroyed without panic.
Questions:
Are users required to tune?
If so, are there reliable guidelines regarding memory and ZFS?
Are the -quickly- chosen values any good? (service window was closing)
---
Leroy
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTi=w0S1mEEwBVVtsY5UoxY0DYU_qxJnaL76VuTKP>