From owner-freebsd-stable@FreeBSD.ORG Fri Sep 24 11:52:50 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E2F9E106566C for ; Fri, 24 Sep 2010 11:52:49 +0000 (UTC) (envelope-from leroy.vanlogchem@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 98A7B8FC19 for ; Fri, 24 Sep 2010 11:52:49 +0000 (UTC) Received: by qyk7 with SMTP id 7so851085qyk.13 for ; Fri, 24 Sep 2010 04:52:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=YOHAij+U/cMWCD/7RitVhC3K164K1U1se2WqXDX4NDA=; b=dR7AG4qQ+Qt09FcIztCM4m6CmjtoakxF6ipupOjbqY2iSu8zkNZh8kGTUaB79dW/fB IziuFXDttJXlOwLpBh6PI9PZ7sxX6FHZNv43kAE7qq2y+r0JQslgXRNQWtBbHR8Xy7Sx FY1j8m9jNoddy0txt91x/p5GRGqJRLxWB05HU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=S8frbi2j77qtf3S8+X0BoEfIufYUquNV8i07vrmWZ1ogQ3+t6POqYQnJYmUpI/Dsia pv5ezvEe61/6d96BhCum6uTfI1LpMwia+QP2uaUcpXFTuckJP8bh/ZpAfSZC+wZcvTzf 49N6+elhrMy81QpWNmNuHuiLWqAfp8DFztI2g= MIME-Version: 1.0 Received: by 10.224.60.136 with SMTP id p8mr2289317qah.386.1285327486138; Fri, 24 Sep 2010 04:24:46 -0700 (PDT) Received: by 10.229.2.25 with HTTP; Fri, 24 Sep 2010 04:24:46 -0700 (PDT) Date: Fri, 24 Sep 2010 13:24:46 +0200 Message-ID: From: Leroy van Logchem To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: 8.1-RELEASE: [zfs] [kmem] zfs destroy snapshot results in panic: kmem_map too small X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Sep 2010 11:52:50 -0000 ----------------------------------------------------------------------------- Problem : Kernel panic "kmem_malloc(114688): kmem_map too small Trigger : Destroy ZFS snapshots (each bigger >80GB) Version : FreeBSD 8.1-RELEASE (GENERIC AMD64 but with DDB) Workaround : kmem / zfs arc tuning in /boot/loader.conf System Info : Manufacturer: Supermicro Mainboard : X8DTN Cpu(s) Two : Intel(R) Xeon(R) E5520 @ 2.27GHz Memory : 10240 MB (speed: 1066 MHz) Storage : mpt0 Adapter: Board Name: SAS3081E Board Assembly: L3-00159-02D Chip Name: C1068E 24 drives are connected to mpt0: /dev/da[0 up to 23] Each drive : 512 # sectorsize 2000398934016 # mediasize in bytes (1.8T) 3907029168 # mediasize in sectors 243201 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. JK1130YAHUSJMT # Disk ident. # zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 43.5T 31.0T 12.5T 71% ONLINE - ----------------------------------------------------------------------------- This server has been running since dec 2009 without downtime. We keep 14 daily zfs snapshots. The problem of our nightly kernel panic started when the size of each snapshot exceeded about 80 GB. A cronjob first creates new snapshots just fine and destroys expired (older >14 days) snapshots. During the command "zfs destroy " the memory free seen in top(1) drops very quickly and soon after that this panic is generated: panic: kmem_malloc(114688): kmem_map too small: 3307884544 total allocated cpuid = 2 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x182 kmem_malloc() at kmem_malloc+0x5b5 uma_large_malloc() at uma_large_malloc+0x4a malloc() at malloc+0x14b zio_compress_data() at zio_compress_data+0xa2 zio_write_bp_init() at zio_write_bp_init+0xc2 zio_exectute() at zio_execute+0x77 taskq_run_safe() at taskq_run_safe+0x13 taskqueue_run() at taskqueue_run+0x91 taskqueue_thread_loop() at taskqueue_thread_loop+0x3f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81261edd30, rbp = 0 --- panic: kmem_malloc(118784): kmem_map too small: 3307884544 total allocated cpuid = 2 During these panics no tuning or /boot/loader.conf values where present. Only after hitting this behaviour yesterday i created boot/loader.conf: vm.kmem_size=18G vfs.zfs.arc_max=6G After which the snapshots could be destroyed without panic. Questions: Are users required to tune? If so, are there reliable guidelines regarding memory and ZFS? Are the -quickly- chosen values any good? (service window was closing) --- Leroy