From owner-freebsd-stable@FreeBSD.ORG Wed Aug 5 05:38:15 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9B667106566B for ; Wed, 5 Aug 2009 05:38:15 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-bw0-f206.google.com (mail-bw0-f206.google.com [209.85.218.206]) by mx1.freebsd.org (Postfix) with ESMTP id 1650C8FC13 for ; Wed, 5 Aug 2009 05:38:14 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: by bwz2 with SMTP id 2so3294564bwz.43 for ; Tue, 04 Aug 2009 22:38:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=HUPCZhCLsBA5b5bEFxib8wp3MTzNPUF8SHBg1hSFk00=; b=ItezyJrJpEmjtuLwLCbsIQ//seE3hK47YPH18VQqyY2Od0FjK3UJ1ueQZP5ute1kY7 PjQwOo5ybdh7KajtIFh6Hafh/yW7KRS5KY7M2f84nHynacO2u1thXzS9xiuy2+MvbITi QbE3EYjNvOVMQbHuv5Q2lwCIyFwNdgZZZpQVg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; b=qufrKM9ZDaEa2Zri/36ft0Wplug5DxrrAzck7NPlauF1l/f2WvD2rkbK6VWkw2zSVN O7siiGXSlruBSKt5CNdER39HLoDjxJYobuWPo4zfJ4RhvRuxrPfZ7uQmOjSu0Vube0OG ayuJ2z5NxbrLmLUwAoxvvSEIcnDLBQ/kegwRY= MIME-Version: 1.0 Received: by 10.204.112.205 with SMTP id x13mr480643bkp.213.1249450693817; Tue, 04 Aug 2009 22:38:13 -0700 (PDT) Date: Wed, 5 Aug 2009 09:38:13 +0400 Message-ID: From: pluknet To: freebsd-stable Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: softdep_setup_freeblocks: kmem_malloc(4096): kmem_map too small X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Aug 2009 05:38:15 -0000 Hi. We have a problem with user running with exceed quota: Disk quotas for user eviluser (uid 9181): Filesystem usage quota limit grace files quota limit grace /home 6172656 6172672 6172672 14723 0 0 Some types of ufs operations running under him lead to kernel panic due to out of kernel memory (tested on 6.2-R, and 6.4-R): db> x/s *panicstr buf.1: kmem_malloc(4096): kmem_map too small: 335544320 total allocated Upping the higher level of vm.kmem_size_max doesn't help much, postponing that panic little farther. db> bt Tracing pid 7242 tid 100772 td 0xca7a57d0 kdb_enter(c0924e28) at kdb_enter+0x2b panic(c093a575,1000,14000000,c17f7818,0,...) at panic+0x127 kmem_malloc(c10680c0,1000,402,ef34e7bc,c07fb86d,...) at kmem_malloc+0x7d page_alloc(c10613c0,1000,ef34e7af,402,0,...) at page_alloc+0x1a slab_zalloc(c10613c0,402,c1061480,c10613c0,da68220c,...) at slab_zalloc+0xdd uma_zone_slab(c10613c0,502) at uma_zone_slab+0xe8 uma_zalloc_bucket(c10613c0,502) at uma_zalloc_bucket+0x15c uma_zalloc_arg(c10613c0,0,502) at uma_zalloc_arg+0x292 malloc(b8,c09d4ba0,502,0,0,...) at malloc+0x46 softdep_setup_freeblocks(cc8fb18c,0,0,800,cc8fb18c,ffffffe0,ffffffff,0,0) at sof tdep_setup_freeblocks+0x48 ffs_truncate(c89e3990,0,0,800,c94f4300,...) at ffs_truncate+0x5cb ffs_write(ef34ebec) at ffs_write+0x603 VOP_WRITE_APV(c09d5960,ef34ebec) at VOP_WRITE_APV+0xce vn_write(caba7000,ef34ecbc,c94f4300,0,ca7a57d0) at vn_write+0x1ee dofilewrite(ca7a57d0,7,caba7000,ef34ecbc,ffffffff,...) at dofilewrite+0x77 kern_writev(ca7a57d0,7,ef34ecbc,7e99c3c,42f6e8,...) at kern_writev+0x3b write(ca7a57d0,ef34ed04) at write+0x45 syscall(3b,82b003b,bfbf003b,8851,82bb000,...) at syscall+0x2bf Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (4, FreeBSD ELF32, write), eip = 0x281ae32f, esp = 0xbfbfbbdc, ebp = 0xbfbfbbf8 --- Always, afaics, the source of panics is a process running apache: db> show proc 7242 Process 7242 (httpd) at 0xca7a3648: state: NORMAL uid: 9181 gids: 999, 999, 9181 parent: pid 3799 at 0xca468000 ABI: FreeBSD ELF32 arguments: /home/eviluser/etc/apache/bin/httpd threads: 1 100772 Run CPU 5 httpd The type of a ufs operations is always the same: a process of that user tries to write data to fs and falls back (due to exceed quotas) to ffs_truncate() where it panics. I couldn't reproduce it, that happens once per several days. freeblks malloc type looks a bit leaky and a cause of panic. db> show malloc Type Allocs Frees Used ... freeblks 3233513 2578006 655507 ... db> show lockedbufs buf at 0xdbd84c88 b_flags = 0x20000220 b_error = 0, b_bufsize = 16384, b_bcount = 16384, b_resid = 0 b_bufobj = (0xcc1bfa50), b_data = 0xdc855000, b_blkno = 1624763616 b_npages = 4, pages(OBJ, IDX, PA): (0xcdb2cce4, 0xc, 0xb638b000),(0xcdb2cce4, 0x d, 0x6fe2c000),(0xcdb2cce4, 0xe, 0xca8d000),(0xcdb2cce4, 0xf, 0xb216e000) lock type bufwait: EXCL (count 1) by thread 0xcb41e190 (pid 11437) buf at 0xdbd86398 b_flags = 0x20000000 b_error = 0, b_bufsize = 16384, b_bcount = 16384, b_resid = 0 b_bufobj = (0xc8165c70), b_data = 0xdc89d000, b_blkno = 545231232 b_npages = 4, pages(OBJ, IDX, PA): (0xc81695ac, 0x40ff230, 0x842dc000),(0xc81695 ac, 0x40ff231, 0xdfd000),(0xc81695ac, 0x40ff232, 0x3b3e000),(0xc81695ac, 0x40ff2 33, 0x201f000) db> show lockedvnods Locked vnodes 0xcc144550: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags () v_object 0xc9c6e000 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xd00e4af0 (pid 11041) ino 203063289, on dev aacd0s1g 0xcc1bf990: tag ufs, type VDIR usecount 1, writecount 0, refcount 6 mountedhere 0 flags () v_object 0xcdb2cce4 ref 0 pages 22 lock type ufs: EXCL (count 1) by thread 0xcb41e190 (pid 11437) with 1 pending ino 203063639, on dev aacd0s1g 0xcdd84dd0: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags () lock type ufs: EXCL (count 1) by thread 0xc8de8000 (pid 3612) ino 263490091, on dev aacd0s1g 0xc89e3990: tag ufs, type VREG usecount 1, writecount 1, refcount 3 mountedhere 0 flags () v_object 0xc98a1294 ref 0 pages 4 lock type ufs: EXCL (count 1) by thread 0xca7a57d0 (pid 7242) ino 140245401, on dev aacd0s1g -- wbr, pluknet