From owner-freebsd-bugs@FreeBSD.ORG Sun Jun 15 09:20:05 2003 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 49FF637B401 for ; Sun, 15 Jun 2003 09:20:05 -0700 (PDT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id C2BD643FA3 for ; Sun, 15 Jun 2003 09:20:03 -0700 (PDT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.12.9/8.12.9) with ESMTP id h5FGK3Up084228 for ; Sun, 15 Jun 2003 09:20:03 -0700 (PDT) (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.12.9/8.12.9/Submit) id h5FGK3Id084227; Sun, 15 Jun 2003 09:20:03 -0700 (PDT) Resent-Date: Sun, 15 Jun 2003 09:20:03 -0700 (PDT) Resent-Message-Id: <200306151620.h5FGK3Id084227@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Jay Kuri Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8507E37B401 for ; Sun, 15 Jun 2003 09:17:19 -0700 (PDT) Received: from redux.oneway.com (redux.oneway.com [216.22.46.103]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9D69543F85 for ; Sun, 15 Jun 2003 09:17:18 -0700 (PDT) (envelope-from jay@redux.oneway.com) Received: from redux.oneway.com (localhost [127.0.0.1]) by redux.oneway.com (8.12.5/8.12.5) with ESMTP id h5FGH84m028890 for ; Sun, 15 Jun 2003 12:17:08 -0400 (EDT) Received: (from root@localhost) by redux.oneway.com (8.12.5/8.12.5/Submit) id h5FGH7T1028889 for FreeBSD-gnats-submit@freebsd.org; Sun, 15 Jun 2003 12:17:07 -0400 (EDT) Message-Id: <200306151558.h5FFwRCK041698@doppel.oneway.com> Date: Sun, 15 Jun 2003 11:58:27 -0400 (EDT) From: Jay Kuri To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Subject: kern/53350: fill up a malloc md-disk on 5.1-R causes panic X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Jay Kuri List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Jun 2003 16:20:05 -0000 >Number: 53350 >Category: kern >Synopsis: fill up a malloc md-disk on 5.1-R causes panic >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sun Jun 15 09:20:03 PDT 2003 >Closed-Date: >Last-Modified: >Originator: Jay Kuri >Release: FreeBSD 5.1-RELEASE i386 >Organization: oneway.com >Environment: System: FreeBSD 5.1-RELEASE i386 >Description: If you have a 5.1-R system with, say, 256M of ram and you mdconfig one or several malloc md-devices totalling around 100M, if you attempt to use them to capacity, the machine will panic. The message given is: panic: kmem_malloc(4096): kmem_map too small: 108347392 total allocated I have taken some time to track down how to repeat the problem at will... so perhaps someone can figure it out. The problem does not occur if you have more memory, or at least, you may have to use larger md drives to do it. I tried it on the exact same configuration, only with 4G of ram, and there is no panic. I have also tried it on several machines, with the same results. I would guess that it has something to do with caching/buffering of the copied files, but that's just a guess. I have attached a shell script that will trigger the error. I've run this several times, usually in single-user mode, and without fail on a 256M machine it will panic. set 'cache_dirs' to any set of directories that will total 100M or more. On my specific config I have a default 5.1R install with a minimal install+catman, the dirs listed in the script total around 110M. The script that got me started on this whole thing worked fine on 5.0-Current (Apr 30), but panic'd the machine as of 5.1R. (it's a different script than the one below, but I removed the non-essential parts of the script to make it less confusing) I should also mention that the kernel is identical to 'GENERIC' except that 'options POLLING' was included. Attached is a script that reliably causes the problem. Below are the results of running the script on a machine with 4G and on a machine with 256M RAM. - 4G log - # df -k Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/acd0 345120 345120 0 100% / devfs 1 1 0 100% /dev /dev/da0s1 253888 84416 169472 33% /opt # sh -x test.sh + PATH=/bin:/sbin:/usr/bin:/usr/sbin + cache_dirs=/bin /sbin /usr/bin /usr/sbin /usr/lib + sysctl+ awk {print int($2 / 1024) - 100000} hw.physmem + space_in_k=3928684 + echo Ram available to us: 3928684 Ram available to us: 3928684 + totaldirsize=0 + type=malloc + /sbin/mdconfig -a -t malloc -s 1M + mddev=md0 + /sbin/newfs /dev/md0 /dev/md0: 1.0MB (2048 sectors) block size 16384, fragment size 2048 using 2 cylinder groups of 0.50MB, 32 blks, 64 inodes. super-block backups (for fsck -b #) at: 160, 1184 + /sbin/mount /dev/md0 /mnt + mkdir /mnt/copy + /bin/cp /bin/cp /mnt/ + /usr/bin/du+ /usr/bin/awk {print int(1024 + ($1 * 1.20))} -sk /bin + dirsize=12050 + /bin/expr 0 + 12050 + totaldirsize=12050 + [ 12050 -le 3928684 ] + echo /bin + dirs_mirrored=/bin + /sbin/mdconfig -a -t malloc -s 12050k + mddevice=md1 + /sbin/newfs -i 1024 -U /dev/md1 + /sbin/tunefs -m 2 /dev/md1 tunefs: minimum percentage of free space changes from 8% to 2% tunefs: should optimize for space with minfree < 8% + /sbin/mount /dev/md1 /mnt/copy + /mnt/cp -Rp /bin/ /mnt/copy + /sbin/umount /mnt/copy + /bin/mkdir -p /mnt/bin + /sbin/mount /dev/md1 /mnt/bin + /usr/bin/du -sk /sbin + /usr/bin/awk {print int(1024 + ($1 * 1.20))} + dirsize=33913 + /bin/expr 12050 + 33913 + totaldirsize=45963 + [ 45963 -le 3928684 ] + echo /bin /sbin + dirs_mirrored=/bin /sbin + /sbin/mdconfig -a -t malloc -s 33913k + mddevice=md2 + /sbin/newfs -i 1024 -U /dev/md2 + /sbin/tunefs -m 2 /dev/md2 tunefs: minimum percentage of free space changes from 8% to 2% tunefs: should optimize for space with minfree < 8% + /sbin/mount /dev/md2 /mnt/copy + /mnt/cp -Rp /sbin/ /mnt/copy + /sbin/umount /mnt/copy + /bin/mkdir -p /mnt/sbin + /sbin/mount /dev/md2 /mnt/sbin + /usr/bin/du -sk /usr/bin + /usr/bin/awk {print int(1024 + ($1 * 1.20))} + dirsize=34375 + /bin/expr 45963 + 34375 + totaldirsize=80338 + [ 80338 -le 3928684 ] + echo /bin /sbin /usr/bin + dirs_mirrored=/bin /sbin /usr/bin + /sbin/mdconfig -a -t malloc -s 34375k + mddevice=md3 + /sbin/newfs -i 1024 -U /dev/md3 + /sbin/tunefs -m 2 /dev/md3 tunefs: minimum percentage of free space changes from 8% to 2% tunefs: should optimize for space with minfree < 8% + /sbin/mount /dev/md3 /mnt/copy + /mnt/cp -Rp /usr/bin/ /mnt/copy /mnt/copy: optimization changed from TIME to SPACE + /sbin/umount /mnt/copy + /bin/mkdir -p /mnt/usr/bin + /sbin/mount /dev/md3 /mnt/usr/bin + /usr/bin/du -sk /usr/sbin + /usr/bin/awk {print int(1024 + ($1 * 1.20))} + dirsize=10420 + /bin/expr 80338 + 10420 + totaldirsize=90758 + [ 90758 -le 3928684 ] + echo /bin /sbin /usr/bin /usr/sbin + dirs_mirrored=/bin /sbin /usr/bin /usr/sbin + /sbin/mdconfig -a -t malloc -s 10420k + mddevice=md4 + /sbin/newfs -i 1024 -U /dev/md4 + /sbin/tunefs -m 2 /dev/md4 tunefs: minimum percentage of free space changes from 8% to 2% tunefs: should optimize for space with minfree < 8% + /sbin/mount /dev/md4 /mnt/copy + /mnt/cp -Rp /usr/sbin/ /mnt/copy /mnt/copy: optimization changed from TIME to SPACE + /sbin/umount /mnt/copy + /bin/mkdir -p /mnt/usr/sbin + /sbin/mount /dev/md4 /mnt/usr/sbin + /usr/bin/du -sk /usr/lib + /usr/bin/awk {print int(1024 + ($1 * 1.20))} + dirsize=49560 + /bin/expr 90758 + 49560 + totaldirsize=140318 + [ 140318 -le 3928684 ] + echo /bin /sbin /usr/bin /usr/sbin /usr/lib + dirs_mirrored=/bin /sbin /usr/bin /usr/sbin /usr/lib + /sbin/mdconfig -a -t malloc -s 49560k + mddevice=md5 + /sbin/newfs -i 1024 -U /dev/md5 + /sbin/tunefs -m 2 /dev/md5 tunefs: minimum percentage of free space changes from 8% to 2% tunefs: should optimize for space with minfree < 8% + /sbin/mount /dev/md5 /mnt/copy + /mnt/cp -Rp /usr/lib/ /mnt/copy + /sbin/umount /mnt/copy + /bin/mkdir -p /mnt/usr/lib + /sbin/mount /dev/md5 /mnt/usr/lib # df -k Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/acd0 345120 345120 0 100% / devfs 1 1 0 100% /dev /dev/da0s1 253888 84416 169472 33% /opt /dev/md0 846 110 670 14% /mnt /dev/md1 10304 9528 570 94% /mnt/bin /dev/md2 29414 28644 182 99% /mnt/sbin /dev/md3 29812 29088 128 100% /mnt/usr/bin /dev/md4 8866 8270 420 95% /mnt/usr/sbin /dev/md5 43142 43074 -794 102% /mnt/usr/lib # top last pid: 136; load averages: 0.01, 0.08, 0.06 up 0+00:05:06 15:40:18 2 processes: 1 running, 1 sleeping Mem: 56M Active, 17M Inact, 168M Wired, 42M Cache, 69M Buf, 3608M Free Swap: PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 44 root 8 0 896K 444K wait 0:00 0.00% 0.00% sh 136 root 96 0 2136K 1300K RUN 0:00 0.00% 0.00% top - test log 256M - # top last pid: 54; load averages: 0.35, 0.24, 0.10 up 0+00:02:05 16:17:12 2 processes: 1 running, 1 sleeping Mem: 1184K Active, 268K Inact, 8328K Wired, 3664K Buf, 235M Free Swap: PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 54 root 96 0 2136K 780K RUN 0:00 0.00% 0.00% top 48 root 8 0 900K 320K wait 0:00 0.00% 0.00% sh # df -k Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/acd0 345120 345120 0 100% / devfs 1 1 0 100% /dev /dev/da0s1 253888 84416 169472 33% /opt # sh -x test.sh + PATH=/bin:/sbin:/usr/bin:/usr/sbin + cache_dirs=/bin /sbin /usr/bin /usr/sbin /usr/lib + sysctl+ awk {print int($2 / 1024) - 100000} hw.physmem + space_in_k=154980 + echo Ram available to us: 154980 Ram available to us: 154980 + totaldirsize=0 + type=malloc + /sbin/mdconfig -a -t malloc -s 1M + mddev=md0 + /sbin/newfs /dev/md0 /dev/md0: 1.0MB (2048 sectors) block size 16384, fragment size 2048 using 2 cylinder groups of 0.50MB, 32 blks, 64 inodes. super-block backups (for fsck -b #) at: 160, 1184 + /sbin/mount /dev/md0 /mnt + mkdir /mnt/copy + /bin/cp /bin/cp /mnt/ + /usr/bin/du+ /usr/bin/awk {print int(1024 + ($1 * 1.20))} -sk /bin + dirsize=12050 + /bin/expr 0 + 12050 + totaldirsize=12050 + [ 12050 -le 154980 ] + echo /bin + dirs_mirrored=/bin + /sbin/mdconfig -a -t malloc -s 12050k + mddevice=md1 + /sbin/newfs -i 1024 -U /dev/md1 + /sbin/tunefs -m 2 /dev/md1 tunefs: minimum percentage of free space changes from 8% to 2% tunefs: should optimize for space with minfree < 8% + /sbin/mount /dev/md1 /mnt/copy + /mnt/cp -Rp /bin/ /mnt/copy + /sbin/umount /mnt/copy + /bin/mkdir -p /mnt/bin + /sbin/mount /dev/md1 /mnt/bin + /usr/bin/du -sk /sbin + /usr/bin/awk {print int(1024 + ($1 * 1.20))} + dirsize=33913 + /bin/expr 12050 + 33913 + totaldirsize=45963 + [ 45963 -le 154980 ] + echo /bin /sbin + dirs_mirrored=/bin /sbin + /sbin/mdconfig -a -t malloc -s 33913k + mddevice=md2 + /sbin/newfs -i 1024 -U /dev/md2 + /sbin/tunefs -m 2 /dev/md2 tunefs: minimum percentage of free space changes from 8% to 2% tunefs: should optimize for space with minfree < 8% + /sbin/mount /dev/md2 /mnt/copy + /mnt/cp -Rp /sbin/ /mnt/copy + /sbin/umount /mnt/copy + /bin/mkdir -p /mnt/sbin + /sbin/mount /dev/md2 /mnt/sbin + /usr/bin/du -sk /usr/bin + /usr/bin/awk {print int(1024 + ($1 * 1.20))} + dirsize=34375 + /bin/expr 45963 + 34375 + totaldirsize=80338 + [ 80338 -le 154980 ] + echo /bin /sbin /usr/bin + dirs_mirrored=/bin /sbin /usr/bin + /sbin/mdconfig -a -t malloc -s 34375k + mddevice=md3 + /sbin/newfs -i 1024 -U /dev/md3 + /sbin/tunefs -m 2 /dev/md3 tunefs: minimum percentage of free space changes from 8% to 2% tunefs: should optimize for space with minfree < 8% + /sbin/mount /dev/md3 /mnt/copy + /mnt/cp -Rp /usr/bin/ /mnt/copy /mnt/copy: optimization changed from TIME to SPACE + /sbin/umount /mnt/copy + /bin/mkdir -p /mnt/usr/bin + /sbin/mount /dev/md3 /mnt/usr/bin + /usr/bin/du -sk /usr/sbin + /usr/bin/awk {print int(1024 + ($1 * 1.20))} + dirsize=10420 + /bin/expr 80338 + 10420 + totaldirsize=90758 + [ 90758 -le 154980 ] + echo /bin /sbin /usr/bin /usr/sbin + dirs_mirrored=/bin /sbin /usr/bin /usr/sbin + /sbin/mdconfig -a -t malloc -s 10420k + mddevice=md4 + /sbin/newfs -i 1024 -U /dev/md4 + /sbin/tunefs -m 2 /dev/md4 tunefs: minimum percentage of free space changes from 8% to 2% tunefs: should optimize for space with minfree < 8% + /sbin/mount /dev/md4 /mnt/copy + /mnt/cp -Rp /usr/sbin/ /mnt/copy /mnt/copy: optimization changed from TIME to SPACE + /sbin/umount /mnt/copy + /bin/mkdir -p /mnt/usr/sbin + /sbin/mount /dev/md4 /mnt/usr/sbin + /usr/bin/du -sk /usr/lib + /usr/bin/awk {print int(1024 + ($1 * 1.20))} + dirsize=49560 + /bin/expr 90758 + 49560 + totaldirsize=140318 + [ 140318 -le 154980 ] + echo /bin /sbin /usr/bin /usr/sbin /usr/lib + dirs_mirrored=/bin /sbin /usr/bin /usr/sbin /usr/lib + /sbin/mdconfig -a -t malloc -s 49560k + mddevice=md5 + /sbin/newfs -i 1024 -U /dev/md5 + /sbin/tunefs -m 2 /dev/md5 tunefs: minimum percentage of free space changes from 8% to 2% tunefs: should optimize for space with minfree < 8% + /sbin/mount /dev/md5 /mnt/copy + /mnt/cp -Rp /usr/lib/ /mnt/copy /mnt/copy: optimization changed from TIME to SPACE panic: kmem_malloc(4096): kmem_map too small: 108347392 total allocated syncing disks, buffers remaining... Fatal trap 12: page fault while in kernel mode fault virtual address = 0x14 fault code = supervisor write, page not present instruction pointer = 0x8:0xc02d1ad9 stack pointer = 0x10:0xd24fb544 frame pointer = 0x10:0xd24fb578 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 137 (cp) trap number = 12 panic: page fault Uptime: 3m12s Terminate ACPI Automatic reboot in 15 seconds - press a key on the console to abort >How-To-Repeat: On a 5.1-R machine with 256M of ram, boot single-user and run this script (ensure that the size of the dirs in cache_dirs is 100M or more) #!/bin/sh ## quick script to show the mdconfig -t malloc + fill = panic bug on 5.1-R PATH=/bin:/sbin:/usr/bin:/usr/sbin # on a machine with 256M of ram, this is enough to crash it. 112M roughly on # my machine. Use any combination of directories that will push the limit # up over 100M cache_dirs="/bin /sbin /usr/bin /usr/sbin /usr/lib" # say we have 100M less than we actually have, to make sure we don't # try to use more memory than we have space_in_k=`sysctl hw.physmem|awk '{print int($2 / 1024) - 100000}'` echo Ram available to us: $space_in_k totaldirsize=0 type="malloc" ## This gives us a writable directory so that we can run this off of the 5.1 ## install or live cd. mddev=`/sbin/mdconfig -a -t malloc -s 1M` /sbin/newfs /dev/${mddev} /sbin/mount /dev/${mddev} /mnt mkdir /mnt/copy /bin/cp /bin/cp /mnt/ for dir in ${cache_dirs} do dirsize=`/usr/bin/du -sk ${dir}|/usr/bin/awk '{print int(1024 + ($1 * 1.20 ))}'` totaldirsize=`/bin/expr ${totaldirsize} + ${dirsize}` if [ ${totaldirsize} -le ${space_in_k} ]; then dirs_mirrored=`echo ${dirs_mirrored} $dir` mddevice=`/sbin/mdconfig -a -t ${type} -s ${dirsize}k` /sbin/newfs -i 1024 -U /dev/${mddevice} >/dev/null /sbin/tunefs -m 2 /dev/${mddevice} /sbin/mount /dev/${mddevice} /mnt/copy >/dev/null /mnt/cp -Rp ${dir}/ /mnt/copy /sbin/umount /mnt/copy >/dev/null /bin/mkdir -p /mnt${dir} /sbin/mount /dev/${mddevice} /mnt${dir} >/dev/null fi done >Fix: None that I know of. >Release-Note: >Audit-Trail: >Unformatted: