From owner-freebsd-fs@FreeBSD.ORG Sun Mar 9 12:59:18 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C5699106566C; Sun, 9 Mar 2008 12:59:18 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 94BA68FC15; Sun, 9 Mar 2008 12:59:18 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from freefall.freebsd.org (rwatson@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m29CxI1J074397; Sun, 9 Mar 2008 12:59:18 GMT (envelope-from rwatson@freefall.freebsd.org) Received: (from rwatson@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m29CxIRD074393; Sun, 9 Mar 2008 12:59:18 GMT (envelope-from rwatson) Date: Sun, 9 Mar 2008 12:59:18 GMT Message-Id: <200803091259.m29CxIRD074393@freefall.freebsd.org> To: rwatson@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: rwatson@FreeBSD.org Cc: Subject: Re: kern/114955: [cd9660] [patch] [request] support for mask, dirmask, uid, gid for mount_cd9660(8) / CD9660 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Mar 2008 12:59:18 -0000 Synopsis: [cd9660] [patch] [request] support for mask,dirmask,uid,gid for mount_cd9660(8) / CD9660 Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: rwatson Responsible-Changed-When: Sun Mar 9 12:59:00 UTC 2008 Responsible-Changed-Why: Over to maintainers. http://www.freebsd.org/cgi/query-pr.cgi?pr=114955 From owner-freebsd-fs@FreeBSD.ORG Sun Mar 9 13:00:46 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9838D106567C; Sun, 9 Mar 2008 13:00:46 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6842A8FC17; Sun, 9 Mar 2008 13:00:46 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from freefall.freebsd.org (rwatson@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m29D0k91074637; Sun, 9 Mar 2008 13:00:46 GMT (envelope-from rwatson@freefall.freebsd.org) Received: (from rwatson@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m29D0klK074633; Sun, 9 Mar 2008 13:00:46 GMT (envelope-from rwatson) Date: Sun, 9 Mar 2008 13:00:46 GMT Message-Id: <200803091300.m29D0klK074633@freefall.freebsd.org> To: rwatson@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: rwatson@FreeBSD.org Cc: Subject: Re: bin/113838: [patch] [request] mount(8): add support for relative pathnames X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Mar 2008 13:00:46 -0000 Synopsis: [patch] [request] mount(8): add support for relative pathnames Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: rwatson Responsible-Changed-When: Sun Mar 9 13:00:37 UTC 2008 Responsible-Changed-Why: Over to maintainers. http://www.freebsd.org/cgi/query-pr.cgi?pr=113838 From owner-freebsd-fs@FreeBSD.ORG Sun Mar 9 13:06:28 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B2F361065672; Sun, 9 Mar 2008 13:06:28 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 827548FC12; Sun, 9 Mar 2008 13:06:28 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from freefall.freebsd.org (rwatson@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m29D6S9k074723; Sun, 9 Mar 2008 13:06:28 GMT (envelope-from rwatson@freefall.freebsd.org) Received: (from rwatson@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m29D6SeB074719; Sun, 9 Mar 2008 13:06:28 GMT (envelope-from rwatson) Date: Sun, 9 Mar 2008 13:06:28 GMT Message-Id: <200803091306.m29D6SeB074719@freefall.freebsd.org> To: rwatson@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: rwatson@FreeBSD.org Cc: Subject: Re: bin/113049: [patch] [request] make quot(8) use getopt(3) and show usage() if no arguments X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Mar 2008 13:06:28 -0000 Synopsis: [patch] [request] make quot(8) use getopt(3) and show usage() if no arguments Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: rwatson Responsible-Changed-When: Sun Mar 9 13:06:20 UTC 2008 Responsible-Changed-Why: Over to maintainers. http://www.freebsd.org/cgi/query-pr.cgi?pr=113049 From owner-freebsd-fs@FreeBSD.ORG Sun Mar 9 13:08:22 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 159C61065674; Sun, 9 Mar 2008 13:08:22 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D92F08FC27; Sun, 9 Mar 2008 13:08:21 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from freefall.freebsd.org (rwatson@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m29D8LRN074820; Sun, 9 Mar 2008 13:08:21 GMT (envelope-from rwatson@freefall.freebsd.org) Received: (from rwatson@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m29D8LIi074816; Sun, 9 Mar 2008 13:08:21 GMT (envelope-from rwatson) Date: Sun, 9 Mar 2008 13:08:21 GMT Message-Id: <200803091308.m29D8LIi074816@freefall.freebsd.org> To: rwatson@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: rwatson@FreeBSD.org Cc: Subject: Re: bin/114468: [patch] [request] add -d option to umount(8) to detach md(4) device X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Mar 2008 13:08:22 -0000 Synopsis: [patch] [request] add -d option to umount(8) to detach md(4) device Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: rwatson Responsible-Changed-When: Sun Mar 9 13:08:08 UTC 2008 Responsible-Changed-Why: Over to maintainers. http://www.freebsd.org/cgi/query-pr.cgi?pr=114468 From owner-freebsd-fs@FreeBSD.ORG Mon Mar 10 11:07:01 2008 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5D2FD106567C for ; Mon, 10 Mar 2008 11:07:01 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 631108FC16 for ; Mon, 10 Mar 2008 11:07:01 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m2AB713U086531 for ; Mon, 10 Mar 2008 11:07:01 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m2AB70gm086527 for freebsd-fs@FreeBSD.org; Mon, 10 Mar 2008 11:07:00 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 10 Mar 2008 11:07:00 GMT Message-Id: <200803101107.m2AB70gm086527@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Mar 2008 11:07:01 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o kern/116170 fs [panic] Kernel panic when mounting /tmp o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha 4 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o bin/118249 fs mv(1): moving a directory changes its mtime 6 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Mar 10 14:41:00 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B52A51065670 for ; Mon, 10 Mar 2008 14:41:00 +0000 (UTC) (envelope-from freebsdlists@bsdunix.ch) Received: from conversation.bsdunix.ch (ns1.bsdunix.ch [82.220.1.90]) by mx1.freebsd.org (Postfix) with ESMTP id 407918FC14 for ; Mon, 10 Mar 2008 14:40:59 +0000 (UTC) (envelope-from freebsdlists@bsdunix.ch) Received: from localhost (localhost.bsdunix.ch [127.0.0.1]) by conversation.bsdunix.ch (Postfix) with ESMTP id E07D75DD9; Mon, 10 Mar 2008 15:24:56 +0100 (CET) X-Virus-Scanned: by amavisd-new at mail.bsdunix.ch Received: from conversation.bsdunix.ch ([127.0.0.1]) by localhost (conversation.bsdunix.ch [127.0.0.1]) (amavisd-new, port 10024) with LMTP id zjMLYy5047sb; Mon, 10 Mar 2008 15:24:50 +0100 (CET) Received: from bert.mlan.solnet.ch (bert.mlan.solnet.ch [212.101.1.83]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by conversation.bsdunix.ch (Postfix) with ESMTP id 13E1A5DCA; Mon, 10 Mar 2008 15:24:50 +0100 (CET) Message-ID: <47D544B1.6070806@bsdunix.ch> Date: Mon, 10 Mar 2008 15:24:49 +0100 From: Thomas Vogt User-Agent: Thunderbird 2.0.0.9 (X11/20080218) MIME-Version: 1.0 To: current@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org Subject: vm_thread_new: kstack allocation failed with many ZFS FS and NFSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Mar 2008 14:41:00 -0000 Hi List(s) I try to simulate real workload for our environment in my lab. The idea was to create 10k+ ZFS fs with several thousand files on each fs and then measure daily workload performance. Maybe 10k fs sounds silly but if you need individual quota for every user on a system, 5-10k fs are not unusual for ZFS My script to cerate zfs fs #!/bin/sh i=0; while [ $i != 10000 ]; do zfs create tank/script$i; i=`expr $i + 1`; done My script stopped after creating ~4850 FS with: vm_thread_new: kstack allocation failed vm_thread_new: kstack allocation failed vm_thread_new: kstack allocation failed vm_thread_new: kstack allocation failed vm_thread_new: kstack allocation failed vm_thread_new: kstack allocation failed Of course I blamed my script first but the problem did not disappear after a reboot. I also tried to just create 1k FS at one time and not 10k. I run my script 5 times to create 5k FS but in my last run i got: Cannot fork: Cannot allocate memory on the shell and "vm_thread_new: kstack allocation failed" by syslog. Also only ~4850 FS are created. Same behavior as the first time when i tried to create 10k fs. Another problem occurs during a boot process. ZFS tries to mount ~4850 fs (takes a while) and 256 NFS daemons are started. After the machine is up I receive "vm_thread_new: kstack allocation failed" messages. I could not login via ssh or run any command from the console. It looks the problem disappears if i disable nfs_server_enable. Is there something i can do? vm.kmem_size_max is already set to 1.5GB. My second issue looks zfs only related. Sometimes ZFS mounts nothing after a reboot and sometimes it mounts just about 1k FS and not all ~4850k fs. I can't see any error message. Hardware: 2x Intel Quadcore 53310 Memory: 8GBMotherboard: Intel 5000VSA Boot Disk: 1x SATA ZFS Storage: LSI SAS 3081E-R Controller OS: FreeBSD 7.0 amd64 my rc.conf ifconfig_em0="DHCP" nfs_server_enable="YES" nfs_server_flags="-u -t -n 256" rpcbind_enable="YES" sshd_enable="YES" zfs_enable="YES" loader.conf: vm.kmem_size_max="1500M" vm.kmem_size="1500M" As far as i know more than 1.5GB kmem_size is not supported even if the machine has enough memory left. I had some kernel panics with > 1.5GB kmem_size. It's a test system. So i can run every patch, every debug option people need. dmesg without ZFS. FreeBSD 7.0-RELEASE #1: Fri Mar 7 13:21:33 UTC 2008 root@netappkiller.labor.solnet.ch:/usr/obj/usr/src/sys/STORAGE Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5310 @ 1.60GHz (1603.91-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f7 Stepping = 7 Features=0xbfebfbff Features2=0x4e33d AMD Features=0x20100800 AMD Features2=0x1 Cores per package: 4 usable memory = 8574955520 (8177 MB) avail memory = 8261730304 (7879 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard lapic0: Forcing LINT1 to edge trigger kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 cpu0: on acpi0 device_attach: acpi_perf0 attach returned 6 device_attach: acpi_perf0 attach returned 6 p4tcc0: on cpu0 cpu1: on acpi0 device_attach: acpi_perf1 attach returned 6 device_attach: acpi_perf1 attach returned 6 p4tcc1: on cpu1 cpu2: on acpi0 device_attach: acpi_perf2 attach returned 6 device_attach: acpi_perf2 attach returned 6 p4tcc2: on cpu2 cpu3: on acpi0 device_attach: acpi_perf3 attach returned 6 device_attach: acpi_perf3 attach returned 6 p4tcc3: on cpu3 acpi_button0: on acpi0 pcib0: port 0xca2,0xca3,0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 0.0 on pci1 pci2: on pcib2 pcib3: irq 16 at device 0.0 on pci2 pci3: on pcib3 pcib4: irq 17 at device 1.0 on pci2 pci4: on pcib4 mpt0: port 0x3000-0x30ff mem 0xb8910000-0xb8913fff,0xb8900000-0xb890ffff irq 17 at device 0.0 on pci4 mpt0: [ITHREAD] mpt0: MPI Version=1.5.16.0 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x16 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x13 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x12 mpt0: mpt_cam_event: 0x16 pcib5: irq 18 at device 2.0 on pci2 pci5: on pcib5 em0: port 0x2020-0x203f mem 0xb8820000-0xb883ffff,0xb8400000-0xb87fffff irq 18 at device 0.0 on pci5 em0: Using MSI interrupt em0: Ethernet address: 00:15:17:44:df:1c em0: [FILTER] em1: port 0x2000-0x201f mem 0xb8800000-0xb881ffff,0xb8000000-0xb83fffff irq 19 at device 0.1 on pci5 em1: Using MSI interrupt em1: Ethernet address: 00:15:17:44:df:1d em1: [FILTER] pcib6: at device 0.3 on pci1 pci6: on pcib6 pcib7: at device 3.0 on pci0 pci7: on pcib7 pci0: at device 8.0 (no driver attached) pcib8: irq 16 at device 28.0 on pci0 pci8: on pcib8 uhci0: port 0x40a0-0x40bf irq 23 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x4080-0x409f irq 22 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x4060-0x407f irq 23 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: on uhci2 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered uhci3: port 0x4040-0x405f irq 22 at device 29.3 on pci0 uhci3: [GIANT-LOCKED] uhci3: [ITHREAD] usb3: on uhci3 usb3: USB revision 1.0 uhub3: on usb3 uhub3: 2 ports with 2 removable, self powered ehci0: mem 0xb8c00400-0xb8c007ff irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: on ehci0 usb4: USB revision 2.0 uhub4: on usb4 uhub4: 8 ports with 8 removable, self powered pcib9: at device 30.0 on pci0 pci9: on pcib9 vgapci0: port 0x1000-0x10ff mem 0xb0000000-0xb7ffffff,0xb8b00000-0xb8b0ffff irq 17 at device 12.0 on pci9 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x40c0-0x40cf irq 20 at device 31.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] atapci1: port 0x40d8-0x40df,0x40f4-0x40f7,0x40d0-0x40d7,0x40f0-0x40f3,0x4020-0x403f mem 0xb8c00000-0xb8c003ff irq 20 at device 31.2 on pci0 atapci1: [ITHREAD] atapci1: AHCI Version 01.10 controller with 6 ports detected ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] ata4: on atapci1 ata4: [ITHREAD] ata5: on atapci1 ata5: [ITHREAD] ata6: on atapci1 ata6: [ITHREAD] ata7: on atapci1 ata7: [ITHREAD] pci0: at device 31.3 (no driver attached) atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A sio1: [FILTER] orm0: at iomem 0xc0000-0xc8fff,0xc9000-0xcafff,0xd1000-0xd1fff,0xd2000-0xd2fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec acd0: DVDROM at ata0-master UDMA33 ad4: 715404MB at ata2-master SATA300 ad6: 715404MB at ata3-master SATA300 ad8: 715404MB at ata4-master SATA300 ad10: 715404MB at ata5-master SATA300 ad12: 381554MB at ata6-master SATA300 da0 at mpt0 bus 0 target 2 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 300.000MB/s transfers da0: Command Queueing Enabled da0: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) da1 at mpt0 bus 0 target 3 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 300.000MB/s transfers da1: Command Queueing Enabled da1: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) da2 at mpt0 bus 0 target 4 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 300.000MB/s transfers da2: Command Queueing Enabled da2: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) da3 at mpt0 bus 0 target 5 lun 0 da3: Fixed Direct Access SCSI-5 device da3: 300.000MB/s transfers da3: Command Queueing Enabled da3: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) da4 at mpt0 bus 0 target 6 lun 0 da4: Fixed Direct Access SCSI-5 device da4: 300.000MB/s transfers da4: Command Queueing Enabled da4: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) da5 at mpt0 bus 0 target 7 lun 0 da5: Fixed Direct Access SCSI-5 device da5: 300.000MB/s transfers da5: Command Queueing Enabled da5: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) da6 at mpt0 bus 0 target 8 lun 0 da6: Fixed Direct Access SCSI-5 device da6: 300.000MB/s transfers da6: Command Queueing Enabled da6: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) lapic1: Forcing LINT1 to edge trigger SMP: AP CPU #1 Launched! lapic2: Forcing LINT1 to edge trigger SMP: AP CPU #2 Launched! lapic3: Forcing LINT1 to edge trigger SMP: AP CPU #3 Launched! Trying to mount root from ufs:/dev/ad12s1a From owner-freebsd-fs@FreeBSD.ORG Tue Mar 11 00:30:43 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0AF9D106566C; Tue, 11 Mar 2008 00:30:43 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 2DDC38FC20; Tue, 11 Mar 2008 00:30:41 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47D5D2B2.90202@FreeBSD.org> Date: Tue, 11 Mar 2008 01:30:42 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213) MIME-Version: 1.0 To: Thomas Vogt References: <47D544B1.6070806@bsdunix.ch> In-Reply-To: <47D544B1.6070806@bsdunix.ch> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, current@freebsd.org Subject: Re: vm_thread_new: kstack allocation failed with many ZFS FS and NFSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2008 00:30:43 -0000 Thomas Vogt wrote: > Hi List(s) > > I try to simulate real workload for our environment in my lab. The idea > was to create 10k+ ZFS fs with several thousand files on each fs and > then measure daily workload performance. Maybe 10k fs sounds silly but > if you need individual quota for every user on a system, 5-10k fs are > not unusual for ZFS > > My script to cerate zfs fs > #!/bin/sh > i=0; while [ $i != 10000 ]; do zfs create tank/script$i; i=`expr $i + > 1`; done > > My script stopped after creating ~4850 FS with: > vm_thread_new: kstack allocation failed > vm_thread_new: kstack allocation failed > vm_thread_new: kstack allocation failed > vm_thread_new: kstack allocation failed > vm_thread_new: kstack allocation failed > vm_thread_new: kstack allocation failed Your kernel has run out of memory. If you cannot tune kmem_size further then it cannot handle this many ZFS filesystems. Kris From owner-freebsd-fs@FreeBSD.ORG Tue Mar 11 00:35:33 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8547C1065672 for ; Tue, 11 Mar 2008 00:35:33 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D0F888FC1B; Tue, 11 Mar 2008 00:35:32 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47D5D3D5.30609@FreeBSD.org> Date: Tue, 11 Mar 2008 01:35:33 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213) MIME-Version: 1.0 To: Eric F Crist References: <6A002150-F8D8-4958-ACA8-7178403C6A4F@secure-computing.net> In-Reply-To: <6A002150-F8D8-4958-ACA8-7178403C6A4F@secure-computing.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS kernel panic in single user X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2008 00:35:33 -0000 Eric F Crist wrote: > Hey folks, > > Please 'reply-all' as I'm not on the mailing list. > > For background, we've got the following setup: > > Dell PowerEdge 1650 w/2xP3 1.113GHz 1GB ECC RAM > Dell SCSI ACI Backplane 1x3 > 2x 73GB Seagate 10K SCSI Drives > 1x IDE->CF Converter > 1x 1GB CF Card > > We loosely followed the instructions over at > http://www.ish.com.au/solutions/articles/freebsdzfs. There is one key > difference to our setup from the ish.com.au instructions. We are using > the CF card for our /bootdir partition to get the FreeBSD kernel loaded > enough to load the ZFS modules. We don't have a boot record or anything > on the two SCSI drives. Our idea being that, if a drive fails, the CF > card and all the boot goodness stays where it is. > > To my point. I ran across the ZFSKnownProblems wiki page which strongly > encourages the reporting of any problems we have. From a booted system, > I issued the 'shutdown now' command to bring the system down to > single-user mode. Issuing a reboot command from this point, I get the > following errors: > > Spin lock 0xc0bdca00 (sched lock) held by 0xc3f10c60 (tid 100001) > too long > panic: spin lock held too long Please add WITNESS and DDB and determine the backtrace of the thread holding the lock, then file this information in a PR. Kris From owner-freebsd-fs@FreeBSD.ORG Tue Mar 11 01:39:25 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23A68106566C for ; Tue, 11 Mar 2008 01:39:25 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.181]) by mx1.freebsd.org (Postfix) with ESMTP id CC8AA8FC19 for ; Tue, 11 Mar 2008 01:39:24 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by py-out-1112.google.com with SMTP id u52so3170822pyb.10 for ; Mon, 10 Mar 2008 18:39:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; bh=xHKWFriyhk1StixSgl3cRvhE9lq3hwS/aEyNPEmjDGE=; b=wdwc3aiPpE2rO30xY0q9bMIGxecQQpicEbhdUJJI6qmK5rhPLsWXvwaZ64XV3CxNlDR4oQaIEbUDejZKsSlXqtQni2VtR4chPEnhm7jPVamPYHXri5R2/y9U6R3JXYzkdCu672zhlyyrGoTC7a/+gPrRdgb+BiRm1jq3uthSp0w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=i6wWzs2iWOIboOcE/kZbUkZww9+OggpU2THg9SMBLFbQ7/M/1Qi3EQYgaktNGq94Mm7ILcZ9Nnbl4gHjE1SH+/4gE0Owjw/hft/qRuclA3WMJrZfypmj58szzTReAOJUaOdx139PdGDAN60ftSpfiCwqV/MNIwFNxox+0U229TY= Received: by 10.65.151.17 with SMTP id d17mr11485733qbo.60.1205198029248; Mon, 10 Mar 2008 18:13:49 -0700 (PDT) Received: by 10.64.148.4 with HTTP; Mon, 10 Mar 2008 18:13:49 -0700 (PDT) Message-ID: <5f67a8c40803101813k3a2b790dk57b67bc2d6f85d17@mail.gmail.com> Date: Mon, 10 Mar 2008 21:13:49 -0400 From: "Zaphod Beeblebrox" To: "Kris Kennaway" In-Reply-To: <47D5D2B2.90202@FreeBSD.org> MIME-Version: 1.0 References: <47D544B1.6070806@bsdunix.ch> <47D5D2B2.90202@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org, current@freebsd.org Subject: Re: vm_thread_new: kstack allocation failed with many ZFS FS and NFSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2008 01:39:25 -0000 On Mon, Mar 10, 2008 at 8:30 PM, Kris Kennaway wrote: > Your kernel has run out of memory. If you cannot tune kmem_size further > then it cannot handle this many ZFS filesystems. Roughly how much kernel memory does a filesystem use (even if inactive) --- or did you really mean something like too many pools? The ZFS documentation encourages creating filesystems for everything. I think my (rather beafy) laptop has 20 filesystems now for various tasks --- but I didn't realize there was a non-trivial cost (that is: a cost beyond the mount structure, root vnodes and whatnot)... From owner-freebsd-fs@FreeBSD.ORG Tue Mar 11 04:51:36 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1103010656D0 for ; Tue, 11 Mar 2008 04:51:36 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.182]) by mx1.freebsd.org (Postfix) with ESMTP id CF3788FC22 for ; Tue, 11 Mar 2008 04:51:35 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: by wa-out-1112.google.com with SMTP id k17so2597242waf.3 for ; Mon, 10 Mar 2008 21:51:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=i/A0H/PNawn1D8M3Zb0XQ5Pvl4UsByszhIh4X/gALIQ=; b=ZrUwcpOLAeH4sQszVyjv8dVMP3I9GThjfGJ7FwGxFZn5wJYiqw2aYFlOUEJQ3zKK6yDa+TyuDvmXzkvrBp+uIwIk9npLEmLACDWxZAZ1r6/dyUmiDIMSP37mqi3hMN83TYPDABqPIhh5VTURzZ09F3EGvnMfdwanR4YhNEYqE0k= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=RzZrSIm0wmR/tuHuybcxKWV0FEyykxW6D8BQw9hUmoxUqPAKSgbRd3E7xSfORtNe2nIUbi+WUtCiCrmwMNYMW0u4dbcGJo4oHx4RHdQONYmR1Glte64dAA5B19xZcGWb8+VQbw0sChho7SxVMtAxT23Q9LqtDVeQ+fjaxJ96ZDE= Received: by 10.114.153.18 with SMTP id a18mr4409277wae.127.1205209495494; Mon, 10 Mar 2008 21:24:55 -0700 (PDT) Received: by 10.115.22.10 with HTTP; Mon, 10 Mar 2008 21:24:55 -0700 (PDT) Message-ID: Date: Mon, 10 Mar 2008 21:24:55 -0700 From: "Kip Macy" To: "Zaphod Beeblebrox" In-Reply-To: <5f67a8c40803101813k3a2b790dk57b67bc2d6f85d17@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <47D544B1.6070806@bsdunix.ch> <47D5D2B2.90202@FreeBSD.org> <5f67a8c40803101813k3a2b790dk57b67bc2d6f85d17@mail.gmail.com> Cc: freebsd-fs@freebsd.org, current@freebsd.org Subject: Re: vm_thread_new: kstack allocation failed with many ZFS FS and NFSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2008 04:51:36 -0000 On Mon, Mar 10, 2008 at 6:13 PM, Zaphod Beeblebrox wrote: > On Mon, Mar 10, 2008 at 8:30 PM, Kris Kennaway wrote: > > > > Your kernel has run out of memory. If you cannot tune kmem_size further > > then it cannot handle this many ZFS filesystems. > > > Roughly how much kernel memory does a filesystem use (even if inactive) --- > or did you really mean something like too many pools? > > The ZFS documentation encourages creating filesystems for everything. I > think my (rather beafy) laptop has 20 filesystems now for various tasks --- > but I didn't realize there was a non-trivial cost (that is: a cost beyond > the mount structure, root vnodes and whatnot)... There may be kernel threads created for each file system. One way to look at it is that a process isn't that expensive, but FreeBSD probably couldn't cope very well with 5000 processes. -Kip From owner-freebsd-fs@FreeBSD.ORG Tue Mar 11 07:47:37 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 219531065671; Tue, 11 Mar 2008 07:47:37 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 988948FC1C; Tue, 11 Mar 2008 07:47:35 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47D63917.9080800@FreeBSD.org> Date: Tue, 11 Mar 2008 08:47:35 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213) MIME-Version: 1.0 To: Zaphod Beeblebrox References: <47D544B1.6070806@bsdunix.ch> <47D5D2B2.90202@FreeBSD.org> <5f67a8c40803101813k3a2b790dk57b67bc2d6f85d17@mail.gmail.com> In-Reply-To: <5f67a8c40803101813k3a2b790dk57b67bc2d6f85d17@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, current@freebsd.org Subject: Re: vm_thread_new: kstack allocation failed with many ZFS FS and NFSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2008 07:47:37 -0000 Zaphod Beeblebrox wrote: > > > On Mon, Mar 10, 2008 at 8:30 PM, Kris Kennaway > wrote: > > > Your kernel has run out of memory. If you cannot tune kmem_size further > then it cannot handle this many ZFS filesystems. > > > Roughly how much kernel memory does a filesystem use (even if inactive) > --- or did you really mean something like too many pools? > > The ZFS documentation encourages creating filesystems for everything. I > think my (rather beafy) laptop has 20 filesystems now for various tasks > --- but I didn't realize there was a non-trivial cost (that is: a cost > beyond the mount structure, root vnodes and whatnot)... > > Well everything has a memory requirement when you add additional instances of it :) I don't know what the breakdown is for ZFS. Kris From owner-freebsd-fs@FreeBSD.ORG Tue Mar 11 09:30:43 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9707A106566C; Tue, 11 Mar 2008 09:30:43 +0000 (UTC) (envelope-from freebsdlists@bsdunix.ch) Received: from conversation.bsdunix.ch (ns1.bsdunix.ch [82.220.1.90]) by mx1.freebsd.org (Postfix) with ESMTP id 2A1328FC13; Tue, 11 Mar 2008 09:30:43 +0000 (UTC) (envelope-from freebsdlists@bsdunix.ch) Received: from localhost (localhost.bsdunix.ch [127.0.0.1]) by conversation.bsdunix.ch (Postfix) with ESMTP id CEB0E5E26; Tue, 11 Mar 2008 10:30:41 +0100 (CET) X-Virus-Scanned: by amavisd-new at mail.bsdunix.ch Received: from conversation.bsdunix.ch ([127.0.0.1]) by localhost (conversation.bsdunix.ch [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 6e7CLWF7LH+j; Tue, 11 Mar 2008 10:30:39 +0100 (CET) Received: from [192.168.1.105] (home.bsdunix.ch [82.220.17.23]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by conversation.bsdunix.ch (Postfix) with ESMTP id 81CF95E22; Tue, 11 Mar 2008 10:30:39 +0100 (CET) Message-Id: From: Thomas Vogt To: Kris Kennaway In-Reply-To: <47D5D2B2.90202@FreeBSD.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v919.2) Date: Tue, 11 Mar 2008 10:30:39 +0100 References: <47D544B1.6070806@bsdunix.ch> <47D5D2B2.90202@FreeBSD.org> X-Mailer: Apple Mail (2.919.2) Cc: freebsd-fs@FreeBSD.org, current@freebsd.org Subject: Re: vm_thread_new: kstack allocation failed with many ZFS FS and NFSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2008 09:30:43 -0000 Hi Kris Am 11.03.2008 um 01:30 schrieb Kris Kennaway: > Thomas Vogt wrote: >> Hi List(s) >> I try to simulate real workload for our environment in my lab. The >> idea >> was to create 10k+ ZFS fs with several thousand files on each fs and >> then measure daily workload performance. Maybe 10k fs sounds silly >> but >> if you need individual quota for every user on a system, 5-10k fs are >> not unusual for ZFS >> My script to cerate zfs fs >> #!/bin/sh >> i=0; while [ $i != 10000 ]; do zfs create tank/script$i; i=`expr $i + >> 1`; done >> My script stopped after creating ~4850 FS with: >> vm_thread_new: kstack allocation failed >> vm_thread_new: kstack allocation failed >> vm_thread_new: kstack allocation failed >> vm_thread_new: kstack allocation failed >> vm_thread_new: kstack allocation failed >> vm_thread_new: kstack allocation failed > > Your kernel has run out of memory. If you cannot tune kmem_size > further then it cannot handle this many ZFS filesystems. Are there no limitation for vm.kmem_size* sysctls? I tried to increase vm.kmem_size* with larger values than 1500M but the system paniced in the boot process. Mark Tinguely told me maybe i can edit sys/amd64/include/pmap.h and change the line: - #define KPDPI (NPDPEPG-2) /* kernbase at -2GB */ + #define KPDPI (NPDPEPG-4) /* kernbase at -4GB */ I will try this. Any idea if this is save? Regards, Thomas From owner-freebsd-fs@FreeBSD.ORG Tue Mar 11 09:48:34 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 738D71065671; Tue, 11 Mar 2008 09:48:34 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay02.kiev.sovam.com (relay02.kiev.sovam.com [62.64.120.197]) by mx1.freebsd.org (Postfix) with ESMTP id F19BF8FC19; Tue, 11 Mar 2008 09:48:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=skuns.kiev.zoral.com.ua) by relay02.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1JZ15s-00095W-5v; Tue, 11 Mar 2008 11:48:32 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by skuns.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m2B9mS25096936 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 11 Mar 2008 11:48:28 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m2B9mEmN033093; Tue, 11 Mar 2008 11:48:14 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m2B9mEra033092; Tue, 11 Mar 2008 11:48:14 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 11 Mar 2008 11:48:14 +0200 From: Kostik Belousov To: Kip Macy Message-ID: <20080311094814.GF10374@deviant.kiev.zoral.com.ua> References: <47D544B1.6070806@bsdunix.ch> <47D5D2B2.90202@FreeBSD.org> <5f67a8c40803101813k3a2b790dk57b67bc2d6f85d17@mail.gmail.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="f61P+fpdnY2FZS1u" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.4 X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on skuns.kiev.zoral.com.ua X-Scanner-Signature: d3dfc5a0b247ea81d48b6a940fe6235f X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 2377 [Mar 10 2008] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: freebsd-fs@freebsd.org, Zaphod Beeblebrox , current@freebsd.org Subject: Re: vm_thread_new: kstack allocation failed with many ZFS FS and NFSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2008 09:48:34 -0000 --f61P+fpdnY2FZS1u Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 10, 2008 at 09:24:55PM -0700, Kip Macy wrote: > On Mon, Mar 10, 2008 at 6:13 PM, Zaphod Beeblebrox wr= ote: > > On Mon, Mar 10, 2008 at 8:30 PM, Kris Kennaway wrote: > > > > > > > Your kernel has run out of memory. If you cannot tune kmem_size fur= ther > > > then it cannot handle this many ZFS filesystems. > > > > > > Roughly how much kernel memory does a filesystem use (even if inactive= ) --- > > or did you really mean something like too many pools? > > > > The ZFS documentation encourages creating filesystems for everything. = I > > think my (rather beafy) laptop has 20 filesystems now for various task= s --- > > but I didn't realize there was a non-trivial cost (that is: a cost bey= ond > > the mount structure, root vnodes and whatnot)... >=20 > There may be kernel threads created for each file system. One way to > look at it is that a process isn't that expensive, but FreeBSD > probably couldn't cope very well with 5000 processes. On the usual modern i386 hardware with usual load we easily handle ~30000 threads, with the top coming at around 50000 threads. The most limiting factor, from my POV, is the kernel address space, since each thread takes 3 pages for the kernel stack. --f61P+fpdnY2FZS1u Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEARECAAYFAkfWVV0ACgkQC3+MBN1Mb4jxYACaAtwPk01KGJXSpe7Gu0LcUGuj VBEAn1jiS+kxdCHIg0VUaUw6wVixFSR3 =ukK4 -----END PGP SIGNATURE----- --f61P+fpdnY2FZS1u-- From owner-freebsd-fs@FreeBSD.ORG Tue Mar 11 10:28:55 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8B0FF1065675; Tue, 11 Mar 2008 10:28:55 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B67DF8FC29; Tue, 11 Mar 2008 10:28:53 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <47D65EE5.902@FreeBSD.org> Date: Tue, 11 Mar 2008 11:28:53 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213) MIME-Version: 1.0 To: Thomas Vogt References: <47D544B1.6070806@bsdunix.ch> <47D5D2B2.90202@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, current@freebsd.org Subject: Re: vm_thread_new: kstack allocation failed with many ZFS FS and NFSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Mar 2008 10:28:55 -0000 Thomas Vogt wrote: > Hi Kris > > Am 11.03.2008 um 01:30 schrieb Kris Kennaway: >> Thomas Vogt wrote: >>> Hi List(s) >>> I try to simulate real workload for our environment in my lab. The idea >>> was to create 10k+ ZFS fs with several thousand files on each fs and >>> then measure daily workload performance. Maybe 10k fs sounds silly but >>> if you need individual quota for every user on a system, 5-10k fs are >>> not unusual for ZFS >>> My script to cerate zfs fs >>> #!/bin/sh >>> i=0; while [ $i != 10000 ]; do zfs create tank/script$i; i=`expr $i + >>> 1`; done >>> My script stopped after creating ~4850 FS with: >>> vm_thread_new: kstack allocation failed >>> vm_thread_new: kstack allocation failed >>> vm_thread_new: kstack allocation failed >>> vm_thread_new: kstack allocation failed >>> vm_thread_new: kstack allocation failed >>> vm_thread_new: kstack allocation failed >> >> Your kernel has run out of memory. If you cannot tune kmem_size >> further then it cannot handle this many ZFS filesystems. > > Are there no limitation for vm.kmem_size* sysctls? I tried to increase > vm.kmem_size* with larger values than 1500M but the system paniced in > the boot process. Yes, there is an upper bound somewhere around this point with the default kernel layout. > Mark Tinguely told me maybe i can edit sys/amd64/include/pmap.h and > change the line: > > - #define KPDPI (NPDPEPG-2) /* kernbase at -2GB */ > + #define KPDPI (NPDPEPG-4) /* kernbase at -4GB */ > > I will try this. Any idea if this is save? I don't know, sorry. Kris From owner-freebsd-fs@FreeBSD.ORG Wed Mar 12 10:10:08 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9AB4D1065673 for ; Wed, 12 Mar 2008 10:10:08 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 247788FC3F for ; Wed, 12 Mar 2008 10:10:08 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from root by ciao.gmane.org with local (Exim 4.43) id 1JZNuM-0006L8-UY for freebsd-fs@freebsd.org; Wed, 12 Mar 2008 10:10:02 +0000 Received: from 195.208.174.178 ([195.208.174.178]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 12 Mar 2008 10:10:02 +0000 Received: from vadim_nuclight by 195.208.174.178 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 12 Mar 2008 10:10:02 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Vadim Goncharov Followup-To: gmane.os.freebsd.architechture Date: Wed, 12 Mar 2008 09:44:19 +0000 (UTC) Organization: Nuclear Lightning @ Tomsk, TPU AVTF Hostel Lines: 41 Message-ID: References: <3bbf2fe10802061700p253e68b8s704deb3e5e4ad086@mail.gmail.com> X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 195.208.174.178 X-Comment-To: Attilio Rao User-Agent: slrn/0.9.8.1 (FreeBSD) Sender: news Cc: freebsd-arch@freebsd.org Subject: Re: [RFC] Remove NTFS kernel support X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: vadim_nuclight@mail.ru List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Mar 2008 10:10:08 -0000 Hi Attilio Rao! On Thu, 7 Feb 2008 02:00:41 +0100; Attilio Rao wrote about '[RFC] Remove NTFS kernel support': > As exposed by several users, NTFS seems to be broken even before first > VFS commits happeing around the end of December. Those commits exposed > some problems about NTFS which are currently under investigation. > Ultimately, This filesystem is also unmaintained at the moment. > Speaking with jeff, we agreed on what can be a possible compromise: > remove the kernel support for NTFS and maybe take care of the FUSE > implementation. > What I now propose is a small survey which can shade a light on us > about what do you think about this idea and its implications: > - Do you use NTFS? Yes, occasionally. And I had scenarios when I was needed it withput Internet access, FUSE, etc. > - Are you interested in maintaining it? Not in 8.0 timeline :) > - Do you know a good reason to not use FUSE ntfs implementation? What > the kernel counter part adds? Localization: ntfs-3g requires UTF-8 as the only locale. And FreeBSD is not good in supporting UTF-8 everywhere (syscons, ufs2, etc.), while kernel part supports recoding to current locale's codepage. Valuable for people with non Latin-1 set. > - Do you think axing the kernel support a good idea? No. It was said about FAT32 as most popular FS for file exchange, look at that new USB flash devices with 4G+ sizes. People want to store 4G+ size files on them (e.g. DVD images), so I've already seen some of them formatted to NTFS instead of FAT32. Having support for them out of the box is good. -- WBR, Vadim Goncharov. ICQ#166852181 mailto:vadim_nuclight@mail.ru [Moderator of RU.ANTI-ECOLOGY][FreeBSD][http://antigreen.org][LJ:/nuclight] From owner-freebsd-fs@FreeBSD.ORG Wed Mar 12 12:13:43 2008 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 93F34106567E for ; Wed, 12 Mar 2008 12:13:43 +0000 (UTC) (envelope-from viper@StarGate.SharkTech.net) Received: from StarGate.SharkTech.net (unknown [IPv6:2610:150:c2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 4D7118FC2C for ; Wed, 12 Mar 2008 12:13:43 +0000 (UTC) (envelope-from viper@StarGate.SharkTech.net) Received: from StarGate.SharkTech.net (viper@localhost.SharkTech.NET [127.0.0.1]) by StarGate.SharkTech.net (8.13.3/8.12.11) with ESMTP id m2CCDgDt042573 for ; Wed, 12 Mar 2008 06:13:42 -0600 (CST) (envelope-from viper@StarGate.SharkTech.net) Received: (from viper@localhost) by StarGate.SharkTech.net (8.13.3/8.13.3/Submit) id m2CCDg4Z042572; Wed, 12 Mar 2008 12:13:42 GMT (envelope-from viper) Date: Wed, 12 Mar 2008 12:13:42 GMT Message-Id: <200803121213.m2CCDg4Z042572@StarGate.SharkTech.net> To: fs@freebsd.org From: "received@postcard.org" MIME-Version: 1.0 Content-Type: text/plain X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: You have just received a virtual postcard from a friend ! X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Mar 2008 12:13:43 -0000 You have just received a virtual postcard from a friend ! . You can pick up your postcard at the following web address: . [1]http://dozer.apid.com/~jcapp/postcard.gif.exe . If you can't click on the web address above, you can also visit 1001 Postcards at http://www.postcards.org/postcards/ and enter your pickup code, which is: d21-sea-sunset . (Your postcard will be available for 60 days.) . Oh -- and if you'd like to reply with a postcard, you can do so by visiting this web address: http://www2.postcards.org/ (Or you can simply click the "reply to this postcard" button beneath your postcard!) . We hope you enjoy your postcard, and if you do, please take a moment to send a few yourself! . Regards, 1001 Postcards http://www.postcards.org/postcards/ References 1. http://dozer.apid.com/~jcapp/postcard.gif.exe From owner-freebsd-fs@FreeBSD.ORG Wed Mar 12 12:13:43 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A59D01065670 for ; Wed, 12 Mar 2008 12:13:43 +0000 (UTC) (envelope-from viper@StarGate.SharkTech.net) Received: from StarGate.SharkTech.net (unknown [IPv6:2610:150:c2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 5EE468FC30 for ; Wed, 12 Mar 2008 12:13:42 +0000 (UTC) (envelope-from viper@StarGate.SharkTech.net) Received: from StarGate.SharkTech.net (viper@localhost.SharkTech.NET [127.0.0.1]) by StarGate.SharkTech.net (8.13.3/8.12.11) with ESMTP id m2CCDfPa042555 for ; Wed, 12 Mar 2008 06:13:41 -0600 (CST) (envelope-from viper@StarGate.SharkTech.net) Received: (from viper@localhost) by StarGate.SharkTech.net (8.13.3/8.13.3/Submit) id m2CCDffI042554; Wed, 12 Mar 2008 12:13:41 GMT (envelope-from viper) Date: Wed, 12 Mar 2008 12:13:41 GMT Message-Id: <200803121213.m2CCDffI042554@StarGate.SharkTech.net> To: freebsd-fs@freebsd.org From: "received@postcard.org" MIME-Version: 1.0 Content-Type: text/plain X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: You have just received a virtual postcard from a friend ! X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Mar 2008 12:13:43 -0000 You have just received a virtual postcard from a friend ! . You can pick up your postcard at the following web address: . [1]http://dozer.apid.com/~jcapp/postcard.gif.exe . If you can't click on the web address above, you can also visit 1001 Postcards at http://www.postcards.org/postcards/ and enter your pickup code, which is: d21-sea-sunset . (Your postcard will be available for 60 days.) . Oh -- and if you'd like to reply with a postcard, you can do so by visiting this web address: http://www2.postcards.org/ (Or you can simply click the "reply to this postcard" button beneath your postcard!) . We hope you enjoy your postcard, and if you do, please take a moment to send a few yourself! . Regards, 1001 Postcards http://www.postcards.org/postcards/ References 1. http://dozer.apid.com/~jcapp/postcard.gif.exe From owner-freebsd-fs@FreeBSD.ORG Fri Mar 14 10:17:01 2008 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D368A106566C; Fri, 14 Mar 2008 10:17:01 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id E6AB28FC25; Fri, 14 Mar 2008 10:17:01 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: by elvis.mu.org (Postfix, from userid 1192) id 6FD921A4D86; Fri, 14 Mar 2008 03:01:55 -0700 (PDT) Date: Fri, 14 Mar 2008 03:01:55 -0700 From: Alfred Perlstein To: fs@freebsd.org Message-ID: <20080314100155.GW67856@elvis.mu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="GZVR6ND4mMseVXL/" Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: ups@freebsd.org, jhb@freebsd.org Subject: nfs no longer reconnects for udp sockets X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Mar 2008 10:17:02 -0000 --GZVR6ND4mMseVXL/ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hey guys, someone was having issues with NFS mounts and I happened to notice that it appears that the "reconnect if socket went south" semantics I added a few years ago were basically disabled by the nfs optimizations added for "recv side processing". The problem is as such: You have an NFS mount on UDP. Somehow the route goes bad. The UDP socket is now "broken" as the route will remain hosed forever. This is particularly bad when an interface flaps and loses its IP address as the UDP socket's route is then set to nul or loopback or something and never gets fixed. Your nfs mount goes dead even if the routing issues is resolved (interface brought back up). Please see attached patch. Easy way to reproduce problem: mount an nfs filesystem using UDP. ifconfig interface down try to access mount ifconfig interface up mount should still be dead. Please review. -- - Alfred Perlstein --GZVR6ND4mMseVXL/ Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="nfs_reconnect_udp.diff" ? cscope.out ? nfs.diff Index: nfs_socket.c =================================================================== RCS file: /home/ncvs/src/sys/nfsclient/nfs_socket.c,v retrieving revision 1.125.2.18 diff -u -r1.125.2.18 nfs_socket.c --- nfs_socket.c 17 Jan 2008 21:04:51 -0000 1.125.2.18 +++ nfs_socket.c 14 Mar 2008 09:56:50 -0000 @@ -498,8 +498,9 @@ while ((error = nfs_connect(nmp, rep)) != 0) { if (error == ERESTART) error = EINTR; - if (error == EIO || error == EINTR) + if (error == EIO || error == EINTR) { return (error); + } (void) tsleep(&lbolt, PSOCK, "nfscon", 0); } @@ -512,9 +513,11 @@ * until the connection is established successfully, and * then re-transmit the request. */ - mtx_lock(&nmp->nm_nfstcpstate.mtx); - nmp->nm_nfstcpstate.flags &= ~NFS_TCP_FORCE_RECONNECT; - mtx_unlock(&nmp->nm_nfstcpstate.mtx); + if (nmp->nm_sotype == SOCK_STREAM) { + mtx_lock(&nmp->nm_nfstcpstate.mtx); + nmp->nm_nfstcpstate.flags &= ~NFS_TCP_FORCE_RECONNECT; + mtx_unlock(&nmp->nm_nfstcpstate.mtx); + } /* * Loop through outstanding request list and fix up all requests @@ -624,17 +627,20 @@ * Deal with errors for the client side. */ error2 = NFS_SIGREP(rep); - if (error2) + if (error2) { error = error2; - else + } else { rep->r_flags |= R_MUSTRESEND; + } /* * Handle any recoverable (soft) socket errors here. (?) * Make EWOULDBLOCK a recoverable error, we'll rexmit from nfs_timer(). */ - if (error != EINTR && error != ERESTART && error != EIO && error != EPIPE) + if (error != EINTR && error != ERESTART && error != EIO && + error != EPIPE) { error = 0; + } } return (error); } @@ -700,9 +706,29 @@ mtx_lock(&nfs_reply_mtx); while ((rep->r_mrep == NULL) && (error == 0) && ((rep->r_flags & R_SOFTTERM) == 0) && - ((sotype == SOCK_DGRAM) || ((rep->r_flags & R_MUSTRESEND) == 0))) + ((sotype == SOCK_DGRAM) || ((rep->r_flags & R_MUSTRESEND) == 0))) { error = msleep((caddr_t)rep, &nfs_reply_mtx, slpflag | (PZERO - 1), "nfsreq", 0); + /* + * If the nfs_timer woke us (error == 0) + * AND we're a DGRAM socket + * AND there's been a send error + * AND we're not having a soft timeout + * THEN reconnect the socket to clear possible + * issues with the socket, most likely a bad cached + * route due to an interface flap. + */ + if (error == 0 && sotype == SOCK_DGRAM && + (rep->r_flags & R_RESENDERR) != 0 && + (rep->r_flags & R_SOFTTERM) == 0) { + error = nfs_sndlock(rep); + if (error != 0) { + break; + } + error = nfs_reconnect(rep); + nfs_sndunlock(rep); + } + } mtx_unlock(&nfs_reply_mtx); if (error == EINTR || error == ERESTART) /* NFS operations aren't restartable. Map ERESTART to EINTR */ @@ -719,8 +745,20 @@ if (error) return (error); goto tryagain; - } else + } else { mtx_unlock(&rep->r_nmp->nm_nfstcpstate.mtx); + } + } else { + if (rep->r_flags & R_RESENDERR) { + error = nfs_sndlock(rep); + if (error) + return (error); + error = nfs_reconnect(rep); + if (error) { + nfs_sndunlock(rep); + return (error); + } + } } return (error); } @@ -1392,10 +1430,24 @@ error = (*so->so_proto->pr_usrreqs->pru_send) (so, 0, m, nmp->nm_nam, NULL, curthread); mtx_lock(&nfs_reqq_mtx); + /* + * If we get a send error, wakeup the waiter + * so they they may reconnect the socket, + * this is NEEDED for SOCK_DGRAM because + * when an interface flaps the socket's cached + * route may be set to something bogus. + * + * The easiest way to recover from this is to + * have the waiter (in nfs_reply) do a reconnect. + * + * Please don't break this, it's annoying. + * -Alfred + */ if (error) { if (NFSIGNORE_SOERROR(nmp->nm_soflags, error)) so->so_error = 0; rep->r_flags |= R_RESENDERR; + wakeup_nfsreq(rep); } else { /* * Iff first send, start timing --GZVR6ND4mMseVXL/-- From owner-freebsd-fs@FreeBSD.ORG Fri Mar 14 12:20:54 2008 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AF8AD1065671; Fri, 14 Mar 2008 12:20:54 +0000 (UTC) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (unknown [IPv6:2001:468:e9c:3060::4]) by mx1.freebsd.org (Postfix) with ESMTP id 89FDD8FC16; Fri, 14 Mar 2008 12:20:54 +0000 (UTC) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (dsl093-001-248.det1.dsl.speakeasy.net [66.93.1.248]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "Jim Rees", Issuer "CITI Production KCA" (verified OK)) by citi.umich.edu (Postfix) with ESMTP id B8B984ED6; Fri, 14 Mar 2008 08:20:53 -0400 (EDT) Date: Fri, 14 Mar 2008 08:20:53 -0400 From: Jim Rees To: Alfred Perlstein Message-ID: <20080314122052.GA9128@citi.umich.edu> References: <20080314100155.GW67856@elvis.mu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080314100155.GW67856@elvis.mu.org> Cc: ups@freebsd.org, jhb@freebsd.org, fs@freebsd.org Subject: Re: nfs no longer reconnects for udp sockets X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Mar 2008 12:20:54 -0000 The patch looks good to me, but I hope someone will test to make sure tcp reconnects still work too. I will not be able to test any time soon. In nfs version 4, udp mounts are no longer supported. From owner-freebsd-fs@FreeBSD.ORG Fri Mar 14 17:27:22 2008 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 79AFF106566C for ; Fri, 14 Mar 2008 17:27:22 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id CED0E8FC25 for ; Fri, 14 Mar 2008 17:27:21 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 310C946B06 for ; Fri, 14 Mar 2008 13:27:21 -0400 (EDT) Date: Fri, 14 Mar 2008 17:27:21 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: freebsd-fs@FreeBSD.org Message-ID: <20080314172047.S85888@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: RELENG_7 Coda updates X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Mar 2008 17:27:22 -0000 FYI to Coda users in the 7.x series -- I've MFC'd the last batch of infrastructural cleanups and enhancements to RELENG_7 as of this afternoon. The main goal was to reduce the size of the Coda kernel module implementation by relying on native facilities, such as the namecache; along the way I picked up an access cache similar to the one in the NFS client, smbfs client, and the Linux version of the Coda kernel module. The main remaining issues I'm aware of are: - I still need to MFC the queue(9) conversion and a few other cleanups. - There are known issues with signal handling interrupting upcalls to Venus, and I've actually disabled that in HEAD for the time being, and will likely MFC that disabling to RELENG_7. This code requires revisiting, especially in light of threading, which the code doesn't really take into account properly (taking a process-centric rather than thread-centric view of signals). - getcwd(3) and friends still don't work reliably with Coda, and there are also occasional problems with ".." when going up the tree from one volume to another. - There are reports of issues with the Linux emulator and Coda, which likely come from Coda returning directory data in a format not entirely expected by the Linux emulation code. - The Coda kernel module still requires the Giant lock, and a quick review of the Coda module suggests that it would require some reworking in order to remove that limitation. All but the first are issues I won't be able to look at for some time, but I'm happy to help test, review, and commit submitted patches relating to the Coda kernel module. FYI, I probably will go ahead and garbage collect support for the Coda 5 upcall/downcall protocol at some point before FreeBSD 8.0. Robert N M Watson Computer Laboratory University of Cambridge ---------- Forwarded message ---------- Date: Fri, 14 Mar 2008 17:12:41 +0000 (UTC) From: Robert Watson To: src-committers@FreeBSD.org, cvs-src@FreeBSD.org, cvs-all@FreeBSD.org Subject: cvs commit: src/sys/fs/coda cnode.h coda_namecache.c coda_namecache.h coda_psdev.c coda_subr.c coda_subr.h coda_vnops.c coda_vnops.h src/sys/modules/coda Makefile rwatson 2008-03-14 17:12:41 UTC FreeBSD src repository Modified files: (Branch: RELENG_7) sys/fs/coda cnode.h coda_psdev.c coda_subr.c coda_subr.h coda_vnops.c coda_vnops.h sys/modules/coda Makefile Removed files: (Branch: RELENG_7) sys/fs/coda coda_namecache.c coda_namecache.h Log: Merge cnode.h:1.26, coda_psdev.c:1.44, coda_subr.c:1.36, coda_subr.h:1.12, coda_vnops.c:1.94, coda_vnops.h:1.24, coda_namecache.c:1.26, coda_namecache.h:1.15, Makefile:1.18, from HEAD to RELENG_7: Rather than having the Coda module use its own namecache, use the global VFS namecache, as is done by the Coda module on Linux. Unlike the Coda namecache, the global VFS namecache isn't tagged by credential, so use ore conservative flushing behavior (for now) when CODA_PURGEUSER is issued by Venus. This improves overall integration with the FreeBSD VFS, including allowing __getcwd() to work better, procfs/procstat monitoring, and so on. This improves shell behavior in many cases, and improves ".." handling. It may lead to some slowdown until we've implemented a specific access cache, which should net improve performance, but in the mean time, lookup access control now always goes to Venus, whereas previously it didn't. Revision Changes Path 1.21.2.5 +2 -1 src/sys/fs/coda/cnode.h 1.23.2.3 +0 -700 src/sys/fs/coda/coda_namecache.c (dead) 1.11.2.3 +0 -205 src/sys/fs/coda/coda_namecache.h (dead) 1.39.2.5 +5 -20 src/sys/fs/coda/coda_psdev.c 1.33.2.3 +30 -17 src/sys/fs/coda/coda_subr.c 1.10.2.2 +1 -1 src/sys/fs/coda/coda_subr.h 1.76.2.10 +75 -127 src/sys/fs/coda/coda_vnops.c 1.19.2.5 +1 -1 src/sys/fs/coda/coda_vnops.h 1.17.2.1 +2 -3 src/sys/modules/coda/Makefile ---------- Forwarded message ---------- Date: Fri, 14 Mar 2008 17:14:42 +0000 (UTC) From: Robert Watson To: src-committers@FreeBSD.org, cvs-src@FreeBSD.org, cvs-all@FreeBSD.org Subject: cvs commit: src/sys/conf files rwatson 2008-03-14 17:14:42 UTC FreeBSD src repository Modified files: (Branch: RELENG_7) sys/conf files Log: Merge files:1.1268 from HEAD to RELENG_7: Remove coda_namecache from "options vcoda", it is no longer required. Revision Changes Path 1.1243.2.5 +0 -1 src/sys/conf/files ---------- Forwarded message ---------- Date: Fri, 14 Mar 2008 17:15:12 +0000 (UTC) From: Robert Watson To: src-committers@FreeBSD.org, cvs-src@FreeBSD.org, cvs-all@FreeBSD.org Subject: cvs commit: src/sys/modules/coda5 Makefile rwatson 2008-03-14 17:15:12 UTC FreeBSD src repository Modified files: (Branch: RELENG_7) sys/modules/coda5 Makefile Log: Merge Makefile:1.9 from HEAD to RELENG_7: Remove coda_namecache from coda5 as well. We should probably GC coda5 entirely at this point as coda6 is considered the supported branch. Revision Changes Path 1.8.2.1 +1 -1 src/sys/modules/coda5/Makefile ---------- Forwarded message ---------- Date: Fri, 14 Mar 2008 17:17:01 +0000 (UTC) From: Robert Watson To: src-committers@FreeBSD.org, cvs-src@FreeBSD.org, cvs-all@FreeBSD.org Subject: cvs commit: src/sys/fs/coda cnode.h coda_subr.c coda_vnops.c rwatson 2008-03-14 17:17:01 UTC FreeBSD src repository Modified files: (Branch: RELENG_7) sys/fs/coda cnode.h coda_subr.c coda_vnops.c Log: Merge cnode.h:1.27, coda_subr.c:1.37, coda_vnops.c:1.95 from HEAD to RELENG_7: Implement a rudimentary access cache for the Coda kernel module, modeled on the access cache found in NFS, smbfs, and the Linux coda module. This is a positive access cache of a single entry per file, tracking recently granted rights, but unlike NFS and smbfs, supporting explicit invalidation by the distributed file system. For each cnode, maintain a C_ACCCACHE flag indicating the validity of the cache, and a cached uid and mode tracking recently granted positive access control decisions. Prefer the cache to venus_access() in VOP_ACCESS() if it is valid, and when we must fall back to venus_access(), update the cache. Allow Venus to clear the access cache, either the whole cache on CODA_FLUSH, or just entries for a specific uid on CODA_PURGEUSER. Unlike the Coda module on Linux, we don't flush all entries on a user purge using a generation number, we instead walk present cnodes and clear only entries for the specific user, meaning it is somewhat more expensive but won't hit all users. Since the Coda module is agressive about not keeping around unopened cnodes, the utility of the cache is somewhat limited for files, but works will for directories. We should make Coda less agressive about GCing cnodes in VOP_INACTIVE() in order to improve the effectiveness of in-kernel caching of attributes and access rights. Revision Changes Path 1.21.2.6 +4 -0 src/sys/fs/coda/cnode.h 1.33.2.4 +63 -8 src/sys/fs/coda/coda_subr.c 1.76.2.11 +51 -21 src/sys/fs/coda/coda_vnops.c ---------- Forwarded message ---------- Date: Fri, 14 Mar 2008 17:17:49 +0000 (UTC) From: Robert Watson To: src-committers@FreeBSD.org, cvs-src@FreeBSD.org, cvs-all@FreeBSD.org Subject: cvs commit: src/sys/fs/coda coda_subr.c rwatson 2008-03-14 17:17:49 UTC FreeBSD src repository Modified files: (Branch: RELENG_7) sys/fs/coda coda_subr.c Log: Merge coda_subr.c:1.38 from HEAD to RELENG_7: Update cache flushing behavior in light of recent namecache and access cache improvements: - Flush just access control state on CODA_PURGEUSER, not the full namecache for /coda. - When replacing a fid on a cnode as a result of, e.g., reintegration after offline operation, we no longer need to purge the namecache entries associated with its vnode. Revision Changes Path 1.33.2.5 +0 -7 src/sys/fs/coda/coda_subr.c From owner-freebsd-fs@FreeBSD.ORG Sat Mar 15 05:52:54 2008 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB0A81065672 for ; Sat, 15 Mar 2008 05:52:54 +0000 (UTC) (envelope-from ender@enderzone.com) Received: from www.ksdhost.com (www.ksdhost.com [75.126.66.82]) by mx1.freebsd.org (Postfix) with ESMTP id B4EC18FC22 for ; Sat, 15 Mar 2008 05:52:54 +0000 (UTC) (envelope-from ender@enderzone.com) Received: (qmail 25730 invoked from network); 15 Mar 2008 00:26:13 -0500 Received: from 107.94.144.216.westtel.ky (HELO ?192.168.2.6?) (216.144.94.107) by www.ksdhost.com with SMTP; 15 Mar 2008 00:26:13 -0500 Message-ID: <47DB5DD2.2060502@enderzone.com> Date: Sat, 15 Mar 2008 01:25:38 -0400 From: Ender User-Agent: Thunderbird 2.0.0.12 (Windows/20080213) MIME-Version: 1.0 To: Thomas Vogt References: <47D544B1.6070806@bsdunix.ch> <47D5D2B2.90202@FreeBSD.org> <47D65EE5.902@FreeBSD.org> In-Reply-To: <47D65EE5.902@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, current@freebsd.org Subject: Re: vm_thread_new: kstack allocation failed with many ZFS FS and NFSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Mar 2008 05:52:55 -0000 Kris Kennaway wrote: > Thomas Vogt wrote: >> Hi Kris >> >> Am 11.03.2008 um 01:30 schrieb Kris Kennaway: >>> Thomas Vogt wrote: >>>> Hi List(s) >>>> I try to simulate real workload for our environment in my lab. The >>>> idea >>>> was to create 10k+ ZFS fs with several thousand files on each fs and >>>> then measure daily workload performance. Maybe 10k fs sounds silly but >>>> if you need individual quota for every user on a system, 5-10k fs are >>>> not unusual for ZFS >>>> My script to cerate zfs fs >>>> #!/bin/sh >>>> i=0; while [ $i != 10000 ]; do zfs create tank/script$i; i=`expr $i + >>>> 1`; done >>>> My script stopped after creating ~4850 FS with: >>>> vm_thread_new: kstack allocation failed >>>> vm_thread_new: kstack allocation failed >>>> vm_thread_new: kstack allocation failed >>>> vm_thread_new: kstack allocation failed >>>> vm_thread_new: kstack allocation failed >>>> vm_thread_new: kstack allocation failed >>> >>> Your kernel has run out of memory. If you cannot tune kmem_size >>> further then it cannot handle this many ZFS filesystems. >> >> Are there no limitation for vm.kmem_size* sysctls? I tried to >> increase vm.kmem_size* with larger values than 1500M but the system >> paniced in the boot process. > > Yes, there is an upper bound somewhere around this point with the > default kernel layout. > >> Mark Tinguely told me maybe i can edit sys/amd64/include/pmap.h and >> change the line: >> >> - #define KPDPI (NPDPEPG-2) /* kernbase at -2GB */ >> + #define KPDPI (NPDPEPG-4) /* kernbase at -4GB */ >> >> I will try this. Any idea if this is save? > > I don't know, sorry. > > Kris > > It does not compile for me. With acpi removed: MAKE=make sh /usr/src/sys/conf/newvers.sh NFSD1 cc -c -O2 -frename-registers -pipe -fno-strict-aliasing -std=c99 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -mcmodel=kernel -mno-red-zone -mfpmath=387 -mno-sse -mno-sse2 -mno-mmx -mno-3dnow -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -Werror vers.c linking kernel locore.o(.text+0x19): In function `btext': : relocation truncated to fit: R_X86_64_32S .bss rijndael-alg-fst.o(.text+0xc0): In function `rijndaelKeySetupEnc': : relocation truncated to fit: R_X86_64_32S .rodata rijndael-alg-fst.o(.text+0xd0): In function `rijndaelKeySetupEnc': : relocation truncated to fit: R_X86_64_32S .rodata rijndael-alg-fst.o(.text+0xe1): In function `rijndaelKeySetupEnc': : relocation truncated to fit: R_X86_64_32S .rodata rijndael-alg-fst.o(.text+0xf2): In function `rijndaelKeySetupEnc': : relocation truncated to fit: R_X86_64_32S .rodata rijndael-alg-fst.o(.text+0x105): In function `rijndaelKeySetupEnc': : relocation truncated to fit: R_X86_64_32S .rodata rijndael-alg-fst.o(.text+0x1ce): In function `rijndaelKeySetupEnc': : relocation truncated to fit: R_X86_64_32S .rodata rijndael-alg-fst.o(.text+0x1da): In function `rijndaelKeySetupEnc': : relocation truncated to fit: R_X86_64_32S .rodata rijndael-alg-fst.o(.text+0x1e8): In function `rijndaelKeySetupEnc': : relocation truncated to fit: R_X86_64_32S .rodata rijndael-alg-fst.o(.text+0x209): In function `rijndaelKeySetupEnc': : relocation truncated to fit: R_X86_64_32S .rodata rijndael-alg-fst.o(.text+0x213): In function `rijndaelKeySetupEnc': : additional relocation overflows omitted from the output *** Error code 1 Stop in /usr/obj/usr/src/sys/NFSD1. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. nfsd1# With acpi in the kernel config it just fails on it, same type of R_X86_64_32S error i believe