From owner-freebsd-stable@FreeBSD.ORG Wed Jun 8 16:06:07 2005 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 03FAD16A41C for ; Wed, 8 Jun 2005 16:06:07 +0000 (GMT) (envelope-from pda@crc.u-strasbg.fr) Received: from mailhost.u-strasbg.fr (mailhost.u-strasbg.fr [130.79.200.155]) by mx1.FreeBSD.org (Postfix) with ESMTP id 217D443D1D for ; Wed, 8 Jun 2005 16:06:05 +0000 (GMT) (envelope-from pda@crc.u-strasbg.fr) Received: from vagabond.ma.maison (vagabond.u-strasbg.fr [IPv6:2001:660:2402:1001:208:2ff:fedc:616a]) by mailhost.u-strasbg.fr (8.13.1/jtpda-5.5pre1) with ESMTP id j58G64YU095395 ; Wed, 8 Jun 2005 18:06:04 +0200 (CEST) Received: from vagabond.ma.maison (localhost [127.0.0.1]) by vagabond.ma.maison (8.13.3/jtpda-5.4) with ESMTP id j58G63i9010915 ; Wed, 8 Jun 2005 18:06:03 +0200 (CEST) Received: (from pda@localhost) by vagabond.ma.maison (8.13.3/8.13.3/Submit) id j58G63Cm010914; Wed, 8 Jun 2005 18:06:03 +0200 (CEST) (envelope-from pda) Date: Wed, 8 Jun 2005 18:06:03 +0200 From: Pierre DAVID To: Xin LI Message-ID: <20050608160603.GA10515@vagabond.ma.maison> References: <20050608123137.GA6872@vagabond.ma.maison> <20050608125239.GA45279@frontfree.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="s/l3CgOIzMHHjg/5" Content-Disposition: inline In-Reply-To: <20050608125239.GA45279@frontfree.net> User-Agent: Mutt/1.5.9i X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-1.6 (mailhost.u-strasbg.fr [IPv6:2001:660:2402::155]); Wed, 08 Jun 2005 18:06:04 +0200 (CEST) X-Antivirus: scanned by sophos at u-strasbg.fr Cc: freebsd-stable@freebsd.org Subject: Re: Show stopper for large disks with 5.4-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jun 2005 16:06:07 -0000 --s/l3CgOIzMHHjg/5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jun 08, 2005 at 08:52:39PM +0800, Xin LI wrote: >=20 > Would you please provide a bit more of information so we can investigate > what was happening, like: >=20 > - first few lines dumpfs(8) output from your storage filesystem > - df -i on your storag filesystem > - dmesg.boot from your /var/run >=20 > I think the delay is too long. Taking a snapshot on a large volume > is slow, but should not be that slow :-) We run similiar mail server > at company, but with ~10x of users and 2x storage (divided into two > RAID groups). Hope I would be able to provide some help. >=20 The server with the MSA 500 is not available at this time (there is another OS on it, sigh...). We reproduced the problem with the planned backup server, with ~2TB on a DAS storage (HP MSA 20). seth ~ # newfs -O 2 -U /dev/da1 ... seth ~ # mount /dev/da1 /mail seth ~ # df -g Filesystem 1G-blocks Used Avail Capacity Mounted on /dev/da0s1a 0 0 0 65% / devfs 0 0 0 100% /dev /dev/da0s1g 9 0 8 0% /local /dev/da0s1f 0 0 0 0% /tmp /dev/da0s1d 7 0 6 9% /usr /dev/da0s1e 9 0 8 0% /var /dev/da1 1983 1 1823 0% /mail And the problem is worse: seth ~ # time mksnap_ffs /mail /mail/.snap/snap1 real 69m18.083s user 0m0.000s sys 0m21.121s First lines of "dumpfs /dev/da1" are: magic 19540119 (UFS2) time Wed Jun 8 17:03:14 2005 superblock location 65536 id [ 42a6f77a a2574d5f ] ncg 11413 size 1073741823 blocks 1039959213 bsize 16384 shift 14 mask 0xffffc000 fsize 2048 shift 11 mask 0xfffff800 frag 8 shift 3 fsbtodb 2 minfree 8% optim time symlinklen 120 maxbsize 16384 maxbpg 2048 maxcontig 8 contigsumsize 8 nbfree 129917894 ndir 2 nifree 268798970 nffree 27 bpg 11761 fpg 94088 ipg 23552 nindir 2048 inopb 64 maxfilesize 140806241583103 sbsize 2048 cgsize 16384 csaddr 3000 cssize 184320 sblkno 40 cblkno 48 iblkno 56 dblkno 3000 cgrotor 8470 fmod 0 ronly 0 clean 0 avgfpdir 64 avgfilesize 16384 flags soft-updates=20 fsmnt /mail volname swuid 0 The complete "dumpfs /dev/da1" output is available on: ftp://ftp8.fr.freebsd.org/pub/tmp/dumpfs Here is the "df -i" output: seth mail # df -i Filesystem 1K-blocks Used Avail Capacity iused ifree %iused= =20 Mounted on /dev/da0s1a 1012974 607370 324568 65% 1399 139911 1% = / devfs 1 1 0 100% 0 0 100%= =20 /dev /dev/da0s1g 10063838 4142 9254590 0% 6 1318904 0%= =20 /local /dev/da0s1f 1012974 14 931924 0% 8 141302 0%= =20 /tmp /dev/da0s1d 8122126 682654 6789702 9% 116342 943496 11%= =20 /usr /dev/da0s1e 10154158 25732 9316094 0% 258 1318652 0%= =20 /var /dev/da1 2079918426 1232068 1912292884 0% 4 268798970 0%= =20 /mail The dmesg.boot is attached to this mail. Just in case, we also included the kernel configuration file to this mail. Thanks in advance for your help, Philippe Pegon & Pierre David --s/l3CgOIzMHHjg/5 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="dmesg.boot" Content-Transfer-Encoding: quoted-printable Copyright (c) 1992-2005 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.4-STABLE #2: Thu May 19 21:43:04 CEST 2005 root@crc.u-strasbg.fr:/usr/obj/usr/src/sys/SETH ACPI APIC Table: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2790.96-MHz 686-class CPU) Origin =3D "GenuineIntel" Id =3D 0xf29 Stepping =3D 9 Features=3D0xbfebfbff Hyperthreading: 2 logical CPUs real memory =3D 1073717248 (1023 MB) avail memory =3D 1041145856 (992 MB) MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-15 on motherboard ioapic1 irqs 16-31 on motherboard ioapic2 irqs 32-47 on motherboard ioapic3 irqs 48-63 on motherboard npx0: on motherboard npx0: INT 16 interface acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x920-0x923 on acpi0 cpu0: on acpi0 pcib0: on acpi0 pci0: on pcib0 pci0: at device 3.0 (no driver attached) pci0: at device 4.0 (no driver attached) pci0: at device 4.2 (no driver attached) isab0: at device 15.0 on pci0 isa0: on isab0 atapci0: port 0x2000-0x200f,0x376,0x1= 70-0x177,0x3f6,0x1f0-0x1f7 at device 15.1 on pci0 ata0: channel #0 on atapci0 ata1: channel #1 on atapci0 ohci0: mem 0xf5ef0000-0xf5ef0fff irq 7 at d= evice 15.2 on pci0 usb0: OHCI version 1.0, legacy support usb0: SMM does not respond, resetting usb0: on ohci0 usb0: USB revision 1.0 uhub0: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 4 ports with 4 removable, self powered pcib1: on acpi0 pci1: on pcib1 ciss0: port 0x3000-0x30ff mem 0xf7bf0000-0xf7bf3fff= ,0xf7cc0000-0xf7cfffff irq 30 at device 3.0 on pci1 pcib2: on acpi0 pci2: on pcib2 bge0: mem 0xf7df0000-= 0xf7dfffff irq 29 at device 1.0 on pci2 miibus0: on bge0 brgphy0: on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000b= aseTX-FDX, auto bge0: Ethernet address: 00:11:0a:9a:ce:7d bge1: mem 0xf7de0000-= 0xf7deffff irq 31 at device 2.0 on pci2 miibus1: on bge1 brgphy1: on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000b= aseTX-FDX, auto bge1: Ethernet address: 00:11:0a:9a:ce:7c pcib3: on acpi0 pci3: on pcib3 pcib4: on acpi0 pci6: on pcib4 ciss1: port 0x4000-0x40ff mem 0xf7f80000-0xf7fbffff,0x= f7ff0000-0xf7ff1fff irq 24 at device 1.0 on pci6 pci6: at device 30.0 (no driver = attached) acpi_tz0: on acpi0 atkbdc0: port 0x64,0x60 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 psm0: irq 12 on atkbdc0 psm0: model IntelliMouse, device ID 3 sio0: port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A fdc0: port 0x3f2-0x3f5 irq 6 drq 2 on acpi0 fd0: <1440-KB 3.5" drive> on fdc0 drive 0 orm0: at iomem 0xee000-0xeffff,0xd0000-0xd17ff,0xcc000-0x= cffff,0xc8000-0xcbfff,0xc0000-0xc7fff on isa0 pmtimer0 on isa0 ppc0: parallel port not found. sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=3D0x300> sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounter "TSC" frequency 2790962048 Hz quality 800 Timecounters tick every 1.000 msec acd0: CDROM at ata0-master PIO4 da1 at ciss1 bus 1 target 0 lun 0 da1: Fixed Direct Access SCSI-5 device=20 da1: 135.168MB/s transfers da1: 2097151MB (4294967295 512 byte sectors: 255H 32S/T 65535C) da2 at ciss1 bus 1 target 1 lun 0 da2: Fixed Direct Access SCSI-5 device=20 da2: 135.168MB/s transfers da2: 287277MB (588344688 512 byte sectors: 255H 32S/T 65535C) da0 at ciss0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-0 device=20 da0: 135.168MB/s transfers da0: 34727MB (71122560 512 byte sectors: 255H 32S/T 8716C) Mounting root from ufs:/dev/da0s1a --s/l3CgOIzMHHjg/5 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=SETH # # SETH # machine i386 #cpu I486_CPU #cpu I586_CPU cpu I686_CPU ident SETH # To statically compile in device wiring instead of /boot/device.hints #hints "GENERIC.hints" # Default places to look for devices. options SCHED_4BSD # 4BSD scheduler options INET # InterNETworking options INET6 # IPv6 communications protocols options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options MD_ROOT # MD is a potential root device options NFSCLIENT # Network Filesystem Client options NFSSERVER # Network Filesystem Server options NFS_ROOT # NFS usable as /, requires NFSCLIENT options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_GPT # GUID Partition Tables. options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!] options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options SCSI_DELAY=15000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options KBD_INSTALL_CDEV # install a CDEV entry in /dev options AHC_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~128k to driver. options AHD_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~215k to driver. options ADAPTIVE_GIANT # Giant mutex is adaptive. device apic # I/O APIC # Bus support. Do not remove isa, even if you have no isa slots device isa device eisa device pci # Floppy drives device fdc # ATA and ATAPI devices device ata device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives device atapifd # ATAPI floppy drives device atapist # ATAPI tape drives options ATA_STATIC_ID # Static device numbering # SCSI Controllers device ahb # EISA AHA1742 family device ahc # AHA2940 and onboard AIC7xxx devices device ahd # AHA39320/29320 and onboard AIC79xx devices device amd # AMD 53C974 (Tekram DC-390(T)) device isp # Qlogic family device mpt # LSI-Logic MPT-Fusion #device ncr # NCR/Symbios Logic device sym # NCR/Symbios Logic (newer chipsets + those of `ncr') device trm # Tekram DC395U/UW/F DC315U adapters device adv # Advansys SCSI adapters device adw # Advansys wide SCSI adapters device aha # Adaptec 154x SCSI adapters device aic # Adaptec 15[012]x SCSI adapters, AIC-6[23]60. device bt # Buslogic/Mylex MultiMaster SCSI adapters device ncv # NCR 53C500 device nsp # Workbit Ninja SCSI-3 device stg # TMC 18C30/18C50 # SCSI peripherals device scbus # SCSI bus (required for SCSI) device ch # SCSI media changers device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct SCSI access) device ses # SCSI Environmental Services (and SAF-TE) # RAID controllers interfaced to the SCSI subsystem device amr # AMI MegaRAID device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID device ciss # Compaq Smart RAID 5* device dpt # DPT Smartcache III, IV - See NOTES for options device hptmv # Highpoint RocketRAID 182x device iir # Intel Integrated RAID device ips # IBM (Adaptec) ServeRAID device mly # Mylex AcceleRAID/eXtremeRAID device twa # 3ware 9000 series PATA/SATA RAID # RAID controllers device aac # Adaptec FSA RAID device aacp # SCSI passthrough for aac (requires CAM) device ida # Compaq Smart RAID device mlx # Mylex DAC960 family device pst # Promise Supertrak SX6000 device twe # 3ware ATA RAID # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device vga # VGA video card driver device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc # Enable this for the pcvt (VT220 compatible) console driver #device vt #options XSERVER # support for X server on a vt console #options FAT_CURSOR # start with block cursor device agp # support several AGP chipsets # Floating point support - do not disable. device npx # Power management support (see NOTES for more options) #device apm # Add suspend/resume support for the i8254. device pmtimer # PCCARD (PCMCIA) support # PCMCIA and cardbus bridge support device cbb # cardbus (yenta) bridge device pccard # PC Card (16-bit) bus device cardbus # CardBus (32-bit) bus # Serial (COM) ports device sio # 8250, 16[45]50 based serial ports # Parallel port device ppc device ppbus # Parallel port bus (required) device lpt # Printer device plip # TCP/IP over parallel device ppi # Parallel port interface device #device vpo # Requires scbus and da # If you've got a "dumb" serial or parallel PCI card that is # supported by the puc(4) glue driver, uncomment the following # line to enable it (connects to the sio and/or ppc drivers): #device puc # PCI Ethernet NICs. device de # DEC/Intel DC21x4x (``Tulip'') device em # Intel PRO/1000 adapter Gigabit Ethernet Card device ixgb # Intel PRO/10GbE Ethernet Card device txp # 3Com 3cR990 (``Typhoon'') device vx # 3Com 3c590, 3c595 (``Vortex'') # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support device bfe # Broadcom BCM440x 10/100 Ethernet device bge # Broadcom BCM570xx Gigabit Ethernet device dc # DEC/Intel 21143 and various workalikes device fxp # Intel EtherExpress PRO/100B (82557, 82558) device lge # Level 1 LXT1001 gigabit ethernet device nge # NatSemi DP83820 gigabit ethernet device pcn # AMD Am79C97x PCI 10/100 (precedence over 'lnc') device re # RealTek 8139C+/8169/8169S/8110S device rl # RealTek 8129/8139 device sf # Adaptec AIC-6915 (``Starfire'') device sis # Silicon Integrated Systems SiS 900/SiS 7016 device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet device ste # Sundance ST201 (D-Link DFE-550TX) device ti # Alteon Networks Tigon I/II gigabit Ethernet device tl # Texas Instruments ThunderLAN device tx # SMC EtherPower II (83c170 ``EPIC'') device vge # VIA VT612x gigabit ethernet device vr # VIA Rhine, Rhine II device wb # Winbond W89C840F device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'') # ISA Ethernet NICs. pccard NICs included. device cs # Crystal Semiconductor CS89x0 NIC # 'device ed' requires 'device miibus' device ed # NE[12]000, SMC Ultra, 3c503, DS8390 cards device ex # Intel EtherExpress Pro/10 and Pro/10+ device ep # Etherlink III based cards device fe # Fujitsu MB8696x based cards device ie # EtherExpress 8/16, 3C507, StarLAN 10 etc. device lnc # NE2100, NE32-VL Lance Ethernet cards device sn # SMC's 9000 series of Ethernet chips device xe # Xircom pccard Ethernet # ISA devices that use the old ISA shims #device le # Wireless NIC cards device wlan # 802.11 support device an # Aironet 4500/4800 802.11 wireless NICs. device awi # BayStack 660 and others device wi # WaveLAN/Intersil/Symbol 802.11 wireless NICs. #device wl # Older non 802.11 Wavelan wireless NIC. # Pseudo devices. device loop # Network loopback device mem # Memory and kernel memory devices device io # I/O device device random # Entropy device device ether # Ethernet support device sl # Kernel SLIP device ppp # Kernel PPP device tun # Packet tunnel. device pty # Pseudo-ttys (telnet etc) device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! device bpf # Berkeley packet filter # USB support device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device usb # USB Bus (required) #device udbp # USB Double Bulk Pipe devices device ugen # Generic device uhid # "Human Interface Devices" device ukbd # Keyboard device ulpt # Printer device umass # Disks/Mass storage - Requires scbus and da device ums # Mouse device urio # Diamond Rio 500 MP3 player device uscanner # Scanners # USB Ethernet, requires mii device aue # ADMtek USB Ethernet device axe # ASIX Electronics USB Ethernet device cue # CATC USB Ethernet device kue # Kawasaki LSI USB Ethernet device rue # RealTek RTL8150 USB Ethernet # FireWire support device firewire # FireWire bus code device sbp # SCSI over FireWire (Requires scbus and da) device fwe # Ethernet over FireWire (non-standard!) # altq(9). Enable the base part of the hooks with the ALTQ option. # Individual disciplines must be built into the base system and can not be # loaded as modules at this point. In order to build a SMP kernel you must # also have the ALTQ_NOPCC option. options ALTQ options ALTQ_CBQ # Class Bases Queueing options ALTQ_RED # Random Early Drop options ALTQ_RIO # RED In/Out options ALTQ_HFSC # Hierarchical Packet Scheduler options ALTQ_CDNR # Traffic conditioner options ALTQ_PRIQ # Priority Queueing options ALTQ_NOPCC # Required for SMP build #options ALTQ_DEBUG # Change syscons colors options SC_KERNEL_CONS_ATTR="(FG_RED|BG_BLACK)" options SC_KERNEL_CONS_REV_ATTR="(FG_BLACK|BG_RED)" # ajout mano device vlan device pf device pflog device pfsync device carp options HZ=1000 # debug #options KDB # Enable kernel debugger support. #options DDB # Support DDB. #options GDB # Support remote GDB. #options INVARIANTS # Enable calls of extra sanity checking #options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS #options WITNESS # Enable checks to detect deadlocks and cycles #options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed --s/l3CgOIzMHHjg/5--