From owner-freebsd-current@FreeBSD.ORG Mon Dec 10 19:10:39 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 55E0516A417 for ; Mon, 10 Dec 2007 19:10:39 +0000 (UTC) (envelope-from oleg@lath.rinet.ru) Received: from lath.rinet.ru (lath.rinet.ru [195.54.192.90]) by mx1.freebsd.org (Postfix) with ESMTP id 3DC8913C46E for ; Mon, 10 Dec 2007 19:10:37 +0000 (UTC) (envelope-from oleg@lath.rinet.ru) Received: from lath.rinet.ru (localhost [127.0.0.1]) by lath.rinet.ru (8.14.2/8.14.1) with ESMTP id lBAIV07c032717 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 10 Dec 2007 21:31:00 +0300 (MSK) (envelope-from oleg@lath.rinet.ru) Received: (from oleg@localhost) by lath.rinet.ru (8.14.2/8.14.1/Submit) id lBAIV0lQ032716; Mon, 10 Dec 2007 21:31:00 +0300 (MSK) (envelope-from oleg) Date: Mon, 10 Dec 2007 21:31:00 +0300 From: Oleg Bulyzhin To: Thomas Vogt Message-ID: <20071210183100.GA32000@lath.rinet.ru> References: <475D5CF7.2020803@bsdunix.ch> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <475D5CF7.2020803@bsdunix.ch> User-Agent: Mutt/1.5.15 (2007-04-06) Cc: freebsd-current@freebsd.org Subject: Re: FreeBSD 7.0Beta4 (amd64) freezes with dummynet enabled X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Dec 2007 19:10:39 -0000 On Mon, Dec 10, 2007 at 04:36:23PM +0100, Thomas Vogt wrote: > Hi, > > I kindly ask for advice for debugging a problem with 7.0-Beta 4 (amd64). > > The machine freezes if dummynet rules are enabled on a gigabit port. The > system runs fine on fast ethernet or with dummynet rules disabled. I > can't trigger the bug if my upstream port is limited to 100mbit. The > machine suddenly freezes after 3-4 minutes if dummynet rules are loaded. > > My Kernel already includes some debug options but I'm not able to > produce a panic. > > Kernel: > options IPFIREWALL > options IPFIREWALL_VERBOSE > options IPFIREWALL_VERBOSE_LIMIT=100 > options IPFIREWALL_DEFAULT_TO_ACCEPT > options DUMMYNET > options ZERO_COPY_SOCKETS > options DEVICE_POLLING > options HZ=1000 > > options INVARIANTS > options INVARIANT_SUPPORT > options WITNESS > options DEBUG_VFS_LOCKS > options DIAGNOSTIC > options KDB > options KDB_UNATTENDED > options DDB > options GDB > options KDB_TRACE > options SOCKBUF_DEBUG > options BREAK_TO_DEBUGGER > > dmesg: > Copyright (c) 1992-2007 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 7.0-BETA4 #4: Fri Dec 7 12:45:37 UTC 2007 > root@lisa.mlan.solnet.ch:/usr/obj/usr/src/sys/SOLNETDEBUG > WARNING: WITNESS option enabled, expect reduced performance. > WARNING: DIAGNOSTIC option enabled, expect reduced performance. > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Xeon(R) CPU 5110 @ 1.60GHz (1739.55-MHz > K8-class CPU) > Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6 > > Features=0xbfebfbff > Features2=0x4e33d > AMD Features=0x20100800 > AMD Features2=0x1 > Cores per package: 2 > usable memory = 4258033664 (4060 MB) > avail memory = 4071751680 (3883 MB) > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > WITNESS: spin lock intrcnt not in order list > ioapic0 irqs 0-23 on motherboard > ioapic1 irqs 24-47 on motherboard > lapic0: Forcing LINT1 to edge trigger > kbd1 at kbdmux0 > acpi0: on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > acpi0: reservation of 0, a0000 (3) failed > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 > acpi_hpet0: iomem 0xfed00000-0xfed003ff on > acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 900 > cpu0: on acpi0 > device_attach: acpi_perf0 attach returned 6 > device_attach: acpi_perf0 attach returned 6 > p4tcc0: on cpu0 > cpu1: on acpi0 > device_attach: acpi_perf1 attach returned 6 > device_attach: acpi_perf1 attach returned 6 > p4tcc1: on cpu1 > acpi_button0: on acpi0 > acpi_button1: on acpi0 > pcib0: port 0xca2,0xca3,0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: at device 2.0 on pci0 > pci1: on pcib1 > pcib2: irq 16 at device 0.0 on pci1 > pci2: on pcib2 > pcib3: irq 16 at device 0.0 on pci2 > pci3: on pcib3 > pcib4: at device 0.0 on pci3 > pci4: on pcib4 > arcmsr0: > mem 0xc3d00000-0xc3d00fff,0xc3000000-0xc33fffff irq 18 at device 14.0 > on pci4 > ARECA RAID ADAPTER0: Driver Version 1.20.00.14 2007-2-05 > ARECA RAID ADAPTER0: FIRMWARE VERSION V1.43 2007-4-17 > arcmsr0: [ITHREAD] > pcib5: at device 0.2 on pci3 > pci5: on pcib5 > pcib6: irq 18 at device 2.0 on pci2 > pci6: on pcib6 > em0: port > 0x2020-0x203f mem 0xc3c20000-0xc3c3ffff,0xc3800000-0xc3bfffff irq 18 at > device 0.0 on pci6 > em0: Using MSI interrupt > em0: Ethernet address: 00:15:17:30:94:30 > em0: [FILTER] > em1: port > 0x2000-0x201f mem 0xc3c00000-0xc3c1ffff,0xc3400000-0xc37fffff irq 19 at > device 0.1 on pci6 > em1: Using MSI interrupt > em1: Ethernet address: 00:15:17:30:94:31 > em1: [FILTER] > pcib7: at device 0.3 on pci1 > pci7: on pcib7 > pcib8: at device 3.0 on pci0 > pci8: on pcib8 > pcib9: at device 4.0 on pci0 > pci9: on pcib9 > vgapci0: port 0x1000-0x107f mem > 0xc2000000-0xc2ffffff,0xb0000000-0xbfffffff,0xc0000000-0xc1ffffff irq 16 > at device 0.0 on pci9 > pcib10: at device 5.0 on pci0 > pci10: on pcib10 > pcib11: at device 6.0 on pci0 > pci11: on pcib11 > pcib12: at device 7.0 on pci0 > pci12: on pcib12 > pci0: at device 8.0 (no driver attached) > pci0: at device 27.0 (no driver attached) > pcib13: irq 16 at device 28.0 on pci0 > pci13: on pcib13 > uhci0: port > 0x3080-0x309f irq 23 at device 29.0 on pci0 > uhci0: [GIANT-LOCKED] > uhci0: [ITHREAD] > usb0: on uhci0 > usb0: USB revision 1.0 > uhub0: on usb0 > uhub0: 2 ports with 2 removable, self powered > uhci1: port > 0x3060-0x307f irq 22 at device 29.1 on pci0 > uhci1: [GIANT-LOCKED] > uhci1: [ITHREAD] > usb1: on uhci1 > usb1: USB revision 1.0 > uhub1: on usb1 > uhub1: 2 ports with 2 removable, self powered > uhci2: port > 0x3040-0x305f irq 23 at device 29.2 on pci0 > uhci2: [GIANT-LOCKED] > uhci2: [ITHREAD] > usb2: on uhci2 > usb2: USB revision 1.0 > uhub2: on usb2 > uhub2: 2 ports with 2 removable, self powered > uhci3: port > 0x3020-0x303f irq 22 at device 29.3 on pci0 > uhci3: [GIANT-LOCKED] > uhci3: [ITHREAD] > usb3: on uhci3 > usb3: USB revision 1.0 > uhub3: on usb3 > uhub3: 2 ports with 2 removable, self powered > ehci0: mem 0xc3f04400-0xc3f047ff irq > 23 at device 29.7 on pci0 > ehci0: [GIANT-LOCKED] > ehci0: [ITHREAD] > usb4: EHCI version 1.0 > usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 > usb4: on ehci0 > usb4: USB revision 2.0 > uhub4: on usb4 > uhub4: 8 ports with 8 removable, self powered > pcib14: at device 30.0 on pci0 > pci14: on pcib14 > isab0: at device 31.0 on pci0 > isa0: on isab0 > atapci0: port > 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30b0-0x30bf irq 20 at device 31.1 > on pci0 > ata0: on atapci0 > ata0: [ITHREAD] > ata1: on atapci0 > ata1: [ITHREAD] > atapci1: port > 0x30c8-0x30cf,0x30e4-0x30e7,0x30c0-0x30c7,0x30e0-0x30e3,0x30a0-0x30af > mem 0xc3f04000-0xc3f043ff irq 20 at device 31.2 on pci0 > atapci1: [ITHREAD] > ata2: on atapci1 > ata2: [ITHREAD] > ata3: on atapci1 > ata3: [ITHREAD] > pci0: at device 31.3 (no driver attached) > atkbdc0: port 0x60,0x64 irq 1 on acpi0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > atkbd0: [ITHREAD] > sio0: configured irq 4 not in bitmap of probed irqs 0 > sio0: port may not be enabled > sio0: configured irq 4 not in bitmap of probed irqs 0 > sio0: port may not be enabled > sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on > acpi0 > sio0: type 16550A, console > sio0: [FILTER] > sio1: configured irq 3 not in bitmap of probed irqs 0 > sio1: port may not be enabled > sio1: configured irq 3 not in bitmap of probed irqs 0 > sio1: port may not be enabled > sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 > sio1: type 16550A > sio1: [FILTER] > orm0: at iomem > 0xcd000-0xcdfff,0xce000-0xcefff,0xcf000-0xcffff on isa0 > ppc0: cannot reserve I/O port range > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > Timecounters tick every 1.000 msec > ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding > disabled, default to accept, logging limited to 100 packets/entry by default > Waiting 5 seconds for SCSI devices to settle > acd0: DVDROM at ata0-master UDMA33 > da0 at arcmsr0 bus 0 target 0 lun 0 > da0: Fixed Direct Access SCSI-5 device > da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da0: 238418MB (488281088 512 byte sectors: 255H 63S/T 30394C) > da1 at arcmsr0 bus 0 target 0 lun 1 > da1: Fixed Direct Access SCSI-5 device > da1: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da1: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) > da2 at arcmsr0 bus 0 target 0 lun 2 > da2: Fixed Direct Access SCSI-5 device > da2: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da2: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) > da3 at arcmsr0 bus 0 target 0 lun 3 > da3: Fixed Direct Access SCSI-5 device > da3: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da3: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) > da4 at arcmsr0 bus 0 target 0 lun 4 > da4: Fixed Direct Access SCSI-5 device > da4: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da4: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) > da5 at arcmsr0 bus 0 target 0 lun 5 > da5: Fixed Direct Access SCSI-5 device > da5: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da5: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) > da6 at arcmsr0 bus 0 target 0 lun 6 > da6: Fixed Direct Access SCSI-5 device > da6: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da6: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) > da7 at arcmsr0 bus 0 target 0 lun 7 > da7: Fixed Direct Access SCSI-5 device > da7: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da7: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) > lapic1: Forcing LINT1 to edge trigger > SMP: AP CPU #1 Launched! > WARNING: WITNESS option enabled, expect reduced performance. > WARNING: DIAGNOSTIC option enabled, expect reduced performance. > Trying to mount root from ufs:/dev/da0s1a > WARNING: ZFS is considered to be an experimental feature in FreeBSD. > ZFS filesystem version 6 > ZFS storage pool version 6 > em1: Hardware Initialization Failed > em1: Unable to initialize the hardware > Expensive timeout(9) function: 0xffffffff80258fe0(0xffffff00012fc000) > 0.008566172 s > em1: link state changed to UP > em1: link state changed to DOWN > em1: link state changed to UP > Expensive timeout(9) function: 0xffffffff80507ae0(0) 0.105003251 s > Expensive timeout(9) function: 0xffffffff80397260(0) 0.372032111 s > Expensive timeout(9) function: 0xffffffff80469350(0xffffff0069b5f888) > 0.403952336 s > > > my ipfw rules: > pipe 2 config bw 600Mbit/s queue 50KBytes > queue 201 config pipe 2 weight 1 mask dst-ip 0x0000000f queue 50 > queue 210 config pipe 2 weight 50 mask dst-ip 0x0000000f queue 50 > > > add 100 permit ip from any to any via lo0 > add 200 deny ip from any to 127.0.0.0/8 > > # Speedup Established > add 5001 skipto 8000 ip from 82.220.2.0/24 to 212.101.0.0/19 via em1 > add 6000 queue 210 ip from any to 212.101.0.0/19 out xmit em1 > add 6001 skipto 8000 ip from any to 212.101.0.0/19 out xmit em1 > add 6002 queue 210 ip from any to 212.41.65.0/18 out xmit em1 > add 6003 skipto 8000 ip from any to 212.41.65.0/18 out xmit em1 > add 6004 queue 210 ip from any to 82.220.0.0/18 out xmit em1 > add 6005 skipto 8000 ip from any to 82.220.0.0/18 out xmit em1 > add 6100 queue 201 ip from any to any out xmit em1 > > add 8000 permit tcp from any to any via em1 established > > # ICMP > add 11700 permit icmp from any to any via em1 icmptype 0,3,4,8,11,12 > > # Traceroute > add 1800 allow udp from any 30000-60000 to me 30000-60000 via em0 > > > # SSH (external connections are allowed for portsnap maintainer) > add 11900 allow ip from me to any ssh setup > add 11901 allow ip from any to me ssh setup > > # HTTP OUT > add 11910 permit tcp from me to any 80 setup via em1 > > # SMTP > add 12100 permit tcp from me to 212.101.0.0/19 25 setup via em1 > add 12110 allow tcp from me to any dst-port smtp via em1 setup > > # DNS UDP SolNet > add 21100 permit udp from 212.101.0.10 domain to me recv em1 > add 21101 permit udp from me to 212.101.0.10 domain out xmit em1 > add 21110 permit udp from 212.101.4.253 domain to me recv em1 > add 21111 permit udp from me to 212.101.4.253 domain out xmit em1 > add 21120 permit udp from 212.101.4.251 domain to me recv em1 > add 21121 permit udp from me to 212.101.4.251 domain out xmit em1 > > # NTP > add 21440 permit udp from me ntp to 212.101.0.0/19 ntp out xmit em1 > add 21445 permit udp from 224.0.1.1 ntp to 212.101.0.0/19 ntp out xmit em1 > add 21450 permit udp from 212.101.0.0/19 ntp to me ntp recv em1 > add 21460 permit udp from 212.101.0.0/19 ntp to 224.0.1.1 ntp recv em1 > add 21470 permit udp from me ntp to 224.0.1.1 ntp out xmit em1 > > > # WEB > add 22250 permit tcp from any to 212.101.4.241 80 setup via em1 limit > src-addr 4 > add 22251 permit tcp from any to 212.101.4.241 443 setup via em1 limit > src-addr 4 > add 22253 permit tcp from 212.101.4.241 to any 1024-65535 setup via em1 > > > # FTP > add 22300 permit tcp from any to 212.101.4.244 21 setup via em1 limit > src-addr 4 > add 22301 permit tcp from 212.101.4.244 to any 21 setup via em1 > add 22302 permit tcp from 212.101.4.244 20 to any 1024-65535 setup via em1 > add 22320 permit tcp from any 20 to 212.101.4.244 1024-65535 setup via em1 > add 22340 permit tcp from any to 212.101.4.244 49152-65535 setup via em1 > > # RSYNC > add 22400 permit tcp from 212.101.4.244 to any rsync setup via em1 > add 22410 permit tcp from any to me rsync setup via em1 > > # CVSUPD > add 22500 permit tcp from 212.101.4.244 to any cvsup setup via em1 > add 22510 permit tcp from any to me cvsup setup via em1 > > # > # NO WINDOWS > # > add 30000 deny udp from any to any 137 via em1 > > # Internal Interface > add 62000 allow tcp from 212.101.0.0/24 to any 22,23,25 via em0 setup > add 62010 allow tcp from 212.101.1.0/24 to any 22,23,25 via em0 setup > add 62020 allow tcp from 212.101.4.0/24 to any 22,25 via em0 setup > add 63000 reset tcp from any to any 22,23,25,3306,111 via em0 setup > add 65000 allow ip from any to any via em0 > > # F*** off the rest > add 65500 deny log ip from any to any > > If i load the ipfw rules i see: > > lock order reversal: > 1st 0xffffffff80839448 PFil hook read/write mutex (PFil hook read/write > mutex) @ /usr/src/sys/net/pfil.c:73 > 2nd 0xffffffff80838440 in_multi_mtx (in_multi_mtx) @ > /usr/src/sys/netinet/ip_output.c:302 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > witness_checkorder() at witness_checkorder+0x605 > _mtx_lock_flags() at _mtx_lock_flags+0x75 > ip_output() at ip_output+0x570 > dummynet_send() at dummynet_send+0x13f > dummynet_io() at dummynet_io+0x322 > ipfw_check_out() at ipfw_check_out+0x20a > pfil_run_hooks() at pfil_run_hooks+0xbc > ip_output() at ip_output+0x372 > udp_send() at udp_send+0x350 > sosend_dgram() at sosend_dgram+0x20f > kern_sendit() at kern_sendit+0x122 > sendit() at sendit+0xdc > sendto() at sendto+0x4d > syscall() at syscall+0x1f6 > Xfast_syscall() at Xfast_syscall+0xab > --- syscall (133, FreeBSD ELF64, sendto), rip = 0x800bebc8c, rsp = > 0x7fffffffe6c8, rbp = 0x551420 --- > Expensive timeout(9) function: 0xffffffff80469350(0xffffff004696e000) > 0.530255650 s > > > FreeBSD is installed on ufs but I have a zfs pool for my data. > Filesystem Size Used Avail Capacity Mounted on > /dev/da0s1a 496M 356M 100M 78% / > devfs 1.0K 1.0K 0B 100% /dev > /dev/da0s1g 169G 4.0K 155G 0% /disk1 > /dev/da0s1f 3.9G 902K 3.6G 0% /tmp > /dev/da0s1e 29G 3.9G 23G 15% /usr > /dev/da0s1d 19G 4.0G 14G 22% /var > pool 1.2T 0B 1.2T 0% /usr/local/data > pool/cvsup 1.2T 5.1G 1.2T 0% /usr/local/data/cvsup > pool/ftp 3.3T 2.2T 1.2T 65% /usr/local/data/ftp > pool/portsnap 1.2T 591M 1.2T 0% /usr/local/data/portsnap > pool/www 1.2T 78M 1.2T 0% /usr/local/data/www > > Any help? > > Best regards, > Thomas > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" Please try this patch: http://www.freebsd.org/cgi/query-pr.cgi?prp=113548-3-diff and report does it help or not. -- Oleg. ================================================================ === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- oleg@rinet.ru === ================================================================