From owner-freebsd-net@FreeBSD.ORG Wed Jul 19 13:34:24 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3D3BF16A4DA for ; Wed, 19 Jul 2006 13:34:24 +0000 (UTC) (envelope-from rrs@cisco.com) Received: from sj-iport-2.cisco.com (sj-iport-2-in.cisco.com [171.71.176.71]) by mx1.FreeBSD.org (Postfix) with ESMTP id 20A7543D49 for ; Wed, 19 Jul 2006 13:34:22 +0000 (GMT) (envelope-from rrs@cisco.com) Received: from sj-dkim-1.cisco.com ([171.71.179.21]) by sj-iport-2.cisco.com with ESMTP; 19 Jul 2006 06:34:23 -0700 X-IronPort-AV: i="4.06,260,1149490800"; d="scan'208"; a="330011425:sNHT29662100" Received: from sj-core-5.cisco.com (sj-core-5.cisco.com [171.71.177.238]) by sj-dkim-1.cisco.com (8.12.11.20060308/8.12.11) with ESMTP id k6JDYMel020671; Wed, 19 Jul 2006 06:34:22 -0700 Received: from xbh-sjc-221.amer.cisco.com (xbh-sjc-221.cisco.com [128.107.191.63]) by sj-core-5.cisco.com (8.12.10/8.12.6) with ESMTP id k6JDYMiB002744; Wed, 19 Jul 2006 06:34:22 -0700 (PDT) Received: from xfe-sjc-211.amer.cisco.com ([171.70.151.174]) by xbh-sjc-221.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 19 Jul 2006 06:34:22 -0700 Received: from [127.0.0.1] ([171.68.225.134]) by xfe-sjc-211.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 19 Jul 2006 06:34:21 -0700 Message-ID: <44BE34E2.7070603@cisco.com> Date: Wed, 19 Jul 2006 09:34:26 -0400 From: Randall Stewart User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060223 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Pawel Worach References: <44BB7A92.9080008@cisco.com> In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 19 Jul 2006 13:34:21.0553 (UTC) FILETIME=[0ADECE10:01C6AB38] DKIM-Signature: a=rsa-sha1; q=dns; l=9534; t=1153316062; x=1154180062; c=relaxed/simple; s=sjdkim1002; h=Content-Type:From:Subject:Content-Transfer-Encoding:MIME-Version; d=cisco.com; i=rrs@cisco.com; z=From:Randall=20Stewart=20 |Subject:Re=3A=20SCTP; X=v=3Dcisco.com=3B=20h=3DeFwn8yHouLxYzxwWS0cbAORTlr0=3D; b=GWl/aIYnhQ+f+dByKrzv/wQ0R+K4EQajnhMzn2A9WFoCX3mBAt6yihiJRsmrCeuRDVArZzmn X0s+OHZyY3E7LNjmPz3iPk5BQ7PIn08jbuKHex4+EenJRmpLRB/lpURm; Authentication-Results: sj-dkim-1.cisco.com; header.From=rrs@cisco.com; dkim=pass ( sig from cisco.com verified; ); Cc: freebsd-net@freebsd.org Subject: Re: SCTP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Jul 2006 13:34:24 -0000 Pawel: I see at least one thing wrong with the sctp_sendmsg() code... I just recently added the iov.... and the order of where bad:/ bad2:/ bad1: goes is wrong.. Now, the MAC stuff I have never enabled (at least I don't think so).. and I see that in this trace it seems the MAC stuff is calling to deallocate the socket directly... I am not sure if the crash is related to the wrong bad calls.. which would do a free() when it should not on the iov.. that can't be good.. but also not sure of the deallocate() stuff... The bad stuff is easy to fix.. and I will get a new patch prepared.. (I also will see if I can't update to current again.. and thus eliminate your syscall conflict).. But I want to look a bit into this mac_destroy_socket() path... R Pawel Worach wrote: > On 7/17/06, Randall Stewart wrote: > >> All: >> >> Just a friendly reminder/prod... if you have started >> testing SCTP.. thats great (any feedback?).. >> and if you have not .. please do so :-D > > > Hi, > > I played around a bit with NetPIPE, FreeBSD-CURRENT in one end and > Linux 2.6.17 in the other over a gigabit crossover cable network, 1500 > MTU. FreeBSD crashes after a while. I do have MAC enabled (no policy > modules loaded at the time), it looks like it is involved. I think I > can reproduce this, made it happen on both attempts. > > For the record, I modified the patch a bit to make it compile, the > syscalls numbers collide with new threading syscalls added recently, > so I moved the thr syscalls up a notch. And I removed this #ifdef MAC > part of the patch due to duplicate sctp_bad labels. > > +#ifdef MAC > +sctp_bad: > +#endif > + sctp_bad: > + free(iov, M_IOV); > > Any more info I can provide ? > > ~/sctp/np> ./NPsctp -h 192.168.10.1 > ... > 68: 16384 bytes 71 times --> 179.87 Mbps in 694.94 usec > 69: 16387 bytes 71 times --> 178.78 Mbps in 699.33 usec > 70: 24573 bytes 71 times --> 198.43 Mbps in 944.80 usec > 71: 24576 bytes 70 times --> 199.18 Mbps in 941.35 usec > 72: 24579 bytes 70 times --> 198.82 Mbps in 943.19 usec > 73: 32765 bytes 35 times --> 210.05 Mbps in 1190.07 usec > 74: 32768 bytes 42 times --> 208.48 Mbps in 1199.15 usec > 75: 32771 bytes 41 times --> 208.00 Mbps in 1202.03 usec > 76: 49149 bytes 41 times --> 234.43 Mbps in 1599.55 usec > 77: 49152 bytes 41 times --> 300.20 Mbps in 1249.17 usec > 78: 49155 bytes 53 times --> 299.66 Mbps in 1251.51 usec > 79: 65533 bytes 26 times --> 4.77 Mbps in 104844.52 usec > 80: 65536 bytes 3 times --> 3.70 Mbps in 135258.48 usec > 81: 65539 bytes 3 times --> 3.70 Mbps in 135257.16 usec > 82: 98301 bytes 3 times --> 7.36 Mbps in 101946.00 usec > 83: 98304 bytes 3 times --> 7.36 Mbps in 101923.51 usec > 84: 98307 bytes 3 times --> 7.36 Mbps in 101945.48 usec > 85: 131069 bytes 3 times --> [stalls here] > > then a couple of seconds later... > > Fatal trap 12: page fault while in kernel mode > fault virtual address = 0x0 > fault code = supervisor write, page not present > instruction pointer = 0x20:0xc06a7e16 > stack pointer = 0x28:0xd35e5174 > frame pointer = 0x28:0xd35e5174 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 12 (swi1: net) > trap number = 12 > panic: page fault > KDB: stack backtrace: > kdb_backtrace(c078488a,c07e2500,c07790c0,d35e5028,100,...) at > kdb_backtrace+0x2e > panic(c07790c0,c079de93,c2466a70,1,1,...) at panic+0xb7 > trap_fatal(d35e5134,0,2,8,e5df6f6e,...) at trap_fatal+0x342 > trap_pfault(d35e5134,0,0,0,0,...) at trap_pfault+0x245 > trap(8,ffff0028,7fff0028,c104db80,0,...) at trap+0x3e3 > calltrap() at calltrap+0x5 > --- trap 0xc, eip = 0xc06a7e16, esp = 0xd35e5174, ebp = 0xd35e5174 --- > mac_labelzone_dtor(0,14,0,0,0,...) at mac_labelzone_dtor+0x6 > uma_zfree_arg(c104db80,0,0,d35e51d0,c06acfc4,...) at uma_zfree_arg+0x2f > mac_labelzone_free(0) at mac_labelzone_free+0x22 > mac_socket_label_free(0,c2ad4000,d35e5200,c0587da8,c2ad4000,...) at > mac_socket_label_free+0x94 > mac_destroy_socket(c2ad4000,40,0,c2ad4000,4,...) at mac_destroy_socket+0x18 > sodealloc(c2ad4000,c2ad4000,0,0,4,...) at sodealloc+0x168 > sofree(c2ad4000,0,0,0,c10372c8,...) at sofree+0x311 > sctp_inpcb_free(c2c98000,0,0,d35e52b4,c060c90d,...) at > sctp_inpcb_free+0x10d6 > sctp_free_assoc(c2c98000,c2cad958,0,c2cafcd0,d35e534c,...) at > sctp_free_assoc+0x1a5b > sctp_handle_shutdown_complete(c2cf3830,c2cad958,c2cafcd0,d35e534c,c0754bbe,...) > > at sctp_handle_shutdown_complete+0x228 > sctp_process_control(c2cea500,14,d35e5bb8,24,c2cf3824,...) at > sctp_process_control+0x1388 > sctp_common_input_processing(d35e5c30,14,20,24,c2cf3824,...) at > sctp_common_input_processing+0x87 > sctp_input(c2cea500,14,c255c800,1,0,...) at sctp_input+0x383 > ip_input(c2cea500,d35e5c88,c0553c65,8,0,...) at ip_input+0x70c > netisr_processqueue(c07e75b8,c2467870,c2467870,c24668d0,d35e5ce4,...) > at netisr_processqueue+0xe9 > swi_net(0,c2467870,80246,b9669622,c2467870,...) at swi_net+0x12f > ithread_execute_handlers(c24668d0,c2463500,c24668d0,c2467870,c24668d0,...) > at ithread_execute_handlers+0x188 > ithread_loop(c2433ad0,d35e5d38,0,0,c2433ad0,...) at ithread_loop+0x76 > fork_exit(c051d900,c2433ad0,d35e5d38) at fork_exit+0x7f > fork_trampoline() at fork_trampoline+0x8 > --- trap 0x1, eip = 0, esp = 0xd35e5d6c, ebp = 0 --- > Uptime: 27m28s > Physical memory: 502 MB > Dumping 68 MB: 53 37 21 5 > > #0 doadump () at pcpu.h:166 > 166 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) bt > #0 doadump () at pcpu.h:166 > #1 0xc053c0b4 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 > #2 0xc053c42d in panic (fmt=0xc07790c0 "%s") > at /usr/src/sys/kern/kern_shutdown.c:565 > #3 0xc074a2d2 in trap_fatal (frame=0xd35e5134, eva=0) > at /usr/src/sys/i386/i386/trap.c:869 > #4 0xc0749f65 in trap_pfault (frame=0xd35e5134, usermode=0, eva=0) > at /usr/src/sys/i386/i386/trap.c:778 > #5 0xc0749ab3 in trap (frame= > {tf_fs = 8, tf_es = -65496, tf_ds = 2147418152, tf_edi = > -1056646272, tf_esi = 0, tf_ebp = -748793484, tf_isp = -748793504, > tf_ebx = 0, tf_edx = 0, tf_ecx = 4, tf_eax = 0, tf_trapno = 12, tf_err > = 2, tf_eip = -1066762730, tf_cs = 32, tf_eflags = 66178, tf_esp = > -748793432, tf_ss = -1066463889}) > at /usr/src/sys/i386/i386/trap.c:463 > #6 0xc0738cfa in calltrap () at /usr/src/sys/i386/i386/exception.s:138 > #7 0xc06a7e16 in mac_labelzone_dtor (mem=0x0, size=20, arg=0x0) > at /usr/src/sys/security/mac/mac_label.c:74 > #8 0xc06f0d6f in uma_zfree_arg (zone=0xc104db80, item=0x0, udata=0x0) > at /usr/src/sys/vm/uma_core.c:2263 > #9 0xc06a7e72 in mac_labelzone_free (label=0x0) at uma.h:303 > #10 0xc06acfc4 in mac_socket_label_free (label=0x0) > at /usr/src/sys/security/mac/mac_socket.c:151 > #11 0xc06ad088 in mac_destroy_socket (socket=0xc2ad4000) > ---Type to continue, or q to quit--- > at /usr/src/sys/security/mac/mac_socket.c:168 > #12 0xc0587da8 in sodealloc (so=0xc2ad4000) > at /usr/src/sys/kern/uipc_socket.c:291 > #13 0xc0588971 in sofree (so=0xc2ad4000) at > /usr/src/sys/kern/uipc_socket.c:592 > #14 0xc0604986 in sctp_inpcb_free (inp=0xc2c98000, immediate=0) > at /usr/src/sys/netinet/sctp_pcb.c:2582 > #15 0xc060817b in sctp_free_assoc (inp=0xc2c98000, stcb=0xc2cad958, > from_inpcbfree=0) at /usr/src/sys/netinet/sctp_pcb.c:3896 > #16 0xc0617b58 in sctp_handle_shutdown_complete (cp=0xc2cf3830, > stcb=0xc2cad958, net=0x0) at /usr/src/sys/netinet/sctp_input.c:2500 > #17 0xc061a7d8 in sctp_process_control (m=0xc2cea500, iphlen=20, > offset=0xd35e5bb8, length=36, sh=0xc2cf3824, ch=0xc2cf3830, > inp=0xc2c98000, stcb=0xc2cad958, netp=0xd35e5bd0, > fwd_tsn_seen=0xd35e5b98) > at /usr/src/sys/netinet/sctp_input.c:4267 > #18 0xc061ad87 in sctp_common_input_processing (mm=0xd35e5c30, iphlen=20, > offset=32, length=36, sh=0xc2cf3824, ch=0xc2cf3830, inp=0xc2c98000, > stcb=0xc2cad958, net=0xc2cafcd0, ecn_bits=2 '\002') > at /usr/src/sys/netinet/sctp_input.c:4583 > #19 0xc061b5e3 in sctp_input (m=0xc2cea500, off=20) > at /usr/src/sys/netinet/sctp_input.c:4994 > #20 0xc05ec1ec in ip_input (m=0xc2cea500) > at /usr/src/sys/netinet/ip_input.c:658 > #21 0xc05d2de9 in netisr_processqueue (ni=0xc07e75b8) > ---Type to continue, or q to quit--- > at /usr/src/sys/net/netisr.c:236 > #22 0xc05d305f in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349 > #23 0xc051d808 in ithread_execute_handlers (p=0xc24668d0, ie=0xc2463500) > at /usr/src/sys/kern/kern_intr.c:662 > #24 0xc051d976 in ithread_loop (arg=0xc2433ad0) > at /usr/src/sys/kern/kern_intr.c:745 > #25 0xc051c38f in fork_exit (callout=0xc051d900 , arg=0x0, > frame=0x0) at /usr/src/sys/kern/kern_fork.c:822 > #26 0xc0738d5c in fork_trampoline () at > /usr/src/sys/i386/i386/exception.s:199 > -- Randall Stewart NSSTG - Cisco Systems Inc. 803-345-0369 815-342-5222 (cell)