Date: Mon, 22 Feb 2010 16:29:40 -0800 From: Jack Vogel <jfvogel@gmail.com> To: Kirk Davis <kirk.davis@epsb.ca> Cc: freebsd-net@freebsd.org, Mike Tancsa <mike@sentex.net> Subject: Re: Intel em0: watchdog timeout Message-ID: <2a41acea1002221629vbe7548am7b5f1ba94d7efa9f@mail.gmail.com> In-Reply-To: <529374128DC1B04D9D037911B8E8F05301C17A57@Exchange26.EDU.epsb.ca> References: <529374128DC1B04D9D037911B8E8F05301C17A51@Exchange26.EDU.epsb.ca> <43416_1266864062_4B82CFBE_43416_81_1_2a41acea1002221043k1b8742c9m8fb484a8e8a4fdda@mail.gmail.com> <529374128DC1B04D9D037911B8E8F05301C17A54@Exchange26.EDU.epsb.ca> <43669_1266865888_4B82D6E0_43669_263_1_2a41acea1002221113v26804200q4f3971c3359dffab@mail.gmail.com> <529374128DC1B04D9D037911B8E8F05301C17A55@Exchange26.EDU.epsb.ca> <201002222107.o1ML7v3Z059734@lava.sentex.ca> <529374128DC1B04D9D037911B8E8F05301C17A56@Exchange26.EDU.epsb.ca> <2a41acea1002221444o6e449602m1830761b21837c41@mail.gmail.com> <529374128DC1B04D9D037911B8E8F05301C17A57@Exchange26.EDU.epsb.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Is your driver static, ie builtin, to the kernel, or do you load/unload it as a module? I ask because perhaps we could try a later driver, and being a module makes that easier. Jack On Mon, Feb 22, 2010 at 3:37 PM, Kirk Davis <kirk.davis@epsb.ca> wrote: > OK. I have the following in /boot/loader.conf (and rebooted) > hw.em.rxd=1024 > hw.em.txd=1024 > > Should this be hw.em2.rxd? Is it set per interface or across all > interfaces? > > nmbcluster=262144 > > # sysctl dev.em.2.stats=1 > Feb 22 16:29:57 inet-gw kernel: em2: Defer count = 20 > Feb 22 16:29:57 inet-gw kernel: em2: Missed Packets = 119947 > Feb 22 16:29:57 inet-gw kernel: em2: Receive No Buffers = 276762 > Feb 22 16:29:57 inet-gw kernel: em2: Receive Length Errors = 0 > Feb 22 16:29:57 inet-gw kernel: em2: Receive errors = 0 > Feb 22 16:29:57 inet-gw kernel: em2: Crc errors = 0 > Feb 22 16:29:57 inet-gw kernel: em2: Alignment errors = 0 > Feb 22 16:29:57 inet-gw kernel: em2: Collision/Carrier extension errors = 0 > Feb 22 16:29:57 inet-gw kernel: em2: RX overruns = 21 > Feb 22 16:29:57 inet-gw kernel: em2: watchdog timeouts = 47 > Feb 22 16:29:57 inet-gw kernel: em2: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK > MSIX IRQ = 0 > Feb 22 16:29:57 inet-gw kernel: em2: XON Rcvd = 22 > Feb 22 16:29:57 inet-gw kernel: em2: XON Xmtd = 8349 > Feb 22 16:29:57 inet-gw kernel: em2: XOFF Rcvd = 31 > Feb 22 16:29:57 inet-gw kernel: em2: XOFF Xmtd = 15779 > Feb 22 16:29:57 inet-gw kernel: em2: Good Packets Rcvd = 966101852 > Feb 22 16:29:57 inet-gw kernel: em2: Good Packets Xmtd = 755993237 > Feb 22 16:29:57 inet-gw kernel: em2: TSO Contexts Xmtd = 0 > Feb 22 16:29:57 inet-gw kernel: em2: TSO Contexts Failed = 0 > > still seeing the watchdog timer and link up/down messages. > > Should I try going higher than 1024 on the hw.em.rxd? I'm not sure the > next time I can schedule another reboot on this production server. > > ---- Kirk > > > *Kirk Davis*** > *Senior Network Analyst, ITS* > *Edmonton Public Schools* > *One Kingsway Ave. * > *Edmonton, Alberta, Canada* > *T5H 4G9* > *phone: 1-780-429-8308* > > > ------------------------------ > *From:* Jack Vogel [mailto:jfvogel@gmail.com] > *Sent:* Monday, February 22, 2010 3:45 PM > *To:* Kirk Davis > *Cc:* Mike Tancsa; freebsd-net@freebsd.org > > *Subject:* Re: Intel em0: watchdog timeout > > OK, so you are still failing to get mbufs in the RX side, increase the > nmbcluster > value, and then what size is your RX ring (number of rx descriptors)? > > If you havent already done so, change that to 1024. > > I am developing a change in the RX code right now that will help > this situation, but am doing so in the 10G driver, once its solid there > I will be backporting it into the 1G drivers, it will make discards > almost unnecessary. > > Jack > > On Mon, Feb 22, 2010 at 1:43 PM, Kirk Davis <kirk.davis@epsb.ca> wrote: > >> >> >> > -----Original Message----- >> > From: Mike Tancsa [mailto:mike@sentex.net] >> > Subject: Re: Intel em0: watchdog timeout >> > >> > At 03:46 PM 2/22/2010, Kirk Davis wrote: >> > >Does this need to be done in loader.conf? It doesn't seem >> > to take from >> > >the command line. >> > ># sysctl dev.em.2.stats=1 >> > >dev.em.2.stats: -1 -> -1 >> > > >> > ># sysctl dev.em.2.stats >> > >dev.em.2.stats: -1 >> > >> > Hi, >> > After you issue those commands, the driver will spit out a >> > lot of useful stats to syslog. It will report something like the >> > following in /var/log/messages >> > >> > Feb 22 16:06:31 offsite kernel: em0: Excessive collisions = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Sequence errors = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Defer count = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Missed Packets = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Receive No Buffers = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Receive Length Errors = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Receive errors = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Crc errors = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Alignment errors = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Collision/Carrier >> > extension errors = 0 >> > Feb 22 16:06:31 offsite kernel: em0: RX overruns = 0 >> > Feb 22 16:06:31 offsite kernel: em0: watchdog timeouts = 0 >> > Feb 22 16:06:31 offsite kernel: em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 >> > LINK MSIX IRQ = 0 >> > Feb 22 16:06:31 offsite kernel: em0: XON Rcvd = 0 >> > Feb 22 16:06:31 offsite kernel: em0: XON Xmtd = 0 >> > Feb 22 16:06:31 offsite kernel: em0: XOFF Rcvd = 0 >> > Feb 22 16:06:31 offsite kernel: em0: XOFF Xmtd = 0 >> > Feb 22 16:06:31 offsite kernel: em0: Good Packets Rcvd = 2559032551 >> > Feb 22 16:06:31 offsite kernel: em0: Good Packets Xmtd = 1568751141 >> > Feb 22 16:06:31 offsite kernel: em0: TSO Contexts Xmtd = 0 >> > Feb 22 16:06:31 offsite kernel: em0: TSO Contexts Failed = 0 >> >> Thanks Mike and Jack. I don't know why I didn'ty notice the output in >> /var/log/messages >> >> Here is the output for the two interfaces that are causing this issue. >> >> Feb 22 13:33:52 inet-gw kernel: em0: Excessive collisions = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Sequence errors = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Defer count = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Missed Packets = 24296 >> Feb 22 13:33:52 inet-gw kernel: em0: Receive No Buffers = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Receive Length Errors = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Receive errors = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Crc errors = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Alignment errors = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Collision/Carrier extension errors >> = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: RX overruns = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: watchdog timeouts = 6 >> Feb 22 13:33:52 inet-gw kernel: em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 >> LINK MSIX IRQ = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: XON Rcvd = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: XON Xmtd = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: XOFF Rcvd = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: XOFF Xmtd = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: Good Packets Rcvd = 424303810 >> Feb 22 13:33:52 inet-gw kernel: em0: Good Packets Xmtd = 576529136 >> Feb 22 13:33:52 inet-gw kernel: em0: TSO Contexts Xmtd = 0 >> Feb 22 13:33:52 inet-gw kernel: em0: TSO Contexts Failed = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: Excessive collisions = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: Sequence errors = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: Defer count = 20 >> Feb 22 13:34:12 inet-gw kernel: em2: Missed Packets = 68059 >> Feb 22 13:34:12 inet-gw kernel: em2: Receive No Buffers = 275612 >> Feb 22 13:34:12 inet-gw kernel: em2: Receive Length Errors = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: Receive errors = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: Crc errors = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: Alignment errors = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: Collision/Carrier extension errors >> = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: RX overruns = 17 >> Feb 22 13:34:12 inet-gw kernel: em2: watchdog timeouts = 38 >> Feb 22 13:34:12 inet-gw kernel: em2: RX MSIX IRQ = 0 TX MSIX IRQ = 0 >> LINK MSIX IRQ = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: XON Rcvd = 21 >> Feb 22 13:34:12 inet-gw kernel: em2: XON Xmtd = 8344 >> Feb 22 13:34:12 inet-gw kernel: em2: XOFF Rcvd = 30 >> Feb 22 13:34:12 inet-gw kernel: em2: XOFF Xmtd = 9159 >> Feb 22 13:34:12 inet-gw kernel: em2: Good Packets Rcvd = 713607509 >> Feb 22 13:34:12 inet-gw kernel: em2: Good Packets Xmtd = 569694020 >> Feb 22 13:34:12 inet-gw kernel: em2: TSO Contexts Xmtd = 0 >> Feb 22 13:34:12 inet-gw kernel: em2: TSO Contexts Failed = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: Excessive collisions = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: Sequence errors = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: Defer count = 20 >> Feb 22 13:35:10 inet-gw kernel: em2: Missed Packets = 68059 >> Feb 22 13:35:10 inet-gw kernel: em2: Receive No Buffers = 275612 >> Feb 22 13:35:10 inet-gw kernel: em2: Receive Length Errors = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: Receive errors = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: Crc errors = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: Alignment errors = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: Collision/Carrier extension errors >> = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: RX overruns = 17 >> Feb 22 13:35:10 inet-gw kernel: em2: watchdog timeouts = 38 >> Feb 22 13:35:10 inet-gw kernel: em2: RX MSIX IRQ = 0 TX MSIX IRQ = 0 >> LINK MSIX IRQ = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: XON Rcvd = 21 >> Feb 22 13:35:10 inet-gw kernel: em2: XON Xmtd = 8344 >> Feb 22 13:35:10 inet-gw kernel: em2: XOFF Rcvd = 30 >> Feb 22 13:35:10 inet-gw kernel: em2: XOFF Xmtd = 9159 >> Feb 22 13:35:10 inet-gw kernel: em2: Good Packets Rcvd = 715555016 >> Feb 22 13:35:10 inet-gw kernel: em2: Good Packets Xmtd = 571157561 >> Feb 22 13:35:10 inet-gw kernel: em2: TSO Contexts Xmtd = 0 >> Feb 22 13:35:10 inet-gw kernel: em2: TSO Contexts Failed = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: Excessive collisions = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: Sequence errors = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: Defer count = 20 >> Feb 22 13:39:12 inet-gw kernel: em2: Missed Packets = 68059 >> Feb 22 13:39:12 inet-gw kernel: em2: Receive No Buffers = 275612 >> Feb 22 13:39:12 inet-gw kernel: em2: Receive Length Errors = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: Receive errors = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: Crc errors = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: Alignment errors = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: Collision/Carrier extension errors >> = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: RX overruns = 17 >> Feb 22 13:39:12 inet-gw kernel: em2: watchdog timeouts = 38 >> Feb 22 13:39:12 inet-gw kernel: em2: RX MSIX IRQ = 0 TX MSIX IRQ = 0 >> LINK MSIX IRQ = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: XON Rcvd = 21 >> Feb 22 13:39:12 inet-gw kernel: em2: XON Xmtd = 8344 >> Feb 22 13:39:12 inet-gw kernel: em2: XOFF Rcvd = 30 >> Feb 22 13:39:12 inet-gw kernel: em2: XOFF Xmtd = 9159 >> Feb 22 13:39:12 inet-gw kernel: em2: Good Packets Rcvd = 723521981 >> Feb 22 13:39:12 inet-gw kernel: em2: Good Packets Xmtd = 577211431 >> Feb 22 13:39:12 inet-gw kernel: em2: TSO Contexts Xmtd = 0 >> Feb 22 13:39:12 inet-gw kernel: em2: TSO Contexts Failed = 0 >> >> >> Can this be the problem? "Receive No Buffers = 275612" >> >> ---- Kirk >> Kirk Davis >> Senior Network Analyst, ITS >> Edmonton Public Schools >> One Kingsway Ave. >> Edmonton, Alberta, Canada >> T5H 4G9 >> phone: 1-780-429-8308 >> >> >> >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2a41acea1002221629vbe7548am7b5f1ba94d7efa9f>