From owner-freebsd-current@FreeBSD.ORG Thu Oct 16 00:00:40 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 643171065696 for ; Thu, 16 Oct 2008 00:00:40 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from ti-out-0910.google.com (ti-out-0910.google.com [209.85.142.189]) by mx1.freebsd.org (Postfix) with ESMTP id A34788FC32 for ; Thu, 16 Oct 2008 00:00:39 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: by ti-out-0910.google.com with SMTP id d27so1632299tid.3 for ; Wed, 15 Oct 2008 17:00:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:received:date:from :to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=PSxpF3Ziy+jDTHx7WTgOlJ2tjc8GiHnscgECR6i4az0=; b=Fgr3etjZ4SgeD+dWNZX8pxML5FK3lJiL/f+X1sGCEbpZQ3DTx6jemdpg/kbDdPguSj E9MX7kUT6p0fMorZBKx2B27nOKgIu3kJTRX4Ltj3v/SjkLLmCVZ5H/6CSmX806B+gN2k Vk6L5J2+FYUWq7sBVKZpjGIVNVpbKA7aXPDZs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=IJPqoghMdrxgoIREfzlsJOurQ0UtgH7/9Uq7zfG0Tm1PY1fYzAR3ejcZyHUcGHtNW1 FtjfKUuf6/YCdlM4bdkDUBf5KidjRG/Hacmo1eaunmn1pcHR0gGmQzWcqPXSDCDFL3VB mA8xYuxMf5TaWn9+JfA9wBAusic4zA7pEEf9g= Received: by 10.110.49.2 with SMTP id w2mr1304247tiw.56.1224115238171; Wed, 15 Oct 2008 17:00:38 -0700 (PDT) Received: from michelle.cdnetworks.co.kr ([211.53.35.84]) by mx.google.com with ESMTPS id d1sm658251tid.13.2008.10.15.17.00.34 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 15 Oct 2008 17:00:35 -0700 (PDT) Received: from michelle.cdnetworks.co.kr (localhost.cdnetworks.co.kr [127.0.0.1]) by michelle.cdnetworks.co.kr (8.13.5/8.13.5) with ESMTP id m9FNwaub023107 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 16 Oct 2008 08:58:36 +0900 (KST) (envelope-from pyunyh@gmail.com) Received: (from yongari@localhost) by michelle.cdnetworks.co.kr (8.13.5/8.13.5/Submit) id m9FNwaCa023106; Thu, 16 Oct 2008 08:58:36 +0900 (KST) (envelope-from pyunyh@gmail.com) Date: Thu, 16 Oct 2008 08:58:36 +0900 From: Pyun YongHyeon To: Koen Martens Message-ID: <20081015235836.GB22992@cdnetworks.co.kr> References: <20071001080058.GE18658@cdnetworks.co.kr> <8bebfc960710010707o4b7e9b2cif086e35b4f8f736d@mail.gmail.com> <20071002004521.GH18658@cdnetworks.co.kr> <8bebfc960710012041h205216ecl75d44e0c9ba69068@mail.gmail.com> <20071002035814.GJ18658@cdnetworks.co.kr> <8bebfc960710022231m4b26a07ahdaef145e07862fb6@mail.gmail.com> <20071004011348.GC30781@cdnetworks.co.kr> <20081015122233.GA11699@latitude.dh2.sono> <20081015123440.GJ14769@cdnetworks.co.kr> <20081015135651.GC12650@latitude.dh2.sono> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="VrqPEDrXMn8OVzN4" Content-Disposition: inline In-Reply-To: <20081015135651.GC12650@latitude.dh2.sono> User-Agent: Mutt/1.4.2.1i Cc: freebsd-current@freebsd.org, Kudo Chien Subject: Re: msk watchdog timeout X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2008 00:00:40 -0000 --VrqPEDrXMn8OVzN4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Wed, Oct 15, 2008 at 03:56:52PM +0200, Koen Martens wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Wed, Oct 15, 2008 at 09:34:40PM +0900, Pyun YongHyeon wrote: > > On Wed, Oct 15, 2008 at 02:22:34PM +0200, Koen Martens wrote: > > > The problem occurs under load (rsyncing tens of gigabytes over > > > gigabit link for example). I tried configuring the switch port > > > down to 100MB, in the hopes that msk0 would be more stable. It > > > is, but it still goes down after a while with watchdog timeouts. > > > > > > I am now running it with msi disabled, it appears it lasts longer > > > than before now. But judging by what others said on this subject > > > already, it might still go wrong after as much as a month. > > > > > > Also, I've never had these problems when the machine was still > > > on 6.x with the myk driver. Only after I upgraded it this tuesday > > > to RELENG_7, trouble started. > > > > > > This is a server that I need to put back into production. I could > > > give you some time on it before I do that, but that'd have to be > > > *right now* so i guess that won't work out really. > > > > > > I'll probably install a nic to be used instead of the built-in > > > yukon interface, to get back the required stability. > > > > > > > I'm not sure whether 88E8050 also has RAM buffer. Youkon > > controllers seems to have silicon bugs for hardwares with RAM > > buffer. msk(4) in HEAD has workaround code for the silicon bug. > > Would you try latest msk(4) from HEAD?(Just copy > > if_msk.c/if_mskreg.h from HEAD to your box and rebuild kernel.) > > Also show me verbosed boot message(msk(4) related one would be > > enough). > > No dice, compile bails out: > > o-pointer-sign -fformat-extensions -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -mno-align-long-strings -mpreferred-stack-boundary=2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding -Werror /usr/src/sys/dev/msk/if_msk.c > /usr/src/sys/dev/msk/if_msk.c:845:50: error: macro "MEXTADD" passed 8 arguments, but takes just 7 > /usr/src/sys/dev/msk/if_msk.c: In function 'msk_jumbo_newbuf': > /usr/src/sys/dev/msk/if_msk.c:844: error: 'MEXTADD' undeclared (first use in this function) > /usr/src/sys/dev/msk/if_msk.c:844: error: (Each undeclared identifier is reported only once > /usr/src/sys/dev/msk/if_msk.c:844: error: for each function it appears in.) > *** Error code 1 > > Stop in /usr/obj/usr/src/sys/GENERIC. > *** Error code 1 > > Stop in /usr/src. > *** Error code 1 > > Stop in /usr/src. > postel# > > I'll have to get back to you about the verbose boot, as i have no remote > console on that machine. > Oops! Sorry, I forgot there is difference in jumbo frame buffer handling. Try attached patch and show me verbosed boot message. -- Regards, Pyun YongHyeon --VrqPEDrXMn8OVzN4 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="msk.watchdog.diff" Index: if_msk.c =================================================================== --- if_msk.c (revision 183165) +++ if_msk.c (working copy) @@ -244,6 +244,9 @@ static int msk_handle_events(struct msk_softc *); static void msk_handle_hwerr(struct msk_if_softc *, uint32_t); static void msk_intr_hwerr(struct msk_softc *); +#ifndef __NO_STRICT_ALIGNMENT +static __inline void msk_fixup_rx(struct mbuf *); +#endif static void msk_rxeof(struct msk_if_softc *, uint32_t, int); static void msk_jumbo_rxeof(struct msk_if_softc *, uint32_t, int); static void msk_txeof(struct msk_if_softc *, int); @@ -783,7 +786,12 @@ return (ENOBUFS); m->m_len = m->m_pkthdr.len = MCLBYTES; - m_adj(m, ETHER_ALIGN); + if ((sc_if->msk_flags & MSK_FLAG_RAMBUF) == 0) + m_adj(m, ETHER_ALIGN); +#ifndef __NO_STRICT_ALIGNMENT + else + m_adj(m, MSK_RX_BUF_ALIGN); +#endif if (bus_dmamap_load_mbuf_sg(sc_if->msk_cdata.msk_rx_tag, sc_if->msk_cdata.msk_rx_sparemap, m, segs, &nsegs, @@ -840,7 +848,12 @@ return (ENOBUFS); } m->m_pkthdr.len = m->m_len = MSK_JLEN; - m_adj(m, ETHER_ALIGN); + if ((sc_if->msk_flags & MSK_FLAG_RAMBUF) == 0) + m_adj(m, ETHER_ALIGN); +#ifndef __NO_STRICT_ALIGNMENT + else + m_adj(m, MSK_RX_BUF_ALIGN); +#endif if (bus_dmamap_load_mbuf_sg(sc_if->msk_cdata.msk_jumbo_rx_tag, sc_if->msk_cdata.msk_jumbo_rx_sparemap, m, segs, &nsegs, @@ -1041,14 +1054,16 @@ { int next; int i; - uint8_t val; /* Get adapter SRAM size. */ - val = CSR_READ_1(sc, B2_E_0); - sc->msk_ramsize = (val == 0) ? 128 : val * 4; + sc->msk_ramsize = CSR_READ_1(sc, B2_E_0) * 4; if (bootverbose) device_printf(sc->msk_dev, "RAM buffer size : %dKB\n", sc->msk_ramsize); + if (sc->msk_ramsize == 0) + return (0); + + sc->msk_pflags |= MSK_FLAG_RAMBUF; /* * Give receiver 2/3 of memory and round down to the multiple * of 1024. Tx/Rx RAM buffer size of Yukon II shoud be multiple @@ -1412,6 +1427,7 @@ sc_if->msk_if_dev = dev; sc_if->msk_port = port; sc_if->msk_softc = sc; + sc_if->msk_flags = sc->msk_pflags; sc->msk_if[port] = sc_if; /* Setup Tx/Rx queue register offsets. */ if (port == MSK_PORT_A) { @@ -1976,6 +1992,7 @@ struct msk_rxdesc *jrxd; struct msk_jpool_entry *entry; uint8_t *ptr; + bus_size_t rxalign; int error, i; mtx_init(&sc_if->msk_jlist_mtx, "msk_jlist_mtx", NULL, MTX_DEF); @@ -2107,9 +2124,16 @@ goto fail; } + rxalign = 1; + /* + * Workaround hardware hang which seems to happen when Rx buffer + * is not aligned on multiple of FIFO word(8 bytes). + */ + if ((sc_if->msk_flags & MSK_FLAG_RAMBUF) != 0) + rxalign = MSK_RX_BUF_ALIGN; /* Create tag for Rx buffers. */ error = bus_dma_tag_create(sc_if->msk_cdata.msk_parent_tag,/* parent */ - 1, 0, /* alignment, boundary */ + rxalign, 0, /* alignment, boundary */ BUS_SPACE_MAXADDR, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ @@ -2918,6 +2942,23 @@ return (0); } +#ifndef __NO_STRICT_ALIGNMENT +static __inline void +msk_fixup_rx(struct mbuf *m) +{ + int i; + uint16_t *src, *dst; + + src = mtod(m, uint16_t *); + dst = src - 3; + + for (i = 0; i < (m->m_len / sizeof(uint16_t) + 1); i++) + *dst++ = *src++; + + m->m_data -= (MSK_RX_BUF_ALIGN - ETHER_ALIGN); +} +#endif + static void msk_rxeof(struct msk_if_softc *sc_if, uint32_t status, int len) { @@ -2955,6 +2996,10 @@ } m->m_pkthdr.rcvif = ifp; m->m_pkthdr.len = m->m_len = len; +#ifndef __NO_STRICT_ALIGNMENT + if ((sc_if->msk_flags & MSK_FLAG_RAMBUF) != 0) + msk_fixup_rx(m); +#endif ifp->if_ipackets++; /* Check for VLAN tagged packets. */ if ((status & GMR_FS_VLAN) != 0 && @@ -3008,6 +3053,10 @@ } m->m_pkthdr.rcvif = ifp; m->m_pkthdr.len = m->m_len = len; +#ifndef __NO_STRICT_ALIGNMENT + if ((sc_if->msk_flags & MSK_FLAG_RAMBUF) != 0) + msk_fixup_rx(m); +#endif ifp->if_ipackets++; /* Check for VLAN tagged packets. */ if ((status & GMR_FS_VLAN) != 0 && @@ -3677,7 +3726,7 @@ /* Configure hardware VLAN tag insertion/stripping. */ msk_setvlan(sc_if, ifp); - if (sc->msk_hw_id == CHIP_ID_YUKON_EC_U) { + if ((sc_if->msk_flags & MSK_FLAG_RAMBUF) == 0) { /* Set Rx Pause threshould. */ CSR_WRITE_1(sc, MR_ADDR(sc_if->msk_port, RX_GMF_LP_THR), MSK_ECU_LLPP); @@ -3790,6 +3839,8 @@ int ltpp, utpp; sc = sc_if->msk_softc; + if ((sc_if->msk_flags & MSK_FLAG_RAMBUF) == 0) + return; /* Setup Rx Queue. */ CSR_WRITE_1(sc, RB_ADDR(sc_if->msk_rxq, RB_CTRL), RB_RST_CLR); Index: if_mskreg.h =================================================================== --- if_mskreg.h (revision 183165) +++ if_mskreg.h (working copy) @@ -2158,6 +2158,7 @@ #define MSK_TX_RING_CNT 256 #define MSK_RX_RING_CNT 256 +#define MSK_RX_BUF_ALIGN 8 #define MSK_JUMBO_RX_RING_CNT MSK_RX_RING_CNT #define MSK_STAT_RING_CNT ((1 + 3) * (MSK_TX_RING_CNT + MSK_RX_RING_CNT)) #define MSK_MAXTXSEGS 32 @@ -2307,6 +2308,7 @@ uint32_t msk_coppertype; uint32_t msk_intrmask; uint32_t msk_intrhwemask; + uint32_t msk_pflags; int msk_suspended; int msk_clock; int msk_msi; @@ -2348,6 +2350,8 @@ int msk_phytype; int msk_phyaddr; int msk_link; + uint32_t msk_flags; +#define MSK_FLAG_RAMBUF 0x0010 struct callout msk_tick_ch; int msk_watchdog_timer; uint32_t msk_txq; /* Tx. Async Queue offset */ --VrqPEDrXMn8OVzN4--