From owner-freebsd-net@FreeBSD.ORG Wed Oct 12 14:07:09 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 12F64106564A for ; Wed, 12 Oct 2011 14:07:09 +0000 (UTC) (envelope-from fodillemlinkarim@gmail.com) Received: from mail-qy0-f175.google.com (mail-qy0-f175.google.com [209.85.216.175]) by mx1.freebsd.org (Postfix) with ESMTP id C2EDD8FC08 for ; Wed, 12 Oct 2011 14:07:08 +0000 (UTC) Received: by qyk10 with SMTP id 10so4895109qyk.13 for ; Wed, 12 Oct 2011 07:07:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=5fNJ6ctOKZxfxK2rhzzza8vRx6LtXJNcZojV6xEstds=; b=q0PTsxjPXorecX9WQN6+/LXBLvN+hgiZGQP806ux8u/6tfKq7mNgu55aTwVwKjS2lc iRjNpY3IIgJQsOWmbHD3jWGTYeAvRCGqsThdWTIPaCY/owYCMZHmN1SW3YXrXnudEhNY Lm8IRyWk4zZyN63onXhD1d1iI1gTkJIfERODs= Received: by 10.229.87.137 with SMTP id w9mr5708493qcl.284.1318428427913; Wed, 12 Oct 2011 07:07:07 -0700 (PDT) Received: from [192.168.1.71] ([208.85.112.101]) by mx.google.com with ESMTPS id d20sm2988075qak.10.2011.10.12.07.07.06 (version=SSLv3 cipher=OTHER); Wed, 12 Oct 2011 07:07:07 -0700 (PDT) Message-ID: <4E959F06.6040906@gmail.com> Date: Wed, 12 Oct 2011 10:07:02 -0400 From: Karim User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15 MIME-Version: 1.0 To: Kevin Oberman References: <4E94637A.5090607@gmail.com> <20111011171029.GA5661@michelle.cdnetworks.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: pyunyh@gmail.com, freebsd-net@freebsd.org Subject: Re: if_msk.c link negotiation / packet drops X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2011 14:07:09 -0000 On 11-10-11 01:31 PM, Kevin Oberman wrote: > On Tue, Oct 11, 2011 at 10:10 AM, YongHyeon PYUN wrote: >> On Tue, Oct 11, 2011 at 11:40:42AM -0400, Karim wrote: >>> Hi List, >>> >>> Using a Marvell NIC plugged into a CISCO switch I see the >>> auto-negotiation failing and even when forcing the device to full-duplex >>> we sometimes see packet drops. >>> >>> Here is the device description from dmesg: >>> >>> mskc0: port 0xbe00-0xbeff mem >>> 0xfdefc000-0xfdefffff irq 16 at device 0.0 on pci1 >>> msk0: on mskc0 >>> msk0: Ethernet address: 00:03:2d:09:94:52 >>> miibus0: on msk0 >>> e1000phy0: PHY 0 on miibus0 >>> e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, >>> 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow >>> mskc0: [ITHREAD] >>> >>> The switch its plugged in (Cisco) is configured for 100baseTX full-duplex. >>> >>> ifconfig reports: >>> >>> msk0: >>> flags=608843 >>> metric 0 mtu 1500 >>> options=40018 >> The flags and options show that you're using very customized >> driver, right? >> >>> ether 00:03:2d:09:94:52 >>> inet 192.168.122.7 netmask 0xffffff00 broadcast 192.168.122.255 >>> media: Ethernet autoselect (100baseTX) >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> >> Resolved duplex is half so I guess it would be normal to see >> dropped frames which may be triggered by collision. > You have a duplex mis-match. If you are hard setting the remote end to > full, the local end must also be configured o full. Auto-configuration > of duplex requires that both ends run auto-config. When one end is set > to not do auto-config, the other end SHOULD always set to half-duplex. > This is part of 802.3 that is a carry-over from the days when hubs and > coax dominated, so the default was declared to be half. Since so much > hardware now exists with that default, changing it ill never happen. > > Either set your computer to full duplex or turn on auto-configuration > on the Cisco. > > Very little hardware now in service fails to auto-config correctly, > but the practice of lacking down the duplex setting became common in > the early days of full-duplex when it was not yet a standard and many > Ethernet chip-sets didn't play nice with others. Things would be much > better if people would just stop hard-setting the duplex, but old, old > habits and memes die hard. > > Also, contrary to common belief, collisions are NOT errors. They are a > normal part of half-dulpex Ethernet operation and do NOT result in > packets being dropped. Only "excessive collisions do and they ARE a > real error and a clear indication that something is wrong. Hi, Thanks for the feedback and detailed information. I have to clarify here; I get the issue even with forcing full duplex. The driver modifications are minor and shouldn't affect link negotiation or link state. Its also interesting that this problem only shows up with Cisco's switches and faced with other types of switches the issue does not come up. Also, nothing like collisions or missed/dropped packets can be found in msk_stats (see below) that somewhat relate to the issue I'm seeing. One thing interesting is the ifm_status value from msk_mediastatus. It often changes from (IFM_AVALID | IFM_ACTIVE) which is 0x3 to 0x1 (IFM_AVALID) for no reason I can tell. From the code in e1000phy_status: static void e1000phy_status(struct mii_softc *sc) { struct mii_data *mii = sc->mii_pdata; int bmcr, bmsr, ssr; mii->mii_media_status = IFM_AVALID; mii->mii_media_active = IFM_ETHER; bmsr = PHY_READ(sc, E1000_SR) | PHY_READ(sc, E1000_SR); bmcr = PHY_READ(sc, E1000_CR); ssr = PHY_READ(sc, E1000_SSR); if (bmsr & E1000_SR_LINK_STATUS) mii->mii_media_status |= IFM_ACTIVE; I can see the bmsr & E1000_SR_LINK_STATUS check failing when the problem occurs. As a side note why are we ORing the same call twice isn't the same thing as calling it once: bmsr = PHY_READ(sc, E1000_SR) | PHY_READ(sc, E1000_SR); As requested here is my msk0 stats output right after the problem showed up: dev.msk.0.stats.rx.ucast_frames: 58103886 dev.msk.0.stats.rx.bcast_frames: 0 dev.msk.0.stats.rx.pause_frames: 0 dev.msk.0.stats.rx.mcast_frames: 0 dev.msk.0.stats.rx.crc_errs: 0 dev.msk.0.stats.rx.good_octets: 5927739395 dev.msk.0.stats.rx.bad_octets: 0 dev.msk.0.stats.rx.frames_64: 53 dev.msk.0.stats.rx.frames_65_127: 58091128 dev.msk.0.stats.rx.frames_128_255: 12545 dev.msk.0.stats.rx.frames_256_511: 40 dev.msk.0.stats.rx.frames_512_1023: 89 dev.msk.0.stats.rx.frames_1024_1518: 31 dev.msk.0.stats.rx.frames_1519_max: 0 dev.msk.0.stats.rx.frames_too_long: 0 dev.msk.0.stats.rx.jabbers: 0 dev.msk.0.stats.rx.overflows: 0 dev.msk.0.stats.tx.ucast_frames: 58104799 dev.msk.0.stats.tx.bcast_frames: 53 dev.msk.0.stats.tx.pause_frames: 0 dev.msk.0.stats.tx.mcast_frames: 0 dev.msk.0.stats.tx.octets: 5927439760 dev.msk.0.stats.tx.frames_64: 547 dev.msk.0.stats.tx.frames_65_127: 58091680 dev.msk.0.stats.tx.frames_128_255: 12576 dev.msk.0.stats.tx.frames_256_511: 17 dev.msk.0.stats.tx.frames_512_1023: 1 dev.msk.0.stats.tx.frames_1024_1518: 32 dev.msk.0.stats.tx.frames_1519_max: 0 dev.msk.0.stats.tx.colls: 2 dev.msk.0.stats.tx.late_colls: 2 dev.msk.0.stats.tx.excess_colls: 0 dev.msk.0.stats.tx.multi_colls: 0 dev.msk.0.stats.tx.single_colls: 2 dev.msk.0.stats.tx.underflows: 0 The 2 collisions occurred before I forced the interface to full-duplex. Karim.