From owner-freebsd-stable@FreeBSD.ORG Sun Jan 24 01:22:11 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE9C81065692 for ; Sun, 24 Jan 2010 01:22:11 +0000 (UTC) (envelope-from sam@errno.com) Received: from ebb.errno.com (ebb.errno.com [69.12.149.25]) by mx1.freebsd.org (Postfix) with ESMTP id AB2BA8FC08 for ; Sun, 24 Jan 2010 01:22:11 +0000 (UTC) Received: from Macintosh-4.local ([10.0.0.195]) (authenticated bits=0) by ebb.errno.com (8.13.6/8.12.6) with ESMTP id o0O1MAHf026085 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 23 Jan 2010 17:22:10 -0800 (PST) (envelope-from sam@errno.com) Message-ID: <4B5BA0C1.8010901@errno.com> Date: Sat, 23 Jan 2010 17:22:09 -0800 From: Sam Leffler Organization: Errno Consulting User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: Russell Yount References: <4B521FC2.4050402@errno.com> <4B535AAE.3060308@errno.com> In-Reply-To: X-Enigmail-Version: 0.96.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-DCC-sonic.net-Metrics: ebb.errno.com; whitelist Cc: freebsd-stable@freebsd.org Subject: Re: atheros broadcast/multicast corruption with multiple hostap's X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jan 2010 01:22:12 -0000 Russell Yount wrote: > On Sun, Jan 17, 2010 at 1:45 PM, Sam Leffler > wrote: > > Russell Yount wrote: > > > > On Sat, Jan 16, 2010 at 3:21 PM, Sam Leffler >> wrote: > > Russell Yount wrote: > > It seems AP to client broadcasts/multicasts traffic is > broken when using WPA2/802.11i with multiple hostapds in 8.0. > > Only the SSID associated with the last hostapd to be > started has > AP to client broadcasts/multicasts being delivered correctly. > > The AP and client are 8.0 freebsd systems althought I see > same > problems with windows XP as a client. > > The AP has 4 hostapds configured to use TLS with client > certificates for > authentication. (hostapd recompiled with > HOSTAPD_CFLAGS=-DEAP_SERVER) > The AP and client radio are shown as ath0: AR5212 mac 5.9 > RF5112 > phy 4.3 > in dmesg. > > Client authenticate using client certificates associate > correctly > to all 4 SSIDs. Unicast traffic flows correctly between > clients > and AP > for all for 4 SSIDs. Client to AP broadcast/multicast > traffic works > on of 4 SSIDs. AP to client broadcast/multicast traffic > only works > on 1 of the SSIDs. I have documented this using ARP > broadcasts, > but normal IP broadcasts also observed to corrupted. > > When an ARP request is send through the AP to an > associated client > it seems to be trashed on any of the SSID except the one > associated > with the last hostapd to be started. Here is the output of > client side > tcpdump showing the problems. > > In the first client side tcpdump with the hostapd associated > with the SSID > being associaed with the last hostapd started and the traffic > flowing > normally. > > In the second client side tcpdump with the hostapd associated > with the SSID > being not the last hostapd started the ARP request is resent > multiple times > and appears corrupted. > > I would really like to find a fix for this. > Any help would be greatly appreciated. > > > This sounds like the crypto encap of the frame is clobbering the > mbuf contents. You can verify this by setting up multiple > vaps w/o > WPA. If this is the problem look for the mbuf copy logic for > mcast > frames and make sure a deep copy is done. > > Sam > > The four VAPs broadcast traffic works find without WPA if I > do not start hostapds on them > I have been trying to discovery why broadcast traffic only > works correctly on the VAP associated with the last hostapd to > be started. I have move with VAP has the working broadcast > traffic by restarting the hostapd > associated with it. > It would seem something in the WPA/802.1x layer initialization > remembers which hostapd was started last and that affected the > crypto encap. > I keep looking but do not see any place in the code that could > account for this. > It seems the corrupt crypto encap also happens on broadcast > between stations. > Please correct me if I am wrong: > but when using hostapd normally traffic is bridged withing the card. > So if a station sends to the VAP a broadcast it is actaully > sending a non- broadcast frame to the AP > and the AP sends the frame to all the other stations. > > > I told you waht the likely problem is. Look in the net80211 layer > in the kernel for the problem. > > Sam > > > > > I tried to find problems in mbuf corruption > in ieee80211_output.c by placing > > m = m_unshare(m,M_NOWAIT); > if (m == NULL) { > IEEE80211_DPRINTF(vap, IEEE80211_MSG_OUTPUT, > "%s: cannot get writable mbuf\n", __func__); > return NULL; > } > > at begining ieee80211_mbuf_adjust() and at > beginning of ieee80211_encap() with no change > in the broadcast traffic behaviour. > > I tried then to in ieee80211_crypto.c substituting > > flags |= IEEE80211_KEY_SWCRYPT; > > for the encryption capabilities test code > > if ((ic->ic_cryptocaps & (1< IEEE80211_DPRINTF(vap, IEEE80211_MSG_CRYPTO, > "%s: no h/w support for cipher %s, falling back to s/w\n", > __func__, cip->ic_name); > flags |= IEEE80211_KEY_SWCRYPT; > } > > to force all the encryption to be done in software. > > This fixed the broadcast traffic problem but without > hardware support its very slow and really loads machine. > Enabling in the debug code to ath and net80211 > > and enabled ATH_DEBUG_KEYCACHE in if_ath.c and > IEEE80211_MSG_CRYPTO in net80211 code. > > It seems that all the VAPS sets the broadcast key for > mac ff:ff:ff:ff:ff:ff in the ath device so I assume > they conflict and the last one setting the key is the > working one; that would explain why the last hostapd > started is the only one with working broadcast code > to clients. > > In if_ath.c the code > > if ((k->wk_flags & IEEE80211_KEY_GROUP) && sc->sc_mcastkey) { > /* > * Group keys on hardware that supports multicast frame > * key search use a mac that is the sender's address with > * the high bit set instead of the app-specified address. > */ > IEEE80211_ADDR_COPY(gmac, bss->ni_macaddr); > gmac[0] |= 0x80; > mac = gmac; > } else > mac = k->wk_macaddr; > > seems to indicate that for multiple VAPs the ath chips > needs to be able to distinguish between broadcast keys > by using a permutation of VAPs bssid. > > But in if_athvar.h the code does not seem complete > and sc->sc_mcastkey is also set false. > > #ifdef notyet > #define ath_hal_hasmcastkeysearch(_ah) \ > (ath_hal_getcapability(_ah, HAL_CAP_MCAST_KEYSRCH, 0, NULL) == > HAL_OK) > #define ath_hal_getmcastkeysearch(_ah) \ > (ath_hal_getcapability(_ah, HAL_CAP_MCAST_KEYSRCH, 1, NULL) == > HAL_OK) > #else > #define ath_hal_getmcastkeysearch(_ah) 0 > #endif > > I am using cards with an AR5212 which does seem to have > multiple bssid support working so I hope they should also > support mcastkeysearch capability. Maybe only some > firmware revisions have this? > > Do you know what the status of the HAL code is for > supporting this? Why it is commented out in if_athvar.h? Good work analyzing things; been a long time since I looked at this. I vaguely recall disabling mcastkey searching because of problems with WEP. I'm surprised WPA is broken as that was a standard test case. You can try enabling the notyet code and see if the right thing happens. I don't see any indication of the mac rev for your part but I expect it supports this as it was only very early parts that had issues. If enabling the mcastkey search mechanism doesn't fix this I might've broken things with changes to explicitly mark group keys when hostapd plumbs their contents. I recall doing this for mwl which doesn't have an indexed key table like ath (it uses the mac address of the local bss and/or associated station to find the data structure where crypto keys are stored). I haven't looked at this stuff in a long time and can't setup a system to test but if you keep pushing on this I'll try to help w/ advise. Sam