Date: Sat, 4 Dec 2010 21:32:27 +0800 From: Adrian Chadd <adrian@freebsd.org> To: Russell Yount <russell.yount@gmail.com> Cc: freebsd-stable@freebsd.org Subject: Re: atheros broadcast/multicast corruption with multiple hostap's Message-ID: <AANLkTinHoSv3yX_73HFaO8wg8tNzoyDbfaTH_y9vLOMQ@mail.gmail.com> In-Reply-To: <c62ff5ca1001241616p166238e8ie8a8eff911605d44@mail.gmail.com> References: <c62ff5ca0912302316o59c01ec5wd9efd008afd59c7f@mail.gmail.com> <4B521FC2.4050402@errno.com> <c62ff5ca1001171010v5ed0458dg7f066e4ef9a15de4@mail.gmail.com> <4B535AAE.3060308@errno.com> <c62ff5ca1001181929j3de9818ct785bfbb18883c55e@mail.gmail.com> <4B5BA0C1.8010901@errno.com> <c62ff5ca1001241616p166238e8ie8a8eff911605d44@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Just FYI, (and sorry for resurrecting such an old thread!)
The mcast keysearch changes you've suggested have broken CCMP handling
on at least the AR9160 in -HEAD.
kern/150148 also notes that between 8.0-REL and 8.1-REL CCMP WPA for
the poster also broke. The main change here related to crypto/key
handling is the mcast key search enable that's been enabled.
Reverting those changes so mcast key search is enabled fixes 11n WPA
(which requires CCMP.) I bet it'd also fix kern/150148.
I'm guessing there's some work needed in the key handling code in if_ath.
Russell, are you still interested in this problem? Would you be
interested in giving me a hand?
Thanks Russell/Sam,
Adrian
On 25 January 2010 08:16, Russell Yount <russell.yount@gmail.com> wrote:
> On Sat, Jan 23, 2010 at 8:22 PM, Sam Leffler <sam@errno.com> wrote:
>
>> Russell Yount wrote:
>> > On Sun, Jan 17, 2010 at 1:45 PM, Sam Leffler <sam@errno.com
>> > <mailto:sam@errno.com>> wrote:
>> >
>> > =A0 =A0 Russell Yount wrote:
>> >
>> >
>> >
>> > =A0 =A0 =A0 =A0 On Sat, Jan 16, 2010 at 3:21 PM, Sam Leffler <sam@errn=
o.com
>> > =A0 =A0 =A0 =A0 <mailto:sam@errno.com> <mailto:sam@errno.com
>> =A0> =A0 =A0 =A0 =A0 <mailto:sam@errno.com>>> wrote:
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0Russell Yount wrote:
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0It seems AP to client broadcasts/multic=
asts traffic is
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0broken when using WPA2/802.11i with mul=
tiple hostapds in
>> 8.0.
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Only the SSID associated with the last =
hostapd to be
>> > =A0 =A0 =A0 =A0 started has
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0AP to client broadcasts/multicasts bein=
g delivered
>> correctly.
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0The AP and client are 8.0 freebsd syste=
ms althought I see
>> > =A0 =A0 =A0 =A0 same
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0problems with windows XP as a client.
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0The AP has 4 hostapds configured to use=
TLS with client
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0certificates for
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0authentication. (hostapd recompiled wit=
h
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0HOSTAPD_CFLAGS=3D-DEAP_SERVER)
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0The AP and client radio are shown as at=
h0: AR5212 mac 5.9
>> > =A0 =A0 =A0 =A0 RF5112
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0phy 4.3
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0in dmesg.
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Client authenticate using client certif=
icates associate
>> > =A0 =A0 =A0 =A0 correctly
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0to all 4 SSIDs. Unicast traffic flows c=
orrectly between
>> > =A0 =A0 =A0 =A0 clients
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0and AP
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0for all for 4 SSIDs. Client to AP broad=
cast/multicast
>> > =A0 =A0 =A0 =A0 traffic works
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0on of 4 SSIDs. AP to client broadcast/m=
ulticast traffic
>> > =A0 =A0 =A0 =A0 only works
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0on 1 of the SSIDs. I have documented th=
is using ARP
>> > =A0 =A0 =A0 =A0 broadcasts,
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0but normal IP broadcasts also observed =
to corrupted.
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0When an ARP request is send through the=
AP to an
>> > =A0 =A0 =A0 =A0 associated client
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0it seems to be trashed on any of the SS=
ID except the one
>> > =A0 =A0 =A0 =A0 associated
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0with the last hostapd to be started. He=
re is the output of
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0client side
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0tcpdump showing the problems.
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0In the first client side tcpdump with t=
he hostapd
>> associated
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0with the SSID
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0being associaed with the last hostapd s=
tarted and the
>> traffic
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0flowing
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0normally.
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0In the second client side tcpdump with =
the hostapd
>> associated
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0with the SSID
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0being not the last hostapd started the =
ARP request is
>> resent
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0multiple times
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0and appears corrupted.
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0I would really like to find a fix for t=
his.
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Any help would be greatly appreciated.
>> >
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0This sounds like the crypto encap of the frame =
is clobbering
>> the
>> > =A0 =A0 =A0 =A0 =A0 =A0mbuf contents. =A0You can verify this by settin=
g up multiple
>> > =A0 =A0 =A0 =A0 vaps w/o
>> > =A0 =A0 =A0 =A0 =A0 =A0WPA. =A0If this is the problem look for the mbu=
f copy logic for
>> > =A0 =A0 =A0 =A0 mcast
>> > =A0 =A0 =A0 =A0 =A0 =A0frames and make sure a deep copy is done.
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Sam
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0The four VAPs broadcast traffic works find with=
out WPA if I
>> > =A0 =A0 =A0 =A0 do not start hostapds on them
>> > =A0 =A0 =A0 =A0 =A0I have been trying to discovery why broadcast traff=
ic only
>> > =A0 =A0 =A0 =A0 works correctly on the VAP associated with the last ho=
stapd to
>> > =A0 =A0 =A0 =A0 be started. I have move with VAP has the working broad=
cast
>> > =A0 =A0 =A0 =A0 traffic by restarting the hostapd
>> > =A0 =A0 =A0 =A0 associated with it.
>> > =A0 =A0 =A0 =A0 =A0It would seem something in the WPA/802.1x layer ini=
tialization
>> > =A0 =A0 =A0 =A0 remembers which hostapd was started last and that affe=
cted the
>> > =A0 =A0 =A0 =A0 crypto encap.
>> > =A0 =A0 =A0 =A0 =A0I keep looking but do not see any place in the code=
that could
>> > =A0 =A0 =A0 =A0 account for this.
>> > =A0 =A0 =A0 =A0 =A0It seems the corrupt crypto encap also happens on b=
roadcast
>> > =A0 =A0 =A0 =A0 between stations.
>> > =A0 =A0 =A0 =A0 Please correct me if I am wrong:
>> > =A0 =A0 =A0 =A0 but when using hostapd normally traffic is bridged wit=
hing the
>> card.
>> > =A0 =A0 =A0 =A0 So if a station sends to the VAP a broadcast it is act=
aully
>> > =A0 =A0 =A0 =A0 sending a non- broadcast frame to the AP
>> > =A0 =A0 =A0 =A0 and the AP sends the frame to all the other stations.
>> >
>> >
>> > =A0 =A0 I told you waht the likely problem is. =A0Look in the net80211=
layer
>> > =A0 =A0 in the kernel for the problem.
>> >
>> > =A0 =A0 =A0 =A0 =A0 =A0Sam
>> >
>> >
>> >
>> >
>> > =A0I tried to find problems in mbuf corruption
>> > in ieee80211_output.c by placing
>> >
>> > =A0 m =3D m_unshare(m,M_NOWAIT);
>> > =A0 if (m =3D=3D NULL) {
>> > =A0 =A0 =A0 IEEE80211_DPRINTF(vap, IEEE80211_MSG_OUTPUT,
>> > =A0 =A0 =A0 "%s: cannot get writable mbuf\n", __func__);
>> > =A0 =A0 =A0 return NULL;
>> > =A0 }
>> >
>> > at begining ieee80211_mbuf_adjust() and at
>> > beginning of ieee80211_encap() with no change
>> > in the broadcast traffic behaviour.
>> >
>> > I tried then to in ieee80211_crypto.c substituting
>> >
>> > =A0 flags |=3D IEEE80211_KEY_SWCRYPT;
>> >
>> > for the encryption capabilities test code
>> >
>> > =A0 if ((ic->ic_cryptocaps & (1<<cipher)) =3D=3D 0) {
>> > =A0 =A0 =A0 =A0IEEE80211_DPRINTF(vap, IEEE80211_MSG_CRYPTO,
>> > =A0 =A0 =A0 =A0"%s: no h/w support for cipher %s, falling back to s/w\=
n",
>> > =A0 =A0 =A0 =A0 =A0 =A0__func__, cip->ic_name);
>> > =A0 =A0 =A0 =A0flags |=3D IEEE80211_KEY_SWCRYPT;
>> > =A0 =A0}
>> >
>> > to force all the encryption to be done in software.
>> >
>> > This fixed the broadcast traffic problem but without
>> > hardware support its very slow and really loads machine.
>> > Enabling in the debug code to ath and net80211
>> >
>> > and enabled ATH_DEBUG_KEYCACHE in if_ath.c and
>> > IEEE80211_MSG_CRYPTO in net80211 code.
>> >
>> > It seems that all the VAPS sets the broadcast key for
>> > mac ff:ff:ff:ff:ff:ff in the ath device so I assume
>> > they conflict and the last one setting the key is the
>> > working one; that would explain why the last hostapd
>> > started is the only one with working broadcast code
>> > to clients.
>> >
>> > In if_ath.c the code
>> >
>> > =A0if ((k->wk_flags & IEEE80211_KEY_GROUP) && sc->sc_mcastkey) {
>> > =A0 =A0/*
>> > =A0 =A0 * Group keys on hardware that supports multicast frame
>> > =A0 =A0 * key search use a mac that is the sender's address with
>> > =A0 =A0 * the high bit set instead of the app-specified address.
>> > =A0 =A0 */
>> > =A0 =A0 =A0IEEE80211_ADDR_COPY(gmac, bss->ni_macaddr);
>> > =A0 =A0 =A0gmac[0] |=3D 0x80;
>> > =A0 =A0 =A0mac =3D gmac;
>> > =A0} else
>> > =A0 =A0 mac =3D k->wk_macaddr;
>> >
>> > seems to indicate that for multiple VAPs the ath chips
>> > needs to be able to distinguish between broadcast keys
>> > by using a permutation of VAPs bssid.
>> >
>> > But in if_athvar.h the code does not seem complete
>> > and sc->sc_mcastkey is also set false.
>> >
>> > =A0#ifdef notyet
>> > =A0#define ath_hal_hasmcastkeysearch(_ah) \
>> > =A0 =A0 =A0 =A0 (ath_hal_getcapability(_ah, HAL_CAP_MCAST_KEYSRCH, 0, =
NULL) =3D=3D
>> > HAL_OK)
>> > =A0#define ath_hal_getmcastkeysearch(_ah) \
>> > =A0 =A0 =A0 =A0 (ath_hal_getcapability(_ah, HAL_CAP_MCAST_KEYSRCH, 1, =
NULL) =3D=3D
>> > HAL_OK)
>> > =A0#else
>> > =A0#define ath_hal_getmcastkeysearch(_ah) =A00
>> > =A0#endif
>> >
>> > I am using cards with an AR5212 which does seem to have
>> > multiple bssid support working so I hope they should also
>> > support mcastkeysearch capability. Maybe only some
>> > firmware revisions have this?
>> >
>> > Do you know what the status of the HAL code is for
>> > supporting this? Why it is commented out in if_athvar.h?
>>
>> Good work analyzing things; been a long time since I looked at this. =A0=
I
>> vaguely recall disabling mcastkey searching because of problems with
>> WEP. =A0I'm surprised WPA is broken as that was a standard test case.
>>
>> You can try enabling the notyet code and see if the right thing happens.
>> =A0I don't see any indication of the mac rev for your part but I expect =
it
>> supports this as it was only very early parts that had issues.
>>
>> If enabling the mcastkey search mechanism doesn't fix this I might've
>> broken things with changes to explicitly mark group keys when hostapd
>> plumbs their contents. =A0I recall doing this for mwl which doesn't have
>> an indexed key table like ath (it uses the mac address of the local bss
>> and/or associated station to find the data structure where crypto keys
>> are stored).
>>
>> I haven't looked at this stuff in a long time and can't setup a system
>> to test but if you keep pushing on this I'll try to help w/ advise.
>>
>> =A0 =A0 =A0 =A0Sam
>>
>
> Sam,
> I have the ath working with multiple hostap's and multicast key search an=
d
> also
> have the station side working, but I do not like how I got the station si=
de
> working.
> Let me explain what I have done and what I am tryinig to figure out now.
> I would like to submit a patch once I get the last issues resolved.
>
> In if_athvar.h I uncommented your macros
>
> =A0#define ath_hal_hasmcastkeysearch(_ah) \
> =A0 =A0 =A0 =A0(ath_hal_getcapability(_ah, HAL_CAP_MCAST_KEYSRCH, 0, NULL=
) =3D=3D
> HAL_OK)
> =A0#define ath_hal_getmcastkeysearch(_ah) \
> =A0 =A0 =A0 =A0(ath_hal_getcapability(_ah, HAL_CAP_MCAST_KEYSRCH, 1, NULL=
) =3D=3D
> HAL_OK)
>
> and added a macro
>
> #define ath_hal_setmcastkeysearch(_ah, _v) \
> =A0 =A0 =A0 ath_hal_setcapability(_ah, HAL_CAP_MCAST_KEYSRCH, 0, _v, NULL=
)
> In if_ath.c I added
>
> =A0 =A0 =A0/*
> =A0 =A0 =A0 =A0* if multicast key search is supported by device enable it
> =A0 =A0 =A0 =A0*/
> =A0 =A0 =A0 if (ath_hal_hasmcastkeysearch(sc->sc_ah)) {
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (!ath_hal_getmcastkeysearch(sc->sc_ah)) {
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ath_hal_setmcastkeysearch(sc-=
>sc_ah,1);
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 }
> =A0 =A0 =A0 }
>
> just before
>
> =A0 =A0 =A0sc->sc_mcastkey =3D ath_hal_getmcastkeysearch(ah);
>
> in ath_attach().
>
> Then in ieee80211_ioctl.c I changed ieee80211_ioctl_setkey() so
> =A0not to assign a slot when opering in hostap mode by changing
>
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0/*
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * Global slots start off w/o any assigned=
key index.
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * Force one here for consistency with IEE=
E80211_IOC_WEPKEY.
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 */
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (wk->wk_keyix =3D=3D IEEE80211_KEYIX_NO=
NE)
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0wk->wk_keyix =3D kid;
> to be
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0/*
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * Global slots start off w/o any assigned=
key index.
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * Force one here for consistency with IEE=
E80211_IOC_WEPKEY.
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 */
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (vap->iv_opmode !=3D IEEE80211_M_HOSTAP
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0&& wk->wk_keyix =3D=3D IEEE80211_KEYIX=
_NONE)
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0wk->wk_keyix =3D kid;
>
> to preserve the station mode operation I had to keep the key index
> assignment
> when not VAP is not operating in hostapd mode. This seems wrong to me.
>
> Now here is the question I am trying to understand. Group keys are specia=
l
> as they are used in only one direction only; sending on hostap and receiv=
ing
> on a station.
>
> The multicast key search code when operating in hostap mode permits the
> lookup
> of the key for encryption by sending VAP bssid on transmission of multica=
st
> traffic.
>
> Is there a corresponding station side multicast key search that would let
> the lookup
> of the encryption key be done on receive by looking up the sending VAP fo=
r
> received
> multicast traffic. If so it seems it could enable multiple stations also =
to
> work on one
> device.
>
> I noticed in the linux legacy hal ar5212_keycache.c the following
> code/comment
>
> =A0 =A0 =A0 =A0u_int32_t validBit =3D AR_KEYTABLE_VALID;
> =A0 =A0 =A0 =A0......
> =A0 =A0 =A0 =A0/*
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * =A0If upper layers have requested mcast=
MACaddr lookup,
> then
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * =A0signify this to the hw by setting th=
e (poorly named)
> validBit
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * =A0to 0. =A0Yes, really 0. The hardware=
specs,
> pcu_registers.txt, is
> =A0 =A0 =A0 =A0 * =A0has incorrectly named ValidBit. It should be called =
"Unicast".
> =A0 =A0 =A0 =A0 * =A0When the Key Cache entry is to decrypt Unicast frame=
s, then this
> =A0 =A0 =A0 =A0 * =A0bit should be '1'; for multicast and broadcast frame=
s, this bit
> is '0'.
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 */
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (mac[0] & 0x01) {
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0validBit =3D 0;
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
> =A0 =A0 =A0 =A0.....
> =A0 =A0 =A0 =A0OS_KC_WRITE(ah, AR_KEYTABLE_MAC0(entry), macLo);
> =A0 =A0 =A0 =A0OS_KC_WRITE(ah, AR_KEYTABLE_MAC1(entry), macHi | validBit)=
;
>
> where as in the freebsd hal in ar5212_keycache.c the AR_KEYTABLE_VALID
> is always set
>
> =A0 =A0 =A0 =A0OS_REG_WRITE(ah, AR_KEYTABLE_MAC0(entry), macLo);
> =A0 =A0 =A0 =A0OS_REG_WRITE(ah, AR_KEYTABLE_MAC1(entry), macHi |
> AR_KEYTABLE_VALID);
> by not setting the AR_KEYTABLE_VALID bit would the multicast key search c=
ode
> work for station mode? I am just guessing here, but seemed like a likely
> explaination?
>
> -Russ
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTinHoSv3yX_73HFaO8wg8tNzoyDbfaTH_y9vLOMQ>