Date: Sat, 27 Dec 2014 16:43:24 -0800 From: "David P. Discher" <dpd@dpdtech.com> To: freebsd-infiniband@freebsd.org Subject: FreeBSD/mthca (Mellanox MT25218 InfiniHostEx ) [poor] Performance, Firmware Upgrade, and SDP Message-ID: <C06A80D7-AE40-4987-9BB3-FC1D20100F9C@dpdtech.com>
next in thread | raw e-mail | index | archive | help
--Apple-Mail=_EE4F8816-B7C3-4E84-B92D-B8D826AF8202 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 On the performance side for Oded=E2=80=99s reply might be the answer to = this : - = https://lists.freebsd.org/pipermail/freebsd-infiniband/2014-December/00012= 4.html This hardware, especially some of the lower end cards like the MT25218 - = are trickling into the used marked, and are really attractively priced = for some small clusters (or in my case storage + Xen/Hypervisor) setups. = So, I=E2=80=99d love to see this code updated, even if its not = upstreamed right away, I=E2=80=99d be happy to do some testing with some = patches to -head or 10-stable. Using both 10-stable and -head, with IPoIB - I can=E2=80=99t get more = than 1.3 Gbps (in one direction). Even bringing up the second port, = doesn=E2=80=99t increase bandwidth. Even though the IPoIB is old, what = I=E2=80=99ve notice is that I=E2=80=99m maxing the CPU for one of the = three MSI-X IRQs (IRQ 260, the first one, which I think is = MTHCA_EQ_COMP, if they are allocated in order ). dmesg + boot verbose: ib_mthca0: HCA FW version 5.2.916 is old (5.3.000 is current). ib_mthca0: If you have problems, try updating your HCA FW. ib_mthca0: attempting to allocate 3 MSI-X vectors (32 supported) msi: routing MSI-X IRQ 260 to local APIC 0 vector 62 msi: routing MSI-X IRQ 261 to local APIC 0 vector 63 msi: routing MSI-X IRQ 262 to local APIC 0 vector 64 ib_mthca0: using IRQs 260-262 for MSI-X I assume this has nothing to really with the IPoIB part, as much as = interrupt handling in the hardware driver. Looking at the code, it = looks like the MSI-X vectors are hard-coded and can=E2=80=99t be = increased, event thought it seems like the card can support up to 32. = So, I think I know the answer, which is no, but can the number of MSI-X = vectors be increased, so the interrupt load can be spread over more CPUs = ? 2) Firmware upgrade for the HCA. I got all the tools from Mellanox, = however the syntax/directions for using flint says to point it to a = non-exisiting /dev path in FreeBSD. Can one use FreeBSD to flash the = firmware ? I could use some help getting the tools to talk to the card. and 3) SDP - Sockets Direct Protocol - while I=E2=80=99m waiting for = other things, I=E2=80=99d like to play with SDP. Quick googling : - http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-7.html > env LD_PRELOAD=3D/usr/lib/libibsdp.so iperf -s /usr/lib/libibsdp.so: Undefined symbol = =E2=80=9Cprogram_invocation_short_name=E2=80=9D However, doing this in FreeBSD, gives me a missing symbol. =46rom = searching around, program_invocation_short_name, is a gcc-libc thing. = I=E2=80=99ve searched around, can=E2=80=99t seem to find an easy way to = work around this. Any suggestions ? Thanks ! - David P. Discher http://davidpdischer.com/ AIM: DavidDPD | Y!M: daviddpdz --Apple-Mail=_EE4F8816-B7C3-4E84-B92D-B8D826AF8202 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUn1ItAAoJEEmwU6XuhYWOyVYIALA1ETjN4F1inbQz2h/LvyQN ZsPKbH8VLgbAG9r3iKos7aGDy7tIIpWOK6myaQ2nI6adyYI2VXMegKLrUzCg1ess J/l13jlqCbKRtrAFiKKQS4p/mL1OxnKWK8wbLHnbbt6X+CNH/AnUenbj+rqMYp4t 6aMFXPYHj4piFjT1qCH1872vNm0DWF888S0r+164oBcUJgLeTL/jIZTL0Mfv2p2V /pZYgxsVnefyC3k0lBJ1JlSE7teaL40MQNbD+3geWIYlAfW72phzvUzUk91mW191 GBAPJa45YlN+XzV+U2U5jJtPxSlZBrDCTNZyUkRyM0rVeniwGsgj3q8w2hcicZM= =Ns9v -----END PGP SIGNATURE----- --Apple-Mail=_EE4F8816-B7C3-4E84-B92D-B8D826AF8202--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C06A80D7-AE40-4987-9BB3-FC1D20100F9C>