Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 28 Mar 2019 23:43:38 +0000 (UTC)
From:      John Baldwin <jhb@FreeBSD.org>
To:        src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org
Subject:   svn commit: r345664 - in stable/11/sys: dev/cxgbe dev/cxgbe/crypto dev/cxgbe/cxgbei dev/cxgbe/firmware dev/cxgbe/tom modules/cxgbe/tom
Message-ID:  <201903282343.x2SNhcKH079312@repo.freebsd.org>

next in thread | raw e-mail | index | archive | help
Author: jhb
Date: Thu Mar 28 23:43:38 2019
New Revision: 345664
URL: https://svnweb.freebsd.org/changeset/base/345664

Log:
  MFC 330040,330041,330079,330884,330946,330947,331649,333068,333810,337722,
  340466,340468,340469,340473: Add TOE-based TLS offload.
  
  Note that this requires a modified OpenSSL library.
  
  330040:
  Fetch TLS key parameters from the firmware.
  
  The parameters describe how much of the adapter's memory is reserved for
  storing TLS keys.  The 'meminfo' sysctl now lists this region of adapter
  memory as 'TLS keys' if present.
  
  330041:
  Move ccr_aes_getdeckey() from ccr(4) to the cxgbe(4) driver.
  
  This routine will also be used by the TOE module to manage TLS keys.
  
  330079:
  Move #include for rijndael.h out of x86-specific region.
  
  The #include was added inside of the conditional by accident and the lack
  of it broke non-x86 builds.
  
  330884:
  Support for TLS offload of TOE connections on T6 adapters.
  
  The TOE engine in Chelsio T6 adapters supports offloading of TLS
  encryption and TCP segmentation for offloaded connections.  Sockets
  using TLS are required to use a set of custom socket options to upload
  RX and TX keys to the NIC and to enable RX processing.  Currently
  these socket options are implemented as TCP options in the vendor
  specific range.  A patched OpenSSL library will be made available in a
  port / package for use with the TLS TOE support.
  
  TOE sockets can either offload both transmit and reception of TLS
  records or just transmit.  TLS offload (both RX and TX) is enabled by
  setting the dev.t6nex.<x>.tls sysctl to 1 and requires TOE to be
  enabled on the relevant interface.  Transmit offload can be used on
  any "normal" or TLS TOE socket by using the custom socket option to
  program a transmit key.  This permits most TOE sockets to
  transparently offload TLS when applications use a patched SSL library
  (e.g. using LD_LIBRARY_PATH to request use of a patched OpenSSL
  library).  Receive offload can only be used with TOE sockets using the
  TLS mode.  The dev.t6nex.0.toe.tls_rx_ports sysctl can be set to a
  list of TCP port numbers.  Any connection with either a local or
  remote port number in that list will be created as a TLS socket rather
  than a plain TOE socket.  Note that although this sysctl accepts an
  arbitrary list of port numbers, the sysctl(8) tool is only able to set
  sysctl nodes to a single value.  A TLS socket will hang without
  receiving data if used by an application that is not using a patched
  SSL library.  Thus, the tls_rx_ports node should be used with care.
  For a server mostly concerned with offloading TLS transmit, this node
  is not needed as plain TOE sockets will fall back to software crypto
  when using an unpatched SSL library.
  
  New per-interface statistics nodes are added giving counts of TLS
  packets and payload bytes (payload bytes do not include TLS headers or
  authentication tags/MACs) offloaded via the TOE engine, e.g.:
  
  dev.cc.0.stats.rx_tls_octets: 149
  dev.cc.0.stats.rx_tls_records: 13
  dev.cc.0.stats.tx_tls_octets: 26501823
  dev.cc.0.stats.tx_tls_records: 1620
  
  TLS transmit work requests are constructed by a new variant of
  t4_push_frames() called t4_push_tls_records() in tom/t4_tls.c.
  
  TLS transmit work requests require a buffer containing IVs.  If the
  IVs are too large to fit into the work request, a separate buffer is
  allocated when constructing a work request.  This buffer is associated
  with the transmit descriptor and freed when the descriptor is ACKed by
  the adapter.
  
  Received TLS frames use two new CPL messages.  The first message is a
  CPL_TLS_DATA containing the decryped payload of a single TLS record.
  The handler places the mbuf containing the received payload on an
  mbufq in the TOE pcb.  The second message is a CPL_RX_TLS_CMP message
  which includes a copy of the TLS header and indicates if there were
  any errors.  The handler for this message places the TLS header into
  the socket buffer followed by the saved mbuf with the payload data.
  Both of these handlers are contained in tom/t4_tls.c.
  
  A few routines were exposed from t4_cpl_io.c for use by t4_tls.c
  including send_rx_credits(), a new send_rx_modulate(), and
  t4_close_conn().
  
  TLS keys for both transmit and receive are stored in onboard memory
  in the NIC in the "TLS keys" memory region.
  
  In some cases a TLS socket can hang with pending data available in the
  NIC that is not delivered to the host.  As a workaround, TLS sockets
  are more aggressive about sending CPL_RX_DATA_ACK messages anytime that
  any data is read from a TLS socket.  In addition, a fallback timer will
  periodically send CPL_RX_DATA_ACK messages to the NIC for connections
  that are still in the handshake phase.  Once the connection has
  finished the handshake and programmed RX keys via the socket option,
  the timer is stopped.
  
  A new function select_ulp_mode() is used to determine what sub-mode a
  given TOE socket should use (plain TOE, DDP, or TLS).  The existing
  set_tcpddp_ulp_mode() function has been renamed to set_ulp_mode() and
  handles initialization of TLS-specific state when necessary in
  addition to DDP-specific state.
  
  Since TLS sockets do not receive individual TCP segments but always
  receive full TLS records, they can receive more data than is available
  in the current window (e.g. if a 16k TLS record is received but the
  socket buffer is itself 16k).  To cope with this, just drop the window
  to 0 when this happens, but track the overage and "eat" the overage as
  it is read from the socket buffer not opening the window (or adding
  rx_credits) for the overage bytes.
  
  330946:
  Remove TLS-related inlines from t4_tom.h to fix iw_cxgbe(4) build.
  
  - Remove the one use of is_tls_offload() and the function.  AIO special
    handling only needs to be disabled when a TOE socket is actively doing
    TLS offload on transmit.  The TOE socket's mode (which affects receive
    operation) doesn't matter, so remove the check for the socket's mode and
    only check if a TOE socket has TLS transmit keys configured to determine
    if an AIO write request should fall back to the normal socket handling
    instead of the TOE fast path.
  - Move can_tls_offload() into t4_tls.c.  It is not used in critical paths,
    so inlining isn't that important.  Change return type to bool while here.
  
  330947:
  Fix the check for an empty send socket buffer on a TOE TLS socket.
  
  Compare sbavail() with the cached sb_off of already-sent data instead of
  always comparing with zero.  This will correctly close the connection and
  send the FIN if the socket buffer contains some previously-sent data but
  no unsent data.
  
  331649:
  Use the offload transmit queue to set flags on TLS connections.
  
  Requests to modify the state of TLS connections need to be sent on the
  same queue as TLS record transmit requests to ensure ordering.
  
  However, in order to use the offload transmit queue in t4_set_tcb_field(),
  the function needs to be updated to do proper flow control / credit
  management when queueing a request to an offload queue.  This required
  passing a pointer to the toepcb itself to this function, so while here
  remove the 'tid' and 'iqid' parameters and obtain those values from the
  toepcb in t4_set_tcb_field() itself.
  
  333068:
  Use the correct key address when renegotiating the transmit key.
  
  Previously, get_keyid() was returning the address of the receive key
  instead of the transmit key when renegotiating the transmit key.  This
  could either hang the card (if a connection was only offloading TLS TX
  and thus had a receive key address of -1) or cause the connection to
  fail by overwriting the wrong key (if both RX and TX TLS were
  offloaded).
  
  333810:
  Be more robust against garbage input on a TOE TLS TX socket.
  
  If a socket is closed or shutdown and a partial record (or what
  appears to be a partial record) is waiting in the socket buffer,
  discard the partial record and close the connection rather than
  waiting forever for the rest of the record.
  
  337722:
  Whitespace nit in t4_tom.h
  
  340466:
  Move the TLS key map into the adapter softc so non-TOE code can use it.
  
  340468:
  Change the quantum for TLS key addresses to 32 bytes.
  
  The addresses passed when reading and writing keys are always shifted
  right by 5 as the memory locations are addressed in 32-byte chunks, so
  the quantum needs to be 32, not 8.
  
  340469:
  Remove bogus roundup2() of the key programming work request header.
  
  The key context is always placed immediately after the work request
  header.  The total work request length has to be rounded up by 16
  however.
  
  340473:
  Restore the <sys/vmem.h> header to fix build of cxgbe(4) TOM.
  
  vmem's are not just used for TLS memory in TOM and the #include actually
  predates the TLS code so should not have been removed when the TLS vmem
  moved in r340466.
  
  Sponsored by:	Chelsio Communications

Added:
  stable/11/sys/dev/cxgbe/tom/t4_tls.c
     - copied, changed from r330884, head/sys/dev/cxgbe/tom/t4_tls.c
  stable/11/sys/dev/cxgbe/tom/t4_tls.h
     - copied unchanged from r330884, head/sys/dev/cxgbe/tom/t4_tls.h
Modified:
  stable/11/sys/dev/cxgbe/adapter.h
  stable/11/sys/dev/cxgbe/crypto/t4_crypto.c
  stable/11/sys/dev/cxgbe/cxgbei/icl_cxgbei.c
  stable/11/sys/dev/cxgbe/firmware/t6fw_cfg.txt
  stable/11/sys/dev/cxgbe/offload.h
  stable/11/sys/dev/cxgbe/t4_main.c
  stable/11/sys/dev/cxgbe/tom/t4_connect.c
  stable/11/sys/dev/cxgbe/tom/t4_cpl_io.c
  stable/11/sys/dev/cxgbe/tom/t4_ddp.c
  stable/11/sys/dev/cxgbe/tom/t4_listen.c
  stable/11/sys/dev/cxgbe/tom/t4_tom.c
  stable/11/sys/dev/cxgbe/tom/t4_tom.h
  stable/11/sys/modules/cxgbe/tom/Makefile
Directory Properties:
  stable/11/   (props changed)

Modified: stable/11/sys/dev/cxgbe/adapter.h
==============================================================================
--- stable/11/sys/dev/cxgbe/adapter.h	Thu Mar 28 22:19:09 2019	(r345663)
+++ stable/11/sys/dev/cxgbe/adapter.h	Thu Mar 28 23:43:38 2019	(r345664)
@@ -39,6 +39,7 @@
 #include <sys/malloc.h>
 #include <sys/rwlock.h>
 #include <sys/sx.h>
+#include <sys/vmem.h>
 #include <vm/uma.h>
 
 #include <dev/pci/pcivar.h>
@@ -297,6 +298,10 @@ struct port_info {
  	struct port_stats stats;
 	u_int tnl_cong_drops;
 	u_int tx_parse_error;
+	u_long	tx_tls_records;
+	u_long	tx_tls_octets;
+	u_long	rx_tls_records;
+	u_long	rx_tls_octets;
 
 	struct callout tick;
 };
@@ -805,6 +810,7 @@ struct adapter {
 	void *ccr_softc;	/* (struct ccr_softc *) */
 	struct l2t_data *l2t;	/* L2 table */
 	struct tid_info tids;
+	vmem_t *key_map;
 
 	uint8_t doorbells;
 	int offload_map;	/* ports with IFCAP_TOE enabled */
@@ -1091,6 +1097,7 @@ void t4_os_link_changed(struct port_info *);
 void t4_iterate(void (*)(struct adapter *, void *), void *);
 void t4_init_devnames(struct adapter *);
 void t4_add_adapter(struct adapter *);
+void t4_aes_getdeckey(void *, const void *, unsigned int);
 int t4_detach_common(device_t);
 int t4_filter_rpl(struct sge_iq *, const struct rss_header *, struct mbuf *);
 int t4_map_bars_0_and_4(struct adapter *);

Modified: stable/11/sys/dev/cxgbe/crypto/t4_crypto.c
==============================================================================
--- stable/11/sys/dev/cxgbe/crypto/t4_crypto.c	Thu Mar 28 22:19:09 2019	(r345663)
+++ stable/11/sys/dev/cxgbe/crypto/t4_crypto.c	Thu Mar 28 23:43:38 2019	(r345664)
@@ -1815,46 +1815,7 @@ ccr_aes_check_keylen(int alg, int klen)
 	return (0);
 }
 
-/*
- * Borrowed from cesa_prep_aes_key().  We should perhaps have a public
- * function to generate this instead.
- *
- * NB: The crypto engine wants the words in the decryption key in reverse
- * order.
- */
 static void
-ccr_aes_getdeckey(void *dec_key, const void *enc_key, unsigned int kbits)
-{
-	uint32_t ek[4 * (RIJNDAEL_MAXNR + 1)];
-	uint32_t *dkey;
-	int i;
-
-	rijndaelKeySetupEnc(ek, enc_key, kbits);
-	dkey = dec_key;
-	dkey += (kbits / 8) / 4;
-
-	switch (kbits) {
-	case 128:
-		for (i = 0; i < 4; i++)
-			*--dkey = htobe32(ek[4 * 10 + i]);
-		break;
-	case 192:
-		for (i = 0; i < 2; i++)
-			*--dkey = htobe32(ek[4 * 11 + 2 + i]);
-		for (i = 0; i < 4; i++)
-			*--dkey = htobe32(ek[4 * 12 + i]);
-		break;
-	case 256:
-		for (i = 0; i < 4; i++)
-			*--dkey = htobe32(ek[4 * 13 + i]);
-		for (i = 0; i < 4; i++)
-			*--dkey = htobe32(ek[4 * 14 + i]);
-		break;
-	}
-	MPASS(dkey == dec_key);
-}
-
-static void
 ccr_aes_setkey(struct ccr_session *s, int alg, const void *key, int klen)
 {
 	unsigned int ck_size, iopad_size, kctx_flits, kctx_len, kbits, mk_size;
@@ -1883,7 +1844,7 @@ ccr_aes_setkey(struct ccr_session *s, int alg, const v
 	switch (alg) {
 	case CRYPTO_AES_CBC:
 	case CRYPTO_AES_XTS:
-		ccr_aes_getdeckey(s->blkcipher.deckey, key, kbits);
+		t4_aes_getdeckey(s->blkcipher.deckey, key, kbits);
 		break;
 	}
 

Modified: stable/11/sys/dev/cxgbe/cxgbei/icl_cxgbei.c
==============================================================================
--- stable/11/sys/dev/cxgbe/cxgbei/icl_cxgbei.c	Thu Mar 28 22:19:09 2019	(r345663)
+++ stable/11/sys/dev/cxgbe/cxgbei/icl_cxgbei.c	Thu Mar 28 23:43:38 2019	(r345664)
@@ -595,9 +595,9 @@ set_ulp_mode_iscsi(struct adapter *sc, struct toepcb *
 	CTR4(KTR_CXGBE, "%s: tid %u, ULP_MODE_ISCSI, CRC hdr=%d data=%d",
 	    __func__, toep->tid, hcrc, dcrc);
 
-	t4_set_tcb_field(sc, toep->ctrlq, toep->tid, W_TCB_ULP_TYPE,
+	t4_set_tcb_field(sc, toep->ctrlq, toep, W_TCB_ULP_TYPE,
 	    V_TCB_ULP_TYPE(M_TCB_ULP_TYPE) | V_TCB_ULP_RAW(M_TCB_ULP_RAW), val,
-	    0, 0, toep->ofld_rxq->iq.abs_id);
+	    0, 0);
 }
 
 /*

Modified: stable/11/sys/dev/cxgbe/firmware/t6fw_cfg.txt
==============================================================================
--- stable/11/sys/dev/cxgbe/firmware/t6fw_cfg.txt	Thu Mar 28 22:19:09 2019	(r345663)
+++ stable/11/sys/dev/cxgbe/firmware/t6fw_cfg.txt	Thu Mar 28 23:43:38 2019	(r345664)
@@ -163,10 +163,12 @@
 	nserver = 512
 	nhpfilter = 0
 	nhash = 16384
-	protocol = ofld, rddp, rdmac, iscsi_initiator_pdu, iscsi_target_pdu, iscsi_t10dif, crypto_lookaside
+	protocol = ofld, rddp, rdmac, iscsi_initiator_pdu, iscsi_target_pdu, iscsi_t10dif, tlskeys, crypto_lookaside
 	tp_l2t = 4096
 	tp_ddp = 2
 	tp_ddp_iscsi = 2
+	tp_tls_key = 3
+	tp_tls_mxrxsize = 17408    # 16384 + 1024, governs max rx data, pm max xfer len, rx coalesce sizes
 	tp_stag = 2
 	tp_pbl = 5
 	tp_rq = 7
@@ -273,7 +275,7 @@
 
 [fini]
 	version = 0x1
-	checksum = 0x7191019f
+	checksum = 0x9e8952d2
 #
 # $FreeBSD$
 #

Modified: stable/11/sys/dev/cxgbe/offload.h
==============================================================================
--- stable/11/sys/dev/cxgbe/offload.h	Thu Mar 28 22:19:09 2019	(r345663)
+++ stable/11/sys/dev/cxgbe/offload.h	Thu Mar 28 23:43:38 2019	(r345664)
@@ -124,6 +124,7 @@ struct t4_virt_res {                      /* virtualiz
 	struct t4_range srq;
 	struct t4_range ocq;
 	struct t4_range l2t;
+	struct t4_range key;
 };
 
 enum {
@@ -148,6 +149,9 @@ struct tom_tunables {
 	int sndbuf;
 	int ddp;
 	int rx_coalesce;
+	int tls;
+	int *tls_rx_ports;
+	int num_tls_rx_ports;
 	int tx_align;
 	int tx_zcopy;
 };

Modified: stable/11/sys/dev/cxgbe/t4_main.c
==============================================================================
--- stable/11/sys/dev/cxgbe/t4_main.c	Thu Mar 28 22:19:09 2019	(r345663)
+++ stable/11/sys/dev/cxgbe/t4_main.c	Thu Mar 28 23:43:38 2019	(r345664)
@@ -66,6 +66,7 @@ __FBSDID("$FreeBSD$");
 #include <vm/vm.h>
 #include <vm/pmap.h>
 #endif
+#include <crypto/rijndael/rijndael.h>
 #ifdef DDB
 #include <ddb/ddb.h>
 #include <ddb/db_lex.h>
@@ -667,6 +668,7 @@ static int sysctl_wcwr_stats(SYSCTL_HANDLER_ARGS);
 static int sysctl_tc_params(SYSCTL_HANDLER_ARGS);
 #endif
 #ifdef TCP_OFFLOAD
+static int sysctl_tls_rx_ports(SYSCTL_HANDLER_ARGS);
 static int sysctl_tp_tick(SYSCTL_HANDLER_ARGS);
 static int sysctl_tp_dack_timer(SYSCTL_HANDLER_ARGS);
 static int sysctl_tp_timer(SYSCTL_HANDLER_ARGS);
@@ -1215,6 +1217,9 @@ t4_attach(device_t dev)
 
 	t4_init_l2t(sc, M_WAITOK);
 	t4_init_tx_sched(sc);
+	if (sc->vres.key.size != 0)
+		sc->key_map = vmem_create("T4TLS key map", sc->vres.key.start,
+		    sc->vres.key.size, 32, 0, M_FIRSTFIT | M_WAITOK);
 
 	/*
 	 * Second pass over the ports.  This time we know the number of rx and
@@ -1499,6 +1504,8 @@ t4_detach_common(device_t dev)
 
 	if (sc->l2t)
 		t4_free_l2t(sc->l2t);
+	if (sc->key_map)
+		vmem_destroy(sc->key_map);
 
 #ifdef TCP_OFFLOAD
 	free(sc->sge.ofld_rxq, M_CXGBE);
@@ -1515,6 +1522,7 @@ t4_detach_common(device_t dev)
 	free(sc->sge.iqmap, M_CXGBE);
 	free(sc->sge.eqmap, M_CXGBE);
 	free(sc->tids.ftid_tab, M_CXGBE);
+	free(sc->tt.tls_rx_ports, M_CXGBE);
 	t4_destroy_dma_tag(sc);
 	if (mtx_initialized(&sc->sc_lock)) {
 		sx_xlock(&t4_list_lock);
@@ -3981,6 +3989,18 @@ get_params__post_init(struct adapter *sc)
 		sc->vres.iscsi.start = val[0];
 		sc->vres.iscsi.size = val[1] - val[0] + 1;
 	}
+	if (sc->cryptocaps & FW_CAPS_CONFIG_TLSKEYS) {
+		param[0] = FW_PARAM_PFVF(TLS_START);
+		param[1] = FW_PARAM_PFVF(TLS_END);
+		rc = -t4_query_params(sc, sc->mbox, sc->pf, 0, 2, param, val);
+		if (rc != 0) {
+			device_printf(sc->dev,
+			    "failed to query TLS parameters: %d.\n", rc);
+			return (rc);
+		}
+		sc->vres.key.start = val[0];
+		sc->vres.key.size = val[1] - val[0] + 1;
+	}
 
 	t4_init_sge_params(sc);
 
@@ -5780,6 +5800,14 @@ t4_sysctls(struct adapter *sc)
 		SYSCTL_ADD_INT(ctx, children, OID_AUTO, "rx_coalesce",
 		    CTLFLAG_RW, &sc->tt.rx_coalesce, 0, "receive coalescing");
 
+		sc->tt.tls = 0;
+		SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tls", CTLFLAG_RW,
+		    &sc->tt.tls, 0, "Inline TLS allowed");
+
+		SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "tls_rx_ports",
+		    CTLTYPE_INT | CTLFLAG_RW, sc, 0, sysctl_tls_rx_ports,
+		    "I", "TCP ports that use inline TLS+TOE RX");
+
 		sc->tt.tx_align = 1;
 		SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_align",
 		    CTLFLAG_RW, &sc->tt.tx_align, 0, "chop and align payload");
@@ -6183,6 +6211,19 @@ cxgbe_sysctls(struct port_info *pi)
 	    "# of buffer-group 3 truncated packets");
 
 #undef SYSCTL_ADD_T4_PORTSTAT
+
+	SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "tx_tls_records",
+	    CTLFLAG_RD, &pi->tx_tls_records,
+	    "# of TLS records transmitted");
+	SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "tx_tls_octets",
+	    CTLFLAG_RD, &pi->tx_tls_octets,
+	    "# of payload octets in transmitted TLS records");
+	SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "rx_tls_records",
+	    CTLFLAG_RD, &pi->rx_tls_records,
+	    "# of TLS records received");
+	SYSCTL_ADD_ULONG(ctx, children, OID_AUTO, "rx_tls_octets",
+	    CTLFLAG_RD, &pi->rx_tls_octets,
+	    "# of payload octets in received TLS records");
 }
 
 static int
@@ -7443,7 +7484,7 @@ sysctl_meminfo(SYSCTL_HANDLER_ARGS)
 		"TDDP region:", "TPT region:", "STAG region:", "RQ region:",
 		"RQUDP region:", "PBL region:", "TXPBL region:",
 		"DBVFIFO region:", "ULPRX state:", "ULPTX state:",
-		"On-chip queues:"
+		"On-chip queues:", "TLS keys:",
 	};
 	struct mem_desc avail[4];
 	struct mem_desc mem[nitems(region) + 3];	/* up to 3 holes */
@@ -7583,6 +7624,13 @@ sysctl_meminfo(SYSCTL_HANDLER_ARGS)
 		md->idx = nitems(region);  /* hide it */
 	md++;
 
+	md->base = sc->vres.key.start;
+	if (sc->vres.key.size)
+		md->limit = md->base + sc->vres.key.size - 1;
+	else
+		md->idx = nitems(region);  /* hide it */
+	md++;
+
 	/* add any address-space holes, there can be up to 3 */
 	for (n = 0; n < i - 1; n++)
 		if (avail[n].limit < avail[n + 1].base)
@@ -8674,6 +8722,68 @@ done:
 #endif
 
 #ifdef TCP_OFFLOAD
+static int
+sysctl_tls_rx_ports(SYSCTL_HANDLER_ARGS)
+{
+	struct adapter *sc = arg1;
+	int *old_ports, *new_ports;
+	int i, new_count, rc;
+
+	if (req->newptr == NULL && req->oldptr == NULL)
+		return (SYSCTL_OUT(req, NULL, imax(sc->tt.num_tls_rx_ports, 1) *
+		    sizeof(sc->tt.tls_rx_ports[0])));
+
+	rc = begin_synchronized_op(sc, NULL, SLEEP_OK | INTR_OK, "t4tlsrx");
+	if (rc)
+		return (rc);
+
+	if (sc->tt.num_tls_rx_ports == 0) {
+		i = -1;
+		rc = SYSCTL_OUT(req, &i, sizeof(i));
+	} else
+		rc = SYSCTL_OUT(req, sc->tt.tls_rx_ports,
+		    sc->tt.num_tls_rx_ports * sizeof(sc->tt.tls_rx_ports[0]));
+	if (rc == 0 && req->newptr != NULL) {
+		new_count = req->newlen / sizeof(new_ports[0]);
+		new_ports = malloc(new_count * sizeof(new_ports[0]), M_CXGBE,
+		    M_WAITOK);
+		rc = SYSCTL_IN(req, new_ports, new_count *
+		    sizeof(new_ports[0]));
+		if (rc)
+			goto err;
+
+		/* Allow setting to a single '-1' to clear the list. */
+		if (new_count == 1 && new_ports[0] == -1) {
+			ADAPTER_LOCK(sc);
+			old_ports = sc->tt.tls_rx_ports;
+			sc->tt.tls_rx_ports = NULL;
+			sc->tt.num_tls_rx_ports = 0;
+			ADAPTER_UNLOCK(sc);
+			free(old_ports, M_CXGBE);
+		} else {
+			for (i = 0; i < new_count; i++) {
+				if (new_ports[i] < 1 ||
+				    new_ports[i] > IPPORT_MAX) {
+					rc = EINVAL;
+					goto err;
+				}
+			}
+
+			ADAPTER_LOCK(sc);
+			old_ports = sc->tt.tls_rx_ports;
+			sc->tt.tls_rx_ports = new_ports;
+			sc->tt.num_tls_rx_ports = new_count;
+			ADAPTER_UNLOCK(sc);
+			free(old_ports, M_CXGBE);
+			new_ports = NULL;
+		}
+	err:
+		free(new_ports, M_CXGBE);
+	}
+	end_synchronized_op(sc, 0);
+	return (rc);
+}
+
 static void
 unit_conv(char *buf, size_t len, u_int val, u_int factor)
 {
@@ -10628,6 +10738,44 @@ DB_FUNC(tcb, db_show_t4tcb, db_t4_table, CS_OWN, NULL)
 	t4_dump_tcb(device_get_softc(dev), tid);
 }
 #endif
+
+/*
+ * Borrowed from cesa_prep_aes_key().
+ *
+ * NB: The crypto engine wants the words in the decryption key in reverse
+ * order.
+ */
+void
+t4_aes_getdeckey(void *dec_key, const void *enc_key, unsigned int kbits)
+{
+	uint32_t ek[4 * (RIJNDAEL_MAXNR + 1)];
+	uint32_t *dkey;
+	int i;
+
+	rijndaelKeySetupEnc(ek, enc_key, kbits);
+	dkey = dec_key;
+	dkey += (kbits / 8) / 4;
+
+	switch (kbits) {
+	case 128:
+		for (i = 0; i < 4; i++)
+			*--dkey = htobe32(ek[4 * 10 + i]);
+		break;
+	case 192:
+		for (i = 0; i < 2; i++)
+			*--dkey = htobe32(ek[4 * 11 + 2 + i]);
+		for (i = 0; i < 4; i++)
+			*--dkey = htobe32(ek[4 * 12 + i]);
+		break;
+	case 256:
+		for (i = 0; i < 4; i++)
+			*--dkey = htobe32(ek[4 * 13 + i]);
+		for (i = 0; i < 4; i++)
+			*--dkey = htobe32(ek[4 * 14 + i]);
+		break;
+	}
+	MPASS(dkey == dec_key);
+}
 
 static struct sx mlu;	/* mod load unload */
 SX_SYSINIT(cxgbe_mlu, &mlu, "cxgbe mod load/unload");

Modified: stable/11/sys/dev/cxgbe/tom/t4_connect.c
==============================================================================
--- stable/11/sys/dev/cxgbe/tom/t4_connect.c	Thu Mar 28 22:19:09 2019	(r345663)
+++ stable/11/sys/dev/cxgbe/tom/t4_connect.c	Thu Mar 28 23:43:38 2019	(r345664)
@@ -140,6 +140,10 @@ do_act_establish(struct sge_iq *iq, const struct rss_h
 	}
 
 	make_established(toep, cpl->snd_isn, cpl->rcv_isn, cpl->tcp_opt);
+
+	if (toep->ulp_mode == ULP_MODE_TLS)
+		tls_establish(toep);
+
 done:
 	INP_WUNLOCK(inp);
 	CURVNET_RESTORE();
@@ -266,6 +270,11 @@ calc_opt2a(struct socket *so, struct toepcb *toep)
 	if (toep->ulp_mode == ULP_MODE_TCPDDP)
 		opt2 |= F_RX_FC_VALID | F_RX_FC_DDP;
 #endif
+	if (toep->ulp_mode == ULP_MODE_TLS) {
+		opt2 |= F_RX_FC_VALID;
+		opt2 &= ~V_RX_COALESCE(M_RX_COALESCE);
+		opt2 |= F_RX_FC_DISABLE;
+	}
 
 	return (htobe32(opt2));
 }
@@ -376,10 +385,7 @@ t4_connect(struct toedev *tod, struct socket *so, stru
 		DONT_OFFLOAD_ACTIVE_OPEN(ENOMEM);
 
 	toep->vnet = so->so_vnet;
-	if (sc->tt.ddp && (so->so_options & SO_NO_DDP) == 0)
-		set_tcpddp_ulp_mode(toep);
-	else
-		toep->ulp_mode = ULP_MODE_NONE;
+	set_ulp_mode(toep, select_ulp_mode(so, sc));
 	SOCKBUF_LOCK(&so->so_rcv);
 	/* opt0 rcv_bufsiz initially, assumes its normal meaning later */
 	toep->rx_credits = min(select_rcv_wnd(so) >> 10, M_RCV_BUFSIZ);

Modified: stable/11/sys/dev/cxgbe/tom/t4_cpl_io.c
==============================================================================
--- stable/11/sys/dev/cxgbe/tom/t4_cpl_io.c	Thu Mar 28 22:19:09 2019	(r345663)
+++ stable/11/sys/dev/cxgbe/tom/t4_cpl_io.c	Thu Mar 28 23:43:38 2019	(r345664)
@@ -82,9 +82,6 @@ VNET_DECLARE(int, tcp_autorcvbuf_inc);
 VNET_DECLARE(int, tcp_autorcvbuf_max);
 #define V_tcp_autorcvbuf_max VNET(tcp_autorcvbuf_max)
 
-#define	IS_AIOTX_MBUF(m)						\
-	((m)->m_flags & M_EXT && (m)->m_ext.ext_flags & EXT_FLAG_AIOTX)
-
 static void	t4_aiotx_cancel(struct kaiocb *job);
 static void	t4_aiotx_queue_toep(struct toepcb *toep);
 
@@ -115,7 +112,7 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_par
 {
 	struct wrqe *wr;
 	struct fw_flowc_wr *flowc;
-	unsigned int nparams = ftxp ? 8 : 6, flowclen;
+	unsigned int nparams, flowclen, paramidx;
 	struct vi_info *vi = toep->vi;
 	struct port_info *pi = vi->pi;
 	struct adapter *sc = pi->adapter;
@@ -125,6 +122,15 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_par
 	KASSERT(!(toep->flags & TPF_FLOWC_WR_SENT),
 	    ("%s: flowc for tid %u sent already", __func__, toep->tid));
 
+	if (ftxp != NULL)
+		nparams = 8;
+	else
+		nparams = 6;
+	if (toep->ulp_mode == ULP_MODE_TLS)
+		nparams++;
+	if (toep->tls.fcplenmax != 0)
+		nparams++;
+
 	flowclen = sizeof(*flowc) + nparams * sizeof(struct fw_flowc_mnemval);
 
 	wr = alloc_wrqe(roundup2(flowclen, 16), toep->ofld_txq);
@@ -140,39 +146,45 @@ send_flowc_wr(struct toepcb *toep, struct flowc_tx_par
 	flowc->flowid_len16 = htonl(V_FW_WR_LEN16(howmany(flowclen, 16)) |
 	    V_FW_WR_FLOWID(toep->tid));
 
-	flowc->mnemval[0].mnemonic = FW_FLOWC_MNEM_PFNVFN;
-	flowc->mnemval[0].val = htobe32(pfvf);
-	flowc->mnemval[1].mnemonic = FW_FLOWC_MNEM_CH;
-	flowc->mnemval[1].val = htobe32(pi->tx_chan);
-	flowc->mnemval[2].mnemonic = FW_FLOWC_MNEM_PORT;
-	flowc->mnemval[2].val = htobe32(pi->tx_chan);
-	flowc->mnemval[3].mnemonic = FW_FLOWC_MNEM_IQID;
-	flowc->mnemval[3].val = htobe32(toep->ofld_rxq->iq.abs_id);
+#define FLOWC_PARAM(__m, __v) \
+	do { \
+		flowc->mnemval[paramidx].mnemonic = FW_FLOWC_MNEM_##__m; \
+		flowc->mnemval[paramidx].val = htobe32(__v); \
+		paramidx++; \
+	} while (0)
+
+	paramidx = 0;
+
+	FLOWC_PARAM(PFNVFN, pfvf);
+	FLOWC_PARAM(CH, pi->tx_chan);
+	FLOWC_PARAM(PORT, pi->tx_chan);
+	FLOWC_PARAM(IQID, toep->ofld_rxq->iq.abs_id);
 	if (ftxp) {
 		uint32_t sndbuf = min(ftxp->snd_space, sc->tt.sndbuf);
 
-		flowc->mnemval[4].mnemonic = FW_FLOWC_MNEM_SNDNXT;
-		flowc->mnemval[4].val = htobe32(ftxp->snd_nxt);
-		flowc->mnemval[5].mnemonic = FW_FLOWC_MNEM_RCVNXT;
-		flowc->mnemval[5].val = htobe32(ftxp->rcv_nxt);
-		flowc->mnemval[6].mnemonic = FW_FLOWC_MNEM_SNDBUF;
-		flowc->mnemval[6].val = htobe32(sndbuf);
-		flowc->mnemval[7].mnemonic = FW_FLOWC_MNEM_MSS;
-		flowc->mnemval[7].val = htobe32(ftxp->mss);
+		FLOWC_PARAM(SNDNXT, ftxp->snd_nxt);
+		FLOWC_PARAM(RCVNXT, ftxp->rcv_nxt);
+		FLOWC_PARAM(SNDBUF, sndbuf);
+		FLOWC_PARAM(MSS, ftxp->mss);
 
 		CTR6(KTR_CXGBE,
 		    "%s: tid %u, mss %u, sndbuf %u, snd_nxt 0x%x, rcv_nxt 0x%x",
 		    __func__, toep->tid, ftxp->mss, sndbuf, ftxp->snd_nxt,
 		    ftxp->rcv_nxt);
 	} else {
-		flowc->mnemval[4].mnemonic = FW_FLOWC_MNEM_SNDBUF;
-		flowc->mnemval[4].val = htobe32(512);
-		flowc->mnemval[5].mnemonic = FW_FLOWC_MNEM_MSS;
-		flowc->mnemval[5].val = htobe32(512);
+		FLOWC_PARAM(SNDBUF, 512);
+		FLOWC_PARAM(MSS, 512);
 
 		CTR2(KTR_CXGBE, "%s: tid %u", __func__, toep->tid);
 	}
+	if (toep->ulp_mode == ULP_MODE_TLS)
+		FLOWC_PARAM(ULP_MODE, toep->ulp_mode);
+	if (toep->tls.fcplenmax != 0)
+		FLOWC_PARAM(TXDATAPLEN_MAX, toep->tls.fcplenmax);
+#undef FLOWC_PARAM
 
+	KASSERT(paramidx == nparams, ("nparams mismatch"));
+
 	txsd->tx_credits = howmany(flowclen, 16);
 	txsd->plen = 0;
 	KASSERT(toep->tx_credits >= txsd->tx_credits && toep->txsd_avail > 0,
@@ -360,7 +372,7 @@ make_established(struct toepcb *toep, uint32_t snd_isn
 	soisconnected(so);
 }
 
-static int
+int
 send_rx_credits(struct adapter *sc, struct toepcb *toep, int credits)
 {
 	struct wrqe *wr;
@@ -382,6 +394,23 @@ send_rx_credits(struct adapter *sc, struct toepcb *toe
 }
 
 void
+send_rx_modulate(struct adapter *sc, struct toepcb *toep)
+{
+	struct wrqe *wr;
+	struct cpl_rx_data_ack *req;
+
+	wr = alloc_wrqe(sizeof(*req), toep->ctrlq);
+	if (wr == NULL)
+		return;
+	req = wrtod(wr);
+
+	INIT_TP_WR_MIT_CPL(req, CPL_RX_DATA_ACK, toep->tid);
+	req->credit_dack = htobe32(F_RX_MODULATE_RX);
+
+	t4_wrq_tx(sc, wr);
+}
+
+void
 t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
 {
 	struct adapter *sc = tod->tod_softc;
@@ -398,8 +427,18 @@ t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
 	    ("%s: sb %p has more data (%d) than last time (%d).",
 	    __func__, sb, sbused(sb), toep->sb_cc));
 
-	toep->rx_credits += toep->sb_cc - sbused(sb);
+	credits = toep->sb_cc - sbused(sb);
 	toep->sb_cc = sbused(sb);
+	if (toep->ulp_mode == ULP_MODE_TLS) {
+		if (toep->tls.rcv_over >= credits) {
+			toep->tls.rcv_over -= credits;
+			credits = 0;
+		} else {
+			credits -= toep->tls.rcv_over;
+			toep->tls.rcv_over = 0;
+		}
+	}
+	toep->rx_credits += credits;
 
 	if (toep->rx_credits > 0 &&
 	    (tp->rcv_wnd <= 32 * 1024 || toep->rx_credits >= 64 * 1024 ||
@@ -410,7 +449,8 @@ t4_rcvd_locked(struct toedev *tod, struct tcpcb *tp)
 		toep->rx_credits -= credits;
 		tp->rcv_wnd += credits;
 		tp->rcv_adv += credits;
-	}
+	} else if (toep->flags & TPF_FORCE_CREDITS)
+		send_rx_modulate(sc, toep);
 }
 
 void
@@ -428,8 +468,8 @@ t4_rcvd(struct toedev *tod, struct tcpcb *tp)
 /*
  * Close a connection by sending a CPL_CLOSE_CON_REQ message.
  */
-static int
-close_conn(struct adapter *sc, struct toepcb *toep)
+int
+t4_close_conn(struct adapter *sc, struct toepcb *toep)
 {
 	struct wrqe *wr;
 	struct cpl_close_con_req *req;
@@ -630,6 +670,7 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep
 
 	KASSERT(toep->ulp_mode == ULP_MODE_NONE ||
 	    toep->ulp_mode == ULP_MODE_TCPDDP ||
+	    toep->ulp_mode == ULP_MODE_TLS ||
 	    toep->ulp_mode == ULP_MODE_RDMA,
 	    ("%s: ulp_mode %u for toep %p", __func__, toep->ulp_mode, toep));
 
@@ -836,7 +877,7 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep
 
 	/* Send a FIN if requested, but only if there's no more data to send */
 	if (m == NULL && toep->flags & TPF_SEND_FIN)
-		close_conn(sc, toep);
+		t4_close_conn(sc, toep);
 }
 
 static inline void
@@ -1028,7 +1069,7 @@ t4_push_pdus(struct adapter *sc, struct toepcb *toep, 
 
 	/* Send a FIN if requested, but only if there are no more PDUs to send */
 	if (mbufq_first(pduq) == NULL && toep->flags & TPF_SEND_FIN)
-		close_conn(sc, toep);
+		t4_close_conn(sc, toep);
 }
 
 int
@@ -1047,6 +1088,8 @@ t4_tod_output(struct toedev *tod, struct tcpcb *tp)
 
 	if (toep->ulp_mode == ULP_MODE_ISCSI)
 		t4_push_pdus(sc, toep, 0);
+	else if (tls_tx_key(toep))
+		t4_push_tls_records(sc, toep, 0);
 	else
 		t4_push_frames(sc, toep, 0);
 
@@ -1071,6 +1114,8 @@ t4_send_fin(struct toedev *tod, struct tcpcb *tp)
 	if (tp->t_state >= TCPS_ESTABLISHED) {
 		if (toep->ulp_mode == ULP_MODE_ISCSI)
 			t4_push_pdus(sc, toep, 0);
+		else if (tls_tx_key(toep))
+			t4_push_tls_records(sc, toep, 0);
 		else
 			t4_push_frames(sc, toep, 0);
 	}
@@ -1703,6 +1748,10 @@ do_fw4_ack(struct sge_iq *iq, const struct rss_header 
 		credits -= txsd->tx_credits;
 		toep->tx_credits += txsd->tx_credits;
 		plen += txsd->plen;
+		if (txsd->iv_buffer) {
+			free(txsd->iv_buffer, M_CXGBE);
+			txsd->iv_buffer = NULL;
+		}
 		txsd++;
 		toep->txsd_avail++;
 		KASSERT(toep->txsd_avail <= toep->txsd_total,
@@ -1728,6 +1777,8 @@ do_fw4_ack(struct sge_iq *iq, const struct rss_header 
 		CURVNET_SET(toep->vnet);
 		if (toep->ulp_mode == ULP_MODE_ISCSI)
 			t4_push_pdus(sc, toep, plen);
+		else if (tls_tx_key(toep))
+			t4_push_tls_records(sc, toep, plen);
 		else
 			t4_push_frames(sc, toep, plen);
 		CURVNET_RESTORE();
@@ -1757,6 +1808,12 @@ do_fw4_ack(struct sge_iq *iq, const struct rss_header 
 			    tid, plen);
 #endif
 			sbdrop_locked(sb, plen);
+			if (tls_tx_key(toep)) {
+				struct tls_ofld_info *tls_ofld = &toep->tls;
+
+				MPASS(tls_ofld->sb_off >= plen);
+				tls_ofld->sb_off -= plen;
+			}
 			if (!TAILQ_EMPTY(&toep->aiotx_jobq))
 				t4_aiotx_queue_toep(toep);
 			sowwakeup_locked(so);	/* unlocks so_snd */
@@ -1808,14 +1865,14 @@ do_set_tcb_rpl(struct sge_iq *iq, const struct rss_hea
 }
 
 void
-t4_set_tcb_field(struct adapter *sc, struct sge_wrq *wrq, int tid,
-    uint16_t word, uint64_t mask, uint64_t val, int reply, int cookie, int iqid)
+t4_set_tcb_field(struct adapter *sc, struct sge_wrq *wrq, struct toepcb *toep,
+    uint16_t word, uint64_t mask, uint64_t val, int reply, int cookie)
 {
 	struct wrqe *wr;
 	struct cpl_set_tcb_field *req;
+	struct ofld_tx_sdesc *txsd;
 
 	MPASS((cookie & ~M_COOKIE) == 0);
-	MPASS((iqid & ~M_QUEUENO) == 0);
 
 	wr = alloc_wrqe(sizeof(*req), wrq);
 	if (wr == NULL) {
@@ -1824,13 +1881,26 @@ t4_set_tcb_field(struct adapter *sc, struct sge_wrq *w
 	}
 	req = wrtod(wr);
 
-	INIT_TP_WR_MIT_CPL(req, CPL_SET_TCB_FIELD, tid);
-	req->reply_ctrl = htobe16(V_QUEUENO(iqid));
+	INIT_TP_WR_MIT_CPL(req, CPL_SET_TCB_FIELD, toep->tid);
+	req->reply_ctrl = htobe16(V_QUEUENO(toep->ofld_rxq->iq.abs_id));
 	if (reply == 0)
 		req->reply_ctrl |= htobe16(F_NO_REPLY);
 	req->word_cookie = htobe16(V_WORD(word) | V_COOKIE(cookie));
 	req->mask = htobe64(mask);
 	req->val = htobe64(val);
+	if ((wrq->eq.flags & EQ_TYPEMASK) == EQ_OFLD) {
+		txsd = &toep->txsd[toep->txsd_pidx];
+		txsd->tx_credits = howmany(sizeof(*req), 16);
+		txsd->plen = 0;
+		KASSERT(toep->tx_credits >= txsd->tx_credits &&
+		    toep->txsd_avail > 0,
+		    ("%s: not enough credits (%d)", __func__,
+		    toep->tx_credits));
+		toep->tx_credits -= txsd->tx_credits;
+		if (__predict_false(++toep->txsd_pidx == toep->txsd_total))
+			toep->txsd_pidx = 0;
+		toep->txsd_avail--;
+	}
 
 	t4_wrq_tx(sc, wr);
 }
@@ -2229,6 +2299,9 @@ t4_aio_queue_aiotx(struct socket *so, struct kaiocb *j
 		return (EOPNOTSUPP);
 
 	if (!sc->tt.tx_zcopy)
+		return (EOPNOTSUPP);
+
+	if (tls_tx_key(toep))
 		return (EOPNOTSUPP);
 
 	SOCKBUF_LOCK(&so->so_snd);

Modified: stable/11/sys/dev/cxgbe/tom/t4_ddp.c
==============================================================================
--- stable/11/sys/dev/cxgbe/tom/t4_ddp.c	Thu Mar 28 22:19:09 2019	(r345663)
+++ stable/11/sys/dev/cxgbe/tom/t4_ddp.c	Thu Mar 28 23:43:38 2019	(r345664)
@@ -816,14 +816,13 @@ enable_ddp(struct adapter *sc, struct toepcb *toep)
 
 	DDP_ASSERT_LOCKED(toep);
 	toep->ddp.flags |= DDP_SC_REQ;
-	t4_set_tcb_field(sc, toep->ctrlq, toep->tid, W_TCB_RX_DDP_FLAGS,
+	t4_set_tcb_field(sc, toep->ctrlq, toep, W_TCB_RX_DDP_FLAGS,
 	    V_TF_DDP_OFF(1) | V_TF_DDP_INDICATE_OUT(1) |
 	    V_TF_DDP_BUF0_INDICATE(1) | V_TF_DDP_BUF1_INDICATE(1) |
 	    V_TF_DDP_BUF0_VALID(1) | V_TF_DDP_BUF1_VALID(1),
-	    V_TF_DDP_BUF0_INDICATE(1) | V_TF_DDP_BUF1_INDICATE(1), 0, 0,
-	    toep->ofld_rxq->iq.abs_id);
-	t4_set_tcb_field(sc, toep->ctrlq, toep->tid, W_TCB_T_FLAGS,
-	    V_TF_RCV_COALESCE_ENABLE(1), 0, 0, 0, toep->ofld_rxq->iq.abs_id);
+	    V_TF_DDP_BUF0_INDICATE(1) | V_TF_DDP_BUF1_INDICATE(1), 0, 0);
+	t4_set_tcb_field(sc, toep->ctrlq, toep, W_TCB_T_FLAGS,
+	    V_TF_RCV_COALESCE_ENABLE(1), 0, 0, 0);
 }
 
 static int
@@ -1873,10 +1872,9 @@ t4_aio_cancel_active(struct kaiocb *job)
 			 */
 			valid_flag = i == 0 ? V_TF_DDP_BUF0_VALID(1) :
 			    V_TF_DDP_BUF1_VALID(1);
-			t4_set_tcb_field(sc, toep->ctrlq, toep->tid,
+			t4_set_tcb_field(sc, toep->ctrlq, toep,
 			    W_TCB_RX_DDP_FLAGS, valid_flag, 0, 1,
-			    i + DDP_BUF0_INVALIDATED,
-			    toep->ofld_rxq->iq.abs_id);
+			    i + DDP_BUF0_INVALIDATED);
 			toep->ddp.db[i].cancel_pending = 1;
 			CTR2(KTR_CXGBE, "%s: request %p marked pending",
 			    __func__, job);

Modified: stable/11/sys/dev/cxgbe/tom/t4_listen.c
==============================================================================
--- stable/11/sys/dev/cxgbe/tom/t4_listen.c	Thu Mar 28 22:19:09 2019	(r345663)
+++ stable/11/sys/dev/cxgbe/tom/t4_listen.c	Thu Mar 28 23:43:38 2019	(r345664)
@@ -1054,6 +1054,11 @@ calc_opt2p(struct adapter *sc, struct port_info *pi, i
 	if (ulp_mode == ULP_MODE_TCPDDP)
 		opt2 |= F_RX_FC_VALID | F_RX_FC_DDP;
 #endif
+	if (ulp_mode == ULP_MODE_TLS) {
+		opt2 |= F_RX_FC_VALID;
+		opt2 &= ~V_RX_COALESCE(M_RX_COALESCE);
+		opt2 |= F_RX_FC_DISABLE;
+	}
 
 	return htobe32(opt2);
 }
@@ -1344,11 +1349,15 @@ found:
 
 		INIT_TP_WR_MIT_CPL(rpl5, CPL_PASS_ACCEPT_RPL, tid);
 	}
-	if (sc->tt.ddp && (so->so_options & SO_NO_DDP) == 0) {
-		ulp_mode = ULP_MODE_TCPDDP;
+	ulp_mode = select_ulp_mode(so, sc);
+	switch (ulp_mode) {
+	case ULP_MODE_TCPDDP:
 		synqe->flags |= TPF_SYNQE_TCPDDP;
-	} else
-		ulp_mode = ULP_MODE_NONE;
+		break;
+	case ULP_MODE_TLS:
+		synqe->flags |= TPF_SYNQE_TLS;
+		break;
+	}
 	rpl->opt0 = calc_opt0(so, vi, e, mtu_idx, rscale, rx_credits, ulp_mode);
 	rpl->opt2 = calc_opt2p(sc, pi, rxqid, &cpl->tcpopt, &th, ulp_mode);
 
@@ -1404,8 +1413,8 @@ found:
 		REJECT_PASS_ACCEPT();
 	}
 
-	CTR5(KTR_CXGBE, "%s: stid %u, tid %u, lctx %p, synqe %p, SYNACK",
-	    __func__, stid, tid, lctx, synqe);
+	CTR6(KTR_CXGBE, "%s: stid %u, tid %u, lctx %p, synqe %p, SYNACK mode %d",
+	    __func__, stid, tid, lctx, synqe, ulp_mode);
 
 	INP_WLOCK(inp);
 	synqe->flags |= TPF_SYNQE_HAS_L2TE;
@@ -1554,9 +1563,11 @@ reset:
 	toep->tid = tid;
 	toep->l2te = &sc->l2t->l2tab[synqe->l2e_idx];
 	if (synqe->flags & TPF_SYNQE_TCPDDP)
-		set_tcpddp_ulp_mode(toep);
+		set_ulp_mode(toep, ULP_MODE_TCPDDP);
+	else if (synqe->flags & TPF_SYNQE_TLS)
+		set_ulp_mode(toep, ULP_MODE_TLS);
 	else
-		toep->ulp_mode = ULP_MODE_NONE;
+		set_ulp_mode(toep, ULP_MODE_NONE);
 	/* opt0 rcv_bufsiz initially, assumes its normal meaning later */
 	toep->rx_credits = synqe->rcv_bufsize;
 

Copied and modified: stable/11/sys/dev/cxgbe/tom/t4_tls.c (from r330884, head/sys/dev/cxgbe/tom/t4_tls.c)
==============================================================================
--- head/sys/dev/cxgbe/tom/t4_tls.c	Tue Mar 13 23:05:51 2018	(r330884, copy source)
+++ stable/11/sys/dev/cxgbe/tom/t4_tls.c	Thu Mar 28 23:43:38 2019	(r345664)
@@ -48,6 +48,19 @@ __FBSDID("$FreeBSD$");
 #include "tom/t4_tom_l2t.h"
 #include "tom/t4_tom.h"
 
+VNET_DECLARE(int, tcp_do_autosndbuf);
+#define V_tcp_do_autosndbuf VNET(tcp_do_autosndbuf)
+VNET_DECLARE(int, tcp_autosndbuf_inc);
+#define V_tcp_autosndbuf_inc VNET(tcp_autosndbuf_inc)
+VNET_DECLARE(int, tcp_autosndbuf_max);
+#define V_tcp_autosndbuf_max VNET(tcp_autosndbuf_max)
+VNET_DECLARE(int, tcp_do_autorcvbuf);
+#define V_tcp_do_autorcvbuf VNET(tcp_do_autorcvbuf)
+VNET_DECLARE(int, tcp_autorcvbuf_inc);
+#define V_tcp_autorcvbuf_inc VNET(tcp_autorcvbuf_inc)
+VNET_DECLARE(int, tcp_autorcvbuf_max);
+#define V_tcp_autorcvbuf_max VNET(tcp_autorcvbuf_max)
+
 /*
  * The TCP sequence number of a CPL_TLS_DATA mbuf is saved here while
  * the mbuf is in the ulp_pdu_reclaimq.
@@ -68,11 +81,17 @@ t4_set_tls_tcb_field(struct toepcb *toep, uint16_t wor
 {
 	struct adapter *sc = td_adapter(toep->td);
 
-	t4_set_tcb_field(sc, toep->ctrlq, toep->tid, word, mask, val, 0, 0,
-	    toep->ofld_rxq->iq.abs_id);
+	t4_set_tcb_field(sc, toep->ofld_txq, toep, word, mask, val, 0, 0);
 }
 
 /* TLS and DTLS common routines */
+bool
+can_tls_offload(struct adapter *sc)
+{
+
+	return (sc->tt.tls && sc->cryptocaps & FW_CAPS_CONFIG_TLSKEYS);
+}
+
 int
 tls_tx_key(struct toepcb *toep)
 {
@@ -423,32 +442,13 @@ prepare_txkey_wr(struct tls_keyctx *kwr, struct tls_ke
 }
 
 /* TLS Key memory management */
-int
-tls_init_kmap(struct adapter *sc, struct tom_data *td)
-{
-
-	td->key_map = vmem_create("T4TLS key map", sc->vres.key.start,
-	    sc->vres.key.size, 8, 0, M_FIRSTFIT | M_NOWAIT);
-	if (td->key_map == NULL)
-		return (ENOMEM);
-	return (0);
-}
-
-void
-tls_free_kmap(struct tom_data *td)
-{
-
-	if (td->key_map != NULL)
-		vmem_destroy(td->key_map);
-}
-
 static int
 get_new_keyid(struct toepcb *toep, struct tls_key_context *k_ctx)
 {
-	struct tom_data *td = toep->td;
+	struct adapter *sc = td_adapter(toep->td);
 	vmem_addr_t addr;
 
-	if (vmem_alloc(td->key_map, TLS_KEY_CONTEXT_SZ, M_NOWAIT | M_FIRSTFIT,
+	if (vmem_alloc(sc->key_map, TLS_KEY_CONTEXT_SZ, M_NOWAIT | M_FIRSTFIT,
 	    &addr) != 0)
 		return (-1);
 
@@ -458,9 +458,9 @@ get_new_keyid(struct toepcb *toep, struct tls_key_cont
 static void
 free_keyid(struct toepcb *toep, int keyid)
 {
-	struct tom_data *td = toep->td;
+	struct adapter *sc = td_adapter(toep->td);
 
-	vmem_free(td->key_map, keyid, TLS_KEY_CONTEXT_SZ);
+	vmem_free(sc->key_map, keyid, TLS_KEY_CONTEXT_SZ);
 }
 
 static void
@@ -482,7 +482,7 @@ static int
 get_keyid(struct tls_ofld_info *tls_ofld, unsigned int ops)
 {
 	return (ops & KEY_WRITE_RX ? tls_ofld->rx_key_addr :
-		((ops & KEY_WRITE_TX) ? tls_ofld->rx_key_addr : -1));
+		((ops & KEY_WRITE_TX) ? tls_ofld->tx_key_addr : -1));
 }
 
 static int
@@ -505,9 +505,9 @@ tls_program_key_id(struct toepcb *toep, struct tls_key
 	struct tls_key_req *kwr;
 	struct tls_keyctx *kctx;
 
-	kwrlen = roundup2(sizeof(*kwr), 16);
+	kwrlen = sizeof(*kwr);
 	kctxlen = roundup2(sizeof(*kctx), 32);
-	len = kwrlen + kctxlen;
+	len = roundup2(kwrlen + kctxlen, 16);
 
 	if (toep->txsd_avail == 0)
 		return (EAGAIN);
@@ -549,7 +549,6 @@ tls_program_key_id(struct toepcb *toep, struct tls_key
 	kwr->sc_more = htobe32(V_ULPTX_CMD(ULP_TX_SC_IMM));
 	kwr->sc_len = htobe32(kctxlen);
 
-	/* XXX: This assumes that kwrlen == sizeof(*kwr). */
 	kctx = (struct tls_keyctx *)(kwr + 1);
 	memset(kctx, 0, kctxlen);
 
@@ -1168,7 +1167,8 @@ t4_push_tls_records(struct adapter *sc, struct toepcb 
 		 * Send a FIN if requested, but only if there's no
 		 * more data to send.
 		 */
-		if (sbavail(sb) == 0 && toep->flags & TPF_SEND_FIN) {
+		if (sbavail(sb) == tls_ofld->sb_off &&
+		    toep->flags & TPF_SEND_FIN) {
 			if (sowwakeup)
 				sowwakeup_locked(so);
 			else
@@ -1182,17 +1182,23 @@ t4_push_tls_records(struct adapter *sc, struct toepcb 
 			/*
 			 * A full TLS header is not yet queued, stop
 			 * for now until more data is added to the
-			 * socket buffer.
+			 * socket buffer.  However, if the connection
+			 * has been closed, we will never get the rest
+			 * of the header so just discard the partial
+			 * header and close the connection.
 			 */
 #ifdef VERBOSE_TRACES
-			CTR4(KTR_CXGBE, "%s: tid %d sbavail %d sb_off %d",
-			    __func__, toep->tid, sbavail(sb), tls_ofld->sb_off);
+			CTR5(KTR_CXGBE, "%s: tid %d sbavail %d sb_off %d%s",
+			    __func__, toep->tid, sbavail(sb), tls_ofld->sb_off,
+			    toep->flags & TPF_SEND_FIN ? "" : " SEND_FIN");
 #endif
 			if (sowwakeup)
 				sowwakeup_locked(so);
 			else
 				SOCKBUF_UNLOCK(sb);
 			SOCKBUF_UNLOCK_ASSERT(sb);
+			if (toep->flags & TPF_SEND_FIN)
+				t4_close_conn(sc, toep);
 			return;
 		}
 
@@ -1209,19 +1215,25 @@ t4_push_tls_records(struct adapter *sc, struct toepcb 
 			/*
 			 * The full TLS record is not yet queued, stop
 			 * for now until more data is added to the
-			 * socket buffer.
+			 * socket buffer.  However, if the connection
+			 * has been closed, we will never get the rest
+			 * of the record so just discard the partial
+			 * record and close the connection.
 			 */
 #ifdef VERBOSE_TRACES
-			CTR5(KTR_CXGBE,
-			    "%s: tid %d sbavail %d sb_off %d plen %d",
+			CTR6(KTR_CXGBE,

*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201903282343.x2SNhcKH079312>