Date: Mon, 24 Mar 2014 11:43:35 +0000 (UTC) From: Randall Stewart <rrs@FreeBSD.org> To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r263682 - in projects/rrs_mqueue: . sys/conf sys/dev/bxe sys/dev/cxgb sys/dev/cxgbe sys/dev/e1000 sys/dev/fdt sys/dev/ixgbe sys/dev/mxge sys/dev/oce sys/dev/virtio/network sys/dev/vxge ... Message-ID: <201403241143.s2OBhZ9m057529@svn.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: rrs Date: Mon Mar 24 11:43:35 2014 New Revision: 263682 URL: http://svnweb.freebsd.org/changeset/base/263682 Log: Ok first downward drop of my old mqueue patch that I had laying on an old version of head (plus a little read me to describe what I am doing in this project branch). I am not sure, so far, if this even compiles yet. It used to but that was a large leap ago.. I will do a large make to bring this machine up to current and make sure everything compiles (which will probably be my next commit here :-) Sponsored by: Adara Networks Added: projects/rrs_mqueue/README_RRS projects/rrs_mqueue/sys/net/drbr.c (contents, props changed) projects/rrs_mqueue/sys/net/drbr.h (contents, props changed) Modified: projects/rrs_mqueue/sys/conf/files projects/rrs_mqueue/sys/dev/bxe/bxe.c projects/rrs_mqueue/sys/dev/bxe/bxe.h projects/rrs_mqueue/sys/dev/cxgb/cxgb_adapter.h projects/rrs_mqueue/sys/dev/cxgb/cxgb_main.c projects/rrs_mqueue/sys/dev/cxgb/cxgb_sge.c projects/rrs_mqueue/sys/dev/cxgbe/adapter.h projects/rrs_mqueue/sys/dev/cxgbe/t4_main.c projects/rrs_mqueue/sys/dev/cxgbe/t4_sge.c projects/rrs_mqueue/sys/dev/e1000/if_em.c projects/rrs_mqueue/sys/dev/e1000/if_em.h projects/rrs_mqueue/sys/dev/e1000/if_igb.c projects/rrs_mqueue/sys/dev/e1000/if_igb.h projects/rrs_mqueue/sys/dev/fdt/fdt_common.c projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.c projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.h projects/rrs_mqueue/sys/dev/ixgbe/ixv.c projects/rrs_mqueue/sys/dev/ixgbe/ixv.h projects/rrs_mqueue/sys/dev/mxge/if_mxge.c projects/rrs_mqueue/sys/dev/mxge/if_mxge_var.h projects/rrs_mqueue/sys/dev/oce/oce_hw.c projects/rrs_mqueue/sys/dev/oce/oce_if.c projects/rrs_mqueue/sys/dev/oce/oce_if.h projects/rrs_mqueue/sys/dev/oce/oce_mbox.c projects/rrs_mqueue/sys/dev/oce/oce_queue.c projects/rrs_mqueue/sys/dev/virtio/network/if_vtnet.c projects/rrs_mqueue/sys/dev/virtio/network/if_vtnetvar.h projects/rrs_mqueue/sys/dev/vxge/vxge.c projects/rrs_mqueue/sys/dev/vxge/vxge.h projects/rrs_mqueue/sys/kern/kern_mbuf.c projects/rrs_mqueue/sys/kern/subr_bufring.c projects/rrs_mqueue/sys/kern/subr_bus.c projects/rrs_mqueue/sys/net/if_var.h projects/rrs_mqueue/sys/netinet/if_ether.c projects/rrs_mqueue/sys/ofed/drivers/net/mlx4/en_tx.c projects/rrs_mqueue/sys/ofed/drivers/net/mlx4/mlx4_en.h projects/rrs_mqueue/sys/sys/buf_ring.h Added: projects/rrs_mqueue/README_RRS ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ projects/rrs_mqueue/README_RRS Mon Mar 24 11:43:35 2014 (r263682) @@ -0,0 +1,57 @@ +Welcome to my little side project. It has taken +me a while to get around to finding the time to +put this back in.. whew. + +What we have here currently is some basic changes that were +done at Adara Networks to the way the lower level stack works +with device drivers. Basically all mbufs have a priority +field (cosqos), this field gets set to the highest number +by default (usually queue 7 by default if we have +8 queues defined 0-7). Now whenever a driver goes +to send out data, it starts at queue 0, and works +its way out. This means that *yes* you can get +packet starvation of things placed in queue 7 while +other things are going to queue 0. + +There are also some new methods added to if_net that +deal with managment of the queues/rings. This is +the start of where I plan on putting an AQM module +in place. Right now its just drop-tail if the queue +gets full, at some point I will try to add in both +codel and possibly Cisco's Pi. Pi will need to be a +klm since C has patents on it (the typical C patent +of you can have a free license unless you sue me). +Codel is open created by Van Jacobson and Kathy Nichols +only it was done in Linux first, so I will have to +read the latest RFC's and try to do a clean implementation +of that so we have no GNU contamination (this may take +a while depending on how much time I can pull away +from Adara, or if I can justify doing it as an +Adara project). + +Now once I have all this settled in to head and know it +all compiles (not sure since I was merging old code from +quite a while ago that this branch yet compiles :-D), I +want to move on and make the IP packet processing stack +lock-less. What I mean by that is I would like to see +*no* locks in the packet forwarding path. Only when you +change things (radix trees etc) will locks be applied. +How I will do this is by adding a garbage collector entity +that "defers" removal of things long enough for packets +being forwarded (during a change) to be out of the system +*before* removal of the old entity. This sounds a bit +strange but it works quite well as long as one can +bound the time period needed before the garbage +collector gets active. + +This will come later assuming I get time.. of course +my life as been full of ENOTIME errors, but hopefully +is slowing down here a bit :-) + +You are welcome to play in here and contribute. You can +email me at rrs at freebsd.org (or rrs at lakerest.net.. thats +where the mail goes anyway :D) + +Best wishes +Randall Stewart + Modified: projects/rrs_mqueue/sys/conf/files ============================================================================== --- projects/rrs_mqueue/sys/conf/files Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/conf/files Mon Mar 24 11:43:35 2014 (r263682) @@ -3081,6 +3081,7 @@ net/bridgestp.c optional bridge | if_b net/flowtable.c optional flowtable inet | flowtable inet6 net/ieee8023ad_lacp.c optional lagg net/if.c standard +net/drbr.c standard net/if_arcsubr.c optional arcnet net/if_atmsubr.c optional atm net/if_bridge.c optional bridge inet | if_bridge inet Modified: projects/rrs_mqueue/sys/dev/bxe/bxe.c ============================================================================== --- projects/rrs_mqueue/sys/dev/bxe/bxe.c Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/bxe/bxe.c Mon Mar 24 11:43:35 2014 (r263682) @@ -5946,10 +5946,11 @@ bxe_tx_mq_start_locked(struct bxe_softc struct bxe_fastpath *fp, struct mbuf *m) { - struct buf_ring *tx_br = fp->tx_br; + struct drbr_ring *tx_br = fp->tx_br; struct mbuf *next; int depth, rc, tx_count; uint16_t tx_bd_avail; + uint8_t qused; rc = tx_count = 0; @@ -5966,25 +5967,16 @@ bxe_tx_mq_start_locked(struct bxe_softc BXE_FP_TX_LOCK_ASSERT(fp); - if (m == NULL) { - /* no new work, check for pending frames */ - next = drbr_dequeue(ifp, tx_br); - } else if (drbr_needs_enqueue(ifp, tx_br)) { - /* have both new and pending work, maintain packet order */ - rc = drbr_enqueue(ifp, tx_br, m); - if (rc != 0) { - fp->eth_q_stats.tx_soft_errors++; - goto bxe_tx_mq_start_locked_exit; - } - next = drbr_dequeue(ifp, tx_br); - } else { - /* new work only and nothing pending */ - next = m; + if (m != NULL) { + rc = drbr_enqueue(ifp, tx_br, m); + if (rc != 0) { + fp->eth_q_stats.tx_soft_errors++; + goto bxe_tx_mq_start_locked_exit; + } } /* keep adding entries while there are frames to send */ - while (next != NULL) { - + while ((next = drbr_peek(ifp, fp->tx_br, &qused)) != NULL) { /* the mbuf now belongs to us */ fp->eth_q_stats.mbuf_alloc_tx++; @@ -5996,19 +5988,22 @@ bxe_tx_mq_start_locked(struct bxe_softc rc = bxe_tx_encap(fp, &next); if (__predict_false(rc != 0)) { fp->eth_q_stats.tx_encap_failures++; - if (next != NULL) { - /* mark the TX queue as full and save the frame */ - ifp->if_drv_flags |= IFF_DRV_OACTIVE; - /* XXX this may reorder the frame */ - rc = drbr_enqueue(ifp, tx_br, next); - fp->eth_q_stats.mbuf_alloc_tx--; - fp->eth_q_stats.tx_frames_deferred++; - } - + if (next == NULL) { + drbr_advance(ifp, fp->tx_br, qused); + } else { + drbr_putback(ifp, fp->tx_br, next, qused); + /* + * Mark the TX queue as full and save + * the frame. + */ + ifp->if_drv_flags |= IFF_DRV_OACTIVE; + fp->eth_q_stats.mbuf_alloc_tx--; + fp->eth_q_stats.tx_frames_deferred++; + } /* stop looking for more work */ break; } - + drbr_advance(ifp, fp->tx_br, qused); /* the transmit frame was enqueued successfully */ tx_count++; @@ -6087,7 +6082,6 @@ bxe_mq_flush(struct ifnet *ifp) { struct bxe_softc *sc = ifp->if_softc; struct bxe_fastpath *fp; - struct mbuf *m; int i; for (i = 0; i < sc->num_queues; i++) { @@ -6102,9 +6096,7 @@ bxe_mq_flush(struct ifnet *ifp) if (fp->tx_br != NULL) { BLOGD(sc, DBG_LOAD, "Clearing fp[%02d] buf_ring\n", fp->index); BXE_FP_TX_LOCK(fp); - while ((m = buf_ring_dequeue_sc(fp->tx_br)) != NULL) { - m_freem(m); - } + drbr_flush(ifp, fp->tx_br); BXE_FP_TX_UNLOCK(fp); } } @@ -6505,12 +6497,9 @@ bxe_free_fp_buffers(struct bxe_softc *sc #if __FreeBSD_version >= 800000 if (fp->tx_br != NULL) { - struct mbuf *m; /* just in case bxe_mq_flush() wasn't called */ - while ((m = buf_ring_dequeue_sc(fp->tx_br)) != NULL) { - m_freem(m); - } - buf_ring_free(fp->tx_br, M_DEVBUF); + drbr_flush(sc->ifnet, fp->tx_br); + drbr_free(fp->tx_br, M_DEVBUF); fp->tx_br = NULL; } #endif @@ -6771,8 +6760,7 @@ bxe_alloc_fp_buffers(struct bxe_softc *s fp = &sc->fp[i]; #if __FreeBSD_version >= 800000 - fp->tx_br = buf_ring_alloc(BXE_BR_SIZE, M_DEVBUF, - M_NOWAIT, &fp->tx_mtx); + fp->tx_br = drbr_alloc(M_DEVBUF, M_NOWAIT, &fp->tx_mtx); if (fp->tx_br == NULL) { BLOGE(sc, "buf_ring alloc fail for fp[%02d]\n", i); goto bxe_alloc_fp_buffers_error; Modified: projects/rrs_mqueue/sys/dev/bxe/bxe.h ============================================================================== --- projects/rrs_mqueue/sys/dev/bxe/bxe.h Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/bxe/bxe.h Mon Mar 24 11:43:35 2014 (r263682) @@ -69,6 +69,7 @@ __FBSDID("$FreeBSD$"); #include <net/if_vlan_var.h> #include <net/zlib.h> #include <net/bpf.h> +#include <net/drbr.h> #include <netinet/in.h> #include <netinet/ip.h> @@ -734,7 +735,7 @@ struct bxe_fastpath { #if __FreeBSD_version >= 800000 #define BXE_BR_SIZE 4096 - struct buf_ring *tx_br; + struct drbr_ring *tx_br; #endif }; /* struct bxe_fastpath */ Modified: projects/rrs_mqueue/sys/dev/cxgb/cxgb_adapter.h ============================================================================== --- projects/rrs_mqueue/sys/dev/cxgb/cxgb_adapter.h Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/cxgb/cxgb_adapter.h Mon Mar 24 11:43:35 2014 (r263682) @@ -252,7 +252,7 @@ struct sge_txq { bus_dma_tag_t entry_tag; struct mbuf_head sendq; - struct buf_ring *txq_mr; + struct drbr_ring *txq_mr; struct ifaltq *txq_ifq; struct callout txq_timer; struct callout txq_watchdog; Modified: projects/rrs_mqueue/sys/dev/cxgb/cxgb_main.c ============================================================================== --- projects/rrs_mqueue/sys/dev/cxgb/cxgb_main.c Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/cxgb/cxgb_main.c Mon Mar 24 11:43:35 2014 (r263682) @@ -66,6 +66,7 @@ __FBSDID("$FreeBSD$"); #include <net/if_media.h> #include <net/if_types.h> #include <net/if_vlan_var.h> +#include <net/drbr.h> #include <netinet/in_systm.h> #include <netinet/in.h> @@ -2361,7 +2362,7 @@ cxgb_tick_handler(void *arg, int count) drops = 0; for (j = pi->first_qset; j < pi->first_qset + pi->nqsets; j++) - drops += sc->sge.qs[j].txq[TXQ_ETH].txq_mr->br_drops; + drops += drbr_get_dropcnt(sc->sge.qs[j].txq[TXQ_ETH].txq_mr); ifp->if_snd.ifq_drops = drops; ifp->if_oerrors = Modified: projects/rrs_mqueue/sys/dev/cxgb/cxgb_sge.c ============================================================================== --- projects/rrs_mqueue/sys/dev/cxgb/cxgb_sge.c Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/cxgb/cxgb_sge.c Mon Mar 24 11:43:35 2014 (r263682) @@ -61,6 +61,7 @@ __FBSDID("$FreeBSD$"); #include <net/bpf.h> #include <net/ethernet.h> #include <net/if_vlan_var.h> +#include <net/drbr.h> #include <netinet/in_systm.h> #include <netinet/in.h> @@ -1684,7 +1685,7 @@ cxgb_transmit_locked(struct ifnet *ifp, { struct port_info *pi = qs->port; struct sge_txq *txq = &qs->txq[TXQ_ETH]; - struct buf_ring *br = txq->txq_mr; + struct drbr_ring *br = txq->txq_mr; int error, avail; avail = txq->size - txq->in_use; @@ -1980,7 +1981,7 @@ t3_free_qset(adapter_t *sc, struct sge_q reclaim_completed_tx(q, 0, TXQ_ETH); if (q->txq[TXQ_ETH].txq_mr != NULL) - buf_ring_free(q->txq[TXQ_ETH].txq_mr, M_DEVBUF); + drbr_free(q->txq[TXQ_ETH].txq_mr, M_DEVBUF); if (q->txq[TXQ_ETH].txq_ifq != NULL) { ifq_delete(q->txq[TXQ_ETH].txq_ifq); free(q->txq[TXQ_ETH].txq_ifq, M_DEVBUF); @@ -2430,8 +2431,8 @@ t3_sge_alloc_qset(adapter_t *sc, u_int i q->port = pi; q->adap = sc; - if ((q->txq[TXQ_ETH].txq_mr = buf_ring_alloc(cxgb_txq_buf_ring_size, - M_DEVBUF, M_WAITOK, &q->lock)) == NULL) { + if ((q->txq[TXQ_ETH].txq_mr = drbr_alloc(M_DEVBUF, M_WAITOK, + &q->lock)) == NULL) { device_printf(sc->dev, "failed to allocate mbuf ring\n"); goto err; } @@ -3523,9 +3524,9 @@ t3_add_configured_sysctls(adapter_t *sc) CTLTYPE_STRING | CTLFLAG_RD, &qs->rspq, 0, t3_dump_rspq, "A", "dump of the response queue"); - SYSCTL_ADD_UQUAD(ctx, txqpoidlist, OID_AUTO, "dropped", +/* RRS FIXME SYSCTL_ADD_UQUAD(ctx, txqpoidlist, OID_AUTO, "dropped", CTLFLAG_RD, &qs->txq[TXQ_ETH].txq_mr->br_drops, - "#tunneled packets dropped"); + "#tunneled packets dropped");*/ SYSCTL_ADD_UINT(ctx, txqpoidlist, OID_AUTO, "sendqlen", CTLFLAG_RD, &qs->txq[TXQ_ETH].sendq.qlen, 0, "#tunneled packets waiting to be sent"); Modified: projects/rrs_mqueue/sys/dev/cxgbe/adapter.h ============================================================================== --- projects/rrs_mqueue/sys/dev/cxgbe/adapter.h Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/cxgbe/adapter.h Mon Mar 24 11:43:35 2014 (r263682) @@ -445,7 +445,7 @@ struct sge_txq { struct ifnet *ifp; /* the interface this txq belongs to */ bus_dma_tag_t tx_tag; /* tag for transmit buffers */ - struct buf_ring *br; /* tx buffer ring */ + struct drbr_ring *br; /* tx buffer ring */ struct tx_sdesc *sdesc; /* KVA of software descriptor ring */ struct mbuf *m; /* held up due to temporary resource shortage */ Modified: projects/rrs_mqueue/sys/dev/cxgbe/t4_main.c ============================================================================== --- projects/rrs_mqueue/sys/dev/cxgbe/t4_main.c Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/cxgbe/t4_main.c Mon Mar 24 11:43:35 2014 (r263682) @@ -54,6 +54,7 @@ __FBSDID("$FreeBSD$"); #include <net/if.h> #include <net/if_types.h> #include <net/if_dl.h> +#include <net/drbr.h> #include <net/if_vlan_var.h> #if defined(__i386__) || defined(__amd64__) #include <vm/vm.h> @@ -1284,7 +1285,7 @@ cxgbe_transmit(struct ifnet *ifp, struct struct port_info *pi = ifp->if_softc; struct adapter *sc = pi->adapter; struct sge_txq *txq = &sc->sge.txq[pi->first_txq]; - struct buf_ring *br; + struct drbr_ring *br; int rc; M_ASSERTPKTHDR(m); @@ -1326,7 +1327,7 @@ cxgbe_transmit(struct ifnet *ifp, struct */ TXQ_LOCK_ASSERT_OWNED(txq); - if (drbr_needs_enqueue(ifp, br) || txq->m) { + if (txq->m) { /* Queued for transmission. */ @@ -1352,7 +1353,6 @@ cxgbe_qflush(struct ifnet *ifp) struct port_info *pi = ifp->if_softc; struct sge_txq *txq; int i; - struct mbuf *m; /* queues do not exist if !PORT_INIT_DONE. */ if (pi->flags & PORT_INIT_DONE) { @@ -1360,8 +1360,7 @@ cxgbe_qflush(struct ifnet *ifp) TXQ_LOCK(txq); m_freem(txq->m); txq->m = NULL; - while ((m = buf_ring_dequeue_sc(txq->br)) != NULL) - m_freem(m); + drbr_flush(ifp, txq->br); TXQ_UNLOCK(txq); } } @@ -4125,7 +4124,7 @@ cxgbe_tick(void *arg) drops = s->tx_drop; for_each_txq(pi, i, txq) - drops += txq->br->br_drops; + drops += drbr_get_dropcnt(txq->br); ifp->if_oqdrops = drops; ifp->if_oerrors = s->tx_error_frames; @@ -6616,7 +6615,7 @@ sysctl_wcwr_stats(SYSCTL_HANDLER_ARGS) static inline void txq_start(struct ifnet *ifp, struct sge_txq *txq) { - struct buf_ring *br; + struct drbr_ring *br; struct mbuf *m; TXQ_LOCK_ASSERT_OWNED(txq); @@ -7857,7 +7856,6 @@ t4_ioctl(struct cdev *dev, unsigned long txq->txpkt_wrs = 0; txq->txpkts_wrs = 0; txq->txpkts_pkts = 0; - txq->br->br_drops = 0; txq->no_dmamap = 0; txq->no_desc = 0; } Modified: projects/rrs_mqueue/sys/dev/cxgbe/t4_sge.c ============================================================================== --- projects/rrs_mqueue/sys/dev/cxgbe/t4_sge.c Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/cxgbe/t4_sge.c Mon Mar 24 11:43:35 2014 (r263682) @@ -48,6 +48,7 @@ __FBSDID("$FreeBSD$"); #include <net/ethernet.h> #include <net/if.h> #include <net/if_vlan_var.h> +#include <net/drbr.h> #include <netinet/in.h> #include <netinet/ip.h> #include <netinet/ip6.h> @@ -1799,9 +1800,10 @@ t4_eth_tx(struct ifnet *ifp, struct sge_ struct port_info *pi = (void *)ifp->if_softc; struct adapter *sc = pi->adapter; struct sge_eq *eq = &txq->eq; - struct buf_ring *br = txq->br; + struct drbr_ring *br = txq->br; struct mbuf *next; int rc, coalescing, can_reclaim; + uint8_t qused; struct txpkts txpkts; struct sgl sgl; @@ -1828,8 +1830,7 @@ t4_eth_tx(struct ifnet *ifp, struct sge_ if (__predict_false(eq->flags & EQ_DOOMED)) { m_freem(m); - while ((m = buf_ring_dequeue_sc(txq->br)) != NULL) - m_freem(m); + drbr_flush(ifp, br); return (ENETDOWN); } @@ -1844,7 +1845,7 @@ t4_eth_tx(struct ifnet *ifp, struct sge_ next = m->m_nextpkt; m->m_nextpkt = NULL; - if (next || buf_ring_peek(br)) + if (next || drbr_peek(ifp, br, &qused)) coalescing = 1; rc = get_pkt_sgl(txq, &m, &sgl, coalescing); @@ -2868,7 +2869,7 @@ alloc_txq(struct port_info *pi, struct s txq->sdesc = malloc(eq->cap * sizeof(struct tx_sdesc), M_CXGBE, M_ZERO | M_WAITOK); - txq->br = buf_ring_alloc(eq->qsize, M_CXGBE, M_WAITOK, &eq->eq_lock); + txq->br = drbr_alloc(M_CXGBE, M_WAITOK, &eq->eq_lock); rc = bus_dma_tag_create(sc->dmat, 1, 0, BUS_SPACE_MAXADDR, BUS_SPACE_MAXADDR, NULL, NULL, 64 * 1024, TX_SGL_SEGS, @@ -2923,8 +2924,8 @@ alloc_txq(struct port_info *pi, struct s SYSCTL_ADD_UQUAD(&pi->ctx, children, OID_AUTO, "txpkts_pkts", CTLFLAG_RD, &txq->txpkts_pkts, "# of frames tx'd using txpkts work requests"); - SYSCTL_ADD_UQUAD(&pi->ctx, children, OID_AUTO, "br_drops", CTLFLAG_RD, - &txq->br->br_drops, "# of drops in the buf_ring for this queue"); +/* SYSCTL_ADD_UQUAD(&pi->ctx, children, OID_AUTO, "br_drops", CTLFLAG_RD, + &txq->br->br_drops, "# of drops in the buf_ring for this queue");*/ SYSCTL_ADD_UINT(&pi->ctx, children, OID_AUTO, "no_dmamap", CTLFLAG_RD, &txq->no_dmamap, 0, "# of times txq ran out of DMA maps"); SYSCTL_ADD_UINT(&pi->ctx, children, OID_AUTO, "no_desc", CTLFLAG_RD, @@ -2953,7 +2954,7 @@ free_txq(struct port_info *pi, struct sg if (txq->txmaps.maps) t4_free_tx_maps(&txq->txmaps, txq->tx_tag); - buf_ring_free(txq->br, M_CXGBE); + drbr_free(txq->br, M_CXGBE); if (txq->tx_tag) bus_dma_tag_destroy(txq->tx_tag); Modified: projects/rrs_mqueue/sys/dev/e1000/if_em.c ============================================================================== --- projects/rrs_mqueue/sys/dev/e1000/if_em.c Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/e1000/if_em.c Mon Mar 24 11:43:35 2014 (r263682) @@ -67,6 +67,7 @@ #include <net/if_arp.h> #include <net/if_dl.h> #include <net/if_media.h> +#include <net/drbr.h> #include <net/if_types.h> #include <net/if_vlan_var.h> @@ -273,6 +274,9 @@ static int em_is_valid_ether_addr(u8 *) static int em_sysctl_int_delay(SYSCTL_HANDLER_ARGS); static void em_add_int_delay_sysctl(struct adapter *, const char *, const char *, struct em_int_delay_info *, int, int); +static void em_max_bytes(struct ifnet *, uint64_t max); +static struct drbr_ring *em_get_ring(struct ifnet *ifp, int num); +static int em_ring_query(struct ifnet *ifp, struct mbuf *); /* Management and WOL Support */ static void em_init_manageability(struct adapter *); static void em_release_manageability(struct adapter *); @@ -897,6 +901,37 @@ em_resume(device_t dev) return bus_generic_resume(dev); } +void +em_max_bytes(struct ifnet *ifp, uint64_t max) +{ + struct adapter *adapter = ifp->if_softc; + adapter->ring_bytes_max = max; +} + +struct drbr_ring * +em_get_ring(struct ifnet *ifp, int num) +{ + struct adapter *adapter = ifp->if_softc; + struct tx_ring *txr; + if (num >= adapter->num_queues) { + return (NULL); + } + if (adapter->tx_rings) { + txr = &adapter->tx_rings[num]; + return (txr->br); + } else { + return (NULL); + } +} + +int +em_ring_query(struct ifnet *ifp, struct mbuf *m) +{ + struct adapter *adapter = ifp->if_softc; + struct tx_ring *txr; + txr = &adapter->tx_rings[0]; + return(drbr_is_on_ring(txr->br, m)); +} #ifdef EM_MULTIQUEUE /********************************************************************* @@ -913,6 +948,7 @@ em_mq_start_locked(struct ifnet *ifp, st struct adapter *adapter = txr->adapter; struct mbuf *next; int err = 0, enq = 0; + uint8_t qused; if ((ifp->if_drv_flags & (IFF_DRV_RUNNING | IFF_DRV_OACTIVE)) != IFF_DRV_RUNNING || adapter->link_active == 0) { @@ -929,20 +965,26 @@ em_mq_start_locked(struct ifnet *ifp, st } /* Process the queue */ - while ((next = drbr_peek(ifp, txr->br)) != NULL) { + while ((next = drbr_peek(ifp, txr->br, &qused)) != NULL) { if ((err = em_xmit(txr, &next)) != 0) { if (next == NULL) - drbr_advance(ifp, txr->br); + drbr_advance(ifp, txr->br, qused); else - drbr_putback(ifp, txr->br, next); + drbr_putback(ifp, txr->br, next, qused); break; } - drbr_advance(ifp, txr->br); + drbr_advance(ifp, txr->br, qused); + atomic_add_long(&txr->bytes_on_ring, + (uint64_t)next->m_pkthdr.len); enq++; ifp->if_obytes += next->m_pkthdr.len; if (next->m_flags & M_MCAST) ifp->if_omcasts++; ETHER_BPF_MTAP(ifp, next); + if (adapter->ring_bytes_max && + (txr->bytes_on_ring >= adapter->ring_bytes_max)) { + break; + } if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) break; } @@ -991,8 +1033,7 @@ em_qflush(struct ifnet *ifp) for (int i = 0; i < adapter->num_queues; i++, txr++) { EM_TX_LOCK(txr); - while ((m = buf_ring_dequeue_sc(txr->br)) != NULL) - m_freem(m); + drbr_flush(ifp, txr->br); EM_TX_UNLOCK(txr); } if_qflush(ifp); @@ -2984,6 +3025,9 @@ em_setup_interface(device_t dev, struct ifp->if_softc = adapter; ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST; ifp->if_ioctl = em_ioctl; + ifp->if_maxbytes = em_max_bytes; + ifp->if_getdrbr_ring = em_get_ring; + ifp->if_mbuf_on_ring = em_ring_query; #ifdef EM_MULTIQUEUE /* Multiqueue stack interface */ ifp->if_transmit = em_mq_start; @@ -3222,7 +3266,7 @@ em_allocate_queues(struct adapter *adapt } #if __FreeBSD_version >= 800000 /* Allocate a buf ring */ - txr->br = buf_ring_alloc(4096, M_DEVBUF, + txr->br = drbr_alloc(M_DEVBUF, M_WAITOK, &txr->tx_mtx); #endif } @@ -3272,7 +3316,7 @@ err_tx_desc: free(adapter->rx_rings, M_DEVBUF); rx_fail: #if __FreeBSD_version >= 800000 - buf_ring_free(txr->br, M_DEVBUF); + drbr_free(txr->br, M_DEVBUF); #endif free(adapter->tx_rings, M_DEVBUF); fail: @@ -3396,6 +3440,7 @@ em_setup_transmit_ring(struct tx_ring *t /* Set number of descriptors available */ txr->tx_avail = adapter->num_tx_desc; + txr->bytes_on_ring = 0; txr->queue_status = EM_QUEUE_IDLE; /* Clear checksum offload context. */ @@ -3579,7 +3624,7 @@ em_free_transmit_buffers(struct tx_ring } #if __FreeBSD_version >= 800000 if (txr->br != NULL) - buf_ring_free(txr->br, M_DEVBUF); + drbr_free(txr->br, M_DEVBUF); #endif if (txr->tx_buffers != NULL) { free(txr->tx_buffers, M_DEVBUF); @@ -3876,6 +3921,8 @@ em_txeof(struct tx_ring *txr) ++processed; if (tx_buffer->m_head) { + atomic_subtract_long(&txr->bytes_on_ring, + (u_long)tx_buffer->m_head->m_pkthdr.len); bus_dmamap_sync(txr->txtag, tx_buffer->map, BUS_DMASYNC_POSTWRITE); @@ -5332,7 +5379,7 @@ em_add_hw_stats(struct adapter *adapter) queue_node = SYSCTL_ADD_NODE(ctx, child, OID_AUTO, namebuf, CTLFLAG_RD, NULL, "Queue Name"); queue_list = SYSCTL_CHILDREN(queue_node); - + drbr_add_sysctl_stats(dev, queue_list, txr->br); SYSCTL_ADD_PROC(ctx, queue_list, OID_AUTO, "txd_head", CTLTYPE_UINT | CTLFLAG_RD, adapter, E1000_TDH(txr->me), Modified: projects/rrs_mqueue/sys/dev/e1000/if_em.h ============================================================================== --- projects/rrs_mqueue/sys/dev/e1000/if_em.h Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/e1000/if_em.h Mon Mar 24 11:43:35 2014 (r263682) @@ -298,8 +298,9 @@ struct tx_ring { u8 last_hw_tucso; u8 last_hw_tucss; #if __FreeBSD_version >= 800000 - struct buf_ring *br; + struct drbr_ring *br; #endif + volatile u_long bytes_on_ring; /* Interrupt resources */ bus_dma_tag_t txtag; void *tag; @@ -346,6 +347,7 @@ struct rx_ring { /* Our adapter structure */ struct adapter { struct ifnet *ifp; + uint64_t ring_bytes_max; struct e1000_hw hw; /* FreeBSD operating-system-specific structures. */ Modified: projects/rrs_mqueue/sys/dev/e1000/if_igb.c ============================================================================== --- projects/rrs_mqueue/sys/dev/e1000/if_igb.c Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/e1000/if_igb.c Mon Mar 24 11:43:35 2014 (r263682) @@ -72,6 +72,7 @@ #include <net/if_arp.h> #include <net/if_dl.h> #include <net/if_media.h> +#include <net/drbr.h> #include <net/if_types.h> #include <net/if_vlan_var.h> @@ -216,6 +217,9 @@ static void igb_reset(struct adapter *); static int igb_setup_interface(device_t, struct adapter *); static int igb_allocate_queues(struct adapter *); static void igb_configure_queues(struct adapter *); +static void igb_max_bytes(struct ifnet *, uint64_t max); +static struct drbr_ring *igb_get_ring(struct ifnet *ifp, int num); +static int igb_ring_query(struct ifnet *ifp, struct mbuf *m); static int igb_allocate_transmit_buffers(struct tx_ring *); static void igb_setup_transmit_structures(struct adapter *); @@ -883,6 +887,42 @@ igb_resume(device_t dev) return bus_generic_resume(dev); } +void +igb_max_bytes(struct ifnet *ifp, uint64_t max) +{ + struct adapter *adapter = ifp->if_softc; + adapter->ring_bytes_max = max; + +} + +struct drbr_ring * +igb_get_ring(struct ifnet *ifp, int num) +{ + struct adapter *adapter = ifp->if_softc; + struct tx_ring *txr; + + if (num >= adapter->num_queues) { + return (NULL); + } + if (adapter->tx_rings) { + txr = &adapter->tx_rings[num]; + return (txr->br); + } else { + return (NULL); + } +} + +int +igb_ring_query(struct ifnet *ifp, struct mbuf *m) +{ + struct adapter *adapter = ifp->if_softc; + struct tx_ring *txr; + /* For this hack, we only use 0, since adara stuff + * sends out on queue 0 always. + */ + txr = &adapter->tx_rings[0]; + return(drbr_is_on_ring(txr->br, m)); +} #ifdef IGB_LEGACY_TX @@ -1003,6 +1043,7 @@ igb_mq_start_locked(struct ifnet *ifp, s struct adapter *adapter = txr->adapter; struct mbuf *next; int err = 0, enq = 0; + uint8_t qused; IGB_TX_LOCK_ASSERT(txr); @@ -1012,22 +1053,24 @@ igb_mq_start_locked(struct ifnet *ifp, s /* Process the queue */ - while ((next = drbr_peek(ifp, txr->br)) != NULL) { + while ((next = drbr_peek(ifp, txr->br, &qused)) != NULL) { if ((err = igb_xmit(txr, &next)) != 0) { if (next == NULL) { /* It was freed, move forward */ - drbr_advance(ifp, txr->br); + drbr_advance(ifp, txr->br, qused); } else { /* * Still have one left, it may not be * the same since the transmit function * may have changed it. */ - drbr_putback(ifp, txr->br, next); + drbr_putback(ifp, txr->br, next, qused); } break; } - drbr_advance(ifp, txr->br); + drbr_advance(ifp, txr->br, qused); + atomic_add_long(&txr->bytes_on_ring, + (u_long)next->m_pkthdr.len); enq++; ifp->if_obytes += next->m_pkthdr.len; if (next->m_flags & M_MCAST) @@ -1035,6 +1078,11 @@ igb_mq_start_locked(struct ifnet *ifp, s ETHER_BPF_MTAP(ifp, next); if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) break; + if (adapter->ring_bytes_max && + (txr->bytes_on_ring >= adapter->ring_bytes_max)) { + break; + } + } if (enq > 0) { /* Set the watchdog */ @@ -1072,12 +1120,10 @@ igb_qflush(struct ifnet *ifp) { struct adapter *adapter = ifp->if_softc; struct tx_ring *txr = adapter->tx_rings; - struct mbuf *m; for (int i = 0; i < adapter->num_queues; i++, txr++) { IGB_TX_LOCK(txr); - while ((m = buf_ring_dequeue_sc(txr->br)) != NULL) - m_freem(m); + drbr_flush(ifp, txr->br); IGB_TX_UNLOCK(txr); } if_qflush(ifp); @@ -3119,6 +3165,9 @@ igb_setup_interface(device_t dev, struct #ifndef IGB_LEGACY_TX ifp->if_transmit = igb_mq_start; ifp->if_qflush = igb_qflush; + ifp->if_maxbytes = igb_max_bytes; + ifp->if_getdrbr_ring = igb_get_ring; + ifp->if_mbuf_on_ring = igb_ring_query; #else ifp->if_start = igb_start; IFQ_SET_MAXLEN(&ifp->if_snd, adapter->num_tx_desc - 1); @@ -3363,7 +3412,7 @@ igb_allocate_queues(struct adapter *adap } #ifndef IGB_LEGACY_TX /* Allocate a buf ring */ - txr->br = buf_ring_alloc(igb_buf_ring_size, M_DEVBUF, + txr->br = drbr_alloc(M_DEVBUF, M_WAITOK, &txr->tx_mtx); #endif } @@ -3423,7 +3472,7 @@ err_tx_desc: free(adapter->rx_rings, M_DEVBUF); rx_fail: #ifndef IGB_LEGACY_TX - buf_ring_free(txr->br, M_DEVBUF); + drbr_free(txr->br, M_DEVBUF); #endif free(adapter->tx_rings, M_DEVBUF); tx_fail: @@ -3541,6 +3590,7 @@ igb_setup_transmit_ring(struct tx_ring * /* Set number of descriptors available */ txr->tx_avail = adapter->num_tx_desc; + txr->bytes_on_ring = 0; bus_dmamap_sync(txr->txdma.dma_tag, txr->txdma.dma_map, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); @@ -3682,7 +3732,7 @@ igb_free_transmit_buffers(struct tx_ring } #ifndef IGB_LEGACY_TX if (txr->br != NULL) - buf_ring_free(txr->br, M_DEVBUF); + drbr_free(txr->br, M_DEVBUF); #endif if (txr->tx_buffers != NULL) { free(txr->tx_buffers, M_DEVBUF); @@ -4016,6 +4066,8 @@ igb_txeof(struct tx_ring *txr) if (buf->m_head) { txr->bytes += buf->m_head->m_pkthdr.len; + atomic_subtract_long(&txr->bytes_on_ring, + (uint64_t)buf->m_head->m_pkthdr.len); bus_dmamap_sync(txr->txtag, buf->map, BUS_DMASYNC_POSTWRITE); @@ -5639,7 +5691,7 @@ igb_add_hw_stats(struct adapter *adapter queue_node = SYSCTL_ADD_NODE(ctx, child, OID_AUTO, namebuf, CTLFLAG_RD, NULL, "Queue Name"); queue_list = SYSCTL_CHILDREN(queue_node); - + drbr_add_sysctl_stats(dev, queue_list, txr->br); SYSCTL_ADD_PROC(ctx, queue_list, OID_AUTO, "interrupt_rate", CTLFLAG_RD, &adapter->queues[i], sizeof(&adapter->queues[i]), Modified: projects/rrs_mqueue/sys/dev/e1000/if_igb.h ============================================================================== --- projects/rrs_mqueue/sys/dev/e1000/if_igb.h Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/e1000/if_igb.h Mon Mar 24 11:43:35 2014 (r263682) @@ -309,12 +309,14 @@ struct tx_ring { IGB_QUEUE_DEPLETED = 8, } queue_status; u32 txd_cmd; - bus_dma_tag_t txtag; char mtx_name[16]; #ifndef IGB_LEGACY_TX - struct buf_ring *br; + struct drbr_ring *br; struct task txq_task; #endif + bus_dma_tag_t txtag; + volatile u_long bytes_on_ring; + u32 bytes; /* used for AIM */ u32 packets; /* Soft Stats */ @@ -371,17 +373,17 @@ struct adapter { struct device *dev; struct cdev *led_dev; - struct resource *pci_mem; - struct resource *msix_mem; - int memrid; - + struct resource *pci_mem; + struct resource *msix_mem; + uint64_t ring_bytes_max; + int memrid; /* * Interrupt resources: this set is * either used for legacy, or for Link * when doing MSIX */ - void *tag; - struct resource *res; + void *tag; + struct resource *res; struct ifmedia media; struct callout timer; Modified: projects/rrs_mqueue/sys/dev/fdt/fdt_common.c ============================================================================== --- projects/rrs_mqueue/sys/dev/fdt/fdt_common.c Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/fdt/fdt_common.c Mon Mar 24 11:43:35 2014 (r263682) @@ -182,7 +182,6 @@ fdt_is_compatible(phandle_t node, const compat += l; len -= l; } - return (rv); } @@ -544,15 +543,18 @@ fdt_get_phyaddr(phandle_t node, device_t if (OF_getencprop(node, "phy-handle", (void *)&phy_handle, sizeof(phy_handle)) <= 0) return (ENXIO); - phy_node = OF_xref_phandle(phy_handle); + device_printf(dev, "phy-handle:0x%x phy_ihandle:0x%x phy_node:0x%x\n", + (uint32_t)phy_handle, (uint32_t)phy_ihandle, + (uint32_t)phy_node); if (OF_getprop(phy_node, "reg", (void *)&phy_reg, sizeof(phy_reg)) <= 0) return (ENXIO); + device_printf(dev, "reg:0x%x\n", (uint32_t)phy_reg); *phy_addr = fdt32_to_cpu(phy_reg); - + device_printf(dev, "tran to reg:0x%x\n", (uint32_t)*phy_addr); /* * Search for softc used to communicate with phy. */ Modified: projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.c ============================================================================== --- projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.c Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.c Mon Mar 24 11:43:35 2014 (r263682) @@ -845,7 +845,8 @@ ixgbe_mq_start_locked(struct ifnet *ifp, struct adapter *adapter = txr->adapter; struct mbuf *next; int enqueued = 0, err = 0; - + uint8_t qused; + if (((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) || adapter->link_active == 0) return (ENETDOWN); @@ -858,18 +859,18 @@ ixgbe_mq_start_locked(struct ifnet *ifp, if (next != NULL) err = drbr_enqueue(ifp, txr->br, next); #else - while ((next = drbr_peek(ifp, txr->br)) != NULL) { + while ((next = drbr_peek(ifp, txr->br, &qused)) != NULL) { if ((err = ixgbe_xmit(txr, &next)) != 0) { if (next == NULL) { - drbr_advance(ifp, txr->br); + drbr_advance(ifp, txr->br, qused); } else { - drbr_putback(ifp, txr->br, next); + drbr_putback(ifp, txr->br, next, qused); } #endif break; } #if __FreeBSD_version >= 901504 - drbr_advance(ifp, txr->br); + drbr_advance(ifp, txr->br, qused); #endif enqueued++; /* Send a copy of the frame to the BPF listener */ @@ -917,12 +918,10 @@ ixgbe_qflush(struct ifnet *ifp) { struct adapter *adapter = ifp->if_softc; struct tx_ring *txr = adapter->tx_rings; - struct mbuf *m; for (int i = 0; i < adapter->num_queues; i++, txr++) { IXGBE_TX_LOCK(txr); - while ((m = buf_ring_dequeue_sc(txr->br)) != NULL) - m_freem(m); + drbr_flush(ifp, txr->br); IXGBE_TX_UNLOCK(txr); } if_qflush(ifp); @@ -2887,7 +2886,7 @@ ixgbe_allocate_queues(struct adapter *ad } #ifndef IXGBE_LEGACY_TX /* Allocate a buf ring */ - txr->br = buf_ring_alloc(IXGBE_BR_SIZE, M_DEVBUF, + txr->br = drbr_alloc(M_DEVBUF, M_WAITOK, &txr->tx_mtx); if (txr->br == NULL) { device_printf(dev, @@ -3249,7 +3248,7 @@ ixgbe_free_transmit_buffers(struct tx_ri } #ifdef IXGBE_LEGACY_TX if (txr->br != NULL) - buf_ring_free(txr->br, M_DEVBUF); + drbr_free(txr->br, M_DEVBUF); #endif if (txr->tx_buffers != NULL) { free(txr->tx_buffers, M_DEVBUF); Modified: projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.h ============================================================================== --- projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.h Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.h Mon Mar 24 11:43:35 2014 (r263682) @@ -58,6 +58,7 @@ #include <net/ethernet.h> #include <net/if_dl.h> #include <net/if_media.h> +#include <net/drbr.h> #include <net/bpf.h> #include <net/if_types.h> @@ -313,7 +314,7 @@ struct tx_ring { bus_dma_tag_t txtag; char mtx_name[16]; #ifndef IXGBE_LEGACY_TX - struct buf_ring *br; + struct drbr_ring *br; struct task txq_task; #endif #ifdef IXGBE_FDIR Modified: projects/rrs_mqueue/sys/dev/ixgbe/ixv.c ============================================================================== --- projects/rrs_mqueue/sys/dev/ixgbe/ixv.c Mon Mar 24 11:19:25 2014 (r263681) +++ projects/rrs_mqueue/sys/dev/ixgbe/ixv.c Mon Mar 24 11:43:35 2014 (r263682) @@ -603,6 +603,7 @@ ixv_mq_start_locked(struct ifnet *ifp, s struct adapter *adapter = txr->adapter; struct mbuf *next; int enqueued, err = 0; + uint8_t qused; if ((ifp->if_drv_flags & (IFF_DRV_RUNNING | IFF_DRV_OACTIVE)) != IFF_DRV_RUNNING || adapter->link_active == 0) { @@ -623,16 +624,16 @@ ixv_mq_start_locked(struct ifnet *ifp, s } *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201403241143.s2OBhZ9m057529>