From owner-svn-src-user@FreeBSD.ORG Mon Nov 17 03:04:46 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 51CD21065673; Mon, 17 Nov 2008 03:04:46 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 279248FC17; Mon, 17 Nov 2008 03:04:46 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAH34kdW071362; Mon, 17 Nov 2008 03:04:46 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAH34k4A071361; Mon, 17 Nov 2008 03:04:46 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811170304.mAH34k4A071361@svn.freebsd.org> From: Lawrence Stewart Date: Mon, 17 Nov 2008 03:04:46 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185016 - user/lstewart/dummynet_7.x X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Nov 2008 03:04:46 -0000 Author: lstewart Date: Mon Nov 17 03:04:45 2008 New Revision: 185016 URL: http://svn.freebsd.org/changeset/base/185016 Log: Somewhere to maintain the backport of my dummynet changes to 7-STABLE. Added: user/lstewart/dummynet_7.x/ - copied from r185015, stable/7/ From owner-svn-src-user@FreeBSD.ORG Mon Nov 17 05:19:19 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3DBB9106564A; Mon, 17 Nov 2008 05:19:19 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 2A38A8FC0A; Mon, 17 Nov 2008 05:19:19 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAH5JJnL073992; Mon, 17 Nov 2008 05:19:19 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAH5JJTl073991; Mon, 17 Nov 2008 05:19:19 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811170519.mAH5JJTl073991@svn.freebsd.org> From: Kip Macy Date: Mon, 17 Nov 2008 05:19:19 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185019 - user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Nov 2008 05:19:19 -0000 Author: kmacy Date: Mon Nov 17 05:19:18 2008 New Revision: 185019 URL: http://svn.freebsd.org/changeset/base/185019 Log: enable tx coalescing by default Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Mon Nov 17 04:16:12 2008 (r185018) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Mon Nov 17 05:19:18 2008 (r185019) @@ -88,7 +88,7 @@ __FBSDID("$FreeBSD$"); extern int txq_fills; int multiq_tx_enable = 1; -int coalesce_tx_enable = 0; +int coalesce_tx_enable = 1; extern struct sysctl_oid_list sysctl__hw_cxgb_children; static int sleep_ticks = 1; From owner-svn-src-user@FreeBSD.ORG Mon Nov 17 07:03:05 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D7114106564A; Mon, 17 Nov 2008 07:03:05 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 8E7428FC16; Mon, 17 Nov 2008 07:03:05 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAH735qE075819; Mon, 17 Nov 2008 07:03:05 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAH735xT075818; Mon, 17 Nov 2008 07:03:05 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811170703.mAH735xT075818@svn.freebsd.org> From: Kip Macy Date: Mon, 17 Nov 2008 07:03:05 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185020 - user/kmacy/HEAD_fast_multi_xmit/sys/netinet X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Nov 2008 07:03:05 -0000 Author: kmacy Date: Mon Nov 17 07:03:05 2008 New Revision: 185020 URL: http://svn.freebsd.org/changeset/base/185020 Log: Make ipv4 pcpu and global flowtable size tunables Modified: user/kmacy/HEAD_fast_multi_xmit/sys/netinet/ip_input.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/netinet/ip_input.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/netinet/ip_input.c Mon Nov 17 05:19:18 2008 (r185019) +++ user/kmacy/HEAD_fast_multi_xmit/sys/netinet/ip_input.c Mon Nov 17 07:03:05 2008 (r185020) @@ -206,6 +206,23 @@ SYSCTL_V_INT(V_NET, vnet_inet, _net_inet ipstealth, 0, "IP stealth mode, no TTL decrementation on forwarding"); #endif +static int ipv4_pcpu_flowtable_size = 2048; +TUNABLE_INT("net.inet.ip.pcpu_flowtable_size", &ipv4_pcpu_flowtable_size); +SYSCTL_V_INT(V_NET, vnet_inet, _net_inet_ip, OID_AUTO, pcpu_flowtable_size, + CTLFLAG_RDTUN, ipv4_pcpu_flowtable_size, 0, + "number of entries in the per cpu flow caches"); + +#ifdef RADIX_MPATH +static int ipv4_global_flowtable_size = 128*1024; +#else +static int ipv4_global_flowtable_size = 16*1024; +#endif +TUNABLE_INT("net.inet.ip.global_flowtable_size", &ipv4_global_flowtable_size); +SYSCTL_V_INT(V_NET, vnet_inet, _net_inet_ip, OID_AUTO, global_flowtable_size, + CTLFLAG_RDTUN, ipv4_global_flowtable_size, 0, + "number of entries in the global flow cache"); + + /* * ipfw_ether and ipfw_bridge hooks. * XXX: Temporary until those are converted to pfil_hooks as well. @@ -216,7 +233,6 @@ int fw_one_pass = 1; struct flowtable *ipv4_ft; struct flowtable *ipv4_forward_ft; - static void ip_freef(struct ipqhead *, struct ipq *); /* @@ -283,13 +299,8 @@ ip_init(void) mtx_init(&ipintrq.ifq_mtx, "ip_inq", NULL, MTX_DEF); netisr_register(NETISR_IP, ip_input, &ipintrq, 0); - ipv4_ft = flowtable_alloc(2048, FL_PCPU); -#ifdef RADIX_MPATH - ipv4_forward_ft = flowtable_alloc(128*1024, FL_HASH_PORTS); -#else - ipv4_forward_ft = flowtable_alloc(16*1024, 0); -#endif - + ipv4_ft = flowtable_alloc(ipv4_pcpu_flowtable_size, FL_PCPU); + ipv4_forward_ft = flowtable_alloc(ipv4_global_flowtable_size, FL_HASH_PORTS); } void From owner-svn-src-user@FreeBSD.ORG Mon Nov 17 07:39:42 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2B6F51065674; Mon, 17 Nov 2008 07:39:39 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id D81F98FC18; Mon, 17 Nov 2008 07:39:39 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAH7ddVQ076510; Mon, 17 Nov 2008 07:39:39 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAH7ddOe076509; Mon, 17 Nov 2008 07:39:39 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811170739.mAH7ddOe076509@svn.freebsd.org> From: Kip Macy Date: Mon, 17 Nov 2008 07:39:39 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185022 - user/kmacy/HEAD_fast_multi_xmit/sys/netinet X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Nov 2008 07:39:42 -0000 Author: kmacy Date: Mon Nov 17 07:39:39 2008 New Revision: 185022 URL: http://svn.freebsd.org/changeset/base/185022 Log: If ip_output_fast is being called from ip_output and there is no cached ARP, fall back to ip_output_legacy Modified: user/kmacy/HEAD_fast_multi_xmit/sys/netinet/ip_output.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/netinet/ip_output.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/netinet/ip_output.c Mon Nov 17 07:09:40 2008 (r185021) +++ user/kmacy/HEAD_fast_multi_xmit/sys/netinet/ip_output.c Mon Nov 17 07:39:39 2008 (r185022) @@ -663,6 +663,13 @@ ip_output_fast(struct mbuf *m, struct mb route_to_rtentry_info(ro, NULL, ri); else if (flowtable_lookup(ipv4_ft, m, ri)) return (ENETUNREACH); + + /* + * Rather than reverting to non-standard ARP + * lookup semantics simply fallback to ip_output_legacy + */ + if ((ri->ri_flags & RTF_DESTEN_VALID) == 0) + return (EINVAL); } if (opt) { @@ -1151,10 +1158,15 @@ int ip_output(struct mbuf *m, struct mbuf *opt, struct route *ro, int flags, struct ip_moptions *imo, struct inpcb *inp) { + int error; + if (flowtable_enable) - return (ip_output_fast(m, opt, ro, flags, imo, inp)); - else - return (ip_output_legacy(m, opt, ro, flags, imo, inp)); + error = ip_output_fast(m, opt, ro, flags, imo, inp); + + if ((flowtable_enable == 0) || (error == EINVAL)) + error = ip_output_legacy(m, opt, ro, flags, imo, inp); + + return (error); } /* From owner-svn-src-user@FreeBSD.ORG Tue Nov 18 13:36:02 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17FD21065672; Tue, 18 Nov 2008 13:36:02 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 0705D8FC0A; Tue, 18 Nov 2008 13:36:02 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAIDa1S2079355; Tue, 18 Nov 2008 13:36:01 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAIDa1Ld079350; Tue, 18 Nov 2008 13:36:01 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811181336.mAIDa1Ld079350@svn.freebsd.org> From: Lawrence Stewart Date: Tue, 18 Nov 2008 13:36:01 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185051 - in user/lstewart/dummynet_8.x: sbin/ipfw sys sys/kern sys/modules sys/modules/alq sys/netinet sys/sys X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Nov 2008 13:36:02 -0000 Author: lstewart Date: Tue Nov 18 13:36:01 2008 New Revision: 185051 URL: http://svn.freebsd.org/changeset/base/185051 Log: - Merge in my alq varlen patch for use by my dummynet logging mods - Fix up the ipfw man page change from my previous DPD commit (pointed out by brueffer@) - Add first pass attempt at detailed logging to dummynet based on a patch I use in house at work. Needs more polishing, but should be functional as is (haven't really tested this version of the patch at all yet). Added: user/lstewart/dummynet_8.x/sys/modules/alq/ - copied from r184954, user/lstewart/alq_varlen_8.x/sys/modules/alq/ Modified: user/lstewart/dummynet_8.x/sbin/ipfw/ipfw.8 user/lstewart/dummynet_8.x/sys/ (props changed) user/lstewart/dummynet_8.x/sys/kern/kern_alq.c user/lstewart/dummynet_8.x/sys/modules/Makefile user/lstewart/dummynet_8.x/sys/netinet/ip_dummynet.c user/lstewart/dummynet_8.x/sys/netinet/ip_dummynet.h user/lstewart/dummynet_8.x/sys/sys/alq.h Modified: user/lstewart/dummynet_8.x/sbin/ipfw/ipfw.8 ============================================================================== --- user/lstewart/dummynet_8.x/sbin/ipfw/ipfw.8 Tue Nov 18 13:24:38 2008 (r185050) +++ user/lstewart/dummynet_8.x/sbin/ipfw/ipfw.8 Tue Nov 18 13:36:01 2008 (r185051) @@ -1994,12 +1994,13 @@ Packet loss set. Argument .Ar packet-loss-set is a comma-delimited string of the form 10,30-31,1000 identifying the specific -packets entering a queue/pipe to drop. In the given example, the 10th, 30th, -31st and 1000th packet to enter the pipe/queue would be dropped. Clearing the -counters on a pipe will cause the +packets entering a queue/pipe to drop. +In the given example, the 10th, 30th, 31st and 1000th packet to enter the +pipe/queue would be dropped. +Clearing the counters on a pipe will cause the .Ar packet-loss-set -to be evaluated again from scratch. Use of this option mutually excludes use of -the +to be evaluated again from scratch. +Use of this option mutually excludes use of the .Nm plr option. .Pp Modified: user/lstewart/dummynet_8.x/sys/kern/kern_alq.c ============================================================================== --- user/lstewart/dummynet_8.x/sys/kern/kern_alq.c Tue Nov 18 13:24:38 2008 (r185050) +++ user/lstewart/dummynet_8.x/sys/kern/kern_alq.c Tue Nov 18 13:36:01 2008 (r185051) @@ -1,5 +1,6 @@ /*- * Copyright (c) 2002, Jeffrey Roberson + * Copyright (c) 2008, Lawrence Stewart * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -51,14 +52,18 @@ __FBSDID("$FreeBSD$"); struct alq { int aq_entmax; /* Max entries */ int aq_entlen; /* Entry length */ + int aq_freebytes; /* Bytes available in buffer */ + int aq_buflen; /* Total length of our buffer */ char *aq_entbuf; /* Buffer for stored entries */ + int aq_writehead; + int aq_writetail; int aq_flags; /* Queue flags */ struct mtx aq_mtx; /* Queue lock */ struct vnode *aq_vp; /* Open vnode handle */ struct ucred *aq_cred; /* Credentials of the opening thread */ - struct ale *aq_first; /* First ent */ - struct ale *aq_entfree; /* First free ent */ - struct ale *aq_entvalid; /* First ent valid for writing */ + //struct ale *aq_first; /* First ent */ + //struct ale *aq_entfree; /* First free ent */ + //struct ale *aq_entvalid; /* First ent valid for writing */ LIST_ENTRY(alq) aq_act; /* List of active queues */ LIST_ENTRY(alq) aq_link; /* List of all queues */ }; @@ -182,8 +187,14 @@ ald_daemon(void) ALD_LOCK(); for (;;) { - while ((alq = LIST_FIRST(&ald_active)) == NULL) - msleep(&ald_active, &ald_mtx, PWAIT, "aldslp", 0); + while ((alq = LIST_FIRST(&ald_active)) == NULL + && !ald_shutingdown) + mtx_sleep(&ald_active, &ald_mtx, PWAIT, "aldslp", 0); + + if (ald_shutingdown) { + ALD_UNLOCK(); + break; + } ALQ_LOCK(alq); ald_deactivate(alq); @@ -191,9 +202,11 @@ ald_daemon(void) needwakeup = alq_doio(alq); ALQ_UNLOCK(alq); if (needwakeup) - wakeup(alq); + wakeup_one(alq); ALD_LOCK(); } + + kthread_exit(); } static void @@ -204,6 +217,12 @@ ald_shutdown(void *arg, int howto) ALD_LOCK(); ald_shutingdown = 1; + /* wake ald_daemon so that it exits*/ + wakeup(&ald_active); + + /* wait for ald_daemon to exit */ + mtx_sleep(ald_thread, &ald_mtx, PWAIT, "aldslp", 0); + while ((alq = LIST_FIRST(&ald_queues)) != NULL) { LIST_REMOVE(alq, aq_link); ALD_UNLOCK(); @@ -244,41 +263,45 @@ alq_doio(struct alq *alq) struct vnode *vp; struct uio auio; struct iovec aiov[2]; - struct ale *ale; - struct ale *alstart; int totlen; int iov; int vfslocked; + KASSERT(alq->aq_freebytes != alq->aq_buflen, + ("%s: queue emtpy!", __func__) + ); + vp = alq->aq_vp; td = curthread; totlen = 0; iov = 0; - alstart = ale = alq->aq_entvalid; - alq->aq_entvalid = NULL; - bzero(&aiov, sizeof(aiov)); bzero(&auio, sizeof(auio)); - do { - if (aiov[iov].iov_base == NULL) - aiov[iov].iov_base = ale->ae_data; - aiov[iov].iov_len += alq->aq_entlen; - totlen += alq->aq_entlen; - /* Check to see if we're wrapping the buffer */ - if (ale->ae_data + alq->aq_entlen != ale->ae_next->ae_data) - iov++; - ale->ae_flags &= ~AE_VALID; - ale = ale->ae_next; - } while (ale->ae_flags & AE_VALID); + /* start the write from the location of our buffer tail pointer */ + aiov[iov].iov_base = alq->aq_entbuf + alq->aq_writetail; + + if (alq->aq_writetail < alq->aq_writehead) { + /* buffer not wrapped */ + totlen = aiov[iov].iov_len = alq->aq_writehead - + alq->aq_writetail; + } else { + /* + * buffer wrapped, requires 2 aiov entries: + * - first is from writetail to end of buffer + * - second is from start of buffer to writehead + */ + aiov[iov].iov_len = alq->aq_buflen - alq->aq_writetail; + iov++; + aiov[iov].iov_base = alq->aq_entbuf; + aiov[iov].iov_len = alq->aq_writehead; + totlen = aiov[0].iov_len + aiov[1].iov_len; + } alq->aq_flags |= AQ_FLUSHING; ALQ_UNLOCK(alq); - if (iov == 2 || aiov[iov].iov_base == NULL) - iov--; - auio.uio_iov = &aiov[0]; auio.uio_offset = 0; auio.uio_segflg = UIO_SYSSPACE; @@ -308,8 +331,17 @@ alq_doio(struct alq *alq) ALQ_LOCK(alq); alq->aq_flags &= ~AQ_FLUSHING; - if (alq->aq_entfree == NULL) - alq->aq_entfree = alstart; + /* Adjust writetail as required, taking into account wrapping */ + alq->aq_writetail += (iov == 2) ? aiov[1].iov_len : totlen; + alq->aq_freebytes += totlen; + + /* + * If we just flushed the buffer completely, + * reset indexes to 0 to minimise buffer wraps + * This is also required to ensure alq_getn() can't wedge itself + */ + if (alq->aq_freebytes == alq->aq_buflen) + alq->aq_writehead = alq->aq_writetail = 0; if (alq->aq_flags & AQ_WANTED) { alq->aq_flags &= ~AQ_WANTED; @@ -340,13 +372,13 @@ alq_open(struct alq **alqp, const char * { struct thread *td; struct nameidata nd; - struct ale *ale; - struct ale *alp; struct alq *alq; - char *bufp; int flags; int error; - int i, vfslocked; + int vfslocked; + + KASSERT(size > 0, ("%s: size <= 0", __func__)); + KASSERT(count >= 0, ("%s: count < 0", __func__)); *alqp = NULL; td = curthread; @@ -365,31 +397,27 @@ alq_open(struct alq **alqp, const char * VFS_UNLOCK_GIANT(vfslocked); alq = malloc(sizeof(*alq), M_ALD, M_WAITOK|M_ZERO); - alq->aq_entbuf = malloc(count * size, M_ALD, M_WAITOK|M_ZERO); - alq->aq_first = malloc(sizeof(*ale) * count, M_ALD, M_WAITOK|M_ZERO); alq->aq_vp = nd.ni_vp; alq->aq_cred = crhold(cred); - alq->aq_entmax = count; - alq->aq_entlen = size; - alq->aq_entfree = alq->aq_first; mtx_init(&alq->aq_mtx, "ALD Queue", NULL, MTX_SPIN|MTX_QUIET); - bufp = alq->aq_entbuf; - ale = alq->aq_first; - alp = NULL; - - /* Match up entries with buffers */ - for (i = 0; i < count; i++) { - if (alp) - alp->ae_next = ale; - ale->ae_data = bufp; - alp = ale; - ale++; - bufp += size; + if (count > 0) { + /* fixed length messages */ + alq->aq_buflen = size * count; + alq->aq_entmax = count; + alq->aq_entlen = size; + } else { + /* variable length messages */ + alq->aq_buflen = size; + alq->aq_entmax = 0; + alq->aq_entlen = 0; } - alp->ae_next = alq->aq_first; + alq->aq_freebytes = alq->aq_buflen; + alq->aq_entbuf = malloc(alq->aq_buflen, M_ALD, M_WAITOK|M_ZERO); + + alq->aq_writehead = alq->aq_writetail = 0; if ((error = ald_add(alq)) != 0) return (error); @@ -403,46 +431,180 @@ alq_open(struct alq **alqp, const char * * wait or return an error depending on the value of waitok. */ int -alq_write(struct alq *alq, void *data, int waitok) +alq_write(struct alq *alq, void *data, int flags) { - struct ale *ale; + /* should only be called in fixed length message (legacy) mode */ + KASSERT(alq->aq_entmax > 0 && alq->aq_entlen > 0, + ("%s: fixed length write on variable length queue", __func__) + ); + return (alq_writen(alq, data, alq->aq_entlen, flags)); +} + +int +alq_writen(struct alq *alq, void *data, int len, int flags) +{ + int activate = 0; + int copy = len; + + KASSERT(len > 0 && len < alq->aq_buflen, + ("%s: len <= 0 || len > alq->aq_buflen", __func__) + ); + + ALQ_LOCK(alq); + + /* + * If the message is larger than our underlying buffer or + * there is not enough free space in our underlying buffer + * to accept the message and the user can't wait, return + */ + if ((len > alq->aq_buflen) || + ((flags & ALQ_NOWAIT) && (alq->aq_freebytes < len))) { + ALQ_UNLOCK(alq); + return (EWOULDBLOCK); + } + + /* + * ALQ_WAITOK or alq->aq_freebytes > len, + * either spin until we have enough free bytes (former) or skip (latter) + */ + while (alq->aq_freebytes < len && (alq->aq_flags & AQ_SHUTDOWN) == 0) { + alq->aq_flags |= AQ_WANTED; + msleep_spin(alq, &alq->aq_mtx, "alqwriten", 0); + } - if ((ale = alq_get(alq, waitok)) == NULL) + /* + * we need to serialise wakups to ensure records remain in order... + * therefore, wakeup the next thread in the queue waiting for + * alq resources to be available + * (technically this is only required if we actually entered the above + * while loop) + */ + wakeup_one(alq); + + /* bail if we're shutting down */ + if (alq->aq_flags & AQ_SHUTDOWN) { + ALQ_UNLOCK(alq); return (EWOULDBLOCK); + } + + /* + * if we need to wrap the buffer to accommodate the write, + * we'll need 2 calls to bcopy + */ + if ((alq->aq_buflen - alq->aq_writehead) < len) + copy = alq->aq_buflen - alq->aq_writehead; + + /* copy (part of) message to the buffer */ + bcopy(data, alq->aq_entbuf + alq->aq_writehead, copy); + alq->aq_writehead += copy; + + if (copy != len) { + /* + * wrap the buffer by copying the remainder of our message + * to the start of the buffer and resetting the head ptr + */ + bcopy(data, alq->aq_entbuf, len - copy); + alq->aq_writehead = copy; + } - bcopy(data, ale->ae_data, alq->aq_entlen); - alq_post(alq, ale); + alq->aq_freebytes -= len; + + if ((alq->aq_flags & AQ_ACTIVE) == 0) { + alq->aq_flags |= AQ_ACTIVE; + activate = 1; + } + + ALQ_UNLOCK(alq); + + if (activate) { + ALD_LOCK(); + ald_activate(alq); + ALD_UNLOCK(); + } return (0); } struct ale * -alq_get(struct alq *alq, int waitok) +alq_get(struct alq *alq, int flags) +{ + /* should only be called in fixed length message (legacy) mode */ + KASSERT(alq->aq_entmax > 0 && alq->aq_entlen > 0, + ("%s: fixed length get on variable length queue", __func__) + ); + return (alq_getn(alq, alq->aq_entlen, flags)); +} + +struct ale * +alq_getn(struct alq *alq, int len, int flags) { struct ale *ale; - struct ale *aln; + int contigbytes; - ale = NULL; + ale = malloc( sizeof(struct ale), + M_ALD, + (flags & ALQ_NOWAIT) ? M_NOWAIT : M_WAITOK + ); + + if (ale == NULL) + return (NULL); ALQ_LOCK(alq); - /* Loop until we get an entry or we're shutting down */ - while ((alq->aq_flags & AQ_SHUTDOWN) == 0 && - (ale = alq->aq_entfree) == NULL && - (waitok & ALQ_WAITOK)) { - alq->aq_flags |= AQ_WANTED; - msleep_spin(alq, &alq->aq_mtx, "alqget", 0); + /* determine the number of free contiguous bytes */ + if (alq->aq_writehead <= alq->aq_writetail) + contigbytes = alq->aq_freebytes; + else + contigbytes = alq->aq_buflen - alq->aq_writehead; + + /* + * If the message is larger than our underlying buffer or + * there is not enough free contiguous space in our underlying buffer + * to accept the message and the user can't wait, return + */ + if ((len > alq->aq_buflen) || + ((flags & ALQ_NOWAIT) && (contigbytes < len))) { + ALQ_UNLOCK(alq); + return (NULL); } - if (ale != NULL) { - aln = ale->ae_next; - if ((aln->ae_flags & AE_VALID) == 0) - alq->aq_entfree = aln; + /* + * ALQ_WAITOK or contigbytes > len, + * either spin until we have enough free contiguous bytes (former) + * or skip (latter) + */ + while (contigbytes < len && (alq->aq_flags & AQ_SHUTDOWN) == 0) { + alq->aq_flags |= AQ_WANTED; + msleep_spin(alq, &alq->aq_mtx, "alqgetn", 0); + if (alq->aq_writehead <= alq->aq_writetail) + contigbytes = alq->aq_freebytes; else - alq->aq_entfree = NULL; - } else + contigbytes = alq->aq_buflen - alq->aq_writehead; + } + + /* + * we need to serialise wakups to ensure records remain in order... + * therefore, wakeup the next thread in the queue waiting for + * alq resources to be available + * (technically this is only required if we actually entered the above + * while loop) + */ + wakeup_one(alq); + + /* bail if we're shutting down */ + if (alq->aq_flags & AQ_SHUTDOWN) { ALQ_UNLOCK(alq); + return (NULL); + } + /* + * If we are here, we have a contiguous number of bytes >= len + * available in our buffer starting at aq_writehead. + */ + ale->ae_data = alq->aq_entbuf + alq->aq_writehead; + ale->ae_datalen = len; + alq->aq_writehead += len; + alq->aq_freebytes -= len; return (ale); } @@ -452,11 +614,6 @@ alq_post(struct alq *alq, struct ale *al { int activate; - ale->ae_flags |= AE_VALID; - - if (alq->aq_entvalid == NULL) - alq->aq_entvalid = ale; - if ((alq->aq_flags & AQ_ACTIVE) == 0) { alq->aq_flags |= AQ_ACTIVE; activate = 1; @@ -464,11 +621,14 @@ alq_post(struct alq *alq, struct ale *al activate = 0; ALQ_UNLOCK(alq); + if (activate) { ALD_LOCK(); ald_activate(alq); ALD_UNLOCK(); } + + free(ale, M_ALD); } void @@ -487,7 +647,7 @@ alq_flush(struct alq *alq) ALQ_UNLOCK(alq); if (needwakeup) - wakeup(alq); + wakeup_one(alq); } /* @@ -509,7 +669,49 @@ alq_close(struct alq *alq) alq_shutdown(alq); mtx_destroy(&alq->aq_mtx); - free(alq->aq_first, M_ALD); free(alq->aq_entbuf, M_ALD); free(alq, M_ALD); } + +static int alq_load_handler(module_t mod, int what, void *arg) +{ + int ret = 0; + + switch(what) { + case MOD_LOAD: + case MOD_UNLOAD: + case MOD_SHUTDOWN: + break; + + case MOD_QUIESCE: + ALD_LOCK(); + /* only allow unload if there are no open queues */ + if (LIST_FIRST(&ald_queues) == NULL) { + ald_shutingdown = 1; + ALD_UNLOCK(); + ald_shutdown(NULL, 0); + mtx_destroy(&ald_mtx); + } else { + ALD_UNLOCK(); + ret = EBUSY; + } + break; + + default: + ret = EINVAL; + break; + } + + return (ret); +} + +/* basic module data */ +static moduledata_t alq_mod = +{ + "alq", + alq_load_handler, /* execution entry point for the module */ + NULL +}; + +DECLARE_MODULE(alq, alq_mod, SI_SUB_SMP, SI_ORDER_ANY); +MODULE_VERSION(alq, 1); Modified: user/lstewart/dummynet_8.x/sys/modules/Makefile ============================================================================== --- user/lstewart/dummynet_8.x/sys/modules/Makefile Tue Nov 18 13:24:38 2008 (r185050) +++ user/lstewart/dummynet_8.x/sys/modules/Makefile Tue Nov 18 13:36:01 2008 (r185051) @@ -17,6 +17,7 @@ SUBDIR= ${_3dfx} \ ${_aic} \ aic7xxx \ aio \ + alq \ ${_amd} \ ale \ amr \ Modified: user/lstewart/dummynet_8.x/sys/netinet/ip_dummynet.c ============================================================================== --- user/lstewart/dummynet_8.x/sys/netinet/ip_dummynet.c Tue Nov 18 13:24:38 2008 (r185050) +++ user/lstewart/dummynet_8.x/sys/netinet/ip_dummynet.c Tue Nov 18 13:36:01 2008 (r185051) @@ -64,28 +64,39 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include #include #include #include +#include +#include +#include +#include + #include #include #include #include #include #include +#include #include #include #include #include +#include +#include #include /* for struct arpcom */ #include /* for ip6_input, ip6_output prototypes */ #include +#include + /* * We keep a private variable for the simulation time, but we could * probably use an existing one ("softticks" in sys/kern/kern_timeout.c) @@ -154,6 +165,73 @@ static struct callout dn_timeout; extern void (*bridge_dn_p)(struct mbuf *, struct ifnet *); +#define DN_LOG(fs, p, q, m, dropped, dir) \ + if (dn_log_enable) \ + dn_log((fs), (p), (q), (m), (dropped), (dir)); + +#define CAST_PTR_INT(X) (*((int*)(X))) + +struct log_node { + /* log msg creation timestamp */ + struct timeval tval; + /* + * direction of packet after dummynet finishes processing it + * (defined in ip_dummynet.h DN_TO_IP_OUT, DN_TO_IP_IN, ...) + */ + int direction; + /* + * pkt dropped yes/no + reason if dropped (see DN_DROP_X defines in + * ip_dummynet.h) + */ + uint32_t dropped; + /* hash of the pkt which triggered the log msg */ + uint32_t hash; + /* IP version log_node relates to; either INP_IPV4 or INP_IPV6 */ + uint8_t ipver; + /* flow set number */ + int fs_num; + /* flags set on the flow set */ + uint16_t fs_flags; + /* pipe number */ + int p_num; + /* current pipe occupancy */ + int p_len; + /* + * max queue len in either pkts or bytes (depending on whether + * DN_QSIZE_IS_BYTES is set in fs_flags) + */ + int q_max_len; + /* current queue occupancy in pkts */ + int q_len_pkts; + /* current queue occupancy in bytes */ + int q_len_bytes; + + STAILQ_ENTRY(log_node) nodes; +}; + +/* + * in_pcb.h defines INP_IPV4 as 0x1 and INP_IPV6 as 0x2, + * which we use as an index into this array + */ +static char ipver[3] = {'\0', '4', '6'}; +static int dn_sysctl_log_enable_handler(SYSCTL_HANDLER_ARGS); +static int dn_sysctl_logfile_name_handler(SYSCTL_HANDLER_ARGS); +static u_int dn_log_enable = 0; +static char dn_logfile[PATH_MAX] = "/var/log/dummynet.log\0"; +STAILQ_HEAD(loghead, log_node) log_queue = STAILQ_HEAD_INITIALIZER(log_queue); +static struct mtx dn_log_queue_mtx; +static int wait_for_log; +static struct alq *dn_alq = NULL; +static volatile uint32_t dn_exit_log_manager_thread = 0; +static struct thread *dn_log_manager_thr = NULL; + +#define DN_LOG_FILE_MODE 0644 +#define DN_ALQ_BUFLEN 200000 +#define DN_MAX_LOG_MSG_LEN 60 + +#define DN_LOG_DISABLE 0 +#define DN_LOG_ENABLE 1 + #ifdef SYSCTL_NODE SYSCTL_NODE(_net_inet_ip, OID_AUTO, dummynet, CTLFLAG_RW, 0, "Dummynet"); SYSCTL_INT(_net_inet_ip_dummynet, OID_AUTO, hash_size, @@ -206,6 +284,12 @@ SYSCTL_LONG(_net_inet_ip_dummynet, OID_A CTLFLAG_RW, &pipe_slot_limit, 0, "Upper limit in slots for pipe queue."); SYSCTL_LONG(_net_inet_ip_dummynet, OID_AUTO, pipe_byte_limit, CTLFLAG_RW, &pipe_byte_limit, 0, "Upper limit in bytes for pipe queue."); +SYSCTL_OID(_net_inet_ip_dummynet, OID_AUTO, log_enable, CTLTYPE_UINT|CTLFLAG_RW, + &dn_log_enable, 0, &dn_sysctl_log_enable_handler, "IU", + "switch dummynet data logging on/off"); +SYSCTL_PROC(_net_inet_ip_dummynet, OID_AUTO, logfile, + CTLTYPE_STRING|CTLFLAG_RW, &dn_logfile, sizeof(dn_logfile), + &dn_sysctl_logfile_name_handler, "A", "file to save dummynet log data to"); #endif #ifdef DUMMYNET_DEBUG @@ -451,6 +535,362 @@ heap_free(struct dn_heap *h) * --- end of heap management functions --- */ +static __inline void +dn_process_log_node(struct log_node * log_node) +{ + char dn_log_msg[DN_MAX_LOG_MSG_LEN]; + + /* construct our log message */ + snprintf( dn_log_msg, + DN_MAX_LOG_MSG_LEN, + "%d,0x%08x,%u.%06u,%u,%u,%d,0x%04x,%d,%d,%d,%d,%d\n", + log_node->direction, + log_node->hash, + (unsigned int)log_node->tval.tv_sec, + (unsigned int)log_node->tval.tv_usec, + ipver[log_node->ipver], + log_node->dropped, + log_node->fs_num, + log_node->fs_flags, + log_node->p_num, + log_node->p_len, + log_node->q_max_len, + log_node->q_len_pkts, + log_node->q_len_bytes + ); + + alq_writen(dn_alq, dn_log_msg, strlen(dn_log_msg), ALQ_WAITOK); +} + +static void +dn_log_manager_thread(void *arg) +{ + struct log_node *log_node, *log_node_temp; + + /* loop until thread is signalled to exit */ + while (!dn_exit_log_manager_thread) { + /* + * sleep until we are signalled to wake because thread has + * been told to exit or until 1 tick has passed + */ + tsleep(&wait_for_log, PWAIT, "logwait", 1); + + /* Process logs until the queue is empty */ + do { + log_node = NULL; + + /* gain exclusive access to the queue */ + mtx_lock(&dn_log_queue_mtx); + + /* get the element at the head of the list */ + if ((log_node = STAILQ_FIRST(&log_queue)) != NULL) { + /* + * list wasn't empty, so let's remove the first + * element from the list. + * Note that STAILQ_REMOVE_HEAD doesn't delete + * the log_node struct itself. It just + * disentangles it from the list structure. + * We have a copy of the node's ptr stored + * in log_node. + */ + STAILQ_REMOVE_HEAD(&log_queue, nodes); + } + /* + * We've finished making changes to the list. Unlock it + * so the pfil hooks can continue queuing pkt_nodes + */ + mtx_unlock(&dn_log_queue_mtx); + + /* if we successfully get a log_node from the list */ + if (log_node != NULL) { + dn_process_log_node(log_node); + /* + * free the memory that was + * malloc'd in dn_log() + */ + free(log_node, M_DUMMYNET); + } + + } while (log_node != NULL); + } + + /* Flush all remaining log_nodes to the log file */ + + /* Lock the mutex so we gain exclusive access to the queue */ + mtx_lock(&dn_log_queue_mtx); + + STAILQ_FOREACH_SAFE(log_node, &log_queue, nodes, log_node_temp) { + dn_process_log_node(log_node); + STAILQ_REMOVE_HEAD(&log_queue, nodes); + free(log_node, M_DUMMYNET); + } + + /* Reinit the list to mark it as empty and virgin */ + STAILQ_INIT(&log_queue); + + /* We've finished making changes to the list. Safe to unlock it. */ + mtx_unlock(&dn_log_queue_mtx); + + /* kthread_exit calls wakeup on our thread's struct pointer */ + kthread_exit(); +} + +static int +dn_sysctl_logfile_name_handler(SYSCTL_HANDLER_ARGS) +{ + struct alq *new_alq; + + if (!req->newptr) + goto skip; + + /* if old filename and new filename are different */ + if (strncmp(dn_logfile, (char *)req->newptr, PATH_MAX)) { + + int error = alq_open( &new_alq, + req->newptr, + curthread->td_ucred, + DN_LOG_FILE_MODE, + DN_ALQ_BUFLEN, + 0 + ); + + /* bail if unable to create new alq */ + if (error) + return 1; + + /* + * If disabled, dn_alq == NULL so we simply close + * the alq as we've proved it can be opened. + * If enabled, close the existing alq and switch the old for the new + */ + if (dn_alq == NULL) + alq_close(new_alq); + else { + alq_close(dn_alq); + dn_alq = new_alq; + } + } + +skip: + return sysctl_handle_string(oidp, arg1, arg2, req); +} + +static int +dn_manage_logging(uint8_t action) +{ + int ret, error = 0; + struct timeval tval; + struct sbuf *s = NULL; + + /* init an autosizing sbuf that initially holds 200 chars */ + if ((s = sbuf_new(NULL, NULL, 200, SBUF_AUTOEXTEND)) == NULL) + return -1; + + if (action == DN_LOG_ENABLE) { + + /* create our alq */ + alq_open( &dn_alq, + dn_logfile, + curthread->td_ucred, + DN_LOG_FILE_MODE, + DN_ALQ_BUFLEN, + 0 + ); + + STAILQ_INIT(&log_queue); + + dn_exit_log_manager_thread = 0; + + ret = kthread_add( &dn_log_manager_thread, + NULL, + NULL, + &dn_log_manager_thr, + RFNOWAIT, + 0, + "dn_log_manager_thr" + ); + + microtime(&tval); + + sbuf_printf(s, + "enable_time_secs=%u\tenable_time_usecs=%06ld\thz=%u\tsysname=%s\tsysver=%u\n", + tval.tv_sec, + tval.tv_usec, + hz, + "FreeBSD", + __FreeBSD_version + ); + + sbuf_finish(s); + alq_writen(dn_alq, sbuf_data(s), sbuf_len(s), ALQ_WAITOK); + } + else if (action == DN_LOG_DISABLE && dn_log_manager_thr != NULL) { + + /* tell the log manager thread that it should exit now */ + dn_exit_log_manager_thread = 1; + + /* + * wake the pkt_manager thread so it realises that + * dn_exit_log_manager_thread = 1 and exits gracefully + */ + wakeup(&wait_for_log); + + /* wait for the pkt_manager thread to exit */ + tsleep(dn_log_manager_thr, PWAIT, "thrwait", 0); + + dn_log_manager_thr = NULL; + + microtime(&tval); + + sbuf_printf(s, + "disable_time_secs=%u\tdisable_time_usecs=%06ld", + tval.tv_sec, + tval.tv_usec + ); + + sbuf_printf(s, "\n"); + sbuf_finish(s); + alq_writen(dn_alq, sbuf_data(s), sbuf_len(s), ALQ_WAITOK); + alq_close(dn_alq); + dn_alq = NULL; + } + + sbuf_delete(s); + + /* + * XXX: Should be using ret to check if any functions fail + * and set error appropriately + */ + return error; +} + +static int +dn_sysctl_log_enable_handler(SYSCTL_HANDLER_ARGS) +{ + if (!req->newptr) + goto skip; + + /* if the value passed in isn't DISABLE or ENABLE, return an error */ + if (CAST_PTR_INT(req->newptr) != DN_LOG_DISABLE && + CAST_PTR_INT(req->newptr) != DN_LOG_ENABLE) + return 1; + + /* if we are changing state (DISABLE to ENABLE or vice versa) */ + if (CAST_PTR_INT(req->newptr) != dn_log_enable ) + if (dn_manage_logging(CAST_PTR_INT(req->newptr))) { + dn_manage_logging(DN_LOG_DISABLE); + return 1; + } + +skip: + return sysctl_handle_int(oidp, arg1, arg2, req); +} + +static uint32_t +hash_pkt(struct mbuf *m, uint32_t offset) +{ + register uint32_t hash = 0; + + while ((m != NULL) && (offset > m->m_len)) { + /* + * the IP packet payload does not start in this mbuf + * need to figure out which mbuf it starts in and what offset + * into the mbuf's data region the payload starts at + */ + offset -= m->m_len; + m = m->m_next; + } + + while (m != NULL) { + /* ensure there is data in the mbuf */ + if ((m->m_len - offset) > 0) { + hash = hash32_buf( m->m_data + offset, + m->m_len - offset, + hash + ); + } + + m = m->m_next; + offset = 0; + } + + return hash; +} + +static void +dn_log( struct dn_flow_set *fs, + struct dn_pipe *p, + struct dn_flow_queue *q, + struct mbuf *pkt, + u_int dropped, + int dir) +{ + struct log_node *log_node; + + DUMMYNET_LOCK_ASSERT(); + + /* M_NOWAIT flag required here */ + log_node = malloc(sizeof(struct log_node), M_DUMMYNET, M_NOWAIT); + + if (log_node == NULL) + return; + + /* set log_node struct members */ + microtime(&(log_node->tval)); + log_node->direction = dir; + log_node->dropped = dropped; + log_node->ipver = INP_IPV4; + log_node->fs_num = (dropped == DN_DROP_NOFS) ? + -1 : fs->fs_nr; + log_node->fs_flags = (dropped == DN_DROP_NOFS) ? + 0 : fs->flags_fs; + log_node->q_max_len = (dropped == DN_DROP_NOFS) ? + -1 : fs->qsize; + log_node->p_num = (dropped == DN_DROP_NOFS || + dropped == DN_DROP_NOP4Q) ? + -1 : p->pipe_nr; + log_node->p_len = (dropped == DN_DROP_NOFS || + dropped == DN_DROP_NOP4Q) ? + -1 : p->len; + log_node->q_len_pkts = (dropped == DN_DROP_NOFS || + dropped == DN_DROP_NOQ) ? + -1 : q->len; + log_node->q_len_bytes = (dropped == DN_DROP_NOFS || + dropped == DN_DROP_NOQ) ? + -1 : q->len_bytes; + + /* *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-user@FreeBSD.ORG Tue Nov 18 21:55:50 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DAA4C1065672; Tue, 18 Nov 2008 21:55:50 +0000 (UTC) (envelope-from adrian@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id AF0458FC12; Tue, 18 Nov 2008 21:55:50 +0000 (UTC) (envelope-from adrian@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAILtoOs089997; Tue, 18 Nov 2008 21:55:50 GMT (envelope-from adrian@svn.freebsd.org) Received: (from adrian@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAILtojh089996; Tue, 18 Nov 2008 21:55:50 GMT (envelope-from adrian@svn.freebsd.org) Message-Id: <200811182155.mAILtojh089996@svn.freebsd.org> From: Adrian Chadd Date: Tue, 18 Nov 2008 21:55:50 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185062 - user/adrian X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Nov 2008 21:55:50 -0000 Author: adrian Date: Tue Nov 18 21:55:50 2008 New Revision: 185062 URL: http://svn.freebsd.org/changeset/base/185062 Log: My playpen. Added: user/adrian/ From owner-svn-src-user@FreeBSD.ORG Tue Nov 18 21:56:30 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F4591065672; Tue, 18 Nov 2008 21:56:30 +0000 (UTC) (envelope-from adrian@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 0367C8FC18; Tue, 18 Nov 2008 21:56:30 +0000 (UTC) (envelope-from adrian@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAILuTDp090043; Tue, 18 Nov 2008 21:56:29 GMT (envelope-from adrian@svn.freebsd.org) Received: (from adrian@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAILuTdP090042; Tue, 18 Nov 2008 21:56:29 GMT (envelope-from adrian@svn.freebsd.org) Message-Id: <200811182156.mAILuTdP090042@svn.freebsd.org> From: Adrian Chadd Date: Tue, 18 Nov 2008 21:56:29 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185063 - user/adrian/spoof_bind X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Nov 2008 21:56:30 -0000 Author: adrian Date: Tue Nov 18 21:56:29 2008 New Revision: 185063 URL: http://svn.freebsd.org/changeset/base/185063 Log: add a project. Added: user/adrian/spoof_bind/ From owner-svn-src-user@FreeBSD.ORG Tue Nov 18 21:58:32 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 13D291065672; Tue, 18 Nov 2008 21:58:32 +0000 (UTC) (envelope-from adrian@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id DC4668FC08; Tue, 18 Nov 2008 21:58:31 +0000 (UTC) (envelope-from adrian@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAILwV07090126; Tue, 18 Nov 2008 21:58:31 GMT (envelope-from adrian@svn.freebsd.org) Received: (from adrian@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAILwVq5090125; Tue, 18 Nov 2008 21:58:31 GMT (envelope-from adrian@svn.freebsd.org) Message-Id: <200811182158.mAILwVq5090125@svn.freebsd.org> From: Adrian Chadd Date: Tue, 18 Nov 2008 21:58:31 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185064 - user/adrian/spoof_bind/sys X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Nov 2008 21:58:32 -0000 Author: adrian Date: Tue Nov 18 21:58:31 2008 New Revision: 185064 URL: http://svn.freebsd.org/changeset/base/185064 Log: Copy the kernel source over. Added: user/adrian/spoof_bind/sys/ - copied from r185063, head/sys/ From owner-svn-src-user@FreeBSD.ORG Tue Nov 18 22:01:38 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 36827106564A; Tue, 18 Nov 2008 22:01:38 +0000 (UTC) (envelope-from adrian@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 0AA358FC0C; Tue, 18 Nov 2008 22:01:38 +0000 (UTC) (envelope-from adrian@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAIM1bPP090267; Tue, 18 Nov 2008 22:01:37 GMT (envelope-from adrian@svn.freebsd.org) Received: (from adrian@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAIM1bpO090266; Tue, 18 Nov 2008 22:01:37 GMT (envelope-from adrian@svn.freebsd.org) Message-Id: <200811182201.mAIM1bpO090266@svn.freebsd.org> From: Adrian Chadd Date: Tue, 18 Nov 2008 22:01:37 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185065 - user/adrian/spoof_bind/sbin X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Nov 2008 22:01:38 -0000 Author: adrian Date: Tue Nov 18 22:01:37 2008 New Revision: 185065 URL: http://svn.freebsd.org/changeset/base/185065 Log: Bring over sbin/ to modify. Added: user/adrian/spoof_bind/sbin/ - copied from r185064, head/sbin/ From owner-svn-src-user@FreeBSD.ORG Tue Nov 18 22:11:33 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC7161065670; Tue, 18 Nov 2008 22:11:33 +0000 (UTC) (envelope-from adrian@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 903038FC19; Tue, 18 Nov 2008 22:11:33 +0000 (UTC) (envelope-from adrian@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAIMBXS1090515; Tue, 18 Nov 2008 22:11:33 GMT (envelope-from adrian@svn.freebsd.org) Received: (from adrian@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAIMBXTF090514; Tue, 18 Nov 2008 22:11:33 GMT (envelope-from adrian@svn.freebsd.org) Message-Id: <200811182211.mAIMBXTF090514@svn.freebsd.org> From: Adrian Chadd Date: Tue, 18 Nov 2008 22:11:33 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185066 - user/adrian/spoof_bind/share X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Nov 2008 22:11:33 -0000 Author: adrian Date: Tue Nov 18 22:11:33 2008 New Revision: 185066 URL: http://svn.freebsd.org/changeset/base/185066 Log: bring over 'share' Added: user/adrian/spoof_bind/share/ - copied from r185065, head/share/ From owner-svn-src-user@FreeBSD.ORG Tue Nov 18 22:17:18 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B266106564A; Tue, 18 Nov 2008 22:17:18 +0000 (UTC) (envelope-from adrian@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 65F478FC17; Tue, 18 Nov 2008 22:17:18 +0000 (UTC) (envelope-from adrian@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAIMHING090653; Tue, 18 Nov 2008 22:17:18 GMT (envelope-from adrian@svn.freebsd.org) Received: (from adrian@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAIMHHjm090641; Tue, 18 Nov 2008 22:17:17 GMT (envelope-from adrian@svn.freebsd.org) Message-Id: <200811182217.mAIMHHjm090641@svn.freebsd.org> From: Adrian Chadd Date: Tue, 18 Nov 2008 22:17:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185067 - in user/adrian/spoof_bind: sbin/ipfw share/man/man4 sys/conf sys/net sys/netinet X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Nov 2008 22:17:18 -0000 Author: adrian Date: Tue Nov 18 22:17:17 2008 New Revision: 185067 URL: http://svn.freebsd.org/changeset/base/185067 Log: Bring over Julian's non-local bind stuff into a subversion branch. Modified: user/adrian/spoof_bind/sbin/ipfw/ipfw2.c user/adrian/spoof_bind/share/man/man4/ip.4 user/adrian/spoof_bind/sys/conf/NOTES user/adrian/spoof_bind/sys/conf/options user/adrian/spoof_bind/sys/net/if_bridge.c user/adrian/spoof_bind/sys/netinet/in.h user/adrian/spoof_bind/sys/netinet/in_pcb.c user/adrian/spoof_bind/sys/netinet/in_pcb.h user/adrian/spoof_bind/sys/netinet/ip_fw.h user/adrian/spoof_bind/sys/netinet/ip_fw2.c user/adrian/spoof_bind/sys/netinet/ip_output.c Modified: user/adrian/spoof_bind/sbin/ipfw/ipfw2.c ============================================================================== --- user/adrian/spoof_bind/sbin/ipfw/ipfw2.c Tue Nov 18 22:11:33 2008 (r185066) +++ user/adrian/spoof_bind/sbin/ipfw/ipfw2.c Tue Nov 18 22:17:17 2008 (r185067) @@ -301,6 +301,7 @@ enum tokens { TOK_ANTISPOOF, TOK_IPSEC, TOK_COMMENT, + TOK_FOR_ME, TOK_PLR, TOK_NOERROR, @@ -433,6 +434,7 @@ struct _s_x rule_options[] = { { "uid", TOK_UID }, { "gid", TOK_GID }, { "jail", TOK_JAIL }, + { "for-me", TOK_FOR_ME }, { "in", TOK_IN }, { "limit", TOK_LIMIT }, { "keep-state", TOK_KEEPSTATE }, @@ -2042,6 +2044,10 @@ show_ipfw(struct ip_fw *rule, int pcwidt O_TAGGED); break; + case O_FOR_ME: + printf(" for-me"); + break; + default: printf(" [opcode %d len %d]", cmd->opcode, cmd->len); @@ -5663,6 +5669,10 @@ read_options: ac--; av++; break; + case TOK_FOR_ME: + fill_cmd(cmd, O_FOR_ME, 0, 0); + break; + default: errx(EX_USAGE, "unrecognised option [%d] %s\n", i, s); } Modified: user/adrian/spoof_bind/share/man/man4/ip.4 ============================================================================== --- user/adrian/spoof_bind/share/man/man4/ip.4 Tue Nov 18 22:11:33 2008 (r185066) +++ user/adrian/spoof_bind/share/man/man4/ip.4 Tue Nov 18 22:17:17 2008 (r185067) @@ -292,6 +292,29 @@ cmsg_level = IPPROTO_IP cmsg_type = IP_RECVIF .Ed .Pp +If the +.Dv IP_NONLOCALOK +options is set then the checking of local bind addresses against addresses +assigned to local interfaces is disabled. +The kernel must have been compiled with the +.Dv IP_NONLOCALBIND option, and the sysctl +.Va net.inet.ip.nonlocalok +should be set to 1. +The option needs to be set on the socket before the +.Xr bind 2 +system call is used on it. +.Bd -literal +u_char spoofing = 1; /* 0 = disable (default), 1 = enable */ + +setsockopt(s, IPPROTO_IP, IP_NONLOCALOK, &spoofing, sizeof(spoofing)); +ret = bind (...); +.Ed +.Pp +This behaviour is not for general use and is +included for use in servers that are implementing fully +transparent proxies. Use of this option on general purpose +systems is strongly discouraged. +.Pp .Dv IP_PORTRANGE may be used to set the port range used for selecting a local port number on a socket with an unspecified (zero) port number. Modified: user/adrian/spoof_bind/sys/conf/NOTES ============================================================================== --- user/adrian/spoof_bind/sys/conf/NOTES Tue Nov 18 22:11:33 2008 (r185066) +++ user/adrian/spoof_bind/sys/conf/NOTES Tue Nov 18 22:17:17 2008 (r185067) @@ -633,6 +633,14 @@ options ALTQ_PRIQ # Priority Queueing options ALTQ_NOPCC # Required if the TSC is unusable options ALTQ_DEBUG +# IP optional behaviour. +# IP_NONLOCALBIND disables the check that bind() usually makes that the +# Address is one that is assigned to an interface on this machine. +# It allows transparent proxies to pretend to be other machines. +# How the packet GET to that machine is a problem solved elsewhere, +# smart routers, ipfw fwd, etc. +options IP_NONLOCALBIND #Allow impersonation for proxies. + # netgraph(4). Enable the base netgraph code with the NETGRAPH option. # Individual node types can be enabled with the corresponding option # listed below; however, this is not strictly necessary as netgraph Modified: user/adrian/spoof_bind/sys/conf/options ============================================================================== --- user/adrian/spoof_bind/sys/conf/options Tue Nov 18 22:11:33 2008 (r185066) +++ user/adrian/spoof_bind/sys/conf/options Tue Nov 18 22:17:17 2008 (r185067) @@ -392,6 +392,7 @@ IPFIREWALL_VERBOSE opt_ipfw.h IPFIREWALL_VERBOSE_LIMIT opt_ipfw.h IPSEC opt_ipsec.h IPSEC_DEBUG opt_ipsec.h +IP_NONLOCALBIND opt_inet.h IPSEC_FILTERTUNNEL opt_ipsec.h IPSTEALTH IPX Modified: user/adrian/spoof_bind/sys/net/if_bridge.c ============================================================================== --- user/adrian/spoof_bind/sys/net/if_bridge.c Tue Nov 18 22:11:33 2008 (r185066) +++ user/adrian/spoof_bind/sys/net/if_bridge.c Tue Nov 18 22:17:17 2008 (r185067) @@ -2938,6 +2938,10 @@ bridge_pfil(struct mbuf **mp, struct ifn struct ip *ip; struct llc llc1; u_int16_t ether_type; + int is_ip = 0; +#ifdef IPFIREWALL_FORWARD + struct m_tag *fwd_tag; +#endif snap = 0; error = -1; /* Default error if not error == 0 */ @@ -2997,6 +3001,7 @@ bridge_pfil(struct mbuf **mp, struct ifn #ifdef INET6 case ETHERTYPE_IPV6: #endif /* INET6 */ + is_ip = 1; break; default: /* @@ -3057,6 +3062,25 @@ bridge_pfil(struct mbuf **mp, struct ifn if (*mp == NULL) return (error); +#ifdef IPFIREWALL_FORWARD + /* + * Did the firewall want to forward it somewhere? + * If so, let the ip stack handle it. + */ + if (i == 0 && args.next_hop != NULL && is_ip /* && src != NULL */) { + fwd_tag = m_tag_get(PACKET_TAG_IPFORWARD, sizeof(struct sockaddr_in), + M_NOWAIT); + if (fwd_tag == NULL) + goto drop; + bcopy(args.next_hop, (fwd_tag+1), sizeof(struct sockaddr_in)); + m_tag_prepend(*mp, fwd_tag); + if (in_localip(args.next_hop->sin_addr)) + (*mp)->m_flags |= M_FASTFWD_OURS; + ether_demux(src, *mp); + return (NULL); + } +#endif + if (DUMMYNET_LOADED && (i == IP_FW_DUMMYNET)) { /* put the Ethernet header back on */ Modified: user/adrian/spoof_bind/sys/netinet/in.h ============================================================================== --- user/adrian/spoof_bind/sys/netinet/in.h Tue Nov 18 22:11:33 2008 (r185066) +++ user/adrian/spoof_bind/sys/netinet/in.h Tue Nov 18 22:17:17 2008 (r185067) @@ -441,6 +441,7 @@ __END_DECLS #define IP_FAITH 22 /* bool; accept FAITH'ed connections */ #define IP_ONESBCAST 23 /* bool: send all-ones broadcast */ +#define IP_NONLOCALOK 24 /* allow bind to spoof other machines */ #define IP_FW_TABLE_ADD 40 /* add entry */ #define IP_FW_TABLE_DEL 41 /* delete entry */ Modified: user/adrian/spoof_bind/sys/netinet/in_pcb.c ============================================================================== --- user/adrian/spoof_bind/sys/netinet/in_pcb.c Tue Nov 18 22:11:33 2008 (r185066) +++ user/adrian/spoof_bind/sys/netinet/in_pcb.c Tue Nov 18 22:17:17 2008 (r185067) @@ -35,6 +35,7 @@ __FBSDID("$FreeBSD$"); #include "opt_ddb.h" +#include "opt_inet.h" #include "opt_ipsec.h" #include "opt_inet6.h" #include "opt_mac.h" @@ -343,7 +344,11 @@ in_pcbbind_setup(struct inpcb *inp, stru } else if (sin->sin_addr.s_addr != INADDR_ANY) { sin->sin_port = 0; /* yech... */ bzero(&sin->sin_zero, sizeof(sin->sin_zero)); - if (ifa_ifwithaddr((struct sockaddr *)sin) == 0) + if ( +#if defined(IP_NONLOCALBIND) + ((inp->inp_flags & INP_NONLOCALOK) == 0) && +#endif + (ifa_ifwithaddr((struct sockaddr *)sin) == 0)) return (EADDRNOTAVAIL); } laddr = sin->sin_addr; Modified: user/adrian/spoof_bind/sys/netinet/in_pcb.h ============================================================================== --- user/adrian/spoof_bind/sys/netinet/in_pcb.h Tue Nov 18 22:11:33 2008 (r185066) +++ user/adrian/spoof_bind/sys/netinet/in_pcb.h Tue Nov 18 22:17:17 2008 (r185067) @@ -396,6 +396,8 @@ void inp_4tuple_get(struct inpcb *inp, #define INP_FAITH 0x200 /* accept FAITH'ed connections */ #define INP_RECVTTL 0x400 /* receive incoming IP TTL */ #define INP_DONTFRAG 0x800 /* don't fragment packet */ +#define INP_NONLOCALOK 0x1000 /* Allow bind to spoof any address */ + /* - requires options IP_NONLOCALBIND */ #define IN6P_IPV6_V6ONLY 0x008000 /* restrict AF_INET6 socket for v6 */ Modified: user/adrian/spoof_bind/sys/netinet/ip_fw.h ============================================================================== --- user/adrian/spoof_bind/sys/netinet/ip_fw.h Tue Nov 18 22:11:33 2008 (r185066) +++ user/adrian/spoof_bind/sys/netinet/ip_fw.h Tue Nov 18 22:17:17 2008 (r185067) @@ -174,6 +174,7 @@ enum ipfw_opcodes { /* arguments (4 byt O_TAG, /* arg1=tag number */ O_TAGGED, /* arg1=tag number */ + O_FOR_ME, /* check for a PCB here for this packet */ O_SETFIB, /* arg1=FIB number */ O_FIB, /* arg1=FIB desired fib number */ Modified: user/adrian/spoof_bind/sys/netinet/ip_fw2.c ============================================================================== --- user/adrian/spoof_bind/sys/netinet/ip_fw2.c Tue Nov 18 22:11:33 2008 (r185066) +++ user/adrian/spoof_bind/sys/netinet/ip_fw2.c Tue Nov 18 22:17:17 2008 (r185067) @@ -2068,6 +2068,46 @@ check_uidgid(ipfw_insn_u32 *insn, int pr } /* + * This function looks to see if the packet (regardless of direction) + * is destined for a socket on this machine. This is regardless of + * whether the address of the socket is a legal address of this + * machine or not. This is used for transparent proxying where + * the proxy masquerades as both the server and the client. + * It is not really needed if the IP_NONLOCALBIND is not dependent + * on that. It is not quite the same as 'from any to me'. + * in that it checks ports too and doesn't check if the address + * is 'legally owned by this machine. Do NOT insist there is still + * a socket.. It could be timing out. It IS STILL OURS. + * XXX check whether we should only match if there is a + * socket on the pcb. + */ +static int +packet_for_me(ipfw_insn_u32 *insn, + int proto, + struct in_addr dst_ip, u_int16_t dst_port, + struct in_addr src_ip, u_int16_t src_port ) +{ + struct inpcbinfo *pi; + int wildcard; + struct inpcb *pcb; + + if (proto == IPPROTO_TCP) { + wildcard = 0; + pi = &tcbinfo; + } else if (proto == IPPROTO_UDP) { + wildcard = INPLOOKUP_WILDCARD; + pi = &udbinfo; + } else { + return (0); + } + INP_INFO_RLOCK(pi); + pcb = in_pcblookup_hash(pi, src_ip, htons(src_port), + dst_ip, htons(dst_port), wildcard, NULL); + INP_INFO_RUNLOCK(pi); + return (pcb?1:0); +} + +/* * The main check routine for the firewall. * * All arguments are in args so we can modify them and return them @@ -2090,6 +2130,7 @@ check_uidgid(ipfw_insn_u32 *insn, int pr * args->next_hop Socket we are forwarding to (out). * args->f_id Addresses grabbed from the packet (out) * args->cookie a cookie depending on rule action + * args->inp A pcb associated with a packet if known * * Return value: * @@ -2604,6 +2645,24 @@ check_body: &ugid_lookup, args->inp); break; + case O_FOR_ME: + /* + * We only check offset == 0 && TCP or UDP + * as this ensures that we have a + * packet with the ports info. + */ + if (offset!=0) + break; + if (is_ipv6) /* XXX to be fixed later */ + break; + if (proto == IPPROTO_TCP || proto == IPPROTO_UDP) + match = packet_for_me( + (ipfw_insn_u32 *)cmd, + proto, + dst_ip, dst_port, + src_ip, src_port ); + break; + case O_RECV: match = iface_match(m->m_pkthdr.rcvif, (ipfw_insn_if *)cmd); @@ -3294,8 +3353,11 @@ check_body: case O_FORWARD_IP: { struct sockaddr_in *sa; sa = &(((ipfw_insn_sa *)cmd)->sa); +/* XXX julian's patch disabled this; "why" needs to be thought about. -adrian */ +#if 0 if (args->eh) /* not valid on layer2 pkts */ break; +#endif if (!q || dyn_dir == MATCH_FORWARD) { if (sa->sin_addr.s_addr == INADDR_ANY) { bcopy(sa, &args->hopstore, @@ -3814,6 +3876,7 @@ check_ipfw_struct(struct ip_fw *rule, in case O_PROBE_STATE: case O_KEEP_STATE: case O_PROTO: + case O_FOR_ME: case O_IP_SRC_ME: case O_IP_DST_ME: case O_LAYER2: Modified: user/adrian/spoof_bind/sys/netinet/ip_output.c ============================================================================== --- user/adrian/spoof_bind/sys/netinet/ip_output.c Tue Nov 18 22:11:33 2008 (r185066) +++ user/adrian/spoof_bind/sys/netinet/ip_output.c Tue Nov 18 22:17:17 2008 (r185067) @@ -33,6 +33,7 @@ __FBSDID("$FreeBSD$"); #include "opt_ipfw.h" +#include "opt_inet.h" #include "opt_ipsec.h" #include "opt_mac.h" #include "opt_mbuf_stress_test.h" @@ -91,6 +92,12 @@ SYSCTL_INT(_net_inet_ip, OID_AUTO, mbuf_ &mbuf_frag_size, 0, "Fragment outgoing mbufs to this size"); #endif +#if defined(IP_NONLOCALBIND) +static int ip_nonlocalok = 0; +SYSCTL_INT(_net_inet_ip, OID_AUTO, nonlocalok, + CTLFLAG_RW|CTLFLAG_SECURE, &ip_nonlocalok, 0, ""); +#endif + static void ip_mloopback (struct ifnet *, struct mbuf *, struct sockaddr_in *, int); @@ -856,6 +863,13 @@ ip_ctloutput(struct socket *so, struct s return (error); } +#if defined(IP_NONLOCALBIND) + case IP_NONLOCALOK: + if (! ip_nonlocalok) { + error = ENOPROTOOPT; + break; + } +#endif case IP_TOS: case IP_TTL: case IP_MINTTL: @@ -927,6 +941,11 @@ ip_ctloutput(struct socket *so, struct s case IP_DONTFRAG: OPTSET(INP_DONTFRAG); break; +#if defined(IP_NONLOCALBIND) + case IP_NONLOCALOK: + OPTSET(INP_NONLOCALOK); + break; +#endif } break; #undef OPTSET From owner-svn-src-user@FreeBSD.ORG Wed Nov 19 01:35:05 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3FEF01065672; Wed, 19 Nov 2008 01:35:05 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 277A28FC0A; Wed, 19 Nov 2008 01:35:05 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAJ1Z4O4099898; Wed, 19 Nov 2008 01:35:04 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAJ1Z47o099891; Wed, 19 Nov 2008 01:35:04 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811190135.mAJ1Z47o099891@svn.freebsd.org> From: Lawrence Stewart Date: Wed, 19 Nov 2008 01:35:04 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185076 - in user/lstewart/dummynet_7.x: sbin/ipfw sys sys/netinet X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Nov 2008 01:35:05 -0000 Author: lstewart Date: Wed Nov 19 01:35:04 2008 New Revision: 185076 URL: http://svn.freebsd.org/changeset/base/185076 Log: Merge r184953 (DPD patch) from dummynet_8.x Modified: user/lstewart/dummynet_7.x/sbin/ipfw/ (props changed) user/lstewart/dummynet_7.x/sbin/ipfw/Makefile user/lstewart/dummynet_7.x/sbin/ipfw/ipfw.8 user/lstewart/dummynet_7.x/sbin/ipfw/ipfw2.c user/lstewart/dummynet_7.x/sys/ (props changed) user/lstewart/dummynet_7.x/sys/netinet/in.h user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.c user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.h user/lstewart/dummynet_7.x/sys/netinet/raw_ip.c Modified: user/lstewart/dummynet_7.x/sbin/ipfw/Makefile ============================================================================== --- user/lstewart/dummynet_7.x/sbin/ipfw/Makefile Wed Nov 19 00:25:15 2008 (r185075) +++ user/lstewart/dummynet_7.x/sbin/ipfw/Makefile Wed Nov 19 01:35:04 2008 (r185076) @@ -5,4 +5,6 @@ SRCS= ipfw2.c WARNS?= 0 MAN= ipfw.8 +CFLAGS+= -I../../sys + .include Modified: user/lstewart/dummynet_7.x/sbin/ipfw/ipfw.8 ============================================================================== --- user/lstewart/dummynet_7.x/sbin/ipfw/ipfw.8 Wed Nov 19 00:25:15 2008 (r185075) +++ user/lstewart/dummynet_7.x/sbin/ipfw/ipfw.8 Wed Nov 19 01:35:04 2008 (r185076) @@ -61,7 +61,7 @@ .Nm .Op Fl s Op Ar field .Brq Cm pipe | queue -.Brq Cm delete | list | show +.Brq Cm delete | zero | list | show .Op Ar number ... .Pp .Nm @@ -1962,6 +1962,20 @@ is a floating-point number between 0 and loss, 1 meaning 100% loss. The loss rate is internally represented on 31 bits. .Pp +.It Cm pls Ar packet-loss-set +Packet loss set. +Argument +.Ar packet-loss-set +is a comma-delimited string of the form 10,30-31,1000 identifying the specific +packets entering a queue/pipe to drop. In the given example, the 10th, 30th, +31st and 1000th packet to enter the pipe/queue would be dropped. Clearing the +counters on a pipe will cause the +.Ar packet-loss-set +to be evaluated again from scratch. Use of this option mutually excludes use of +the +.Nm plr +option. +.Pp .It Cm queue Brq Ar slots | size Ns Cm Kbytes Queue size, in .Ar slots Modified: user/lstewart/dummynet_7.x/sbin/ipfw/ipfw2.c ============================================================================== --- user/lstewart/dummynet_7.x/sbin/ipfw/ipfw2.c Wed Nov 19 00:25:15 2008 (r185075) +++ user/lstewart/dummynet_7.x/sbin/ipfw/ipfw2.c Wed Nov 19 01:35:04 2008 (r185076) @@ -83,6 +83,9 @@ int comment_only, /* only print action and comment */ verbose; +#define CHARPTR_IS_INT(X) \ + (((unsigned int)(((unsigned char)(*(X)))) - (unsigned char)'0') < 11) + #define IP_MASK_ALL 0xffffffff /* * the following macro returns an error message if we run out of @@ -128,6 +131,10 @@ int printf("%u", (uint32_t)arg); \ } while (0) +#define flush_range_array(PLS) \ + if((PLS)->arr != NULL) \ + free((PLS)->arr); + /* * _s_x is a structure that stores a string <-> token pairs, used in * various places in the parser. Entries are stored in arrays, @@ -303,6 +310,7 @@ enum tokens { TOK_COMMENT, TOK_PLR, + TOK_PLS, TOK_NOERROR, TOK_BUCKETS, TOK_DSTIP, @@ -348,6 +356,7 @@ enum tokens { struct _s_x dummynet_params[] = { { "plr", TOK_PLR }, + { "pls", TOK_PLS }, { "noerror", TOK_NOERROR }, { "buckets", TOK_BUCKETS }, { "dst-ip", TOK_DSTIP }, @@ -626,6 +635,151 @@ _substrcmp2(const char *str1, const char } /* + * Generates a string of the form "1,2,3,6-10,1000,1500-2000" from a specified + * range list. If the range list is too large to fit into the provided string + * buffer,"..." will be appended to the string to indicate there was more + * information that could not be displayed. + */ +static void +gen_pls_str(char *str, u_int len, struct pls_range_array *ranges) +{ + struct pls_range *node; + u_int i = 0, stroffset = 0, ret = 0, str_truncated = 0; + + for(; i < ranges->count; i++) { + + if(ranges->arr[i].start == ranges->arr[i].end) + ret = snprintf(str + stroffset, len - stroffset, "%d,", ranges->arr[i].start); + else + ret = snprintf(str + stroffset, len - stroffset, "%d-%d,", ranges->arr[i].start, ranges->arr[i].end); + + if(ret >= (len - stroffset)) { + str_truncated = 1; + break; + } + + stroffset = strlen(str); + } + + if(str_truncated) { + *(str+len-3) = '.'; + *(str+len-2) = '.'; + *(str+len-1) = '.'; + *(str+len) = '\0'; + } + else + /* remove the trailing comma from the list */ + *(str+stroffset-1) = '\0'; +} + +/* + * This function parses a string of the form "1,2,6-10,3,1000,2000-1500" and turns it + * into an array of pls_range structs. The array is sorted in order of range start + * value from lowest to highest, and the range start and end values within a range struct + * are ordered from lowest to highest for consistency. In the example above, + * the range node list extracted from the parsed string would look like this: + * + * pls_range1.start = 1, pls_range1.end = 1 + * pls_range2.start = 2, pls_range2.end = 2 + * pls_range3.start = 3, pls_range3.end = 3 + * pls_range4.start = 6, pls_range4.end = 10 + * pls_range5.start = 1000, pls_range5.end = 1000 + * pls_range6.start = 1500, pls_range6.end = 2000 + */ +static int +parse_pls_str(char *str, struct pls_range_array *ranges) +{ + char *ch, *tmpstr; + char tmp[16]; + int i = 0, j = 0, comma_count = 0, arr_index = 0; + struct pls_range *current_range = NULL; + + for(i = strlen(str), tmpstr = str; i >= 0; i--) { + ch = tmpstr++; + if(*ch == ',') + comma_count++; + } + + ranges->size = comma_count + 1; + ranges->arr = (struct pls_range *)malloc(ranges->size * sizeof(struct pls_range)); + bzero(ranges->arr, ranges->size * sizeof(struct pls_range)); + + if(ranges->arr == NULL) + return (ENOMEM); + + for(i = strlen(str), tmpstr = str; i >= 0; i--) { + ch = tmpstr++; + memset(tmp, '\0', sizeof(tmp)); + + j = 0; + + /* if the character is a number 0-9 */ + while(CHARPTR_IS_INT(ch)) { + /* read until the number ends */ + tmp[j++] = *ch; + ch = tmpstr++; + i--; + } + + /* + * if we are at the end of a range, or the end of the string and + * we have a number stored in tmp + */ + if((*ch == ',' && j > 0) || (i == 0 && j > 0)) { + if(ranges->arr[arr_index].start == 0) { + ranges->arr[arr_index].start = strtol(tmp, NULL, 10); + } + + ranges->arr[arr_index].end = strtol(tmp, NULL, 10); + + /* + * reorder the start and end of the range if they were + * specified out of order + */ + if(ranges->arr[arr_index].end < ranges->arr[arr_index].start) { + u_int tmp = ranges->arr[arr_index].end; + ranges->arr[arr_index].end = ranges->arr[arr_index].start; + ranges->arr[arr_index].start = tmp; + } + + arr_index++; + } + else if(*ch == '-' && j > 0) { + /* + * we are half way through parsing a range, so let's set + * create a new range and set its start value to the + * first number we parsed in the range + */ + ranges->arr[arr_index].start = strtol(tmp, NULL, 10); + } + else { + flush_range_array(ranges); + return 1; /* failed parsing the string */ + } + } + + ranges->count = arr_index; + + struct pls_range tmp_range; + + /* + * bubble sort of the list to put them in numerical order of range start + * values + */ + for(i = 0; i < ranges->count; i++) { + for(j = 1; j < ranges->count; j++) { + if(ranges->arr[j-1].start > ranges->arr[j].start) { + tmp_range = ranges->arr[j-1]; + ranges->arr[j-1] = ranges->arr[j]; + ranges->arr[j] = tmp_range; + } + } + } + + return 0; +} + +/* * prints one port, symbolic or numeric */ static void @@ -2254,7 +2408,7 @@ print_flowset_parms(struct dn_flow_set * { int l; char qs[30]; - char plr[30]; + char pl[30]; char red[90]; /* Display RED parameters */ l = fs->qsize; @@ -2266,9 +2420,13 @@ print_flowset_parms(struct dn_flow_set * } else sprintf(qs, "%3d sl.", l); if (fs->plr) - sprintf(plr, "plr %f", 1.0 * fs->plr / (double)(0x7fffffff)); + sprintf(pl, "plr %f", 1.0 * fs->plr / (double)(0x7fffffff)); + else if (fs->pls.count > 0) { + sprintf(pl, "pls "); + gen_pls_str(pl+strlen(pl), sizeof(pl)-strlen(pl), &(fs->pls)); + } else - plr[0] = '\0'; + pl[0] = '\0'; if (fs->flags_fs & DN_IS_RED) /* RED parameters */ sprintf(red, "\n\t %cRED w_q %f min_th %d max_th %d max_p %f", @@ -2281,7 +2439,7 @@ print_flowset_parms(struct dn_flow_set * sprintf(red, "droptail"); printf("%s %s%s %d queues (%d buckets) %s\n", - prefix, qs, plr, fs->rq_elements, fs->rq_size, red); + prefix, qs, pl, fs->rq_elements, fs->rq_size, red); } static void @@ -2309,7 +2467,7 @@ list_pipes(void *data, uint nbytes, int /* * compute length, as pipe have variable size */ - l = sizeof(*p) + p->fs.rq_elements * sizeof(*q); + l = sizeof(*p) + p->fs.rq_elements * sizeof(*q) + p->fs.pls.count * sizeof(struct pls_range); next = (char *)p + l; nbytes -= l; @@ -2332,11 +2490,21 @@ list_pipes(void *data, uint nbytes, int sprintf(prefix, "%05d: %s %4d ms ", p->pipe_nr, buf, p->delay); + + /* + * the pls array data is tucked in between the dn_pipe struct + * and fs queue data because the pointer isn't recalculated + * during the copy from kernel mem into userspace mem, we have + * to manually set the array pointer we work back from the end + * of the pipe's data boundary (calculated above as "next") + */ + p->fs.pls.arr = next - (p->fs.pls.count * sizeof(struct pls_range)) - (p->fs.rq_elements * sizeof(*q)); + print_flowset_parms(&(p->fs), prefix); if (verbose) printf(" V %20qd\n", p->V >> MY_M); - q = (struct dn_flow_queue *)(p+1); + q = (struct dn_flow_queue *)(next - p->fs.rq_elements * sizeof(*q)); list_queues(&(p->fs), q); } for (fs = next; nbytes >= sizeof *fs; fs = next) { @@ -4113,11 +4281,21 @@ config_pipe(int ac, char **av) ac--; av++; break; + case TOK_PLS: + NEED1("pls needs argument x,y-z\n"); + if(parse_pls_str(av[0], &(p.fs.pls))) + errx(EX_DATAERR, "invalid packet loss set"); + ac--; av++; + break; + case TOK_QUEUE: NEED1("queue needs queue size\n"); end = NULL; p.fs.qsize = strtoul(av[0], &end, 0); - if (*end == 'K' || *end == 'k') { + if (*end == 'M' || *end == 'm') { + p.fs.flags_fs |= DN_QSIZE_IS_BYTES; + p.fs.qsize *= 1048576; + } else if (*end == 'K' || *end == 'k') { p.fs.flags_fs |= DN_QSIZE_IS_BYTES; p.fs.qsize *= 1024; } else if (*end == 'B' || @@ -4374,6 +4552,9 @@ end_mask: if (p.fs.qsize > limit) errx(EX_DATAERR, "2 <= queue size <= %ld", limit); } + if (p.fs.pls.count > 0 && p.fs.plr > 0) { + errx(EX_DATAERR, "plr and pls options are mutually exclusive"); + } if (p.fs.flags_fs & DN_IS_RED) { size_t len; int lookup_depth, avg_pkt_size; @@ -5763,21 +5944,36 @@ done: } static void -zero(int ac, char *av[], int optname /* IP_FW_ZERO or IP_FW_RESETLOG */) +zero(int ac, char *av[], int optname /* IP_FW_ZERO or IP_FW_RESETLOG or IP_DUMMYNET_ZERO */) { uint32_t arg, saved_arg; int failed = EX_OK; - char const *name = optname == IP_FW_ZERO ? "ZERO" : "RESETLOG"; + char const *name; char const *errstr; + struct dn_pipe p; + + if(optname == IP_FW_ZERO || optname == IP_DUMMYNET_ZERO) + name = "ZERO"; + else + name = "RESETLOG"; av++; ac--; + memset(&p, 0, sizeof p); + if (!ac) { /* clear all entries */ - if (do_cmd(optname, NULL, 0) < 0) - err(EX_UNAVAILABLE, "setsockopt(IP_FW_%s)", name); + if (do_pipe) + failed = do_cmd(optname, &p, sizeof p); + else + failed = do_cmd(optname, NULL, 0); + + if (failed < 0) + err(EX_UNAVAILABLE, "setsockopt(IP_%s_%s)", + do_pipe ? "DUMMYNET" : "FW", name); + if (!do_quiet) - printf("%s.\n", optname == IP_FW_ZERO ? + printf("%s.\n", (optname == IP_FW_ZERO || optname == IP_DUMMYNET_ZERO) ? "Accounting cleared":"Logging counts reset"); return; @@ -5795,13 +5991,14 @@ zero(int ac, char *av[], int optname /* arg |= (1 << 24) | ((use_set - 1) << 16); av++; ac--; - if (do_cmd(optname, &arg, sizeof(arg))) { - warn("rule %u: setsockopt(IP_FW_%s)", - saved_arg, name); + p.pipe_nr = arg; + if (do_pipe ? do_cmd(optname, &p, sizeof p) : do_cmd(optname, &arg, sizeof(arg))) { + warn("%s %u: setsockopt(IP_%s_%s)", do_pipe ? "pipe" : "rule", + saved_arg, do_pipe ? "DUMMYNET" : "FW", name); failed = EX_UNAVAILABLE; } else if (!do_quiet) printf("Entry %d %s.\n", saved_arg, - optname == IP_FW_ZERO ? + (optname == IP_FW_ZERO || optname == IP_DUMMYNET_ZERO) ? "cleared" : "logging count reset"); } else { errx(EX_USAGE, "invalid rule number ``%s''", *av); @@ -6294,6 +6491,8 @@ ipfw_main(int oldac, char **oldav) delete(ac, av); else if (_substrcmp(*av, "flush") == 0) flush(do_force); + else if (do_pipe && _substrcmp(*av, "zero") == 0) + zero(ac, av, IP_DUMMYNET_ZERO); else if (_substrcmp(*av, "zero") == 0) zero(ac, av, IP_FW_ZERO); else if (_substrcmp(*av, "resetlog") == 0) Modified: user/lstewart/dummynet_7.x/sys/netinet/in.h ============================================================================== --- user/lstewart/dummynet_7.x/sys/netinet/in.h Wed Nov 19 00:25:15 2008 (r185075) +++ user/lstewart/dummynet_7.x/sys/netinet/in.h Wed Nov 19 01:35:04 2008 (r185076) @@ -460,10 +460,11 @@ __END_DECLS #define IP_DUMMYNET_DEL 61 /* delete a dummynet pipe from chain */ #define IP_DUMMYNET_FLUSH 62 /* flush dummynet */ #define IP_DUMMYNET_GET 64 /* get entire dummynet pipes */ +#define IP_DUMMYNET_ZERO 65 /* clear single/all dummynet counter(s) */ -#define IP_RECVTTL 65 /* bool; receive IP TTL w/dgram */ #define IP_MINTTL 66 /* minimum TTL for packet or drop */ #define IP_DONTFRAG 67 /* don't fragment packet */ +#define IP_RECVTTL 68 /* bool; receive IP TTL w/dgram */ /* IPv4 Source Filter Multicast API [RFC3678] */ #define IP_ADD_SOURCE_MEMBERSHIP 70 /* join a source-specific group */ Modified: user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.c ============================================================================== --- user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.c Wed Nov 19 00:25:15 2008 (r185075) +++ user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.c Wed Nov 19 01:35:04 2008 (r185076) @@ -241,6 +241,10 @@ void dummynet_drain(void); static ip_dn_io_t dummynet_io; static void dn_rule_delete(void *); +#define flush_range_array(PLS) \ + if((PLS)->arr != NULL) \ + free((PLS)->arr, M_DUMMYNET); + /* * Heap management functions. * @@ -1313,6 +1317,23 @@ dummynet_io(struct mbuf **m0, int dir, s q->tot_pkts++; if (fs->plr && random() < fs->plr) goto dropit; /* Random pkt drop. */ + else { + while (fs->pls_index < fs->pls.count) { + if (q->tot_pkts >= fs->pls.arr[fs->pls_index].start) { + if (q->tot_pkts <= + fs->pls.arr[fs->pls_index].end) + goto dropit; /* Controlled pkt drop. */ + else + fs->pls_index++; + } + else + /* + * q->tot_pkts is lower than the start of the + * current range, so let the packet through + */ + break; + } + } if (fs->flags_fs & DN_QSIZE_IS_BYTES) { if (q->len_bytes > fs->qsize) goto dropit; /* Queue size overflow. */ @@ -1492,12 +1513,31 @@ purge_flow_set(struct dn_flow_set *fs, i free(fs->w_q_lookup, M_DUMMYNET); if (fs->rq != NULL) free(fs->rq, M_DUMMYNET); + flush_range_array(&(fs->pls)); /* If this fs is not part of a pipe, free it. */ if (fs->pipe == NULL || fs != &(fs->pipe->fs)) free(fs, M_DUMMYNET); } } +static void +zero_flow_set(struct dn_flow_set *fs) +{ + int i; + + DUMMYNET_LOCK_ASSERT(); + + fs->pls_index = 0; + + for (i = 0; i <= fs->rq_size; i++) { + if (fs->rq[i] != NULL) { + fs->rq[i]->tot_pkts = 0; + fs->rq[i]->tot_bytes = 0; + fs->rq[i]->drops = 0; + } + } +} + /* * Dispose all packets queued on a pipe (not a flow_set). * Also free all resources associated to a pipe, which is about @@ -1693,12 +1733,31 @@ alloc_hash(struct dn_flow_set *x, struct return 0 ; } +static int +set_pls(struct pls_range_array *x, struct pls_range_array *src) +{ + flush_range_array(x); + + MALLOC(x->arr, struct pls_range *, src->count * sizeof(struct pls_range), M_DUMMYNET, M_NOWAIT | M_ZERO); + + if(x->arr == NULL) + return (ENOMEM); + + x->count = x->size = src->count; + copyin(src->arr, x->arr, src->count * sizeof(struct pls_range)); + + return 0; +} + + static void set_fs_parms(struct dn_flow_set *x, struct dn_flow_set *src) { x->flags_fs = src->flags_fs; x->qsize = src->qsize; x->plr = src->plr; + x->pls_index = 0; + set_pls(&(x->pls), &(src->pls)); x->flow_mask = src->flow_mask; if (x->flags_fs & DN_QSIZE_IS_BYTES) { if (x->qsize > pipe_byte_limit) @@ -1985,16 +2044,63 @@ delete_pipe(struct dn_pipe *p) } /* + * Zeroes counters of a specified pipe/queue or all pipes/queues. + */ +static int +zero_pipe(struct dn_pipe *p) +{ + struct dn_pipe *pipe; + struct dn_flow_set *fs; + int i; + + DUMMYNET_LOCK(); + + if (p->pipe_nr > 0) { + pipe = locate_pipe(p->pipe_nr); /* locate pipe */ + if (pipe == NULL) { + DUMMYNET_UNLOCK(); + return (ENOENT); /* not found */ + } + zero_flow_set(&pipe->fs); + } else { + /* zero all pipes */ + for (i = 0; i < HASHSIZE; i++) { + SLIST_FOREACH(pipe, &pipehash[i], next) { + zero_flow_set(&pipe->fs); + } + SLIST_FOREACH(fs, &flowsethash[i], next) { + zero_flow_set(fs); + } + } + } + + DUMMYNET_UNLOCK(); + + return 0 ; +} + +static char * +dn_copy_pls(struct pls_range_array *src_pls, char *bp) +{ + bcopy(src_pls->arr, bp, src_pls->count * sizeof(struct pls_range)); + + return (bp + src_pls->count * sizeof(struct pls_range)); +} + +/* * helper function used to copy data from kernel in DUMMYNET_GET */ static char * dn_copy_set(struct dn_flow_set *set, char *bp) { int i, copied = 0 ; - struct dn_flow_queue *q, *qp = (struct dn_flow_queue *)bp; DUMMYNET_LOCK_ASSERT(); + bp = dn_copy_pls(&(set->pls), bp); + + struct dn_flow_queue *q, *qp = (struct dn_flow_queue *)bp; + for (i = 0 ; i <= set->rq_size ; i++) for (q = set->rq[i] ; q ; q = q->next, qp++ ) { if (q->hash_slot != i) @@ -2030,10 +2136,12 @@ dn_calc_size(void) */ for (i = 0; i < HASHSIZE; i++) { SLIST_FOREACH(pipe, &pipehash[i], next) - size += sizeof(*pipe) + + size += sizeof(*pipe) + pipe->fs.pls.count * sizeof(struct + pls_range) + pipe->fs.rq_elements * sizeof(struct dn_flow_queue); SLIST_FOREACH(fs, &flowsethash[i], next) - size += sizeof (*fs) + + size += sizeof (*fs) + pipe->fs.pls.count * sizeof(struct + pls_range) + fs->rq_elements * sizeof(struct dn_flow_queue); } return size; @@ -2171,6 +2279,15 @@ ip_dn_ctl(struct sockopt *sopt) error = delete_pipe(p); break ; + + case IP_DUMMYNET_ZERO : + p = &tmp_pipe ; + error = sooptcopyin(sopt, p, sizeof *p, sizeof *p); + if (error) + break ; + + error = zero_pipe(p); + break ; } return error ; } Modified: user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.h ============================================================================== --- user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.h Wed Nov 19 00:25:15 2008 (r185075) +++ user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.h Wed Nov 19 01:35:04 2008 (r185076) @@ -230,6 +230,17 @@ struct dn_flow_queue { */ } ; +struct pls_range { + unsigned int start; + unsigned int end; +}; + +struct pls_range_array { + unsigned int size; + unsigned int count; + struct pls_range *arr; +}; + /* * flow_set descriptor. Contains the "template" parameters for the * queue configuration, and pointers to the hash table of dn_flow_queue's. @@ -261,6 +272,8 @@ struct dn_flow_set { int weight ; /* WFQ queue weight */ int qsize ; /* queue size in slots or bytes */ int plr ; /* pkt loss rate (2^31-1 means 100%) */ + int pls_index ; /* index into pls array */ + struct pls_range_array pls; /* pkt loss set as an array of pls_range structs */ struct ipfw_flow_id flow_mask ; Modified: user/lstewart/dummynet_7.x/sys/netinet/raw_ip.c ============================================================================== --- user/lstewart/dummynet_7.x/sys/netinet/raw_ip.c Wed Nov 19 00:25:15 2008 (r185075) +++ user/lstewart/dummynet_7.x/sys/netinet/raw_ip.c Wed Nov 19 01:35:04 2008 (r185076) @@ -529,6 +529,7 @@ rip_ctloutput(struct socket *so, struct case IP_DUMMYNET_CONFIGURE: case IP_DUMMYNET_DEL: case IP_DUMMYNET_FLUSH: + case IP_DUMMYNET_ZERO: if (ip_dn_ctl_ptr != NULL) error = ip_dn_ctl_ptr(sopt); else From owner-svn-src-user@FreeBSD.ORG Wed Nov 19 01:47:47 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ABF2D106564A; Wed, 19 Nov 2008 01:47:47 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 95A518FC0C; Wed, 19 Nov 2008 01:47:47 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAJ1llQu000243; Wed, 19 Nov 2008 01:47:47 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAJ1llEf000242; Wed, 19 Nov 2008 01:47:47 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811190147.mAJ1llEf000242@svn.freebsd.org> From: Lawrence Stewart Date: Wed, 19 Nov 2008 01:47:47 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185077 - user/lstewart/dummynet_7.x/sbin/ipfw X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Nov 2008 01:47:47 -0000 Author: lstewart Date: Wed Nov 19 01:47:47 2008 New Revision: 185077 URL: http://svn.freebsd.org/changeset/base/185077 Log: Remove a testing change that snuck into the previous commit Modified: user/lstewart/dummynet_7.x/sbin/ipfw/Makefile Modified: user/lstewart/dummynet_7.x/sbin/ipfw/Makefile ============================================================================== --- user/lstewart/dummynet_7.x/sbin/ipfw/Makefile Wed Nov 19 01:35:04 2008 (r185076) +++ user/lstewart/dummynet_7.x/sbin/ipfw/Makefile Wed Nov 19 01:47:47 2008 (r185077) @@ -5,6 +5,4 @@ SRCS= ipfw2.c WARNS?= 0 MAN= ipfw.8 -CFLAGS+= -I../../sys - .include From owner-svn-src-user@FreeBSD.ORG Wed Nov 19 01:52:28 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E4815106564A; Wed, 19 Nov 2008 01:52:28 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id B51838FC08; Wed, 19 Nov 2008 01:52:28 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAJ1qSg0000361; Wed, 19 Nov 2008 01:52:28 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAJ1qSpt000360; Wed, 19 Nov 2008 01:52:28 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811190152.mAJ1qSpt000360@svn.freebsd.org> From: Lawrence Stewart Date: Wed, 19 Nov 2008 01:52:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185078 - user/lstewart/alq_varlen_7.x X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Nov 2008 01:52:29 -0000 Author: lstewart Date: Wed Nov 19 01:52:28 2008 New Revision: 185078 URL: http://svn.freebsd.org/changeset/base/185078 Log: Somewhere to maintain the backport of my alq changes to 7-STABLE. Added: user/lstewart/alq_varlen_7.x/ - copied from r185077, stable/7/ From owner-svn-src-user@FreeBSD.ORG Wed Nov 19 02:55:26 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7D7F91065672; Wed, 19 Nov 2008 02:55:26 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 647068FC0C; Wed, 19 Nov 2008 02:55:26 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAJ2tQTY001694; Wed, 19 Nov 2008 02:55:26 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAJ2tPjq001690; Wed, 19 Nov 2008 02:55:25 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811190255.mAJ2tPjq001690@svn.freebsd.org> From: Lawrence Stewart Date: Wed, 19 Nov 2008 02:55:25 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185080 - in user/lstewart/alq_varlen_7.x/sys: . kern modules modules/alq sys X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Nov 2008 02:55:26 -0000 Author: lstewart Date: Wed Nov 19 02:55:25 2008 New Revision: 185080 URL: http://svn.freebsd.org/changeset/base/185080 Log: Merge r184066 from alq_varlen_8.x and pull in changes from head r180196 whilst we're at it. Compile tested only so far. Added: user/lstewart/alq_varlen_7.x/sys/modules/alq/ user/lstewart/alq_varlen_7.x/sys/modules/alq/Makefile Modified: user/lstewart/alq_varlen_7.x/sys/ (props changed) user/lstewart/alq_varlen_7.x/sys/kern/kern_alq.c user/lstewart/alq_varlen_7.x/sys/modules/Makefile user/lstewart/alq_varlen_7.x/sys/sys/alq.h Modified: user/lstewart/alq_varlen_7.x/sys/kern/kern_alq.c ============================================================================== --- user/lstewart/alq_varlen_7.x/sys/kern/kern_alq.c Wed Nov 19 02:22:34 2008 (r185079) +++ user/lstewart/alq_varlen_7.x/sys/kern/kern_alq.c Wed Nov 19 02:55:25 2008 (r185080) @@ -1,5 +1,6 @@ /*- * Copyright (c) 2002, Jeffrey Roberson + * Copyright (c) 2008, Lawrence Stewart * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -51,14 +52,18 @@ __FBSDID("$FreeBSD$"); struct alq { int aq_entmax; /* Max entries */ int aq_entlen; /* Entry length */ + int aq_freebytes; /* Bytes available in buffer */ + int aq_buflen; /* Total length of our buffer */ char *aq_entbuf; /* Buffer for stored entries */ + int aq_writehead; + int aq_writetail; int aq_flags; /* Queue flags */ struct mtx aq_mtx; /* Queue lock */ struct vnode *aq_vp; /* Open vnode handle */ struct ucred *aq_cred; /* Credentials of the opening thread */ - struct ale *aq_first; /* First ent */ - struct ale *aq_entfree; /* First free ent */ - struct ale *aq_entvalid; /* First ent valid for writing */ + //struct ale *aq_first; /* First ent */ + //struct ale *aq_entfree; /* First free ent */ + //struct ale *aq_entvalid; /* First ent valid for writing */ LIST_ENTRY(alq) aq_act; /* List of active queues */ LIST_ENTRY(alq) aq_link; /* List of all queues */ }; @@ -182,8 +187,14 @@ ald_daemon(void) ALD_LOCK(); for (;;) { - while ((alq = LIST_FIRST(&ald_active)) == NULL) - msleep(&ald_active, &ald_mtx, PWAIT, "aldslp", 0); + while ((alq = LIST_FIRST(&ald_active)) == NULL + && !ald_shutingdown) + mtx_sleep(&ald_active, &ald_mtx, PWAIT, "aldslp", 0); + + if (ald_shutingdown) { + ALD_UNLOCK(); + break; + } ALQ_LOCK(alq); ald_deactivate(alq); @@ -191,9 +202,11 @@ ald_daemon(void) needwakeup = alq_doio(alq); ALQ_UNLOCK(alq); if (needwakeup) - wakeup(alq); + wakeup_one(alq); ALD_LOCK(); } + + kthread_exit(0); } static void @@ -204,6 +217,12 @@ ald_shutdown(void *arg, int howto) ALD_LOCK(); ald_shutingdown = 1; + /* wake ald_daemon so that it exits*/ + wakeup(&ald_active); + + /* wait for ald_daemon to exit */ + mtx_sleep(ald_thread, &ald_mtx, PWAIT, "aldslp", 0); + while ((alq = LIST_FIRST(&ald_queues)) != NULL) { LIST_REMOVE(alq, aq_link); ALD_UNLOCK(); @@ -224,9 +243,7 @@ alq_shutdown(struct alq *alq) /* Drain IO */ while (alq->aq_flags & (AQ_FLUSHING|AQ_ACTIVE)) { alq->aq_flags |= AQ_WANTED; - ALQ_UNLOCK(alq); - tsleep(alq, PWAIT, "aldclose", 0); - ALQ_LOCK(alq); + msleep_spin(alq, &alq->aq_mtx, "aldclose", 0); } ALQ_UNLOCK(alq); @@ -246,41 +263,45 @@ alq_doio(struct alq *alq) struct vnode *vp; struct uio auio; struct iovec aiov[2]; - struct ale *ale; - struct ale *alstart; int totlen; int iov; int vfslocked; + KASSERT(alq->aq_freebytes != alq->aq_buflen, + ("%s: queue emtpy!", __func__) + ); + vp = alq->aq_vp; td = curthread; totlen = 0; iov = 0; - alstart = ale = alq->aq_entvalid; - alq->aq_entvalid = NULL; - bzero(&aiov, sizeof(aiov)); bzero(&auio, sizeof(auio)); - do { - if (aiov[iov].iov_base == NULL) - aiov[iov].iov_base = ale->ae_data; - aiov[iov].iov_len += alq->aq_entlen; - totlen += alq->aq_entlen; - /* Check to see if we're wrapping the buffer */ - if (ale->ae_data + alq->aq_entlen != ale->ae_next->ae_data) - iov++; - ale->ae_flags &= ~AE_VALID; - ale = ale->ae_next; - } while (ale->ae_flags & AE_VALID); + /* start the write from the location of our buffer tail pointer */ + aiov[iov].iov_base = alq->aq_entbuf + alq->aq_writetail; + + if (alq->aq_writetail < alq->aq_writehead) { + /* buffer not wrapped */ + totlen = aiov[iov].iov_len = alq->aq_writehead - + alq->aq_writetail; + } else { + /* + * buffer wrapped, requires 2 aiov entries: + * - first is from writetail to end of buffer + * - second is from start of buffer to writehead + */ + aiov[iov].iov_len = alq->aq_buflen - alq->aq_writetail; + iov++; + aiov[iov].iov_base = alq->aq_entbuf; + aiov[iov].iov_len = alq->aq_writehead; + totlen = aiov[0].iov_len + aiov[1].iov_len; + } alq->aq_flags |= AQ_FLUSHING; ALQ_UNLOCK(alq); - if (iov == 2 || aiov[iov].iov_base == NULL) - iov--; - auio.uio_iov = &aiov[0]; auio.uio_offset = 0; auio.uio_segflg = UIO_SYSSPACE; @@ -310,8 +331,17 @@ alq_doio(struct alq *alq) ALQ_LOCK(alq); alq->aq_flags &= ~AQ_FLUSHING; - if (alq->aq_entfree == NULL) - alq->aq_entfree = alstart; + /* Adjust writetail as required, taking into account wrapping */ + alq->aq_writetail += (iov == 2) ? aiov[1].iov_len : totlen; + alq->aq_freebytes += totlen; + + /* + * If we just flushed the buffer completely, + * reset indexes to 0 to minimise buffer wraps + * This is also required to ensure alq_getn() can't wedge itself + */ + if (alq->aq_freebytes == alq->aq_buflen) + alq->aq_writehead = alq->aq_writetail = 0; if (alq->aq_flags & AQ_WANTED) { alq->aq_flags &= ~AQ_WANTED; @@ -342,13 +372,13 @@ alq_open(struct alq **alqp, const char * { struct thread *td; struct nameidata nd; - struct ale *ale; - struct ale *alp; struct alq *alq; - char *bufp; int flags; int error; - int i, vfslocked; + int vfslocked; + + KASSERT(size > 0, ("%s: size <= 0", __func__)); + KASSERT(count >= 0, ("%s: count < 0", __func__)); *alqp = NULL; td = curthread; @@ -367,31 +397,27 @@ alq_open(struct alq **alqp, const char * VFS_UNLOCK_GIANT(vfslocked); alq = malloc(sizeof(*alq), M_ALD, M_WAITOK|M_ZERO); - alq->aq_entbuf = malloc(count * size, M_ALD, M_WAITOK|M_ZERO); - alq->aq_first = malloc(sizeof(*ale) * count, M_ALD, M_WAITOK|M_ZERO); alq->aq_vp = nd.ni_vp; alq->aq_cred = crhold(cred); - alq->aq_entmax = count; - alq->aq_entlen = size; - alq->aq_entfree = alq->aq_first; mtx_init(&alq->aq_mtx, "ALD Queue", NULL, MTX_SPIN|MTX_QUIET); - bufp = alq->aq_entbuf; - ale = alq->aq_first; - alp = NULL; - - /* Match up entries with buffers */ - for (i = 0; i < count; i++) { - if (alp) - alp->ae_next = ale; - ale->ae_data = bufp; - alp = ale; - ale++; - bufp += size; + if (count > 0) { + /* fixed length messages */ + alq->aq_buflen = size * count; + alq->aq_entmax = count; + alq->aq_entlen = size; + } else { + /* variable length messages */ + alq->aq_buflen = size; + alq->aq_entmax = 0; + alq->aq_entlen = 0; } - alp->ae_next = alq->aq_first; + alq->aq_freebytes = alq->aq_buflen; + alq->aq_entbuf = malloc(alq->aq_buflen, M_ALD, M_WAITOK|M_ZERO); + + alq->aq_writehead = alq->aq_writetail = 0; if ((error = ald_add(alq)) != 0) return (error); @@ -405,48 +431,180 @@ alq_open(struct alq **alqp, const char * * wait or return an error depending on the value of waitok. */ int -alq_write(struct alq *alq, void *data, int waitok) +alq_write(struct alq *alq, void *data, int flags) { - struct ale *ale; + /* should only be called in fixed length message (legacy) mode */ + KASSERT(alq->aq_entmax > 0 && alq->aq_entlen > 0, + ("%s: fixed length write on variable length queue", __func__) + ); + return (alq_writen(alq, data, alq->aq_entlen, flags)); +} + +int +alq_writen(struct alq *alq, void *data, int len, int flags) +{ + int activate = 0; + int copy = len; + + KASSERT(len > 0 && len < alq->aq_buflen, + ("%s: len <= 0 || len > alq->aq_buflen", __func__) + ); - if ((ale = alq_get(alq, waitok)) == NULL) + ALQ_LOCK(alq); + + /* + * If the message is larger than our underlying buffer or + * there is not enough free space in our underlying buffer + * to accept the message and the user can't wait, return + */ + if ((len > alq->aq_buflen) || + ((flags & ALQ_NOWAIT) && (alq->aq_freebytes < len))) { + ALQ_UNLOCK(alq); + return (EWOULDBLOCK); + } + + /* + * ALQ_WAITOK or alq->aq_freebytes > len, + * either spin until we have enough free bytes (former) or skip (latter) + */ + while (alq->aq_freebytes < len && (alq->aq_flags & AQ_SHUTDOWN) == 0) { + alq->aq_flags |= AQ_WANTED; + msleep_spin(alq, &alq->aq_mtx, "alqwriten", 0); + } + + /* + * we need to serialise wakups to ensure records remain in order... + * therefore, wakeup the next thread in the queue waiting for + * alq resources to be available + * (technically this is only required if we actually entered the above + * while loop) + */ + wakeup_one(alq); + + /* bail if we're shutting down */ + if (alq->aq_flags & AQ_SHUTDOWN) { + ALQ_UNLOCK(alq); return (EWOULDBLOCK); + } - bcopy(data, ale->ae_data, alq->aq_entlen); - alq_post(alq, ale); + /* + * if we need to wrap the buffer to accommodate the write, + * we'll need 2 calls to bcopy + */ + if ((alq->aq_buflen - alq->aq_writehead) < len) + copy = alq->aq_buflen - alq->aq_writehead; + + /* copy (part of) message to the buffer */ + bcopy(data, alq->aq_entbuf + alq->aq_writehead, copy); + alq->aq_writehead += copy; + + if (copy != len) { + /* + * wrap the buffer by copying the remainder of our message + * to the start of the buffer and resetting the head ptr + */ + bcopy(data, alq->aq_entbuf, len - copy); + alq->aq_writehead = copy; + } + + alq->aq_freebytes -= len; + + if ((alq->aq_flags & AQ_ACTIVE) == 0) { + alq->aq_flags |= AQ_ACTIVE; + activate = 1; + } + + ALQ_UNLOCK(alq); + + if (activate) { + ALD_LOCK(); + ald_activate(alq); + ALD_UNLOCK(); + } return (0); } struct ale * -alq_get(struct alq *alq, int waitok) +alq_get(struct alq *alq, int flags) +{ + /* should only be called in fixed length message (legacy) mode */ + KASSERT(alq->aq_entmax > 0 && alq->aq_entlen > 0, + ("%s: fixed length get on variable length queue", __func__) + ); + return (alq_getn(alq, alq->aq_entlen, flags)); +} + +struct ale * +alq_getn(struct alq *alq, int len, int flags) { struct ale *ale; - struct ale *aln; + int contigbytes; + + ale = malloc( sizeof(struct ale), + M_ALD, + (flags & ALQ_NOWAIT) ? M_NOWAIT : M_WAITOK + ); - ale = NULL; + if (ale == NULL) + return (NULL); ALQ_LOCK(alq); - /* Loop until we get an entry or we're shutting down */ - while ((alq->aq_flags & AQ_SHUTDOWN) == 0 && - (ale = alq->aq_entfree) == NULL && - (waitok & ALQ_WAITOK)) { - alq->aq_flags |= AQ_WANTED; + /* determine the number of free contiguous bytes */ + if (alq->aq_writehead <= alq->aq_writetail) + contigbytes = alq->aq_freebytes; + else + contigbytes = alq->aq_buflen - alq->aq_writehead; + + /* + * If the message is larger than our underlying buffer or + * there is not enough free contiguous space in our underlying buffer + * to accept the message and the user can't wait, return + */ + if ((len > alq->aq_buflen) || + ((flags & ALQ_NOWAIT) && (contigbytes < len))) { ALQ_UNLOCK(alq); - tsleep(alq, PWAIT, "alqget", 0); - ALQ_LOCK(alq); + return (NULL); } - if (ale != NULL) { - aln = ale->ae_next; - if ((aln->ae_flags & AE_VALID) == 0) - alq->aq_entfree = aln; + /* + * ALQ_WAITOK or contigbytes > len, + * either spin until we have enough free contiguous bytes (former) + * or skip (latter) + */ + while (contigbytes < len && (alq->aq_flags & AQ_SHUTDOWN) == 0) { + alq->aq_flags |= AQ_WANTED; + msleep_spin(alq, &alq->aq_mtx, "alqgetn", 0); + if (alq->aq_writehead <= alq->aq_writetail) + contigbytes = alq->aq_freebytes; else - alq->aq_entfree = NULL; - } else + contigbytes = alq->aq_buflen - alq->aq_writehead; + } + + /* + * we need to serialise wakups to ensure records remain in order... + * therefore, wakeup the next thread in the queue waiting for + * alq resources to be available + * (technically this is only required if we actually entered the above + * while loop) + */ + wakeup_one(alq); + + /* bail if we're shutting down */ + if (alq->aq_flags & AQ_SHUTDOWN) { ALQ_UNLOCK(alq); + return (NULL); + } + /* + * If we are here, we have a contiguous number of bytes >= len + * available in our buffer starting at aq_writehead. + */ + ale->ae_data = alq->aq_entbuf + alq->aq_writehead; + ale->ae_datalen = len; + alq->aq_writehead += len; + alq->aq_freebytes -= len; return (ale); } @@ -456,11 +614,6 @@ alq_post(struct alq *alq, struct ale *al { int activate; - ale->ae_flags |= AE_VALID; - - if (alq->aq_entvalid == NULL) - alq->aq_entvalid = ale; - if ((alq->aq_flags & AQ_ACTIVE) == 0) { alq->aq_flags |= AQ_ACTIVE; activate = 1; @@ -468,11 +621,14 @@ alq_post(struct alq *alq, struct ale *al activate = 0; ALQ_UNLOCK(alq); + if (activate) { ALD_LOCK(); ald_activate(alq); ALD_UNLOCK(); } + + free(ale, M_ALD); } void @@ -491,7 +647,7 @@ alq_flush(struct alq *alq) ALQ_UNLOCK(alq); if (needwakeup) - wakeup(alq); + wakeup_one(alq); } /* @@ -513,7 +669,49 @@ alq_close(struct alq *alq) alq_shutdown(alq); mtx_destroy(&alq->aq_mtx); - free(alq->aq_first, M_ALD); free(alq->aq_entbuf, M_ALD); free(alq, M_ALD); } + +static int alq_load_handler(module_t mod, int what, void *arg) +{ + int ret = 0; + + switch(what) { + case MOD_LOAD: + case MOD_UNLOAD: + case MOD_SHUTDOWN: + break; + + case MOD_QUIESCE: + ALD_LOCK(); + /* only allow unload if there are no open queues */ + if (LIST_FIRST(&ald_queues) == NULL) { + ald_shutingdown = 1; + ALD_UNLOCK(); + ald_shutdown(NULL, 0); + mtx_destroy(&ald_mtx); + } else { + ALD_UNLOCK(); + ret = EBUSY; + } + break; + + default: + ret = EINVAL; + break; + } + + return (ret); +} + +/* basic module data */ +static moduledata_t alq_mod = +{ + "alq", + alq_load_handler, /* execution entry point for the module */ + NULL +}; + +DECLARE_MODULE(alq, alq_mod, SI_SUB_SMP, SI_ORDER_ANY); +MODULE_VERSION(alq, 1); Modified: user/lstewart/alq_varlen_7.x/sys/modules/Makefile ============================================================================== --- user/lstewart/alq_varlen_7.x/sys/modules/Makefile Wed Nov 19 02:22:34 2008 (r185079) +++ user/lstewart/alq_varlen_7.x/sys/modules/Makefile Wed Nov 19 02:55:25 2008 (r185080) @@ -16,6 +16,7 @@ SUBDIR= ${_3dfx} \ ${_aic} \ aic7xxx \ aio \ + alq \ ${_amd} \ amr \ ${_an} \ Added: user/lstewart/alq_varlen_7.x/sys/modules/alq/Makefile ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/lstewart/alq_varlen_7.x/sys/modules/alq/Makefile Wed Nov 19 02:55:25 2008 (r185080) @@ -0,0 +1,10 @@ +# $FreeBSD$ + +.include + +.PATH: ${.CURDIR}/../../kern +KMOD=alq +SRCS=opt_mac.h vnode_if.h kern_alq.c + +.include + Modified: user/lstewart/alq_varlen_7.x/sys/sys/alq.h ============================================================================== --- user/lstewart/alq_varlen_7.x/sys/sys/alq.h Wed Nov 19 02:22:34 2008 (r185079) +++ user/lstewart/alq_varlen_7.x/sys/sys/alq.h Wed Nov 19 02:55:25 2008 (r185080) @@ -41,15 +41,15 @@ extern struct thread *ald_thread; * Async. Logging Entry */ struct ale { - struct ale *ae_next; /* Next Entry */ + //struct ale *ae_next; /* Next Entry */ char *ae_data; /* Entry buffer */ - int ae_flags; /* Entry flags */ + int ae_datalen; /* Length of buffer */ + //int ae_flags; /* Entry flags */ }; -#define AE_VALID 0x0001 /* Entry has valid data */ - +//#define AE_VALID 0x0001 /* Entry has valid data */ -/* waitok options */ +/* flags options */ #define ALQ_NOWAIT 0x0001 #define ALQ_WAITOK 0x0002 @@ -88,7 +88,8 @@ int alq_open(struct alq **, const char * * The system is shutting down. * 0 on success. */ -int alq_write(struct alq *alq, void *data, int waitok); +int alq_write(struct alq *alq, void *data, int flags); +int alq_writen(struct alq *alq, void *data, int len, int flags); /* * alq_flush: Flush the queue out to disk @@ -115,13 +116,14 @@ void alq_close(struct alq *alq); * * This leaves the queue locked until a subsequent alq_post. */ -struct ale *alq_get(struct alq *alq, int waitok); +struct ale *alq_get(struct alq *alq, int flags); +struct ale *alq_getn(struct alq *alq, int len, int flags); /* * alq_post: Schedule the ale retrieved by alq_get for writing. * alq The queue to post the entry to. * ale An asynch logging entry returned by alq_get. */ -void alq_post(struct alq *, struct ale *); +void alq_post(struct alq *alq, struct ale *ale); #endif /* _SYS_ALQ_H_ */ From owner-svn-src-user@FreeBSD.ORG Wed Nov 19 03:14:42 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A1BF1065677; Wed, 19 Nov 2008 03:14:42 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id E4CCF8FC16; Wed, 19 Nov 2008 03:14:41 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAJ3EfFs002426; Wed, 19 Nov 2008 03:14:41 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAJ3EfuA002423; Wed, 19 Nov 2008 03:14:41 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811190314.mAJ3EfuA002423@svn.freebsd.org> From: Lawrence Stewart Date: Wed, 19 Nov 2008 03:14:41 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185081 - in user/lstewart/dummynet_7.x/sys: . kern modules modules/alq sys X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Nov 2008 03:14:42 -0000 Author: lstewart Date: Wed Nov 19 03:14:41 2008 New Revision: 185081 URL: http://svn.freebsd.org/changeset/base/185081 Log: Pull in the 7.x alq module. Added: user/lstewart/dummynet_7.x/sys/modules/alq/ - copied from r185080, user/lstewart/alq_varlen_7.x/sys/modules/alq/ Modified: user/lstewart/dummynet_7.x/sys/ (props changed) user/lstewart/dummynet_7.x/sys/kern/kern_alq.c user/lstewart/dummynet_7.x/sys/modules/Makefile user/lstewart/dummynet_7.x/sys/sys/alq.h Modified: user/lstewart/dummynet_7.x/sys/kern/kern_alq.c ============================================================================== --- user/lstewart/dummynet_7.x/sys/kern/kern_alq.c Wed Nov 19 02:55:25 2008 (r185080) +++ user/lstewart/dummynet_7.x/sys/kern/kern_alq.c Wed Nov 19 03:14:41 2008 (r185081) @@ -1,5 +1,6 @@ /*- * Copyright (c) 2002, Jeffrey Roberson + * Copyright (c) 2008, Lawrence Stewart * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -51,14 +52,18 @@ __FBSDID("$FreeBSD$"); struct alq { int aq_entmax; /* Max entries */ int aq_entlen; /* Entry length */ + int aq_freebytes; /* Bytes available in buffer */ + int aq_buflen; /* Total length of our buffer */ char *aq_entbuf; /* Buffer for stored entries */ + int aq_writehead; + int aq_writetail; int aq_flags; /* Queue flags */ struct mtx aq_mtx; /* Queue lock */ struct vnode *aq_vp; /* Open vnode handle */ struct ucred *aq_cred; /* Credentials of the opening thread */ - struct ale *aq_first; /* First ent */ - struct ale *aq_entfree; /* First free ent */ - struct ale *aq_entvalid; /* First ent valid for writing */ + //struct ale *aq_first; /* First ent */ + //struct ale *aq_entfree; /* First free ent */ + //struct ale *aq_entvalid; /* First ent valid for writing */ LIST_ENTRY(alq) aq_act; /* List of active queues */ LIST_ENTRY(alq) aq_link; /* List of all queues */ }; @@ -182,8 +187,14 @@ ald_daemon(void) ALD_LOCK(); for (;;) { - while ((alq = LIST_FIRST(&ald_active)) == NULL) - msleep(&ald_active, &ald_mtx, PWAIT, "aldslp", 0); + while ((alq = LIST_FIRST(&ald_active)) == NULL + && !ald_shutingdown) + mtx_sleep(&ald_active, &ald_mtx, PWAIT, "aldslp", 0); + + if (ald_shutingdown) { + ALD_UNLOCK(); + break; + } ALQ_LOCK(alq); ald_deactivate(alq); @@ -191,9 +202,11 @@ ald_daemon(void) needwakeup = alq_doio(alq); ALQ_UNLOCK(alq); if (needwakeup) - wakeup(alq); + wakeup_one(alq); ALD_LOCK(); } + + kthread_exit(0); } static void @@ -204,6 +217,12 @@ ald_shutdown(void *arg, int howto) ALD_LOCK(); ald_shutingdown = 1; + /* wake ald_daemon so that it exits*/ + wakeup(&ald_active); + + /* wait for ald_daemon to exit */ + mtx_sleep(ald_thread, &ald_mtx, PWAIT, "aldslp", 0); + while ((alq = LIST_FIRST(&ald_queues)) != NULL) { LIST_REMOVE(alq, aq_link); ALD_UNLOCK(); @@ -224,9 +243,7 @@ alq_shutdown(struct alq *alq) /* Drain IO */ while (alq->aq_flags & (AQ_FLUSHING|AQ_ACTIVE)) { alq->aq_flags |= AQ_WANTED; - ALQ_UNLOCK(alq); - tsleep(alq, PWAIT, "aldclose", 0); - ALQ_LOCK(alq); + msleep_spin(alq, &alq->aq_mtx, "aldclose", 0); } ALQ_UNLOCK(alq); @@ -246,41 +263,45 @@ alq_doio(struct alq *alq) struct vnode *vp; struct uio auio; struct iovec aiov[2]; - struct ale *ale; - struct ale *alstart; int totlen; int iov; int vfslocked; + KASSERT(alq->aq_freebytes != alq->aq_buflen, + ("%s: queue emtpy!", __func__) + ); + vp = alq->aq_vp; td = curthread; totlen = 0; iov = 0; - alstart = ale = alq->aq_entvalid; - alq->aq_entvalid = NULL; - bzero(&aiov, sizeof(aiov)); bzero(&auio, sizeof(auio)); - do { - if (aiov[iov].iov_base == NULL) - aiov[iov].iov_base = ale->ae_data; - aiov[iov].iov_len += alq->aq_entlen; - totlen += alq->aq_entlen; - /* Check to see if we're wrapping the buffer */ - if (ale->ae_data + alq->aq_entlen != ale->ae_next->ae_data) - iov++; - ale->ae_flags &= ~AE_VALID; - ale = ale->ae_next; - } while (ale->ae_flags & AE_VALID); + /* start the write from the location of our buffer tail pointer */ + aiov[iov].iov_base = alq->aq_entbuf + alq->aq_writetail; + + if (alq->aq_writetail < alq->aq_writehead) { + /* buffer not wrapped */ + totlen = aiov[iov].iov_len = alq->aq_writehead - + alq->aq_writetail; + } else { + /* + * buffer wrapped, requires 2 aiov entries: + * - first is from writetail to end of buffer + * - second is from start of buffer to writehead + */ + aiov[iov].iov_len = alq->aq_buflen - alq->aq_writetail; + iov++; + aiov[iov].iov_base = alq->aq_entbuf; + aiov[iov].iov_len = alq->aq_writehead; + totlen = aiov[0].iov_len + aiov[1].iov_len; + } alq->aq_flags |= AQ_FLUSHING; ALQ_UNLOCK(alq); - if (iov == 2 || aiov[iov].iov_base == NULL) - iov--; - auio.uio_iov = &aiov[0]; auio.uio_offset = 0; auio.uio_segflg = UIO_SYSSPACE; @@ -310,8 +331,17 @@ alq_doio(struct alq *alq) ALQ_LOCK(alq); alq->aq_flags &= ~AQ_FLUSHING; - if (alq->aq_entfree == NULL) - alq->aq_entfree = alstart; + /* Adjust writetail as required, taking into account wrapping */ + alq->aq_writetail += (iov == 2) ? aiov[1].iov_len : totlen; + alq->aq_freebytes += totlen; + + /* + * If we just flushed the buffer completely, + * reset indexes to 0 to minimise buffer wraps + * This is also required to ensure alq_getn() can't wedge itself + */ + if (alq->aq_freebytes == alq->aq_buflen) + alq->aq_writehead = alq->aq_writetail = 0; if (alq->aq_flags & AQ_WANTED) { alq->aq_flags &= ~AQ_WANTED; @@ -342,13 +372,13 @@ alq_open(struct alq **alqp, const char * { struct thread *td; struct nameidata nd; - struct ale *ale; - struct ale *alp; struct alq *alq; - char *bufp; int flags; int error; - int i, vfslocked; + int vfslocked; + + KASSERT(size > 0, ("%s: size <= 0", __func__)); + KASSERT(count >= 0, ("%s: count < 0", __func__)); *alqp = NULL; td = curthread; @@ -367,31 +397,27 @@ alq_open(struct alq **alqp, const char * VFS_UNLOCK_GIANT(vfslocked); alq = malloc(sizeof(*alq), M_ALD, M_WAITOK|M_ZERO); - alq->aq_entbuf = malloc(count * size, M_ALD, M_WAITOK|M_ZERO); - alq->aq_first = malloc(sizeof(*ale) * count, M_ALD, M_WAITOK|M_ZERO); alq->aq_vp = nd.ni_vp; alq->aq_cred = crhold(cred); - alq->aq_entmax = count; - alq->aq_entlen = size; - alq->aq_entfree = alq->aq_first; mtx_init(&alq->aq_mtx, "ALD Queue", NULL, MTX_SPIN|MTX_QUIET); - bufp = alq->aq_entbuf; - ale = alq->aq_first; - alp = NULL; - - /* Match up entries with buffers */ - for (i = 0; i < count; i++) { - if (alp) - alp->ae_next = ale; - ale->ae_data = bufp; - alp = ale; - ale++; - bufp += size; + if (count > 0) { + /* fixed length messages */ + alq->aq_buflen = size * count; + alq->aq_entmax = count; + alq->aq_entlen = size; + } else { + /* variable length messages */ + alq->aq_buflen = size; + alq->aq_entmax = 0; + alq->aq_entlen = 0; } - alp->ae_next = alq->aq_first; + alq->aq_freebytes = alq->aq_buflen; + alq->aq_entbuf = malloc(alq->aq_buflen, M_ALD, M_WAITOK|M_ZERO); + + alq->aq_writehead = alq->aq_writetail = 0; if ((error = ald_add(alq)) != 0) return (error); @@ -405,48 +431,180 @@ alq_open(struct alq **alqp, const char * * wait or return an error depending on the value of waitok. */ int -alq_write(struct alq *alq, void *data, int waitok) +alq_write(struct alq *alq, void *data, int flags) { - struct ale *ale; + /* should only be called in fixed length message (legacy) mode */ + KASSERT(alq->aq_entmax > 0 && alq->aq_entlen > 0, + ("%s: fixed length write on variable length queue", __func__) + ); + return (alq_writen(alq, data, alq->aq_entlen, flags)); +} + +int +alq_writen(struct alq *alq, void *data, int len, int flags) +{ + int activate = 0; + int copy = len; + + KASSERT(len > 0 && len < alq->aq_buflen, + ("%s: len <= 0 || len > alq->aq_buflen", __func__) + ); - if ((ale = alq_get(alq, waitok)) == NULL) + ALQ_LOCK(alq); + + /* + * If the message is larger than our underlying buffer or + * there is not enough free space in our underlying buffer + * to accept the message and the user can't wait, return + */ + if ((len > alq->aq_buflen) || + ((flags & ALQ_NOWAIT) && (alq->aq_freebytes < len))) { + ALQ_UNLOCK(alq); + return (EWOULDBLOCK); + } + + /* + * ALQ_WAITOK or alq->aq_freebytes > len, + * either spin until we have enough free bytes (former) or skip (latter) + */ + while (alq->aq_freebytes < len && (alq->aq_flags & AQ_SHUTDOWN) == 0) { + alq->aq_flags |= AQ_WANTED; + msleep_spin(alq, &alq->aq_mtx, "alqwriten", 0); + } + + /* + * we need to serialise wakups to ensure records remain in order... + * therefore, wakeup the next thread in the queue waiting for + * alq resources to be available + * (technically this is only required if we actually entered the above + * while loop) + */ + wakeup_one(alq); + + /* bail if we're shutting down */ + if (alq->aq_flags & AQ_SHUTDOWN) { + ALQ_UNLOCK(alq); return (EWOULDBLOCK); + } - bcopy(data, ale->ae_data, alq->aq_entlen); - alq_post(alq, ale); + /* + * if we need to wrap the buffer to accommodate the write, + * we'll need 2 calls to bcopy + */ + if ((alq->aq_buflen - alq->aq_writehead) < len) + copy = alq->aq_buflen - alq->aq_writehead; + + /* copy (part of) message to the buffer */ + bcopy(data, alq->aq_entbuf + alq->aq_writehead, copy); + alq->aq_writehead += copy; + + if (copy != len) { + /* + * wrap the buffer by copying the remainder of our message + * to the start of the buffer and resetting the head ptr + */ + bcopy(data, alq->aq_entbuf, len - copy); + alq->aq_writehead = copy; + } + + alq->aq_freebytes -= len; + + if ((alq->aq_flags & AQ_ACTIVE) == 0) { + alq->aq_flags |= AQ_ACTIVE; + activate = 1; + } + + ALQ_UNLOCK(alq); + + if (activate) { + ALD_LOCK(); + ald_activate(alq); + ALD_UNLOCK(); + } return (0); } struct ale * -alq_get(struct alq *alq, int waitok) +alq_get(struct alq *alq, int flags) +{ + /* should only be called in fixed length message (legacy) mode */ + KASSERT(alq->aq_entmax > 0 && alq->aq_entlen > 0, + ("%s: fixed length get on variable length queue", __func__) + ); + return (alq_getn(alq, alq->aq_entlen, flags)); +} + +struct ale * +alq_getn(struct alq *alq, int len, int flags) { struct ale *ale; - struct ale *aln; + int contigbytes; + + ale = malloc( sizeof(struct ale), + M_ALD, + (flags & ALQ_NOWAIT) ? M_NOWAIT : M_WAITOK + ); - ale = NULL; + if (ale == NULL) + return (NULL); ALQ_LOCK(alq); - /* Loop until we get an entry or we're shutting down */ - while ((alq->aq_flags & AQ_SHUTDOWN) == 0 && - (ale = alq->aq_entfree) == NULL && - (waitok & ALQ_WAITOK)) { - alq->aq_flags |= AQ_WANTED; + /* determine the number of free contiguous bytes */ + if (alq->aq_writehead <= alq->aq_writetail) + contigbytes = alq->aq_freebytes; + else + contigbytes = alq->aq_buflen - alq->aq_writehead; + + /* + * If the message is larger than our underlying buffer or + * there is not enough free contiguous space in our underlying buffer + * to accept the message and the user can't wait, return + */ + if ((len > alq->aq_buflen) || + ((flags & ALQ_NOWAIT) && (contigbytes < len))) { ALQ_UNLOCK(alq); - tsleep(alq, PWAIT, "alqget", 0); - ALQ_LOCK(alq); + return (NULL); } - if (ale != NULL) { - aln = ale->ae_next; - if ((aln->ae_flags & AE_VALID) == 0) - alq->aq_entfree = aln; + /* + * ALQ_WAITOK or contigbytes > len, + * either spin until we have enough free contiguous bytes (former) + * or skip (latter) + */ + while (contigbytes < len && (alq->aq_flags & AQ_SHUTDOWN) == 0) { + alq->aq_flags |= AQ_WANTED; + msleep_spin(alq, &alq->aq_mtx, "alqgetn", 0); + if (alq->aq_writehead <= alq->aq_writetail) + contigbytes = alq->aq_freebytes; else - alq->aq_entfree = NULL; - } else + contigbytes = alq->aq_buflen - alq->aq_writehead; + } + + /* + * we need to serialise wakups to ensure records remain in order... + * therefore, wakeup the next thread in the queue waiting for + * alq resources to be available + * (technically this is only required if we actually entered the above + * while loop) + */ + wakeup_one(alq); + + /* bail if we're shutting down */ + if (alq->aq_flags & AQ_SHUTDOWN) { ALQ_UNLOCK(alq); + return (NULL); + } + /* + * If we are here, we have a contiguous number of bytes >= len + * available in our buffer starting at aq_writehead. + */ + ale->ae_data = alq->aq_entbuf + alq->aq_writehead; + ale->ae_datalen = len; + alq->aq_writehead += len; + alq->aq_freebytes -= len; return (ale); } @@ -456,11 +614,6 @@ alq_post(struct alq *alq, struct ale *al { int activate; - ale->ae_flags |= AE_VALID; - - if (alq->aq_entvalid == NULL) - alq->aq_entvalid = ale; - if ((alq->aq_flags & AQ_ACTIVE) == 0) { alq->aq_flags |= AQ_ACTIVE; activate = 1; @@ -468,11 +621,14 @@ alq_post(struct alq *alq, struct ale *al activate = 0; ALQ_UNLOCK(alq); + if (activate) { ALD_LOCK(); ald_activate(alq); ALD_UNLOCK(); } + + free(ale, M_ALD); } void @@ -491,7 +647,7 @@ alq_flush(struct alq *alq) ALQ_UNLOCK(alq); if (needwakeup) - wakeup(alq); + wakeup_one(alq); } /* @@ -513,7 +669,49 @@ alq_close(struct alq *alq) alq_shutdown(alq); mtx_destroy(&alq->aq_mtx); - free(alq->aq_first, M_ALD); free(alq->aq_entbuf, M_ALD); free(alq, M_ALD); } + +static int alq_load_handler(module_t mod, int what, void *arg) +{ + int ret = 0; + + switch(what) { + case MOD_LOAD: + case MOD_UNLOAD: + case MOD_SHUTDOWN: + break; + + case MOD_QUIESCE: + ALD_LOCK(); + /* only allow unload if there are no open queues */ + if (LIST_FIRST(&ald_queues) == NULL) { + ald_shutingdown = 1; + ALD_UNLOCK(); + ald_shutdown(NULL, 0); + mtx_destroy(&ald_mtx); + } else { + ALD_UNLOCK(); + ret = EBUSY; + } + break; + + default: + ret = EINVAL; + break; + } + + return (ret); +} + +/* basic module data */ +static moduledata_t alq_mod = +{ + "alq", + alq_load_handler, /* execution entry point for the module */ + NULL +}; + +DECLARE_MODULE(alq, alq_mod, SI_SUB_SMP, SI_ORDER_ANY); +MODULE_VERSION(alq, 1); Modified: user/lstewart/dummynet_7.x/sys/modules/Makefile ============================================================================== --- user/lstewart/dummynet_7.x/sys/modules/Makefile Wed Nov 19 02:55:25 2008 (r185080) +++ user/lstewart/dummynet_7.x/sys/modules/Makefile Wed Nov 19 03:14:41 2008 (r185081) @@ -16,6 +16,7 @@ SUBDIR= ${_3dfx} \ ${_aic} \ aic7xxx \ aio \ + alq \ ${_amd} \ amr \ ${_an} \ Modified: user/lstewart/dummynet_7.x/sys/sys/alq.h ============================================================================== --- user/lstewart/dummynet_7.x/sys/sys/alq.h Wed Nov 19 02:55:25 2008 (r185080) +++ user/lstewart/dummynet_7.x/sys/sys/alq.h Wed Nov 19 03:14:41 2008 (r185081) @@ -41,15 +41,15 @@ extern struct thread *ald_thread; * Async. Logging Entry */ struct ale { - struct ale *ae_next; /* Next Entry */ + //struct ale *ae_next; /* Next Entry */ char *ae_data; /* Entry buffer */ - int ae_flags; /* Entry flags */ + int ae_datalen; /* Length of buffer */ + //int ae_flags; /* Entry flags */ }; -#define AE_VALID 0x0001 /* Entry has valid data */ - +//#define AE_VALID 0x0001 /* Entry has valid data */ -/* waitok options */ +/* flags options */ #define ALQ_NOWAIT 0x0001 #define ALQ_WAITOK 0x0002 @@ -88,7 +88,8 @@ int alq_open(struct alq **, const char * * The system is shutting down. * 0 on success. */ -int alq_write(struct alq *alq, void *data, int waitok); +int alq_write(struct alq *alq, void *data, int flags); +int alq_writen(struct alq *alq, void *data, int len, int flags); /* * alq_flush: Flush the queue out to disk @@ -115,13 +116,14 @@ void alq_close(struct alq *alq); * * This leaves the queue locked until a subsequent alq_post. */ -struct ale *alq_get(struct alq *alq, int waitok); +struct ale *alq_get(struct alq *alq, int flags); +struct ale *alq_getn(struct alq *alq, int len, int flags); /* * alq_post: Schedule the ale retrieved by alq_get for writing. * alq The queue to post the entry to. * ale An asynch logging entry returned by alq_get. */ -void alq_post(struct alq *, struct ale *); +void alq_post(struct alq *alq, struct ale *ale); #endif /* _SYS_ALQ_H_ */ From owner-svn-src-user@FreeBSD.ORG Wed Nov 19 03:57:11 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D44811065679; Wed, 19 Nov 2008 03:57:11 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id BA0348FC17; Wed, 19 Nov 2008 03:57:11 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAJ3vBt7003285; Wed, 19 Nov 2008 03:57:11 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAJ3vBtv003282; Wed, 19 Nov 2008 03:57:11 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811190357.mAJ3vBtv003282@svn.freebsd.org> From: Lawrence Stewart Date: Wed, 19 Nov 2008 03:57:11 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185083 - in user/lstewart/dummynet_7.x: sbin/ipfw sys sys/netinet X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Nov 2008 03:57:12 -0000 Author: lstewart Date: Wed Nov 19 03:57:11 2008 New Revision: 185083 URL: http://svn.freebsd.org/changeset/base/185083 Log: Merge the dummynet and ipfw man page bits of dummynet_8.x r185051. Couple of minor tweaks required to make it compile (namely kthread_* kpi differences). Modified: user/lstewart/dummynet_7.x/sbin/ipfw/ (props changed) user/lstewart/dummynet_7.x/sbin/ipfw/ipfw.8 user/lstewart/dummynet_7.x/sys/ (props changed) user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.c user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.h Modified: user/lstewart/dummynet_7.x/sbin/ipfw/ipfw.8 ============================================================================== --- user/lstewart/dummynet_7.x/sbin/ipfw/ipfw.8 Wed Nov 19 03:24:35 2008 (r185082) +++ user/lstewart/dummynet_7.x/sbin/ipfw/ipfw.8 Wed Nov 19 03:57:11 2008 (r185083) @@ -1967,12 +1967,13 @@ Packet loss set. Argument .Ar packet-loss-set is a comma-delimited string of the form 10,30-31,1000 identifying the specific -packets entering a queue/pipe to drop. In the given example, the 10th, 30th, -31st and 1000th packet to enter the pipe/queue would be dropped. Clearing the -counters on a pipe will cause the +packets entering a queue/pipe to drop. +In the given example, the 10th, 30th, 31st and 1000th packet to enter the +pipe/queue would be dropped. +Clearing the counters on a pipe will cause the .Ar packet-loss-set -to be evaluated again from scratch. Use of this option mutually excludes use of -the +to be evaluated again from scratch. +Use of this option mutually excludes use of the .Nm plr option. .Pp Modified: user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.c ============================================================================== --- user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.c Wed Nov 19 03:24:35 2008 (r185082) +++ user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.c Wed Nov 19 03:57:11 2008 (r185083) @@ -64,28 +64,39 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include #include #include #include +#include +#include +#include +#include + #include #include #include #include #include #include +#include #include #include #include #include +#include +#include #include /* for struct arpcom */ #include /* for ip6_input, ip6_output prototypes */ #include +#include + /* * We keep a private variable for the simulation time, but we could * probably use an existing one ("softticks" in sys/kern/kern_timeout.c) @@ -154,6 +165,74 @@ static struct callout dn_timeout; extern void (*bridge_dn_p)(struct mbuf *, struct ifnet *); +#define DN_LOG(fs, p, q, m, dropped, dir) \ + if (dn_log_enable) \ + dn_log((fs), (p), (q), (m), (dropped), (dir)); + +#define CAST_PTR_INT(X) (*((int*)(X))) + +struct log_node { + /* log msg creation timestamp */ + struct timeval tval; + /* + * direction of packet after dummynet finishes processing it + * (defined in ip_dummynet.h DN_TO_IP_OUT, DN_TO_IP_IN, ...) + */ + int direction; + /* + * pkt dropped yes/no + reason if dropped (see DN_DROP_X defines in + * ip_dummynet.h) + */ + uint32_t dropped; + /* hash of the pkt which triggered the log msg */ + uint32_t hash; + /* IP version log_node relates to; either INP_IPV4 or INP_IPV6 */ + uint8_t ipver; + /* flow set number */ + int fs_num; + /* flags set on the flow set */ + uint16_t fs_flags; + /* pipe number */ + int p_num; + /* current pipe occupancy */ + int p_len; + /* + * max queue len in either pkts or bytes (depending on whether + * DN_QSIZE_IS_BYTES is set in fs_flags) + */ + int q_max_len; + /* current queue occupancy in pkts */ + int q_len_pkts; + /* current queue occupancy in bytes */ + int q_len_bytes; + + STAILQ_ENTRY(log_node) nodes; +}; + +/* + * in_pcb.h defines INP_IPV4 as 0x1 and INP_IPV6 as 0x2, + * which we use as an index into this array + */ +static char ipver[3] = {'\0', '4', '6'}; +static int dn_sysctl_log_enable_handler(SYSCTL_HANDLER_ARGS); +static int dn_sysctl_logfile_name_handler(SYSCTL_HANDLER_ARGS); +static u_int dn_log_enable = 0; +static char dn_logfile[PATH_MAX] = "/var/log/dummynet.log\0"; +STAILQ_HEAD(loghead, log_node) log_queue = STAILQ_HEAD_INITIALIZER(log_queue); +static struct mtx dn_log_queue_mtx; +static int wait_for_log; +static struct alq *dn_alq = NULL; +static volatile uint32_t dn_exit_log_manager_thread = 0; +static struct thread *dn_log_manager_thr = NULL; +static struct proc *dn_log_manager_proc = NULL; + +#define DN_LOG_FILE_MODE 0644 +#define DN_ALQ_BUFLEN 200000 +#define DN_MAX_LOG_MSG_LEN 60 + +#define DN_LOG_DISABLE 0 +#define DN_LOG_ENABLE 1 + #ifdef SYSCTL_NODE SYSCTL_NODE(_net_inet_ip, OID_AUTO, dummynet, CTLFLAG_RW, 0, "Dummynet"); SYSCTL_INT(_net_inet_ip_dummynet, OID_AUTO, hash_size, @@ -206,6 +285,12 @@ SYSCTL_LONG(_net_inet_ip_dummynet, OID_A CTLFLAG_RW, &pipe_slot_limit, 0, "Upper limit in slots for pipe queue."); SYSCTL_LONG(_net_inet_ip_dummynet, OID_AUTO, pipe_byte_limit, CTLFLAG_RW, &pipe_byte_limit, 0, "Upper limit in bytes for pipe queue."); +SYSCTL_OID(_net_inet_ip_dummynet, OID_AUTO, log_enable, CTLTYPE_UINT|CTLFLAG_RW, + &dn_log_enable, 0, &dn_sysctl_log_enable_handler, "IU", + "switch dummynet data logging on/off"); +SYSCTL_PROC(_net_inet_ip_dummynet, OID_AUTO, logfile, + CTLTYPE_STRING|CTLFLAG_RW, &dn_logfile, sizeof(dn_logfile), + &dn_sysctl_logfile_name_handler, "A", "file to save dummynet log data to"); #endif #ifdef DUMMYNET_DEBUG @@ -451,6 +536,363 @@ heap_free(struct dn_heap *h) * --- end of heap management functions --- */ +static __inline void +dn_process_log_node(struct log_node * log_node) +{ + char dn_log_msg[DN_MAX_LOG_MSG_LEN]; + + /* construct our log message */ + snprintf( dn_log_msg, + DN_MAX_LOG_MSG_LEN, + "%d,0x%08x,%u.%06u,%u,%u,%d,0x%04x,%d,%d,%d,%d,%d\n", + log_node->direction, + log_node->hash, + (unsigned int)log_node->tval.tv_sec, + (unsigned int)log_node->tval.tv_usec, + ipver[log_node->ipver], + log_node->dropped, + log_node->fs_num, + log_node->fs_flags, + log_node->p_num, + log_node->p_len, + log_node->q_max_len, + log_node->q_len_pkts, + log_node->q_len_bytes + ); + + alq_writen(dn_alq, dn_log_msg, strlen(dn_log_msg), ALQ_WAITOK); +} + +static void +dn_log_manager_thread(void *arg) +{ + struct log_node *log_node, *log_node_temp; + + /* loop until thread is signalled to exit */ + while (!dn_exit_log_manager_thread) { + /* + * sleep until we are signalled to wake because thread has + * been told to exit or until 1 tick has passed + */ + tsleep(&wait_for_log, PWAIT, "logwait", 1); + + /* Process logs until the queue is empty */ + do { + log_node = NULL; + + /* gain exclusive access to the queue */ + mtx_lock(&dn_log_queue_mtx); + + /* get the element at the head of the list */ + if ((log_node = STAILQ_FIRST(&log_queue)) != NULL) { + /* + * list wasn't empty, so let's remove the first + * element from the list. + * Note that STAILQ_REMOVE_HEAD doesn't delete + * the log_node struct itself. It just + * disentangles it from the list structure. + * We have a copy of the node's ptr stored + * in log_node. + */ + STAILQ_REMOVE_HEAD(&log_queue, nodes); + } + /* + * We've finished making changes to the list. Unlock it + * so the pfil hooks can continue queuing pkt_nodes + */ + mtx_unlock(&dn_log_queue_mtx); + + /* if we successfully get a log_node from the list */ + if (log_node != NULL) { + dn_process_log_node(log_node); + /* + * free the memory that was + * malloc'd in dn_log() + */ + free(log_node, M_DUMMYNET); + } + + } while (log_node != NULL); + } + + /* Flush all remaining log_nodes to the log file */ + + /* Lock the mutex so we gain exclusive access to the queue */ + mtx_lock(&dn_log_queue_mtx); + + STAILQ_FOREACH_SAFE(log_node, &log_queue, nodes, log_node_temp) { + dn_process_log_node(log_node); + STAILQ_REMOVE_HEAD(&log_queue, nodes); + free(log_node, M_DUMMYNET); + } + + /* Reinit the list to mark it as empty and virgin */ + STAILQ_INIT(&log_queue); + + /* We've finished making changes to the list. Safe to unlock it. */ + mtx_unlock(&dn_log_queue_mtx); + + /* kthread_exit calls wakeup on our thread's struct pointer */ + kthread_exit(0); +} + +static int +dn_sysctl_logfile_name_handler(SYSCTL_HANDLER_ARGS) +{ + struct alq *new_alq; + + if (!req->newptr) + goto skip; + + /* if old filename and new filename are different */ + if (strncmp(dn_logfile, (char *)req->newptr, PATH_MAX)) { + + int error = alq_open( &new_alq, + req->newptr, + curthread->td_ucred, + DN_LOG_FILE_MODE, + DN_ALQ_BUFLEN, + 0 + ); + + /* bail if unable to create new alq */ + if (error) + return 1; + + /* + * If disabled, dn_alq == NULL so we simply close + * the alq as we've proved it can be opened. + * If enabled, close the existing alq and switch the old for the new + */ + if (dn_alq == NULL) + alq_close(new_alq); + else { + alq_close(dn_alq); + dn_alq = new_alq; + } + } + +skip: + return sysctl_handle_string(oidp, arg1, arg2, req); +} + +static int +dn_manage_logging(uint8_t action) +{ + int ret, error = 0; + struct timeval tval; + struct sbuf *s = NULL; + + /* init an autosizing sbuf that initially holds 200 chars */ + if ((s = sbuf_new(NULL, NULL, 200, SBUF_AUTOEXTEND)) == NULL) + return -1; + + if (action == DN_LOG_ENABLE) { + + /* create our alq */ + alq_open( &dn_alq, + dn_logfile, + curthread->td_ucred, + DN_LOG_FILE_MODE, + DN_ALQ_BUFLEN, + 0 + ); + + STAILQ_INIT(&log_queue); + + dn_exit_log_manager_thread = 0; + + ret = kthread_create( &dn_log_manager_thread, + NULL, + &dn_log_manager_proc, + RFNOWAIT, + 0, + "dn_log_manager_thr" + ); + dn_log_manager_thr = + FIRST_THREAD_IN_PROC(dn_log_manager_proc); + + microtime(&tval); + + sbuf_printf(s, + "enable_time_secs=%ld\tenable_time_usecs=%06ld\thz=%d\tsysname=%s\tsysver=%u\n", + tval.tv_sec, + tval.tv_usec, + hz, + "FreeBSD", + __FreeBSD_version + ); + + sbuf_finish(s); + alq_writen(dn_alq, sbuf_data(s), sbuf_len(s), ALQ_WAITOK); + } + else if (action == DN_LOG_DISABLE && dn_log_manager_thr != NULL) { + + /* tell the log manager thread that it should exit now */ + dn_exit_log_manager_thread = 1; + + /* + * wake the pkt_manager thread so it realises that + * dn_exit_log_manager_thread = 1 and exits gracefully + */ + wakeup(&wait_for_log); + + /* wait for the pkt_manager thread to exit */ + tsleep(dn_log_manager_thr, PWAIT, "thrwait", 0); + + dn_log_manager_thr = NULL; + + microtime(&tval); + + sbuf_printf(s, + "disable_time_secs=%ld\tdisable_time_usecs=%06ld", + tval.tv_sec, + tval.tv_usec + ); + + sbuf_printf(s, "\n"); + sbuf_finish(s); + alq_writen(dn_alq, sbuf_data(s), sbuf_len(s), ALQ_WAITOK); + alq_close(dn_alq); + dn_alq = NULL; + } + + sbuf_delete(s); + + /* + * XXX: Should be using ret to check if any functions fail + * and set error appropriately + */ + return error; +} + +static int +dn_sysctl_log_enable_handler(SYSCTL_HANDLER_ARGS) +{ + if (!req->newptr) + goto skip; + + /* if the value passed in isn't DISABLE or ENABLE, return an error */ + if (CAST_PTR_INT(req->newptr) != DN_LOG_DISABLE && + CAST_PTR_INT(req->newptr) != DN_LOG_ENABLE) + return 1; + + /* if we are changing state (DISABLE to ENABLE or vice versa) */ + if (CAST_PTR_INT(req->newptr) != dn_log_enable ) + if (dn_manage_logging(CAST_PTR_INT(req->newptr))) { + dn_manage_logging(DN_LOG_DISABLE); + return 1; + } + +skip: + return sysctl_handle_int(oidp, arg1, arg2, req); +} + +static uint32_t +hash_pkt(struct mbuf *m, uint32_t offset) +{ + register uint32_t hash = 0; + + while ((m != NULL) && (offset > m->m_len)) { + /* + * the IP packet payload does not start in this mbuf + * need to figure out which mbuf it starts in and what offset + * into the mbuf's data region the payload starts at + */ + offset -= m->m_len; + m = m->m_next; + } + + while (m != NULL) { + /* ensure there is data in the mbuf */ + if ((m->m_len - offset) > 0) { + hash = hash32_buf( m->m_data + offset, + m->m_len - offset, + hash + ); + } + + m = m->m_next; + offset = 0; + } + + return hash; +} + +static void +dn_log( struct dn_flow_set *fs, + struct dn_pipe *p, + struct dn_flow_queue *q, + struct mbuf *pkt, + u_int dropped, + int dir) +{ + struct log_node *log_node; + + DUMMYNET_LOCK_ASSERT(); + + /* M_NOWAIT flag required here */ + log_node = malloc(sizeof(struct log_node), M_DUMMYNET, M_NOWAIT); + + if (log_node == NULL) + return; + + /* set log_node struct members */ + microtime(&(log_node->tval)); + log_node->direction = dir; + log_node->dropped = dropped; + log_node->ipver = INP_IPV4; + log_node->fs_num = (dropped == DN_DROP_NOFS) ? + -1 : fs->fs_nr; + log_node->fs_flags = (dropped == DN_DROP_NOFS) ? + 0 : fs->flags_fs; + log_node->q_max_len = (dropped == DN_DROP_NOFS) ? + -1 : fs->qsize; + log_node->p_num = (dropped == DN_DROP_NOFS || + dropped == DN_DROP_NOP4Q) ? + -1 : p->pipe_nr; + log_node->p_len = (dropped == DN_DROP_NOFS || + dropped == DN_DROP_NOP4Q) ? + -1 : p->len; + log_node->q_len_pkts = (dropped == DN_DROP_NOFS || + dropped == DN_DROP_NOQ) ? + -1 : q->len; + log_node->q_len_bytes = (dropped == DN_DROP_NOFS || + dropped == DN_DROP_NOQ) ? + -1 : q->len_bytes; + + /* + * calc a hash of the pkt which triggered this log message + * hash is calculated over the IP payload (not IP header) so as to + * be invariant to changes in the IP header + * XXX: Handle IPv6 + */ + struct ip *ip = mtod(pkt, struct ip *); + uint32_t ip_hl = (ip->ip_hl << 2); + + if (pkt->m_pkthdr.csum_flags & (CSUM_TCP | CSUM_UDP)) { + /* + * This is a TCP or UDP packet without its checksum field + * Manually calculate the checksum so that we generate a correct pkt hash + */ + if (pkt->m_pkthdr.csum_flags & CSUM_TCP) { + struct tcphdr *th = (struct tcphdr *)((caddr_t)ip + ip_hl); + th->th_sum = in_cksum_skip(pkt, ip->ip_len, ip_hl); + pkt->m_pkthdr.csum_flags &= ~CSUM_TCP; + } else { + struct udphdr *uh = (struct udphdr *)((caddr_t)ip + ip_hl); + uh->uh_sum = in_cksum_skip(pkt, ip->ip_len, ip_hl); + pkt->m_pkthdr.csum_flags &= ~CSUM_UDP; + } + } + + log_node->hash = hash_pkt(pkt, ip_hl); + + mtx_lock(&dn_log_queue_mtx); + STAILQ_INSERT_TAIL(&log_queue, log_node, nodes); + mtx_unlock(&dn_log_queue_mtx); +} + /* * Return the mbuf tag holding the dummynet state. As an optimization * this is assumed to be the first tag on the list. If this turns out @@ -504,6 +946,7 @@ transmit_event(struct dn_pipe *pipe, str else *head = m; *tail = m; + pipe->len--; } if (*tail != NULL) (*tail)->m_nextpkt = NULL; @@ -550,6 +993,7 @@ move_pkt(struct mbuf *pkt, struct dn_flo p->tail->m_nextpkt = pkt; p->tail = pkt; p->tail->m_nextpkt = NULL; + p->len++; } /* @@ -1295,34 +1739,43 @@ dummynet_io(struct mbuf **m0, int dir, s } else fs = locate_flowset(fwa->cookie); - if (fs == NULL) + if (fs == NULL) { + DN_LOG(NULL, NULL, NULL, m, DN_DROP_NOFS, dir); goto dropit; /* This queue/pipe does not exist! */ + } pipe = fs->pipe; if (pipe == NULL) { /* Must be a queue, try find a matching pipe. */ pipe = locate_pipe(fs->parent_nr); if (pipe != NULL) fs->pipe = pipe; else { + DN_LOG(fs, NULL, NULL, m, DN_DROP_NOP4Q, dir); printf("dummynet: no pipe %d for queue %d, drop pkt\n", fs->parent_nr, fs->fs_nr); goto dropit; } } q = find_queue(fs, &(fwa->f_id)); - if (q == NULL) + if (q == NULL) { + DN_LOG(fs, pipe, NULL, m, DN_DROP_NOQ, dir); goto dropit; /* Cannot allocate queue. */ + } /* Update statistics, then check reasons to drop pkt. */ q->tot_bytes += len; q->tot_pkts++; - if (fs->plr && random() < fs->plr) + if (fs->plr && random() < fs->plr) { + DN_LOG(fs, pipe, q, m, DN_DROP_PLR, dir); goto dropit; /* Random pkt drop. */ + } else { while (fs->pls_index < fs->pls.count) { if (q->tot_pkts >= fs->pls.arr[fs->pls_index].start) { if (q->tot_pkts <= - fs->pls.arr[fs->pls_index].end) + fs->pls.arr[fs->pls_index].end) { + DN_LOG(fs, pipe, q, m, DN_DROP_PLS, dir); goto dropit; /* Controlled pkt drop. */ + } else fs->pls_index++; } @@ -1335,20 +1788,28 @@ dummynet_io(struct mbuf **m0, int dir, s } } if (fs->flags_fs & DN_QSIZE_IS_BYTES) { - if (q->len_bytes > fs->qsize) + if (q->len_bytes > fs->qsize) { + DN_LOG(fs, pipe, q, m, DN_DROP_QOVERFLOW, dir); goto dropit; /* Queue size overflow. */ + } } else { - if (q->len >= fs->qsize) + if (q->len >= fs->qsize) { + DN_LOG(fs, pipe, q, m, DN_DROP_QOVERFLOW, dir); goto dropit; /* Queue count overflow. */ + } } - if (fs->flags_fs & DN_IS_RED && red_drops(fs, q, len)) + if (fs->flags_fs & DN_IS_RED && red_drops(fs, q, len)) { + DN_LOG(fs, pipe, q, m, DN_DROP_RED, dir); goto dropit; + } /* XXX expensive to zero, see if we can remove it. */ mtag = m_tag_get(PACKET_TAG_DUMMYNET, sizeof(struct dn_pkt_tag), M_NOWAIT | M_ZERO); - if (mtag == NULL) + if (mtag == NULL) { + DN_LOG(fs, pipe, q, m, DN_DROP_MALLOC, dir); goto dropit; /* Cannot allocate packet header. */ + } m_tag_prepend(m, mtag); /* Attach to mbuf chain. */ pkt = (struct dn_pkt_tag *)(mtag + 1); @@ -1369,6 +1830,8 @@ dummynet_io(struct mbuf **m0, int dir, s q->len++; q->len_bytes += len; + DN_LOG(fs, pipe, q, m, DN_NO_DROP, dir); + if (q->head != m) /* Flow was not idle, we are done. */ goto done; @@ -2301,6 +2764,7 @@ ip_dn_init(void) printf("DUMMYNET with IPv6 initialized (040826)\n"); DUMMYNET_LOCK_INIT(); + mtx_init(&dn_log_queue_mtx, "dummynet__queue_mtx", NULL, MTX_DEF); for (i = 0; i < HASHSIZE; i++) { SLIST_INIT(&pipehash[i]); @@ -2339,6 +2803,7 @@ ip_dn_destroy(void) ip_dn_io_ptr = NULL; ip_dn_ruledel_ptr = NULL; + dn_manage_logging(DN_LOG_DISABLE); DUMMYNET_LOCK(); callout_stop(&dn_timeout); DUMMYNET_UNLOCK(); @@ -2386,4 +2851,5 @@ static moduledata_t dummynet_mod = { }; DECLARE_MODULE(dummynet, dummynet_mod, SI_SUB_PROTO_IFATTACHDOMAIN, SI_ORDER_ANY); MODULE_DEPEND(dummynet, ipfw, 2, 2, 2); +MODULE_DEPEND(idummynet, alq, 1, 1, 1); MODULE_VERSION(dummynet, 1); Modified: user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.h ============================================================================== --- user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.h Wed Nov 19 03:24:35 2008 (r185082) +++ user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.h Wed Nov 19 03:57:11 2008 (r185083) @@ -30,6 +30,33 @@ #ifndef _IP_DUMMYNET_H #define _IP_DUMMYNET_H +/* Packet was not dropped by dummynet */ +#define DN_NO_DROP 0 + +/* Packet was dropped because there was no corresponding flow set */ +#define DN_DROP_NOFS 1 + +/* Packet was dropped because there was no corresponding pipe for the queue */ +#define DN_DROP_NOP4Q 2 + +/* Packet was dropped because we could not allocate a queue for the packet */ +#define DN_DROP_NOQ 3 + +/* Packet was dropped because PLR was set and this packet won the lucky dip */ +#define DN_DROP_PLR 4 + +/* Packet was dropped because PLS was set and this packet was selected */ +#define DN_DROP_PLS 5 + +/* Packet was dropped because the queue it was destined for is full */ +#define DN_DROP_QOVERFLOW 6 + +/* Packet was dropped because RED was configured on the queue */ +#define DN_DROP_RED 7 + +/* Packet was dropped because of a malloc failure */ +#define DN_DROP_MALLOC 8 + /* * Definition of dummynet data structures. In the structures, I decided * not to use the macros in in the hope of making the code @@ -328,6 +355,7 @@ struct dn_pipe { /* a pipe */ int bandwidth; /* really, bytes/tick. */ int delay ; /* really, ticks */ + u_int len; /* number of pkts in delay line */ struct mbuf *head, *tail ; /* packets in delay line */ /* WF2Q+ */ From owner-svn-src-user@FreeBSD.ORG Wed Nov 19 04:02:08 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E67C1065673; Wed, 19 Nov 2008 04:02:08 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 065128FC12; Wed, 19 Nov 2008 04:02:08 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAJ4276v003431; Wed, 19 Nov 2008 04:02:07 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAJ427Is003430; Wed, 19 Nov 2008 04:02:07 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811190402.mAJ427Is003430@svn.freebsd.org> From: Lawrence Stewart Date: Wed, 19 Nov 2008 04:02:07 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185084 - user/lstewart/dummynet_8.x/sys/netinet X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Nov 2008 04:02:08 -0000 Author: lstewart Date: Wed Nov 19 04:02:07 2008 New Revision: 185084 URL: http://svn.freebsd.org/changeset/base/185084 Log: Correct a typo I just spotted. Modified: user/lstewart/dummynet_8.x/sys/netinet/ip_dummynet.c Modified: user/lstewart/dummynet_8.x/sys/netinet/ip_dummynet.c ============================================================================== --- user/lstewart/dummynet_8.x/sys/netinet/ip_dummynet.c Wed Nov 19 03:57:11 2008 (r185083) +++ user/lstewart/dummynet_8.x/sys/netinet/ip_dummynet.c Wed Nov 19 04:02:07 2008 (r185084) @@ -2848,5 +2848,5 @@ static moduledata_t dummynet_mod = { }; DECLARE_MODULE(dummynet, dummynet_mod, SI_SUB_PROTO_IFATTACHDOMAIN, SI_ORDER_ANY); MODULE_DEPEND(dummynet, ipfw, 2, 2, 2); -MODULE_DEPEND(idummynet, alq, 1, 1, 1); +MODULE_DEPEND(dummynet, alq, 1, 1, 1); MODULE_VERSION(dummynet, 1); From owner-svn-src-user@FreeBSD.ORG Wed Nov 19 04:05:45 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A1EB61065675; Wed, 19 Nov 2008 04:05:45 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 8970B8FC08; Wed, 19 Nov 2008 04:05:45 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAJ45juH003540; Wed, 19 Nov 2008 04:05:45 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAJ45jFh003539; Wed, 19 Nov 2008 04:05:45 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811190405.mAJ45jFh003539@svn.freebsd.org> From: Lawrence Stewart Date: Wed, 19 Nov 2008 04:05:45 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185085 - in user/lstewart/dummynet_7.x/sys: . netinet X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Nov 2008 04:05:45 -0000 Author: lstewart Date: Wed Nov 19 04:05:44 2008 New Revision: 185085 URL: http://svn.freebsd.org/changeset/base/185085 Log: Merge r185084 from dummynet_8.x Modified: user/lstewart/dummynet_7.x/sys/ (props changed) user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.c Modified: user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.c ============================================================================== --- user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.c Wed Nov 19 04:02:07 2008 (r185084) +++ user/lstewart/dummynet_7.x/sys/netinet/ip_dummynet.c Wed Nov 19 04:05:44 2008 (r185085) @@ -2851,5 +2851,5 @@ static moduledata_t dummynet_mod = { }; DECLARE_MODULE(dummynet, dummynet_mod, SI_SUB_PROTO_IFATTACHDOMAIN, SI_ORDER_ANY); MODULE_DEPEND(dummynet, ipfw, 2, 2, 2); -MODULE_DEPEND(idummynet, alq, 1, 1, 1); +MODULE_DEPEND(dummynet, alq, 1, 1, 1); MODULE_VERSION(dummynet, 1); From owner-svn-src-user@FreeBSD.ORG Thu Nov 20 01:09:00 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ED344106564A; Thu, 20 Nov 2008 01:09:00 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id C51C78FC08; Thu, 20 Nov 2008 01:09:00 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAK190AX033801; Thu, 20 Nov 2008 01:09:00 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAK1909N033800; Thu, 20 Nov 2008 01:09:00 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811200109.mAK1909N033800@svn.freebsd.org> From: Lawrence Stewart Date: Thu, 20 Nov 2008 01:09:00 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185111 - user/lstewart/misc_7.x X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Nov 2008 01:09:01 -0000 Author: lstewart Date: Thu Nov 20 01:09:00 2008 New Revision: 185111 URL: http://svn.freebsd.org/changeset/base/185111 Log: Somewhere for small, random, 7.x related patch development Added: user/lstewart/misc_7.x/ - copied from r185110, stable/7/ From owner-svn-src-user@FreeBSD.ORG Thu Nov 20 01:11:35 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 33BFA1065672; Thu, 20 Nov 2008 01:11:35 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 0B8C38FC0A; Thu, 20 Nov 2008 01:11:35 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAK1BYwU033912; Thu, 20 Nov 2008 01:11:34 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAK1BYQS033911; Thu, 20 Nov 2008 01:11:34 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811200111.mAK1BYQS033911@svn.freebsd.org> From: Lawrence Stewart Date: Thu, 20 Nov 2008 01:11:34 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185112 - user/lstewart/misc_8.x X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Nov 2008 01:11:35 -0000 Author: lstewart Date: Thu Nov 20 01:11:34 2008 New Revision: 185112 URL: http://svn.freebsd.org/changeset/base/185112 Log: Somewhere for small, random, 8.x related patch development Added: user/lstewart/misc_8.x/ - copied from r185111, head/ From owner-svn-src-user@FreeBSD.ORG Thu Nov 20 02:06:05 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7C135106564A; Thu, 20 Nov 2008 02:06:05 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 6CE8B8FC12; Thu, 20 Nov 2008 02:06:05 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAK265hs035098; Thu, 20 Nov 2008 02:06:05 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAK265QH035097; Thu, 20 Nov 2008 02:06:05 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811200206.mAK265QH035097@svn.freebsd.org> From: Kip Macy Date: Thu, 20 Nov 2008 02:06:05 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185113 - user/kmacy/HEAD_fast_multi_xmit/sys/kern X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Nov 2008 02:06:05 -0000 Author: kmacy Date: Thu Nov 20 02:06:05 2008 New Revision: 185113 URL: http://svn.freebsd.org/changeset/base/185113 Log: Don't use knlist_mtx_lock routine Modified: user/kmacy/HEAD_fast_multi_xmit/sys/kern/kern_event.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/kern/kern_event.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/kern/kern_event.c Thu Nov 20 01:11:34 2008 (r185112) +++ user/kmacy/HEAD_fast_multi_xmit/sys/kern/kern_event.c Thu Nov 20 02:06:05 2008 (r185113) @@ -86,6 +86,10 @@ MTX_SYSINIT(kq_global, &kq_global, "kque TASKQUEUE_DEFINE_THREAD(kqueue); +static void knlist_mtx_lock(void *arg); +static void knlist_mtx_unlock(void *arg); +static int knlist_mtx_locked(void *arg); + static int kevent_copyout(void *arg, struct kevent *kevp, int count); static int kevent_copyin(void *arg, struct kevent *kevp, int count); static int kqueue_register(struct kqueue *kq, struct kevent *kev, @@ -193,8 +197,14 @@ SYSCTL_INT(_kern, OID_AUTO, kq_calloutma mtx_assert(&(kq)->kq_lock, MA_NOTOWNED); \ } while (0) #define KN_LIST_LOCK(kn) do { \ - if (kn->kn_knlist != NULL) \ - kn->kn_knlist->kl_lock(kn->kn_knlist->kl_lockarg); \ + struct knlist *list = kn->kn_knlist; \ + \ + if (list != NULL){ \ + if (list->kl_lock != knlist_mtx_lock) \ + list->kl_lock(list->kl_lockarg); \ + else \ + mtx_lock(list->kl_lockarg); \ + } \ } while (0) #define KN_LIST_UNLOCK(kn) do { \ if (kn->kn_knlist != NULL) \ @@ -424,8 +434,12 @@ knote_fork(struct knlist *list, int pid) if (list == NULL) return; - list->kl_lock(list->kl_lockarg); - + + if (list->kl_lock != knlist_mtx_lock) + list->kl_lock(list->kl_lockarg); + else + mtx_lock(list->kl_lockarg); + SLIST_FOREACH(kn, &list->kl_list, kn_selnext) { if ((kn->kn_status & KN_INFLUX) == KN_INFLUX) continue; @@ -476,7 +490,10 @@ knote_fork(struct knlist *list, int pid) KQ_LOCK(kq); kn->kn_status &= ~KN_INFLUX; KQ_UNLOCK_FLUX(kq); - list->kl_lock(list->kl_lockarg); + if (list->kl_lock != knlist_mtx_lock) + list->kl_lock(list->kl_lockarg); + else + mtx_lock(list->kl_lockarg); } list->kl_unlock(list->kl_lockarg); } @@ -1618,8 +1635,12 @@ knote(struct knlist *list, long hint, in KNL_ASSERT_LOCK(list, islocked); - if (!islocked) - list->kl_lock(list->kl_lockarg); + if (!islocked){ + if (list->kl_lock != knlist_mtx_lock) + list->kl_lock(list->kl_lockarg); + else + mtx_lock(list->kl_lockarg); + } /* * If we unlock the list lock (and set KN_INFLUX), we can eliminate @@ -1657,8 +1678,14 @@ knlist_add(struct knlist *knl, struct kn KQ_NOTOWNED(kn->kn_kq); KASSERT((kn->kn_status & (KN_INFLUX|KN_DETACHED)) == (KN_INFLUX|KN_DETACHED), ("knote not KN_INFLUX and KN_DETACHED")); - if (!islocked) - knl->kl_lock(knl->kl_lockarg); + if (!islocked) { + struct knlist *list = kn->kn_knlist; + + if (list->kl_lock != knlist_mtx_lock) + list->kl_lock(list->kl_lockarg); + else + mtx_lock(list->kl_lockarg); + } SLIST_INSERT_HEAD(&knl->kl_list, kn, kn_selnext); if (!islocked) knl->kl_unlock(knl->kl_lockarg); @@ -1677,8 +1704,15 @@ knlist_remove_kq(struct knlist *knl, str if (!kqislocked) KASSERT((kn->kn_status & (KN_INFLUX|KN_DETACHED)) == KN_INFLUX, ("knlist_remove called w/o knote being KN_INFLUX or already removed")); - if (!knlislocked) - knl->kl_lock(knl->kl_lockarg); + if (!knlislocked) { + struct knlist *list = kn->kn_knlist; + + if (list->kl_lock != knlist_mtx_lock) + list->kl_lock(list->kl_lockarg); + else + mtx_lock(list->kl_lockarg); + } + SLIST_REMOVE(&knl->kl_list, kn, knote, kn_selnext); kn->kn_knlist = NULL; if (!knlislocked) @@ -1721,9 +1755,6 @@ knlist_empty(struct knlist *knl) static struct mtx knlist_lock; MTX_SYSINIT(knlist_lock, &knlist_lock, "knlist lock for lockless objects", MTX_DEF); -static void knlist_mtx_lock(void *arg); -static void knlist_mtx_unlock(void *arg); -static int knlist_mtx_locked(void *arg); static void knlist_mtx_lock(void *arg) @@ -1752,11 +1783,12 @@ knlist_init(struct knlist *knl, void *lo knl->kl_lockarg = &knlist_lock; else knl->kl_lockarg = lock; - + if (kl_lock == NULL) knl->kl_lock = knlist_mtx_lock; else knl->kl_lock = kl_lock; + if (kl_unlock == NULL) knl->kl_unlock = knlist_mtx_unlock; else @@ -1801,7 +1833,11 @@ knlist_cleardel(struct knlist *knl, stru else { KNL_ASSERT_UNLOCKED(knl); again: /* need to reacquire lock since we have dropped it */ - knl->kl_lock(knl->kl_lockarg); + + if (knl->kl_lock != knlist_mtx_lock) + knl->kl_lock(knl->kl_lockarg); + else + mtx_lock(knl->kl_lockarg); } SLIST_FOREACH_SAFE(kn, &knl->kl_list, kn_selnext, kn2) { From owner-svn-src-user@FreeBSD.ORG Thu Nov 20 04:04:22 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C0BF11065674; Thu, 20 Nov 2008 04:04:22 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id B2CD18FC16; Thu, 20 Nov 2008 04:04:22 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAK44Mlk038538; Thu, 20 Nov 2008 04:04:22 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAK44M1n038537; Thu, 20 Nov 2008 04:04:22 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811200404.mAK44M1n038537@svn.freebsd.org> From: Lawrence Stewart Date: Thu, 20 Nov 2008 04:04:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185116 - user/lstewart/misc_7.x/sys/kern X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Nov 2008 04:04:22 -0000 Author: lstewart Date: Thu Nov 20 04:04:22 2008 New Revision: 185116 URL: http://svn.freebsd.org/changeset/base/185116 Log: Fix a bug in kthread_exit() which led to potential deadlock in code that was relying on the documented behaviour in the kthread(9) man page. This brings the wakeup() related behaviour of kthread_exit in 7.x inline with 8.x. I've tested the fix with a simple scenario that was previously deadlocking. A waiting thread waits on a terminating thread by calling mtx_sleep() with the "chan" argument set to the *thread obtained by calling FIRST_THREAD_IN_PROC(*proc_that_will_exit). Without this patch, the waiting thread never gets woken because kthread_exit() calls wakeup() on the *proc. With this patch, the wait thread gets woken and all is good in the world again. For more details, refer to: http://lists.freebsd.org/pipermail/freebsd-arch/2008-November/008688.html Discussed with: attilio@, julian@ Modified: user/lstewart/misc_7.x/sys/kern/kern_kthread.c Modified: user/lstewart/misc_7.x/sys/kern/kern_kthread.c ============================================================================== --- user/lstewart/misc_7.x/sys/kern/kern_kthread.c Thu Nov 20 03:34:36 2008 (r185115) +++ user/lstewart/misc_7.x/sys/kern/kern_kthread.c Thu Nov 20 04:04:22 2008 (r185116) @@ -144,6 +144,7 @@ kthread_exit(int ecode) * Wakeup anyone waiting for us to exit. */ wakeup(p); + wakeup(td); /* Buh-bye! */ exit1(td, W_EXITCODE(ecode, 0)); From owner-svn-src-user@FreeBSD.ORG Thu Nov 20 04:11:22 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 127ED1065672; Thu, 20 Nov 2008 04:11:22 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 047BC8FC1C; Thu, 20 Nov 2008 04:11:22 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAK4BLGD038732; Thu, 20 Nov 2008 04:11:21 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAK4BLIq038730; Thu, 20 Nov 2008 04:11:21 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811200411.mAK4BLIq038730@svn.freebsd.org> From: Lawrence Stewart Date: Thu, 20 Nov 2008 04:11:21 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185117 - user/lstewart/misc_8.x/share/man/man9 X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Nov 2008 04:11:22 -0000 Author: lstewart Date: Thu Nov 20 04:11:21 2008 New Revision: 185117 URL: http://svn.freebsd.org/changeset/base/185117 Log: In light of recent discussions regarding the kthread(9)/kproc(9) KPI, clairfy the behaviour of kthread_exit(). Whilst we're here, remove some bogus man page references. Modified: user/lstewart/misc_8.x/share/man/man9/kproc.9 user/lstewart/misc_8.x/share/man/man9/kthread.9 Modified: user/lstewart/misc_8.x/share/man/man9/kproc.9 ============================================================================== --- user/lstewart/misc_8.x/share/man/man9/kproc.9 Thu Nov 20 04:04:22 2008 (r185116) +++ user/lstewart/misc_8.x/share/man/man9/kproc.9 Thu Nov 20 04:11:21 2008 (r185117) @@ -157,11 +157,17 @@ letting the main function return to its The .Fa ecode argument specifies the exit status of the process. -While exiting, the function -.Xr exit1 9 -will initiate a call to +While exiting, the function will call .Xr wakeup 9 -on the process handle. +on the calling process's handle. +If another thread/process needs to wait for the calling process to exit, +.Xr mtx_sleep 9 +can be used in the waiting thread/process with the +.Fa chan +argument set to the exiting process's handle. +This will put the waiting thread/process to sleep until the exiting process +has made its final call to +.Fn kproc_exit . .Pp The .Fn kproc_resume , @@ -332,9 +338,8 @@ parameter. .El .Sh SEE ALSO .Xr rfork 2 , -.Xr exit1 9 , .Xr kthread 9 , -.Xr SYSINIT 9 , +.Xr mtx_sleep 9 , .Xr wakeup 9 .Sh HISTORY The Modified: user/lstewart/misc_8.x/share/man/man9/kthread.9 ============================================================================== --- user/lstewart/misc_8.x/share/man/man9/kthread.9 Thu Nov 20 04:04:22 2008 (r185116) +++ user/lstewart/misc_8.x/share/man/man9/kthread.9 Thu Nov 20 04:11:21 2008 (r185117) @@ -171,7 +171,17 @@ The function is used to terminate kernel threads. It should be called by the main function of the kernel thread rather than letting the main function return to its caller. -.\" XXX "int ecode" argument isn't documented. +While exiting, the function will call +.Xr wakeup 9 +on the calling thread's handle. +If another thread/process needs to wait for the calling thread to exit, +.Xr mtx_sleep 9 +can be used in the waiting thread/process with the +.Fa chan +argument set to the exiting thread's handle. +This will put the waiting thread/process to sleep until the exiting thread +has made its final call to +.Fn kthread_exit . .Pp The .Fn kthread_resume , @@ -290,9 +300,8 @@ parameter. .El .Sh SEE ALSO .Xr rfork 2 , -.Xr exit1 9 , .Xr kproc 9 , -.Xr SYSINIT 9 , +.Xr mtx_sleep 9 , .Xr wakeup 9 .Sh HISTORY The From owner-svn-src-user@FreeBSD.ORG Thu Nov 20 04:23:18 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B00FC106564A; Thu, 20 Nov 2008 04:23:18 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id A1F8E8FC16; Thu, 20 Nov 2008 04:23:18 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAK4NISm039040; Thu, 20 Nov 2008 04:23:18 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAK4NIZV039039; Thu, 20 Nov 2008 04:23:18 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811200423.mAK4NIZV039039@svn.freebsd.org> From: Kip Macy Date: Thu, 20 Nov 2008 04:23:18 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185118 - user/kmacy/HEAD_fast_multi_xmit/sys/dev/hwpmc X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Nov 2008 04:23:18 -0000 Author: kmacy Date: Thu Nov 20 04:23:18 2008 New Revision: 185118 URL: http://svn.freebsd.org/changeset/base/185118 Log: be more aggressive about disabling events Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/hwpmc/hwpmc_ipm.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/hwpmc/hwpmc_ipm.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/hwpmc/hwpmc_ipm.c Thu Nov 20 04:11:21 2008 (r185117) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/hwpmc/hwpmc_ipm.c Thu Nov 20 04:23:18 2008 (r185118) @@ -1325,7 +1325,8 @@ ipm_stop_pmc(int cpu, int ri) static int ipm_v1_intr(int cpu, struct trapframe *tf) { - int i, error, retval, ri; + int i, error, did_interrupt, ri; + uint32_t ipm_evsel; struct pmc *pm; struct ipm_cpu *pc; struct pmc_hw *phw; @@ -1334,66 +1335,65 @@ ipm_v1_intr(int cpu, struct trapframe *t KASSERT(cpu >= 0 && cpu < mp_ncpus, ("[ipm-v1,%d] CPU %d out of range", __LINE__, cpu)); - retval = 0; + did_interrupt = 0; pc = (struct ipm_cpu *) pmc_pcpu[cpu]; for (i = 0; i < IPM_NPMCS_V1 - 1; i++) { + error = 0; ri = i + 1; phw = pc->pc_common.pc_hwpmcs[ri]; + pm = phw->phw_pmc; - if ((pm = phw->phw_pmc) == NULL || + if (pm) + ipm_evsel = pm->pm_md.pm_ipm.pm_ipm_evsel; + else + ipm_evsel = rdmsr(IPM_MSR_EVSEL0 + i); + did_interrupt = 1; + + /* + * Stop the PMC + */ + wrmsr(IPM_MSR_EVSEL0 + i, ipm_evsel & (~IPM_EVSEL_EN)); + + if (pm == NULL || pm->pm_state != PMC_STATE_RUNNING || !PMC_IS_SAMPLING_MODE(PMC_TO_MODE(pm)) || IPM_PMC_STOPPED(pc, ri)) { continue; } - if (!IPM_PMC_HAS_OVERFLOWED(i)) - continue; - /* - * Stop the PMC - */ - wrmsr(IPM_MSR_EVSEL0 + i, - pm->pm_md.pm_ipm.pm_ipm_evsel & (~IPM_EVSEL_EN)); - - retval = 1; - - error = pmc_process_interrupt(cpu, pm, tf, - TRAPF_USERMODE(tf)); - if (error) - IPM_PMC_MARK_STOPPED(pc,ri); - + if (IPM_PMC_HAS_OVERFLOWED(i)) + error = pmc_process_interrupt(cpu, pm, tf, + TRAPF_USERMODE(tf)); + /* reload sampling count */ v = pm->pm_sc.pm_reloadcount; wrmsr(IPM_MSR_PERFCTR0 + i, IPM_RELOAD_COUNT_TO_PERFCTR_VALUE(v)); + + if (pm->pm_state != PMC_STATE_RUNNING || + !PMC_IS_SAMPLING_MODE(PMC_TO_MODE(pm)) || + IPM_PMC_STOPPED(pc, ri)) { + continue; + } + if (error) + IPM_PMC_MARK_STOPPED(pc,ri); + else + wrmsr(IPM_MSR_EVSEL0 + i, + ipm_evsel | IPM_EVSEL_EN); } /* * The LAPIC needs to have its PMC interrupt * unmasked after a PMC interrupt. */ - if (retval) + if (did_interrupt) pmc_x86_lapic_enable_pmc_interrupt(); - atomic_add_int(retval ? &pmc_stats.pm_intr_processed : + atomic_add_int(did_interrupt ? &pmc_stats.pm_intr_processed : &pmc_stats.pm_intr_ignored, 1); - /* restart counters */ - for (i = 0; i < IPM_NPMCS_V1 - 1; i++) { - ri = i + 1; - phw = pc->pc_common.pc_hwpmcs[ri]; - if ((pm = phw->phw_pmc) == NULL || - pm->pm_state != PMC_STATE_RUNNING || - !PMC_IS_SAMPLING_MODE(PMC_TO_MODE(pm)) || - IPM_PMC_STOPPED(pc, ri)) { - continue; - } - wrmsr(IPM_MSR_EVSEL0 + i, - pm->pm_md.pm_ipm.pm_ipm_evsel | IPM_EVSEL_EN); - } - - return (retval); + return (did_interrupt); } static int @@ -1427,6 +1427,11 @@ ipm_v2_intr(int cpu, struct trapframe *t phw = pc->pc_common.pc_hwpmcs[ri]; + if (!(ipm_v2_pmc_has_overflowed(ovf_status, ri))) { + continue; + } + retval = 1; + if ((pm = phw->phw_pmc) == NULL || pm->pm_state != PMC_STATE_RUNNING || !PMC_IS_SAMPLING_MODE(PMC_TO_MODE(pm)) || @@ -1434,12 +1439,6 @@ ipm_v2_intr(int cpu, struct trapframe *t continue; } - if (!(ipm_v2_pmc_has_overflowed(ovf_status, ri))) { - continue; - } - - retval = 1; - error = pmc_process_interrupt(cpu, pm, tf, TRAPF_USERMODE(tf)); if (error) @@ -1582,8 +1581,11 @@ pmc_initialize_ipm(struct pmc_mdep *pmc_ pmc_mdep->pmd_describe = ipm_describe; pmc_mdep->pmd_get_msr = ipm_get_msr; /* i386 */ if (ipm_cputype == PMC_CPU_INTEL_CORE) { + printf("using v1 intr\n"); + pmc_mdep->pmd_intr = ipm_v1_intr; } else { + printf("using v2 intr\n"); pmc_mdep->pmd_intr = ipm_v2_intr; } From owner-svn-src-user@FreeBSD.ORG Thu Nov 20 04:27:44 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0108A1065673; Thu, 20 Nov 2008 04:27:44 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id E76008FC1D; Thu, 20 Nov 2008 04:27:43 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAK4Rh5d039164; Thu, 20 Nov 2008 04:27:43 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAK4Rhbc039163; Thu, 20 Nov 2008 04:27:43 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811200427.mAK4Rhbc039163@svn.freebsd.org> From: Lawrence Stewart Date: Thu, 20 Nov 2008 04:27:43 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185119 - user/lstewart/misc_7.x/share/man/man9 X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Nov 2008 04:27:44 -0000 Author: lstewart Date: Thu Nov 20 04:27:43 2008 New Revision: 185119 URL: http://svn.freebsd.org/changeset/base/185119 Log: Merge r185117 kthread(9) man page changes from misc_8.x Modified: user/lstewart/misc_7.x/share/man/man9/ (props changed) user/lstewart/misc_7.x/share/man/man9/kthread.9 Modified: user/lstewart/misc_7.x/share/man/man9/kthread.9 ============================================================================== --- user/lstewart/misc_7.x/share/man/man9/kthread.9 Thu Nov 20 04:23:18 2008 (r185118) +++ user/lstewart/misc_7.x/share/man/man9/kthread.9 Thu Nov 20 04:27:43 2008 (r185119) @@ -141,11 +141,17 @@ letting the main function return to its The .Fa ecode argument specifies the exit status of the thread. -While exiting, the function -.Xr exit1 9 -will initiate a call to +While exiting, the function will call .Xr wakeup 9 -on the thread handle. +on the calling thread's handle. +If another thread/process needs to wait for the calling thread to exit, +.Xr mtx_sleep 9 +can be used in the waiting thread/process with the +.Fa chan +argument set to the exiting thread's handle. +This will put the waiting thread/process to sleep until the exiting thread +has made its final call to +.Fn kthread_exit . .Pp The .Fn kthread_resume , @@ -264,8 +270,7 @@ parameter. .El .Sh SEE ALSO .Xr rfork 2 , -.Xr exit1 9 , -.Xr SYSINIT 9 , +.Xr mtx_sleep 9 , .Xr wakeup 9 .Sh HISTORY The From owner-svn-src-user@FreeBSD.ORG Thu Nov 20 04:46:09 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6DDD2106564A; Thu, 20 Nov 2008 04:46:09 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 608D68FC08; Thu, 20 Nov 2008 04:46:09 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAK4k9iC039611; Thu, 20 Nov 2008 04:46:09 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAK4k9UC039610; Thu, 20 Nov 2008 04:46:09 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811200446.mAK4k9UC039610@svn.freebsd.org> From: Kip Macy Date: Thu, 20 Nov 2008 04:46:09 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185120 - user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Nov 2008 04:46:09 -0000 Author: kmacy Date: Thu Nov 20 04:46:09 2008 New Revision: 185120 URL: http://svn.freebsd.org/changeset/base/185120 Log: only scan with invariants Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Thu Nov 20 04:27:43 2008 (r185119) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Thu Nov 20 04:46:09 2008 (r185120) @@ -1880,8 +1880,10 @@ t3_free_tx_desc(struct sge_txq *q, int r bus_dmamap_unload(q->entry_tag, txsd->map); txsd->flags &= ~TX_SW_DESC_MAPPED; } - m_freem_iovec(&txsd->mi); + m_freem_iovec(&txsd->mi); +#ifdef INVARIANTS buf_ring_scan(&q->txq_mr, txsd->mi.mi_base, __FILE__, __LINE__); +#endif txsd->mi.mi_base = NULL; /* * XXX check for cache hit rate here From owner-svn-src-user@FreeBSD.ORG Thu Nov 20 10:47:21 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B9A441065672; Thu, 20 Nov 2008 10:47:21 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id A79888FC19; Thu, 20 Nov 2008 10:47:21 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAKAlLrC049288; Thu, 20 Nov 2008 10:47:21 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAKAlLij049286; Thu, 20 Nov 2008 10:47:21 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811201047.mAKAlLij049286@svn.freebsd.org> From: Kip Macy Date: Thu, 20 Nov 2008 10:47:21 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185126 - in user/kmacy/HEAD_fast_multi_xmit/sys: kern sys X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Nov 2008 10:47:21 -0000 Author: kmacy Date: Thu Nov 20 10:47:21 2008 New Revision: 185126 URL: http://svn.freebsd.org/changeset/base/185126 Log: Add lock-less ring buffer as primitive for replacing ifq Added: user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_bufr.c (contents, props changed) user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h (contents, props changed) Added: user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_bufr.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_bufr.c Thu Nov 20 10:47:21 2008 (r185126) @@ -0,0 +1,66 @@ +/************************************************************************** + * + * Copyright (c) 2007,2008 Kip Macy kmacy@freebsd.org + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * + * 2. The name of Kip Macy nor the names of other + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * + * + ***************************************************************************/ +#include +__FBSDID("$FreeBSD$"); + + +#include +#include +#include +#include +#include +#include + + +struct buf_ring * +buf_ring_alloc(int count, struct malloc_type *type, int flags) +{ + struct buf_ring *br; + + KASSERT(powerof2(count), ("buf ring must be size power of 2")); + + br = malloc(sizeof(struct buf_ring) + count*sizeof(caddr_t), + type, flags|M_ZERO); + if (br == NULL) + return (NULL); + + br->br_prod_size = br->br_cons_size = count; + br->br_prod_mask = br->br_cons_mask = count-1; + br->br_prod_head = br->br_cons_head = 0; + br->br_prod_tail = br->br_cons_tail = 0; + + return (br); +} + +void +buf_ring_free(struct buf_ring *br, struct malloc_type *type) +{ + free(br, type); +} Added: user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Thu Nov 20 10:47:21 2008 (r185126) @@ -0,0 +1,195 @@ +/************************************************************************** + * + * Copyright (c) 2007,2008 Kip Macy kmacy@freebsd.org + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * + * 2. The name of Kip Macy nor the names of other + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * + * $FreeBSD$ + * + ***************************************************************************/ + +#ifndef _SYS_BUF_RING_H_ +#define _SYS_BUF_RING_H_ + +#include + +#if defined (__GNUC__) + #if #cpu(i386) || defined __i386 || defined i386 || defined __i386__ || #cpu(x86_64) || defined __x86_64__ + #define mb() __asm__ __volatile__ ("mfence;": : :"memory") + #define wmb() __asm__ __volatile__ ("sfence;": : :"memory") + #define rmb() __asm__ __volatile__ ("lfence;": : :"memory") + #elif #cpu(sparc64) || defined sparc64 || defined __sparcv9 + #define mb() __asm__ __volatile__ ("membar #MemIssue": : :"memory") + #define wmb() mb() + #define rmb() mb() + #elif #cpu(sparc) || defined sparc || defined __sparc__ + #define mb() __asm__ __volatile__ ("stbar;": : :"memory") + #define wmb() mb() + #define rmb() mb() +#else + #define wmb() mb() + #define rmb() mb() + #define mb() /* XXX just to make this compile */ + #endif +#else + #error "unknown compiler" +#endif + +struct buf_ring { + volatile uint32_t br_prod_head; + volatile uint32_t br_prod_tail; + int br_prod_size; + int br_prod_mask; + /* + * Pad out to next L2 cache line + */ + uint64_t _pad0[14]; + + volatile uint32_t br_cons_head; + volatile uint32_t br_cons_tail; + int br_cons_size; + int br_cons_mask; + + /* + * Pad out to next L2 cache line + */ + uint64_t _pad1[14]; + void *br_ring[0]; +}; + + +static __inline int +buf_ring_enqueue(struct buf_ring *br, void *buf) +{ + uint32_t prod_head, prod_next; + uint32_t cons_tail; + int success; + + do { + prod_head = br->br_prod_head; + cons_tail = br->br_cons_tail; + + prod_next = (prod_head + 1) & br->br_prod_mask; + + if (prod_next == cons_tail) + return (ENOSPC); + + success = atomic_cmpset_int(&br->br_prod_head, prod_head, + prod_next); + } while (success == 0); + + KASSERT(br->br_ring[prod_head] == NULL, ("dangling value in enqueue")); + + br->br_ring[prod_head] = buf; + wmb(); + + /* + * If there are other enqueues in progress + * that preceeded us, we need to wait for them + * to complete - this is only a problem if they + * have been preempted + */ + while (br->br_prod_tail != prod_head) + cpu_spinwait(); + br->br_prod_tail = prod_next; + + return (0); +} + +static __inline void * +buf_ring_dequeue(struct buf_ring *br) +{ + uint32_t cons_head, cons_next; + uint32_t prod_tail; + void *buf; + int success; + + do { + cons_head = br->br_cons_head; + prod_tail = br->br_prod_tail; + + cons_next = (cons_head + 1) & br->br_cons_mask; + + if (cons_head == prod_tail) + return (NULL); + + success = atomic_cmpset_int(&br->br_cons_head, cons_head, + cons_next); + } while (success == 0); + + buf = br->br_ring[cons_head]; +#ifdef INVARIANTS + br->br_ring[cons_head] = NULL; +#endif + mb(); + + /* + * If there are other dequeues in progress + * that preceeded us, we need to wait for them + * to complete - this is only a problem if they + * have been preempted + */ + while (br->br_cons_tail != cons_head) + cpu_spinwait(); + + br->br_cons_tail = cons_next; + + return (buf); +} + +static __inline void * +buf_ring_peek(struct buf_ring *br) +{ + + return (br->br_ring[br->br_cons_tail]); +} + +static __inline int +buf_ring_full(struct buf_ring *br) +{ + + return (((br->br_prod_head + 1) & br->br_prod_mask) == br->br_cons_tail); +} + +static __inline int +buf_ring_empty(struct buf_ring *br) +{ + + return (br->br_cons_head == br->br_prod_tail); +} + +static __inline int +buf_ring_count(struct buf_ring *br) +{ + + return ((br->br_prod_size + br->br_prod_tail - br->br_cons_tail) + & br->br_prod_mask); +} + +struct buf_ring *buf_ring_alloc(int count, struct malloc_type *type, int flags); +void buf_ring_free(struct buf_ring *br, struct malloc_type *type); + + + +#endif From owner-svn-src-user@FreeBSD.ORG Thu Nov 20 10:48:01 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F1D1C106567C; Thu, 20 Nov 2008 10:48:01 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id DFEB18FC13; Thu, 20 Nov 2008 10:48:01 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAKAm1Io049346; Thu, 20 Nov 2008 10:48:01 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAKAm1ra049339; Thu, 20 Nov 2008 10:48:01 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811201048.mAKAm1ra049339@svn.freebsd.org> From: Kip Macy Date: Thu, 20 Nov 2008 10:48:01 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185127 - in user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb: . sys ulp/iw_cxgb X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Nov 2008 10:48:02 -0000 Author: kmacy Date: Thu Nov 20 10:48:01 2008 New Revision: 185127 URL: http://svn.freebsd.org/changeset/base/185127 Log: Fix cxgb to compile against new lock-less ring buffer Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_adapter.h user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_osdep.h user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/sys/cxgb_support.c user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/ulp/iw_cxgb/iw_cxgb_resource.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_adapter.h ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_adapter.h Thu Nov 20 10:47:21 2008 (r185126) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_adapter.h Thu Nov 20 10:48:01 2008 (r185127) @@ -41,6 +41,7 @@ $FreeBSD$ #include #include #include +#include #include #include @@ -258,7 +259,7 @@ struct sge_txq { * mbuf touches */ struct mbuf_head cleanq; - struct buf_ring txq_mr; + struct buf_ring *txq_mr; struct mbuf *immpkt; uint32_t txq_drops; uint32_t txq_skipped; Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Thu Nov 20 10:47:21 2008 (r185126) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Thu Nov 20 10:48:01 2008 (r185127) @@ -122,7 +122,7 @@ cxgb_pcpu_enqueue_packet_(struct sge_qse return (ENXIO); } txq = &qs->txq[TXQ_ETH]; - err = buf_ring_enqueue(&txq->txq_mr, m); + err = buf_ring_enqueue(txq->txq_mr, m); if (err) { txq->txq_drops++; m_freem(m); @@ -187,7 +187,7 @@ cxgb_dequeue_packet(struct sge_txq *txq, } sc = qs->port->adapter; - m = buf_ring_dequeue(&txq->txq_mr); + m = buf_ring_dequeue(txq->txq_mr); if (m == NULL) return (0); @@ -202,14 +202,14 @@ cxgb_dequeue_packet(struct sge_txq *txq, } size = m->m_pkthdr.len; - for (m = buf_ring_peek(&txq->txq_mr); m != NULL; - m = buf_ring_peek(&txq->txq_mr)) { + for (m = buf_ring_peek(txq->txq_mr); m != NULL; + m = buf_ring_peek(txq->txq_mr)) { if (m->m_pkthdr.tso_segsz > 0 || size + m->m_pkthdr.len > TX_WR_SIZE_MAX || m->m_next != NULL) break; - buf_ring_dequeue(&txq->txq_mr); + buf_ring_dequeue(txq->txq_mr); size += m->m_pkthdr.len; m_vec[count++] = m; @@ -232,7 +232,7 @@ cxgb_pcpu_free(struct sge_qset *qs) mtx_lock(&txq->lock); while ((m = mbufq_dequeue(&txq->sendq)) != NULL) m_freem(m); - while ((m = buf_ring_dequeue(&txq->txq_mr)) != NULL) + while ((m = buf_ring_dequeue(txq->txq_mr)) != NULL) m_freem(m); t3_free_tx_desc_all(txq); @@ -290,7 +290,7 @@ cxgb_pcpu_start_(struct sge_qset *qs, st initerr = ENXIO; else if (immpkt) { - if (!buf_ring_empty(&txq->txq_mr)) + if (!buf_ring_empty(txq->txq_mr)) initerr = cxgb_pcpu_enqueue_packet_(qs, immpkt); else txq->immpkt = immpkt; @@ -321,7 +321,7 @@ cxgb_pcpu_start_(struct sge_qset *qs, st } stopped = isset(&qs->txq_stopped, TXQ_ETH); - flush = (((!buf_ring_empty(&txq->txq_mr) || (!IFQ_DRV_IS_EMPTY(&pi->ifp->if_snd))) && !stopped) || txq->immpkt); + flush = (((!buf_ring_empty(txq->txq_mr) || (!IFQ_DRV_IS_EMPTY(&pi->ifp->if_snd))) && !stopped) || txq->immpkt); max_desc = tx_flush ? TX_ETH_Q_SIZE : TX_START_MAX_DESC; if (cxgb_debug) @@ -332,7 +332,7 @@ cxgb_pcpu_start_(struct sge_qset *qs, st if ((tx_flush && flush && err == 0) && - (!buf_ring_empty(&txq->txq_mr) || + (!buf_ring_empty(txq->txq_mr) || !IFQ_DRV_IS_EMPTY(&pi->ifp->if_snd))) { struct thread *td = curthread; @@ -382,7 +382,7 @@ cxgb_pcpu_transmit(struct ifnet *ifp, st txq = &qs->txq[TXQ_ETH]; if (((sc->tunq_coalesce == 0) || - (buf_ring_count(&txq->txq_mr) >= TX_WR_COUNT_MAX) || + (buf_ring_count(txq->txq_mr) >= TX_WR_COUNT_MAX) || (coalesce_tx_enable == 0)) && mtx_trylock(&txq->lock)) { if (cxgb_debug) printf("doing immediate transmit\n"); @@ -390,12 +390,12 @@ cxgb_pcpu_transmit(struct ifnet *ifp, st txq->flags |= TXQ_TRANSMITTING; err = cxgb_pcpu_start_(qs, immpkt, FALSE); txq->flags &= ~TXQ_TRANSMITTING; - resid = (buf_ring_count(&txq->txq_mr) > 64) || (desc_reclaimable(txq) > 64); + resid = (buf_ring_count(txq->txq_mr) > 64) || (desc_reclaimable(txq) > 64); mtx_unlock(&txq->lock); } else if (immpkt) { if (cxgb_debug) printf("deferred coalesce=%jx ring_count=%d mtx_owned=%d\n", - sc->tunq_coalesce, buf_ring_count(&txq->txq_mr), mtx_owned(&txq->lock)); + sc->tunq_coalesce, buf_ring_count(txq->txq_mr), mtx_owned(&txq->lock)); err = cxgb_pcpu_enqueue_packet_(qs, immpkt); } @@ -444,7 +444,7 @@ cxgb_pcpu_start_proc(void *arg) if ((qs->port->ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) { idleticks = hz; - if (!buf_ring_empty(&txq->txq_mr) || + if (!buf_ring_empty(txq->txq_mr) || !mbufq_empty(&txq->sendq)) cxgb_pcpu_free(qs); goto done; @@ -469,11 +469,13 @@ cxgb_pcpu_start_proc(void *arg) mtx_unlock(&qs->rspq.lock); } #endif - if ((!buf_ring_empty(&txq->txq_mr)) && err == 0) { + if ((!buf_ring_empty(txq->txq_mr)) && err == 0) { +#if 0 if (cxgb_debug) printf("head=%p cons=%d prod=%d\n", txq->sendq.head, txq->txq_mr.br_cons, txq->txq_mr.br_prod); +#endif continue; } done: Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_osdep.h ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_osdep.h Thu Nov 20 10:47:21 2008 (r185126) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_osdep.h Thu Nov 20 10:48:01 2008 (r185127) @@ -156,9 +156,6 @@ struct t3_mbuf_hdr { #if defined(__i386__) || defined(__amd64__) -#define mb() __asm volatile("mfence":::"memory") -#define rmb() __asm volatile("lfence":::"memory") -#define wmb() __asm volatile("sfence" ::: "memory") #define smp_mb() mb() #define L1_CACHE_BYTES 128 @@ -187,164 +184,6 @@ extern void kdb_backtrace(void); #define L1_CACHE_BYTES 32 #endif -struct buf_ring { - caddr_t *br_ring; - volatile uint32_t br_cons; - volatile uint32_t br_prod; - int br_size; - struct mtx br_lock; -}; - -struct buf_ring *buf_ring_alloc(int count, int flags); -void buf_ring_free(struct buf_ring *); - -static __inline int -buf_ring_count(struct buf_ring *mr) -{ - int size = mr->br_size; - uint32_t mask = size - 1; - - return ((size + mr->br_prod - mr->br_cons) & mask); -} - -static __inline int -buf_ring_empty(struct buf_ring *mr) -{ - return (mr->br_cons == mr->br_prod); -} - -static __inline int -buf_ring_full(struct buf_ring *mr) -{ - uint32_t mask; - - mask = mr->br_size - 1; - return (mr->br_cons == ((mr->br_prod + 1) & mask)); -} - -/* - * The producer and consumer are independently locked - * this relies on the consumer providing his own serialization - * - */ -static __inline void * -__buf_ring_dequeue(struct buf_ring *mr) -{ - uint32_t prod, cons, mask; - caddr_t *ring, m; - - ring = (caddr_t *)mr->br_ring; - mask = mr->br_size - 1; - cons = mr->br_cons; - prod = mr->br_prod; - m = NULL; - if (cons != prod) { - m = ring[cons]; - ring[cons] = NULL; - mr->br_cons = (cons + 1) & mask; - } - return (m); -} - -static __inline void * -buf_ring_dequeue(struct buf_ring *mr) -{ - void *buf; - - mtx_lock(&mr->br_lock); - buf = __buf_ring_dequeue(mr); - mtx_unlock(&mr->br_lock); - return (buf); -} - -#ifdef INVARIANT_SUPPORT -static __inline void -__buf_ring_scan(struct buf_ring *mr, void *m, char *file, int line) -{ - int i; - - for (i = 0; i < mr->br_size; i++) - if (m == mr->br_ring[i]) - panic("%s:%d m=%p present prod=%d cons=%d idx=%d", file, - line, m, mr->br_prod, mr->br_cons, i); -} - -static __inline void -buf_ring_scan(struct buf_ring *mr, void *m, char *file, int line) -{ - mtx_lock(&mr->br_lock); - __buf_ring_scan(mr, m, file, line); - mtx_unlock(&mr->br_lock); -} - -#else -static __inline void -__buf_ring_scan(struct buf_ring *mr, void *m, char *file, int line) -{ -} - -static __inline void -buf_ring_scan(struct buf_ring *mr, void *m, char *file, int line) -{ -} -#endif - -static __inline int -__buf_ring_enqueue(struct buf_ring *mr, void *m, char *file, int line) -{ - - uint32_t prod, cons, mask; - int err; - - mask = mr->br_size - 1; - prod = mr->br_prod; - mb(); - cons = mr->br_cons; - __buf_ring_scan(mr, m, file, line); - if (((prod + 1) & mask) != cons) { - KASSERT(mr->br_ring[prod] == NULL, ("overwriting entry")); - mr->br_ring[prod] = m; - mb(); - mr->br_prod = (prod + 1) & mask; - err = 0; - } else - err = ENOBUFS; - - return (err); -} - -static __inline int -buf_ring_enqueue_(struct buf_ring *mr, void *m, char *file, int line) -{ - int err; - - mtx_lock(&mr->br_lock); - err = __buf_ring_enqueue(mr, m, file, line); - mtx_unlock(&mr->br_lock); - - return (err); -} - -#define buf_ring_enqueue(mr, m) buf_ring_enqueue_((mr), (m), __FILE__, __LINE__) - - -static __inline void * -buf_ring_peek(struct buf_ring *mr) -{ - int prod, cons, mask; - caddr_t *ring, m; - - ring = (caddr_t *)mr->br_ring; - mask = mr->br_size - 1; - cons = mr->br_cons; - prod = mr->br_prod; - m = NULL; - if (cons != prod) - m = ring[cons]; - - return (m); -} - #define DBG_RX (1 << 0) static const int debug_flags = DBG_RX; Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Thu Nov 20 10:47:21 2008 (r185126) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Thu Nov 20 10:48:01 2008 (r185127) @@ -1715,10 +1715,8 @@ t3_free_qset(adapter_t *sc, struct sge_q t3_free_tx_desc_all(&q->txq[TXQ_ETH]); for (i = 0; i < SGE_TXQ_PER_SET; i++) - if (q->txq[i].txq_mr.br_ring != NULL) { - free(q->txq[i].txq_mr.br_ring, M_DEVBUF); - mtx_destroy(&q->txq[i].txq_mr.br_lock); - } + if (q->txq[i].txq_mr != NULL) + buf_ring_free(q->txq[i].txq_mr, M_DEVBUF); for (i = 0; i < SGE_RXQ_PER_SET; ++i) { if (q->fl[i].desc) { mtx_lock_spin(&sc->sge.reg_lock); @@ -2277,14 +2275,12 @@ t3_sge_alloc_qset(adapter_t *sc, u_int i int i, header_size, ret = 0; for (i = 0; i < SGE_TXQ_PER_SET; i++) { - if ((q->txq[i].txq_mr.br_ring = malloc(cxgb_txq_buf_ring_size*sizeof(struct mbuf *), - M_DEVBUF, M_WAITOK|M_ZERO)) == NULL) { + + if ((q->txq[i].txq_mr = buf_ring_alloc(cxgb_txq_buf_ring_size, + M_DEVBUF, M_WAITOK)) == NULL) { device_printf(sc->dev, "failed to allocate mbuf ring\n"); goto err; } - q->txq[i].txq_mr.br_prod = q->txq[i].txq_mr.br_cons = 0; - q->txq[i].txq_mr.br_size = cxgb_txq_buf_ring_size; - mtx_init(&q->txq[i].txq_mr.br_lock, "txq mbuf ring", NULL, MTX_DEF); } init_qset_cntxt(q, id); @@ -3509,12 +3505,14 @@ t3_add_configured_sysctls(adapter_t *sc) SYSCTL_ADD_INT(ctx, txqpoidlist, OID_AUTO, "sendqlen", CTLFLAG_RD, &qs->txq[TXQ_ETH].sendq.qlen, 0, "#tunneled packets waiting to be sent"); +#if 0 SYSCTL_ADD_UINT(ctx, txqpoidlist, OID_AUTO, "queue_pidx", CTLFLAG_RD, (uint32_t *)(uintptr_t)&qs->txq[TXQ_ETH].txq_mr.br_prod, 0, "#tunneled packets queue producer index"); SYSCTL_ADD_UINT(ctx, txqpoidlist, OID_AUTO, "queue_cidx", CTLFLAG_RD, (uint32_t *)(uintptr_t)&qs->txq[TXQ_ETH].txq_mr.br_cons, 0, "#tunneled packets queue consumer index"); +#endif SYSCTL_ADD_INT(ctx, txqpoidlist, OID_AUTO, "processed", CTLFLAG_RD, &qs->txq[TXQ_ETH].processed, 0, "#tunneled packets processed by the card"); Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/sys/cxgb_support.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/sys/cxgb_support.c Thu Nov 20 10:47:21 2008 (r185126) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/sys/cxgb_support.c Thu Nov 20 10:48:01 2008 (r185127) @@ -303,33 +303,3 @@ free: uma_zfree(zone, vec[i]); } -struct buf_ring * -buf_ring_alloc(int count, int flags) -{ - struct buf_ring *br; - - KASSERT(powerof2(count), ("buf ring must be size power of 2")); - - br = malloc(sizeof(struct buf_ring), M_DEVBUF, flags|M_ZERO); - if (br == NULL) - return (NULL); - - br->br_ring = malloc(sizeof(caddr_t)*count, M_DEVBUF, flags|M_ZERO); - if (br->br_ring == NULL) { - free(br, M_DEVBUF); - return (NULL); - } - - mtx_init(&br->br_lock, "buf ring", NULL, MTX_DUPOK|MTX_DEF); - br->br_size = count; - br->br_prod = br->br_cons = 0; - - return (br); -} - -void -buf_ring_free(struct buf_ring *br) -{ - free(br->br_ring, M_DEVBUF); - free(br, M_DEVBUF); -} Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/ulp/iw_cxgb/iw_cxgb_resource.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/ulp/iw_cxgb/iw_cxgb_resource.c Thu Nov 20 10:47:21 2008 (r185126) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/ulp/iw_cxgb/iw_cxgb_resource.c Thu Nov 20 10:48:01 2008 (r185127) @@ -90,7 +90,7 @@ static int __cxio_init_resource_fifo(str u32 rarray[16]; mtx_init(fifo_lock, "cxio fifo", NULL, MTX_DEF|MTX_DUPOK); - *fifo = buf_ring_alloc(nr, M_NOWAIT); + *fifo = buf_ring_alloc(nr, M_DEVBUF, M_NOWAIT); if (*fifo == NULL) return (-ENOMEM); #if 0 @@ -149,7 +149,7 @@ static int cxio_init_qpid_fifo(struct cx mtx_init(&rdev_p->rscp->qpid_fifo_lock, "qpid fifo", NULL, MTX_DEF); - rdev_p->rscp->qpid_fifo = buf_ring_alloc(T3_MAX_NUM_QP, M_NOWAIT); + rdev_p->rscp->qpid_fifo = buf_ring_alloc(T3_MAX_NUM_QP, M_DEVBUF, M_NOWAIT); if (rdev_p->rscp->qpid_fifo == NULL) return (-ENOMEM); @@ -168,7 +168,7 @@ int cxio_hal_init_rhdl_resource(u32 nr_r void cxio_hal_destroy_rhdl_resource(void) { - buf_ring_free(rhdl_fifo); + buf_ring_free(rhdl_fifo, M_DEVBUF); } #endif @@ -202,11 +202,11 @@ int cxio_hal_init_resource(struct cxio_r goto pdid_err; return 0; pdid_err: - buf_ring_free(rscp->cqid_fifo); + buf_ring_free(rscp->cqid_fifo, M_DEVBUF); cqid_err: - buf_ring_free(rscp->qpid_fifo); + buf_ring_free(rscp->qpid_fifo, M_DEVBUF); qpid_err: - buf_ring_free(rscp->tpt_fifo); + buf_ring_free(rscp->tpt_fifo, M_DEVBUF); tpt_err: return (-ENOMEM); } @@ -276,10 +276,10 @@ void cxio_hal_put_pdid(struct cxio_hal_r void cxio_hal_destroy_resource(struct cxio_hal_resource *rscp) { - buf_ring_free(rscp->tpt_fifo); - buf_ring_free(rscp->cqid_fifo); - buf_ring_free(rscp->qpid_fifo); - buf_ring_free(rscp->pdid_fifo); + buf_ring_free(rscp->tpt_fifo, M_DEVBUF); + buf_ring_free(rscp->cqid_fifo, M_DEVBUF); + buf_ring_free(rscp->qpid_fifo, M_DEVBUF); + buf_ring_free(rscp->pdid_fifo, M_DEVBUF); free(rscp, M_DEVBUF); } From owner-svn-src-user@FreeBSD.ORG Thu Nov 20 12:40:54 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 68278106564A; Thu, 20 Nov 2008 12:40:54 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 5309E8FC08; Thu, 20 Nov 2008 12:40:54 +0000 (UTC) (envelope-from lstewart@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAKCesYi051793; Thu, 20 Nov 2008 12:40:54 GMT (envelope-from lstewart@svn.freebsd.org) Received: (from lstewart@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAKCesYa051792; Thu, 20 Nov 2008 12:40:54 GMT (envelope-from lstewart@svn.freebsd.org) Message-Id: <200811201240.mAKCesYa051792@svn.freebsd.org> From: Lawrence Stewart Date: Thu, 20 Nov 2008 12:40:52 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185129 - user/lstewart/alq_varlen_8.x/sys/kern X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Nov 2008 12:40:54 -0000 Author: lstewart Date: Thu Nov 20 12:40:52 2008 New Revision: 185129 URL: http://svn.freebsd.org/changeset/base/185129 Log: Clean up some comments/code and fix some bugs I noticed doing a read through. Modified: user/lstewart/alq_varlen_8.x/sys/kern/kern_alq.c Modified: user/lstewart/alq_varlen_8.x/sys/kern/kern_alq.c ============================================================================== --- user/lstewart/alq_varlen_8.x/sys/kern/kern_alq.c Thu Nov 20 12:23:44 2008 (r185128) +++ user/lstewart/alq_varlen_8.x/sys/kern/kern_alq.c Thu Nov 20 12:40:52 2008 (r185129) @@ -85,7 +85,6 @@ static struct mtx ald_mtx; static LIST_HEAD(, alq) ald_queues; static LIST_HEAD(, alq) ald_active; static int ald_shutingdown = 0; -struct thread *ald_thread; static struct proc *ald_proc; #define ALD_LOCK() mtx_lock(&ald_mtx) @@ -179,8 +178,6 @@ ald_daemon(void) int needwakeup; struct alq *alq; - ald_thread = FIRST_THREAD_IN_PROC(ald_proc); - EVENTHANDLER_REGISTER(shutdown_pre_sync, ald_shutdown, NULL, SHUTDOWN_PRI_FIRST); @@ -206,7 +203,7 @@ ald_daemon(void) ALD_LOCK(); } - kthread_exit(); + kproc_exit(0); } static void @@ -217,11 +214,14 @@ ald_shutdown(void *arg, int howto) ALD_LOCK(); ald_shutingdown = 1; - /* wake ald_daemon so that it exits*/ + /* + * Wake ald_daemon so that it exits. It won't be able to do + * anything until we mtx_sleep because we hold the ald_mtx + */ wakeup(&ald_active); - /* wait for ald_daemon to exit */ - mtx_sleep(ald_thread, &ald_mtx, PWAIT, "aldslp", 0); + /* Wait for ald_daemon to exit */ + mtx_sleep(ald_proc, &ald_mtx, PWAIT, "aldslp", 0); while ((alq = LIST_FIRST(&ald_queues)) != NULL) { LIST_REMOVE(alq, aq_link); @@ -414,7 +414,7 @@ alq_open(struct alq **alqp, const char * alq->aq_entlen = 0; } - alq->aq_freebytes = alq->aq_buflen; + alq->aq_freebytes = alq->aq_buflen; alq->aq_entbuf = malloc(alq->aq_buflen, M_ALD, M_WAITOK|M_ZERO); alq->aq_writehead = alq->aq_writetail = 0; @@ -446,7 +446,7 @@ alq_writen(struct alq *alq, void *data, int activate = 0; int copy = len; - KASSERT(len > 0 && len < alq->aq_buflen, + KASSERT(len > 0 && len <= alq->aq_buflen, ("%s: len <= 0 || len > alq->aq_buflen", __func__) ); @@ -501,10 +501,10 @@ alq_writen(struct alq *alq, void *data, if (copy != len) { /* * wrap the buffer by copying the remainder of our message - * to the start of the buffer and resetting the head ptr + * to the start of the buffer and resetting aq_writehead */ bcopy(data, alq->aq_entbuf, len - copy); - alq->aq_writehead = copy; + alq->aq_writehead = len - copy; } alq->aq_freebytes -= len; @@ -541,6 +541,10 @@ alq_getn(struct alq *alq, int len, int f struct ale *ale; int contigbytes; + KASSERT(len > 0 && len <= alq->aq_buflen, + ("%s: len <= 0 || len > alq->aq_buflen", __func__) + ); + ale = malloc( sizeof(struct ale), M_ALD, (flags & ALQ_NOWAIT) ? M_NOWAIT : M_WAITOK @@ -551,7 +555,14 @@ alq_getn(struct alq *alq, int len, int f ALQ_LOCK(alq); - /* determine the number of free contiguous bytes */ + /* + * Determine the number of free contiguous bytes. + * We ensure elsewhere that if aq_writehead == aq_writetail because + * the buffer is empty, they will both be set to 0 and therefore + * aq_freebytes == aq_buflen and is fully contiguous. + * If they are equal and the buffer is not empty, aq_freebytes will + * be 0 indicating the buffer is full. + */ if (alq->aq_writehead <= alq->aq_writetail) contigbytes = alq->aq_freebytes; else @@ -569,7 +580,7 @@ alq_getn(struct alq *alq, int len, int f } /* - * ALQ_WAITOK or contigbytes > len, + * ALQ_WAITOK or contigbytes >= len, * either spin until we have enough free contiguous bytes (former) * or skip (latter) */ @@ -583,15 +594,15 @@ alq_getn(struct alq *alq, int len, int f } /* - * we need to serialise wakups to ensure records remain in order... - * therefore, wakeup the next thread in the queue waiting for - * alq resources to be available + * We need to serialise wakups to ensure records remain in order. + * Therefore, wakeup the next thread in the queue waiting for + * alq resources to be available. * (technically this is only required if we actually entered the above * while loop) */ wakeup_one(alq); - /* bail if we're shutting down */ + /* Bail if we're shutting down */ if (alq->aq_flags & AQ_SHUTDOWN) { ALQ_UNLOCK(alq); return (NULL); @@ -606,6 +617,10 @@ alq_getn(struct alq *alq, int len, int f alq->aq_writehead += len; alq->aq_freebytes -= len; + /* Wrap aq_writehead if we've filled to the end of the buffer */ + if (alq->aq_writehead == alq->aq_buflen) + alq->aq_writehead = 0; + return (ale); } From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 00:57:50 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7DE021065672; Fri, 21 Nov 2008 00:57:50 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 683348FC1A; Fri, 21 Nov 2008 00:57:50 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL0voMX068442; Fri, 21 Nov 2008 00:57:50 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL0voKZ068441; Fri, 21 Nov 2008 00:57:50 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210057.mAL0voKZ068441@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 00:57:50 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185136 - user/kmacy/HEAD_fast_multi_xmit/sys/sys X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 00:57:50 -0000 Author: kmacy Date: Fri Nov 21 00:57:50 2008 New Revision: 185136 URL: http://svn.freebsd.org/changeset/base/185136 Log: - Add single consumer variant of dequeue for cases where the consumer is already protected by a lock - move enqueue/dequeue to critical sections to avoid starvation by preemption Modified: user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Modified: user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Fri Nov 21 00:48:57 2008 (r185135) +++ user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Fri Nov 21 00:57:50 2008 (r185136) @@ -86,14 +86,17 @@ buf_ring_enqueue(struct buf_ring *br, vo uint32_t cons_tail; int success; + critical_enter(); do { prod_head = br->br_prod_head; cons_tail = br->br_cons_tail; prod_next = (prod_head + 1) & br->br_prod_mask; - if (prod_next == cons_tail) + if (prod_next == cons_tail) { + critical_exit(); return (ENOSPC); + } success = atomic_cmpset_int(&br->br_prod_head, prod_head, prod_next); @@ -107,32 +110,38 @@ buf_ring_enqueue(struct buf_ring *br, vo /* * If there are other enqueues in progress * that preceeded us, we need to wait for them - * to complete - this is only a problem if they - * have been preempted + * to complete */ while (br->br_prod_tail != prod_head) cpu_spinwait(); br->br_prod_tail = prod_next; - + critical_exit(); return (0); } +/* + * multi-consumer safe dequeue + * + */ static __inline void * -buf_ring_dequeue(struct buf_ring *br) +buf_ring_dequeue_mc(struct buf_ring *br) { uint32_t cons_head, cons_next; uint32_t prod_tail; void *buf; int success; - + + critical_enter(); do { cons_head = br->br_cons_head; prod_tail = br->br_prod_tail; cons_next = (cons_head + 1) & br->br_cons_mask; - if (cons_head == prod_tail) + if (cons_head == prod_tail) { + critical_exit(); return (NULL); + } success = atomic_cmpset_int(&br->br_cons_head, cons_head, cons_next); @@ -147,14 +156,53 @@ buf_ring_dequeue(struct buf_ring *br) /* * If there are other dequeues in progress * that preceeded us, we need to wait for them - * to complete - this is only a problem if they - * have been preempted + * to complete */ while (br->br_cons_tail != cons_head) cpu_spinwait(); br->br_cons_tail = cons_next; + mb(); + critical_exit(); + + return (buf); +} + +/* + * Single-Consumer dequeue for uses where dequeue + * is protected by a lock + */ +static __inline void * +buf_ring_dequeue_sc(struct buf_ring *br) +{ + uint32_t cons_head, cons_next; + uint32_t prod_tail; + void *buf; + + critical_enter(); + cons_head = br->br_cons_head; + prod_tail = br->br_prod_tail; + + cons_next = (cons_head + 1) & br->br_cons_mask; + + if (cons_head == prod_tail) { + critical_exit(); + return (NULL); + } + + br->br_cons_head = cons_next; + buf = br->br_ring[cons_head]; +#ifdef INVARIANTS + br->br_ring[cons_head] = NULL; +#endif + mb(); + KASSERT(br->br_cons_tail == cons_head, + ("inconsistent list cons_tail=%d cons_head=%d", + br->br_cons_tail, cons_head)); + br->br_cons_tail = cons_next; + mb(); + critical_exit(); return (buf); } From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 01:04:22 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF701106564A; Fri, 21 Nov 2008 01:04:22 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id B92948FC18; Fri, 21 Nov 2008 01:04:22 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL14MEe068624; Fri, 21 Nov 2008 01:04:22 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL14M2w068622; Fri, 21 Nov 2008 01:04:22 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210104.mAL14M2w068622@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 01:04:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185137 - in user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb: . ulp/iw_cxgb X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 01:04:22 -0000 Author: kmacy Date: Fri Nov 21 01:04:22 2008 New Revision: 185137 URL: http://svn.freebsd.org/changeset/base/185137 Log: Update to buf_ring_dequeue changes Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/ulp/iw_cxgb/iw_cxgb_resource.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Fri Nov 21 00:57:50 2008 (r185136) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Fri Nov 21 01:04:22 2008 (r185137) @@ -187,7 +187,7 @@ cxgb_dequeue_packet(struct sge_txq *txq, } sc = qs->port->adapter; - m = buf_ring_dequeue(txq->txq_mr); + m = buf_ring_dequeue_sc(txq->txq_mr); if (m == NULL) return (0); @@ -209,7 +209,7 @@ cxgb_dequeue_packet(struct sge_txq *txq, size + m->m_pkthdr.len > TX_WR_SIZE_MAX || m->m_next != NULL) break; - buf_ring_dequeue(txq->txq_mr); + buf_ring_dequeue_sc(txq->txq_mr); size += m->m_pkthdr.len; m_vec[count++] = m; @@ -232,7 +232,7 @@ cxgb_pcpu_free(struct sge_qset *qs) mtx_lock(&txq->lock); while ((m = mbufq_dequeue(&txq->sendq)) != NULL) m_freem(m); - while ((m = buf_ring_dequeue(txq->txq_mr)) != NULL) + while ((m = buf_ring_dequeue_sc(txq->txq_mr)) != NULL) m_freem(m); t3_free_tx_desc_all(txq); Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/ulp/iw_cxgb/iw_cxgb_resource.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/ulp/iw_cxgb/iw_cxgb_resource.c Fri Nov 21 00:57:50 2008 (r185136) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/ulp/iw_cxgb/iw_cxgb_resource.c Fri Nov 21 01:04:22 2008 (r185137) @@ -122,7 +122,7 @@ static int __cxio_init_resource_fifo(str buf_ring_enqueue(*fifo, (void *) (uintptr_t)i); #if 0 for (i = 0; i < skip_low + skip_high; i++) - buf_ring_dequeue(*fifo); + buf_ring_dequeue_sc(*fifo); #endif return 0; } @@ -219,7 +219,7 @@ static u32 cxio_hal_get_resource(struct u32 entry; mtx_lock(lock); - entry = (u32)(uintptr_t)buf_ring_dequeue(fifo); + entry = (u32)(uintptr_t)buf_ring_dequeue_sc(fifo); mtx_unlock(lock); return entry; } From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 01:11:22 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 99AEC1065675; Fri, 21 Nov 2008 01:11:22 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 83ECD8FC1D; Fri, 21 Nov 2008 01:11:22 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL1BMZE068860; Fri, 21 Nov 2008 01:11:22 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL1BMX0068859; Fri, 21 Nov 2008 01:11:22 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210111.mAL1BMX0068859@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 01:11:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185138 - user/kmacy/HEAD_fast_multi_xmit/sys/conf X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 01:11:22 -0000 Author: kmacy Date: Fri Nov 21 01:11:22 2008 New Revision: 185138 URL: http://svn.freebsd.org/changeset/base/185138 Log: adb buf ring alloc / free to build Modified: user/kmacy/HEAD_fast_multi_xmit/sys/conf/files Modified: user/kmacy/HEAD_fast_multi_xmit/sys/conf/files ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/conf/files Fri Nov 21 01:04:22 2008 (r185137) +++ user/kmacy/HEAD_fast_multi_xmit/sys/conf/files Fri Nov 21 01:11:22 2008 (r185138) @@ -1805,6 +1805,7 @@ kern/subr_acl_posix1e.c standard kern/subr_autoconf.c standard kern/subr_blist.c standard kern/subr_bus.c standard +kern/subr_bufr.c standard kern/subr_clist.c standard kern/subr_clock.c standard kern/subr_devstat.c standard From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 04:37:38 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E8CF51065670; Fri, 21 Nov 2008 04:37:38 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id D25BC8FC08; Fri, 21 Nov 2008 04:37:38 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL4bcrn073890; Fri, 21 Nov 2008 04:37:38 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL4bcnF073889; Fri, 21 Nov 2008 04:37:38 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210437.mAL4bcnF073889@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 04:37:38 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185141 - user/kmacy/HEAD_fast_multi_xmit/sys/sys X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 04:37:39 -0000 Author: kmacy Date: Fri Nov 21 04:37:38 2008 New Revision: 185141 URL: http://svn.freebsd.org/changeset/base/185141 Log: add sanity checking of buf ring functions under DEBUG_BUFRING Modified: user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Modified: user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Fri Nov 21 03:03:57 2008 (r185140) +++ user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Fri Nov 21 04:37:38 2008 (r185141) @@ -56,6 +56,15 @@ #error "unknown compiler" #endif +#if defined(INVARIANTS) && !defined(DEBUG_BUFRING) +#define DEBUG_BUFRING 1 +#endif + +#ifdef DEBUG_BUFRING +#include +#include +#endif + struct buf_ring { volatile uint32_t br_prod_head; volatile uint32_t br_prod_tail; @@ -70,11 +79,14 @@ struct buf_ring { volatile uint32_t br_cons_tail; int br_cons_size; int br_cons_mask; - + /* * Pad out to next L2 cache line */ uint64_t _pad1[14]; +#ifdef DEBUG_BUFRING + struct mtx *br_lock; +#endif void *br_ring[0]; }; @@ -85,7 +97,14 @@ buf_ring_enqueue(struct buf_ring *br, vo uint32_t prod_head, prod_next; uint32_t cons_tail; int success; - +#ifdef DEBUG_BUFRING + int i; + for (i = br->br_cons_head; i != br->br_prod_head; + i = ((i + 1) & br->br_cons_mask)) + if(br->br_ring[i] == buf) + panic("buf=%p already enqueue at %d prod=%d cons=%d", + buf, i, br->br_prod_tail, br->br_cons_tail); +#endif critical_enter(); do { prod_head = br->br_prod_head; @@ -101,9 +120,10 @@ buf_ring_enqueue(struct buf_ring *br, vo success = atomic_cmpset_int(&br->br_prod_head, prod_head, prod_next); } while (success == 0); - - KASSERT(br->br_ring[prod_head] == NULL, ("dangling value in enqueue")); - +#ifdef DEBUG_BUFRING + if (br->br_ring[prod_head] != NULL) + panic("dangling value in enqueue"); +#endif br->br_ring[prod_head] = buf; wmb(); @@ -115,6 +135,7 @@ buf_ring_enqueue(struct buf_ring *br, vo while (br->br_prod_tail != prod_head) cpu_spinwait(); br->br_prod_tail = prod_next; + mb(); critical_exit(); return (0); } @@ -148,7 +169,7 @@ buf_ring_dequeue_mc(struct buf_ring *br) } while (success == 0); buf = br->br_ring[cons_head]; -#ifdef INVARIANTS +#ifdef DEBUG_BUFRING br->br_ring[cons_head] = NULL; #endif mb(); @@ -192,14 +213,16 @@ buf_ring_dequeue_sc(struct buf_ring *br) br->br_cons_head = cons_next; buf = br->br_ring[cons_head]; -#ifdef INVARIANTS + mb(); + +#ifdef DEBUG_BUFRING br->br_ring[cons_head] = NULL; + if (!mtx_owned(br->br_lock)) + panic("lock not held on single consumer dequeue"); + if (br->br_cons_tail != cons_head) + panic("inconsistent list cons_tail=%d cons_head=%d", + br->br_cons_tail, cons_head); #endif - mb(); - KASSERT(br->br_cons_tail == cons_head, - ("inconsistent list cons_tail=%d cons_head=%d", - br->br_cons_tail, cons_head)); - br->br_cons_tail = cons_next; mb(); critical_exit(); @@ -235,7 +258,8 @@ buf_ring_count(struct buf_ring *br) & br->br_prod_mask); } -struct buf_ring *buf_ring_alloc(int count, struct malloc_type *type, int flags); +struct buf_ring *buf_ring_alloc(int count, struct malloc_type *type, int flags, + struct mtx *); void buf_ring_free(struct buf_ring *br, struct malloc_type *type); From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 04:38:20 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7D3071065672; Fri, 21 Nov 2008 04:38:20 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 667B88FC08; Fri, 21 Nov 2008 04:38:20 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL4cKcn073947; Fri, 21 Nov 2008 04:38:20 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL4cKk5073945; Fri, 21 Nov 2008 04:38:20 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210438.mAL4cKk5073945@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 04:38:20 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185142 - in user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb: . ulp/iw_cxgb X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 04:38:20 -0000 Author: kmacy Date: Fri Nov 21 04:38:20 2008 New Revision: 185142 URL: http://svn.freebsd.org/changeset/base/185142 Log: update buf_ring_alloc calls Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/ulp/iw_cxgb/iw_cxgb_resource.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Fri Nov 21 04:37:38 2008 (r185141) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Fri Nov 21 04:38:20 2008 (r185142) @@ -1879,7 +1879,7 @@ t3_free_tx_desc(struct sge_txq *q, int r txsd->flags &= ~TX_SW_DESC_MAPPED; } m_freem_iovec(&txsd->mi); -#ifdef INVARIANTS +#if 0 buf_ring_scan(&q->txq_mr, txsd->mi.mi_base, __FILE__, __LINE__); #endif txsd->mi.mi_base = NULL; @@ -2277,15 +2277,13 @@ t3_sge_alloc_qset(adapter_t *sc, u_int i for (i = 0; i < SGE_TXQ_PER_SET; i++) { if ((q->txq[i].txq_mr = buf_ring_alloc(cxgb_txq_buf_ring_size, - M_DEVBUF, M_WAITOK)) == NULL) { + M_DEVBUF, M_WAITOK, &q->txq[i].lock)) == NULL) { device_printf(sc->dev, "failed to allocate mbuf ring\n"); goto err; } } - init_qset_cntxt(q, id); q->idx = id; - if ((ret = alloc_ring(sc, p->fl_size, sizeof(struct rx_desc), sizeof(struct rx_sw_desc), &q->fl[0].phys_addr, &q->fl[0].desc, &q->fl[0].sdesc, Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/ulp/iw_cxgb/iw_cxgb_resource.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/ulp/iw_cxgb/iw_cxgb_resource.c Fri Nov 21 04:37:38 2008 (r185141) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/ulp/iw_cxgb/iw_cxgb_resource.c Fri Nov 21 04:38:20 2008 (r185142) @@ -90,7 +90,7 @@ static int __cxio_init_resource_fifo(str u32 rarray[16]; mtx_init(fifo_lock, "cxio fifo", NULL, MTX_DEF|MTX_DUPOK); - *fifo = buf_ring_alloc(nr, M_DEVBUF, M_NOWAIT); + *fifo = buf_ring_alloc(nr, M_DEVBUF, M_NOWAIT, fifo_lock); if (*fifo == NULL) return (-ENOMEM); #if 0 @@ -149,7 +149,8 @@ static int cxio_init_qpid_fifo(struct cx mtx_init(&rdev_p->rscp->qpid_fifo_lock, "qpid fifo", NULL, MTX_DEF); - rdev_p->rscp->qpid_fifo = buf_ring_alloc(T3_MAX_NUM_QP, M_DEVBUF, M_NOWAIT); + rdev_p->rscp->qpid_fifo = buf_ring_alloc(T3_MAX_NUM_QP, M_DEVBUF, M_NOWAIT, + &rdev_p->rscp->qpid_fifo_lock); if (rdev_p->rscp->qpid_fifo == NULL) return (-ENOMEM); From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 04:41:53 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 96DEE1065670; Fri, 21 Nov 2008 04:41:53 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 8081C8FC1A; Fri, 21 Nov 2008 04:41:53 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL4fr24074058; Fri, 21 Nov 2008 04:41:53 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL4frTb074057; Fri, 21 Nov 2008 04:41:53 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210441.mAL4frTb074057@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 04:41:53 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185143 - user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/sys X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 04:41:53 -0000 Author: kmacy Date: Fri Nov 21 04:41:53 2008 New Revision: 185143 URL: http://svn.freebsd.org/changeset/base/185143 Log: centralize free of mbuf tags Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/sys/uipc_mvec.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/sys/uipc_mvec.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/sys/uipc_mvec.c Fri Nov 21 04:38:20 2008 (r185142) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/sys/uipc_mvec.c Fri Nov 21 04:41:53 2008 (r185143) @@ -128,6 +128,8 @@ _mcl_collapse_mbuf(struct mbuf_iovec *mi #ifdef IFNET_MULTIQ mi->mi_rss_hash = m->m_pkthdr.rss_hash; #endif + if(!SLIST_EMPTY(&m->m_pkthdr.tags)) + m_tag_delete_chain(m, NULL); } if (m->m_type != MT_DATA) { mi->mi_data = NULL; @@ -201,9 +203,6 @@ busdma_map_sg_collapse(struct mbuf **m, KASSERT(n->m_pkthdr.len, ("packet has zero header len")); - if (n->m_flags & M_PKTHDR && !SLIST_EMPTY(&n->m_pkthdr.tags)) - m_tag_delete_chain(n, NULL); - if (n->m_pkthdr.len <= PIO_LEN) return (0); retry: @@ -319,9 +318,6 @@ busdma_map_sg_vec(struct mbuf **m, struc mv->mv_first = 0; for (mp = m, i = 0, mi = mv->mv_vec; i < pkt_count; mp++, segs++, mi++, i++) { - if ((*mp)->m_flags & M_PKTHDR - && !SLIST_EMPTY(&(*mp)->m_pkthdr.tags)) - m_tag_delete_chain(*mp, NULL); busdma_map_mbuf_fast(*mp, segs); _mcl_collapse_mbuf(mi, *mp); KASSERT(mi->mi_len, ("empty packet")); From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 04:44:50 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7681A1065670; Fri, 21 Nov 2008 04:44:50 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 5FE968FC08; Fri, 21 Nov 2008 04:44:50 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL4iofa074176; Fri, 21 Nov 2008 04:44:50 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL4ioW6074175; Fri, 21 Nov 2008 04:44:50 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210444.mAL4ioW6074175@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 04:44:50 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185144 - user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 04:44:50 -0000 Author: kmacy Date: Fri Nov 21 04:44:49 2008 New Revision: 185144 URL: http://svn.freebsd.org/changeset/base/185144 Log: check that dequeue and peek match Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Fri Nov 21 04:41:53 2008 (r185143) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Fri Nov 21 04:44:49 2008 (r185144) @@ -88,7 +88,7 @@ __FBSDID("$FreeBSD$"); extern int txq_fills; int multiq_tx_enable = 1; -int coalesce_tx_enable = 1; +int coalesce_tx_enable = 0; extern struct sysctl_oid_list sysctl__hw_cxgb_children; static int sleep_ticks = 1; @@ -155,16 +155,17 @@ cxgb_pcpu_enqueue_packet(struct ifnet *i static int cxgb_dequeue_packet(struct sge_txq *txq, struct mbuf **m_vec) { - struct mbuf *m; + struct mbuf *m, *m0; struct sge_qset *qs; int count, size, coalesced; struct adapter *sc; + #ifndef IFNET_MULTIQUEUE struct port_info *pi = txq->port; + mtx_assert(&txq->lock, MA_OWNED); if (txq->immpkt != NULL) panic("immediate packet set"); - mtx_assert(&txq->lock, MA_OWNED); IFQ_DRV_DEQUEUE(&pi->ifp->if_snd, m); if (m == NULL) @@ -174,6 +175,7 @@ cxgb_dequeue_packet(struct sge_txq *txq, return (1); #endif + mtx_assert(&txq->lock, MA_OWNED); coalesced = count = size = 0; qs = txq_to_qset(txq, TXQ_ETH); if (qs->qs_flags & QS_EXITING) @@ -192,9 +194,7 @@ cxgb_dequeue_packet(struct sge_txq *txq, return (0); count = 1; - KASSERT(m->m_type == MT_DATA, - ("m=%p is bad mbuf type %d from ring cons=%d prod=%d", m, - m->m_type, txq->txq_mr.br_cons, txq->txq_mr.br_prod)); + m_vec[0] = m; if (m->m_pkthdr.tso_segsz > 0 || m->m_pkthdr.len > TX_WR_SIZE_MAX || m->m_next != NULL || (coalesce_tx_enable == 0)) { @@ -209,7 +209,11 @@ cxgb_dequeue_packet(struct sge_txq *txq, size + m->m_pkthdr.len > TX_WR_SIZE_MAX || m->m_next != NULL) break; - buf_ring_dequeue_sc(txq->txq_mr); + m0 = buf_ring_dequeue_sc(txq->txq_mr); +#ifdef DEBUG_BUFRING + if (m0 != m) + panic("peek and dequeue don't match"); +#endif size += m->m_pkthdr.len; m_vec[count++] = m; From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 04:45:54 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 26EF61065674; Fri, 21 Nov 2008 04:45:54 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 10DE18FC0A; Fri, 21 Nov 2008 04:45:54 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL4jres074235; Fri, 21 Nov 2008 04:45:53 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL4jrfi074234; Fri, 21 Nov 2008 04:45:53 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210445.mAL4jrfi074234@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 04:45:53 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185145 - user/kmacy/HEAD_fast_multi_xmit/sys/sys X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 04:45:54 -0000 Author: kmacy Date: Fri Nov 21 04:45:53 2008 New Revision: 185145 URL: http://svn.freebsd.org/changeset/base/185145 Log: safety check buf_ring_peek - assumes single-consumer Modified: user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Modified: user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Fri Nov 21 04:44:49 2008 (r185144) +++ user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Fri Nov 21 04:45:53 2008 (r185145) @@ -233,6 +233,11 @@ static __inline void * buf_ring_peek(struct buf_ring *br) { +#ifdef DEBUG_BUFRING + if (!mtx_owned(br->br_lock)) + panic("lock not held on single consumer dequeue"); +#endif + mb(); return (br->br_ring[br->br_cons_tail]); } From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 04:47:11 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6476C1065672; Fri, 21 Nov 2008 04:47:11 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 4E4FB8FC23; Fri, 21 Nov 2008 04:47:11 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL4lBUq074303; Fri, 21 Nov 2008 04:47:11 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL4lBuG074302; Fri, 21 Nov 2008 04:47:11 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210447.mAL4lBuG074302@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 04:47:11 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185146 - user/kmacy/HEAD_fast_multi_xmit/sys/sys X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 04:47:11 -0000 Author: kmacy Date: Fri Nov 21 04:47:11 2008 New Revision: 185146 URL: http://svn.freebsd.org/changeset/base/185146 Log: generalize sanity check in buf_ring_peek to multi-consumer case Modified: user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Modified: user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Fri Nov 21 04:45:53 2008 (r185145) +++ user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Fri Nov 21 04:47:11 2008 (r185146) @@ -234,7 +234,7 @@ buf_ring_peek(struct buf_ring *br) { #ifdef DEBUG_BUFRING - if (!mtx_owned(br->br_lock)) + if ((br->br_lock != NULL) && !mtx_owned(br->br_lock)) panic("lock not held on single consumer dequeue"); #endif mb(); From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 04:48:08 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E1D01065672; Fri, 21 Nov 2008 04:48:08 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 87E338FC1A; Fri, 21 Nov 2008 04:48:08 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL4m8lF074361; Fri, 21 Nov 2008 04:48:08 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL4m8Dg074360; Fri, 21 Nov 2008 04:48:08 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210448.mAL4m8Dg074360@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 04:48:08 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185147 - user/kmacy/HEAD_fast_multi_xmit/sys/kern X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 04:48:08 -0000 Author: kmacy Date: Fri Nov 21 04:48:08 2008 New Revision: 185147 URL: http://svn.freebsd.org/changeset/base/185147 Log: initialize br_lock Modified: user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_bufr.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_bufr.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_bufr.c Fri Nov 21 04:47:11 2008 (r185146) +++ user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_bufr.c Fri Nov 21 04:48:08 2008 (r185147) @@ -40,7 +40,7 @@ __FBSDID("$FreeBSD$"); struct buf_ring * -buf_ring_alloc(int count, struct malloc_type *type, int flags) +buf_ring_alloc(int count, struct malloc_type *type, int flags, struct mtx *lock) { struct buf_ring *br; @@ -50,7 +50,9 @@ buf_ring_alloc(int count, struct malloc_ type, flags|M_ZERO); if (br == NULL) return (NULL); - +#ifdef DEBUG_BUFRING + br->br_lock = lock; +#endif br->br_prod_size = br->br_cons_size = count; br->br_prod_mask = br->br_cons_mask = count-1; br->br_prod_head = br->br_cons_head = 0; From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 04:49:15 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8304D106564A; Fri, 21 Nov 2008 04:49:15 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 531B58FC0A; Fri, 21 Nov 2008 04:49:15 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL4nFP6074425; Fri, 21 Nov 2008 04:49:15 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL4nFX7074424; Fri, 21 Nov 2008 04:49:15 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210449.mAL4nFX7074424@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 04:49:15 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185148 - user/kmacy/HEAD_fast_multi_xmit/sys/net X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 04:49:15 -0000 Author: kmacy Date: Fri Nov 21 04:49:15 2008 New Revision: 185148 URL: http://svn.freebsd.org/changeset/base/185148 Log: check that ifp is set for rtentry Modified: user/kmacy/HEAD_fast_multi_xmit/sys/net/flowtable.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/net/flowtable.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/net/flowtable.c Fri Nov 21 04:48:08 2008 (r185147) +++ user/kmacy/HEAD_fast_multi_xmit/sys/net/flowtable.c Fri Nov 21 04:49:15 2008 (r185148) @@ -408,6 +408,9 @@ ipv4_flow_lookup_hash_internal(struct mb ((uint16_t *)key)[1] = dport; hash = hashword(key, 3, hashjitter + proto); + if (m->m_pkthdr.flowid == 0) + m->m_pkthdr.flowid = hash; + CTR5(KTR_SPARE3, "proto=%d hash=%x key[0]=%x sport=%d dport=%d\n", proto, hash, key[0], sport, dport); return (hash); @@ -465,7 +468,8 @@ flow_stale(struct flowtable *ft, struct || (fle->f_uptime <= fle->f_rt->rt_llinfo_uptime) || ((fle->f_rt->rt_flags & RTF_GATEWAY) && ((fle->f_rt->rt_gwroute->rt_flags & (RTF_UP|RTF_LLINFO)) - != (RTF_UP|RTF_LLINFO)))) + != (RTF_UP|RTF_LLINFO))) + || (fle->f_rt->rt_ifp == NULL)) return (1); idle_time = time_uptime - fle->f_uptime; @@ -602,7 +606,6 @@ gw_valid(struct rtentry *rt) && (rt->rt_gwroute->rt_flags & RTF_UP))); } - int flowtable_lookup(struct flowtable *ft, struct mbuf *m, struct rtentry_info *ri) @@ -626,8 +629,6 @@ flowtable_lookup(struct flowtable *ft, s hash = ipv4_flow_lookup_hash_internal(m, &ro, key, &flags, &proto); - if (m->m_pkthdr.flowid == 0) - m->m_pkthdr.flowid = hash; /* * Ports are zero and this isn't a transmit cache * - thus not a protocol for which we need to keep @@ -646,6 +647,7 @@ flowtable_lookup(struct flowtable *ft, s && flowtable_key_equal(fle, key, flags) && (proto == fle->f_proto) && (rt->rt_flags & RTF_UP) + && (rt->rt_ifp != NULL) && (fle->f_uptime > rt->rt_llinfo_uptime) && gw_valid(rt)) { fle->f_uptime = time_uptime; @@ -674,7 +676,7 @@ uncached: if (ro.ro_rt == NULL) error = ENETUNREACH; else { - int finsert = 0; + int finsert; if (ro.ro_rt->rt_flags & RTF_GATEWAY) error = arpresolve(ro.ro_rt->rt_ifp, ro.ro_rt, @@ -683,7 +685,8 @@ uncached: error = arpresolve(ro.ro_rt->rt_ifp, ro.ro_rt, NULL, &ro.ro_dst, desten); #ifdef DIAGNOSTICS - log(LOG_WARNING, "dst=%s gw=%s proto=%d hash=%x " + if (error) + log(LOG_WARNING, "dst=%s gw=%s proto=%d hash=%x " "rt_flags=%lx error=%d\n", inet_ntoa(((struct sockaddr_in*)&ro.ro_dst)->sin_addr), inet_ntoa(((struct sockaddr_in*)&ro.ro_rt->rt_gateway)->sin_addr), From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 06:01:05 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ECD321065670; Fri, 21 Nov 2008 06:01:05 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id A3A388FC0C; Fri, 21 Nov 2008 06:01:05 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL615Yi075919; Fri, 21 Nov 2008 06:01:05 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL615QN075918; Fri, 21 Nov 2008 06:01:05 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210601.mAL615QN075918@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 06:01:05 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185149 - user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 06:01:06 -0000 Author: kmacy Date: Fri Nov 21 06:01:05 2008 New Revision: 185149 URL: http://svn.freebsd.org/changeset/base/185149 Log: enable ALTQ support using ifp's sendq Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Fri Nov 21 04:49:15 2008 (r185148) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Fri Nov 21 06:01:05 2008 (r185149) @@ -148,7 +148,11 @@ cxgb_pcpu_enqueue_packet(struct ifnet *i qidx = 0; #endif qs = &pi->adapter->sge.qs[qidx]; - err = cxgb_pcpu_enqueue_packet_(qs, m); + if (ALTQ_IS_ENABLED(&ifp->if_snd)) { + IFQ_ENQUEUE(&ifp->if_snd, m, err); + } else { + err = cxgb_pcpu_enqueue_packet_(qs, m); + } return (err); } @@ -174,7 +178,15 @@ cxgb_dequeue_packet(struct sge_txq *txq, m_vec[0] = m; return (1); #endif - + if (ALTQ_ENABLED(&pi->ifp->if_snd)) { + IFQ_DRV_DEQUEUE(&pi->ifp->if_snd, m); + if (m == NULL) + return (0); + + m_vec[0] = m; + return (1); + } + mtx_assert(&txq->lock, MA_OWNED); coalesced = count = size = 0; qs = txq_to_qset(txq, TXQ_ETH); @@ -294,7 +306,8 @@ cxgb_pcpu_start_(struct sge_qset *qs, st initerr = ENXIO; else if (immpkt) { - if (!buf_ring_empty(txq->txq_mr)) + if (!buf_ring_empty(txq->txq_mr) + || ALTQ_ENABLED(&pi->ifp->if_snd)) initerr = cxgb_pcpu_enqueue_packet_(qs, immpkt); else txq->immpkt = immpkt; From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 08:00:41 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8CA75106564A; Fri, 21 Nov 2008 08:00:41 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 7B1EC8FC08; Fri, 21 Nov 2008 08:00:41 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL80fM5078489; Fri, 21 Nov 2008 08:00:41 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL80fkl078487; Fri, 21 Nov 2008 08:00:41 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210800.mAL80fkl078487@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 08:00:41 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185150 - user/kmacy/HEAD_fast_multi_xmit/sys/net X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 08:00:41 -0000 Author: kmacy Date: Fri Nov 21 08:00:41 2008 New Revision: 185150 URL: http://svn.freebsd.org/changeset/base/185150 Log: abstract out ifq initialization and teardown to allow drivers to maintain their own Modified: user/kmacy/HEAD_fast_multi_xmit/sys/net/if.c user/kmacy/HEAD_fast_multi_xmit/sys/net/if_var.h Modified: user/kmacy/HEAD_fast_multi_xmit/sys/net/if.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/net/if.c Fri Nov 21 06:01:05 2008 (r185149) +++ user/kmacy/HEAD_fast_multi_xmit/sys/net/if.c Fri Nov 21 08:00:41 2008 (r185150) @@ -480,6 +480,28 @@ if_free_type(struct ifnet *ifp, u_char t free(ifp, M_IFNET); }; +void +ifq_attach(struct ifaltq *ifq, char *buf) +{ + + mtx_init(ifq->ifq_mtx, buf, "if send queue", MTX_DEF); + + if (ifq->ifq_maxlen == 0) + ifq->ifq_maxlen = ifqmaxlen; + + ifq->altq_type = 0; + ifq->altq_disc = NULL; + ifq->altq_flags &= ALTQF_CANTCHANGE; + ifq->altq_tbr = NULL; + ifq->altq_ifp = ifp; +} + +void +ifq_detach(struct ifaltq *ifq) +{ + mtx_destroy(ifq->ifq_mtx); +} + /* * Perform generic interface initalization tasks and attach the interface * to the list of "active" interfaces. @@ -522,6 +544,7 @@ if_attach(struct ifnet *ifp) ifp->if_data.ifi_epoch = time_uptime; ifp->if_data.ifi_datalen = sizeof(struct if_data); ifp->if_transmit = if_transmit; + ifp->if_qflush = if_qflush; #ifdef MAC mac_ifnet_init(ifp); mac_ifnet_create(ifp); @@ -533,7 +556,7 @@ if_attach(struct ifnet *ifp) make_dev_alias(ifdev_byindex(ifp->if_index), "%s%d", net_cdevsw.d_name, ifp->if_index); - mtx_init(&ifp->if_snd.ifq_mtx, ifp->if_xname, "if send queue", MTX_DEF); + ifq_attach(&ifp->if_snd, ifp->if_xname); /* * create a Link Level name for this device @@ -571,19 +594,6 @@ if_attach(struct ifnet *ifp) TAILQ_INSERT_HEAD(&ifp->if_addrhead, ifa, ifa_link); ifp->if_broadcastaddr = NULL; /* reliably crash if used uninitialized */ - /* - * XXX: why do we warn about this? We're correcting it and most - * drivers just set the value the way we do. - */ - if (ifp->if_snd.ifq_maxlen == 0) { - if_printf(ifp, "XXX: driver didn't set ifq_maxlen\n"); - ifp->if_snd.ifq_maxlen = ifqmaxlen; - } - ifp->if_snd.altq_type = 0; - ifp->if_snd.altq_disc = NULL; - ifp->if_snd.altq_flags &= ALTQF_CANTCHANGE; - ifp->if_snd.altq_tbr = NULL; - ifp->if_snd.altq_ifp = ifp; IFNET_WLOCK(); TAILQ_INSERT_TAIL(&V_ifnet, ifp, if_link); @@ -825,7 +835,7 @@ if_detach(struct ifnet *ifp) KNOTE_UNLOCKED(&ifp->if_klist, NOTE_EXIT); knlist_clear(&ifp->if_klist, 0); knlist_destroy(&ifp->if_klist); - mtx_destroy(&ifp->if_snd.ifq_mtx); + ifq_detach(&ifp->if_snd); IF_AFDATA_DESTROY(ifp); splx(s); } @@ -1376,7 +1386,8 @@ if_unroute(struct ifnet *ifp, int flag, TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) if (fam == PF_UNSPEC || (fam == ifa->ifa_addr->sa_family)) pfctlinput(PRC_IFDOWN, ifa->ifa_addr); - if_qflush(&ifp->if_snd); + ifp->if_qflush(ifp); + #ifdef DEV_CARP if (ifp->if_carp) carp_carpdev_state(ifp->if_carp); @@ -1506,10 +1517,12 @@ if_up(struct ifnet *ifp) * Flush an interface queue. */ static void -if_qflush(struct ifaltq *ifq) +if_qflush(struct ifnet *ifp) { struct mbuf *m, *n; - + struct ifaltq *ifq; + + ifq = &ifp->if_snd; IFQ_LOCK(ifq); #ifdef ALTQ if (ALTQ_IS_ENABLED(ifq)) Modified: user/kmacy/HEAD_fast_multi_xmit/sys/net/if_var.h ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/net/if_var.h Fri Nov 21 06:01:05 2008 (r185149) +++ user/kmacy/HEAD_fast_multi_xmit/sys/net/if_var.h Fri Nov 21 08:00:41 2008 (r185150) @@ -186,7 +186,9 @@ struct ifnet { /* protected by if_addr_mtx */ void *if_pf_kif; void *if_lagg; /* lagg glue */ - void *if_pspare[9]; /* multiq/TOE 3; vimage 3; general use 4 */ + void *if_pspare[8]; /* multiq/TOE 3; vimage 3; general use 4 */ + int (*if_qflush) /* flush any queues */ + (struct ifnet *); int (*if_transmit) /* initiate output routine */ (struct ifnet *, struct mbuf *); int if_ispare[2]; /* general use 2 */ @@ -688,6 +690,9 @@ int ifioctl(struct socket *, u_long, cad int ifpromisc(struct ifnet *, int); struct ifnet *ifunit(const char *); +void ifq_attach(struct ifaltq *); +void ifq_detach(struct ifaltq *); + struct ifaddr *ifa_ifwithaddr(struct sockaddr *); struct ifaddr *ifa_ifwithbroadaddr(struct sockaddr *); struct ifaddr *ifa_ifwithdstaddr(struct sockaddr *); From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 08:19:21 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F71D1065673; Fri, 21 Nov 2008 08:19:21 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 3EC648FC1B; Fri, 21 Nov 2008 08:19:21 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL8JL8e078955; Fri, 21 Nov 2008 08:19:21 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL8JLu0078953; Fri, 21 Nov 2008 08:19:21 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210819.mAL8JLu0078953@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 08:19:21 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185152 - user/kmacy/HEAD_fast_multi_xmit/sys/net X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 08:19:21 -0000 Author: kmacy Date: Fri Nov 21 08:19:20 2008 New Revision: 185152 URL: http://svn.freebsd.org/changeset/base/185152 Log: compile fixes for previous commit Modified: user/kmacy/HEAD_fast_multi_xmit/sys/net/if.c user/kmacy/HEAD_fast_multi_xmit/sys/net/if_var.h Modified: user/kmacy/HEAD_fast_multi_xmit/sys/net/if.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/net/if.c Fri Nov 21 08:10:07 2008 (r185151) +++ user/kmacy/HEAD_fast_multi_xmit/sys/net/if.c Fri Nov 21 08:19:20 2008 (r185152) @@ -113,7 +113,7 @@ static int ifconf(u_long, caddr_t); static void if_freemulti(struct ifmultiaddr *); static void if_grow(void); static void if_init(void *); -static void if_qflush(struct ifaltq *); +static void if_qflush(struct ifnet *); static void if_route(struct ifnet *, int flag, int fam); static int if_setflag(struct ifnet *, int, int, int *, int); static void if_slowtimo(void *); @@ -481,10 +481,10 @@ if_free_type(struct ifnet *ifp, u_char t }; void -ifq_attach(struct ifaltq *ifq, char *buf) +ifq_attach(struct ifaltq *ifq, struct ifnet *ifp) { - mtx_init(ifq->ifq_mtx, buf, "if send queue", MTX_DEF); + mtx_init(&ifq->ifq_mtx, ifp->if_xname, "if send queue", MTX_DEF); if (ifq->ifq_maxlen == 0) ifq->ifq_maxlen = ifqmaxlen; @@ -499,7 +499,7 @@ ifq_attach(struct ifaltq *ifq, char *buf void ifq_detach(struct ifaltq *ifq) { - mtx_destroy(ifq->ifq_mtx); + mtx_destroy(&ifq->ifq_mtx); } /* @@ -556,7 +556,7 @@ if_attach(struct ifnet *ifp) make_dev_alias(ifdev_byindex(ifp->if_index), "%s%d", net_cdevsw.d_name, ifp->if_index); - ifq_attach(&ifp->if_snd, ifp->if_xname); + ifq_attach(&ifp->if_snd, ifp); /* * create a Link Level name for this device Modified: user/kmacy/HEAD_fast_multi_xmit/sys/net/if_var.h ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/net/if_var.h Fri Nov 21 08:10:07 2008 (r185151) +++ user/kmacy/HEAD_fast_multi_xmit/sys/net/if_var.h Fri Nov 21 08:19:20 2008 (r185152) @@ -187,7 +187,7 @@ struct ifnet { void *if_pf_kif; void *if_lagg; /* lagg glue */ void *if_pspare[8]; /* multiq/TOE 3; vimage 3; general use 4 */ - int (*if_qflush) /* flush any queues */ + void (*if_qflush) /* flush any queues */ (struct ifnet *); int (*if_transmit) /* initiate output routine */ (struct ifnet *, struct mbuf *); @@ -690,7 +690,7 @@ int ifioctl(struct socket *, u_long, cad int ifpromisc(struct ifnet *, int); struct ifnet *ifunit(const char *); -void ifq_attach(struct ifaltq *); +void ifq_attach(struct ifaltq *, struct ifnet *ifp); void ifq_detach(struct ifaltq *); struct ifaddr *ifa_ifwithaddr(struct sockaddr *); From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 08:20:28 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E2FBA1065670; Fri, 21 Nov 2008 08:20:28 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id D268E8FC12; Fri, 21 Nov 2008 08:20:28 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAL8KSAF079017; Fri, 21 Nov 2008 08:20:28 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAL8KS8l079014; Fri, 21 Nov 2008 08:20:28 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811210820.mAL8KS8l079014@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 08:20:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185153 - user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 08:20:29 -0000 Author: kmacy Date: Fri Nov 21 08:20:28 2008 New Revision: 185153 URL: http://svn.freebsd.org/changeset/base/185153 Log: add per-hardware queue ifaltq to enable altq with multiple transmit queues Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_adapter.h user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_adapter.h ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_adapter.h Fri Nov 21 08:19:20 2008 (r185152) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_adapter.h Fri Nov 21 08:20:28 2008 (r185153) @@ -260,7 +260,9 @@ struct sge_txq { */ struct mbuf_head cleanq; struct buf_ring *txq_mr; + struct ifaltq *txq_ifq; struct mbuf *immpkt; + uint32_t txq_drops; uint32_t txq_skipped; uint32_t txq_coalesced; Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Fri Nov 21 08:19:20 2008 (r185152) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Fri Nov 21 08:20:28 2008 (r185153) @@ -149,7 +149,7 @@ cxgb_pcpu_enqueue_packet(struct ifnet *i #endif qs = &pi->adapter->sge.qs[qidx]; if (ALTQ_IS_ENABLED(&ifp->if_snd)) { - IFQ_ENQUEUE(&ifp->if_snd, m, err); + IFQ_ENQUEUE(qs->txq[0].txq_ifq, m, err); } else { err = cxgb_pcpu_enqueue_packet_(qs, m); } @@ -178,8 +178,8 @@ cxgb_dequeue_packet(struct sge_txq *txq, m_vec[0] = m; return (1); #endif - if (ALTQ_ENABLED(&pi->ifp->if_snd)) { - IFQ_DRV_DEQUEUE(&pi->ifp->if_snd, m); + if (ALTQ_IS_ENABLED(txq->txq_ifq)) { + IFQ_DRV_DEQUEUE(txq->txq_ifq, m); if (m == NULL) return (0); @@ -307,7 +307,7 @@ cxgb_pcpu_start_(struct sge_qset *qs, st else if (immpkt) { if (!buf_ring_empty(txq->txq_mr) - || ALTQ_ENABLED(&pi->ifp->if_snd)) + || ALTQ_IS_ENABLED(&pi->ifp->if_snd)) initerr = cxgb_pcpu_enqueue_packet_(qs, immpkt); else txq->immpkt = immpkt; Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Fri Nov 21 08:19:20 2008 (r185152) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Fri Nov 21 08:20:28 2008 (r185153) @@ -1714,9 +1714,15 @@ t3_free_qset(adapter_t *sc, struct sge_q t3_free_tx_desc_all(&q->txq[TXQ_ETH]); - for (i = 0; i < SGE_TXQ_PER_SET; i++) + for (i = 0; i < SGE_TXQ_PER_SET; i++) { if (q->txq[i].txq_mr != NULL) buf_ring_free(q->txq[i].txq_mr, M_DEVBUF); + if (q->txq[i].txq_ifq != NULL) { + ifq_detach(q->txq[i].txq_ifq); + free(q->txq[i].txq_ifq, M_DEVBUF); + } + } + for (i = 0; i < SGE_RXQ_PER_SET; ++i) { if (q->fl[i].desc) { mtx_lock_spin(&sc->sge.reg_lock); @@ -2281,6 +2287,13 @@ t3_sge_alloc_qset(adapter_t *sc, u_int i device_printf(sc->dev, "failed to allocate mbuf ring\n"); goto err; } + if ((q->txq[i].txq_ifq = + malloc(sizeof(struct ifaltq), M_DEVBUF, M_NOWAIT|M_ZERO)) + == NULL) { + device_printf(sc->dev, "failed to allocate ifq\n"); + goto err; + } + ifq_attach(q->txq[i].txq_ifq, pi->ifp); } init_qset_cntxt(q, id); q->idx = id; From owner-svn-src-user@FreeBSD.ORG Fri Nov 21 23:59:54 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A2764106564A; Fri, 21 Nov 2008 23:59:54 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 8A8248FC14; Fri, 21 Nov 2008 23:59:54 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mALNxsHl000100; Fri, 21 Nov 2008 23:59:54 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mALNxs5x099998; Fri, 21 Nov 2008 23:59:54 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811212359.mALNxs5x099998@svn.freebsd.org> From: Kip Macy Date: Fri, 21 Nov 2008 23:59:54 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185158 - user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 23:59:54 -0000 Author: kmacy Date: Fri Nov 21 23:59:54 2008 New Revision: 185158 URL: http://svn.freebsd.org/changeset/base/185158 Log: - Dramatically reduce the number of context switches by not calling wakeup In practice the worst case extra millisecond we wait for the service thread to push out stragglers rarely hurts. Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Fri Nov 21 19:22:25 2008 (r185157) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Fri Nov 21 23:59:54 2008 (r185158) @@ -127,9 +127,6 @@ cxgb_pcpu_enqueue_packet_(struct sge_qse txq->txq_drops++; m_freem(m); } - if ((qs->txq[TXQ_ETH].flags & TXQ_TRANSMITTING) == 0) - wakeup(qs); - return (err); } @@ -415,10 +412,6 @@ cxgb_pcpu_transmit(struct ifnet *ifp, st sc->tunq_coalesce, buf_ring_count(txq->txq_mr), mtx_owned(&txq->lock)); err = cxgb_pcpu_enqueue_packet_(qs, immpkt); } - - if (resid && (txq->flags & TXQ_TRANSMITTING) == 0) - wakeup(qs); - return ((err == ENOSPC) ? 0 : err); } From owner-svn-src-user@FreeBSD.ORG Sat Nov 22 00:24:23 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C354D106564A; Sat, 22 Nov 2008 00:24:23 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id AA47F8FC0C; Sat, 22 Nov 2008 00:24:23 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAM0ONRE000636; Sat, 22 Nov 2008 00:24:23 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAM0ONho000634; Sat, 22 Nov 2008 00:24:23 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811220024.mAM0ONho000634@svn.freebsd.org> From: Kip Macy Date: Sat, 22 Nov 2008 00:24:23 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185159 - user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Nov 2008 00:24:23 -0000 Author: kmacy Date: Sat Nov 22 00:24:23 2008 New Revision: 185159 URL: http://svn.freebsd.org/changeset/base/185159 Log: although it ends up sucking up substantially more cpu waking up the transmit thread gives slightly better peak throughput Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Fri Nov 21 23:59:54 2008 (r185158) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_multiq.c Sat Nov 22 00:24:23 2008 (r185159) @@ -89,6 +89,7 @@ __FBSDID("$FreeBSD$"); extern int txq_fills; int multiq_tx_enable = 1; int coalesce_tx_enable = 0; +int wakeup_tx_thread = 0; extern struct sysctl_oid_list sysctl__hw_cxgb_children; static int sleep_ticks = 1; @@ -127,6 +128,9 @@ cxgb_pcpu_enqueue_packet_(struct sge_qse txq->txq_drops++; m_freem(m); } + if (wakeup_tx_thread && ((txq->flags & TXQ_TRANSMITTING) == 0)) + wakeup(qs); + return (err); } Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Fri Nov 21 23:59:54 2008 (r185158) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Sat Nov 22 00:24:23 2008 (r185159) @@ -86,6 +86,7 @@ extern int nmbjumbo9; extern int nmbjumbo16; extern int multiq_tx_enable; extern int coalesce_tx_enable; +extern int wakeup_tx_thread; #define USE_GTS 0 @@ -3377,6 +3378,10 @@ t3_add_attach_sysctls(adapter_t *sc) CTLFLAG_RW, &coalesce_tx_enable, 0, "coalesce small packets in work requests - WARNING ALPHA"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, + "wakeup_tx_thread", + CTLFLAG_RW, &wakeup_tx_thread, + 0, "wakeup tx thread if no transmitter running"); + SYSCTL_ADD_INT(ctx, children, OID_AUTO, "cache_alloc", CTLFLAG_RD, &cxgb_cached_allocations, 0, "#times a cluster was allocated from cache"); From owner-svn-src-user@FreeBSD.ORG Sat Nov 22 01:35:59 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DA292106564A; Sat, 22 Nov 2008 01:35:59 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id BE4D88FC0C; Sat, 22 Nov 2008 01:35:59 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAM1Zxmg002129; Sat, 22 Nov 2008 01:35:59 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAM1Zx2g002127; Sat, 22 Nov 2008 01:35:59 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811220135.mAM1Zx2g002127@svn.freebsd.org> From: Kip Macy Date: Sat, 22 Nov 2008 01:35:59 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185160 - in user/kmacy/HEAD_fast_multi_xmit/sys: conf kern X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Nov 2008 01:36:00 -0000 Author: kmacy Date: Sat Nov 22 01:35:59 2008 New Revision: 185160 URL: http://svn.freebsd.org/changeset/base/185160 Log: rename subr_bufr.c to subr_bufring.c Added: user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_bufring.c (props changed) - copied unchanged from r185147, user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_bufr.c Deleted: user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_bufr.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/conf/files Modified: user/kmacy/HEAD_fast_multi_xmit/sys/conf/files ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/conf/files Sat Nov 22 00:24:23 2008 (r185159) +++ user/kmacy/HEAD_fast_multi_xmit/sys/conf/files Sat Nov 22 01:35:59 2008 (r185160) @@ -1805,7 +1805,7 @@ kern/subr_acl_posix1e.c standard kern/subr_autoconf.c standard kern/subr_blist.c standard kern/subr_bus.c standard -kern/subr_bufr.c standard +kern/subr_bufring.c standard kern/subr_clist.c standard kern/subr_clock.c standard kern/subr_devstat.c standard Copied: user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_bufring.c (from r185147, user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_bufr.c) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_bufring.c Sat Nov 22 01:35:59 2008 (r185160, copy of r185147, user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_bufr.c) @@ -0,0 +1,68 @@ +/************************************************************************** + * + * Copyright (c) 2007,2008 Kip Macy kmacy@freebsd.org + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * 1. Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * + * 2. The name of Kip Macy nor the names of other + * contributors may be used to endorse or promote products derived from + * this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * + * + ***************************************************************************/ +#include +__FBSDID("$FreeBSD$"); + + +#include +#include +#include +#include +#include +#include + + +struct buf_ring * +buf_ring_alloc(int count, struct malloc_type *type, int flags, struct mtx *lock) +{ + struct buf_ring *br; + + KASSERT(powerof2(count), ("buf ring must be size power of 2")); + + br = malloc(sizeof(struct buf_ring) + count*sizeof(caddr_t), + type, flags|M_ZERO); + if (br == NULL) + return (NULL); +#ifdef DEBUG_BUFRING + br->br_lock = lock; +#endif + br->br_prod_size = br->br_cons_size = count; + br->br_prod_mask = br->br_cons_mask = count-1; + br->br_prod_head = br->br_cons_head = 0; + br->br_prod_tail = br->br_cons_tail = 0; + + return (br); +} + +void +buf_ring_free(struct buf_ring *br, struct malloc_type *type) +{ + free(br, type); +} From owner-svn-src-user@FreeBSD.ORG Sat Nov 22 08:48:01 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ED8BB1065670; Sat, 22 Nov 2008 08:48:01 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id D6B4D8FC16; Sat, 22 Nov 2008 08:48:01 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAM8m1LJ011463; Sat, 22 Nov 2008 08:48:01 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAM8m1QJ011461; Sat, 22 Nov 2008 08:48:01 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200811220848.mAM8m1QJ011461@svn.freebsd.org> From: Kip Macy Date: Sat, 22 Nov 2008 08:48:01 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185167 - in user/kmacy/HEAD_fast_multi_xmit/sys: dev/cxgb sys X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Nov 2008 08:48:02 -0000 Author: kmacy Date: Sat Nov 22 08:48:01 2008 New Revision: 185167 URL: http://svn.freebsd.org/changeset/base/185167 Log: add buf ring counter for debugging Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Modified: user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Sat Nov 22 08:46:16 2008 (r185166) +++ user/kmacy/HEAD_fast_multi_xmit/sys/dev/cxgb/cxgb_sge.c Sat Nov 22 08:48:01 2008 (r185167) @@ -3513,22 +3513,19 @@ t3_add_configured_sysctls(adapter_t *sc) SYSCTL_ADD_PROC(ctx, rspqpoidlist, OID_AUTO, "qdump", CTLTYPE_STRING | CTLFLAG_RD, &qs->rspq, 0, t3_dump_rspq, "A", "dump of the response queue"); +#ifdef DEBUG_BUFRING - + SYSCTL_ADD_INT(ctx, txqpoidlist, OID_AUTO, + "buffered_count", CTLFLAG_RD, + &qs->txq[TXQ_ETH].txq_mr->br_count, + 0, "#buf ring packets enqueued"); +#endif SYSCTL_ADD_INT(ctx, txqpoidlist, OID_AUTO, "dropped", CTLFLAG_RD, &qs->txq[TXQ_ETH].txq_drops, 0, "#tunneled packets dropped"); SYSCTL_ADD_INT(ctx, txqpoidlist, OID_AUTO, "sendqlen", CTLFLAG_RD, &qs->txq[TXQ_ETH].sendq.qlen, 0, "#tunneled packets waiting to be sent"); -#if 0 - SYSCTL_ADD_UINT(ctx, txqpoidlist, OID_AUTO, "queue_pidx", - CTLFLAG_RD, (uint32_t *)(uintptr_t)&qs->txq[TXQ_ETH].txq_mr.br_prod, - 0, "#tunneled packets queue producer index"); - SYSCTL_ADD_UINT(ctx, txqpoidlist, OID_AUTO, "queue_cidx", - CTLFLAG_RD, (uint32_t *)(uintptr_t)&qs->txq[TXQ_ETH].txq_mr.br_cons, - 0, "#tunneled packets queue consumer index"); -#endif SYSCTL_ADD_INT(ctx, txqpoidlist, OID_AUTO, "processed", CTLFLAG_RD, &qs->txq[TXQ_ETH].processed, 0, "#tunneled packets processed by the card"); Modified: user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Sat Nov 22 08:46:16 2008 (r185166) +++ user/kmacy/HEAD_fast_multi_xmit/sys/sys/buf_ring.h Sat Nov 22 08:48:01 2008 (r185167) @@ -56,6 +56,8 @@ #error "unknown compiler" #endif +#define DEBUG_BUFRING 1 + #if defined(INVARIANTS) && !defined(DEBUG_BUFRING) #define DEBUG_BUFRING 1 #endif @@ -86,6 +88,7 @@ struct buf_ring { uint64_t _pad1[14]; #ifdef DEBUG_BUFRING struct mtx *br_lock; + uint32_t br_count; #endif void *br_ring[0]; }; @@ -123,7 +126,9 @@ buf_ring_enqueue(struct buf_ring *br, vo #ifdef DEBUG_BUFRING if (br->br_ring[prod_head] != NULL) panic("dangling value in enqueue"); -#endif + + atomic_add_int(&br->br_count, 1); +#endif br->br_ring[prod_head] = buf; wmb(); @@ -171,6 +176,7 @@ buf_ring_dequeue_mc(struct buf_ring *br) buf = br->br_ring[cons_head]; #ifdef DEBUG_BUFRING br->br_ring[cons_head] = NULL; + atomic_subtract_int(&br->br_count, 1); #endif mb(); @@ -216,6 +222,7 @@ buf_ring_dequeue_sc(struct buf_ring *br) mb(); #ifdef DEBUG_BUFRING + atomic_subtract_int(&br->br_count, 1); br->br_ring[cons_head] = NULL; if (!mtx_owned(br->br_lock)) panic("lock not held on single consumer dequeue"); From owner-svn-src-user@FreeBSD.ORG Sat Nov 22 16:12:58 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 605421065673; Sat, 22 Nov 2008 16:12:58 +0000 (UTC) (envelope-from dfr@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 278018FC33; Sat, 22 Nov 2008 16:12:57 +0000 (UTC) (envelope-from dfr@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAMGCvvE022487; Sat, 22 Nov 2008 16:12:57 GMT (envelope-from dfr@svn.freebsd.org) Received: (from dfr@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAMGCvAO022486; Sat, 22 Nov 2008 16:12:57 GMT (envelope-from dfr@svn.freebsd.org) Message-Id: <200811221612.mAMGCvAO022486@svn.freebsd.org> From: Doug Rabson Date: Sat, 22 Nov 2008 16:12:57 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185180 - user/dfr/xenhvm X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Nov 2008 16:12:58 -0000 Author: dfr Date: Sat Nov 22 16:12:57 2008 New Revision: 185180 URL: http://svn.freebsd.org/changeset/base/185180 Log: A place to work on Xen HVM mode support. Added: user/dfr/xenhvm/ From owner-svn-src-user@FreeBSD.ORG Sat Nov 22 16:14:52 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB645106564A; Sat, 22 Nov 2008 16:14:52 +0000 (UTC) (envelope-from dfr@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 7DC7D8FC17; Sat, 22 Nov 2008 16:14:52 +0000 (UTC) (envelope-from dfr@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mAMGEqv7022580; Sat, 22 Nov 2008 16:14:52 GMT (envelope-from dfr@svn.freebsd.org) Received: (from dfr@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mAMGEqvD022579; Sat, 22 Nov 2008 16:14:52 GMT (envelope-from dfr@svn.freebsd.org) Message-Id: <200811221614.mAMGEqvD022579@svn.freebsd.org> From: Doug Rabson Date: Sat, 22 Nov 2008 16:14:52 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185181 - user/dfr/xenhvm/6 X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Nov 2008 16:14:52 -0000 Author: dfr Date: Sat Nov 22 16:14:52 2008 New Revision: 185181 URL: http://svn.freebsd.org/changeset/base/185181 Log: Clone Kip's Xen on stable/6 tree so that I can work on improving FreeBSD/amd64 performance in Xen's HVM mode. Added: user/dfr/xenhvm/6/ - copied from r185180, projects/releng_6_xen/