From owner-freebsd-net@FreeBSD.ORG Tue Sep 14 22:02:43 2004 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DFADB16A4CE for ; Tue, 14 Sep 2004 22:02:42 +0000 (GMT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7992043D39 for ; Tue, 14 Sep 2004 22:02:42 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.10/8.12.10) with ESMTP id i8EM2gJt000955 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 14 Sep 2004 18:02:42 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id i8EM2a8R069146; Tue, 14 Sep 2004 18:02:36 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MyLggxmwdT" Content-Transfer-Encoding: 7bit Message-ID: <16711.27260.667013.411961@grasshopper.cs.duke.edu> Date: Tue, 14 Sep 2004 18:02:36 -0400 (EDT) To: "Don Bowman" In-Reply-To: <16710.3441.49431.847790@grasshopper.cs.duke.edu> References: <16710.3441.49431.847790@grasshopper.cs.duke.edu> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid cc: freebsd-net@freebsd.org Subject: RE: packet generator X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Sep 2004 22:02:43 -0000 --MyLggxmwdT Content-Type: text/plain; charset=us-ascii Content-Description: message body text Content-Transfer-Encoding: 7bit Andrew Gallatin writes: > xmit routine was called 683441 times. This means that the queue was > only a little over two packets deep on average, and vmstat shows idle > time. I've tried piping additional packets to nghook mx0:orphans > input, but that does not seem to increase the queue depth. > The problem here seems to be that rather than just slapping the packets onto the driver's queue, ng_source passes the mbuf down to more of netgraph, where there is at least one spinlock, and the driver's ifq lock is taken and released a zillion times by ether_output_frame(), etc. A quick hack (appended) to just slap the mbufs onto the if_snd queue gets me from ~410Kpps to 1020Kpps. I also see very deep queues with this (because I'm slamming 4K pkts onto the queue at once..). This is nearly identical to the linux pktgen figure on the same hardware, which makes me feel comfortable that there is a lot of headroom in the driver/firmware API and I'm not botching something in the FreeBSD driver. BTW, did you see your 800Kpps on 4.x or 5.x? If it was 4.x, what do you see on 5.x if you still have the same setup handy? Thanks, Drew --MyLggxmwdT Content-Type: application/octet-stream Content-Description: diff to make ng_source skip netgraph detour Content-Disposition: attachment; filename="ng_source.diff" Content-Transfer-Encoding: base64 SW5kZXg6IG5nX3NvdXJjZS5jCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KUkNTIGZpbGU6IC9ob21lL25jdnMv c3JjL3N5cy9uZXRncmFwaC9uZ19zb3VyY2UuYyx2CnJldHJpZXZpbmcgcmV2aXNpb24gMS4x OApkaWZmIC11IC1yMS4xOCBuZ19zb3VyY2UuYwotLS0gbmdfc291cmNlLmMJMjAgSnVsIDIw MDQgMTc6MTU6MzcgLTAwMDAJMS4xOAorKysgbmdfc291cmNlLmMJMTQgU2VwIDIwMDQgMjE6 MzY6MTQgLTAwMDAKQEAgLTYzNyw3ICs2MzcsNyBAQAogc3RhdGljIGludAogbmdfc291cmNl X3NlbmQgKHNjX3Agc2MsIGludCB0b3NlbmQsIGludCAqc2VudF9wKQogewotCXN0cnVjdCBp ZnF1ZXVlIHRtcF9xdWV1ZTsKKwlzdHJ1Y3QgaWZxdWV1ZSB0bXBfcXVldWUsICppZnE7CiAJ c3RydWN0IG1idWYgKm0sICptMjsKIAlpbnQgc2VudCA9IDA7CiAJaW50IGVycm9yID0gMDsK QEAgLTY3Myw2ICs2NzMsOSBAQAogCX0KIAogCXNlbnQgPSAwOworCisJaWZxID0gJnNjLT5v dXRwdXRfaWZwLT5pZl9zbmQ7CQorCUlGX0xPQ0soaWZxKTsKIAlmb3IgKDs7KSB7CiAJCV9J Rl9ERVFVRVVFKCZ0bXBfcXVldWUsIG0yKTsKIAkJaWYgKG0yID09IE5VTEwpCkBAIC02ODEs MTMgKzY4NCwxNyBAQAogCQkJKytzZW50OwogCQkJc2MtPnN0YXRzLm91dEZyYW1lcysrOwog CQkJc2MtPnN0YXRzLm91dE9jdGV0cyArPSBtMi0+bV9wa3RoZHIubGVuOwotCQkJTkdfU0VO RF9EQVRBX09OTFkoZXJyb3IsIHNjLT5vdXRwdXQuaG9vaywgbTIpOworCQkvKglOR19TRU5E X0RBVEFfT05MWShlcnJvciwgc2MtPm91dHB1dC5ob29rLCBtMik7Ki8KKwkJIAlfSUZfRU5R VUVVRShpZnEsIG0yKTsKIAkJCWlmIChlcnJvcikKIAkJCQlwcmludGYoIiVzOiBlcnJvcj0l ZFxuIiwgX19mdW5jX18sIGVycm9yKTsKIAkJfSBlbHNlIHsKIAkJCU5HX0ZSRUVfTShtMik7 CiAJCX0KIAl9CisJSUZfVU5MT0NLKGlmcSk7CisJaWYgKChzYy0+b3V0cHV0X2lmcC0+aWZf ZmxhZ3MgJiBJRkZfT0FDVElWRSkgPT0gMCkKKwkJaWZfc3RhcnQoc2MtPm91dHB1dF9pZnAp OwogCiAJc2MtPnBhY2tldHMgLT0gc2VudDsKIAlpZiAoc2VudF9wICE9IE5VTEwpCg== --MyLggxmwdT--