Date: Tue, 14 Sep 2004 18:02:36 -0400 (EDT) From: Andrew Gallatin <gallatin@cs.duke.edu> To: "Don Bowman" <don@sandvine.com> Cc: freebsd-net@freebsd.org Subject: RE: packet generator Message-ID: <16711.27260.667013.411961@grasshopper.cs.duke.edu> In-Reply-To: <16710.3441.49431.847790@grasshopper.cs.duke.edu> References: <A8535F8D62F3644997E91F4F66E341FC150AE4@exchange.sandvine.com> <16710.3441.49431.847790@grasshopper.cs.duke.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
--MyLggxmwdT Content-Type: text/plain; charset=us-ascii Content-Description: message body text Content-Transfer-Encoding: 7bit Andrew Gallatin writes: > xmit routine was called 683441 times. This means that the queue was > only a little over two packets deep on average, and vmstat shows idle > time. I've tried piping additional packets to nghook mx0:orphans > input, but that does not seem to increase the queue depth. > The problem here seems to be that rather than just slapping the packets onto the driver's queue, ng_source passes the mbuf down to more of netgraph, where there is at least one spinlock, and the driver's ifq lock is taken and released a zillion times by ether_output_frame(), etc. A quick hack (appended) to just slap the mbufs onto the if_snd queue gets me from ~410Kpps to 1020Kpps. I also see very deep queues with this (because I'm slamming 4K pkts onto the queue at once..). This is nearly identical to the linux pktgen figure on the same hardware, which makes me feel comfortable that there is a lot of headroom in the driver/firmware API and I'm not botching something in the FreeBSD driver. BTW, did you see your 800Kpps on 4.x or 5.x? If it was 4.x, what do you see on 5.x if you still have the same setup handy? Thanks, Drew --MyLggxmwdT Content-Type: application/octet-stream Content-Description: diff to make ng_source skip netgraph detour Content-Disposition: attachment; filename="ng_source.diff" Content-Transfer-Encoding: base64 SW5kZXg6IG5nX3NvdXJjZS5jCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KUkNTIGZpbGU6IC9ob21lL25jdnMv c3JjL3N5cy9uZXRncmFwaC9uZ19zb3VyY2UuYyx2CnJldHJpZXZpbmcgcmV2aXNpb24gMS4x OApkaWZmIC11IC1yMS4xOCBuZ19zb3VyY2UuYwotLS0gbmdfc291cmNlLmMJMjAgSnVsIDIw MDQgMTc6MTU6MzcgLTAwMDAJMS4xOAorKysgbmdfc291cmNlLmMJMTQgU2VwIDIwMDQgMjE6 MzY6MTQgLTAwMDAKQEAgLTYzNyw3ICs2MzcsNyBAQAogc3RhdGljIGludAogbmdfc291cmNl X3NlbmQgKHNjX3Agc2MsIGludCB0b3NlbmQsIGludCAqc2VudF9wKQogewotCXN0cnVjdCBp ZnF1ZXVlIHRtcF9xdWV1ZTsKKwlzdHJ1Y3QgaWZxdWV1ZSB0bXBfcXVldWUsICppZnE7CiAJ c3RydWN0IG1idWYgKm0sICptMjsKIAlpbnQgc2VudCA9IDA7CiAJaW50IGVycm9yID0gMDsK QEAgLTY3Myw2ICs2NzMsOSBAQAogCX0KIAogCXNlbnQgPSAwOworCisJaWZxID0gJnNjLT5v dXRwdXRfaWZwLT5pZl9zbmQ7CQorCUlGX0xPQ0soaWZxKTsKIAlmb3IgKDs7KSB7CiAJCV9J Rl9ERVFVRVVFKCZ0bXBfcXVldWUsIG0yKTsKIAkJaWYgKG0yID09IE5VTEwpCkBAIC02ODEs MTMgKzY4NCwxNyBAQAogCQkJKytzZW50OwogCQkJc2MtPnN0YXRzLm91dEZyYW1lcysrOwog CQkJc2MtPnN0YXRzLm91dE9jdGV0cyArPSBtMi0+bV9wa3RoZHIubGVuOwotCQkJTkdfU0VO RF9EQVRBX09OTFkoZXJyb3IsIHNjLT5vdXRwdXQuaG9vaywgbTIpOworCQkvKglOR19TRU5E X0RBVEFfT05MWShlcnJvciwgc2MtPm91dHB1dC5ob29rLCBtMik7Ki8KKwkJIAlfSUZfRU5R VUVVRShpZnEsIG0yKTsKIAkJCWlmIChlcnJvcikKIAkJCQlwcmludGYoIiVzOiBlcnJvcj0l ZFxuIiwgX19mdW5jX18sIGVycm9yKTsKIAkJfSBlbHNlIHsKIAkJCU5HX0ZSRUVfTShtMik7 CiAJCX0KIAl9CisJSUZfVU5MT0NLKGlmcSk7CisJaWYgKChzYy0+b3V0cHV0X2lmcC0+aWZf ZmxhZ3MgJiBJRkZfT0FDVElWRSkgPT0gMCkKKwkJaWZfc3RhcnQoc2MtPm91dHB1dF9pZnAp OwogCiAJc2MtPnBhY2tldHMgLT0gc2VudDsKIAlpZiAoc2VudF9wICE9IE5VTEwpCg== --MyLggxmwdT--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?16711.27260.667013.411961>