Date: Wed, 15 Mar 2006 10:32:52 -0800 From: Wes Santee <wsantee@gmail.com> To: freebsd-pf@freebsd.org Subject: cbq and borrowing not working Message-ID: <44185DD4.8010009@gmail.com>
next in thread | raw e-mail | index | archive | help
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Greetings. I asked this same question on the pf@benzedrine.cx list and so far the only answer I've received is "yeah, it happens here also, anyone know what's going on?" Perhaps someone here can shed some light. My apologies if you're on that list also and this question is a repeat. I'm on 6-STABLE if that makes any difference. Short Version _____________ Assume altq is using only the cbq scheduler. When a parent queue "A" is not labeled for borrowing from its root queue, the child queues of "A" will not fully borrow from their parent, even if there is plenty of bandwidth available to the parent queue. Does anyone know what is happening here, or how to solve it? Long Version ____________ The setup from pf.conf is this: altq on $ext_if bandwidth 54Mb cbq queue { internal, external } queue internal bandwidth 53104Kb priority 7 cbq(borrow) queue external bandwidth 896Kb priority 4 cbq { highq, defaultq, lowq } queue highq bandwidth 40% priority 3 cbq(borrow) queue defaultq bandwidth 40% priority 2 cbq(default borrow) queue lowq bandwidth 20% priority 1 qlimit 300 cbq(borrow) The internal queue is for wireless traffic coming from and going to the internal network. The external queue (and its children) are for traffic going upstream to my ISP. The "external" queue is set to my max upstream bandwidth to my ISP, so it can't borrow from the root queue. All the child queue's under "external" can (and should) borrow against the available upstream bandwidth amount if available. The problem is that packets in lowq are filling up the queue, even though there is plenty of bandwidth available to borrow from the parent queue. Here's a snapshot from pfctl -vvsqueue: queue internal bandwidth 53.10Mb priority 7 cbq( borrow ) [ pkts: 401 bytes: 216532 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 0.0 packets/s, 0 b/s ] queue external bandwidth 896Kb priority 4 {highq, defaultq, lowq} [ pkts: 0 bytes: 0 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 0.0 packets/s, 0 b/s ] queue highq bandwidth 358.40Kb priority 3 cbq( borrow ) [ pkts: 0 bytes: 0 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 0.0 packets/s, 0 b/s ] queue defaultq bandwidth 358.40Kb priority 2 cbq( borrow default ) [ pkts: 7397 bytes: 3230740 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 10 suspends: 9 ] [ measured: 1.0 packets/s, 4.16Kb/s ] queue lowq bandwidth 179.20Kb qlimit 300 cbq( borrow ) [ pkts: 41643 bytes: 48966175 dropped pkts: 0 bytes: 0 ] [ qlength: 117/300 borrows: 31513 suspends: 6282 ] [ measured: 21.1 packets/s, 193.23Kb/s ] Notice that queue "external", the parent of lowq, has plenty of available bandwidth (if you add up the usage for its child queues). However, lowq has still got a backlog of 117 packets, and refuses to go over 193Kbit/sec. It is borrowing, but not enough to keep the queue in check. If I don't set a qlimit of 300 on lowq, the queue will fill up in about 5 seconds after starting pf. Okay, now what happens if I change queue "external" to allow borrowing from the root, ala: altq on $ext_if bandwidth 54Mb cbq queue { internal, external } queue internal bandwidth 53104Kb priority 7 cbq(borrow) queue external bandwidth 896Kb priority 4 cbq(borrow) { highq, defaultq, lowq } queue highq bandwidth 40% priority 3 cbq(borrow) queue defaultq bandwidth 40% priority 2 cbq(default borrow) queue lowq bandwidth 20% priority 1 cbq(borrow) "pfctl -vvsqueue" paints a different picture: queue internal bandwidth 53.10Mb priority 7 qlimit 200 cbq( borrow ) [ pkts: 0 bytes: 0 dropped pkts: 0 bytes: 0 ] [ qlength: 0/200 borrows: 0 suspends: 0 ] [ measured: 0.0 packets/s, 0 b/s ] queue external bandwidth 896Kb priority 4 cbq( borrow ) {highq, defaultq, lowq} [ pkts: 0 bytes: 0 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 1285 suspends: 0 ] [ measured: 0.0 packets/s, 0 b/s ] queue highq bandwidth 358.40Kb priority 3 cbq( borrow ) [ pkts: 0 bytes: 0 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 0 suspends: 0 ] [ measured: 0.0 packets/s, 0 b/s ] queue defaultq bandwidth 358.40Kb priority 2 cbq( borrow default ) [ pkts: 2781 bytes: 1226812 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 180 suspends: 0 ] [ measured: 17.5 packets/s, 66.61Kb/s ] queue lowq bandwidth 179.20Kb cbq( borrow ) [ pkts: 5189 bytes: 6068440 dropped pkts: 0 bytes: 0 ] [ qlength: 0/ 50 borrows: 4341 suspends: 0 ] [ measured: 40.9 packets/s, 398.70Kb/s ] Now the queue "lowq" is borrowing heavily from its parent queue (external) and the queue is not backing up even at the default of 50 elements. The strange thing here is that the queue "external" now shows that it is borrowing from its parent (the root queue), although the amount of bandwidth going through its queue is never getting close to its max of 896Kbit (about the highest I've seen it go is ~700Kbit/sec). At any rate, having "queue" external borrow from the root queue is a recipe for disaster as I'll just end up saturating my upstream bandwidth. The question is the same: does anyone know what is happening here, or how to solve it? Cheers, - -Wes -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (MingW32) iQIVAwUBRBhdxorq8W17hxGfAQiaeQ/8DJD0Yh8bLwwx9QiwhIUx9jwr/iyDaLMP z1wTvvGhGmE2Lksa8Y0ewhIbHHhNiAxnQ0q5bZGq6nmR1A54WntJ7DWfxKIfecSH C45K0WYz536fo+XipkEb0OHV6YwKuaOf+cLMObiKcVvWo6fLblU/hLNg6yYwv/PS rhZGr5Y/oul0pFgrLEWOTYmPXnn2wzUjoVZU5B4U1uNL+IZX0YBVkhlACm2yd/vc wSVNP7abpZb8tPQ6yZ5MafFrIFDapuou2nqKhtBOGECytSCdRihR8jtBOEsTYtlh V+Gzcwac7x4DNkbagMYuv888F+by7W1Ao9vTFU9oa0IBawtHhet1ihONRo+nbOMv gMYs4UW50JxkflQL3OGBZw5bypRh945m1gLzwMdaBdVzvVouduiz2N/EEAxFqNMo zcjkQiUToZAK+iZSLz5ptPVjaJnDX5NILQvu59IujpqdbDFbdORtWHpUixJnRxZh eCm3kptfPvbNVeWlHVapraQZcdF2/grJ8gM2+bi1jOlqvBbJUCHRjmrOfs4jnv1O 5cdqex5G3gob9terfTi02iwOylnACKPxkJzFGyyjjG+4gnendLgW7EAhoIWrC4WC 20dJfLg3BbC2uyr+lJR9vP0rrS5ShKxwwXUmtKe3U8PYeN83mGAc9kwJ/SvGT+2L GpJFrnQR4UU= =WeJv -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?44185DD4.8010009>