Date: Wed, 3 Sep 2008 00:04:40 -0700 From: Jeremy Chadwick <koitsu@FreeBSD.org> To: Lance Murdock <lance@theouterdarkness.com> Cc: freebsd-pf@freebsd.org Subject: Re: ALTQ & Multiple Connections Message-ID: <20080903070440.GA28260@icarus.home.lan> In-Reply-To: <20080903053916.GA81677@theouterdarkness.com> References: <20080903020843.GA70612@theouterdarkness.com> <200809030444.31690.max@love2party.net> <20080903053916.GA81677@theouterdarkness.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Sep 02, 2008 at 10:39:16PM -0700, Lance Murdock wrote: > On Wed, Sep 03, 2008 at 04:44:31AM +0200, Max Laier wrote: > > No and I don't know of any software that would make that > > possible - probably because it's a horrible idea. > > I wouldn't say it is a horrible idea. It may have some implementation > details, but the idea of maximally utilizing available resources at > minimum cost is not fundamentally horrible. > > Also, this is for negotiation purposes as much as any technical reason. > Our carriers feel there is no need to negotiate on price because they're > going to get paid on the overages anyway. They figure the router and > construction expenses of pulling in more fiber are pretty much a lock-in, > and they're pretty much right. So I'd like to put a shot across > their bow that not only do we have the power to control how much they > get paid without scuttling our own site, but also we don't need to pull > more fiber to do it. :-) If I understand your situation correctly, you pay for a connection to two different peering providers or LECs, and the bandwidth (likely 95th-percentile) that you're billed against is different per provider (e.g. Provider ABC gives you a gigE port with 4mbit/sec 95th-percentile for X dollars a month, while provider XYZ gives you a gigE port with 512kbit/sec 95th-percentile for X dollars a month). Is this correct? If so, I'm not even sure commercial routers (e.g. Junipers) can solve your predicament. Ideally you'd be better off with symmetric bandwidth amounts to both of your peers (e.g. you pay both Provider ABC and Provider XYZ for 4mbit/95th of traffic). In that situation, you might be able to used a "load-balanced" solution for packets, which *might* work and meet your needs. I say "might" because Internet routing does not guarantee you'll be using both connections 50/50. I'm not sure why people think that; it really doesn't work that way in practise. Assuming the two ISPs are different and you have different IPs per peer, you're screwed. BGP preferencing and route priority will ensure you probably will not utilise each connection equally. My comment applies to incoming traffic, not outbound. Outbound you can preference/balance at your leisure, as Max described. > Ideally we would let as much traffic as possible take its best path > according to the route table and only shape the minimum necessary > to meet our utilization objectives. But even I am confident I have > crossed irretrievably into fantasyland at that point. Like I said: why do you think the rest of the world will adhere to what your routing table prefers? This is one of the common caveats to BGP. Just because you preference a route through provider ABC doesn't mean some ISP in Malaysia is going to honour that preference. I deal with this situation at my day job on a daily basis. > I'm thinking of something along the lines of good old fashioned > multilink PPP, which brought up more channels based on utilization. > The only difference here is that we're not going to get protocol > cooperation from the far end. Okay, so multilink PPP implies that you're able to get at least some sort of common IP block assigned to you through both peers, and get both peers to comply with your routing policies? Let me know if you manage to do that, as I'd be interesting in knowing what providers are *that* flexible. > > The only thing you can do is > > some level of *per-flow* round-robin (with weights) onto your outgoing > > connections - maybe adjusting the weights according to ALTQ usage stats. > > I'm sorry, I don't know enough about ALTQ to know if this is intended to > be a practical suggestion. If so, where would I look for documentation? > I've got the Reed book and it's been massively helpful but doesn't > appear to cover the sort of crazy misuse that I have in mind. > > > But > > that's a very rough estimate - but you can't do better than that, anyways. > > If we can get within, say, 10% that would be a great start. Since carrier > standard is 95/5 billing, all we have to do is visibly clip the peaks on > an MRTG graph to achieve our objective. See above. I'm pretty sure I understand your predicament, and if I was in your shoes, I'd be dropping my contract with the provider who provides less bandwidth (or costs more) and urging the other provider to provide more reliable redundancy and peering. I've personally been in that situation, though with only one provider used at the time. We pulled our entire infrastructure out of their datacenter one we found out they had no form of switch or router failover/redundancy, and that despite being in California, they were using Telia (a Swedish ISP) to peer with large carriers like AT&T and MCI. Telia connection dies? No failover available, and Telia has no NOC in the US. "Hallo dis Telia NOC" "Talar du Engelska?" "Nej" -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080903070440.GA28260>