From owner-freebsd-net@FreeBSD.ORG Wed Aug 14 12:01:16 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2A008D14; Wed, 14 Aug 2013 12:01:16 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id DEB7924A0; Wed, 14 Aug 2013 12:01:15 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 6F3657300A; Wed, 14 Aug 2013 14:05:51 +0200 (CEST) Date: Wed, 14 Aug 2013 14:05:51 +0200 From: Luigi Rizzo To: Lev Serebryakov Subject: route/arp lifetime (Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux)) Message-ID: <20130814120551.GA64260@onelab2.iet.unipi.it> References: <520A6D07.5080106@freebsd.org> <520AFBE8.1090109@freebsd.org> <520B24A0.4000706@freebsd.org> <520B3056.1000804@freebsd.org> <20130814102109.GA63246@onelab2.iet.unipi.it> <587579055.20130814154713@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <587579055.20130814154713@serebryakov.spb.ru> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: Lawrence Stewart , FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Aug 2013 12:01:16 -0000 On Wed, Aug 14, 2013 at 03:47:13PM +0400, Lev Serebryakov wrote: > Hello, Luigi. > You wrote 14 ?????????????? 2013 ??., 14:21:09: > > LR> Then the problem remains that we should keep a copy of route and > LR> arp information in the socket instead of redoing the lookups on > LR> every single transmission, as they consume some 25% of the time of > LR> a sendto(), and probably even more when it comes to large tcp > LR> segments, sendfile() and the like. > And we should invalidate this info on ARP/route changes, or connection > will be lost in such cases, am I right?.. So, on each such event code > should look into all sockets and check, if routing/ARP information is still > valid for them. Or we should store lists of sockets in routing and ARP > tables... I don't know, what is worse. I think we should start by acknowledging that routing and ARP information is inherently stale, and changes unfrequently. So it is not a disaster if we have incorrect information for some short amount of time (milliseconds) because in the end the remote party that decides to change it and inform us may take much longer than that to distribute the update. Considering that each lookup takes between 100..300ns if you are lucky (not many misses, relatively empty table etc.), one could reasonably do the lookup at most once per millisecond or so (just reading 'ticks', no need for a nanotime() if you have a slow clock), or whenever we get an error related to the socket, either in the forward path (e.g. ifp points to an interface that is down) or in the reverse path (e.g. a dupack because we sent a packet to the wrong place). cheers luigi