From owner-freebsd-current@FreeBSD.ORG Mon Nov 21 17:20:32 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF564106564A; Mon, 21 Nov 2011 17:20:32 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 7E62F8FC08; Mon, 21 Nov 2011 17:20:31 +0000 (UTC) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 067EE73027; Mon, 21 Nov 2011 18:36:15 +0100 (CET) Date: Mon, 21 Nov 2011 18:36:15 +0100 From: Luigi Rizzo To: John Baldwin Message-ID: <20111121173614.GA63552@onelab2.iet.unipi.it> References: <4EC6AEF0.1010402@FreeBSD.org> <20111118220458.GA21152@onelab2.iet.unipi.it> <201111211129.29362.jhb@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201111211129.29362.jhb@freebsd.org> User-Agent: Mutt/1.4.2.3i Cc: Matteo Landi , Doug Barton , freebsd-current@freebsd.org Subject: Re: ixgbe and fast interrupts X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2011 17:20:32 -0000 On Mon, Nov 21, 2011 at 11:29:29AM -0500, John Baldwin wrote: > On Friday, November 18, 2011 5:04:58 pm Luigi Rizzo wrote: > > On Fri, Nov 18, 2011 at 11:16:00AM -0800, Doug Barton wrote: > > > On 11/18/2011 09:54, Luigi Rizzo wrote: > > > > One more thing (i am mentioning it here for archival purposes, > > > > as i keep forgetting to test it). Is entropy harvesting expensive ? > > > > > > No. It was designed to be inexpensive on purpose. :) > > > > hmmm.... > > unfortunately I don't have a chance to test it until monday > > (probably one could see if the ping times change by modifying > > the value of kern.random.sys.harvest.* ). > > > > But in the code i see the following: > > > > - the harvest routine is this: > > > > void > > random_harvest(void *entropy, u_int count, u_int bits, u_int frac, > > enum esource origin) > > { > > if (reap_func) > > (*reap_func)(get_cyclecount(), entropy, count, bits, frac, > > origin); > > } > > > > - the reap_func seems to be bound to > > > > dev/random/randomdev_soft.c::random_harvest_internal() > > > > which internally uses a spinlock and then moves entries between > > two lists. > > > > I am concerned that the get_cyclecount() might end up querying an > > expensive device (is it using kern.timecounter.hardware ?) > > On modern x86 it just does rdtsc(). > > > So between the indirect function call, spinlock, list manipulation > > and the cyclecounter i wouldn't be surprised it the whole thing > > takes a microsecond or so. > > I suspect it is not quite that expensive. > > > Anyways, on monday i'll know better. in the meantime, if someone > > wants to give it a try... in our tests between two machines and > > ixgbe (10G) interfaces, an unmodified 9.0 kernel has a median ping > > time of 30us with "slow" pings (say -i 0.01 or larger) and 17us with > > a ping -f . > > Did you time it with harvest.interrupt disabled? yes, thanks for reminding me to post the results. Using unmodified ping (which has 1us resolution on the reports), there is no measurable difference irrespective of the setting of kern.random.sys.harvest.ethernet, kern.random.sys.harvest.interrupt and kern.timecounter.hardware. Have tried to set hw mitigation to 0 on the NIC (ixgbe on both sides) but there is no visible effect either. However I don't trust my measurements because i cannot explain them. Response times have a min of 20us (about 50 out of 5000 samples) and a median of 27us, and i really don't understand if the low readings are real or the result of some races. Ping does a gettimeofday() for the initial timestamp, and relies on in-kernel timestamp for the response. cheers luigi