From owner-freebsd-hackers@FreeBSD.ORG Fri Aug 20 14:20:19 2004 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2C2C216A4CE for ; Fri, 20 Aug 2004 14:20:19 +0000 (GMT) Received: from mail.gulfgate-inc.com (mail.gulfgate-inc.com [64.1.98.180]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6F3C743D53 for ; Fri, 20 Aug 2004 14:20:18 +0000 (GMT) (envelope-from mpf@inodes.us) Received: (qmail 4919 invoked by uid 85); 20 Aug 2004 14:22:54 -0000 Received: from 192.168.0.4 by mail.gulfgate-inc.com (envelope-from , uid 89) with qmail-scanner-1.23 (clamdscan: 0.75.1. spamassassin: 2.64. Clear:RC:1(192.168.0.4):. Processed in 0.081429 secs); 20 Aug 2004 14:22:54 -0000 X-Qmail-Scanner-Mail-From: mpf@inodes.us via mail.gulfgate-inc.com X-Qmail-Scanner: 1.23 (Clear:RC:1(192.168.0.4):. Processed in 0.081429 secs) Received: from unknown (HELO ?127.0.0.1?) (192.168.0.4) by 192.168.0.40 with SMTP; 20 Aug 2004 14:22:53 -0000 Message-ID: <412608A0.7030704@inodes.us> Date: Fri, 20 Aug 2004 09:20:16 -0500 From: Matt Freitag User-Agent: Mozilla Thunderbird 0.7 (Windows/20040616) X-Accept-Language: en-us, en MIME-Version: 1.0 To: anand@cs.uwaterloo.ca, freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Network Packet drops in FreeBSD 5.2.1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Aug 2004 14:20:19 -0000 Anand Subramanian wrote: > Hi All, > I am using an Intel Celeron box (single CPU,1.7GHz, 495MB real and 472 MB > avail memory, FreeBSD 5.2.1 #15 release), to run a daemon process which > shares a circular queue/buffer with the kernel. The daemon drains objects > off the front of the queue while the queue objects are populated by the > protocol processing function say, XXX_input() in the kernel, called by > ip_input(). If the front and rear indices of the shared buffer are equal > then the protocol stack drops the packet alright. > > This model works fine for packet (60-64 bytes in size) input rates upto > 11500 packets/sec, after which packets are lost. The machine running the > daemon uses the SiS 900 NIC, 10/100Mbps. > > > Try a nic that has hardware support for checksum and fragmentation offloading. If I'm not mistaken that sis900 is an integrated nic? It was somewhat flaky for me.. If you're operating at 100Mbps, your line rate doesn't become saturated at 12kpps. In fact, 100Mbps isn't saturated until around ~140kpps from my own lab results. > The packet loss was detected using the "netstat -I sis0" command, run > both > on starting the daemon and upon shutting the daemon down. The Ipkts field > in the netstat output should indicate the number of packets received by > the interface in question.. > > With an appropriate chosen value for the shared buffer length, no packets > are dropped because the shared queue is full. Hence packets seem to be > dropped at the adapter level. The surprising part seems to be that though > packets are being dropped/lost, top shows a ~70% idle system with peak > interrupt time of ~25%. > > The daemon uses a "hacked" version of the select() call with a timeout > value. The XXX_input() protocol processing function signals the > thread/KSE > waiting on the hacked select() call as soon as it sees that there are > packets in the shared buffer(shared between the daemon and the kernel). > > Question is : > > 1. Is top really accurate in reporting all stats at such workloads, or > input pkt rates? Can the %Idle time reported by top be trusted? > > I've seen some exceptions to the rule, yet overall it seems accurate - but I've never read the inter workings of top, so I'm no expert. > 2. At increasing network input loads (12000 pkts/sec), much of the system > time maybe spent in the hardware interrupt handler, ip processing > functions. With the user daemon calling select(), any time spent in the > select() call would be charged to the daemon's timeslice. Would it be > fairly scheduled to run. It should be(of course depending on the RPLs), > but wanted to confirm this. > > I'm not sure if it's just the NIC you're using, but you should probably enable device polling, perhaps even enable fastforwarding. > 3. When ip_drain() is called and it calls the DEQUEUE macro, it acquires > Giant. DOes this mean other netisr's and handlers are disabled so that > the > queue gets emptied in a sort of batch-mode behavior? > > 4. I am trying different clock speeds by changing kern.hz in loader.conf. > Doesn't seem to help but I am still looking into this. > > When network packets are being dropped at the interface level, is it > really necessary for the system to be ~0% idle??? > > > I'm sure it's all interrupts, again a better NIC could help :> > Any other input is greatly appreciated. > > Best, > Anand > > > > HTH, -mpf > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" > > >