Date: Sun, 21 Mar 1999 02:18:53 -0500 (EST) From: Bill Paul <wpaul@skynet.ctr.columbia.edu> To: julian@whistle.com (Julian Elischer) Cc: dillon@apollo.backplane.com, hackers@freebsd.org Subject: Re: Gigabit ethernet revisited Message-ID: <199903210718.CAA08119@skynet.ctr.columbia.edu> In-Reply-To: <36F2FE8E.7566F4CF@whistle.com> from "Julian Elischer" at Mar 19, 99 05:49:02 pm
next in thread | previous in thread | raw e-mail | index | archive | help
Of all the gin joints in all the towns in all the world, Julian Elischer had to walk into mine and say: > I think Andrew might be right.. > it could well be livelock. > > Matt Thomas implemented a solution for the 100mb dec cards > when 100 was fast. I think that the de drivers responded to the > interrupt and immediatly did SCHEDNETISR() to schedule the rest of > the driver that was running at a lower priority. I don't know if the > if_de driver still does that but is could be worth a look. I took a stab at duplicating this (the code is in the de driver but it's conditional and not enabled by default). I managed to make it work fairly easily, but it doesn't seem to help. I even set the maxsockbuf parameter to a ridiculously large number (20MBytes) and did a -b20000000 with ttcp to have it set the SO_RCVBUF really large, and still the socket buffers filled up (I actually added some debug printf()s to see what happened in sbappendaddr() when sbspace() showed there was no room left and indeed the socket buffer showed up as been filled for several consecutive calls to sbappendaddr().) It seems really strange to me that this can happen: the application should be draining the socket long before 20MB of data accumulates. Let me just see if I understand what happens correctly: - Data arrives at the NIC. - The NIC interrupts. - The driver checks the receive ring and passes the packets to ether_input(). - Ether_input() puts them on a queue and schedules a soft interrupt (netisr). - The netisr eventually causes ip_input() to run. - Ip_input() passes the data to udp_input(). - Udp_input() passes the data to sbappendaddr() and calls sorwakeup(). - Sorwakeup() sets some flags in the socket structure and calls sowakeup(). - Sowakeup() calls wakeup(). - Wakeup() marks the process waiting on the socket as runnable. - The scheduler runs the process so it can receive the data. So, how long after the process is marked runable does the scheduler actually transfer control to the process so that it can handle the received data? Can the process be prevented from running if there are too many interrupts from the NIC? Is there some way to make the scheduler run the process more often (I tried using nice -20 on the receving instance of ttcp; that didn't seem to make a difference)? Has anybody else actually tried to receive data at 600 to 800Mbps speeds on FreeBSD and done it reliably? (Note: I mean actually transmitting UDP packets at, say 80MB/sec and actually receiving _all_ of the transmitted packets on the other side, in the application, at the same speed. No fudging.) Transmit speed doesn't seem to be an issue here, but somehow I get the feeling that the kernel is sabotaging itself on receive. -Bill -- ============================================================================= -Bill Paul (212) 854-6020 | System Manager, Master of Unix-Fu Work: wpaul@ctr.columbia.edu | Center for Telecommunications Research Home: wpaul@skynet.ctr.columbia.edu | Columbia University, New York City ============================================================================= "It is not I who am crazy; it is I who am mad!" - Ren Hoek, "Space Madness" ============================================================================= To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199903210718.CAA08119>