Date: Mon, 22 Mar 1999 15:04:36 -0800 (PST) From: Matthew Dillon <dillon@apollo.backplane.com> To: Dennis <dennis@etinc.com> Cc: hackers@freebsd.org Subject: Re: Gigabit ethernet -- what am I doing wrong? Message-ID: <199903222304.PAA25219@apollo.backplane.com> References: <Your message of "Sun, 21 Mar 1999 20:10:21 EST." <199903220058.TAA17538@etinc.com> <199903221711.MAA20551@etinc.com> <199903222019.PAA21360@etinc.com> <199903222241.RAA21975@etinc.com>
next in thread | previous in thread | raw e-mail | index | archive | help
:>:Dennis
:>
:> Dennis, please read my response more carefully and you will understand
:> why your concerns are unfounded. I will paraphrase: "The FIFOs are
:> large enough such that the burst is going to be relatively efficient
:> no matter what the PCI bus width and no matter how slow the card".
:>
:> The problem that is being solved here is not a card's ability to sustain
:> a long burst, but instead the bandwidth available on the PCI bus when
:> multiple cards are operating. There is no advantage to any single card
:> being able to do long sustained bursts on a faster PCI bus because they
:> are already limited to the speed physical media they were built to
:handle.
:> The requirement is for the card to be able to sustain a reasonably-sized
:> burst so the PCI bus's transactional overhead remains small verses the
:> actual data transfer. This is what the FIFO accomplishes.
:
:What i mean by "sustained" is more than 1 or 2 words...I believe that 32bit
:engines would have a problem doing single cycle transfers at 64bits...and
:that implies substantial setup overhead, more than double or triple, which
:significantly effects available bandwidth and device perfornance. I've done
:a lot of empirical testing and if you miss the burst performance really sucks
:wind. Controllers with external logic on the board, for example, often
:can't do
:single-cycle transfers and bus performance severely suffers. I dont know how
:you can say that it doesnt.
:
:Dennis
Dennis, you are missing something fundamental. Do you know how
a FIFO works? Do you know what a low and high water mark is? Do
you know what a bus-matching FIFO is? Do you know how network or
scsi master-bus DMA works?
Concentrate on the FIFO part of my explanation. These are not
software FIFOs. Maybe you are confused. These are hardware FIFOs.
Do you know how low and high water marks work with FIFOs? Do you know
what a data-pending-idle-timeout is?
-
Let me give you an example: With an IDLE PCI bus it is quite possible
for a card doing DMA to wind up doing extremely inefficient single-word
transfers do to the on-card physical data rate being relatively low
( aka a 10BaseT card ). But since this is an IDLE PCI bus, you don't care.
If the PCI bus has additional load on it ... say from a 100BaseT card
sitting in another slot, any DMA operations performed by the 10BaseT
card get queued to the FIFO while the PCI bus is busy with the other card.
As the PCI bus is loaded down more and more by additional cards or
traffic, PCI bus latency causes the amount of pending data in the various
cards FIFOs to grow, even on cards with slow processors, increasing the
efficiency of the PCI bus because the cards start to do multi-word burst
I/O of the data that has accumulated in the FIFOs. This is a
self-stabilizing situation ( at least theoretically ). If the PCI burst
limits are set properly, the worst case latency occurs when you have
5 PCI cards trying to do DMA simultaniously, and 4 of those cards are
relatively low speed such that they manage to do only a few words per
PCI bus grant despite the bus latencies being incured due to other card's
DMA. In this case the fifth card sees worst-case operation:
Around 16 cycles ( guessing, I don't have the spec on my lap ) of latency
due to single-word transfers from the other four cards, and then 64 cycles
or so worth of burst activity from the 5th ( high speed ) card. As you
can see, it isn't terrible.
-Matt
Matthew Dillon
<dillon@backplane.com>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199903222304.PAA25219>