Date: Mon, 18 Feb 2002 04:06:29 -0800 From: Terry Lambert <tlambert2@mindspring.com> To: Andy Sporner <sporner@nentec.de> Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: Porting a device driver from NetBSD to FreeBSD Message-ID: <3C70EE45.74A43103@mindspring.com> References: <XFMail.020218112456.sporner@nentec.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Andy Sporner wrote: > Yes and no. It *IS* an ethernet device--that's clear, but it has it's > own stack (more or less) optimized for doing policy based routing, so > it doesn't connect to the TCP/IP stack. The application is traffic > management. We are expecting very high bandwidth traffic management. > The NetBSD driver switched 20K new connections a second before running > into our PCI bottleneck (Hence the need to move to FreeBSD). The forwarding > is handled in hardware, so we only get hit during the establishment of > new connections. OK; that it has its own stack hooked to its interrupt, and is not using the TCP stack at all implies that you are immune to most of the normal problems. > > What happens if you just probe the thing, and don't try > > to attach or detach at all (always fail the probe, but > > printf when it would have been successful)? You implied > > that you did this, but it wasn't entirely clear that you > > had not attached it. > > No problems. Works fine... OK, this lets out tickled hardware at the same address. > Sorry, I tried 4.3 4.4 and 4.5. I will keep this point in > mind on future posts. I have largely been lurking. This is important because of a recent bug in the kqueue code, though if you are never going to user space, then that won't be an issue. But a version always narrows the problem field a bit. I guess I should ask you if this is an SMP system and/or is the kernel compiled for SMP, or ist it UP (this is a two by two marix, so the correct answer is not "yes/no" 8-)). > > I recommend looking at the if_tx.c driver, and using that > > as a guide, since it does some of the strange stuff you > > seem to need to do, and it (apparently) works. > > > > I will do this. I had been using if_fxp.c as an example. I recommend this one because of your read and write of configuration space. > > It might just be that you are setting PCI_COMMAND_MASTER_ENABLE, > > rather than setting: > > > > command |= PCI_COMMAND_IO_ENABLE | PCI_COMMAND_MEM_ENABLE | > > PCI_COMMAND_MASTER_ENABLE; > > I can't use the IO mode because the registers of the crossbar are > not available there. The only register that is there is also in MEM > access mode too. The reason I pointed this out is that you had _neither_ of these set in the configuration write, and you will need to set *at least* one of them for things to be happy. > >> 4. The device driver does not use MBUFS at all. > > > > Not relevent, then... though if it's a network driver, it's > > obligated to use mbufs at some point. > > See above... Naturally I would like to say more about the internal > structure of the driver, but I am not able because of confidentiality > or as they say here "Datenschutz". Yep; been there, done that. [ ... shared interrupt ... ] > I found this to be the case at least once and this might be > a point to investigate. But having the Realtek driver installed > causes the PCI memory to be overlapped between the devices. The Realtek is, I think, configured by BIOS, so you may end up in trouble here anyway. Setting the PCI_COMMAND_MEM_ENABLE before the write is probably an even better idea, given that. > We handle shared interrupts OK (just check our cause registers > and do nothing if it isn't for us). It has worked in the Netbsd > world this way OK. But for efficiency reasons I always locate a > slot where I can plug the device so that no sharing occurs. Otherwise > I would get an interrupt for every network packet. The problem I was concerned with was the other driver *not* handling that correctly and/or a possible race in the card, e.g.: 1) Set flag 2) Make data valid 3) Cause interrupt vs. 1) Make data valid 2) Set flag 3) Cause interrupt Could be a firmware/hardware problem, if it's run too fast, or if the interrupt ends up shared. > I can say in response: > > o No sharing of interrupts (that I can see in 'dmesg'). Yep. Not entirely definnitive, if you have unrecognized hardware probed by the BIOS. > o There is no blocking to read the cause register of the > crossbar. If nothing is set, we just leave immediately. Hm. You may want to read this register via the I/O space in order to cause a bus stall (annoying, but maybe more accurate). > o As to the much faster. It seems a different case. > > During initialization we read and write memory > on the switch devices. This requires a PCI data > transfer to send the command to the switch (to > read the memory) and a response. We read every > 64 words (16MB Ram) and it takes 3-4 minutes on > NetBSD. The same code in the initalization routine > on FreeBSD is so quick it almost cannot be counted. This is concerning. Perhaps it is failing because of the lack of the initialization flag for memory commands. Basically, 16MB at 132MB/S (32 bit PCI @ 33 MHz) should be that fast, if all you are doing is a read of that much linear data across the bus. If you are chunking it up with register tweaking and a lot of little reads, then it should take longer. Either way, there's something weird here with the difference between FreeBSD and NetBSD being so large. > We also see this as an impact in establishing new > connections, since establishing a new connection > requires 16 words to be transferred to the switch > memory. Heh. I can see the attraction to FreeBSD, then... > At least on Netbsd, interrupts are not enabled > during the initialization (during attach (nitro_init)). > so would not be subjected to the sharing of interrupt > situation. There shouldn't be an issue here, unless you use an active probe that requires an interrupt to attach or to init. This shouldn't really be necessary, given that it's a PCI device, so that's probably not it... > > Alternately, you could ask Bill Paul, since he's a better > > choice than me on this sort of thing, anyway. 8-) 8-). > > > > Hope this is useful, even if it doesn't come right out > > and say "here's a patch". > > > > Fine enough! I have been *VERY* pleased at the response I have been > getting in the FreeBSD world. I had not had that much success in the > NetBSD world at all (with the exception of a new notable people). I > was even snubbed by Mr. Wasabi himself. So going forward I won't > be doing any new work on NetBSD. You probably just didn't get to the right list or the right people. Being unable to drop details of your hardware can also mean that people will have to turn their diagnostic process over in the email message, and a lot of people are going to be a bit reluctant to do that, if they go by instincts, or if they consider it "secret". Other people might not respond on the principle that the driver will end up being closed. We have a lot of kooks. ;^). In any case, good luck with your project... -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3C70EE45.74A43103>