From owner-freebsd-hackers Mon Feb 18 4:42:48 2002 Delivered-To: freebsd-hackers@freebsd.org Received: from gate.nentec.de (gate2.nentec.de [194.25.215.66]) by hub.freebsd.org (Postfix) with ESMTP id CAAF537B422 for ; Mon, 18 Feb 2002 04:42:35 -0800 (PST) Received: from nenny.nentec.de (root@nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.9.3/8.9.3) with ESMTP id NAA19432; Mon, 18 Feb 2002 13:42:33 +0100 Received: from andromeda (andromeda [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3/SuSE Linux 8.11.1-0.5) with ESMTP id g1ICgTT11984; Mon, 18 Feb 2002 13:42:29 +0100 Message-ID: X-Mailer: XFMail 1.4.0 on Linux X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <3C70EE45.74A43103@mindspring.com> Date: Mon, 18 Feb 2002 13:42:32 +0100 (MET) Reply-To: Andy Sporner Organization: NENTEC Netywerktechnologie GmbH From: Andy Sporner To: Terry Lambert Subject: Re: Porting a device driver from NetBSD to FreeBSD Cc: freebsd-hackers@FreeBSD.ORG X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Hi Terry (and other snoopers--readers :-)) > > I guess I should ask you if this is an SMP system and/or > is the kernel compiled for SMP, or ist it UP (this is a > two by two marix, so the correct answer is not "yes/no" 8-)). Monoproc machine. Standard (OUT of box config file -- GENERIC). Sorry, I have no access to the machine now to tell you direct, but I didn't change anything in this regard in the GENERIC config file. >> > It might just be that you are setting PCI_COMMAND_MASTER_ENABLE, >> > rather than setting: >> > >> > command |= PCI_COMMAND_IO_ENABLE | PCI_COMMAND_MEM_ENABLE | >> > PCI_COMMAND_MASTER_ENABLE; >> >> I can't use the IO mode because the registers of the crossbar are >> not available there. The only register that is there is also in MEM >> access mode too. > > The reason I pointed this out is that you had _neither_ of > these set in the configuration write, and you will need to > set *at least* one of them for things to be happy. > For clarity I will set it direct, but it is already set on the read of the value. > >> o There is no blocking to read the cause register of the >> crossbar. If nothing is set, we just leave immediately. > > Hm. You may want to read this register via the I/O space in > order to cause a bus stall (annoying, but maybe more accurate). Hmmm. I don't think I can get to these registers this way (not available in the IO Map) The documentation (though I know sometimes how good it can be (NOT!)) seems to say that only one register is available through IO reads and the rest (including this special one) are available through MEM read. > >> o As to the much faster. It seems a different case. >> >> During initialization we read and write memory >> on the switch devices. This requires a PCI data >> transfer to send the command to the switch (to >> read the memory) and a response. We read every >> 64 words (16MB Ram) and it takes 3-4 minutes on >> NetBSD. The same code in the initalization routine >> on FreeBSD is so quick it almost cannot be counted. > > This is concerning. Perhaps it is failing because of the > lack of the initialization flag for memory commands. Basically, > 16MB at 132MB/S (32 bit PCI @ 33 MHz) should be that fast, if > all you are doing is a read of that much linear data across the > bus. If you are chunking it up with register tweaking and a lot > of little reads, then it should take longer. > We are doing pattern write/reads and this passes the test herein lies the example: start1 = 0x234000; end1 = sw->dram_size ? 0x3DFFFF : 0x24FFFF; start2 = sw->dram_size ? 0x520000 : 0x390000; end2 = sw->dram_size ? 0x7FFFFF : 0x3FFFFF; for (addr = start1; addr <= end2 - TEST_GAP; addr += TEST_GAP) { reg_data = TEST_PATTERN; dev_write_dram(sc, glink, addr, CONTROL_DRAM, ®_data, 1); reg_data = 0x0; dev_read_dram(sc, glink, addr, CONTROL_DRAM, ®_data, 1); if (reg_data != TEST_PATTERN) { printf("Failed...\n"); return (-1); } /* if */ reg_data = 0x0; dev_write_dram(sc, glink, addr, CONTROL_DRAM, ®_data, 1); which eventually is reduced to the following for reads and writes: /* data is the value to be written to the switch device */ /* addr is the address on the switch device */ for (;;) { r = dev_cb_read_4(sc, CB_STATUS); if ((r & BIT_8) == 0) { break; } /* if */ } /* for */ dev_cb_write_4(sc, CB_PCI_DATA_WORD_0, data); r = ((addr >> 14) & 0xffff); /* bit15 equal to 0 write */ dev_cb_write_4(sc, CB_PCI_DATA_WORD_1, r); r = (sw->dev << 22)|(region << 16)|(addr & 0x3fff); dev_cb_write_4(sc, CB_PCI_ADDRESS_WORD, r); Conversly Reads are similar from the switch device. That is to say we wait for a bit to be set and then read some PCI communications registers on the crossbar to see the results. We use these abstractions to write to the crossbar device: UINT32 dev_cb_write_4(galnet_softc_t *sc, UINT32 ofs, UINT32 val) { bus_space_write_4(sc->sc_st, sc->sc_sh, ofs, val); return (0); } /* dev_cb_write_4() */ UINT32 dev_cb_read_4(galnet_softc_t *sc, UINT32 ofs) { return (bus_space_read_4(sc->sc_st, sc->sc_sh, ofs)); } /* dev_cb_read_4() */ which is using data initialized during the attach function. > Either way, there's something weird here with the difference > between FreeBSD and NetBSD being so large. > >> We also see this as an impact in establishing new >> connections, since establishing a new connection >> requires 16 words to be transferred to the switch >> memory. > > Heh. I can see the attraction to FreeBSD, then... Here is the logic: 1. Memory test on FreeBSD is significantly faster than the the NetBSD counterpart (also boots MUCH faster!) on the exact same hardware (just swapped the boot harddisks). 2. The Memory test transfers high bulks of data using the same methods as the setting of entries to handle hardware forwarding of data. 3. Thus if Memory test is faster this follows that the setting of hardware forwarding entries must also be faster. > You probably just didn't get to the right list or the right > people. I was writting to tech-kern. I would wait weeks for responses. > > Being unable to drop details of your hardware can also mean > that people will have to turn their diagnostic process over > in the email message, and a lot of people are going to be a > bit reluctant to do that, if they go by instincts, or if they > consider it "secret". Other people might not respond on the > principle that the driver will end up being closed. We have > a lot of kooks. ;^). :-) Agreed. BTW: As a reward for people who actually READ posts through completely. I am making an preview type anouncement of my clustering software (Hi-AV) availability at: http://www.sporner.com/bsdclusters. I will (hopefully!!!) be making some more improvements in the next week or so to the documentation. I have to put the stuff though another test round to, but in principle it works. It has to be a lower priority than my regular work so there is no interference. This is more or less an experiment to see how many people read complete messages. I will make a more formal announcment the right way in a week or so. Hopefully I get better luck announcing it this way. I have tried many times to get those TURKEYs at www.bsdtoday.com to put something on their clustering page about it and I have yet to see or hear anything from them. It gets rather frustrating! > > In any case, good luck with your project... > Thanks for your help... Andy To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message