From owner-freebsd-questions Fri Nov 15 14:03:29 1996 Return-Path: owner-questions Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id OAA20143 for questions-outgoing; Fri, 15 Nov 1996 14:03:29 -0800 (PST) Received: (from root@localhost) by freefall.freebsd.org (8.7.5/8.7.3) id OAA20062 for freebsd-hackers-digest-outgoing; Fri, 15 Nov 1996 14:01:45 -0800 (PST) Date: Fri, 15 Nov 1996 14:01:45 -0800 (PST) Message-Id: <199611152201.OAA20062@freefall.freebsd.org> From: owner-hackers-digest To: freebsd-hackers-digest@FreeBSD.ORG Subject: hackers-digest V1 #1645 Reply-To: hackers Sender: owner-questions@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk hackers-digest Friday, 15 November 1996 Volume 01 : Number 1645 In this issue: Re: Sockets question... 2.1.5-stable build fails in gdb, mrouted, named, nslookup Re: Sockets question... Re: Sockets question... Re: Sockets question... Re: Sockets question... earlier "holographic shell" in 2.2-ALPHA install procedure Re: Sockets question... Re: Sockets question... Kernel Commands Re: Sockets question... Re: FreeBSD 2.2-ALPHA is now available. Re: Sockets question... Re: Sockets question... Re: Sockets question... Re: Sockets question... Re: Sockets question... ---------------------------------------------------------------------- From: Joe Greco Date: Fri, 15 Nov 1996 14:21:51 -0600 (CST) Subject: Re: Sockets question... > >> Did you receive a signal? That is known to cause similar behaviour on > >> SunOS... > > > >Can't. Any signal on that process is a SERIOUS error; its a DBMS! We have > >a generic "oh shit" trap for all signals set; it does not go off. > > Don't do that for SIGIO. Anyone can send SIGIO to any process. Bruce, Could you clarify what you mean? Obviously "anyone" can not do that... Sorry, I am just missing your meaning. ... JG ------------------------------ From: Chris Shenton Date: Fri, 15 Nov 1996 15:37:02 -0500 Subject: 2.1.5-stable build fails in gdb, mrouted, named, nslookup Flame me (informatively) if I'm doing this wrong, but... I'm running 2.1.5-RELEASE and recently supped -stable. I do a "make world" from /usr/src, and get a few compilation errors that I probably shouldn't. Nothing fatal, but gdb, mroute, named, and nslookup *should* build. Can anyone enlighten me? Thanks. cd /usr/src/ make -k [...] ===> gnu/usr.bin/gdb/gdb cc -O -I/home/src/gnu/usr.bin/gdb/gdb/. -I/usr/include/readline -I/home/src/gnu/usr.bin/gdb/gdb/../bfd -o gdb annotate.o blockframe.o breakpoint.o buildsym.o c-lang.o c-typeprint.o c-valprint.o ch-lang.o ch-typeprint.o ch-valprint.o coffread.o command.o complaints.o copying.o core.o coredep.o corelow.o cp-valprint.o dcache.o dbxread.o demangle.o disassemble.o dis-buf.o dwarfread.o elfread.o environ.o eval.o exec.o expprint.o findvar.o fork-child.o freebsd-nat.o gdbtypes.o i386-dis.o i386-pinsn.o i386-tdep.o infcmd.o inflow.o infptrace.o infrun.o inftarg.o init.o kcorelow.o language.o m2-lang.o m2-typeprint.o m2-valprint.o main.o maint.o mem-break.o minsyms.o objfiles.o parse.o printcmd.o remote.o remote-utils.o solib.o source.o stabsread.o stack.o symfile.o symmisc.o symtab.o target.o thread.o top.o typeprint.o utils.o valarith.o valops.o valprint.o values.o version.o serial.o ser-unix.o mdebugread.o c-exp.tab.o ch-exp.tab.o m2-exp.tab.o compat_que.o -lreadline -ltermca! p ! - -lgnuregex -L/home/src/gnu/usr.bin/gdb/gdb/../libiberty/obj -liberty -L/home/src/gnu/usr.bin/gdb/gdb/../bfd/obj -lbfd -L/home/src/gnu/usr.bin/gdb/gdb/../mmalloc/obj -lmmalloc utils.o: Undefined symbol `_vasprintf' referenced from text segment utils.o: Undefined symbol `_vasprintf' referenced from text segment *** Error code 1 (continuing) `all' not remade because of errors. ===> usr.sbin/mrouted/mrouted cc -O -I/home/src/usr.sbin/mrouted/mrouted/.. -DRSRR -DRSRR -o mrouted config.o cfparse.o main.o route.o vif.o prune.o callout.o rsrr.o -L/home/src/usr.sbin/mrouted/mrouted/obj/../common -lmrouted main.o: Undefined symbol `_configfilename' referenced from text segment vif.o: Undefined symbol `_configfilename' referenced from text segment vif.o: Undefined symbol `_config_vifs_from_file' referenced from text segment *** Error code 1 (continuing) `all' not remade because of errors. ===> usr.sbin/named cc -O -o named version.o db_dump.o db_load.o db_lookup.o db_reload.o db_save.o db_update.o db_secure.o db_glue.o ns_forw.o ns_init.o ns_main.o ns_maint.o ns_req.o ns_resp.o ns_sort.o ns_stats.o ns_validate.o ns_ncache.o storage.o dmalloc.o tree.o -lutil ns_main.o: Undefined symbol `_Version' referenced from text segment ns_main.o: Undefined symbol `_Version' referenced from text segment *** Error code 1 (continuing) `all' not remade because of errors. ===> usr.sbin/nslookup cc -O -o nslookup main.o commands.o getinfo.o debug.o send.o skip.o list.o subr.o -ll main.o: Undefined symbol `_yylex' referenced from text segment subr.o: Undefined symbol `_yyin' referenced from text segment subr.o: Undefined symbol `_yyrestart' referenced from text segment *** Error code 1 (continuing) `all' not remade because of errors. ------------------------------ From: Joe Greco Date: Fri, 15 Nov 1996 14:37:11 -0600 (CST) Subject: Re: Sockets question... > In message <199611151706.KAA26239@phaeton.artisoft.com> Terry wrote: > >How do I read into a structure on a machine that demands aligned > >data access? > > You read into an intermediate buffer and copy it. You have to convert from > network to machine representation anyway, so this isn't (much) more overhead. > Or you use UDP if you want a record-oriented protocol. Actually... I usually found it easier to do the following. #include /* * One dull variant of Joe's "xread" function. */ int xread(fd, buf, siz) int fd; char *buf; int siz; { int rval; int chrs = 0; while (siz) { if ((rval = read(fd, buf, siz)) < 0) { return(rval); } chrs += rval; siz -= rval; buf += rval; } } int main() { int rval; struct big_ugly { yadda; yadda; yadda; } data; if ((rval = xread(fd, (char *)&data, sizeof(data))) != sizeof(data)) { fprintf(stderr, "Help me, I am on fire, xread returned %d\n", rval); } } This of course assumes you either do not need to do byte reordering, or do it elsewhere. ... JG ------------------------------ From: Bill Fenner Date: Fri, 15 Nov 1996 12:44:03 PST Subject: Re: Sockets question... >Actually... I usually found it easier to do the following. And that's what Terry's complaining about; if you have alignment requirements when writing big_ugly->yadda, those alignment requirements might not be met by the read() inside of xread(). Bill ------------------------------ From: Joe Greco Date: Fri, 15 Nov 1996 14:59:34 -0600 (CST) Subject: Re: Sockets question... > >Actually... I usually found it easier to do the following. > > And that's what Terry's complaining about; if you have alignment > requirements when writing big_ugly->yadda, those alignment requirements > might not be met by the read() inside of xread(). Ummm... and the problem is...? As far as I am aware, byte oriented data can be written to unaligned addresses on any UNIX architecture that I have seen. xread is explicitly called with what is clearly a byte oriented buffer. If you are possibly worried about something such as the atomicity of reads (potentially valid in a threaded environment, or one using shared memory), I agree that there may be some concern. Since it is not clear to _me_ that such atomicity of access would be valid under the same circumstances even with read(), I would probably code around the situation anyways. Is there some other problem that I am missing? I've done this sort of things for several years now... ... JG ------------------------------ From: Bruce Evans Date: Sat, 16 Nov 1996 07:59:35 +1100 Subject: Re: Sockets question... >> on system 1 with 32MB RAM: the kernel panics with a double fault > >This is, I think, expected behaviour for a page which is not present but >for which a kmem mapping exists. This happens because the /kernel file >is mapped to /dev/kmem. Not in FreeBSD. /kernel is an ordinary, unmapped file. >If you try this with a different file (one without an established mapping), >what happens? Same. It was a bug in i586_copyout. > >> on system 2 with 8MB RAM: read returns -1. This worked because it's an i486. The memory size was irrelevant. This shows why everyone should include full information information in bug reports. Bruce ------------------------------ From: Mike Haertel Date: Fri, 15 Nov 1996 13:04:52 -0800 (PST) Subject: earlier "holographic shell" in 2.2-ALPHA install procedure I would like the ability to launch the "emergency holographic shell" earlier in the install process, say from a menu entry somewhere. Mainly I would like to be able to launch it before "ifconfig" is called in network installs. ------------------------------ From: Terry Lambert Date: Fri, 15 Nov 1996 13:58:03 -0700 (MST) Subject: Re: Sockets question... > >So at the "read" interface, you *can* count on it arriving in the > >same sized chunks as you wrote it > > No, you can *never* count on that, since non-blocking reads from a stream > socket return as much data as is available, which could be less than you > asked for. See soo_read() (or soo_rw() in earlier BSD's) and soreceive(). By default, the sockets are blocking. You have to go out of your way to make them non-blocking (ie: loading the gun before you can shoot yourself in the foot). > 4.4BSD introduced the MSG_WAITALL flag, so if you use recv() or any of its > friends you can ask for your whole request to be performed. This is, of > course, not portable, and MSG_WAITALL won't even do the trick if your > request is larger than the socket's high water mark (e.g. SO_RECVBUF). This makes sense. But on a blocking socket, it doesn't make sense to have to issue multiple system calls to read chunks of a whole message when you aren't going to do anything with it until all the reads have been satisfied? It makes sense to do it on the server, if you have a server getting messaged from a bunch of sources, but that's just moving the packet assembly into user space. You'd expect the server to have a higher premium on interleaving operations than on atomicity of transactions. Even so, this only becomes a problem if you send "outrageously large" packets in a single write. You'd think that you could move it back down into kernel space by the select not coming true until all frags had been reassembled. I could see SO_RECVBUF putting a limit on your biggest message size in that case. The MSG_WAITALL seems like, even without the flag, it would be the default behaviour for the read. The alternative is to push the "frag" assembly ("frags" in this case being buffers larger than SO_RECVBUF cut into SO_RECVBUF) up into the user program, an dictate to the user how he must code a state machine in user space in order to use the interface. Ugh, copy overhead. Bleah. 8-(. I suppose one very real possibility is that he's trying to send huge packets... most of the drivers don't support them, and the DEC driver supports them only with a couple of patches that Matt and Amancio beat out between them a while back... If it's huge packets, he's probably SOL, and needs to redesign his stream formatting. Anyway, let's wait and see what clarifications he has to offer before we speculate ourselves to death. 8-). Terry Lambert terry@lambert.org - --- Any opinions in this posting are my own and not those of my present or previous employers. ------------------------------ From: Terry Lambert Date: Fri, 15 Nov 1996 14:09:34 -0700 (MST) Subject: Re: Sockets question... > In message <199611151858.LAA26626@phaeton.artisoft.com> you write: > >So at the "read" interface, you *can* count on it arriving in the > >same sized chunks as you wrote it > > No, you can *never* count on that, since non-blocking reads from a stream > socket return as much data as is available, which could be less than you > asked for. See soo_read() (or soo_rw() in earlier BSD's) and soreceive(). > > 4.4BSD introduced the MSG_WAITALL flag, so if you use recv() or any of its > friends you can ask for your whole request to be performed. This is, of > course, not portable, and MSG_WAITALL won't even do the trick if your > request is larger than the socket's high water mark (e.g. SO_RECVBUF). Maybe this is where I'm confused. I was under the impression that the MSG_WAITALL was for messages that had to be passed up to free up the receive buffer, and was put there to allow the copy to the user buffer to occur in SO_RECVBUF-sized chunks. Again, we aren't really interested in the non-blocking socket case... Terry Lambert terry@lambert.org - --- Any opinions in this posting are my own and not those of my present or previous employers. ------------------------------ From: Tony Sterrett Date: Fri, 15 Nov 1996 13:23:20 -0800 (PST) Subject: Kernel Commands Hello everybody: Does there exist systems calls used in the kernel to do things like disabling the cache(Pentum), allocating memory etc. Can someone do me the flavor of posting and call for disabling cache and maybe provide a reference to this class of commands. Thanks in Advance. Tony ------------------------------ From: Terry Lambert Date: Fri, 15 Nov 1996 14:11:24 -0700 (MST) Subject: Re: Sockets question... > I'm telling you that you have no credibility with your statements. [ ... ] > I never stated it shouldn't. My statement was that your 'blanket' > statements in trying to describe his problem show that you like to see > your email more than you want to fix the problem. [ ... ] > I never stated he didn't know what he was doing, simply that you have > shown an obvious lack of understanding of the problem. Well Nate, you stud, you... If you can solve his problem in 3 notes... SOLVE HIS PROBLEM! Otherwise you are doing exactly what you accuse me of doing. Terry Lambert terry@lambert.org - --- Any opinions in this posting are my own and not those of my present or previous employers. ------------------------------ From: "Rodney W. Grimes" Date: Fri, 15 Nov 1996 13:26:22 -0800 (PST) Subject: Re: FreeBSD 2.2-ALPHA is now available. > > > 4 port does not, 6 port does. > > > > > > Unless the product offering has changed lately. > > > > Thanks Joe, combining your statement with what another person said > > defanitly qualifies the fact that the BB1004 does not have modem > > support, and that the IOAT66 (6-port) does. > > Actually cruising the Boca Web site is handy... ( :-) :-) ) > > http://www.bocaresearch.com/docs/prodlist.htm ... > > Unfortunately their use of "RJ-45" is incorrect, it's actually a ten pin > connector, allowing for full modem control. It is correct, it is called an RJ-45/10, I know, I had to order a bunch of them to make cables when Boca was backorder 30 days on them. You also need a special set of die's for your crimp tool, but it is infact known as an RJ-45, they just left the /10 off :-(. > > Jordan, can you please list this in the serial board section of the > > release/whatever notes: > > > > Boca BB1004 4-Port serial card (Modems NOT supported) > > Boca IOAT66 6-Port serial card (Modems supported) > > Boca BB1008 8-Port serial card (Modems NOT supported) > > Boca BB2016 16-Port serial card (Modems supported) > > That is reasonable. > > I have been toying with the idea of picking up an IOAT66, but I've had > bad experiences with a BB2016 (interrupt problems) and I am a bit > hesitant to try a Boca serial board again. :-) I guess that's fine > since I don't think I know anyone who sells them anyways. But they > ARE very inexpensive! Ahhh... I sell them.... just not a regular item :-) - -- Rod Grimes rgrimes@gndrsh.aac.dev.com Accurate Automation, Inc. Reliable computers for FreeBSD ------------------------------ From: Terry Lambert Date: Fri, 15 Nov 1996 14:06:50 -0700 (MST) Subject: Re: Sockets question... > >How do I read into a structure on a machine that demands aligned > >data access? > > You read into an intermediate buffer and copy it. You have to convert from > network to machine representation anyway, so this isn't (much) more overhead. DCE RPC does not do a conversion if both ends are the same byte order. BSD RPC does not do a conversion on hardware with correct endianess. I think this is why #pragma pack was invented. 8-(. > >nothing would work at all if you couldn't issue a read for n > >bytes that didn't complete until you *got* n bytes. > > Well, I guess the BSD networking code has probably never worked at all. > The read() system call on a socket is based on soreceive(), which > returns up to N bytes. On a non-blocking socket, right? Or in the case of a blocking call for more bytes than SO_RECVBUF (which you can determine via getsockopt and therefore never trigger). Can it return less than N otherwise? The WAITALL case is only for the user buffer larger than SO_RECVBUF, right? Terry Lambert terry@lambert.org - --- Any opinions in this posting are my own and not those of my present or previous employers. ------------------------------ From: Terry Lambert Date: Fri, 15 Nov 1996 14:26:41 -0700 (MST) Subject: Re: Sockets question... > > Otherwise, you have just un-formatted your transport contents. 8-(. > > TCP streams are byte oriented, not record oriented. There is no preservation > of record boundaries with TCP. I am talking about imposing structure on the stream on the basis of forcing the programs to communicate with the stream using record sized chunks. It's irrelevent what happens in the wire, or how the kernel chooses to copy the byte-oriented data into the user buffer on the receiver, so long as the receiver is not falsly notified that the request has been fulfilled. I admit that there is an implication here that I didn't take into account: That two writes to a socket will not be agregated across write buffer boundries into a single packet, OR that the receiver will read an amount such that if agregation occurs, it is irrelevent to the receiver's ability to distinguish the boundries. I have to say that SO_RCVLOWAT and SO_RCVTIMEO screw this considerably, in their roles as vmin/vtime equivalents. 8-(. My advice for someone building a transaction oriented protocol would be to set SO_RCVLOWAT to the same value as SO_RCVBUF, and to never, ever do packets larger than that as part of the formatted data. > > If you want to get technical, according to this description, if you are > > using a SOCK_STREAM, then a read on a blocking socket will act like a > > recv(2) or recvfrom(2) with flags MSG_WAITALL by default. > > read(2), recv(2), and recvfrom(2) all call soreceive() for sockets. read(2) > does NOT set MSG_WAITALL when calling soreceive(). Also: Well, this is probably an error for a blocking socket read, then... 8-(. > * If MSG_WAITALL is set but resid is larger than the receive buffer, > * we have to do the receive in sections, and thus risk returning > * a short count if a timeout or signal occurs after we start. > > So MSG_WAITALL can also return a short count. Only in case of a timeout or a signal, both of which are error conditions. The default behaviour for BSD prior to the introduction of siginterrupt() was to restart all system calls interrupted by signal. If the default behaviour is used, then the only possible failure would be EPIPE, indicating a timeout (loss of connection) (assuming a blocking socket). > > Maybe you should be using SOCK_SEQPACKET instead of SOCK_STREAM? > > That might be difficult. > > A SOCK_SEQPACKET socket [ .... ] is protocol specific, and presently > implemented only for PF_NS. (PF_NS == Xerox Network Systems protocols) Well, bummer deal. 8-(. Terry Lambert terry@lambert.org - --- Any opinions in this posting are my own and not those of my present or previous employers. ------------------------------ From: Karl Denninger Date: Fri, 15 Nov 1996 15:46:43 -0600 (CST) Subject: Re: Sockets question... > > No; what happens for you at the receiving end is that the packets get > > reassembled into the same *stream* of data as they were at the sender. > > The sender can buffer small writes into big chunks, or can fragment big > > writes. If the underlying MTU of the path changes, writes that didn't > > used to have to be fragmented may now have to be. Packet N can get > > lost in the network, while packets N+1, N+2, N+3 and N+4 arrive; when > > the sender retransmits packet N all of a sudden a ton of data "shows > > up" at the receiver. The receiver's job is *solely* to put those > > packets back together into the same stream as the sender sent. > > So at the "read" interface, you *can* count on it arriving in the > same sized chunks as you wrote it, as long as the order and chunk size of > your reads is identical to those of your writes... in other words, > synchronized client and server automatons. > > Right? > > IF I issue a blocking read for 1000 bytes, AND > I issue a blocking write for 1000 bytes, AND > the blocking write returns "1000", > THEN the blocking read will return 1000 bytes OR > it won't return (EPIPE after connection timeout). > > Karl is saying he has synchronized client and server automatons, but > that it's not working for large buffer sizes on the writes. No, Karl is doing this: 1) The *writer* is writing records of variable size with a prefix to indicate how many byte(s) follow. 2) The writer does this ASSUMING that all of the records will get delivered to the reader. 3) When the writer is done, he writes a "no more records follow" flag record. 4) All of those writes return with no errors. 5) The READER gets about 2700 of the records (out of 8500!) and NEVER SEES ANY MORE DATA. It hangs in read()! This does NOT happen with the 2.6.3 development kit and libraries. It RELIABLY happens with -current. - -- - -- Karl Denninger (karl@MCS.Net)| MCSNet - The Finest Internet Connectivity http://www.mcs.net/~karl | T1's from $600 monthly to FULL DS-3 Service | 33 Analog Prefixes, 13 ISDN, Web servers $75/mo Voice: [+1 312 803-MCS1 x219]| Email to "info@mcs.net" WWW: http://www.mcs.net/ Fax: [+1 312 248-9865] | 2 FULL DS-3 Internet links; 400Mbps B/W Internal ------------------------------ From: Bill Fenner Date: Fri, 15 Nov 1996 13:54:58 PST Subject: Re: Sockets question... Terry: >Bill: > Terry: >> >So at the "read" interface, you *can* count on it arriving in the >> >same sized chunks as you wrote it >> >> No, you can *never* count on that, since non-blocking reads from a stream >> socket return as much data as is available, which could be less than you >> asked for. See soo_read() (or soo_rw() in earlier BSD's) and soreceive(). > >By default, the sockets are blocking. You have to go out of your >way to make them non-blocking (ie: loading the gun before you can shoot >yourself in the foot). Terry, STOP SPECULATING. Look at the code. soo_read() is what is called by the read system call, and it calls soreceive(). soreceive() returns whatever is waiting, up to what is requested, unless you set MSG_WAITALL. soo_read() doesn't set MSG_WAITALL, so the semantics of read() on a blocking socket is to return as much data is waiting, up to the amount that was requested. >But on a blocking socket, it doesn't make sense to have to issue multiple >system calls to read chunks of a whole message when you aren't going to >do anything with it until all the reads have been satisfied? The CSRG apparently felt otherwise. Bill ------------------------------ From: Bill Fenner Date: Fri, 15 Nov 1996 13:58:48 PST Subject: Re: Sockets question... In message <199611152146.PAA10779@Jupiter.Mcs.Net> Karl wrote: >This does NOT happen with the 2.6.3 development kit and libraries. It >RELIABLY happens with -current. This points clearly at: a) incorrect code, compiled one way by 2.6.3 and differently by 2.7.2 b) a compiler bug. We should all stop arguing about SOCK_STREAM semantics. Bill ------------------------------ End of hackers-digest V1 #1645 ******************************