Date: Wed, 8 Mar 2017 18:47:24 -0800 From: Alfred Perlstein <alfred@freebsd.org> To: Devin Teske <dteske@freebsd.org> Cc: Julian Elischer <julian@elischer.org>, freebsd-current <freebsd-current@freebsd.org> Subject: Re: I/O semantics of pipe and FIFO. Message-ID: <819671BE-4700-4E73-A697-095BBDE287D4@freebsd.org> In-Reply-To: <B71629F9-A171-4E01-BA0A-39F74C77724B@freebsd.org> References: <20170304214812.GA16845@chaz.gmail.com> <8efdc961-1768-0bc0-715f-4a1e103359d4@elischer.org> <17381773-019a-2181-f00f-1908d04b8d22@freebsd.org> <B71629F9-A171-4E01-BA0A-39F74C77724B@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
I've seen that bug, but I think our bug was more that dd would exit if it go= t back a short read from input. So if you did something like this (maybe not= exactly, but close): dd bs=3D1m | dd bs=3D10m > f.out You might get only 1mb in f.out. This had to do with skipping disklabels and= other such schenanigans.=20 Sent from my iPhone > On Mar 8, 2017, at 4:09 PM, Devin Teske <dteske@freebsd.org> wrote: >=20 > Problem we had found was: >=20 > Executing dd with a closed stdout and stderr would cause the summary messa= ges printed at the end to go into the destination output file. >=20 > For example, >=20 > dd if=3D/dev/zero of=3D/tmp/foo bs=3D1m count=3D1 >=20 > Works fine, but the following: >=20 > dd if=3D/dev/zero of=3D/tmp/foo bs=3D1m count=3D1 >&- 2>&- >=20 > Will cause the summary statistics of dd to appear in /tmp/foo instead of o= n the console. >=20 > The issue is that the summary statistics are send to fd1, which if you clo= se down stdout and stdin, fd1 is actually the output file since it got the l= owest file descriptor available when open(2) was called on the output file. >=20 > This was never fixed because it was deemed =E2=80=9Csilly developer, don=E2= =80=99t close stdout and stderr before invoking dd=E2=80=9D. >=20 > The argument has been made by Jilles T. that it is generally a bad idea to= close down any of the standard file descriptors because it cannot be predic= ted how a particular UNIX utility will react (e.g., in the case of dd, causi= ng a simple printf(3) to go to an unexpected location). > =E2=80=94=20 > Devin >=20 >=20 >> On Mar 4, 2017, at 8:12 PM, Alfred Perlstein <alfred@freebsd.org> wrote: >>=20 >> Devin and I found this when we worked together. I think it was due to so= me situation in dd(1) where short reads would exit pre-maturely, however I m= ay be mis-remembering. Devin, do you recall the specifics? >>=20 >>=20 >>> On 3/4/17 7:44 PM, Julian Elischer wrote: >>>=20 >>> an interesting point to discuss? is our behaviour in this test right? >>> from: "austin-group mailng list (posix standard discussion)" >>>=20 >>> ------ rest of email is quoted ------- >>> On 5/3/17 5:48 am, Stephane Chazelas wrote: >>>=20 >>> 2017-03-04 13:14:08 +0000, Danny Niu: >>>> Hi all. >>>>=20 >>>> I couldn't remember where I saw it saying, that when reading >>>> from a pipe or a FIFO, the read syscall returns the content of >>>> at most one write call. It's a bit similar to the >>>> message-nondiscard semantics of dear old STREAM. >>>>=20 >>>> Currently, I'm reading through the text to find out a bit >>>> more, and I appreciate a bit of pointer on this. >>> [...] >>>=20 >>> (echo x; echo y) | (sleep 1; dd count=3D1 2> /dev/null) >>>=20 >>> outputs both x and y in all of Linux, FreeBSD and Solaris in my >>> tests. >>>=20 >>> That a read wouldn't read what's currently in the pipe would be >>> quite surprising. >>>=20 >>> I also wouldn't expect pipes to store the writes as individual >>> separate message but use one buffer. >>>=20 >>> In: >>>=20 >>> ( >>> dd bs=3D40000 count=3D1 if=3D/dev/zero 2> /dev/null >>> echo first through >&2 >>> dd bs=3D40000 count=3D1 if=3D/dev/zero 2> /dev/null >>> echo second through >&2 >>> ) | (sleep 1; dd bs=3D100000 count=3D1 2> /dev/null) | wc -c >>>=20 >>> That is where the second write blocks because the pipe is full, >>> the reading dd still reads both writes in Linux and Solaris in >>> my tests (on Solaris (10 on amd64 at least), reduce to 20000 >>> instead of 40000 or both writes would block). >>>=20 >>> On FreeBSD, I get only the first write (using 8000 followed by >>> 10000 for instance). >>>=20 >>> FreeBSD is also the only one of the three where >>>=20 >>> dd bs=3D1000000 count=3D1 if=3D/dev/zero | dd bs=3D1000000 count=3D1 | w= c -c >>>=20 >>> Doesn't output 1000000. The others schedule both processes back >>> and forth during their write() and read() system call while the >>> pipe is being filled and emptied several times. >>>=20 >>=20 >=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?819671BE-4700-4E73-A697-095BBDE287D4>