From owner-freebsd-standards@FreeBSD.ORG Sun Apr 7 19:12:59 2013 Return-Path: Delivered-To: freebsd-standards@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 152A491A; Sun, 7 Apr 2013 19:12:59 +0000 (UTC) (envelope-from cedric.blancher@gmail.com) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) by mx1.freebsd.org (Postfix) with ESMTP id D811AAD8; Sun, 7 Apr 2013 19:12:58 +0000 (UTC) Received: by mail-ie0-f181.google.com with SMTP id 17so6035613iea.26 for ; Sun, 07 Apr 2013 12:12:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=rjfbGtvoYwaJdEDHzDy0t2A8GueIK7clV/XT6dnYpKI=; b=tmSPxJm5MLK+RzhGBoF2ZS6CCwEIfBy48XngFtzbldDGLGY/RYhZWzzogCuRwnqmAk 0XuQAv1wtkDisiVhk6gaVRG37gjhrW46hmorOsnKNxNmlyluYD317L6WOhZ8vC+C+hbJ HvzKNOYLjFmEjnJVJTXyxEQrJANTabaWYTi0ufkt5LL3DCTybw+SSqCuaIyFytgy15Iy SnYZ/YVSfWn0NccSAVle73o6epFIytcic61TTR1MOb2NrjpxOvffnwcA+73C6YIg+X/o X7ZvnaLKYE5bYdJK6NruzKb5H3xVVLor7q53cyvuyeXG7bDgjwqA8azsU8IXgFLWXx/7 O3hg== MIME-Version: 1.0 X-Received: by 10.50.153.81 with SMTP id ve17mr4934010igb.24.1365361977954; Sun, 07 Apr 2013 12:12:57 -0700 (PDT) Sender: cedric.blancher@gmail.com Received: by 10.50.40.36 with HTTP; Sun, 7 Apr 2013 12:12:57 -0700 (PDT) In-Reply-To: References: Date: Sun, 7 Apr 2013 21:12:57 +0200 X-Google-Sender-Auth: HnVli7FsgViY2Y6OnjPE8jF2ewY Message-ID: Subject: Fwd: Where does FreeBSD tr -C differ from tr -c? From: Cedric Blancher To: freebsd-hackers@freebsd.org, freebsd-standards@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Apr 2013 19:12:59 -0000 Forwarding to freebsd-hackers@/freebsd-standards@freebsd.org The question remain open and I need help. tr -C is implemented by FreeBSD tr -C but I can't find examples (or a testcase) where tr -c and tr -C differ. Ced PS: Who wrote tr -C and how can I contact the author? ---------- Forwarded message ---------- From: Cedric Blancher Date: 5 April 2013 16:04 Subject: Where does FreeBSD tr -C differ from tr -c? To: freebsd-i18n@freebsd.org Has anyone discovered examples for FreeBSD tr -c producing different output than tr -C? I tried this (ksh93) test script but it NEVER produces a difference in the en_US.utf8 and fr_FR.utf-8 locales: ------ snip ------ builtin rm typeset string typeset -li16 n_ch ; for (( n_ch=1 ; n_ch < 0x5000 ; n_ch++ )) ; do ch="$(printf "\u[${n_ch/~(El)16#/}]")" string+="$ch" done typeset -li16 m1 m2 for (( m1=0x32 ; m1 < 0x3000 ; m1+=7 )) ; do (( m2=m1+1500 )) range="$(printf "\u[${m1/~(El)16#/}]-\u[${m2/~(El)16#/}]")" tr -Cd "$range" <<<"$string" >'res_C' & tr -cd "$range" <<<"$string" >'res_c' & wait res_c="$( <'res_c' )" res_C="$( <'res_C' )" rm 'res_c' 'res_C' if [[ "$res_c" != "$res_C" ]] ; then printf 'DIFFER range=%q\n' "${range}" fi done exit 0 ------ snip ------ So when does tr -C differ from tr -c? I need examples, please... Ced -- Cedric Blancher Institute Pasteur -- Cedric Blancher Institute Pasteur From owner-freebsd-standards@FreeBSD.ORG Sun Apr 7 20:32:06 2013 Return-Path: Delivered-To: freebsd-standards@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B5424A83; Sun, 7 Apr 2013 20:32:06 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (unknown [IPv6:2001:610:1108:5012::107]) by mx1.freebsd.org (Postfix) with ESMTP id 803E5EF0; Sun, 7 Apr 2013 20:32:06 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id AF4C6120207; Sun, 7 Apr 2013 22:31:51 +0200 (CEST) Received: by snail.stack.nl (Postfix, from userid 1677) id 6A0322848C; Sun, 7 Apr 2013 22:31:51 +0200 (CEST) Date: Sun, 7 Apr 2013 22:31:51 +0200 From: Jilles Tjoelker To: Cedric Blancher Subject: Re: Fwd: Where does FreeBSD tr -C differ from tr -c? Message-ID: <20130407203151.GA47134@stack.nl> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hackers@freebsd.org, freebsd-standards@freebsd.org X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Apr 2013 20:32:06 -0000 On Sun, Apr 07, 2013 at 09:12:57PM +0200, Cedric Blancher wrote: > Forwarding to freebsd-hackers@/freebsd-standards@freebsd.org > The question remain open and I need help. tr -C is implemented by > FreeBSD tr -C but I can't find examples (or a testcase) where tr -c > and tr -C differ. Reading the rationale of POSIX, here is an example of a difference: % printf 'a\200'|LC_ALL=en_US.US-ASCII tr -cd '\000-\177'|hd 00000000 61 |a| 00000001 % printf 'a\200'|LC_ALL=en_US.US-ASCII tr -Cd '\000-\177'|hd 00000000 61 80 |a.| 00000002 Because the bytes 128..255 are not characters in us-ascii, they cannot be removed with -Cd, only with -cd. Here is another difference (using LC_CTYPE=en_US.UTF-8, rest C): % echo $'\U0001a000'|tr -cd '\U0001a000'|hd % echo $'\U0001a000'|tr -Cd '\U0001a000'|hd 00000000 f0 9a 80 80 |....| 00000004 The cause is that iswrune(3) returns false for the unassigned code point U+0001A000. This may well contain bugs because Unicode adds new characters from time to time and our tables seem to be updated very rarely. POSIX also says things about collation order. You may not have detected this because FreeBSD does not implement LC_COLLATE for multibyte locales yet. > PS: Who wrote tr -C and how can I contact the author? You can read the Subversion logs but people may no longer be around. -- Jilles Tjoelker From owner-freebsd-standards@FreeBSD.ORG Mon Apr 8 11:06:52 2013 Return-Path: Delivered-To: freebsd-standards@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B316D239 for ; Mon, 8 Apr 2013 11:06:52 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id A48F436A for ; Mon, 8 Apr 2013 11:06:52 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.6/8.14.6) with ESMTP id r38B6qqC057388 for ; Mon, 8 Apr 2013 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.6/8.14.6/Submit) id r38B6qxC057386 for freebsd-standards@FreeBSD.org; Mon, 8 Apr 2013 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 8 Apr 2013 11:06:52 GMT Message-Id: <201304081106.r38B6qxC057386@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-standards@FreeBSD.org Subject: Current problem reports assigned to freebsd-standards@FreeBSD.org X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Apr 2013 11:06:52 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o stand/176683 standards catman pages shall be stored in /var (/usr/local/var,/ o stand/176412 standards newfs writes by default, compare to bsdlabel/disklabel o stand/175711 standards When the server has more than 3 days, rising interrupt o stand/175453 standards Catching C++ std::bad_cast doesn't work in FreeBSD 9.1 o stand/174938 standards Problem statement: iSCSI target failure o stand/173421 standards [libc] [patch] strptime() accepts formats that should o stand/173087 standards pax(1) does not support the pax interchange format o stand/172805 standards Fix catopen(3)'s EINVAL usage and document EFTYPE o stand/172276 standards POSIX: {get,set}groups gidsetsize is u_int not int o stand/172215 standards localeconv() grouping appears not to match POSIX o stand/170403 standards wrong ntohs expression type tickling clang o stand/169697 standards syslogd(8) is not BOM aware o stand/166349 standards Support the assignment-allocation character for fscanf o stand/164787 standards dirfd() function not available when _POSIX_C_SOURCE is o kern/164674 standards [patch] [libc] vfprintf/vfwprintf return error (EOF) o o stand/162434 standards getaddrinfo: addrinfo.ai_family is an address family, o stand/150093 standards C++ std::locale support is broken o stand/130067 standards Wrong numeric limits in system headers? o stand/124860 standards flockfile(3) doesn't work when the memory has been exh o stand/121921 standards [patch] Add leap second support to at(1), atrun(8) o stand/116477 standards rm(1): rm behaves unexpectedly when using -r and relat o bin/116413 standards incorrect getconf(1) handling of unsigned constants gi o stand/116081 standards make does not work with the directive sinclude o stand/100017 standards [Patch] Add fuser(1) functionality to fstat(1) a stand/86484 standards [patch] mkfifo(1) uses wrong permissions o stand/82654 standards C99 long double math functions are missing o stand/81287 standards [patch] fingerd(8) might send a line not ending in CRL a stand/80293 standards sysconf() does not support well-defined unistd values o stand/79056 standards [feature request] [atch] regex(3) regression tests o stand/70813 standards [patch] ls(1) not Posix compliant o stand/66357 standards make POSIX conformance problem ('sh -e' & '+' command- s kern/64875 standards [libc] [patch] [request] add a system call: fdatasync( o stand/56476 standards [patch] cd9660 unicode support simple hack o stand/54410 standards one-true-awk not POSIX compliant (no extended REs) o stand/46119 standards Priority problems for SCHED_OTHER using pthreads o stand/44365 standards [headers] [patch] [request] introduce ulong and unchar a stand/41576 standards ln(1): replacing old dir-symlinks a docs/26003 standards getgroups(2) lists NGROUPS_MAX but not syslimits.h s stand/24590 standards timezone function not compatible witn Single Unix Spec o stand/21519 standards sys/dir.h should be deprecated some more s bin/14925 standards getsubopt isn't poisonous enough 41 problems total. From owner-freebsd-standards@FreeBSD.ORG Wed Apr 10 00:50:01 2013 Return-Path: Delivered-To: freebsd-standards@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 0B7DC788 for ; Wed, 10 Apr 2013 00:50:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id E4BD993D for ; Wed, 10 Apr 2013 00:50:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.6/8.14.6) with ESMTP id r3A0o06g094783 for ; Wed, 10 Apr 2013 00:50:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.6/8.14.6/Submit) id r3A0o0Cd094782; Wed, 10 Apr 2013 00:50:00 GMT (envelope-from gnats) Resent-Date: Wed, 10 Apr 2013 00:50:00 GMT Resent-Message-Id: <201304100050.r3A0o0Cd094782@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-standards@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Matthew Rezny Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 444AC772 for ; Wed, 10 Apr 2013 00:49:05 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 236DD934 for ; Wed, 10 Apr 2013 00:49:05 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.5/8.14.5) with ESMTP id r3A0n3gc060894 for ; Wed, 10 Apr 2013 00:49:03 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.5/8.14.5/Submit) id r3A0n378060893; Wed, 10 Apr 2013 00:49:03 GMT (envelope-from nobody) Message-Id: <201304100049.r3A0n378060893@red.freebsd.org> Date: Wed, 10 Apr 2013 00:49:03 GMT From: Matthew Rezny To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Subject: standards/177742: conflict of dd's bs= option with use of conv=sparse X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 00:50:01 -0000 >Number: 177742 >Category: standards >Synopsis: conflict of dd's bs= option with use of conv=sparse >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-standards >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Apr 10 00:50:00 UTC 2013 >Closed-Date: >Last-Modified: >Originator: Matthew Rezny >Release: 9.1-RELEASE >Organization: RezTek, s.r.o. >Environment: FreeBSD 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243826: Tue Dec 4 06:55:39 UTC 2012 root@obrian.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386 >Description: Invoking dd with a blocksize specified disables sparse conversion. The dd utility has numerous options which affect the behavior of other options. One of the broader reaching options is blocksize as altering the size of blocks alters any options that work in terms of blocks (e.g. seek, skip, count, etc). That change is obvious, but the impact on conversion options in non-obvious. Excerpts from dd's man page: bs=n Set both input and output block size to n bytes, superseding the ibs and obs operands. If no conversion values other than noerror, notrunc or sync are specified, then each input block is copied to the output as a single block without any aggregation of short blocks. conv=value[,value ...] Where value is one of the symbols from the following list. sparse If one or more output blocks would consist solely of NUL bytes, try to seek the output file by the required space instead of filling them with NULs, resulting in a sparse file. The documentation of the bs= option only mentions an affect on aggregation of blocks with some conversions operations but does not state anything about disabling conversion. The sparse conversion is not explicitly documented as being affected by the bs option. As it states the sparse conversion works on blocks of output, it's implied that the effect of bs on sparse would be to set the size of of a run of NULs that must be found to trigger the lseek. Looking at the source code to dd, the function dd_out(int force) works as expected: loop through the output buffer, if sparse option is set then set sparse flag, if output block has any non-NULLs then clear sparse flag, if sparse && !force then seek otherwise write output block, continue through loop. Moving on to the function dd_in() reveals the smoking gun along with an interesting comment. /* * POSIX states that if bs is set and no other conversions * than noerror, notrunc or sync are specified, the block * is output without buffering as it is read. */ if (ddflags & C_BS) { out.dbcnt = in.dbcnt; dd_out(1); in.dbcnt = 0; continue; } The comment says that if we aren't doing conversions then we output without buffering, but that is not quite what the code does. If the bs= option was set, then regardless of conversions that may be specified, dd_out() is called with force=1. When force is set, then dd_out() will skip all the sparse processing. If I want to efficiently image a disk that is not largely free, I would expect to be able to do so with something like dd if=/dev/da0 of=disk.img bs=64k conv=sparse. Unfortunately, doing so will end up with a full size image of the disk. If I want to actually get sparse conversion, I must drop the bs= options and take the speed hit of single sector I/O requests. The difference in total throughput of 1 sector vs 128 sectors per I/O request is more than 10x. Worse than that performance hit, if there is some reason I want to see multiple sectors of NULLs before doing a lseek on the output (perhaps same size as file system clusters) there is just no way to do so. >How-To-Repeat: # dd if=/dev/zero of=test1 count=1024 conv=sparse 1024+0 records in 1024+0 records out 524288 bytes transferred in 0.000758 secs (691734274 bytes/sec) # dd if=/dev/zero of=test2 bs=1k count=512 conv=sparse 512+0 records in 512+0 records out 524288 bytes transferred in 0.001709 secs (306783378 bytes/sec) # ls -l total 1224 drwxrwxr-x 2 root operator 512 Apr 10 02:10 .snap -rw-r--r-- 1 root wheel 524288 Apr 10 02:14 test1 -rw-r--r-- 1 root wheel 524288 Apr 10 02:15 test2 # du -a 8 ./.snap 128 ./test1 1088 ./test2 1232 . With /dev/zero as the input and conv=sparse, the output should be nothing and the space on disk should be minimal. When bs= is part of the invocation, we can see that the output is full size. >Fix: Make the code do what the comment says it should. Call dd_out with force iff no conversion flags are set other than noerror, notrunc or sync. Suggested patch: if (ddflags & C_BS) { out.dbcnt = in.dbcnt; - dd_out(1); + dd_out(dd_flags & !(C_NOERROR | C_NOTRUNC | C_SYNC) == 0); in.dbcnt = 0; continue; } This fix only addresses the sparse conversion as that happens on the dd_out() function. All other conversions are normally done as the last call made by dd_in(). The section of bad code in dd_in() will call dd_out() whenever the bs= option is specified. Without adding additional checks, it appears all other conversions will also be skipped. This block of code look like an attempt at optimization gone bad. >Release-Note: >Audit-Trail: >Unformatted: From owner-freebsd-standards@FreeBSD.ORG Wed Apr 10 14:10:01 2013 Return-Path: Delivered-To: freebsd-standards@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E9B22B21 for ; Wed, 10 Apr 2013 14:10:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id DAC23114 for ; Wed, 10 Apr 2013 14:10:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.6/8.14.6) with ESMTP id r3AEA10t044881 for ; Wed, 10 Apr 2013 14:10:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.6/8.14.6/Submit) id r3AEA1vH044880; Wed, 10 Apr 2013 14:10:01 GMT (envelope-from gnats) Date: Wed, 10 Apr 2013 14:10:01 GMT Message-Id: <201304101410.r3AEA1vH044880@freefall.freebsd.org> To: freebsd-standards@FreeBSD.org Cc: From: Matthew Rezny Subject: Re: standards/177742: conflict of dd's bs= option with use of conv=sparse X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Matthew Rezny List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 14:10:02 -0000 The following reply was made to PR standards/177742; it has been noted by GNATS. From: Matthew Rezny To: bug-followup@FreeBSD.org Cc: Subject: Re: standards/177742: conflict of dd's bs= option with use of conv=sparse Date: Wed, 10 Apr 2013 15:31:08 +0200 The patch I suggested got a little messed up by the web form, and it also contained a typo. Further, I had neglected to consider the C_BS flag itself should be present after masking off the few allowed flags, so the patch should be amended as such follows. if (ddflags & C_BS) { out.dbcnt = in.dbcnt; - dd_out(1); + dd_out((ddflags & !(C_NOERROR | C_NOTRUNC | C_SYNC)) == C_BS); in.dbcnt = 0; continue; } This patch has been tested to confirm conv=sparse now works as expected with bs= set. No other conversions have been checked with the bs= option and from reading the code I don't think they will. From owner-freebsd-standards@FreeBSD.ORG Wed Apr 10 19:01:21 2013 Return-Path: Delivered-To: freebsd-standards@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id C8567880 for ; Wed, 10 Apr 2013 19:01:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 6D2A424C for ; Wed, 10 Apr 2013 19:01:21 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r3AJ1Anl044868; Wed, 10 Apr 2013 22:01:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.2 kib.kiev.ua r3AJ1Anl044868 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r3AJ1A7J044867; Wed, 10 Apr 2013 22:01:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 10 Apr 2013 22:01:10 +0300 From: Konstantin Belousov To: Matthew Rezny Subject: Re: standards/177742: conflict of dd's bs= option with use of conv=sparse Message-ID: <20130410190110.GM2930@kib.kiev.ua> References: <201304101410.r3AEA1vH044880@freefall.freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="bZ2MuwyI/0uB8yuJ" Content-Disposition: inline In-Reply-To: <201304101410.r3AEA1vH044880@freefall.freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-standards@FreeBSD.org X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2013 19:01:21 -0000 --bZ2MuwyI/0uB8yuJ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Apr 10, 2013 at 02:10:01PM +0000, Matthew Rezny wrote: > The following reply was made to PR standards/177742; it has been noted by= GNATS. >=20 > From: Matthew Rezny > To: bug-followup@FreeBSD.org > Cc: =20 > Subject: Re: standards/177742: conflict of dd's bs=3D option with use of > conv=3Dsparse > Date: Wed, 10 Apr 2013 15:31:08 +0200 >=20 > The patch I suggested got a little messed up by the web form, and it > also contained a typo. Further, I had neglected to consider the > C_BS flag itself should be present after masking off the few allowed > flags, so the patch should be amended as such follows. > =20 > if (ddflags & C_BS) { > out.dbcnt =3D in.dbcnt; > - dd_out(1); > + dd_out((ddflags & !(C_NOERROR | C_NOTRUNC | C_SYNC)) =3D=3D C_BS); > in.dbcnt =3D 0; > continue; > } > =20 > This patch has been tested to confirm conv=3Dsparse now works as expected > with bs=3D set. No other conversions have been checked with the bs=3D > option and from reading the code I don't think they will. I somehow followed what you wrote. Still, I think your patch is not correct. Its result is always using the block coalescing mode there. The issue is that the '!' operator result is defined to be either 0 or 1, which mask and-ed with ddflags never could be equal to the C_BS =3D=3D 4. You probably mean '~' ? In fact, the test in the containing if () means that C_BS is guaranteed to be set there. So my question is, do you want to explicitely check that no other flags are set, except noerror, notrunc, sync and bs ? Or is it yet another bug ? --bZ2MuwyI/0uB8yuJ Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJRZbb1AAoJEJDCuSvBvK1BO8UP/1yyeFvOhJJWE+6kd0S8sHUq 0KCIG1h//CyNlHgeSvzWlFdklzCkd08f3/v4lqgrwhsDA7P/39jh4BE/9FD5hM3P izjkTCUKUenKb3SoW+aK6++k3DSx8eXSBCEB5Omon++uZDJkYsPhO6stZ+5MlMEQ 3UYSDZVUUpkVe8m2S6bmDKnx19bRoY4QpDHWr2T0WyfV/gJFvg47L8netIZMA5Tl EBFFVQQAmB2RuktKDBjZazy2C7K7+0500A7VYQjJT8iOCZh8CDE8NjLDSpcPGdcn A62L0a0c+JeiSkfq0ShgtnPcfcF56SnOOt/p/fodOFSuq3bNCSwKFWB/wLfN4RCE rNf8sx/t5gBwwL+Gc1z4ksd9J1Ik77amK0vNnOEvQHdQsaduVUHZ/3CKOR2nUyjL 7Ge3yNk4H3tF0sw/eEzJGmYTJPIFiMJUV+euuU7xHEVIkBQPPdq4mQNe8FXuhkcb sg1CG7npqWqPwEj1c0xKXQp8vLj1ux002MHpxFUMiW2MBApMo/rZ4d2ktIYBQu4/ g3EI75jUulvzy5e6VOd3GEaenKSrmBK5AhocLm8Vd0tCRlVHDeGIIOR8aDdTJYeG HhVC1l/tbnc7UuQ/ReHIe19/XkQEQgFW+EZ+pUqst2Q/7+9358OEXyYS31Gr+ZC+ UGL+syBA6IijCv3VGLYI =OIM8 -----END PGP SIGNATURE----- --bZ2MuwyI/0uB8yuJ-- From owner-freebsd-standards@FreeBSD.ORG Thu Apr 11 04:00:02 2013 Return-Path: Delivered-To: freebsd-standards@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3FF94834 for ; Thu, 11 Apr 2013 04:00:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 228AADB0 for ; Thu, 11 Apr 2013 04:00:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.6/8.14.6) with ESMTP id r3B401wh007851 for ; Thu, 11 Apr 2013 04:00:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.6/8.14.6/Submit) id r3B401UV007850; Thu, 11 Apr 2013 04:00:01 GMT (envelope-from gnats) Date: Thu, 11 Apr 2013 04:00:01 GMT Message-Id: <201304110400.r3B401UV007850@freefall.freebsd.org> To: freebsd-standards@FreeBSD.org Cc: From: Matthew Rezny Subject: Re: standards/177742: conflict of dd's bs= option with use of conv=sparse X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Matthew Rezny List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Apr 2013 04:00:02 -0000 The following reply was made to PR standards/177742; it has been noted by GNATS. From: Matthew Rezny To: Konstantin Belousov , bug-followup@FreeBSD.org Cc: Subject: Re: standards/177742: conflict of dd's bs= option with use of conv=sparse Date: Thu, 11 Apr 2013 05:44:59 +0200 On Wed, 10 Apr 2013 22:01:10 +0300 Konstantin Belousov wrote: > On Wed, Apr 10, 2013 at 02:10:01PM +0000, Matthew Rezny wrote: > > The following reply was made to PR standards/177742; it has been > > noted by GNATS. > > > > From: Matthew Rezny > > To: bug-followup@FreeBSD.org > > Cc: > > Subject: Re: standards/177742: conflict of dd's bs= option with use > > of conv=sparse > > Date: Wed, 10 Apr 2013 15:31:08 +0200 > > > > The patch I suggested got a little messed up by the web form, and > > it also contained a typo. Further, I had neglected to consider the > > C_BS flag itself should be present after masking off the few > > allowed flags, so the patch should be amended as such follows. > > > > if (ddflags & C_BS) { > > out.dbcnt = in.dbcnt; > > - dd_out(1); > > + dd_out((ddflags & !(C_NOERROR | C_NOTRUNC | C_SYNC)) == C_BS); > > in.dbcnt = 0; > > continue; > > } > > > > This patch has been tested to confirm conv=sparse now works as > > expected with bs= set. No other conversions have been checked with > > the bs= option and from reading the code I don't think they will. > > I somehow followed what you wrote. > > Still, I think your patch is not correct. Its result is always using > the block coalescing mode there. The issue is that the '!' operator > result is defined to be either 0 or 1, which mask and-ed with ddflags > never could be equal to the C_BS == 4. You probably mean '~' ? > > In fact, the test in the containing if () means that C_BS is > guaranteed to be set there. > > So my question is, do you want to explicitely check that no other > flags are set, except noerror, notrunc, sync and bs ? Or is it yet > another bug ? > Sorry about that, your guess is right. I meant dd_out((ddflags & ~(C_NOERROR | C_NOTRUNC | C_SYNC)) == C_BS); I don't know what is more embarrassing, the number of errors copying from one screen to another, or the fact I didn't catch the obviously bad syntax. I am e-mailing from a separate PC than the one on which I am doing this work, glancing at text on one a then going to the other a meter away and trying to type it again from memory. Error prone indeed. And yes, the test in the if means C_BS has to be set, which is what I remembered after the initial report. I found the odd behavior, looked at the code via svnweb, wrote up the PR, then went to test the patch. In that process I realize the omission, go to amend the PR and notice I introduced the spurious _ in the middle of ddflags when typing earlier and correct that but completely miss the typo where I hit ! instead of ~. Adjacent keys, both a form of negation, but critically different in use. To get to your question about do I really want to verify that no flags are set other than noerror, notrunc, sync and bs, the answer to that is yes. Yes because that is the test condition the comment alludes to but was not present. Does that make sense overall? Not really, but then again, the flow of dd_in() stops making any sense past this point. I started from the point of figuring out why the force flag would be set and altered the call to dd_out() to not set force when we are doing some conversions. But really, any conversion other than sparse is normally done on the in side, only sparse conversion happens on the out side (which is is what the force flag would bypass). I can't actually figure out why the force flag is needed in any situation except when dd_close() calls to dd_out(). All other cases that should always be set false. Looking not just at my problem but the overall situation, it seems the logic in dd_in() does not hold up. The last line, (*cfunc)();, actually invokes the appropriate conversion(s) and then calls dd_out(0). With the simple test on line 361, we will never get down to that call if bs= option is used. Really, to make the code fit the comment, it should probably be something like if ((ddflags & ~(C_NOERROR | C_NOTRUNC | C_SYNC)) == C_BS) { out.dbcnt = in.dbcnt; dd_out(0); in.dbcnt = 0; continue; } That would actually perform the test early enough to only apply this small optimization when really no conversions are done. The simple if C_BS then dd_out() and continue looks like a small optimization attempt gone bad. Some effort was made to avoid calling the conversion function and not leave anything in the buffer between output rounds, but without sufficient checks it ends up meaning that using the bs= option disables all "conversions" except those that are applied during input read stage (noerro, notrunc, sync). All things considered, completely removing lines 361-366 of dd.c would be the most straightforward path to correct output results (judged by the result on disk, ignoring the comment about POSIX rules about buffering which really shouldn't make a difference, the correctness of the result matters more than the path taken to arrive at it).