From owner-freebsd-questions@freebsd.org Tue Nov 7 19:10:05 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57AF5E626B1 for ; Tue, 7 Nov 2017 19:10:05 +0000 (UTC) (envelope-from freebsd@edvax.de) Received: from mailrelay10.qsc.de (mailrelay10.qsc.de [212.99.163.152]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.antispameurope.com", Issuer "TeleSec ServerPass Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C343A1B18 for ; Tue, 7 Nov 2017 19:10:04 +0000 (UTC) (envelope-from freebsd@edvax.de) Received: from mx01.qsc.de ([213.148.129.14]) by mailrelay10.qsc.de; Tue, 07 Nov 2017 20:09:09 +0100 Received: from r56.edvax.de (port-92-195-23-159.dynamic.qsc.de [92.195.23.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx01.qsc.de (Postfix) with ESMTPS id BC02E3CBF9; Tue, 7 Nov 2017 20:09:08 +0100 (CET) Received: from r56.edvax.de (localhost [127.0.0.1]) by r56.edvax.de (8.14.5/8.14.5) with SMTP id vA7J986x002972; Tue, 7 Nov 2017 20:09:08 +0100 (CET) (envelope-from freebsd@edvax.de) Date: Tue, 7 Nov 2017 20:09:08 +0100 From: Polytropon To: byrnejb@harte-lyne.ca Cc: freebsd-questions@freebsd.org Subject: Re: sed - remove nul lines from file Message-Id: <20171107200908.f9358f33.freebsd@edvax.de> In-Reply-To: <76aef2fd3792a0d9291b90cb74b6924f.squirrel@webmail.harte-lyne.ca> References: <20171107193652.7b0aa08f.freebsd@edvax.de> <76aef2fd3792a0d9291b90cb74b6924f.squirrel@webmail.harte-lyne.ca> Reply-To: Polytropon Organization: EDVAX X-Mailer: Sylpheed 3.1.1 (GTK+ 2.24.5; i386-portbld-freebsd8.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-cloud-security-sender: freebsd@edvax.de X-cloud-security-recipient: freebsd-questions@freebsd.org X-cloud-security-Virusscan: CLEAN X-cloud-security-disclaimer: This E-Mail was scanned by E-Mailservice on mailrelay10.qsc.de with 30A05683483 X-cloud-security-connect: mx01.qsc.de[213.148.129.14], TLS=1, IP=213.148.129.14 X-cloud-security: scantime:.1421 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Nov 2017 19:10:05 -0000 On Tue, 7 Nov 2017 13:54:41 -0500, James B. Byrne wrote: > > On Tue, November 7, 2017 13:36, Polytropon wrote: > > On Tue, 7 Nov 2017 12:12:55 -0500, James B. Byrne via > > freebsd-questions wrote: > >> I have a data file created by an ancient proprietary scripting > >> language called QTP. There is a bug in this program which, on > >> occasion, manifests itself by inserting output records consisting > >> entirely of nul (^@) (\x00) bytes at regular intervals. In the > >> present case every 47th. record consists entirely of nuls. > > > ... > > In this case, awk can also help: > > > > $ awk '(length > 0)' < infile.txt > outfile.txt > > > > This will print all lines which are longer than 0 characters. > > > > Thank you very much. This worked exactly as I required. > > I infer from this that awk does not consider nul a character and its > presence does not count towards the length of a record. Which is > counter intuitive to me. A nul takes up the same space as any other > character so why is it not counted? I would not have tried this > construction for that reason. Even though this example was actually meant for empty lines, i. e., those where the NULs have already been removed (for example with the tr -d command), but it seems that awk does actually ignores the NULs. Let's say this is the test input: foo bar ^@^@^@^@^@^@^@^@ baz meow When fed into the awk command mentioned above, the NULs are magically removed: $ awk '(length > 0)' < nul.txt foo bar baz meow This is an interesting behaviour, but fits the current problem quite well: It removes NULs and emoty lines. :-) -- Polytropon Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ...