From owner-freebsd-current@FreeBSD.ORG Wed Mar 6 10:49:52 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 94942ADD; Wed, 6 Mar 2013 10:49:52 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (lrosenman-1-pt.tunnel.tserv8.dal1.ipv6.he.net [IPv6:2001:470:1f0e:3ad::2]) by mx1.freebsd.org (Postfix) with ESMTP id 4F64EF4; Wed, 6 Mar 2013 10:49:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:Subject:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=tAQAYEVg02koOwrKlJnKERvuAHEpjqEJwR5tvv+9A4Y=; b=cX9zzBg965zk3HjU7EobP/ny6yH/vXnO3ZDlN14lSalE+mBVrrjjnjAcXI3sokd8xdUzeqwk7OAzoVqPuCK6FMVTAAmZHdd77ukCMNswrVyWzTp6fhGOJ2ZLwNOxHKJWe+CMdeUA+JLPTa6fn3TwbTVNdDpglGOcq7/RAFHSOnM=; Received: from localhost.lerctr.org ([127.0.0.1]:55686 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpa (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UDBuv-000HNw-R0; Wed, 06 Mar 2013 04:49:51 -0600 Received: from cpe-72-182-19-162.austin.res.rr.com ([72.182.19.162]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Wed, 06 Mar 2013 04:49:48 -0600 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 06 Mar 2013 04:49:48 -0600 From: Larry Rosenman To: , Subject: Fwd: Re: zfs send/recv invalid data Message-ID: <46d966dd574cd8097d4972213c73e9be@webmail.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/0.8.5 X-Spam-Score: -3.5 (---) X-LERCTR-Spam-Score: -3.5 (---) X-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-LERCTR-Spam-Report: SpamScore (-3.5/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.628 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2013 10:49:52 -0000 I forgot to add current/stable to the list TL;DR: there seems(!) to be something(!) unclean about an ssh path between an 8.3-STABLE(r247820) and 10.0-CURRENT(r247826) box such that a zfs send stream is corrupted in transit. below is the thread from -fs about it, with sshd configs from both sides. If I copy the stream it works, but piping through ssh does NOT. -------- Original Message -------- Subject: Re: zfs send/recv invalid data Date: 2013-03-06 04:46 From: Larry Rosenman To: Steven Hartland Cc: Ronald Klop , On 2013-03-06 02:38, Steven Hartland wrote: > ----- Original Message ----- From: "Larry Rosenman" >>>>>>>> I received an "invalid data" in a zfs send (from 8.3) / zfs >>>>>>>> recv (to 10.0) of a -R -I stream. >>>>>>>> What data do I need to gather to figure out what side and >>>>>>>> what's wrong? >>>>>>>> I've already started zpool scrubs on both sides. >>>>>>>> I can insert a tee to grab the stream on either/both sides if >>>>>>>> that would help. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> Is the problem repeatable or is it just a network glitch? >>>>>>> Ronald. >>>>>> Repeatable....... >>>>> Here is the exact error message: >>>>> receiving incremental stream of vault/home/ctr@2013-03-05-test3 >>>>> into zroot/backups/TBH/home/ctr@2013-03-05-test3 >>>>> cannot receive incremental stream: invalid backup stream >>>>> this is the script I'm running: >>>>> #!/bin/sh >>>>> DATE=`date "+%Y-%m-%d-BUG-REPRO"` >>>>> DATE2=`date -v "-1d" "+%Y-%m-%d"` >>>>> # snap the source >>>>> ssh root@tbh.lerctr.org zfs snapshot -r vault@${DATE} >>>>> # zfs copy the source to here. >>>>> ssh root@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2} >>>>> vault@${DATE} | \ >>>>> tee /tmp/backup.stream.send.${DATE} | \ >>>>> ssh home.lerctr.org \"tee /tmp/backup.stream.receive.${DATE} >>>>> | zfs recv -u -v -d zroot/backups/TBH\"" >>>>> # make sure we NEVER allow the backup stuff to automount. >>>>> /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \ >>>>> awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh >>>>> both streams are in http://www.lerctr.org/~ler/ZFS_RECV >>>> Your send and receive sides differ, which indicates your ssh >>>> shell my not be clean. >>>> Looking at the receive side its got what looks like a mail >>>> message appended. >>>> I suspect if you manually copy the receive copy to the 10 machine >>>> and >>>> the receive it will work fine. >>> we're copying mail files........ >>> and it still fails.... >>> >> I've put more example send/recv files in that directory. >> we're copying home dirs, which include lots of mail. >> (this one is my wife's) >> Ideas? >> I *CAN* give access to both sides via ssh..... > The copy of the data stream on both sides should be identical > though and its not, which leads me to believe something is > corrupting the data on the way. Try the following:- > >> From source:- > zfs send -R -D -I vault@${DATE2} vault@${DATE} > test.stream > scp test.stream home.lerctr.org:~/ >> From target: > zfs recv -u -v -d zroot/backups/TBH < test.stream > If this works then there is something unclean about your ssh > shell. > Regards > Steve > send side: # zfs send -R -D -I vault@2013-03-05 vault@2013-03-06 >/tmp/send.stream # openssl md5 /tmp/send.stream MD5(/tmp/send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33 # scp /tmp/send.stream home:/tmp/send.stream send.stream 100% 1180MB 2.5MB/s 07:44 # uname -a FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 r247820: Mon Mar 4 18:08:11 CST 2013 root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64 # Receive side: # uname -a FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 r247826: Mon Mar 4 19:59:08 CST 2013 root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64 # openssl md5 /tmp/send.stream MD5(/tmp/send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33 # zfs recv -F -u -v -d zroot/backups/TBH < /tmp/send.stream # So, you are correct that something(tm) is unclean about the ssh path. adding -current and -stable for diagnosing ssh issue. sshd config on the 8.3-STABLE box: # cat /etc/ssh/sshd_config # $OpenBSD: sshd_config,v 1.87 2012/07/10 02:19:15 djm Exp $ # $FreeBSD: stable/8/crypto/openssh/sshd_config 247521 2013-03-01 02:06:04Z des $ # This is the sshd server system-wide configuration file. See # sshd_config(5) for more information. # This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin # The strategy used for options in the default sshd_config shipped with # OpenSSH is to specify options with their default value where # possible, but leave them commented. Uncommented options override the # default value. # Note that some of FreeBSD's defaults differ from OpenBSD's, and # FreeBSD has a few additional options. #Port 22 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: # Disable legacy (protocol version 1) support in the server for new # installations. In future the default will change to require explicit # activation of protocol 1 Protocol 2 # HostKey for protocol version 1 #HostKey /etc/ssh/ssh_host_key # HostKeys for protocol version 2 #HostKey /etc/ssh/ssh_host_rsa_key #HostKey /etc/ssh/ssh_host_dsa_key # Lifetime and size of ephemeral version 1 server key #KeyRegenerationInterval 1h #ServerKeyBits 1024 # Logging # obsoletes QuietMode and FascistLogging #SyslogFacility AUTH #LogLevel INFO # Authentication: #LoginGraceTime 2m PermitRootLogin yes #StrictModes yes #MaxAuthTries 6 #MaxSessions 10 #RSAAuthentication yes #PubkeyAuthentication yes # The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2 # but this is overridden so installations will only check .ssh/authorized_keys #AuthorizedKeysFile .ssh/authorized_keys #AuthorizedPrincipalsFile none # For this to work you will also need host keys in /etc/ssh/ssh_known_hosts #RhostsRSAAuthentication no # similar for protocol version 2 #HostbasedAuthentication no # Change to yes if you don't trust ~/.ssh/known_hosts for # RhostsRSAAuthentication and HostbasedAuthentication #IgnoreUserKnownHosts no # Don't read the user's ~/.rhosts and ~/.shosts files #IgnoreRhosts yes # Change to yes to enable built-in password authentication. #PasswordAuthentication no #PermitEmptyPasswords no # Change to no to disable PAM authentication #ChallengeResponseAuthentication yes # Kerberos options #KerberosAuthentication no #KerberosOrLocalPasswd yes #KerberosTicketCleanup yes #KerberosGetAFSToken no # GSSAPI options #GSSAPIAuthentication no #GSSAPICleanupCredentials yes # Set this to 'no' to disable PAM authentication, account processing, # and session processing. If this is enabled, PAM authentication will # be allowed through the ChallengeResponseAuthentication and # PasswordAuthentication. Depending on your PAM configuration, # PAM authentication via ChallengeResponseAuthentication may bypass # the setting of "PermitRootLogin without-password". # If you just want the PAM account and session checks to run without # PAM authentication, then enable this but set PasswordAuthentication # and ChallengeResponseAuthentication to 'no'. #UsePAM yes #AllowAgentForwarding yes #AllowTcpForwarding yes #GatewayPorts no #X11Forwarding yes #X11DisplayOffset 10 #X11UseLocalhost yes #PrintMotd yes #PrintLastLog yes #TCPKeepAlive yes #UseLogin no #UsePrivilegeSeparation sandbox #PermitUserEnvironment no #Compression delayed ClientAliveInterval 120 ClientAliveCountMax 200000 #UseDNS yes #PidFile /var/run/sshd.pid #MaxStartups 10 #PermitTunnel no #ChrootDirectory none #VersionAddendum FreeBSD-20120901 # no default banner path #Banner none # override default of no subsystems Subsystem sftp /usr/libexec/sftp-server # Disable HPN tuning improvements. #HPNDisabled no # Buffer size for HPN to non-HPN connections. #HPNBufferSize 2048 # TCP receive socket buffer polling for HPN. Disable on non autotuning kernels. #TcpRcvBufPoll yes # Allow the use of the NONE cipher. #NoneEnabled no # Example of overriding settings on a per-user basis #Match User anoncvs # X11Forwarding no # AllowTcpForwarding no # ForceCommand cvs server # sshd config on the 10.0-CURRENT: # cat /etc/ssh/sshd_config # $OpenBSD: sshd_config,v 1.87 2012/07/10 02:19:15 djm Exp $ # $FreeBSD: head/crypto/openssh/sshd_config 240075 2012-09-03 16:51:41Z des $ # This is the sshd server system-wide configuration file. See # sshd_config(5) for more information. # This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin # The strategy used for options in the default sshd_config shipped with # OpenSSH is to specify options with their default value where # possible, but leave them commented. Uncommented options override the # default value. # Note that some of FreeBSD's defaults differ from OpenBSD's, and # FreeBSD has a few additional options. #Port 22 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: # The default requires explicit activation of protocol 1 #Protocol 2 # HostKey for protocol version 1 #HostKey /etc/ssh/ssh_host_key # HostKeys for protocol version 2 #HostKey /etc/ssh/ssh_host_rsa_key #HostKey /etc/ssh/ssh_host_dsa_key #HostKey /etc/ssh/ssh_host_ecdsa_key # Lifetime and size of ephemeral version 1 server key #KeyRegenerationInterval 1h #ServerKeyBits 1024 # Logging # obsoletes QuietMode and FascistLogging #SyslogFacility AUTH #LogLevel INFO # Authentication: #LoginGraceTime 2m PermitRootLogin yes #StrictModes yes #MaxAuthTries 6 #MaxSessions 10 #RSAAuthentication yes #PubkeyAuthentication yes # The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2 # but this is overridden so installations will only check .ssh/authorized_keys AuthorizedKeysFile .ssh/authorized_keys #AuthorizedPrincipalsFile none # For this to work you will also need host keys in /etc/ssh/ssh_known_hosts #RhostsRSAAuthentication no # similar for protocol version 2 #HostbasedAuthentication no # Change to yes if you don't trust ~/.ssh/known_hosts for # RhostsRSAAuthentication and HostbasedAuthentication #IgnoreUserKnownHosts no # Don't read the user's ~/.rhosts and ~/.shosts files #IgnoreRhosts yes # Change to yes to enable built-in password authentication. #PasswordAuthentication no #PermitEmptyPasswords no # Change to no to disable PAM authentication #ChallengeResponseAuthentication yes # Kerberos options #KerberosAuthentication no #KerberosOrLocalPasswd yes #KerberosTicketCleanup yes #KerberosGetAFSToken no # GSSAPI options #GSSAPIAuthentication no #GSSAPICleanupCredentials yes # Set this to 'no' to disable PAM authentication, account processing, # and session processing. If this is enabled, PAM authentication will # be allowed through the ChallengeResponseAuthentication and # PasswordAuthentication. Depending on your PAM configuration, # PAM authentication via ChallengeResponseAuthentication may bypass # the setting of "PermitRootLogin without-password". # If you just want the PAM account and session checks to run without # PAM authentication, then enable this but set PasswordAuthentication # and ChallengeResponseAuthentication to 'no'. #UsePAM yes #AllowAgentForwarding yes #AllowTcpForwarding yes #GatewayPorts no #X11Forwarding yes #X11DisplayOffset 10 #X11UseLocalhost yes #PrintMotd yes #PrintLastLog yes #TCPKeepAlive yes #UseLogin no #UsePrivilegeSeparation sandbox #PermitUserEnvironment no #Compression delayed ClientAliveInterval 120 ClientAliveCountMax 200000 #UseDNS yes #PidFile /var/run/sshd.pid #MaxStartups 10 #PermitTunnel no #ChrootDirectory none #VersionAddendum FreeBSD-20120901 # no default banner path #Banner none # override default of no subsystems Subsystem sftp /usr/libexec/sftp-server # Disable HPN tuning improvements. #HPNDisabled no # Buffer size for HPN to non-HPN connections. #HPNBufferSize 2048 # TCP receive socket buffer polling for HPN. Disable on non autotuning kernels. #TcpRcvBufPoll yes # Allow the use of the NONE cipher. #NoneEnabled no # Example of overriding settings on a per-user basis #Match User anoncvs # X11Forwarding no # AllowTcpForwarding no # ForceCommand cvs server # Ideas from the ssh folks? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 430 Valona Loop, Round Rock, TX 78681-3893