Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 06 Mar 2013 04:49:48 -0600
From:      Larry Rosenman <ler@lerctr.org>
To:        <freebsd-stable@freebsd.org>, <freebsd-current@freebsd.org>
Subject:   Fwd: Re: zfs send/recv invalid data
Message-ID:  <46d966dd574cd8097d4972213c73e9be@webmail.lerctr.org>

next in thread | raw e-mail | index | archive | help
I forgot to add current/stable to the list

TL;DR: there seems(!) to be something(!) unclean about an ssh path 
between an 8.3-STABLE(r247820) and
10.0-CURRENT(r247826) box such that a zfs send stream is corrupted in 
transit.

below is the thread from -fs about it, with sshd configs from both 
sides.

If I copy the stream it works, but piping through ssh does NOT.



-------- Original Message --------
Subject: Re: zfs send/recv invalid data
Date: 2013-03-06 04:46
 From: Larry Rosenman <ler@lerctr.org>
To: Steven Hartland <killing@multiplay.co.uk>
Cc: Ronald Klop <ronald-freebsd8@klop.yi.org>, <freebsd-fs@freebsd.org>

On 2013-03-06 02:38, Steven Hartland wrote:
> ----- Original Message ----- From: "Larry Rosenman" <ler@lerctr.org>
>>>>>>>> I received an "invalid data" in a zfs send (from 8.3) / zfs 
>>>>>>>> recv (to 10.0) of a -R -I stream.
>>>>>>>> What data do I need to gather to figure out what side and 
>>>>>>>> what's wrong?
>>>>>>>> I've already started zpool scrubs on both sides.
>>>>>>>> I can insert a tee to grab the stream on either/both sides if 
>>>>>>>> that would  help.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> Is the problem repeatable or is it just a network glitch?
>>>>>>> Ronald.
>>>>>> Repeatable.......
>>>>> Here is the exact error message:
>>>>> receiving incremental stream of vault/home/ctr@2013-03-05-test3 
>>>>> into zroot/backups/TBH/home/ctr@2013-03-05-test3
>>>>> cannot receive incremental stream: invalid backup stream
>>>>> this is the script I'm running:
>>>>> #!/bin/sh
>>>>> DATE=`date "+%Y-%m-%d-BUG-REPRO"`
>>>>> DATE2=`date -v "-1d" "+%Y-%m-%d"`
>>>>> # snap the source
>>>>> ssh root@tbh.lerctr.org zfs snapshot -r vault@${DATE}
>>>>> # zfs copy the source to here.
>>>>> ssh root@tbh.lerctr.org "zfs send  -R -D  -I vault@${DATE2} 
>>>>> vault@${DATE} | \
>>>>>      tee /tmp/backup.stream.send.${DATE} | \
>>>>>      ssh home.lerctr.org \"tee /tmp/backup.stream.receive.${DATE} 
>>>>> | zfs recv  -u -v -d zroot/backups/TBH\""
>>>>> # make sure we NEVER allow the backup stuff to automount.
>>>>> /sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \
>>>>>     awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh
>>>>> both streams are in http://www.lerctr.org/~ler/ZFS_RECV
>>>> Your send and receive sides differ, which indicates your ssh
>>>> shell my not be clean.
>>>> Looking at the receive side its got what looks like a mail
>>>> message appended.
>>>> I suspect if you manually copy the receive copy to the 10 machine 
>>>> and
>>>> the receive it will work fine.
>>> we're copying mail files........
>>> and it still fails....
>>> 
>> I've put more example send/recv files in that directory.
>> we're copying home dirs, which include lots of mail.
>> (this one is my wife's)
>> Ideas?
>> I *CAN* give access to both sides via ssh.....
> The copy of the data stream on both sides should be identical
> though and its not, which leads me to believe something is
> corrupting the data on the way. Try the following:-
> 
>> From source:-
> zfs send -R -D -I vault@${DATE2} vault@${DATE} > test.stream
> scp test.stream home.lerctr.org:~/
>> From target:
> zfs recv  -u -v -d zroot/backups/TBH < test.stream
> If this works then there is something unclean about your ssh
> shell.
> Regards
>    Steve
> send side:
# zfs send -R -D -I vault@2013-03-05 vault@2013-03-06 >/tmp/send.stream
# openssl md5 /tmp/send.stream
MD5(/tmp/send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33
# scp /tmp/send.stream home:/tmp/send.stream
send.stream                                   100% 1180MB   2.5MB/s   
07:44
# uname -a
FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 
r247820: Mon Mar  4 18:08:11 CST 2013     
root@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER  amd64
#

Receive side:
# uname -a
FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 r247826: 
Mon Mar  4 19:59:08 CST 2013     
root@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE  amd64
# openssl md5 /tmp/send.stream
MD5(/tmp/send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33
# zfs recv -F -u -v -d zroot/backups/TBH < /tmp/send.stream
<success>
#

So, you are correct that something(tm) is unclean about the ssh path.

adding -current and -stable for diagnosing ssh issue.

sshd config on the 8.3-STABLE box:
# cat /etc/ssh/sshd_config
#       $OpenBSD: sshd_config,v 1.87 2012/07/10 02:19:15 djm Exp $
#       $FreeBSD: stable/8/crypto/openssh/sshd_config 247521 2013-03-01 
02:06:04Z des $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

# Note that some of FreeBSD's defaults differ from OpenBSD's, and
# FreeBSD has a few additional options.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

# Disable legacy (protocol version 1) support in the server for new
# installations. In future the default will change to require explicit
# activation of protocol 1
Protocol 2

# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key

# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 1024

# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#RSAAuthentication yes
#PubkeyAuthentication yes

# The default is to check both .ssh/authorized_keys and 
.ssh/authorized_keys2
# but this is overridden so installations will only check 
.ssh/authorized_keys
#AuthorizedKeysFile     .ssh/authorized_keys

#AuthorizedPrincipalsFile none

# For this to work you will also need host keys in 
/etc/ssh/ssh_known_hosts
#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# Change to yes to enable built-in password authentication.
#PasswordAuthentication no
#PermitEmptyPasswords no

# Change to no to disable PAM authentication
#ChallengeResponseAuthentication yes

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

# Set this to 'no' to disable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
#UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation sandbox
#PermitUserEnvironment no
#Compression delayed
ClientAliveInterval 120
ClientAliveCountMax 200000
#UseDNS yes
#PidFile /var/run/sshd.pid
#MaxStartups 10
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum FreeBSD-20120901

# no default banner path
#Banner none

# override default of no subsystems
Subsystem       sftp    /usr/libexec/sftp-server

# Disable HPN tuning improvements.
#HPNDisabled no

# Buffer size for HPN to non-HPN connections.
#HPNBufferSize 2048

# TCP receive socket buffer polling for HPN.  Disable on non autotuning 
kernels.
#TcpRcvBufPoll yes

# Allow the use of the NONE cipher.
#NoneEnabled no

# Example of overriding settings on a per-user basis
#Match User anoncvs
#       X11Forwarding no
#       AllowTcpForwarding no
#       ForceCommand cvs server
#


sshd config on the 10.0-CURRENT:
# cat /etc/ssh/sshd_config
#       $OpenBSD: sshd_config,v 1.87 2012/07/10 02:19:15 djm Exp $
#       $FreeBSD: head/crypto/openssh/sshd_config 240075 2012-09-03 
16:51:41Z des $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

# Note that some of FreeBSD's defaults differ from OpenBSD's, and
# FreeBSD has a few additional options.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

# The default requires explicit activation of protocol 1
#Protocol 2

# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key

# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 1024

# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#RSAAuthentication yes
#PubkeyAuthentication yes

# The default is to check both .ssh/authorized_keys and 
.ssh/authorized_keys2
# but this is overridden so installations will only check 
.ssh/authorized_keys
AuthorizedKeysFile      .ssh/authorized_keys

#AuthorizedPrincipalsFile none

# For this to work you will also need host keys in 
/etc/ssh/ssh_known_hosts
#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# Change to yes to enable built-in password authentication.
#PasswordAuthentication no
#PermitEmptyPasswords no

# Change to no to disable PAM authentication
#ChallengeResponseAuthentication yes

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

# Set this to 'no' to disable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
#UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation sandbox
#PermitUserEnvironment no
#Compression delayed
ClientAliveInterval 120
ClientAliveCountMax 200000
#UseDNS yes
#PidFile /var/run/sshd.pid
#MaxStartups 10
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum FreeBSD-20120901

# no default banner path
#Banner none

# override default of no subsystems
Subsystem       sftp    /usr/libexec/sftp-server

# Disable HPN tuning improvements.
#HPNDisabled no

# Buffer size for HPN to non-HPN connections.
#HPNBufferSize 2048

# TCP receive socket buffer polling for HPN.  Disable on non autotuning 
kernels.
#TcpRcvBufPoll yes

# Allow the use of the NONE cipher.
#NoneEnabled no

# Example of overriding settings on a per-user basis
#Match User anoncvs
#       X11Forwarding no
#       AllowTcpForwarding no
#       ForceCommand cvs server
#

Ideas from the ssh folks?

-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c)     E-Mail: ler@lerctr.org
US Mail: 430 Valona Loop, Round Rock, TX 78681-3893



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?46d966dd574cd8097d4972213c73e9be>