I forgot to add current/stable to the list
TL;DR: there seems(!) to be something(!) unclean about an ssh path
between an 8.3-STABLE(r247820) and
10.0-CURRENT(r247826) box such that a zfs send stream is corrupted in
transit.
below is the thread from -fs about it, with sshd configs from both
sides.
If I copy the stream it works, but piping through ssh does NOT.
-------- Original Message --------
Subject: Re: zfs send/recv invalid data
Date: 2013-03-06 04:46
From: Larry Rosenman <l...@lerctr.org>
To: Steven Hartland <kill...@multiplay.co.uk>
Cc: Ronald Klop <ronald-freeb...@klop.yi.org>, <freebsd...@freebsd.org>
On 2013-03-06 02:38, Steven Hartland wrote:
----- Original Message ----- From: "Larry Rosenman" <l...@lerctr.org>
I received an "invalid data" in a zfs send (from 8.3) / zfs
recv (to 10.0) of a -R -I stream.
What data do I need to gather to figure out what side and
what's wrong?
I've already started zpool scrubs on both sides.
I can insert a tee to grab the stream on either/both sides if
that would help.
Is the problem repeatable or is it just a network glitch?
Ronald.
Repeatable.......
Here is the exact error message:
receiving incremental stream of vault/home/ctr@2013-03-05-test3
into zroot/backups/TBH/home/ctr@2013-03-05-test3
cannot receive incremental stream: invalid backup stream
this is the script I'm running:
#!/bin/sh
DATE=`date "+%Y-%m-%d-BUG-REPRO"`
DATE2=`date -v "-1d" "+%Y-%m-%d"`
# snap the source
ssh r...@tbh.lerctr.org zfs snapshot -r vault@${DATE}
# zfs copy the source to here.
ssh r...@tbh.lerctr.org "zfs send -R -D -I vault@${DATE2}
vault@${DATE} | \
tee /tmp/backup.stream.send.${DATE} | \
ssh home.lerctr.org \"tee /tmp/backup.stream.receive.${DATE}
| zfs recv -u -v -d zroot/backups/TBH\""
# make sure we NEVER allow the backup stuff to automount.
/sbin/zfs list -H -t filesystem -r zroot/backups/TBH| \
awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh
both streams are in http://www.lerctr.org/~ler/ZFS_RECV
Your send and receive sides differ, which indicates your ssh
shell my not be clean.
Looking at the receive side its got what looks like a mail
message appended.
I suspect if you manually copy the receive copy to the 10 machine
and
the receive it will work fine.
we're copying mail files........
and it still fails....
I've put more example send/recv files in that directory.
we're copying home dirs, which include lots of mail.
(this one is my wife's)
Ideas?
I *CAN* give access to both sides via ssh.....
The copy of the data stream on both sides should be identical
though and its not, which leads me to believe something is
corrupting the data on the way. Try the following:-
From source:-
zfs send -R -D -I vault@${DATE2} vault@${DATE} > test.stream
scp test.stream home.lerctr.org:~/
From target:
zfs recv -u -v -d zroot/backups/TBH < test.stream
If this works then there is something unclean about your ssh
shell.
Regards
Steve
send side:
# zfs send -R -D -I vault@2013-03-05 vault@2013-03-06 >/tmp/send.stream
# openssl md5 /tmp/send.stream
MD5(/tmp/send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33
# scp /tmp/send.stream home:/tmp/send.stream
send.stream 100% 1180MB 2.5MB/s
07:44
# uname -a
FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54
r247820: Mon Mar 4 18:08:11 CST 2013
r...@thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER amd64
#
Receive side:
# uname -a
FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 r247826:
Mon Mar 4 19:59:08 CST 2013
r...@borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64
# openssl md5 /tmp/send.stream
MD5(/tmp/send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33
# zfs recv -F -u -v -d zroot/backups/TBH < /tmp/send.stream
<success>
#
So, you are correct that something(tm) is unclean about the ssh path.
adding -current and -stable for diagnosing ssh issue.
sshd config on the 8.3-STABLE box:
# cat /etc/ssh/sshd_config
# $OpenBSD: sshd_config,v 1.87 2012/07/10 02:19:15 djm Exp $
# $FreeBSD: stable/8/crypto/openssh/sshd_config 247521 2013-03-01
02:06:04Z des $
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.
# Note that some of FreeBSD's defaults differ from OpenBSD's, and
# FreeBSD has a few additional options.
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
# Disable legacy (protocol version 1) support in the server for new
# installations. In future the default will change to require explicit
# activation of protocol 1
Protocol 2
# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key
# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 1024
# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
#LogLevel INFO
# Authentication:
#LoginGraceTime 2m
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10
#RSAAuthentication yes
#PubkeyAuthentication yes
# The default is to check both .ssh/authorized_keys and
.ssh/authorized_keys2
# but this is overridden so installations will only check
.ssh/authorized_keys
#AuthorizedKeysFile .ssh/authorized_keys
#AuthorizedPrincipalsFile none
# For this to work you will also need host keys in
/etc/ssh/ssh_known_hosts
#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes
# Change to yes to enable built-in password authentication.
#PasswordAuthentication no
#PermitEmptyPasswords no
# Change to no to disable PAM authentication
#ChallengeResponseAuthentication yes
# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
# Set this to 'no' to disable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
#UsePAM yes
#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation sandbox
#PermitUserEnvironment no
#Compression delayed
ClientAliveInterval 120
ClientAliveCountMax 200000
#UseDNS yes
#PidFile /var/run/sshd.pid
#MaxStartups 10
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum FreeBSD-20120901
# no default banner path
#Banner none
# override default of no subsystems
Subsystem sftp /usr/libexec/sftp-server
# Disable HPN tuning improvements.
#HPNDisabled no
# Buffer size for HPN to non-HPN connections.
#HPNBufferSize 2048
# TCP receive socket buffer polling for HPN. Disable on non autotuning
kernels.
#TcpRcvBufPoll yes
# Allow the use of the NONE cipher.
#NoneEnabled no
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs server
#
sshd config on the 10.0-CURRENT:
# cat /etc/ssh/sshd_config
# $OpenBSD: sshd_config,v 1.87 2012/07/10 02:19:15 djm Exp $
# $FreeBSD: head/crypto/openssh/sshd_config 240075 2012-09-03
16:51:41Z des $
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.
# Note that some of FreeBSD's defaults differ from OpenBSD's, and
# FreeBSD has a few additional options.
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
# The default requires explicit activation of protocol 1
#Protocol 2
# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 1024
# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
#LogLevel INFO
# Authentication:
#LoginGraceTime 2m
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10
#RSAAuthentication yes
#PubkeyAuthentication yes
# The default is to check both .ssh/authorized_keys and
.ssh/authorized_keys2
# but this is overridden so installations will only check
.ssh/authorized_keys
AuthorizedKeysFile .ssh/authorized_keys
#AuthorizedPrincipalsFile none
# For this to work you will also need host keys in
/etc/ssh/ssh_known_hosts
#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes
# Change to yes to enable built-in password authentication.
#PasswordAuthentication no
#PermitEmptyPasswords no
# Change to no to disable PAM authentication
#ChallengeResponseAuthentication yes
# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
# Set this to 'no' to disable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
#UsePAM yes
#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation sandbox
#PermitUserEnvironment no
#Compression delayed
ClientAliveInterval 120
ClientAliveCountMax 200000
#UseDNS yes
#PidFile /var/run/sshd.pid
#MaxStartups 10
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum FreeBSD-20120901
# no default banner path
#Banner none
# override default of no subsystems
Subsystem sftp /usr/libexec/sftp-server
# Disable HPN tuning improvements.
#HPNDisabled no
# Buffer size for HPN to non-HPN connections.
#HPNBufferSize 2048
# TCP receive socket buffer polling for HPN. Disable on non autotuning
kernels.
#TcpRcvBufPoll yes
# Allow the use of the NONE cipher.
#NoneEnabled no
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs server
#
Ideas from the ssh folks?
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: l...@lerctr.org
US Mail: 430 Valona Loop, Round Rock, TX 78681-3893
_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
