Date: Thu, 15 Mar 2001 22:41:47 -0800 From: "Jonathan Graehl" <jonathan@graehl.org> To: "freebsd-Arch" <freebsd-arch@FreeBSD.ORG> Subject: end to end file transfer Message-ID: <NCBBLOALCKKINBNNEDDLMEJLDMAA.jonathan@graehl.org> In-Reply-To: <200103160607.XAA04414@usr05.primenet.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> > I don't see your "4 bits in error in your lossy-compressed media > > file is not so bad" argument as an answer to my question, nor are > > all file transfers sending lossy-compressed media where you don't > > care about errors. > > Well, then, if we're concerned over bit errors not capable of being > caught by TCP/IP, rather than fixing TCP/IP, we better add MD5 to > SMTP, UUCP, POP3, IMAP4, FSP, GOPHER, ARCHIE, FTAM, NFS, CODA, HTTP, > AFS, and every other internet protocol, so that TCP/IP can be > lossy for the rest of time and never have to change. > > After all, it's better to "fix" every program that uses a lossy > transport than it is to fix a lossy transport, isn't it? > > Sheesh... The argument for modularity and code reuse is specious. Code can be reused in other ways than stuffing it into the transport layer (shared libraries?). Read the excellent paper Gordon referenced: http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf (don't worry, it's an pleasant trip), you'll understand why people might be interested in true application-level guarantees of succesful transfer. No matter how you fortify the network (using IPSEC AH with a 160 bit hash?), you still face these possible failures: 1) the server sends the wrong bits (disk corruption, cosmic ray twiddles a bit in memory before it is sent out) 2) the client writes the file to disk with errors along the way (same possibilities as the server) Perhaps I am interested in knowing if I managed to succesfully store on my disk the same bits that were originally written to the remote disk by the file's creator. If it were possible to have the filesystem maintain the whole-file checksum of the data when it was originally written, then we could truly guarantee this. No matter how error-unlikely the transport, it is still possible that what I write to disk was corrupted after transport. Reading back the contents of the file and doing a whole-file checksum, and comparing it to the server's is the only way to be sure I have the same file. Common to this thread has been the complaint that "it only happens .1% of the time". If the consequences of "it" happening are really bad, then there should be a mechanism for at least detecting "it", if "it" cannot be prevented. For instance, you say: "if it only happens a few time that a distfile is changed but has the same size and filename, then it is not worth having the server implement SITE MD5". What you are missing is, it may be bad to miss an update to a file. Therefore, if I wish to check every day for new versions, I have to download the same file every day *FOR ALL FILES I AM CHECKING*, not just those which have been changed and happen to have the same size (if I knew which had been changed, I wouldn't need to ask, now would I?) I really do not understand the resistance. Perhaps it would be more constructive if we ask: What is the best protocol for reliable (signed) serving of public files with minimum complexity and server-overhead? What is the best protocol for confidential, reliable transfer of private files? Vanilla FTP is the answer to neither of these. Feel free to invent a new protocol or extend existing ones ;) For the first, http with a signed hash of the content sounds acceptable to me, although I'd rather have a less clunky header format (content-negotiation based on MIME type is a good thing, I suppose, but a more minimalist binary-file-transfer protocol using a signle TCP connection would be nice). For the second, I currently use scp (or rsync over ssh). -- Jonathan Graehl email: jonathan@graehl.org web: http://jonathan.graehl.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?NCBBLOALCKKINBNNEDDLMEJLDMAA.jonathan>