From owner-freebsd-audit Fri Mar 15 11:12:53 2002 Delivered-To: freebsd-audit@freebsd.org Received: from alcatraz.iptelecom.net.ua (alcatraz.iptelecom.net.ua [212.9.224.15]) by hub.freebsd.org (Postfix) with ESMTP id B053237B400; Fri, 15 Mar 2002 11:11:50 -0800 (PST) Received: from ipcard.iptcom.net (ipcard.iptcom.net [212.9.224.5]) by alcatraz.iptelecom.net.ua (8.9.3/8.9.3) with ESMTP id VAA79175; Fri, 15 Mar 2002 21:11:48 +0200 (EET) (envelope-from sobomax@FreeBSD.org) Received: from vega.vega.com (h16.229.dialup.iptcom.net [212.9.229.16]) by ipcard.iptcom.net (8.9.3/8.9.3) with ESMTP id VAA98777; Fri, 15 Mar 2002 21:11:44 +0200 (EET) (envelope-from sobomax@FreeBSD.org) Received: from FreeBSD.org (big_brother.vega.com [192.168.1.1]) by vega.vega.com (8.11.6/8.11.3) with ESMTP id g2FJBDB99955; Fri, 15 Mar 2002 21:11:13 +0200 (EET) (envelope-from sobomax@FreeBSD.org) Message-ID: <3C92478F.2B7C11DB@FreeBSD.org> Date: Fri, 15 Mar 2002 21:12:15 +0200 From: Maxim Sobolev Organization: Vega International Capital X-Mailer: Mozilla 4.79 [en] (Windows NT 5.0; U) X-Accept-Language: en,uk,ru MIME-Version: 1.0 To: Michael Smith Cc: hackers@FreeBSD.org, audit@FreeBSD.org, re@FreeBSD.org Subject: Re: Extending loader(8) for loading kerels/modules split across several disks References: <200203070940.g279eTa04750@mass.dis.org> Content-Type: multipart/mixed; boundary="------------FC04DF02325744190C9B23D5" Sender: owner-freebsd-audit@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG This is a multi-part message in MIME format. --------------FC04DF02325744190C9B23D5 Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 7bit Michael Smith wrote: > > > > > Please review attached patch, which adds long overdue feature to our > > > > loader(8), allowing it to load sequence of files as a single object. > > > > > > I don't like this. I would much rather see support for 'split' files > > > implemented as a stacking filesystem layer like the gzip support, with > > > the simple recognition of 'foo.gz.aa' as the first part of a split > > > version of 'foo.gz', which in turn is recognised as a compressed version > > > of 'foo'. > > > > I am curious how in this case the layer is going to know how many > > parts the file contains? > > The simple way to do it is to keep asking for more parts until there are > no more. > > You can take the NetBSD approach of wrapping the file in a multipart tar > archive. > > Or a more elegant method involves the use of a control file. > > eg. the splitfs code, when asked to open "foo" looks for "foo.split" > which is a text file containing a list of filenames and media names, eg. > > foo.aa "Kernel floppy 1" > foo.ab "Kernel floppy 2" > foo.ac "Kernel and modules floppy" > > For each file segment, the process is: > > - try to open the file > - prompt "please insert the disk labelled '" > - try to open the file > - return error > > At any rate, my key point is that the splitting should be invisible, and > *definitely* not pushed up into the loader. Ok, attached is the path, which does exactly what described. Please review and if there are no objections I would like to commit it shortly, so that our re@ team would be able to consider it for the forthcoming 5.0-DP1 release. Thanks! -Maxim --------------FC04DF02325744190C9B23D5 Content-Type: text/plain; charset=koi8-r; name="loader-multifiles.diff2" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="loader-multifiles.diff2" Index: src/lib/libstand/Makefile =================================================================== RCS file: /home/ncvs/src/lib/libstand/Makefile,v retrieving revision 1.27 diff -d -u -r1.27 Makefile --- src/lib/libstand/Makefile 27 Feb 2002 17:15:37 -0000 1.27 +++ src/lib/libstand/Makefile 15 Mar 2002 08:40:31 -0000 @@ -153,6 +153,7 @@ SRCS+= ufs.c nfs.c cd9660.c tftp.c zipfs.c bzipfs.c SRCS+= netif.c nfs.c SRCS+= dosfs.c ext2fs.c +SRCS+= splitfs.c beforeinstall: ${INSTALL} -C -o ${BINOWN} -g ${BINGRP} -m 444 ${.CURDIR}/stand.h \ Index: src/lib/libstand/bzipfs.c =================================================================== RCS file: /home/ncvs/src/lib/libstand/bzipfs.c,v retrieving revision 1.3 diff -d -u -r1.3 bzipfs.c --- src/lib/libstand/bzipfs.c 1 Feb 2002 16:33:40 -0000 1.3 +++ src/lib/libstand/bzipfs.c 15 Mar 2002 08:40:31 -0000 @@ -150,7 +150,7 @@ /* If the name already ends in .gz or .bz2, ignore it */ if ((cp = strrchr(fname, '.')) && (!strcmp(cp, ".gz") - || !strcmp(cp, ".bz2"))) + || !strcmp(cp, ".bz2") || !strcmp(cp, ".split"))) return(ENOENT); /* Construct new name */ Index: src/lib/libstand/splitfs.c =================================================================== RCS file: src/lib/libstand/splitfs.c diff -N src/lib/libstand/splitfs.c --- /dev/null 1 Jan 1970 00:00:00 -0000 +++ src/lib/libstand/splitfs.c 15 Mar 2002 08:40:31 -0000 @@ -0,0 +1,287 @@ +/* + * Copyright (c) 2002 Maxim Sobolev + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include "stand.h" + +#define NTRIES (3) +#define CONF_BUF (512) +#define SEEK_BUF (512) + +struct split_file +{ + char **filesv; /* Filenames */ + char **descsv; /* Descriptions */ + int filesc; /* Number of parts */ + int curfile; /* Current file number */ + int curfd; /* Current file descriptor */ + off_t tot_pos; /* Offset from the beginning of the sequence */ + off_t file_pos; /* Offset from the beginning of the slice */ +}; + +static int splitfs_open(const char *path, struct open_file *f); +static int splitfs_close(struct open_file *f); +static int splitfs_read(struct open_file *f, void *buf, size_t size, size_t *resid); +static off_t splitfs_seek(struct open_file *f, off_t offset, int where); +static int splitfs_stat(struct open_file *f, struct stat *sb); + +struct fs_ops splitfs_fsops = { + "split", + splitfs_open, + splitfs_close, + splitfs_read, + null_write, + splitfs_seek, + splitfs_stat, + null_readdir +}; + +static void +split_file_destroy(struct split_file *sf) +{ + int i; + + if (sf->filesc > 0) { + for (i = 0; i < sf->filesc; i++) { + free(sf->filesv[i]); + free(sf->descsv[i]); + } + free(sf->filesv); + free(sf->descsv); + } + free(sf); +} + +static int +splitfs_open(const char *fname, struct open_file *f) +{ + char *buf, *confname, *cp; + int conffd; + struct split_file *sf; + struct stat sb; + + printf("%s\n", fname); + /* Have to be in "just read it" mode */ + if (f->f_flags != F_READ) + return(EPERM); + + /* If the name already ends in `.split', ignore it */ + if ((cp = strrchr(fname, '.')) && (!strcmp(cp, ".split"))) + return(ENOENT); + + /* Construct new name */ + confname = malloc(strlen(fname) + 7); + sprintf(confname, "%s.split", fname); + + /* Try to open the configuration file */ + conffd = open(confname, O_RDONLY); + free(confname); + if (conffd == -1) + return(ENOENT); + + if (fstat(conffd, &sb) < 0) { + printf("splitfs_open: stat failed\n"); + close(conffd); + return(ENOENT); + } + if (!S_ISREG(sb.st_mode)) { + printf("splitfs_open: not a file\n"); + close(conffd); + return(EISDIR); /* best guess */ + } + + /* Allocate a split_file structure, populate it from the config file */ + sf = malloc(sizeof(struct split_file)); + bzero(sf, sizeof(struct split_file)); + buf = malloc(CONF_BUF); + while (fgetstr(buf, CONF_BUF, conffd) > 0) { + cp = buf; + while ((*cp != '\0') && (isspace(*cp) == 0)) + cp++; + if (*cp != '\0') { + *cp = '\0'; + cp++; + } + while ((*cp != '\0') && (isspace(*cp) != 0)) + cp++; + if (*cp == '\0') + cp = buf; + sf->filesc++; + sf->filesv = realloc(sf->filesv, sizeof(*(sf->filesv)) * sf->filesc); + sf->descsv = realloc(sf->descsv, sizeof(*(sf->descsv)) * sf->filesc); + sf->filesv[sf->filesc - 1] = strdup(buf); + sf->descsv[sf->filesc - 1] = strdup(cp); + } + free(buf); + close(conffd); + + if ((sf->filesc == 0) || ((sf->curfd = open(sf->filesv[0], O_RDONLY)) == -1)) { + split_file_destroy(sf); + return(ENOENT); + } + + /* Looks OK, we'll take it */ + f->f_fsdata = sf; + return (0); +} + +static int +splitfs_close(struct open_file *f) +{ + int fd; + struct split_file *sf; + + sf = (struct split_file *)f->f_fsdata; + fd = sf->curfd; + split_file_destroy(sf); + return(close(fd)); +} + +static int +splitfs_read(struct open_file *f, void *buf, size_t size, size_t *resid) +{ + int i, nread, totread; + struct split_file *sf; + + sf = (struct split_file *)f->f_fsdata; + totread = 0; + do { + nread = read(sf->curfd, buf, size - totread); + + /* Error? */ + if (nread == -1) + return (errno); + + sf->tot_pos += nread; + sf->file_pos += nread; + totread += nread; + buf += nread; + + if (totread < size) { /* EOF */ + if (sf->curfile == (sf->filesc - 1)) /* Last slice */ + break; + + /* Close previous slice */ + if (close(sf->curfd) != 0) + return (errno); + + sf->curfile++; + for (i = 0;; i++) { + sf->curfd = open(sf->filesv[sf->curfile], O_RDONLY); + if (sf->curfd >= 0) + break; + if ((sf->curfd == -1) && (errno != ENOENT)) + return (errno); + if (i == NTRIES) + return (EIO); + printf("\nInsert disk labelled %s and press any key...", sf->descsv[sf->curfile]); + getchar();putchar('\n'); + } + sf->file_pos = 0; + } + } while (totread < size); + + if (resid != NULL) + *resid = size - totread; + + return (0); +} + +static off_t +splitfs_seek(struct open_file *f, off_t offset, int where) +{ + int nread; + size_t resid; + off_t new_pos, seek_by; + struct split_file *sf; + + sf = (struct split_file *)f->f_fsdata; + + seek_by = offset; + switch (where) { + case SEEK_SET: + seek_by -= sf->tot_pos; + break; + case SEEK_CUR: + break; + case SEEK_END: + panic("splitfs_seek: SEEK_END not supported"); + break; + } + + if (seek_by > 0) { + /* + * Seek forward - implemented using splitfs_read(), because otherwise we'll be + * unable to detect that we have crossed slice boundary and hence + * unable to do a long seek crossing that boundary. + */ + void *tmp; + + tmp = malloc(SEEK_BUF); + if (tmp == NULL) + return (-1); + + nread = 0; + for (; seek_by > 0; seek_by -= nread) { + resid = 0; + errno = splitfs_read(f, tmp, min(seek_by, SEEK_BUF), &resid); + nread = min(seek_by, SEEK_BUF) - resid; + if ((errno != 0) || (nread == 0)) + /* Error or EOF */ + break; + } + free(tmp); + if (errno != 0) + return (-1); + } + + if (seek_by != 0) { + /* Seek backward or seek past the boundary of the last slice */ + if (sf->file_pos + seek_by < 0) + panic("splitfs_seek: can't seek past the beginning of the slice"); + new_pos = lseek(sf->curfd, seek_by, SEEK_CUR); + if (new_pos < 0) + return (-1); + sf->tot_pos += new_pos - sf->file_pos; + sf->file_pos = new_pos; + } + + return (sf->tot_pos); +} + +static int +splitfs_stat(struct open_file *f, struct stat *sb) +{ + int result; + struct split_file *sf = (struct split_file *)f->f_fsdata; + + /* stat as normal, but indicate that size is unknown */ + if ((result = fstat(sf->curfd, sb)) == 0) + sb->st_size = -1; + return (result); +} Index: src/lib/libstand/stand.h =================================================================== RCS file: /home/ncvs/src/lib/libstand/stand.h,v retrieving revision 1.29 diff -d -u -r1.29 stand.h --- src/lib/libstand/stand.h 9 Mar 2002 21:02:11 -0000 1.29 +++ src/lib/libstand/stand.h 15 Mar 2002 08:40:31 -0000 @@ -125,6 +125,7 @@ extern struct fs_ops bzipfs_fsops; extern struct fs_ops dosfs_fsops; extern struct fs_ops ext2fs_fsops; +extern struct fs_ops splitfs_fsops; /* where values for lseek(2) */ #define SEEK_SET 0 /* set file offset to offset */ Index: src/lib/libstand/zipfs.c =================================================================== RCS file: /home/ncvs/src/lib/libstand/zipfs.c,v retrieving revision 1.8 diff -d -u -r1.8 zipfs.c --- src/lib/libstand/zipfs.c 30 Sep 2001 22:28:01 -0000 1.8 +++ src/lib/libstand/zipfs.c 15 Mar 2002 08:40:31 -0000 @@ -175,7 +175,7 @@ /* If the name already ends in .gz or .bz2, ignore it */ if ((cp = strrchr(fname, '.')) && (!strcmp(cp, ".gz") - || !strcmp(cp, ".bz2"))) + || !strcmp(cp, ".bz2") || !strcmp(cp, ".split"))) return(ENOENT); /* Construct new name */ Index: src/sys/boot/i386/loader/conf.c =================================================================== RCS file: /home/ncvs/src/sys/boot/i386/loader/conf.c,v retrieving revision 1.18 diff -d -u -r1.18 conf.c --- src/sys/boot/i386/loader/conf.c 5 Nov 2001 18:59:13 -0000 1.18 +++ src/sys/boot/i386/loader/conf.c 15 Mar 2002 08:40:31 -0000 @@ -60,6 +60,7 @@ &ext2fs_fsops, &dosfs_fsops, &cd9660_fsops, + &splitfs_fsops, #ifdef LOADER_GZIP_SUPPORT &zipfs_fsops, #endif --------------FC04DF02325744190C9B23D5-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-audit" in the body of the message