From owner-freebsd-fs@FreeBSD.ORG Mon Jul 19 06:41:18 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C7C5516A4CE for ; Mon, 19 Jul 2004 06:41:18 +0000 (GMT) Received: from inetmg01.sony.com.sg (inetmg01.sony.com.sg [202.42.154.67]) by mx1.FreeBSD.org (Postfix) with ESMTP id A5FB343D4C for ; Mon, 19 Jul 2004 06:41:15 +0000 (GMT) (envelope-from ravi.nanjundappa@ap.sony.com) Received: from avgw02b.sony.com.sg ([43.68.8.23]) by inetmg01.sony.com.sg (8.11.7+Sun/8.11.6) with SMTP id i6J6k1I17251 for ; Mon, 19 Jul 2004 14:46:06 +0800 (SGT) Received: from unknown(43.68.8.1) by avgw02b.sony.com.sg via csmap id 80d9f958_d94f_11d8_862e_0002b3cb4edc_31153; Mon, 19 Jul 2004 14:47:29 +0800 (SGT) Received: from sapsgssdibh01.ap.sony.com (bh01.ap.sony.com [43.68.15.23]) by seagw01.sony.com.sg (8.11.6+Sun/8.11.6) with ESMTP id i6J6XqE24671 for ; Mon, 19 Jul 2004 14:33:52 +0800 (SGT) Received: from sapinsardexc01.sard.in.sony.com.sg (SAPINSARDEXC01 [43.88.102.8]) by sapsgssdibh01.ap.sony.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2657.72) id PATQCFJZ; Mon, 19 Jul 2004 14:41:08 +0800 Received: from [43.88.102.67] (RAVIN [43.88.102.67]) by sapinsardexc01.sard.in.sony.com.sg with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2655.55) id 37T3DX6A; Mon, 19 Jul 2004 12:22:51 +0530 From: ravi To: freebsd-fs@freebsd.org Content-Type: text/plain Organization: SONY-SARD Message-Id: <1090219655.4730.32.camel@ravin> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.5 (1.4.5-7) Date: Mon, 19 Jul 2004 12:17:35 +0530 Content-Transfer-Encoding: 7bit Subject: Regarding linprocfs in FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jul 2004 06:41:18 -0000 Hi , This is my first mail to the list . Please have a look at the following satements in linprocfs_init() of linprocfs.c file . dir = pfs_create_dir(root, "net", NULL, NULL, 0); pfs_create_file(dir, "dev", &linprocfs_donetdev,NULL, NULL, PFS_WR); Even though we are giving the permission flag as PFS_WR the file is getting created with only read permission . Could any one you tell me the reason for this ? Regards, N Ravi From owner-freebsd-fs@FreeBSD.ORG Mon Jul 19 10:40:29 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 394AE16A5AF for ; Mon, 19 Jul 2004 10:40:29 +0000 (GMT) Received: from inetmg01.sony.com.sg (inetmg01.sony.com.sg [202.42.154.67]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9E8B643D2F for ; Mon, 19 Jul 2004 10:40:27 +0000 (GMT) (envelope-from ravi.nanjundappa@ap.sony.com) Received: from avgw02b.sony.com.sg ([43.68.8.23]) by inetmg01.sony.com.sg (8.11.7+Sun/8.11.6) with SMTP id i6JAjHI10953 for ; Mon, 19 Jul 2004 18:45:17 +0800 (SGT) Received: from unknown(43.68.8.1) by avgw02b.sony.com.sg via csmap id eb5dd512_d970_11d8_9765_0002b3cb4edc_28635; Mon, 19 Jul 2004 18:46:41 +0800 (SGT) Received: from sapsgssdibh01.ap.sony.com (bh01.ap.sony.com [43.68.15.23]) by seagw01.sony.com.sg (8.11.6+Sun/8.11.6) with ESMTP id i6JAX3E02870 for ; Mon, 19 Jul 2004 18:33:03 +0800 (SGT) Received: from sapinsardexc01.sard.in.sony.com.sg (SAPINSARDEXC01 [43.88.102.8]) by sapsgssdibh01.ap.sony.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2657.72) id PATQCFZB; Mon, 19 Jul 2004 18:40:19 +0800 Received: from [43.88.102.67] (RAVIN [43.88.102.67]) by sapinsardexc01.sard.in.sony.com.sg with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2655.55) id 37T3DYAY; Mon, 19 Jul 2004 16:22:02 +0530 From: ravi To: freebsd-fs@freebsd.org Content-Type: text/plain Organization: SONY-SARD Message-Id: <1090234005.4730.50.camel@ravin> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.5 (1.4.5-7) Date: Mon, 19 Jul 2004 16:16:45 +0530 Content-Transfer-Encoding: 7bit Subject: procfs in FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jul 2004 10:40:29 -0000 what is the use of PFS_PROCDEP in the following statement ? dir = pfs_create_dir(root, "pid", procfs_attr, NULL, PFS_PROCDEP); And there is no entry under /proc with the name "pid" . Why is it so ? Regards, N Ravi From owner-freebsd-fs@FreeBSD.ORG Mon Jul 19 11:58:42 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0F10F16A4CE; Mon, 19 Jul 2004 11:58:41 +0000 (GMT) Received: from inetmg01.sony.com.sg (inetmg01.sony.com.sg [202.42.154.67]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4F91743D2D; Mon, 19 Jul 2004 11:58:40 +0000 (GMT) (envelope-from ravi.nanjundappa@ap.sony.com) Received: from avgw02b.sony.com.sg ([43.68.8.23]) by inetmg01.sony.com.sg (8.11.7+Sun/8.11.6) with SMTP id i6JC2RI01047; Mon, 19 Jul 2004 20:02:28 +0800 (SGT) Received: from unknown(43.68.8.1) by avgw02b.sony.com.sg via csmap id b5bbb4aa_d97b_11d8_8391_0002b3cb4edc_11241; Mon, 19 Jul 2004 20:03:56 +0800 (SGT) Received: from sapsgssdibh01.ap.sony.com (bh01.ap.sony.com [43.68.15.23]) by seagw01.sony.com.sg (8.11.6+Sun/8.11.6) with ESMTP id i6JBoIE17504; Mon, 19 Jul 2004 19:50:18 +0800 (SGT) Received: from sapinsardexc01.sard.in.sony.com.sg (SAPINSARDEXC01 [43.88.102.8]) by sapsgssdibh01.ap.sony.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2657.72) id PATQCF5K; Mon, 19 Jul 2004 19:57:34 +0800 Received: from [43.88.102.67] (RAVIN [43.88.102.67]) by sapinsardexc01.sard.in.sony.com.sg with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2655.55) id 37T3DYBT; Mon, 19 Jul 2004 17:39:12 +0530 From: ravi To: Andrey Simonenko In-Reply-To: <20040706114506.GC466@pm514-9.comsys.ntu-kpi.kiev.ua> References: <20040706114506.GC466@pm514-9.comsys.ntu-kpi.kiev.ua> Content-Type: text/plain Organization: SONY-SARD Message-Id: <1090238636.4728.7.camel@ravin> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.5 (1.4.5-7) Date: Mon, 19 Jul 2004 17:33:56 +0530 Content-Transfer-Encoding: 7bit cc: Brian Fundakowski Feldman cc: Raja Guha cc: Dan Nelson cc: freebsd-fs@freebsd.org cc: Aniruddha Bohra Subject: Regarding FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jul 2004 11:58:42 -0000 Hi, 1) Can u explain me the usage of different arguments to pf_create_file () function ? 2) what is the use of PFS_PROCDEP in the following statement ? dir = pfs_create_dir(root, "pid", procfs_attr, NULL, PFS_PROCDEP); 3) there is no entry under /proc with the name "pid" . Why is it so ? 4) And even though for some of the proc entries the permission given is PFS_RDWR , the file is getting created with the read permission only . What is the reason for this ? Regards, N Ravi From owner-freebsd-fs@FreeBSD.ORG Mon Jul 19 12:16:04 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AE88E16A4CE for ; Mon, 19 Jul 2004 12:16:04 +0000 (GMT) Received: from inetmg01.sony.com.sg (inetmg01.sony.com.sg [202.42.154.67]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9958E43D53 for ; Mon, 19 Jul 2004 12:16:03 +0000 (GMT) (envelope-from ravi.nanjundappa@ap.sony.com) Received: from avgw02b.sony.com.sg ([43.68.8.23]) by inetmg01.sony.com.sg (8.11.7+Sun/8.11.6) with SMTP id i6JCKsI05459 for ; Mon, 19 Jul 2004 20:20:54 +0800 (SGT) Received: from unknown(43.68.8.1) by avgw02b.sony.com.sg via csmap id 4959b750_d97e_11d8_85c1_0002b3cb4edc_29304; Mon, 19 Jul 2004 20:22:22 +0800 (SGT) Received: from sapsgssdibh01.ap.sony.com (bh01.ap.sony.com [43.68.15.23]) by seagw01.sony.com.sg (8.11.6+Sun/8.11.6) with ESMTP id i6JC8iE20270 for ; Mon, 19 Jul 2004 20:08:44 +0800 (SGT) Received: from sapinsardexc01.sard.in.sony.com.sg (SAPINSARDEXC01 [43.88.102.8]) by sapsgssdibh01.ap.sony.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2657.72) id PATQCF7Q; Mon, 19 Jul 2004 20:16:01 +0800 Received: from [43.88.102.67] (RAVIN [43.88.102.67]) by sapinsardexc01.sard.in.sony.com.sg with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2655.55) id 37T3DYC1; Mon, 19 Jul 2004 17:57:41 +0530 From: ravi To: freebsd-fs@freebsd.org Content-Type: text/plain Organization: SONY-SARD Message-Id: <1090239745.4728.10.camel@ravin> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.5 (1.4.5-7) Date: Mon, 19 Jul 2004 17:52:25 +0530 Content-Transfer-Encoding: 7bit Subject: Regarding file system in FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jul 2004 12:16:04 -0000 Hi, How to create a file system using a kernel module ? Regards, N Ravi From owner-freebsd-fs@FreeBSD.ORG Mon Jul 19 16:43:38 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9BFBA16A4CF for ; Mon, 19 Jul 2004 16:43:38 +0000 (GMT) Received: from odin.ac.hmc.edu (Odin.AC.HMC.Edu [134.173.32.75]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4625E43D1D for ; Mon, 19 Jul 2004 16:43:38 +0000 (GMT) (envelope-from brdavis@odin.ac.hmc.edu) Received: from odin.ac.hmc.edu (IDENT:brdavis@localhost.localdomain [127.0.0.1]) by odin.ac.hmc.edu (8.12.10/8.12.10) with ESMTP id i6JGhZOF022026; Mon, 19 Jul 2004 09:43:35 -0700 Received: (from brdavis@localhost) by odin.ac.hmc.edu (8.12.10/8.12.3/Submit) id i6JGhZNw022025; Mon, 19 Jul 2004 09:43:35 -0700 Date: Mon, 19 Jul 2004 09:43:34 -0700 From: Brooks Davis To: David Kreil Message-ID: <20040719164334.GA14144@Odin.AC.HMC.Edu> References: <20040716045134.GA28777@Odin.AC.HMC.Edu> <200407170154.i6H1sTm14158@puffin.ebi.ac.uk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="YiEDa0DAkWCtVeE4" Content-Disposition: inline In-Reply-To: <200407170154.i6H1sTm14158@puffin.ebi.ac.uk> User-Agent: Mutt/1.5.4i cc: freebsd-fs@freebsd.org Subject: Re: "sanitizing" disks: wiping swap, non-allocated space, and file-tails X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jul 2004 16:43:38 -0000 --YiEDa0DAkWCtVeE4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jul 17, 2004 at 02:54:29AM +0100, David Kreil wrote: >=20 > Dear Brooks, >=20 > Many thanks for your fast and helpful reply! >=20 > > [Please don't cross post so much.] >=20 > I'm sorry. I had sent it to "ports" two weeks ago, assuming that external= =20 > programs might be required and then didn't quite know where to turn next.= I=20 > guess I should have tried individual groups at a time. In general, just 1-2 lists is best. Idealy, you can send to a list a wait a few days before trying somewhere else. > > The only way to clean a file system I can think of is, to dump the fs, > > scrub the disk, restore the fs. That will keep only real data and > > remove all the junk. >=20 > Ok, I can do that once to start from a clean slate but it would not be=20 > practical to do this regularly. Other then encrypting the disk, there isn't much else you can do at this point without kernel modification. > > You should be able to scrub swap at shutdown if > > you use swapoff to disable it, but you've got to be sure it's all unused > > first. You can't trust this though because you don't know the system > > will shut down cleanly. > >=20 > > There is a currently null operation in the disk code that acts when > > a block is no longer used. It should be possible to hook into that > > code to implement some sort of scrubber. It wouldn't be reliable for > > the same reason that a swap scrubber can't be, but it would be OK most > > of the time. >=20 > Yes, both approaches would already improve my situation considerably. >=20 > The handbook says that a shutdown "will attempt to run the script > /etc/rc.shutdown, and then proceed to send all processes the TERM > signal, and subsequently the KILL signal to any that do not terminate > timely". Would turning off swap and scrubbing in rc.shutdown be > sufficient, or do I need a way of doing this *after* all other > processes have been terminated (and if so, what would be a good > approach)? I believe you would need to hook after process shutdown unless you're 100% sure you won't be using swap since you can't disable swap if it's in use. > You mentioned the possibility of hooking a scrub routine into the fs > code that releases a block. My main worry with such an approach would > be that repeated random writes of an individual block would probably > never really be written to disk due to drive caching etc. Is there a > way around this problem? With ATA disks, you can disable write caching which should work. With SCSI disks, you could force each block to disk individualy, in the correct order. > > The easiest way to scrub a disk is: > >=20 > > dd if=3D3D/dev/random of=3D3D/dev/ bs=3D3D > > > > dd if=3D3D/dev/zero of=3D3D/dev/ bs=3D3D > >=20 > > This doesn't meet DoD requirements, but only for obsolete reasons. >=20 > Interesting - can you say what type of reasons these are? :-) The short form is that the DoD requirements require writing a pattern, a fit flipped version of the pattern, and the pattern again followed by some set of characters, usually zeros. Unfortunaly, this method was designed for an encoding technique that has not been used in >20 years so it doens't work. For the long form, read the standard: http://www.radium.ncsc.mil/tpep/library/rainbow/NCSC-TG-025.2.html and a nice modern analysis: http://www.cs.auckland.ac.nz/~pgut001/pubs/secure_del.html > > > (2) Related to this: > > > I'm also interested in people's personal experiences in using partiti= on > > > or file system encryption options. > > > For performance reasons I'd rather avoid having /tmp and swap and cer= tain=3D > > > work space on an encrypted disk, hence the need for (1). > >=20 > > If you swap, your performance will be suck enough that encrypting it > > won't hurt much, especially with modern CPUs. I wouldn't worry at all > > about that cost. /tmp is probably similar for most applications. >=20 > Really? A loss in disk performance of factor of four should be > noticable I had thought. For swap, I don't think a factor of four will be noticable since that just makes accessing swap 4000 times slower then accessing RAM instead of 1000 times slower (example numbers of course). For file systems, it's going to depend heavily on your application. Many of them just don't use that much disk. I believe phk said buildworld wasn't all that much worse then before (I don't recall an actual number). -- Brooks --=20 Any statement of the form "X is the one, true Y" is FALSE. PGP fingerprint 655D 519C 26A7 82E7 2529 9BF0 5D8E 8BE9 F238 1AD4 --YiEDa0DAkWCtVeE4 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQFA+/ozXY6L6fI4GtQRAqZzAJ467oPJFXvbr55NDM17AuzOF+928QCgsaOl k1IPK1DffMCkYAwxTv8XVUw= =sGx7 -----END PGP SIGNATURE----- --YiEDa0DAkWCtVeE4-- From owner-freebsd-fs@FreeBSD.ORG Tue Jul 20 11:16:40 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 98DEB16A4CE for ; Tue, 20 Jul 2004 11:16:40 +0000 (GMT) Received: from afields.ca (afields.ca [216.194.67.132]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5865743D3F for ; Tue, 20 Jul 2004 11:16:40 +0000 (GMT) (envelope-from afields@afields.ca) Received: from afields.ca (localhost.afields.ca [127.0.0.1]) by afields.ca (8.12.11/8.12.11) with ESMTP id i6KBGcMO034199; Tue, 20 Jul 2004 07:16:38 -0400 (EDT) (envelope-from afields@afields.ca) Received: (from afields@localhost) by afields.ca (8.12.11/8.12.11/Submit) id i6KBGbwM034198; Tue, 20 Jul 2004 07:16:37 -0400 (EDT) (envelope-from afields) Date: Tue, 20 Jul 2004 07:16:37 -0400 From: Allan Fields To: David Kreil Message-ID: <20040720111637.GJ12833@afields.ca> References: <20040716181339.GA18056@afields.ca> <200407170204.i6H24eU16729@puffin.ebi.ac.uk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="X1bOJ3K7DJ5YkBrT" Content-Disposition: inline In-Reply-To: <200407170204.i6H24eU16729@puffin.ebi.ac.uk> User-Agent: Mutt/1.4i cc: freebsd-fs@freebsd.org Subject: Re: "sanitizing" disks: wiping swap, non-allocated space, and file-tails X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jul 2004 11:16:40 -0000 --X1bOJ3K7DJ5YkBrT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jul 17, 2004 at 03:04:40AM +0100, David Kreil wrote: > > I still somewhat worry about the factor four in performance lost that is= =20 > mentioned in the gdbe paper. This is no problem for a set of sensitive pr= ivate=20 > files but at the system level it does cause me worry. As you seem to be s= o=20 > confident about this, however, (or have I misunderstood you?) I'll be hap= py to=20 > give it a go. One approach would be to gather statistics of peak performance requirements or do some stress-testing. phk has added support for statistics collection in GEOM: see gstat(8). You can simulate loads and benchmark with various tools found in ports. Outside of performance concerns: I wasn't suggesting you encrypt the device containing the root partition, as this is currently not supported since GBDE devices are mounted from userland gbde(8) during system startup from /etc/rc.d/gbde . You can create a separate home partition and leave /usr unencrypted if usage cases won't dictate storage of site-specific data such as password files, etc. You can setup /usr such that permissions are restrictive enough to ensure users can't write files to unprotected areas of the disk. What I meant to say was that if you can encrypt any sensitive areas and there is a workable trade-off between security and performance/usability, do so. Even in the case that 98% of your information is mundane, it's the 2% such as private keys, proprietary communication/documents, etc. that ultimately matters. Finally, it's possible to use gbde in a loopback configuration w/ md driver for finer granularity or for incremental addition of secure vnode-backed / temporary mounts. > Thanks for pointing this out. The Handbook describes a basic gdbe setup b= ut=20 > mentions that getting other volumes (like /home) onto a gdbe partition wa= s=20 > trickier. Can you tell me which volumes you have successfully put onto a = gdbe=20 > partition and what was required to get this working? I currently don't use the default script and have tested various configurations. On all systems I've had /home partitioned separate to /usr which is a simple case of changing your /etc/fstab to the corresponding bde devices and setting the noauto flag, pass# to 0 so as not to attempt filesystem check before attach: =2E. /dev/ar0g /usr ufs rw 2 2 /dev/ar0h.bde /home ufs rw,noauto 2 0 =2E. > I wonder, in particular, what issues I have to expect in wanting to keep= =20 > system relevant directories like /var on a gdbe partition. The gbde attach should occur early enough during multiuser startup to avoid such problems, I don't recall if the provided rc script would be sufficient, I'll test a configuration soon, or let me know if you have any luck. There are several approaches to securing /etc, but I can elaborate more after further testing. The short term approach is not storing private keys, etc. on an unencrypted root. Support for encrypted root is possible w/ some work, but there are a few issues to sort out first. > With many thanks again for your help > and best regards, >=20 > David. >=20 > ------------------------------------------------------------------------ > Dr David Philip Kreil ("`-''-/").___..--''"`-._ > Research Fellow `6_ 6 ) `-. ( ).`-.__.`) > University of Cambridge (_Y_.)' ._ ) `._ `. ``-..-' > ++44 1223 764107, fax 333992 _..`--'_..-_/ /--'_.' ,' > www.inference.phy.cam.ac.uk/dpk20 (il),-'' (li),' ((!.-' --=20 Allan Fields, AFRSL - http://afields.ca 2D4F 6806 D307 0889 6125 C31D F745 0D72 39B4 5541 --X1bOJ3K7DJ5YkBrT Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (FreeBSD) iD8DBQFA/P8S90UNcjm0VUERAjMWAKCvT601qPxbW/uEvhDFbrJcCIHh2gCfSWBJ P+gNc8w61zJmGmq5K12Kkgg= =lCa0 -----END PGP SIGNATURE----- --X1bOJ3K7DJ5YkBrT-- From owner-freebsd-fs@FreeBSD.ORG Tue Jul 20 20:03:21 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4429516A4CE for ; Tue, 20 Jul 2004 20:03:21 +0000 (GMT) Received: from maui.ebi.ac.uk (maui.ebi.ac.uk [193.62.196.100]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3338843D48 for ; Tue, 20 Jul 2004 20:03:20 +0000 (GMT) (envelope-from kreil@ebi.ac.uk) Received: from puffin.ebi.ac.uk (puffin.ebi.ac.uk [193.62.196.89]) by maui.ebi.ac.uk (8.11.7+Sun/8.11.7) with ESMTP id i6KK3FF28733; Tue, 20 Jul 2004 21:03:15 +0100 (BST) Received: from puffin.ebi.ac.uk (kreil@localhost) by puffin.ebi.ac.uk (8.11.6/8.11.6) with ESMTP id i6KK3Et20000; Tue, 20 Jul 2004 21:03:15 +0100 Message-Id: <200407202003.i6KK3Et20000@puffin.ebi.ac.uk> X-Mailer: exmh version 2.4 06/23/2000 with nmh-1.0.4 To: Brooks Davis In-Reply-To: Your message of "Mon, 19 Jul 2004 09:43:34 PDT." <20040719164334.GA14144@Odin.AC.HMC.Edu> X-Habeas-SWE-1: winter into spring X-Habeas-SWE-2: brightly anticipated X-Habeas-SWE-3: like Habeas SWE (tm) X-Habeas-SWE-4: Copyright 2002 Habeas (tm) X-Habeas-SWE-5: Sender Warranted Email (SWE) (tm). The sender of this X-Habeas-SWE-6: email in exchange for a license for this Habeas X-Habeas-SWE-7: warrant mark warrants that this is a Habeas Compliant X-Habeas-SWE-8: Message (HCM) and not spam. Please report use of this X-Habeas-SWE-9: mark in spam to . Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Date: Tue, 20 Jul 2004 21:03:14 +0100 From: David Kreil X-EBI-Information: This email is scanned using www.mailscanner.info. X-EBI: Found to be clean X-EBI-SpamCheck: not spam, SpamAssassin (score=-8, required 5, HABEAS_SWE -8.00) cc: freebsd-fs@freebsd.org cc: David Kreil Subject: Re: "sanitizing" disks: wiping swap, non-allocated space, and file-tails X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jul 2004 20:03:21 -0000 Dear Brooks, Thank you very much again for your comments. > > The handbook says that a shutdown "will attempt to run the script > > /etc/rc.shutdown, and then proceed to send all processes the TERM > > signal, and subsequently the KILL signal to any that do not terminate= > > timely". Would turning off swap and scrubbing in rc.shutdown be > > sufficient, or do I need a way of doing this *after* all other > > processes have been terminated (and if so, what would be a good > > approach)? > = > I believe you would need to hook after process shutdown unless you're > 100% sure you won't be using swap since you can't disable swap if it's > in use. Ok, so if I do it after the processes' shutdown, any remaining stuff in = virtual memory should easily fit into physical memory. swapoff would then= move = any swapped out pages back to physical memory before disconnecting swap a= nd I = can proceed. Would you know how I could hook into the shutdown procedure *after* the = processes have been terminated? > > You mentioned the possibility of hooking a scrub routine into the fs > > code that releases a block. My main worry with such an approach would= > > be that repeated random writes of an individual block would probably > > never really be written to disk due to drive caching etc. Is there a > > way around this problem? > = > With ATA disks, you can disable write caching which should work. With > SCSI disks, you could force each block to disk individualy, in the > correct order. That sounds like coding it would require more disk interface knowledge th= an I = have :-/ > > > The easiest way to scrub a disk is: > > > > > > dd if=3D/dev/random of=3D/dev/ bs=3D > > > I noticed that it will refuse to let me do that on swap, even if it is of= f. Of = course, I can edit the disklabel to read "unused", run dd, and restore th= e = swap disklabel to "swap" but is there another way? Also, I've just done some tests, and dd if=3D/dev/random of=3D/dev/ bs=3D1048576 only writes at 6.5MB/s on my system (/dev/zero gives 7.9MB/s). Is that = typical? My drives theoretically should do 30-40MB/s on read, and 20-30MB= /s on = write. If these results are "normal", however, that means, for a 10GB swap file = and, = say 6 wipes, I'd be waiting 3h on shutdown, while a BND-safe thorough 20 = wipes = would take half a day. Not really practical :-/ So unless you tell me that I should be able to achieve much faster write = speeds, I think I'll have to ditch the idea of regularly wiping swap (or = anything else for that matter). > > > This doesn't meet DoD requirements, but only for obsolete reasons. > > > > Interesting - can you say what type of reasons these are? :-) > = > The short form is that the DoD requirements require writing a pattern, = a > bit flipped version of the pattern, and the pattern again followed by > some set of characters, usually zeros. Unfortunaly, this method was > designed for an encoding technique that has not been used in >20 years > so it doens't work. Ah, ok! Thanks for the pointers :) > > > If you swap, your performance will be suck enough that encrypting i= t > > > won't hurt much, especially with modern CPUs. I wouldn't worry at = all > > > about that cost. /tmp is probably similar for most applications. > > > > Really? A loss in disk performance of factor of four should be > > noticable I had thought. > = > For swap, I don't think a factor of four will be noticable since that > just makes accessing swap 4000 times slower then accessing RAM instead > of 1000 times slower (example numbers of course). For file systems, > it's going to depend heavily on your application. Many of them just > don't use that much disk. I believe phk said buildworld wasn't all tha= t > much worse then before (I don't recall an actual number). That's impressive. Ok, I get the impression that in practice having every= thing that's not dedicated fast'n'dirty work-space gbde encrypted might a= fter all be the best solution. With many thanks again for your help, David. ------------------------------------------------------------------------ Dr David Philip Kreil ("`-''-/").___..--''"`-._ Research Fellow `6_ 6 ) `-. ( ).`-.__.`) University of Cambridge (_Y_.)' ._ ) `._ `. ``-..-' ++44 1223 764107, fax 333992 _..`--'_..-_/ /--'_.' ,' www.inference.phy.cam.ac.uk/dpk20 (il),-'' (li),' ((!.-' From owner-freebsd-fs@FreeBSD.ORG Tue Jul 20 20:44:09 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7584816A4CE for ; Tue, 20 Jul 2004 20:44:09 +0000 (GMT) Received: from maui.ebi.ac.uk (maui.ebi.ac.uk [193.62.196.100]) by mx1.FreeBSD.org (Postfix) with ESMTP id D1C3C43D2F for ; Tue, 20 Jul 2004 20:44:08 +0000 (GMT) (envelope-from kreil@ebi.ac.uk) Received: from puffin.ebi.ac.uk (puffin.ebi.ac.uk [193.62.196.89]) by maui.ebi.ac.uk (8.11.7+Sun/8.11.7) with ESMTP id i6KKi1F19767; Tue, 20 Jul 2004 21:44:01 +0100 (BST) Received: from puffin.ebi.ac.uk (kreil@localhost) by puffin.ebi.ac.uk (8.11.6/8.11.6) with ESMTP id i6KKi0725056; Tue, 20 Jul 2004 21:44:01 +0100 Message-Id: <200407202044.i6KKi0725056@puffin.ebi.ac.uk> X-Mailer: exmh version 2.4 06/23/2000 with nmh-1.0.4 To: Allan Fields In-Reply-To: Your message of "Tue, 20 Jul 2004 07:16:37 EDT." <20040720111637.GJ12833@afields.ca> X-Habeas-SWE-1: winter into spring X-Habeas-SWE-2: brightly anticipated X-Habeas-SWE-3: like Habeas SWE (tm) X-Habeas-SWE-4: Copyright 2002 Habeas (tm) X-Habeas-SWE-5: Sender Warranted Email (SWE) (tm). The sender of this X-Habeas-SWE-6: email in exchange for a license for this Habeas X-Habeas-SWE-7: warrant mark warrants that this is a Habeas Compliant X-Habeas-SWE-8: Message (HCM) and not spam. Please report use of this X-Habeas-SWE-9: mark in spam to . Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 20 Jul 2004 21:44:00 +0100 From: David Kreil X-EBI-Information: This email is scanned using www.mailscanner.info. X-EBI: Found to be clean X-EBI-SpamCheck: not spam, SpamAssassin (score=-8, required 5, HABEAS_SWE -8.00) cc: freebsd-fs@freebsd.org cc: David Kreil Subject: Re: "sanitizing" disks: wiping swap, non-allocated space, and file-tails X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jul 2004 20:44:09 -0000 Dear Allan, Thank you very much for your many comments! > > I still somewhat worry about the factor four in performance lost [...] > > One approach would be to gather statistics of peak performance > requirements or do some stress-testing. phk has added support for > statistics collection in GEOM: see gstat(8). You can simulate loads > and benchmark with various tools found in ports. Ta! > Outside of performance concerns: I wasn't suggesting you encrypt > the device containing the root partition, as this is currently not > supported since GBDE devices are mounted from userland gbde(8) during > system startup from /etc/rc.d/gbde . You can create a separate home > partition and leave /usr unencrypted if usage cases won't dictate > storage of site-specific data such as password files, etc. You can > setup /usr such that permissions are restrictive enough to ensure > users can't write files to unprotected areas of the disk. > > What I meant to say was that if you can encrypt any sensitive areas and > there is a workable trade-off between security and performance/usability, > do so. Even in the case that 98% of your information is mundane, it's > the 2% such as private keys, proprietary communication/documents, > etc. that ultimately matters. > > Finally, it's possible to use gbde in a loopback configuration w/ > md driver for finer granularity or for incremental addition of > secure vnode-backed / temporary mounts. I'm not sure I understand - are you suggesting to encrypt more selectively? But which areas are senstive, and which are not? I felt that as soon as I encrpyted /tmp and swap, I might performancewise just as well go for encrypting everything that contains dynamic information, for greatest simplicity. Then I don't have to think about whether there might be leakage, improving the security rating of one of the weakest links in the system - myself :o) > > Thanks for pointing this out. The Handbook describes a basic gdbe > > setup but mentions that getting other volumes (like /home) onto a > > gdbe partition was trickier. Can you tell me which volumes you have > > successfully put onto a gdbe partition and what was required to get > > this working? > > I currently don't use the default script and have tested various > configurations. On all systems I've had /home partitioned separate > to /usr which is a simple case of changing your /etc/fstab to the > corresponding bde devices and setting the noauto flag, pass# to 0 > so as not to attempt filesystem check before attach: > > > /dev/ar0g /usr ufs rw 2 2 > /dev/ar0h.bde /home ufs rw,noauto 2 0 > Ok! > > I wonder, in particular, what issues I have to expect in wanting to keep > > system relevant directories like /var on a gdbe partition. > > The gbde attach should occur early enough during multiuser startup to avoid > such problems, I don't recall if the provided rc script would be sufficient, > I'll test a configuration soon, That would be great. I currently only have the system on and off, as we are still fiddling with the hardware (a disk went down again today). > or let me know if you have any luck. Yes, will report back if I do! :o) > There are several approaches to securing /etc, but I can elaborate > more after further testing. The short term approach is not storing > private keys, etc. on an unencrypted root. Support for encrypted > root is possible w/ some work, but there are a few issues to sort > out first. > I think I don't need root to be encrypted per se, but /var, /etc, /usr/local, and /home would be good. As you say, the question is how to get these mounted early enough. Success stories gratefully received! With best regards, David. ------------------------------------------------------------------------ Dr David Philip Kreil ("`-''-/").___..--''"`-._ Research Fellow `6_ 6 ) `-. ( ).`-.__.`) University of Cambridge (_Y_.)' ._ ) `._ `. ``-..-' ++44 1223 764107, fax 333992 _..`--'_..-_/ /--'_.' ,' www.inference.phy.cam.ac.uk/dpk20 (il),-'' (li),' ((!.-' From owner-freebsd-fs@FreeBSD.ORG Tue Jul 20 22:00:36 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9A4AC16A4CE for ; Tue, 20 Jul 2004 22:00:36 +0000 (GMT) Received: from odin.ac.hmc.edu (Odin.AC.HMC.Edu [134.173.32.75]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6153C43D46 for ; Tue, 20 Jul 2004 22:00:36 +0000 (GMT) (envelope-from brdavis@odin.ac.hmc.edu) Received: from odin.ac.hmc.edu (IDENT:brdavis@localhost.localdomain [127.0.0.1]) by odin.ac.hmc.edu (8.12.10/8.12.10) with ESMTP id i6KM0XOF027350; Tue, 20 Jul 2004 15:00:33 -0700 Received: (from brdavis@localhost) by odin.ac.hmc.edu (8.12.10/8.12.3/Submit) id i6KM0Xnp027349; Tue, 20 Jul 2004 15:00:33 -0700 Date: Tue, 20 Jul 2004 15:00:33 -0700 From: Brooks Davis To: David Kreil Message-ID: <20040720220033.GA12560@Odin.AC.HMC.Edu> References: <20040719164334.GA14144@Odin.AC.HMC.Edu> <200407202003.i6KK3Et20000@puffin.ebi.ac.uk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="huq684BweRXVnRxX" Content-Disposition: inline In-Reply-To: <200407202003.i6KK3Et20000@puffin.ebi.ac.uk> User-Agent: Mutt/1.5.4i cc: freebsd-fs@freebsd.org Subject: Re: "sanitizing" disks: wiping swap, non-allocated space, and file-tails X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jul 2004 22:00:36 -0000 --huq684BweRXVnRxX Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jul 20, 2004 at 09:03:14PM +0100, David Kreil wrote: >=20 > Dear Brooks, >=20 > Thank you very much again for your comments. >=20 > > > The handbook says that a shutdown "will attempt to run the script > > > /etc/rc.shutdown, and then proceed to send all processes the TERM > > > signal, and subsequently the KILL signal to any that do not terminate > > > timely". Would turning off swap and scrubbing in rc.shutdown be > > > sufficient, or do I need a way of doing this *after* all other > > > processes have been terminated (and if so, what would be a good > > > approach)? > >=20 > > I believe you would need to hook after process shutdown unless you're > > 100% sure you won't be using swap since you can't disable swap if it's > > in use. >=20 > Ok, so if I do it after the processes' shutdown, any remaining stuff in= =20 > virtual memory should easily fit into physical memory. swapoff would then= move=20 > any swapped out pages back to physical memory before disconnecting swap a= nd I=20 > can proceed. >=20 > Would you know how I could hook into the shutdown procedure *after* the= =20 > processes have been terminated? Unfortunaly, I don't. I suspect you may have to hack init. > > > > The easiest way to scrub a disk is: > > > > > > > > dd if=3D/dev/random of=3D/dev/ bs=3D > > > > >=20 > I noticed that it will refuse to let me do that on swap, even if it is of= f. Of=20 > course, I can edit the disklabel to read "unused", run dd, and restore th= e=20 > swap disklabel to "swap" but is there another way? That's broken. Which OS are you using? > Also, I've just done some tests, and >=20 > dd if=3D/dev/random of=3D/dev/ bs=3D1048576 >=20 > only writes at 6.5MB/s on my system (/dev/zero gives 7.9MB/s). Is that=20 > typical? My drives theoretically should do 30-40MB/s on read, and 20-30MB= /s on=20 > write. >=20 > If these results are "normal", however, that means, for a 10GB swap file = and,=20 > say 6 wipes, I'd be waiting 3h on shutdown, while a BND-safe thorough 20 = wipes=20 > would take half a day. Not really practical :-/ > So unless you tell me that I should be able to achieve much faster write= =20 > speeds, I think I'll have to ditch the idea of regularly wiping swap (or= =20 > anything else for that matter). You might try a bigger size, but that may be as good as it gets. If you really want performance, you should use arc4random in a custom userland program. That's faster, but expect wiping a 40GB disk to take hours even in that case. I've got such an application, but I haven't had time to clean it up and submit it for release. I'll probably do it some day, but I can't recommend waiting for that. It's only about 800 lines of code including the man page and a fancy composable operations system to allow just about any DoD or non-DoD pattern or writes and verifies to be written on the command line. > > > > If you swap, your performance will be suck enough that encrypting it > > > > won't hurt much, especially with modern CPUs. I wouldn't worry at = all > > > > about that cost. /tmp is probably similar for most applications. > > > > > > Really? A loss in disk performance of factor of four should be > > > noticable I had thought. > >=20 > > For swap, I don't think a factor of four will be noticable since that > > just makes accessing swap 4000 times slower then accessing RAM instead > > of 1000 times slower (example numbers of course). For file systems, > > it's going to depend heavily on your application. Many of them just > > don't use that much disk. I believe phk said buildworld wasn't all that > > much worse then before (I don't recall an actual number). >=20 > That's impressive. Ok, I get the impression that in practice having > everything that's not dedicated fast'n'dirty work-space gbde encrypted > might after all be the best solution. Probably. Remember, modern CPUs are REALLY fast compared to just about everything else in the system including main memory. As long as you stay in cache, you can burn a remarkable number of cycles pretty much for free. For that matter, if you're operations are stream oriented (i.e. producing streams of random numbers) only memory bandwidth matters as long as the working set of the algorithm stays in cache. -- Brooks --=20 Any statement of the form "X is the one, true Y" is FALSE. PGP fingerprint 655D 519C 26A7 82E7 2529 9BF0 5D8E 8BE9 F238 1AD4 --huq684BweRXVnRxX Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQFA/ZYAXY6L6fI4GtQRAg65AKDiI5VWP/3gRvHtISrLBNpS6bylogCg5zR2 KD4SFBK1c72bOW4o7kkTgfU= =AMgA -----END PGP SIGNATURE----- --huq684BweRXVnRxX-- From owner-freebsd-fs@FreeBSD.ORG Tue Jul 20 22:24:53 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AAB9A16A4CE for ; Tue, 20 Jul 2004 22:24:53 +0000 (GMT) Received: from maui.ebi.ac.uk (maui.ebi.ac.uk [193.62.196.100]) by mx1.FreeBSD.org (Postfix) with ESMTP id BC75A43D5F for ; Tue, 20 Jul 2004 22:24:52 +0000 (GMT) (envelope-from kreil@ebi.ac.uk) Received: from puffin.ebi.ac.uk (puffin.ebi.ac.uk [193.62.196.89]) by maui.ebi.ac.uk (8.11.7+Sun/8.11.7) with ESMTP id i6KMOoF04403; Tue, 20 Jul 2004 23:24:50 +0100 (BST) Received: from puffin.ebi.ac.uk (kreil@localhost) by puffin.ebi.ac.uk (8.11.6/8.11.6) with ESMTP id i6KMOoN05374; Tue, 20 Jul 2004 23:24:50 +0100 Message-Id: <200407202224.i6KMOoN05374@puffin.ebi.ac.uk> X-Mailer: exmh version 2.4 06/23/2000 with nmh-1.0.4 To: Brooks Davis In-Reply-To: Your message of "Tue, 20 Jul 2004 15:00:33 PDT." <20040720220033.GA12560@Odin.AC.HMC.Edu> X-Habeas-SWE-1: winter into spring X-Habeas-SWE-2: brightly anticipated X-Habeas-SWE-3: like Habeas SWE (tm) X-Habeas-SWE-4: Copyright 2002 Habeas (tm) X-Habeas-SWE-5: Sender Warranted Email (SWE) (tm). The sender of this X-Habeas-SWE-6: email in exchange for a license for this Habeas X-Habeas-SWE-7: warrant mark warrants that this is a Habeas Compliant X-Habeas-SWE-8: Message (HCM) and not spam. Please report use of this X-Habeas-SWE-9: mark in spam to . Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 20 Jul 2004 23:24:50 +0100 From: David Kreil X-EBI-Information: This email is scanned using www.mailscanner.info. X-EBI: Found to be clean X-EBI-SpamCheck: not spam, SpamAssassin (score=-8, required 5, HABEAS_SWE -8.00) cc: freebsd-fs@freebsd.org cc: David Kreil Subject: Re: "sanitizing" disks: wiping swap, non-allocated space, and file-tails X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jul 2004 22:24:53 -0000 Dear Brooks, Thank you very much for your fast reply. > > Would you know how I could hook into the shutdown procedure *after* the > > processes have been terminated? > > Unfortunaly, I don't. I suspect you may have to hack init. Hmpf, ok, will have a look. > > > > > The easiest way to scrub a disk is: > > > > > > > > > > dd if=/dev/random of=/dev/ bs= > > > > > > > > > I noticed that it will refuse to let me do that on swap, even if it is > > off. Of course, I can edit the disklabel to read "unused", run dd, and > > restore the swap disklabel to "swap" but is there another way? > > That's broken. Which OS are you using? FreeBSD 5.2.1-RELEASE of Feb 23, GENERIC kernel > dd dd if=/dev/random of=/dev/twed0s3a bs=1048576 gives dd: /dev/twed0s3a: Operation not permitted when disklabel shows fstype "swap"; when I change the disklabel to fstype "unused", it works. In either case, the partition is *not* being used as swap, as I can verify with swapon. So if this is not a feature, it is a bug. Who would like to know about such an observation - this list, or should this be sent elsewhere? (Feel like this question should be on newbie :) > You might try a bigger size, but that may be as good as it gets. Hmpf, ok. > If you > really want performance, you should use arc4random in a custom userland > program. That's faster, but expect wiping a 40GB disk to take hours > even in that case. Assuming that /dev/zero is as fast as it gets, the limiting step really seems to be the writing - or would you expect /dev/zero to be slow for some odd reasons? When you use dd on your system, to you get similar write rates? > I've got such an application, but I haven't had time > to clean it up and submit it for release. I'll probably do it some day, > but I can't recommend waiting for that. It's only about 800 lines of > code including the man page and a fancy composable operations system to > allow just about any DoD or non-DoD pattern or writes and verifies to be > written on the command line. I'd be happy to beta-test-drive it if you like! :o) > > > For swap, I don't think a factor of four will be noticable since that > > > just makes accessing swap 4000 times slower then accessing RAM instead > > > of 1000 times slower (example numbers of course). For file systems, > > > it's going to depend heavily on your application. Many of them just > > > don't use that much disk. I believe phk said buildworld wasn't all that > > > much worse then before (I don't recall an actual number). > > > > That's impressive. Ok, I get the impression that in practice having > > everything that's not dedicated fast'n'dirty work-space gbde encrypted > > might after all be the best solution. > > Probably. Remember, modern CPUs are REALLY fast compared to just about > everything else in the system including main memory. As long as you > stay in cache, you can burn a remarkable number of cycles pretty much > for free. For that matter, if you're operations are stream oriented > (i.e. producing streams of random numbers) only memory bandwidth matters > as long as the working set of the algorithm stays in cache. According to the gbde paper, the encryption can up to double the number of disk-accesses compared to a non-encrypted volume. I hence think you are right that the encryption time per se should be negligible, it will be the extra seek/read/write operations that give the slow-down. Now, if buildworld can do most of its stuff in memory and is CPU rather than I/O bound, that would make little difference. If your application needs to swap and/or is I/O intensive, I guess one should feel it strongly. If I can get my system to have most stuff on gbde partitions with the help of this list, including system ones like /var, I think it's worth a go. I can then have a smaller "fast" partition for temporary not-sensitive work that I could wipe with your utility periodically (or /dev/random at worst). With many thanks and best regards, David. ------------------------------------------------------------------------ Dr David Philip Kreil ("`-''-/").___..--''"`-._ Research Fellow `6_ 6 ) `-. ( ).`-.__.`) University of Cambridge (_Y_.)' ._ ) `._ `. ``-..-' ++44 1223 764107, fax 333992 _..`--'_..-_/ /--'_.' ,' www.inference.phy.cam.ac.uk/dpk20 (il),-'' (li),' ((!.-' From owner-freebsd-fs@FreeBSD.ORG Wed Jul 21 02:38:46 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E441216A4CF for ; Wed, 21 Jul 2004 02:38:46 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9F7D343D54 for ; Wed, 21 Jul 2004 02:38:46 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i6L2bx7U015557; Tue, 20 Jul 2004 22:37:59 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i6L2bqiN015554; Tue, 20 Jul 2004 22:37:59 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Tue, 20 Jul 2004 22:37:52 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: ravi In-Reply-To: <1090234005.4730.50.camel@ravin> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-fs@freebsd.org Subject: Re: procfs in FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jul 2004 02:38:47 -0000 On Mon, 19 Jul 2004, ravi wrote: > what is the use of PFS_PROCDEP in the following statement ? > > dir = pfs_create_dir(root, "pid", > procfs_attr, NULL, PFS_PROCDEP); > > And there is no entry under /proc with the name "pid" . > > Why is it so ? I'm not all that familiar with pseudofs, but I believe that the PFS_PROCDEP flag is a "magic" flag that tells procfs to create process-specific directories under the directory in question, and the "pid" is basically place-holder text. I.e., the presence of the directory node created above tells procfs to spit out lots of per-process subdirectories instead. The details here are a bit hazy to me, but perhaps this is enough to point you in the right direction? Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Principal Research Scientist, McAfee Research From owner-freebsd-fs@FreeBSD.ORG Wed Jul 21 02:50:06 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D83C316A4CE; Wed, 21 Jul 2004 02:50:06 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 76E2443D2F; Wed, 21 Jul 2004 02:50:06 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i6L2n9gt015794; Tue, 20 Jul 2004 22:49:09 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i6L2n7vc015790; Tue, 20 Jul 2004 22:49:07 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Tue, 20 Jul 2004 22:49:07 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: ravi In-Reply-To: <1090238636.4728.7.camel@ravin> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Brian Fundakowski Feldman cc: Raja Guha cc: Dan Nelson cc: freebsd-fs@freebsd.org cc: Aniruddha Bohra cc: Andrey Simonenko Subject: Re: Regarding FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jul 2004 02:50:07 -0000 On Mon, 19 Jul 2004, ravi wrote: > Hi, > 1) Can u explain me the usage of different arguments to pf_create_file > () function ? Again, details a bit hazy to me here, but roughly speaking: 'parent' is the parent directory where the object will appear 'name' is the name it will have 'fill' is the function pointer provided by the pseudofs-derived file system to implement the node. I.e., procfs_doproccmdline. 'attr' appears to be a function pointer allowing the pseudo file system to modify the set of attributes returned for a node; this is used by procfs to modify the set of permissions returns for certain objects (looks like the debugging entries in per-process directories). 'vis' is a visibility test function: should the node be visible to a particular caller (i.e., "you can't see that process for security reasons"). 'flags' is a set of node flags, such as PFS_RD, PFS_WR, etc, as found in pseudofs.h. I'd suggest working from examples in src/sys/fs/procfs/procfs.c; pseudofs is largely an abstraction layer to let the bulk of the file system logic be shared between procfs and linprocfs, since they're similar (but different). > 2) what is the use of PFS_PROCDEP in the following statement ? > > dir = pfs_create_dir(root, "pid", > procfs_attr, NULL, PFS_PROCDEP); As I mentioned in a previous e-mail, it's creating a template per-process directory that will be expanded into many perceived directories on demand, one per process. If you look at procfs_init(), you'll see that it creates the template, then places a bunch of nodes in the template that will be valid for each process. > 3) there is no entry under /proc with the name "pid" . Why is it so ? See previous e-mail -- the template has its name replaced with each possible pid, and 'pid' is a place-holder. > 4) And even though for some of the proc entries the permission given is > PFS_RDWR , the file is getting created with the read permission only . > > What is the reason for this ? pfs_getattr() appears to always return read-only entries, except where pn_attr() is implemented on the node. The actual protections appear to be calculated in several places -- for example, pfs_visible() uses various visibility tests, including per-process tests, etc. The permission bits are probably not a perfect guide to access rights. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Principal Research Scientist, McAfee Research From owner-freebsd-fs@FreeBSD.ORG Wed Jul 21 04:36:40 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5FC7A16A4CE; Wed, 21 Jul 2004 04:36:40 +0000 (GMT) Received: from inetmg01.sony.com.sg (inetmg01.sony.com.sg [202.42.154.67]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0A4E743D41; Wed, 21 Jul 2004 04:36:38 +0000 (GMT) (envelope-from ravi.nanjundappa@ap.sony.com) Received: from avgw02b.sony.com.sg ([43.68.8.23]) by inetmg01.sony.com.sg (8.11.7+Sun/8.11.6) with SMTP id i6L4fKI06356; Wed, 21 Jul 2004 12:41:21 +0800 (SGT) Received: from unknown(43.68.8.1) by avgw02b.sony.com.sg via csmap id 6fab8908_dad0_11d8_824c_0002b3cb4edc_16480; Wed, 21 Jul 2004 12:42:56 +0800 (SGT) Received: from sapsgssdibh01.ap.sony.com (bh01.ap.sony.com [43.68.15.23]) by seagw01.sony.com.sg (8.11.6+Sun/8.11.6) with ESMTP id i6L4TBE26533; Wed, 21 Jul 2004 12:29:11 +0800 (SGT) Received: from sapinsardexc01.sard.in.sony.com.sg (SAPINSARDEXC01 [43.88.102.8]) by sapsgssdibh01.ap.sony.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2657.72) id PATQCH6R; Wed, 21 Jul 2004 12:36:27 +0800 Received: from [43.88.102.67] (RAVIN [43.88.102.67]) by sapinsardexc01.sard.in.sony.com.sg with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2655.55) id 37T3DY4Q; Wed, 21 Jul 2004 10:18:05 +0530 From: ravi To: Robert Watson In-Reply-To: References: Content-Type: text/plain Organization: SONY-SARD Message-Id: <1090384970.4727.6.camel@ravin> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.5 (1.4.5-7) Date: Wed, 21 Jul 2004 10:12:50 +0530 Content-Transfer-Encoding: 7bit cc: Brian Fundakowski Feldman cc: Raja Guha cc: Dan Nelson cc: freebsd-fs@freebsd.org cc: Aniruddha Bohra cc: Andrey Simonenko Subject: Re: Regarding FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jul 2004 04:36:40 -0000 hi, I'm able to create a proc file under /proc directory using pfs_create_file() with the read and write permission in a kernel module . But I don't know how to read and write to /proc/file . Can u please tell me how to do it ? Regards, N Ravi On Wed, 2004-07-21 at 08:19, Robert Watson wrote: > On Mon, 19 Jul 2004, ravi wrote: > > > Hi, > > 1) Can u explain me the usage of different arguments to > pf_create_file > > () function ? > > Again, details a bit hazy to me here, but roughly speaking: > > 'parent' is the parent directory where the object will appear > > 'name' is the name it will have > > 'fill' is the function pointer provided by the pseudofs-derived file > system to implement the node. I.e., procfs_doproccmdline. > > 'attr' appears to be a function pointer allowing the pseudo file > system > to modify the set of attributes returned for a node; this is used by > procfs to modify the set of permissions returns for certain objects > (looks like the debugging entries in per-process directories). > > 'vis' is a visibility test function: should the node be visible to a > particular caller (i.e., "you can't see that process for security > reasons"). > > 'flags' is a set of node flags, such as PFS_RD, PFS_WR, etc, as found > in > pseudofs.h. > > I'd suggest working from examples in src/sys/fs/procfs/procfs.c; > pseudofs > is largely an abstraction layer to let the bulk of the file system logic > be shared between procfs and linprocfs, since they're similar (but > different). > > > 2) what is the use of PFS_PROCDEP in the following statement ? > > > > dir = pfs_create_dir(root, "pid", > > procfs_attr, NULL, PFS_PROCDEP); > > As I mentioned in a previous e-mail, it's creating a template > per-process > directory that will be expanded into many perceived directories on > demand, > one per process. If you look at procfs_init(), you'll see that it > creates > the template, then places a bunch of nodes in the template that will be > valid for each process. > > > 3) there is no entry under /proc with the name "pid" . Why is it so ? > > > See previous e-mail -- the template has its name replaced with each > possible pid, and 'pid' is a place-holder. > > > 4) And even though for some of the proc entries the permission given > is > > PFS_RDWR , the file is getting created with the read permission only . > > > > > What is the reason for this ? > > pfs_getattr() appears to always return read-only entries, except where > pn_attr() is implemented on the node. The actual protections appear to > be > calculated in several places -- for example, pfs_visible() uses various > visibility tests, including per-process tests, etc. The permission bits > are probably not a perfect guide to access rights. > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > robert@fledge.watson.org Principal Research Scientist, McAfee > Research > From owner-freebsd-fs@FreeBSD.ORG Wed Jul 21 06:06:34 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7AB0416A4CE; Wed, 21 Jul 2004 06:06:34 +0000 (GMT) Received: from inetmg01.sony.com.sg (inetmg01.sony.com.sg [202.42.154.67]) by mx1.FreeBSD.org (Postfix) with ESMTP id 61D7D43D5A; Wed, 21 Jul 2004 06:06:31 +0000 (GMT) (envelope-from ravi.nanjundappa@ap.sony.com) Received: from avgw02b.sony.com.sg ([43.68.8.23]) by inetmg01.sony.com.sg (8.11.7+Sun/8.11.6) with SMTP id i6L6BFI07313; Wed, 21 Jul 2004 14:11:16 +0800 (SGT) Received: from unknown(43.68.8.1) by avgw02b.sony.com.sg via csmap id ff7e2afc_dadc_11d8_80eb_0002b3cb4edc_30211; Wed, 21 Jul 2004 14:12:52 +0800 (SGT) Received: from sapsgssdibh01.ap.sony.com (bh01.ap.sony.com [43.68.15.23]) by seagw01.sony.com.sg (8.11.6+Sun/8.11.6) with ESMTP id i6L5x4E21667; Wed, 21 Jul 2004 13:59:04 +0800 (SGT) Received: from sapinsardexc01.sard.in.sony.com.sg (SAPINSARDEXC01 [43.88.102.8]) by sapsgssdibh01.ap.sony.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2657.72) id PATQC2CM; Wed, 21 Jul 2004 14:06:22 +0800 Received: from [43.88.102.67] (RAVIN [43.88.102.67]) by sapinsardexc01.sard.in.sony.com.sg with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2655.55) id 37T3DYXQ; Wed, 21 Jul 2004 11:47:57 +0530 From: ravi To: Robert Watson In-Reply-To: References: Content-Type: text/plain Organization: SONY-SARD Message-Id: <1090390362.4727.14.camel@ravin> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.5 (1.4.5-7) Date: Wed, 21 Jul 2004 11:42:42 +0530 Content-Transfer-Encoding: 7bit cc: Brian Fundakowski Feldman cc: Raja Guha cc: Dan Nelson cc: freebsd-fs@freebsd.org cc: Aniruddha Bohra cc: Andrey Simonenko Subject: Regarding FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jul 2004 06:06:34 -0000 Hi, can we write or read to/from a proc file created under /proc using pfs_write() and pfs_read() defined in pseudofs_vnops.c ? Regards, N Ravi From owner-freebsd-fs@FreeBSD.ORG Fri Jul 23 08:56:35 2004 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D260916A4CE for ; Fri, 23 Jul 2004 08:56:35 +0000 (GMT) Received: from inetmg01.sony.com.sg (inetmg01.sony.com.sg [202.42.154.67]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6980243D2D for ; Fri, 23 Jul 2004 08:56:33 +0000 (GMT) (envelope-from ravi.nanjundappa@ap.sony.com) Received: from avgw02b.sony.com.sg (avgw02b [43.68.8.23]) by inetmg01.sony.com.sg (8.11.7+Sun/8.11.6) with SMTP id i6N91OI29457 for ; Fri, 23 Jul 2004 17:01:24 +0800 (SGT) Received: from unknown(43.68.8.1) by avgw02b.sony.com.sg via csmap id 1b4d0c16_dc87_11d8_9dd0_0002b3cb4edc_28951; Fri, 23 Jul 2004 17:03:04 +0800 (SGT) Received: from sapsgssdibh01.ap.sony.com (bh01.ap.sony.com [43.68.15.23]) by seagw01.sony.com.sg (8.11.6+Sun/8.11.6) with ESMTP id i6N8n6E05247 for ; Fri, 23 Jul 2004 16:49:06 +0800 (SGT) Received: from sapinsardexc01.sard.in.sony.com.sg (SAPINSARDEXC01 [43.88.102.8]) by sapsgssdibh01.ap.sony.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2657.72) id P3XJ7SH3; Fri, 23 Jul 2004 16:56:25 +0800 Received: from [43.88.102.67] (RAVIN [43.88.102.67]) by sapinsardexc01.sard.in.sony.com.sg with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2655.55) id 37T3DZRQ; Fri, 23 Jul 2004 14:38:06 +0530 From: ravi To: freebsd-fs@freebsd.org In-Reply-To: <20040719120055.7DEC716A4D0@hub.freebsd.org> References: <20040719120055.7DEC716A4D0@hub.freebsd.org> Content-Type: text/plain Organization: SONY-SARD Message-Id: <1090573381.2960.3.camel@ravin> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.5 (1.4.5-7) Date: Fri, 23 Jul 2004 14:33:02 +0530 Content-Transfer-Encoding: 7bit Subject: Regarding the file size of /proc file entries X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jul 2004 08:56:35 -0000 Hi, Please have a look at the following output . $ ls -l /proc/self/ total 0 -rw-rw-rw- 1 ravi ravi 0 Jul 23 19:56 cmdline lr--r--r-- 1 ravi ravi 0 Jul 23 19:56 exe -> /bin/ls -rw------- 1 ravi ravi 0 Jul 23 19:56 mem -rw-rw-rw- 1 ravi ravi 0 Jul 23 19:56 mem123 -rw-rw-rw- 1 ravi ravi 0 Jul 23 19:56 stat -rw-rw-rw- 1 ravi ravi 0 Jul 23 19:56 status $ And also please have a look at the 5th column . For all the entries the size shown is zero . Which structure of the kernel will store the size of /proc file entries ? Please do the needful . Regards, N Ravi