From owner-freebsd-arch@FreeBSD.ORG Mon Jan 3 10:47:41 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3FD9A16A4CE for ; Mon, 3 Jan 2005 10:47:41 +0000 (GMT) Received: from critter.freebsd.dk (f170.freebsd.dk [212.242.86.170]) by mx1.FreeBSD.org (Postfix) with ESMTP id 811F943D60 for ; Mon, 3 Jan 2005 10:47:40 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.1/8.13.1) with ESMTP id j03AldKG018963 for ; Mon, 3 Jan 2005 11:47:39 +0100 (CET) (envelope-from phk@critter.freebsd.dk) To: arch@freebsd.org From: Poul-Henning Kamp Date: Mon, 03 Jan 2005 11:47:39 +0100 Message-ID: <18962.1104749259@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Subject: making nmdm(4) emulate actual speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jan 2005 10:47:41 -0000 I participated in an "Editor Celebrity Death Match" recently and being the senior combatant my weapon of choice was ed(1). To properly show off ed(1)'s main weakness I wanted to run my slides in ed(1) on a 300 bps line. Rather than use two USB-serial dongles and a usb-hub, I hacked nmdm(4) up to actually respect the baud-rate set with stty. Would this be considered generally useful ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Mon Jan 3 14:57:19 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B20D516A4CE for ; Mon, 3 Jan 2005 14:57:19 +0000 (GMT) Received: from xorpc.icir.org (xorpc.icir.org [192.150.187.68]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9103343D2D for ; Mon, 3 Jan 2005 14:57:19 +0000 (GMT) (envelope-from rizzo@icir.org) Received: from xorpc.icir.org (localhost [127.0.0.1]) by xorpc.icir.org (8.12.11/8.12.8) with ESMTP id j03EvINl067505; Mon, 3 Jan 2005 06:57:18 -0800 (PST) (envelope-from rizzo@xorpc.icir.org) Received: (from rizzo@localhost) by xorpc.icir.org (8.12.11/8.12.3/Submit) id j03EvGO2067504; Mon, 3 Jan 2005 06:57:16 -0800 (PST) (envelope-from rizzo) Date: Mon, 3 Jan 2005 06:57:16 -0800 From: Luigi Rizzo To: Poul-Henning Kamp Message-ID: <20050103065715.A67451@xorpc.icir.org> References: <18962.1104749259@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <18962.1104749259@critter.freebsd.dk>; from phk@phk.freebsd.dk on Mon, Jan 03, 2005 at 11:47:39AM +0100 cc: arch@freebsd.org Subject: Re: making nmdm(4) emulate actual speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jan 2005 14:57:19 -0000 On Mon, Jan 03, 2005 at 11:47:39AM +0100, Poul-Henning Kamp wrote: > > I participated in an "Editor Celebrity Death Match" recently and > being the senior combatant my weapon of choice was ed(1). To > properly show off ed(1)'s main weakness I wanted to run my slides > in ed(1) on a 300 bps line. > > Rather than use two USB-serial dongles and a usb-hub, I hacked nmdm(4) > up to actually respect the baud-rate set with stty. > > Would this be considered generally useful ? being nmdm(4) an emulation tool, i'd say definitely yes, probably even more useful if you provide a knob to enable/disable the speed emulation -- i see a point in actually emulating the wire speed, but also one in not doing so when the application is not speed-sensitive and you just want it to run quickly. cheers luigi > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk@FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-arch@FreeBSD.ORG Mon Jan 3 18:51:37 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 47CB816A4CE for ; Mon, 3 Jan 2005 18:51:37 +0000 (GMT) Received: from mail.vicor-nb.com (bigwoop.vicor-nb.com [208.206.78.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id EF77E43D2F for ; Mon, 3 Jan 2005 18:51:33 +0000 (GMT) (envelope-from julian@elischer.org) Received: from elischer.org (julian.vicor-nb.com [208.206.78.97]) by mail.vicor-nb.com (Postfix) with ESMTP id F0ADA7A403; Mon, 3 Jan 2005 10:51:30 -0800 (PST) Message-ID: <41D99432.7040700@elischer.org> Date: Mon, 03 Jan 2005 10:51:30 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.3.1) Gecko/20030516 X-Accept-Language: en, hu MIME-Version: 1.0 To: Poul-Henning Kamp References: <18962.1104749259@critter.freebsd.dk> In-Reply-To: <18962.1104749259@critter.freebsd.dk> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: arch@freebsd.org Subject: Re: making nmdm(4) emulate actual speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jan 2005 18:51:37 -0000 Poul-Henning Kamp wrote: >I participated in an "Editor Celebrity Death Match" recently and >being the senior combatant my weapon of choice was ed(1). To >properly show off ed(1)'s main weakness I wanted to run my slides >in ed(1) on a 300 bps line. > >Rather than use two USB-serial dongles and a usb-hub, I hacked nmdm(4) >up to actually respect the baud-rate set with stty. > >Would this be considered generally useful ? > > maybe as an option... I want faster tansfer speed when I use it.. but emulated speed could be fun.. and even useful as a way of pacing data in some cases.. From owner-freebsd-arch@FreeBSD.ORG Mon Jan 3 21:02:25 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3557A16A4CF for ; Mon, 3 Jan 2005 21:02:25 +0000 (GMT) Received: from harmony.village.org (rover.village.org [168.103.84.182]) by mx1.FreeBSD.org (Postfix) with ESMTP id A976B43D2D for ; Mon, 3 Jan 2005 21:02:24 +0000 (GMT) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.13.1/8.13.1) with ESMTP id j03L0PZX033400; Mon, 3 Jan 2005 14:00:25 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Mon, 03 Jan 2005 14:00:44 -0700 (MST) Message-Id: <20050103.140044.94756174.imp@bsdimp.com> To: phk@phk.freebsd.dk From: "M. Warner Losh" In-Reply-To: <18962.1104749259@critter.freebsd.dk> References: <18962.1104749259@critter.freebsd.dk> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit cc: arch@freebsd.org Subject: Re: making nmdm(4) emulate actual speed. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jan 2005 21:02:25 -0000 In message: <18962.1104749259@critter.freebsd.dk> Poul-Henning Kamp writes: : Rather than use two USB-serial dongles and a usb-hub, I hacked nmdm(4) : up to actually respect the baud-rate set with stty. : : Would this be considered generally useful ? Yes. It would be useful, but only as an option.... Warner From owner-freebsd-arch@FreeBSD.ORG Mon Jan 3 22:23:39 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E76B516A4CE for ; Mon, 3 Jan 2005 22:23:39 +0000 (GMT) Received: from mail18-ash-R.bigfish.com (mail-ash.bigfish.com [206.16.192.253]) by mx1.FreeBSD.org (Postfix) with ESMTP id 63FD643D45 for ; Mon, 3 Jan 2005 22:23:39 +0000 (GMT) (envelope-from yfeng@verniernetworks.com) Received: from mail18-ash.bigfish.com (localhost.localdomain [127.0.0.1]) by mail18-ash-R.bigfish.com (Postfix) with ESMTP id D61295E572B for ; Mon, 3 Jan 2005 22:23:36 +0000 (UCT) X-BigFish: VC Received: by mail18-ash (MessageSwitch) id 1104791016709629_16166; Mon, 3 Jan 2005 22:23:36 +0000 (UCT) Received: from exch2.verniernetworks.com (dns.verniernetworks.com [65.200.185.165]) by mail18-ash.bigfish.com (Postfix) with ESMTP id 5C9DF5E5BB5 for ; Mon, 3 Jan 2005 22:23:36 +0000 (UCT) X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Date: Mon, 3 Jan 2005 14:23:25 -0800 Message-ID: <12394E9FCB7C8441BB238D7F67B402E407E5C6@exch2.verniernetworks.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: User process starvation in FreeBSD 5.3 Thread-Index: AcTx4teEtGwBgzB/QNOtBKDrs6TYug== From: "Youlin Feng" To: Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.1 Subject: User process starvation in FreeBSD 5.3 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jan 2005 22:23:40 -0000 Hello all, =20 We are building a network appliance running FreeBSD 5.3 and under very heavy network traffic the user processes don't get scheduled for an unacceptable period of time. Marking the user process/thread real-time class doesn't help since the real-time user threads priorities are still lower than the interrupt threads. BTW, in our case, the CPU spends very long time at PI_NET (16) priority level, instead of at the (PI_SOFT + 4) soft intr level due to the packet forwarding nature. Either way, the user processes don't have a chance. In the following discussion I'll use PI_NET to represent the network interrupt threads priority. =20 I am experimenting with improving the real-time threads scheduling performance and would like to hear from you. =20 Here is what I am doing: 1. Raise the minimum real-time user threads priority to a value higher than PI_NET, e.g.=20 #define PRI_MIN_REALTIME 0 And use the rtprio() syscall or command to set the priority to be higher than PI_NET, e.g. "rtprio 10 " =20 This didn't turn out to be enough, since the real-time user process still uses system services or drivers that run at a lower priority than PI_NET, e.g. disk, tty, etc. =20 2. Change the PI_NET value to be the highest of all interrupt threads. Potential performance impact isn't a concern here since we have relatively rare non-network events. So PI_NET is changed to (PRI_MIN_ITHD + 32) from (PRI_MIN_ITHD + 16). This should give preference to all other I/O interrupt threads. I am assuming the software interrupt scheduling isn't the bottleneck. =20 I haven't got a chance to reproduce the problem using the 2nd change yet. I suspect that it isn't enough to get what I want, since lots of kernel code do msleep(...,pri,...) with "pri" bigger than the changed PI_NET. But, if this thinking of changing the priority ranges makes sense, PI_NET can always be changed to the highest value of all kernel priority values, ie the lowest kernel priority. =20 Hopefully this change would work and not be intrusive to the kernel. =20 Thank you for your comments in advance. =20 Youlin Feng From owner-freebsd-arch@FreeBSD.ORG Tue Jan 4 01:06:03 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2B28D16A4CE for ; Tue, 4 Jan 2005 01:06:03 +0000 (GMT) Received: from schlepper.zs64.net (schlepper.zs64.net [212.12.50.230]) by mx1.FreeBSD.org (Postfix) with ESMTP id 09FBF43D1F for ; Tue, 4 Jan 2005 01:06:02 +0000 (GMT) (envelope-from stb@lassitu.de) Received: from [IPv6:::1] (schlepper [212.12.50.230]) by schlepper.zs64.net (8.13.1/8.11.1) with ESMTP id j0415vQF029797; Tue, 4 Jan 2005 02:05:59 +0100 (CET) (envelope-from stb@lassitu.de) In-Reply-To: <12394E9FCB7C8441BB238D7F67B402E407E5C6@exch2.verniernetworks.com> References: <12394E9FCB7C8441BB238D7F67B402E407E5C6@exch2.verniernetworks.com> Mime-Version: 1.0 (Apple Message framework v619) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Stefan Bethke Date: Tue, 4 Jan 2005 02:05:56 +0100 To: "Youlin Feng" X-Mailer: Apple Mail (2.619) cc: freebsd-arch@freebsd.org Subject: Re: User process starvation in FreeBSD 5.3 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jan 2005 01:06:03 -0000 Am 03.01.2005 um 23:23 schrieb Youlin Feng: > We are building a network appliance running FreeBSD 5.3 and under very > heavy network traffic the user processes don't get scheduled for an > unacceptable period of time. Marking the user process/thread real-time > class doesn't help since the real-time user threads priorities are > still > lower than the interrupt threads. The effect you're describing very much sounds like 'livelock': the system is so overwhelmed with interrupts that it has no time to do anything else but servicing them. FreeBSD offers polling(4), which is intended to mitigate the overhead of a high interrupt rate on certain network controllers; especially in high-throughput scenarios, it can improve system load, throughput, and latency quite dramatically. On the other hand, your hardware might just not be able handle the packet rate. I'd suggest asking this on freebsd-net, where I believe quite a few people with experience in high-throughput setups are reading. Cheers, Stefan -- Stefan Bethke Fon +49 170 346 0140 From owner-freebsd-arch@FreeBSD.ORG Tue Jan 4 22:40:46 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3EBC816A4CE; Tue, 4 Jan 2005 22:40:46 +0000 (GMT) Received: from darkness.comp.waw.pl (darkness.comp.waw.pl [195.117.238.136]) by mx1.FreeBSD.org (Postfix) with ESMTP id D879F43D45; Tue, 4 Jan 2005 22:40:45 +0000 (GMT) (envelope-from pjd@darkness.comp.waw.pl) Received: by darkness.comp.waw.pl (Postfix, from userid 1009) id B4E9FACC47; Tue, 4 Jan 2005 23:40:43 +0100 (CET) Date: Tue, 4 Jan 2005 23:40:43 +0100 From: Pawel Jakub Dawidek To: arch@freebsd.org Message-ID: <20050104224043.GM784@darkness.comp.waw.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Z1OTrj3C7qypP14j" Content-Disposition: inline User-Agent: Mutt/1.4.2i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 5.2.1-RC2 i386 cc: scottl@freebsd.org Subject: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jan 2005 22:40:46 -0000 --Z1OTrj3C7qypP14j Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi. I want you to look at two patches which makes du(1) 64bit clean. This work is part of the BigDisk project: http://www.freebsd.org/projects/bigdisk/ The main problem here is that du(1) uses fts(3) and fts_number field from one of its structures to store size. This field is defined as 'long' so it doesn't give us what we want (on 32bit archs). So first of all, we need this patch: http://people.freebsd.org/~pjd/patches/fts.h.patch It adds 64bit fts_bignum field, but because it is hiden under union, it doesn't break ABI. It also doesn't break API, thanks to #defines. Patch for du(1) is here: http://people.freebsd.org/~pjd/patches/du-64bit-clean.patch We should decide if we want fts.h.patch also in HEAD or only in RELENG_5, because we can make it much cleaner in HEAD, but we will break ABI (API should be quite ok) - we need to change size of fts_number field. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --Z1OTrj3C7qypP14j Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQFB2xtrForvXbEpPzQRAlAAAKCex0u69e6lWz3SwIrZFG/eINDBhwCg4Jo7 HEAbpGuTwXdsbdqhHB+5Jn0= =aM1R -----END PGP SIGNATURE----- --Z1OTrj3C7qypP14j-- From owner-freebsd-arch@FreeBSD.ORG Tue Jan 4 23:47:49 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3643216A4CE; Tue, 4 Jan 2005 23:47:49 +0000 (GMT) Received: from mail.vicor-nb.com (bigwoop.vicor-nb.com [208.206.78.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 744C843D3F; Tue, 4 Jan 2005 23:47:48 +0000 (GMT) (envelope-from julian@elischer.org) Received: from elischer.org (julian.vicor-nb.com [208.206.78.97]) by mail.vicor-nb.com (Postfix) with ESMTP id 41F847A425; Tue, 4 Jan 2005 15:47:48 -0800 (PST) Message-ID: <41DB2B24.6050005@elischer.org> Date: Tue, 04 Jan 2005 15:47:48 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.3.1) Gecko/20030516 X-Accept-Language: en, hu MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <20050104224043.GM784@darkness.comp.waw.pl> In-Reply-To: <20050104224043.GM784@darkness.comp.waw.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: arch@freebsd.org cc: scottl@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jan 2005 23:47:49 -0000 Pawel Jakub Dawidek wrote: >Hi. > >I want you to look at two patches which makes du(1) 64bit clean. >This work is part of the BigDisk project: > > http://www.freebsd.org/projects/bigdisk/ > > One thing that needs to be done is an 2ndary storage fsck. that doesn't try put everything in RAM. Basically this will mean extracting all the metadata from filesystems into files and running sort operations of various kinds on them to order the data in ways that allows consistencies to be checked. It will take a bit longer than a RAM fsck but maybe not as much as one might fear. We all remember those "sort a mag-tape larger than RAM" lessons from CS101 don't we? At least it doesn't have to be "in place" so merge sorts are OK. :-) why? A bitmap of 1TB of 512 byte records is 244MB so with a 4BG machine with 3GB available to the process you can't even fit the bitmaps into memory for a 12TB Filesystem let alone other metadata. Going to 2048 byte frags helps but you still run into a limit. last I tried it, you need about 600MB per TB of fileysstem to check. So I think a special fsck that uses files is a must for really big filesystems, unless they (the filesystems) can be broken up in a logical way (IBM did that many years ago I believe). I think you should add that to your list. From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 02:58:33 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 79FFA16A4CE; Wed, 5 Jan 2005 02:58:33 +0000 (GMT) Received: from carver.gumbysoft.com (carver.gumbysoft.com [66.220.23.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6660543D55; Wed, 5 Jan 2005 02:58:33 +0000 (GMT) (envelope-from dwhite@gumbysoft.com) Received: by carver.gumbysoft.com (Postfix, from userid 1000) id 5833372DD4; Tue, 4 Jan 2005 18:58:33 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by carver.gumbysoft.com (Postfix) with ESMTP id 5359172DCB; Tue, 4 Jan 2005 18:58:33 -0800 (PST) Date: Tue, 4 Jan 2005 18:58:33 -0800 (PST) From: Doug White To: Pawel Jakub Dawidek In-Reply-To: <20050104224043.GM784@darkness.comp.waw.pl> Message-ID: <20050104183627.O20855@carver.gumbysoft.com> References: <20050104224043.GM784@darkness.comp.waw.pl> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@freebsd.org cc: scottl@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 02:58:33 -0000 On Tue, 4 Jan 2005, Pawel Jakub Dawidek wrote: > I want you to look at two patches which makes du(1) 64bit clean. > This work is part of the BigDisk project: > > http://www.freebsd.org/projects/bigdisk/ > > The main problem here is that du(1) uses fts(3) and fts_number field from > one of its structures to store size. > This field is defined as 'long' so it doesn't give us what we want > (on 32bit archs). No offense intended, but can we avoid introducing LP64 bugs, please? Particularly when the goal is "ABI compatibility."* dwlab3,ttyp1,~,24>uname -m i386 dwlab3,ttyp1,~,25>./test sizeof(long) [4] + sizeof(void *) [4] == 8 == sizeof(int64_t) [8] ok .. but: dwlab4,ttyp1,~,20>uname -m amd64 dwlab4,ttyp1,~,21>./test sizeof(long) [8] + sizeof(void *) [8] == 16 != sizeof(int64_t) [8] oops! The struct just grew by 8 bytes! (*) On the same platform, obviously. -- Doug White | FreeBSD: The Power to Serve dwhite@gumbysoft.com | www.FreeBSD.org From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 03:03:01 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2BA7F16A4CE; Wed, 5 Jan 2005 03:03:01 +0000 (GMT) Received: from odin.ac.hmc.edu (Odin.AC.HMC.Edu [134.173.32.75]) by mx1.FreeBSD.org (Postfix) with ESMTP id C765543D1F; Wed, 5 Jan 2005 03:03:00 +0000 (GMT) (envelope-from brdavis@odin.ac.hmc.edu) Received: from odin.ac.hmc.edu (localhost.localdomain [127.0.0.1]) by odin.ac.hmc.edu (8.13.0/8.13.0) with ESMTP id j0534QTD026367; Tue, 4 Jan 2005 19:04:26 -0800 Received: (from brdavis@localhost) by odin.ac.hmc.edu (8.13.0/8.13.0/Submit) id j0534QU6026366; Tue, 4 Jan 2005 19:04:26 -0800 Date: Tue, 4 Jan 2005 19:04:26 -0800 From: Brooks Davis To: Pawel Jakub Dawidek Message-ID: <20050105030426.GB24604@odin.ac.hmc.edu> References: <20050104224043.GM784@darkness.comp.waw.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="aVD9QWMuhilNxW9f" Content-Disposition: inline In-Reply-To: <20050104224043.GM784@darkness.comp.waw.pl> User-Agent: Mutt/1.4.1i X-Virus-Scanned: by amavisd-new X-Spam-Status: No, hits=0.0 required=8.0 tests=none autolearn=no version=2.63 X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on odin.ac.hmc.edu cc: arch@freebsd.org cc: scottl@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 03:03:01 -0000 --aVD9QWMuhilNxW9f Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 04, 2005 at 11:40:43PM +0100, Pawel Jakub Dawidek wrote: > Hi. >=20 > I want you to look at two patches which makes du(1) 64bit clean. > This work is part of the BigDisk project: >=20 > http://www.freebsd.org/projects/bigdisk/ >=20 > The main problem here is that du(1) uses fts(3) and fts_number field from > one of its structures to store size. > This field is defined as 'long' so it doesn't give us what we want > (on 32bit archs). >=20 > So first of all, we need this patch: >=20 > http://people.freebsd.org/~pjd/patches/fts.h.patch >=20 > It adds 64bit fts_bignum field, but because it is hiden under union, > it doesn't break ABI. It also doesn't break API, thanks to #defines. >=20 > Patch for du(1) is here: >=20 > http://people.freebsd.org/~pjd/patches/du-64bit-clean.patch >=20 > We should decide if we want fts.h.patch also in HEAD or only in RELENG_5, > because we can make it much cleaner in HEAD, but we will break ABI > (API should be quite ok) - we need to change size of fts_number field. I'd be inclined to use the somewhat gross fix in PR 74567 in RELENG_5 and do it right in HEAD. bde suggested changing fts_num to intmax_t. -- Brooks --=20 Any statement of the form "X is the one, true Y" is FALSE. PGP fingerprint 655D 519C 26A7 82E7 2529 9BF0 5D8E 8BE9 F238 1AD4 --aVD9QWMuhilNxW9f Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQFB21k6XY6L6fI4GtQRAjaZAJ9tniH3xspI4bx/8mJt6YABLOvXoACghYHi 1Rt4YC2eV0zvBt5S/PYCBdg= =vYru -----END PGP SIGNATURE----- --aVD9QWMuhilNxW9f-- From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 03:12:09 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6FC3416A4CE; Wed, 5 Jan 2005 03:12:09 +0000 (GMT) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0FD4B43D3F; Wed, 5 Jan 2005 03:12:09 +0000 (GMT) (envelope-from scottl@freebsd.org) Received: from [192.168.254.11] (junior-wifi.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.12.11/8.12.10) with ESMTP id j053F0rF015546; Tue, 4 Jan 2005 20:15:01 -0700 (MST) (envelope-from scottl@freebsd.org) Message-ID: <41DB5AAB.9080705@freebsd.org> Date: Tue, 04 Jan 2005 20:10:35 -0700 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.2) Gecko/20040929 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Doug White References: <20050104224043.GM784@darkness.comp.waw.pl> <20050104183627.O20855@carver.gumbysoft.com> In-Reply-To: <20050104183627.O20855@carver.gumbysoft.com> X-Enigmail-Version: 0.86.1.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, hits=0.0 required=3.8 tests=none autolearn=no version=2.63 X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on pooker.samsco.org cc: arch@freebsd.org cc: Pawel Jakub Dawidek Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 03:12:09 -0000 Doug White wrote: > On Tue, 4 Jan 2005, Pawel Jakub Dawidek wrote: > > >>I want you to look at two patches which makes du(1) 64bit clean. >>This work is part of the BigDisk project: >> >> http://www.freebsd.org/projects/bigdisk/ >> >>The main problem here is that du(1) uses fts(3) and fts_number field from >>one of its structures to store size. >>This field is defined as 'long' so it doesn't give us what we want >>(on 32bit archs). > > > No offense intended, but can we avoid introducing LP64 bugs, please? > Particularly when the goal is "ABI compatibility."* > > dwlab3,ttyp1,~,24>uname -m > i386 > dwlab3,ttyp1,~,25>./test > sizeof(long) [4] + sizeof(void *) [4] == 8 == sizeof(int64_t) [8] > > ok .. but: > > dwlab4,ttyp1,~,20>uname -m > amd64 > dwlab4,ttyp1,~,21>./test > sizeof(long) [8] + sizeof(void *) [8] == 16 != sizeof(int64_t) [8] > > oops! The struct just grew by 8 bytes! > > (*) On the same platform, obviously. > I'm not sure how you are getting this. The structure goes from long fts_number; /* local numeric value */ void *fts_pointer; /* local address value */ to union { struct { long __fts_number; /* local numeric value */ void *__fts_pointer; /* local address value */ } __struct_ftsent; int64_t __fts_bignum; } __union_ftsent; Regardless of how big a pointer or a long is on your platform, the two fields are going to combine to consume at least 64 bits. All that the change does is overlay those >= 64bits with a int64_t. Scott From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 03:36:11 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AD0ED16A4CE; Wed, 5 Jan 2005 03:36:11 +0000 (GMT) Received: from harmony.village.org (rover.village.org [168.103.84.182]) by mx1.FreeBSD.org (Postfix) with ESMTP id B9EC543D3F; Wed, 5 Jan 2005 03:36:10 +0000 (GMT) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.13.1/8.13.1) with ESMTP id j053Yg0w052603; Tue, 4 Jan 2005 20:34:42 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Tue, 04 Jan 2005 20:35:02 -0700 (MST) Message-Id: <20050104.203502.85411551.imp@bsdimp.com> To: julian@elischer.org From: "M. Warner Losh" In-Reply-To: <41DB2B24.6050005@elischer.org> References: <20050104224043.GM784@darkness.comp.waw.pl> <41DB2B24.6050005@elischer.org> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit cc: arch@freebsd.org cc: scottl@freebsd.org cc: pjd@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 03:36:11 -0000 In message: <41DB2B24.6050005@elischer.org> Julian Elischer writes: : : : Pawel Jakub Dawidek wrote: : : >Hi. : > : >I want you to look at two patches which makes du(1) 64bit clean. : >This work is part of the BigDisk project: : > : > http://www.freebsd.org/projects/bigdisk/ : > : > : One thing that needs to be done is an 2ndary storage fsck. : that doesn't try put everything in RAM. : Basically this will mean extracting all the metadata from filesystems into : files and running sort operations of various kinds on them : to order the data in ways that allows consistencies to be checked. : It will take a bit longer than a RAM fsck but maybe not as much as : one might fear. : We all remember those "sort a mag-tape larger than RAM" : lessons from CS101 don't we? : At least it doesn't have to be "in place" so merge sorts are OK. :-) : : why? : : A bitmap of 1TB of 512 byte records is 244MB so with a 4BG machine : with 3GB available to the process you can't even fit the bitmaps into : memory for a 12TB Filesystem let alone other metadata. : : Going to 2048 byte frags helps but you still run into a limit. : last I tried it, you need about 600MB per TB of fileysstem to check. : : So I think a special fsck that uses files is a must for really big : filesystems, unless they (the filesystems) can be broken up in : a logical way (IBM did that many years ago I believe). : I think you should add that to your list. I think that a big amount of this could be reduced by using simple arrays rather than lists which are more memory efficient... Warner From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 03:46:06 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EEE1D16A4CE; Wed, 5 Jan 2005 03:46:06 +0000 (GMT) Received: from carver.gumbysoft.com (carver.gumbysoft.com [66.220.23.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id D5A9D43D1F; Wed, 5 Jan 2005 03:46:06 +0000 (GMT) (envelope-from dwhite@gumbysoft.com) Received: by carver.gumbysoft.com (Postfix, from userid 1000) id C62E872DD4; Tue, 4 Jan 2005 19:46:06 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by carver.gumbysoft.com (Postfix) with ESMTP id C12C272DCB; Tue, 4 Jan 2005 19:46:06 -0800 (PST) Date: Tue, 4 Jan 2005 19:46:06 -0800 (PST) From: Doug White To: Scott Long In-Reply-To: <41DB5AAB.9080705@freebsd.org> Message-ID: <20050104194302.O21516@carver.gumbysoft.com> References: <20050104224043.GM784@darkness.comp.waw.pl> <20050104183627.O20855@carver.gumbysoft.com> <41DB5AAB.9080705@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@freebsd.org cc: Pawel Jakub Dawidek Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 03:46:07 -0000 On Tue, 4 Jan 2005, Scott Long wrote: > > dwlab3,ttyp1,~,24>uname -m > > i386 > > dwlab3,ttyp1,~,25>./test > > sizeof(long) [4] + sizeof(void *) [4] == 8 == sizeof(int64_t) [8] > > > > ok .. but: > > > > dwlab4,ttyp1,~,20>uname -m > > amd64 > > dwlab4,ttyp1,~,21>./test > > sizeof(long) [8] + sizeof(void *) [8] == 16 != sizeof(int64_t) [8] > > > > oops! The struct just grew by 8 bytes! > > > > (*) On the same platform, obviously. > > > > I'm not sure how you are getting this. The structure goes from > > long fts_number; /* local numeric value */ > void *fts_pointer; /* local address value */ > > to > > union { > struct { > long __fts_number; /* local numeric value */ > void *__fts_pointer; /* local address value */ > } __struct_ftsent; > int64_t __fts_bignum; > } __union_ftsent; > > > Regardless of how big a pointer or a long is on your platform, the two > fields are going to combine to consume at least 64 bits. All that the > change does is overlay those >= 64bits with a int64_t. Oops, you're right. Since the new member is <= in size to the previous members, strictly speaking it shouldn't change size. Now there is a question if gcc will do something odd to that embedded struct for alignment purposes. I haven't checked the original code for __packed, however... I'm also not clear on if endianness plays a part, but I need to read the code. -- Doug White | FreeBSD: The Power to Serve dwhite@gumbysoft.com | www.FreeBSD.org From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 04:46:38 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 57E5C16A4CE for ; Wed, 5 Jan 2005 04:46:38 +0000 (GMT) Received: from mail03.syd.optusnet.com.au (mail03.syd.optusnet.com.au [211.29.132.184]) by mx1.FreeBSD.org (Postfix) with ESMTP id 77DED43D39 for ; Wed, 5 Jan 2005 04:46:37 +0000 (GMT) (envelope-from PeterJeremy@optushome.com.au) Received: from cirb503493.alcatel.com.au (c211-30-75-229.belrs2.nsw.optusnet.com.au [211.30.75.229]) j054kUaf032622 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Wed, 5 Jan 2005 15:46:31 +1100 Received: from cirb503493.alcatel.com.au (localhost.alcatel.com.au [127.0.0.1])j054kUxP038432; Wed, 5 Jan 2005 15:46:30 +1100 (EST) (envelope-from pjeremy@cirb503493.alcatel.com.au) Received: (from pjeremy@localhost)j054kQKh038431; Wed, 5 Jan 2005 15:46:26 +1100 (EST) (envelope-from pjeremy) Date: Wed, 5 Jan 2005 15:46:26 +1100 From: Peter Jeremy To: Julian Elischer Message-ID: <20050105044626.GI34222@cirb503493.alcatel.com.au> References: <20050104224043.GM784@darkness.comp.waw.pl> <41DB2B24.6050005@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <41DB2B24.6050005@elischer.org> User-Agent: Mutt/1.4.2i cc: arch@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 04:46:38 -0000 On Tue, 2005-Jan-04 15:47:48 -0800, Julian Elischer wrote: >One thing that needs to be done is an 2ndary storage fsck. >that doesn't try put everything in RAM. Agreed but VM is different to physical RAM. >Basically this will mean extracting all the metadata from filesystems into >files and running sort operations of various kinds on them >to order the data in ways that allows consistencies to be checked. How do you determine where the disk space comes from? You can't safely use the filesystem being checked because you can't safely write to it until the fsck has completed. _Any_ filesystem picked automatically may not have sufficient free space to do an fsck. (Older Unices prompted for a temporary directory for work files). Overall, the most likely location for free space is swap. I'd therefore suggest that rather than moving back to using temporary files, we look at how to better use swap space: - Look into reducing VM requirements (600MB per TB seems high) - Tweak data structures to maximise locality of reference (traversing linked lists is extremely slow when your list won't fit into RAM) - Give fsck the ability to split into multiple processes to avoid process size limits - usr socketpair()s to allow a "master" process to delegate work ala dump(8). The downside of using swap is the possibility of over-writing a crash dump. -- Peter Jeremy From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 05:17:02 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BB04A16A4CE; Wed, 5 Jan 2005 05:17:02 +0000 (GMT) Received: from mail12.syd.optusnet.com.au (mail12.syd.optusnet.com.au [211.29.132.193]) by mx1.FreeBSD.org (Postfix) with ESMTP id 24DE443D39; Wed, 5 Jan 2005 05:17:02 +0000 (GMT) (envelope-from PeterJeremy@optushome.com.au) Received: from cirb503493.alcatel.com.au (c211-30-75-229.belrs2.nsw.optusnet.com.au [211.30.75.229]) j055H0JG013754 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO); Wed, 5 Jan 2005 16:17:01 +1100 Received: from cirb503493.alcatel.com.au (localhost.alcatel.com.au [127.0.0.1])j055H0xP038476; Wed, 5 Jan 2005 16:17:00 +1100 (EST) (envelope-from pjeremy@cirb503493.alcatel.com.au) Received: (from pjeremy@localhost)j055H0Ut038475; Wed, 5 Jan 2005 16:17:00 +1100 (EST) (envelope-from pjeremy) Date: Wed, 5 Jan 2005 16:17:00 +1100 From: Peter Jeremy To: Pawel Jakub Dawidek Message-ID: <20050105051700.GJ34222@cirb503493.alcatel.com.au> References: <20050104224043.GM784@darkness.comp.waw.pl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20050104224043.GM784@darkness.comp.waw.pl> User-Agent: Mutt/1.4.2i cc: arch@freebsd.org cc: scottl@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 05:17:02 -0000 On Tue, 2005-Jan-04 23:40:43 +0100, Pawel Jakub Dawidek wrote: >I want you to look at two patches which makes du(1) 64bit clean. You might also like to look at the thread beginning http://lists.freebsd.org/pipermail/freebsd-hackers/2004-December/009427.html > http://people.freebsd.org/~pjd/patches/fts.h.patch > >It adds 64bit fts_bignum field, but because it is hiden under union, >it doesn't break ABI. It also doesn't break API, thanks to #defines. Very nice. Much cleaner than my suggestions. All of our supported architectures are either I32LP64 or ILP32 (ignoring bde's IP32L64 i386 variant) therefore the union won't affect alignment or alter padding. Since NULL is physically represented by all zero bits on all our architectures, fts_bignum is even correctly initialised to zero - though this bit isn't standards compliant. >We should decide if we want fts.h.patch also in HEAD or only in RELENG_5, >because we can make it much cleaner in HEAD, but we will break ABI >(API should be quite ok) - we need to change size of fts_number field. I gather you're suggesting that fts_number be changed to int64_t in HEAD. I'm less convinced about this - though if this change is made, it should be done sooner rather than later. The only downside is that ls(1) will bloat further on 32-bit architectures (I recall seeing at least one complaint about running ls on a ludicrously big directory). fts(3) is a BSD4.4 invention. The 4.x man page suggests future standardisation by POSIX but this statement disappeared in v1.14 without any comment in the commit logs. Rather than changing fts_number to int64, I suggest leaving the patch and documenting that fts_bignum overlays both fts_number and fts_pointer. The need for fts_bignum will probably disappear in time. -- Peter Jeremy From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 06:57:34 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 58EA216A4CE; Wed, 5 Jan 2005 06:57:34 +0000 (GMT) Received: from pimout2-ext.prodigy.net (pimout2-ext.prodigy.net [207.115.63.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id E1B4B43D49; Wed, 5 Jan 2005 06:57:33 +0000 (GMT) (envelope-from julian@elischer.org) Received: from [192.168.1.102] (adsl-216-100-134-143.dsl.snfc21.pacbell.net [216.100.134.143])j056vOXO011736; Wed, 5 Jan 2005 01:57:29 -0500 Message-ID: <41DB8FD3.3030101@elischer.org> Date: Tue, 04 Jan 2005 22:57:23 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8a3) Gecko/20041017 X-Accept-Language: en, hu MIME-Version: 1.0 To: "M. Warner Losh" References: <20050104224043.GM784@darkness.comp.waw.pl> <41DB2B24.6050005@elischer.org> <20050104.203502.85411551.imp@bsdimp.com> In-Reply-To: <20050104.203502.85411551.imp@bsdimp.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: arch@freebsd.org cc: scottl@freebsd.org cc: pjd@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 06:57:34 -0000 M. Warner Losh wrote: > In message: <41DB2B24.6050005@elischer.org> > Julian Elischer writes: > : > : > : Pawel Jakub Dawidek wrote: > : > : >Hi. > : > > : >I want you to look at two patches which makes du(1) 64bit clean. > : >This work is part of the BigDisk project: > : > > : > http://www.freebsd.org/projects/bigdisk/ > : > > : > > : One thing that needs to be done is an 2ndary storage fsck. > : that doesn't try put everything in RAM. > : Basically this will mean extracting all the metadata from filesystems into > : files and running sort operations of various kinds on them > : to order the data in ways that allows consistencies to be checked. > : It will take a bit longer than a RAM fsck but maybe not as much as > : one might fear. > : We all remember those "sort a mag-tape larger than RAM" > : lessons from CS101 don't we? > : At least it doesn't have to be "in place" so merge sorts are OK. :-) > : > : why? > : > : A bitmap of 1TB of 512 byte records is 244MB so with a 4BG machine > : with 3GB available to the process you can't even fit the bitmaps into > : memory for a 12TB Filesystem let alone other metadata. > : > : Going to 2048 byte frags helps but you still run into a limit. > : last I tried it, you need about 600MB per TB of fileysstem to check. > : > : So I think a special fsck that uses files is a must for really big > : filesystems, unless they (the filesystems) can be broken up in > : a logical way (IBM did that many years ago I believe). > : I think you should add that to your list. > > I think that a big amount of this could be reduced by using simple > arrays rather than lists which are more memory efficient... > > Warner but that just puts the problem a bit further away. From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 07:05:26 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2E32416A4CE; Wed, 5 Jan 2005 07:05:26 +0000 (GMT) Received: from darkness.comp.waw.pl (darkness.comp.waw.pl [195.117.238.136]) by mx1.FreeBSD.org (Postfix) with ESMTP id C65A643D48; Wed, 5 Jan 2005 07:05:25 +0000 (GMT) (envelope-from pjd@darkness.comp.waw.pl) Received: by darkness.comp.waw.pl (Postfix, from userid 1009) id 2EEF6ACAEE; Wed, 5 Jan 2005 08:05:24 +0100 (CET) Date: Wed, 5 Jan 2005 08:05:24 +0100 From: Pawel Jakub Dawidek To: Doug White Message-ID: <20050105070524.GN784@darkness.comp.waw.pl> References: <20050104224043.GM784@darkness.comp.waw.pl> <20050104183627.O20855@carver.gumbysoft.com> <41DB5AAB.9080705@freebsd.org> <20050104194302.O21516@carver.gumbysoft.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ICrdrp3pM9DyZLTK" Content-Disposition: inline In-Reply-To: <20050104194302.O21516@carver.gumbysoft.com> User-Agent: Mutt/1.4.2i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 5.2.1-RC2 i386 cc: arch@freebsd.org cc: Scott Long Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 07:05:26 -0000 --ICrdrp3pM9DyZLTK Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 04, 2005 at 07:46:06PM -0800, Doug White wrote: +> Now there is a question if gcc will do something odd to that embedded +> struct for alignment purposes. I haven't checked the original code for +> __packed, however... I checked where fields are placed after and before this change on all our archs and ABI is preserved. I wrote a small tool for this purpose if you want to play: http://people.freebsd.org/~pjd/ftstest.tbz % tar -jvxf ftstest.tbz % cd ftstest % make % old/oldfts % new/newfts It shows addresses of few fields in the structure. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --ICrdrp3pM9DyZLTK Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQFB25G0ForvXbEpPzQRAodgAJ0ZiJPbNSUHCEiDpdQ2BlZapv1MkwCgl2X1 Lv9lO4pj4TxwKv31jEMZ2K0= =QU/D -----END PGP SIGNATURE----- --ICrdrp3pM9DyZLTK-- From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 07:10:08 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DE84516A4CE for ; Wed, 5 Jan 2005 07:10:08 +0000 (GMT) Received: from pimout3-ext.prodigy.net (pimout3-ext.prodigy.net [207.115.63.102]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6761543D2F for ; Wed, 5 Jan 2005 07:10:08 +0000 (GMT) (envelope-from julian@elischer.org) Received: from [192.168.1.102] (adsl-216-100-134-143.dsl.snfc21.pacbell.net [216.100.134.143])j057A5Hg386634; Wed, 5 Jan 2005 02:10:06 -0500 Message-ID: <41DB92C9.50801@elischer.org> Date: Tue, 04 Jan 2005 23:10:01 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8a3) Gecko/20041017 X-Accept-Language: en, hu MIME-Version: 1.0 To: Peter Jeremy References: <20050104224043.GM784@darkness.comp.waw.pl> <41DB2B24.6050005@elischer.org> <20050105044626.GI34222@cirb503493.alcatel.com.au> In-Reply-To: <20050105044626.GI34222@cirb503493.alcatel.com.au> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: arch@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 07:10:09 -0000 Peter Jeremy wrote: > On Tue, 2005-Jan-04 15:47:48 -0800, Julian Elischer wrote: > >>One thing that needs to be done is an 2ndary storage fsck. >>that doesn't try put everything in RAM. > > > Agreed but VM is different to physical RAM. > > >>Basically this will mean extracting all the metadata from filesystems into >>files and running sort operations of various kinds on them >>to order the data in ways that allows consistencies to be checked. > > > How do you determine where the disk space comes from? You can't > safely use the filesystem being checked because you can't safely write > to it until the fsck has completed. _Any_ filesystem picked > automatically may not have sufficient free space to do an fsck. > (Older Unices prompted for a temporary directory for work files). One assumes you also have a smaller (say 100GB temp filesystem that you can newfs on boot.. or some other similar system.. Here we have our own fsck that acts differently, and we do /var with the regular fsck first and then use it as a place to store the logfiles of the main fsck runs on our TB raids using our modified fsck. > > Overall, the most likely location for free space is swap. I'd therefore > suggest that rather than moving back to using temporary files, we look > at how to better use swap space: swap may not be big enough either.. > - Look into reducing VM requirements (600MB per TB seems high) I only report what it does.. > - Tweak data structures to maximise locality of reference (traversing > linked lists is extremely slow when your list won't fit into RAM) > - Give fsck the ability to split into multiple processes to avoid process > size limits - usr socketpair()s to allow a "master" process to delegate > work ala dump(8). still only DELAYs the problem. > > The downside of using swap is the possibility of over-writing a crash dump. > From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 08:22:56 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CE52316A4CE for ; Wed, 5 Jan 2005 08:22:56 +0000 (GMT) Received: from mail.freebsd.org.cn (dns3.freebsd.org.cn [61.129.66.75]) by mx1.FreeBSD.org (Postfix) with SMTP id 6424F43D2F for ; Wed, 5 Jan 2005 08:22:48 +0000 (GMT) (envelope-from delphij@frontfree.net) Received: (qmail 62506 invoked by uid 0); 5 Jan 2005 08:15:24 -0000 Received: from unknown (HELO beastie.frontfree.net) (219.239.99.7) by mail.freebsd.org.cn with SMTP; 5 Jan 2005 08:15:24 -0000 Received: from localhost (localhost.frontfree.net [127.0.0.1]) by beastie.frontfree.net (Postfix) with ESMTP id 8C21913195B for ; Wed, 5 Jan 2005 16:22:38 +0800 (CST) Received: from beastie.frontfree.net ([127.0.0.1]) by localhost (beastie.frontfree.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 00147-02 for ; Wed, 5 Jan 2005 16:22:27 +0800 (CST) Received: by beastie.frontfree.net (Postfix, from userid 1001) id 4E826134B96; Wed, 5 Jan 2005 16:22:26 +0800 (CST) Date: Wed, 5 Jan 2005 16:22:26 +0800 From: Xin LI To: freebsd-arch@FreeBSD.org Message-ID: <20050105082226.GA1899@frontfree.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BOKacYhQ+x31HxR3" Content-Disposition: inline User-Agent: Mutt/1.4.2.1i X-GPG-key-ID/Fingerprint: 0xCAEEB8C0 / 43B8 B703 B8DD 0231 B333 DC28 39FB 93A0 CAEE B8C0 X-GPG-Public-Key: http://www.delphij.net/delphij.asc X-Operating-System: FreeBSD beastie.frontfree.net 5.3-delphij FreeBSD 5.3-delphij #11: Tue Oct 26 14:12:03 CST 2004 delphij@beastie.frontfree.net:/usr/obj/usr/src/sys/BEASTIE i386 X-URL: http://www.delphij.net X-By: delphij@beastie.frontfree.net X-Location: Beijing, China X-Virus-Scanned: by amavisd-new at frontfree.net Subject: RFC: Two fields in kld_sym_lookup and nlist structures X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 08:22:57 -0000 --BOKacYhQ+x31HxR3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Dear folks, While I was traversing our source tree, I found that kld_sym_lookup's symname and nlist's n_name are defined as char *. On the other hand, it seems that the usual usage of them looks like: nlist[0].n_name =3D "foo"; Which generates warnings on higher warning levels, since "foo" is a string constatnt, and n_name is supposed to be a variable char *. Can we change these fields into const char *? A preliminary patch looks like: Index: sys/sys/linker.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /home/ncvs/src/sys/sys/linker.h,v retrieving revision 1.39 diff -u -r1.39 linker.h --- sys/sys/linker.h 27 Aug 2004 01:10:16 -0000 1.39 +++ sys/sys/linker.h 5 Jan 2005 07:46:03 -0000 @@ -270,7 +270,7 @@ =20 struct kld_sym_lookup { int version; /* set to sizeof(struct kld_sym_lookup) */ - char *symname; /* Symbol name we are looking up */ + const char *symname; /* Symbol name we are looking up */ u_long symvalue; size_t symsize; }; Index: sys/sys/nlist_aout.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /home/ncvs/src/sys/sys/nlist_aout.h,v retrieving revision 1.11 diff -u -r1.11 nlist_aout.h --- sys/sys/nlist_aout.h 7 Apr 2004 04:19:49 -0000 1.11 +++ sys/sys/nlist_aout.h 5 Jan 2005 07:56:06 -0000 @@ -51,11 +51,11 @@ struct nlist { #ifdef _AOUT_INCLUDE_ union { - char *n_name; /* symbol name (in memory) */ + const char *n_name; /* symbol name (in memory) */ long n_strx; /* file string table offset (on disk) */ } n_un; #else - char *n_name; /* symbol name (in memory) */ + const char *n_name; /* symbol name (in memory) */ int : 8 * (sizeof(long) > sizeof(char *) ? sizeof(long) - sizeof(char *) : sizeof(char *) - sizeof(long)); #endif Cheers, --=20 Xin LI http://www.delphij.net/ See complete headers for GPG key and other information. --BOKacYhQ+x31HxR3 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (FreeBSD) iD8DBQFB26PC/cVsHxFZiIoRAjjVAJ0SZ4sE66b/NX9yAMwa05Ej2pOwTQCgiOh5 kfSW0HMfC2VtAQI2+oYJI+I= =VPsU -----END PGP SIGNATURE----- --BOKacYhQ+x31HxR3-- From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 10:20:58 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2C46716A4CF; Wed, 5 Jan 2005 10:20:58 +0000 (GMT) Received: from darkness.comp.waw.pl (darkness.comp.waw.pl [195.117.238.136]) by mx1.FreeBSD.org (Postfix) with ESMTP id C52BE43D39; Wed, 5 Jan 2005 10:20:57 +0000 (GMT) (envelope-from pjd@darkness.comp.waw.pl) Received: by darkness.comp.waw.pl (Postfix, from userid 1009) id 23455ACB34; Wed, 5 Jan 2005 11:20:56 +0100 (CET) Date: Wed, 5 Jan 2005 11:20:56 +0100 From: Pawel Jakub Dawidek To: Brooks Davis Message-ID: <20050105102056.GO784@darkness.comp.waw.pl> References: <20050104224043.GM784@darkness.comp.waw.pl> <20050105030426.GB24604@odin.ac.hmc.edu> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="AA9g+nFNFPYNJKiL" Content-Disposition: inline In-Reply-To: <20050105030426.GB24604@odin.ac.hmc.edu> User-Agent: Mutt/1.4.2i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 5.2.1-RC2 i386 cc: arch@freebsd.org cc: scottl@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 10:20:58 -0000 --AA9g+nFNFPYNJKiL Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 04, 2005 at 07:04:26PM -0800, Brooks Davis wrote: +> I'd be inclined to use the somewhat gross fix in PR 74567 in RELENG_5 +> and do it right in HEAD. [...] It allocates memory and we don't need it. Proposed fix is actually for RELENG_5. We can also do some magic inside du(1) to split 64bit value between two fields (fts_number/fts_pointer) when needed (on 32bit archs), but it would be really hackish. +> [...] bde suggested changing fts_num to intmax_t. The only issue here is that we break ABI if intmax_t will be bumped to 128bits in gcc, but we probably will have much bigger problems then:) I like this idea. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --AA9g+nFNFPYNJKiL Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQFB27+IForvXbEpPzQRAjziAJ9rxx5bioUTnVhgFh96w/dazJxcTQCdGm/b oLexIIsnaIK8DLsKTUWKYqc= =AYpl -----END PGP SIGNATURE----- --AA9g+nFNFPYNJKiL-- From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 16:37:18 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F091116A4CE; Wed, 5 Jan 2005 16:37:18 +0000 (GMT) Received: from harmony.village.org (rover.village.org [168.103.84.182]) by mx1.FreeBSD.org (Postfix) with ESMTP id 988D843D31; Wed, 5 Jan 2005 16:37:17 +0000 (GMT) (envelope-from imp@bsdimp.com) Received: from localhost (warner@rover2.village.org [10.0.0.1]) by harmony.village.org (8.13.1/8.13.1) with ESMTP id j05GYksD061289; Wed, 5 Jan 2005 09:34:46 -0700 (MST) (envelope-from imp@bsdimp.com) Date: Wed, 05 Jan 2005 09:35:08 -0700 (MST) Message-Id: <20050105.093508.102576805.imp@bsdimp.com> To: julian@elischer.org From: "M. Warner Losh" In-Reply-To: <41DB8FD3.3030101@elischer.org> References: <41DB2B24.6050005@elischer.org> <20050104.203502.85411551.imp@bsdimp.com> <41DB8FD3.3030101@elischer.org> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit cc: arch@freebsd.org cc: scottl@freebsd.org cc: pjd@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 16:37:19 -0000 In message: <41DB8FD3.3030101@elischer.org> Julian Elischer writes: : > I think that a big amount of this could be reduced by using simple : > arrays rather than lists which are more memory efficient... : : but that just puts the problem a bit further away. We have to do it. If you don't have a file system to store a file, then you cna't do this at all. When you are fscking /, you don't have a file system. And if you have finished, nothing is yet mounted r/w. Warner From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 19:04:05 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 43E4616A4CE; Wed, 5 Jan 2005 19:04:05 +0000 (GMT) Received: from odin.ac.hmc.edu (Odin.AC.HMC.Edu [134.173.32.75]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1B44443D39; Wed, 5 Jan 2005 19:04:05 +0000 (GMT) (envelope-from brdavis@odin.ac.hmc.edu) Received: from odin.ac.hmc.edu (localhost.localdomain [127.0.0.1]) by odin.ac.hmc.edu (8.13.0/8.13.0) with ESMTP id j05J5cIb009836; Wed, 5 Jan 2005 11:05:38 -0800 Received: (from brdavis@localhost) by odin.ac.hmc.edu (8.13.0/8.13.0/Submit) id j05J5cKQ009835; Wed, 5 Jan 2005 11:05:38 -0800 Date: Wed, 5 Jan 2005 11:05:38 -0800 From: Brooks Davis To: Pawel Jakub Dawidek Message-ID: <20050105190538.GA8054@odin.ac.hmc.edu> References: <20050104224043.GM784@darkness.comp.waw.pl> <20050105030426.GB24604@odin.ac.hmc.edu> <20050105102056.GO784@darkness.comp.waw.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="SUOF0GtieIMvvwua" Content-Disposition: inline In-Reply-To: <20050105102056.GO784@darkness.comp.waw.pl> User-Agent: Mutt/1.4.1i X-Virus-Scanned: by amavisd-new X-Spam-Status: No, hits=0.0 required=8.0 tests=none autolearn=no version=2.63 X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on odin.ac.hmc.edu cc: arch@FreeBSD.org cc: scottl@FreeBSD.org Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 19:04:05 -0000 --SUOF0GtieIMvvwua Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jan 05, 2005 at 11:20:56AM +0100, Pawel Jakub Dawidek wrote: > On Tue, Jan 04, 2005 at 07:04:26PM -0800, Brooks Davis wrote: > +> I'd be inclined to use the somewhat gross fix in PR 74567 in RELENG_5 > +> and do it right in HEAD. [...] >=20 > It allocates memory and we don't need it. > Proposed fix is actually for RELENG_5. We can also do some magic inside > du(1) to split 64bit value between two fields (fts_number/fts_pointer) > when needed (on 32bit archs), but it would be really hackish. I missed the fact this this was for RELENG_5 only, this seems like a good solution there. > +> [...] bde suggested changing fts_num to intmax_t. >=20 > The only issue here is that we break ABI if intmax_t will be bumped to > 128bits in gcc, but we probably will have much bigger problems then:) > I like this idea. We'd have to bump nearly all library revs if we changed the the size of intmax_t since we'd break printing of 64-bit numbers. :-) -- Brooks --=20 Any statement of the form "X is the one, true Y" is FALSE. PGP fingerprint 655D 519C 26A7 82E7 2529 9BF0 5D8E 8BE9 F238 1AD4 --SUOF0GtieIMvvwua Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQFB3DqCXY6L6fI4GtQRAua9AJ9XXp6JAGn2J+5cSubpY/pNBKP8LQCcCOae wsuORnnHA0USZc9T9QqJ9nA= =Uytx -----END PGP SIGNATURE----- --SUOF0GtieIMvvwua-- From owner-freebsd-arch@FreeBSD.ORG Wed Jan 5 23:12:47 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AE78516A4CE; Wed, 5 Jan 2005 23:12:47 +0000 (GMT) Received: from mail.vicor-nb.com (bigwoop.vicor-nb.com [208.206.78.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 934B343D1F; Wed, 5 Jan 2005 23:12:47 +0000 (GMT) (envelope-from julian@elischer.org) Received: from elischer.org (julian.vicor-nb.com [208.206.78.97]) by mail.vicor-nb.com (Postfix) with ESMTP id 7A3F27A403; Wed, 5 Jan 2005 15:12:47 -0800 (PST) Message-ID: <41DC746F.4090409@elischer.org> Date: Wed, 05 Jan 2005 15:12:47 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.3.1) Gecko/20030516 X-Accept-Language: en, hu MIME-Version: 1.0 To: "M. Warner Losh" References: <41DB2B24.6050005@elischer.org> <20050104.203502.85411551.imp@bsdimp.com> <41DB8FD3.3030101@elischer.org> <20050105.093508.102576805.imp@bsdimp.com> In-Reply-To: <20050105.093508.102576805.imp@bsdimp.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: arch@freebsd.org cc: scottl@freebsd.org cc: pjd@freebsd.org Subject: Re: BigDisk project: du(1) 64bit clean. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jan 2005 23:12:47 -0000 M. Warner Losh wrote: >In message: <41DB8FD3.3030101@elischer.org> > Julian Elischer writes: >: > I think that a big amount of this could be reduced by using simple >: > arrays rather than lists which are more memory efficient... >: >: but that just puts the problem a bit further away. > >We have to do it. If you don't have a file system to store a file, >then you cna't do this at all. When you are fscking /, you don't have >a file system. And if you have finished, nothing is yet mounted r/w. > I'm suggesting two version.. one tha tuses RAM and oen that uses files.. the filesystem that the files are written to would be done first with the RAM version.. We already do that here, though for different reasons. > >Warner > >