From owner-freebsd-hubs Sat Mar 16 05:00:59 1996 Return-Path: owner-hubs Received: (from root@localhost) by freefall.freebsd.org (8.7.3/8.7.3) id FAA01791 for hubs-outgoing; Sat, 16 Mar 1996 05:00:59 -0800 (PST) Received: from mail.rwth-aachen.de (mail.RWTH-Aachen.DE [137.226.144.9]) by freefall.freebsd.org (8.7.3/8.7.3) with ESMTP id FAA01785 for ; Sat, 16 Mar 1996 05:00:54 -0800 (PST) Received: from gilberto.physik.rwth-aachen.de (gilberto.physik.rwth-aachen.de) by mail.rwth-aachen.de (PMDF V5.0-4 #13110) id <01I2EQGVQOV40016OQ@mail.rwth-aachen.de>; Sat, 16 Mar 1996 14:04:05 +0100 Received: (from kuku@localhost) by gilberto.physik.rwth-aachen.de (8.6.11/8.6.9) id OAA06397; Sat, 16 Mar 1996 14:07:17 +0100 Date: Sat, 16 Mar 1996 14:07:17 +0100 From: "Christoph P. Kukulies" Subject: mirror and perl To: de-bsd-hubs@blues.physik.rwth-aachen.de, freebsd-hubs@freebsd.org Message-id: <199603161307.OAA06397@gilberto.physik.rwth-aachen.de> Content-transfer-encoding: 7BIT Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk My freebsd.org mirroring is still very flaky using mirror.2.3 and perl5.001. Once, when trying to mirror larger trees, I got an Out of Memory! message in my mirror log file and this happened during scanning the remote site. I'm now running the mirror script with the -F option - btw there is a bug in the mirror script, bigtmp is mixed up with bigtemp. I have subdivided the scan now into scanning smaller sub trees but there still seems to be a memory leak or something eating abnormally much memory. I'm using the -F option on mirror to allows the use of .db files but still I get sometimes pieces in the wrong places. E.g., I scanned /pub/FreeBSD/distfiles/ but the subdirs of distfiles landed in /pub/FreeBSD/ (!). Anyone knowing a stable combination of mirror (version/pl) and perl version/pl ? --Chris Christoph P. U. Kukulies kuku@gil.physik.rwth-aachen.de From owner-freebsd-hubs Sat Mar 16 05:12:39 1996 Return-Path: owner-hubs Received: (from root@localhost) by freefall.freebsd.org (8.7.3/8.7.3) id FAA02298 for hubs-outgoing; Sat, 16 Mar 1996 05:12:39 -0800 (PST) Received: from grumble.grondar.za (root@grumble.grondar.za [196.7.18.130]) by freefall.freebsd.org (8.7.3/8.7.3) with ESMTP id FAA02291 for ; Sat, 16 Mar 1996 05:12:24 -0800 (PST) Received: from grumble.grondar.za (mark@localhost [127.0.0.1]) by grumble.grondar.za (8.7.4/8.7.3) with ESMTP id PAA22977; Sat, 16 Mar 1996 15:11:05 +0200 (SAT) Message-Id: <199603161311.PAA22977@grumble.grondar.za> To: "Christoph P. Kukulies" cc: de-bsd-hubs@blues.physik.rwth-aachen.de, freebsd-hubs@freebsd.org Subject: Re: mirror and perl Date: Sat, 16 Mar 1996 15:11:04 +0200 From: Mark Murray Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk "Christoph P. Kukulies" wrote: > > My freebsd.org mirroring is still very flaky using mirror.2.3 > and perl5.001. > Once, when trying to mirror larger trees, I got an Out of Memory! > message in my mirror log file and this happened during scanning the > remote site. I'm now running the mirror script with the -F option - > btw there is a bug in the mirror script, bigtmp is mixed up with > bigtemp. It sounds like maybe you have either 1) a bug in mirror (I vaguely remember something about memory leaks here. I know it is only scripts, but this memory is quite persistent for reasons I cant quite fathom) 2) non-PHKmalloc linked in. > I have subdivided the scan now into scanning smaller sub trees > but there still seems to be a memory leak or something eating > abnormally much memory. I'm using the -F option on mirror > to allows the use of .db files but still I get sometimes > pieces in the wrong places. E.g., I scanned /pub/FreeBSD/distfiles/ > but the subdirs of distfiles landed in /pub/FreeBSD/ (!). > > Anyone knowing a stable combination of mirror (version/pl) and perl > version/pl ? I Have _just_ submitted perl5.002. Please try that... M -- Mark Murray 46 Harvey Rd, Claremont, Cape Town 7700, South Africa +27 21 61-3768 GMT+0200 Finger mark@grondar.za for PGP key From owner-freebsd-hubs Sat Mar 16 05:23:02 1996 Return-Path: owner-hubs Received: (from root@localhost) by freefall.freebsd.org (8.7.3/8.7.3) id FAA02609 for hubs-outgoing; Sat, 16 Mar 1996 05:23:02 -0800 (PST) Received: from epochal (epochal.doc.ic.ac.uk [146.169.19.14]) by freefall.freebsd.org (8.7.3/8.7.3) with SMTP id FAA02604 for ; Sat, 16 Mar 1996 05:23:00 -0800 (PST) Received: from triumph.doc.ic.ac.uk [146.169.24.1] by epochal with smtp (Exim 0.41 #1) id E0txvqY-0006Ld-00; Sat, 16 Mar 1996 13:16:39 +0000 Message-ID: <314AC042.1E713C5A@doc.ic.ac.uk> Date: Sat, 16 Mar 1996 13:21:06 +0000 From: Lee McLoughlin Organization: IC-Parc, Imperial College, London. U.K. X-Mailer: Mozilla 2.0 (X11; I; SunOS 4.1.3_U1 sun4m) MIME-Version: 1.0 To: Mark Murray CC: "Christoph P. Kukulies" , de-bsd-hubs@blues.physik.rwth-aachen.de, freebsd-hubs@freebsd.org Subject: Re: mirror and perl References: <199603161311.PAA22977@grumble.grondar.za> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Mark Murray wrote: > > "Christoph P. Kukulies" wrote: > > > > My freebsd.org mirroring is still very flaky using mirror.2.3 > > and perl5.001. > > Once, when trying to mirror larger trees, I got an Out of Memory! > > message in my mirror log file and this happened during scanning the > > remote site. I'm now running the mirror script with the -F option - > > btw there is a bug in the mirror script, bigtmp is mixed up with > > bigtemp. Please try again with mirror-2.8 (sunsite.doc.ic.ac.uk packages/mirror/mirror-2.8.tar.gz) -- Lee McLoughlin. Phone: +44 171 594 8388 IC-Parc, Imperial College, Fax: +44 171 594 8449 South Kensington, London. SW7 2BZ. UK. Email: L.McLoughlin@doc.ic.ac.uk From owner-freebsd-hubs Sat Mar 16 05:51:07 1996 Return-Path: owner-hubs Received: (from root@localhost) by freefall.freebsd.org (8.7.3/8.7.3) id FAA03627 for hubs-outgoing; Sat, 16 Mar 1996 05:51:07 -0800 (PST) Received: from mail.rwth-aachen.de (mail.RWTH-Aachen.DE [137.226.144.9]) by freefall.freebsd.org (8.7.3/8.7.3) with ESMTP id FAA03622 for ; Sat, 16 Mar 1996 05:51:03 -0800 (PST) Received: from gilberto.physik.rwth-aachen.de (gilberto.physik.rwth-aachen.de) by mail.rwth-aachen.de (PMDF V5.0-4 #13110) id <01I2ES6I7RI80015RB@mail.rwth-aachen.de>; Sat, 16 Mar 1996 14:53:00 +0100 Received: (from kuku@localhost) by gilberto.physik.rwth-aachen.de (8.6.11/8.6.9) id OAA06516; Sat, 16 Mar 1996 14:56:11 +0100 Date: Sat, 16 Mar 1996 14:56:11 +0100 (MET) From: "Christoph P. Kukulies" Subject: Re: mirror and perl In-reply-to: <314AC042.1E713C5A@doc.ic.ac.uk> To: lmjm@doc.ic.ac.uk (Lee McLoughlin) Cc: mark@grondar.za, kuku@gilberto.physik.rwth-aachen.de, de-bsd-hubs@blues.physik.rwth-aachen.de, freebsd-hubs@freebsd.org Reply-to: Christoph Kukulies Message-id: <199603161356.OAA06516@gilberto.physik.rwth-aachen.de> MIME-version: 1.0 X-Mailer: ELM [version 2.4 PL25 ME8b] Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: 7bit Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > > Mark Murray wrote: > > > > "Christoph P. Kukulies" wrote: > > > > > > My freebsd.org mirroring is still very flaky using mirror.2.3 > > > and perl5.001. > > > Once, when trying to mirror larger trees, I got an Out of Memory! > > > message in my mirror log file and this happened during scanning the > > > remote site. I'm now running the mirror script with the -F option - > > > btw there is a bug in the mirror script, bigtmp is mixed up with > > > bigtemp. > > Please try again with mirror-2.8 (sunsite.doc.ic.ac.uk > packages/mirror/mirror-2.8.tar.gz) Sorry, I *am* using mirror 2.8 (2.3 was a typo on my side). > > > > -- > Lee McLoughlin. Phone: +44 171 594 8388 > IC-Parc, Imperial College, Fax: +44 171 594 8449 > South Kensington, London. SW7 2BZ. UK. Email: L.McLoughlin@doc.ic.ac.uk > --Chris Christoph P. U. Kukulies kuku@gil.physik.rwth-aachen.de From owner-freebsd-hubs Sat Mar 16 05:53:58 1996 Return-Path: owner-hubs Received: (from root@localhost) by freefall.freebsd.org (8.7.3/8.7.3) id FAA03733 for hubs-outgoing; Sat, 16 Mar 1996 05:53:58 -0800 (PST) Received: from epochal (epochal.doc.ic.ac.uk [146.169.19.14]) by freefall.freebsd.org (8.7.3/8.7.3) with SMTP id FAA03728 for ; Sat, 16 Mar 1996 05:53:54 -0800 (PST) Received: from triumph.doc.ic.ac.uk [146.169.24.1] by epochal with smtp (Exim 0.41 #1) id E0txwLv-0006M4-00; Sat, 16 Mar 1996 13:49:03 +0000 Message-ID: <314AC7DB.7E364E8B@doc.ic.ac.uk> Date: Sat, 16 Mar 1996 13:53:31 +0000 From: Lee McLoughlin Organization: IC-Parc, Imperial College, London. U.K. X-Mailer: Mozilla 2.0 (X11; I; SunOS 4.1.3_U1 sun4m) MIME-Version: 1.0 To: Christoph Kukulies CC: mark@grondar.za, de-bsd-hubs@blues.physik.rwth-aachen.de, freebsd-hubs@freebsd.org Subject: Re: mirror and perl References: <199603161356.OAA06516@gilberto.physik.rwth-aachen.de> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Christoph P. Kukulies wrote: > > > > My freebsd.org mirroring is still very flaky using mirror.2.3 > > > > and perl5.001. > > > > Once, when trying to mirror larger trees, I got an Out of Memory! > > > > message in my mirror log file and this happened during scanning the > > > > remote site. I'm now running the mirror script with the -F option - > > > > btw there is a bug in the mirror script, bigtmp is mixed up with > > > > bigtemp. > > > > Please try again with mirror-2.8 (sunsite.doc.ic.ac.uk > > packages/mirror/mirror-2.8.tar.gz) > > Sorry, I *am* using mirror 2.8 (2.3 was a typo on my side). Oh dear! I've found mirror-2.8 + perl5.001m to be pretty good. I'm working on mirror-2.9 which can do things a different way, to over come this kind of problem. Touch wood it will be out in a couple of weeks. -- Lee McLoughlin. Phone: +44 171 594 8388 IC-Parc, Imperial College, Fax: +44 171 594 8449 South Kensington, London. SW7 2BZ. UK. Email: L.McLoughlin@doc.ic.ac.uk From owner-freebsd-hubs Sat Mar 16 05:55:21 1996 Return-Path: owner-hubs Received: (from root@localhost) by freefall.freebsd.org (8.7.3/8.7.3) id FAA03769 for hubs-outgoing; Sat, 16 Mar 1996 05:55:21 -0800 (PST) Received: from mail.rwth-aachen.de (mail.RWTH-Aachen.DE [137.226.144.9]) by freefall.freebsd.org (8.7.3/8.7.3) with ESMTP id FAA03759 for ; Sat, 16 Mar 1996 05:55:15 -0800 (PST) Received: from gilberto.physik.rwth-aachen.de (gilberto.physik.rwth-aachen.de) by mail.rwth-aachen.de (PMDF V5.0-4 #13110) id <01I2ESCWAQ8G0016Y3@mail.rwth-aachen.de>; Sat, 16 Mar 1996 14:58:09 +0100 Received: (from kuku@localhost) by gilberto.physik.rwth-aachen.de (8.6.11/8.6.9) id PAA06549; Sat, 16 Mar 1996 15:01:26 +0100 Date: Sat, 16 Mar 1996 15:01:26 +0100 (MET) From: "Christoph P. Kukulies" Subject: Re: mirror and perl In-reply-to: <199603161311.PAA22977@grumble.grondar.za> To: mark@grondar.za (Mark Murray) Cc: kuku@gilberto.physik.rwth-aachen.de, de-bsd-hubs@blues.physik.rwth-aachen.de, freebsd-hubs@freebsd.org Reply-to: Christoph Kukulies Message-id: <199603161401.PAA06549@gilberto.physik.rwth-aachen.de> MIME-version: 1.0 X-Mailer: ELM [version 2.4 PL25 ME8b] Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 8bit Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > > > "Christoph P. Kukulies" wrote: > > > > My freebsd.org mirroring is still very flaky using mirror.2.3 Correction, 2.8, of course :) > > and perl5.001. > > Once, when trying to mirror larger trees, I got an Out of Memory! > > message in my mirror log file and this happened during scanning the > > remote site. I'm now running the mirror script with the -F option - > > btw there is a bug in the mirror script, bigtmp is mixed up with > > bigtemp. > > It sounds like maybe you have either > 1) a bug in mirror (I vaguely remember something about memory leaks > here. I know it is only scripts, but this memory is quite persistent > for reasons I cant quite fathom) > 2) non-PHKmalloc linked in. With a stock 2.1 system I would believe, phk malloc is enabled, isn't it? Hhhmm, thinking about it, my perl5.001 might stem from pre 2.1 times though otoh phkmalloc goes into libc.so.2.2, let me see... blues> cd /usr/local/bin blues> ldd perl5.001 perl5.001: -lm.2 => /usr/lib/libm.so.2.0 (0x8073000) -lc.2 => /usr/lib/libc.so.2.2 (0x808a000) -lcrypt.2 => /usr/lib/libcrypt.so.2.0 (0x80f7000) blues> ls -l /usr/lib/libc.so* -rw-rw-r-- 1 root bin 384333 Aug 7 1995 /usr/lib/libc.so.1.0 -r--r--r-- 1 bin bin 403106 Jul 3 1994 /usr/lib/libc.so.1.1 -r--r--r-- 1 bin bin 430501 Aug 9 1995 /usr/lib/libc.so.2.1 -r--r--r-- 1 bīn bin 463947 Dec 13 11:19 /usr/lib/libc.so.2.2 blues> > > > I have subdivided the scan now into scanning smaller sub trees > > but there still seems to be a memory leak or something eating > > abnormally much memory. I'm using the -F option on mirror > > to allows the use of .db files but still I get sometimes > > pieces in the wrong places. E.g., I scanned /pub/FreeBSD/distfiles/ > > but the subdirs of distfiles landed in /pub/FreeBSD/ (!). > > > > Anyone knowing a stable combination of mirror (version/pl) and perl > > version/pl ? > > I Have _just_ submitted perl5.002. Please try that... > > M > -- > Mark Murray > 46 Harvey Rd, Claremont, Cape Town 7700, South Africa > +27 21 61-3768 GMT+0200 > Finger mark@grondar.za for PGP key > --Chris Christoph P. U. Kukulies kuku@gil.physik.rwth-aachen.de From owner-freebsd-hubs Sat Mar 16 06:18:07 1996 Return-Path: owner-hubs Received: (from root@localhost) by freefall.freebsd.org (8.7.3/8.7.3) id GAA04978 for hubs-outgoing; Sat, 16 Mar 1996 06:18:07 -0800 (PST) Received: from grumble.grondar.za (root@grumble.grondar.za [196.7.18.130]) by freefall.freebsd.org (8.7.3/8.7.3) with ESMTP id GAA04968 for ; Sat, 16 Mar 1996 06:17:54 -0800 (PST) Received: from grumble.grondar.za (mark@localhost [127.0.0.1]) by grumble.grondar.za (8.7.5/8.7.3) with ESMTP id QAA00672; Sat, 16 Mar 1996 16:16:02 +0200 (SAT) Message-Id: <199603161416.QAA00672@grumble.grondar.za> To: Christoph Kukulies cc: mark@grondar.za (Mark Murray), de-bsd-hubs@blues.physik.rwth-aachen.de, freebsd-hubs@freebsd.org Subject: Re: mirror and perl Date: Sat, 16 Mar 1996 16:16:01 +0200 From: Mark Murray Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk "Christoph P. Kukulies" wrote: > > 2) non-PHKmalloc linked in. > > With a stock 2.1 system I would believe, phk malloc is enabled, isn't it? > Hhhmm, thinking about it, my perl5.001 might stem from pre 2.1 times > though otoh phkmalloc goes into libc.so.2.2, let me see... Er, no. PHKmalloc is a 2.2 thing. I cannot remember dates, but it may be worth a recompile of perl5 on your current system to see if this goes away. > blues> cd /usr/local/bin > blues> ldd perl5.001 > perl5.001: > -lm.2 => /usr/lib/libm.so.2.0 (0x8073000) > -lc.2 => /usr/lib/libc.so.2.2 (0x808a000) > -lcrypt.2 => /usr/lib/libcrypt.so.2.0 (0x80f7000) > blues> ls -l /usr/lib/libc.so* > -rw-rw-r-- 1 root bin 384333 Aug 7 1995 /usr/lib/libc.so.1.0 > -r--r--r-- 1 bin bin 403106 Jul 3 1994 /usr/lib/libc.so.1.1 > -r--r--r-- 1 bin bin 430501 Aug 9 1995 /usr/lib/libc.so.2.1 > -r--r--r-- 1 bīn bin 463947 Dec 13 11:19 /usr/lib/libc.so.2.2 ^^^^^^ Dates, someone? When did PHKmalloc go in? M -- Mark Murray 46 Harvey Rd, Claremont, Cape Town 7700, South Africa +27 21 61-3768 GMT+0200 Finger mark@grondar.za for PGP key From owner-freebsd-hubs Sat Mar 16 06:18:33 1996 Return-Path: owner-hubs Received: (from root@localhost) by freefall.freebsd.org (8.7.3/8.7.3) id GAA05013 for hubs-outgoing; Sat, 16 Mar 1996 06:18:33 -0800 (PST) Received: from time.cdrom.com (time.cdrom.com [204.216.27.226]) by freefall.freebsd.org (8.7.3/8.7.3) with ESMTP id GAA05008 for ; Sat, 16 Mar 1996 06:18:32 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by time.cdrom.com (8.7.5/8.6.9) with SMTP id GAA01136; Sat, 16 Mar 1996 06:15:44 -0800 (PST) To: Christoph Kukulies cc: mark@grondar.za (Mark Murray), de-bsd-hubs@blues.physik.rwth-aachen.de, freebsd-hubs@freebsd.org Subject: Re: mirror and perl In-reply-to: Your message of "Sat, 16 Mar 1996 15:01:26 +0100." <199603161401.PAA06549@gilberto.physik.rwth-aachen.de> Date: Sat, 16 Mar 1996 06:15:44 -0800 Message-ID: <1134.826985744@time.cdrom.com> From: "Jordan K. Hubbard" Sender: owner-hubs@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > With a stock 2.1 system I would believe, phk malloc is enabled, isn't it? Nope! That didn't come along until later. It's not enabled in any version of 2.1 (not even -stable). Jordan From owner-freebsd-hubs Sat Mar 16 22:41:39 1996 Return-Path: owner-hubs Received: (from root@localhost) by freefall.freebsd.org (8.7.3/8.7.3) id WAA25794 for hubs-outgoing; Sat, 16 Mar 1996 22:41:39 -0800 (PST) Received: from ibp.ibp.fr (ibp.ibp.fr [132.227.60.30]) by freefall.freebsd.org (8.7.3/8.7.3) with ESMTP id WAA25784 for ; Sat, 16 Mar 1996 22:41:31 -0800 (PST) Received: from blaise.ibp.fr (blaise.ibp.fr [132.227.60.1]) by ibp.ibp.fr (8.6.12/jtpda-5.0) with ESMTP id HAA12678 ; Sun, 17 Mar 1996 07:41:27 +0100 Received: from (uucp@localhost) by blaise.ibp.fr (8.6.12/jtpda-5.0) with UUCP id HAA04943 ; Sun, 17 Mar 1996 07:41:26 +0100 Received: (from roberto@localhost) by keltia.freenix.fr (8.7.4/keltia-uucp-2.7) id GAA08811; Sun, 17 Mar 1996 06:13:25 +0100 (MET) From: Ollivier Robert Message-Id: <199603170513.GAA08811@keltia.freenix.fr> Subject: Re: mirror and perl To: mark@grondar.za (Mark Murray) Date: Sun, 17 Mar 1996 06:13:24 +0100 (MET) Cc: kuku@gilberto.physik.rwth-aachen.de, mark@grondar.za, de-bsd-hubs@blues.physik.rwth-aachen.de, freebsd-hubs@FreeBSD.org In-Reply-To: <199603161416.QAA00672@grumble.grondar.za> from Mark Murray at "Mar 16, 96 04:16:01 pm" X-Operating-System: FreeBSD 2.2-CURRENT ctm#1762 X-Mailer: ELM [version 2.4ME+ PL11 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-hubs@FreeBSD.org X-Loop: FreeBSD.org Precedence: bulk It seems that Mark Murray said: > > Er, no. PHKmalloc is a 2.2 thing. I cannot remember dates, but it may be > worth a recompile of perl5 on your current system to see if this goes away. Probably but ensure that when you configure Perl not to use the Perl's malloc (which derived from the BSD one if I remember well). > > Dates, someone? When did PHKmalloc go in? >From the CVS log, the first commit is the following: phk 95/09/16 02:28:15 Modified: lib/libc/stdlib Makefile.inc malloc.3 malloc.c Removed: lib/libc/stdlib free.3 realloc.3 Log: ``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details. It was enhanced shortly after. phk 95/10/08 11:44:21 Modified: lib/libc/stdlib malloc.3 malloc.c Log: phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: -- Ollivier ROBERT -=- The daemon is FREE! -=- roberto@keltia.frmug.fr.net FreeBSD keltia.freenix.fr 2.2-CURRENT #6: Mon Mar 11 20:18:10 MET 1996