From owner-freebsd-questions Wed Mar 13 3:35:24 2002 Delivered-To: freebsd-questions@freebsd.org Received: from famine.e-raist.com (famine.e-raist.com [65.100.40.90]) by hub.freebsd.org (Postfix) with ESMTP id 1B84037B416 for ; Wed, 13 Mar 2002 03:35:16 -0800 (PST) Received: from e-raist.com (war.e-raist.com [65.100.40.89]) (authenticated bits=0) by famine.e-raist.com (8.12.2/8.12.2) with ESMTP id g2DBZLNb059190 for ; Wed, 13 Mar 2002 03:35:21 -0800 (PST) Message-ID: <3C8F3F2F.D6B46631@e-raist.com> Date: Wed, 13 Mar 2002 03:59:43 -0800 From: Raistlin Majere X-Mailer: Mozilla 4.79 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: freebsd-questions@freebsd.org Subject: SOLVED: slow perl string concats, malloc, and other performance issues Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 7bit Sender: owner-freebsd-questions@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG The plague of my slow perl performance has been removed from my life and FreeBSD once again reigns in speed. I ran the following perl script on a linux system and on a freebsd system both running identical hardware and perl 5.6.1 to find that linux was able to execute the script in 9 seconds and freebsd was taking almost 56 seconds. #!/usr/bin/perl $result =""; $begin = time; for ($i = 0; $i < 1000000;) { $i++; #comment the following line to get freebsd perl to run as fast as linux perl $result .= "$i\n"; } $duration = time - $begin; print "duration = $duration.\n"; I called in my genius friend and mentor Marc Frajola to assist me in debugging this odd slowness and he discovered through profiling (recompiling perl with the -pg option to the compiler AND to the linker) where the majority of the clock cycles were going. We then ran the script in the background and fired up a vmstat session to find the system was thrashing. procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad0 md0 in sy cs us sy id 1 0 0 189240 62440 71 0 0 0 73 1 0 0 235 1788 76 1 1 98 1 0 0 189980 62072 11965 0 0 0 11868 0 0 0 1145 1968 262 82 18 0 1 0 0 190676 62332 12280 0 0 0 12342 0 0 0 1149 1845 241 82 18 0 1 0 0 191380 61400 12296 0 0 0 12060 0 2 0 1150 1816 241 79 21 0 1 0 0 189036 63744 12277 0 0 0 12860 0 0 0 1148 1949 260 81 19 0 1 0 0 191608 60792 12999 0 0 0 12258 0 0 0 1147 1820 247 78 22 0 The enormous amount of page faults led us to look at the memory allocation parts of the perl program and we identified that malloc was the culprit. Malloc wasn't able to keep up with the gigantic string which kept growing and growing through the for loop. We did a "make clean" in the perl source directory and did a new ./Configure, this time we told it to use its own malloc, (-Dusemymalloc) ran "make; make test; make install;" and found the above script which took 56 seconds now took 5 seconds. (4 seconds faster than the linux system!) The stock build of perl from the port tree does NOT implement this important change. You have to build from source and tell the Configure script to use its own malloc. The default when running ./Configure is also NOT to use its own malloc, so you have to be looking for that question in the mess of others. (or edit the config.sh file) I -=highly=- recommend recompiling your version of perl today. I believe you'll notice a significant performance increase. Special Thanks to Marc for his wonderful help! -- Raistlin Alexander Majere --------------------------------------------------------------- E R A I S T S O F T W A R E Raistlin Majere http://www.e-raist.com raistlin@e-raist.com * Knowledge for Hire * PGPFP: 1ED8 AF84 9E90 3CFC E4A4 8E69 728D D193 00CF 70FF --------------------------------------------------------------- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message