From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 28 02:51:35 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65F671065673 for ; Sun, 28 Dec 2008 02:51:35 +0000 (UTC) (envelope-from unixmania@gmail.com) Received: from fk-out-0910.google.com (fk-out-0910.google.com [209.85.128.185]) by mx1.freebsd.org (Postfix) with ESMTP id EBE3F8FC19 for ; Sun, 28 Dec 2008 02:51:29 +0000 (UTC) (envelope-from unixmania@gmail.com) Received: by fk-out-0910.google.com with SMTP id k31so3234902fkk.11 for ; Sat, 27 Dec 2008 18:51:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=aZZwX96jNMoVtnw5YSKBok7Fx0y0IWEf/A15R3eIHmo=; b=Dgbs/25IDgeuWGPm9g6zT4iyUWtdIdhUpNf5hauuknccjcvWoPQ6olN7tczzyXF3u1 GUQ02uD0lU5spwizhvrKFXEsws5VQZge0pdwaPHpDZ/1ddYILU5BWNo3mPBPMlmlAYnr yA7Eb4O5KW7ZMpjAuqmZkrJE8KqrPYynbUc+M= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=crSzJ7hAOuEwC5qCQJW5EBQ6IRf9isGSA5vfhFOXyS9NYkWRCbIgPs9HR1bANrK2+Z rAbz3LiRQn/m9h6lrUK8a5ktbnZiSgCjKuQPv/55UUbhLLIH9BEefHs8MurQkF9V9h7q XNSCWqBynDONSuREsOHKTMHC8CCZOZYC68mg4= Received: by 10.102.218.5 with SMTP id q5mr4400533mug.99.1230432688626; Sat, 27 Dec 2008 18:51:28 -0800 (PST) Received: by 10.103.137.8 with HTTP; Sat, 27 Dec 2008 18:51:28 -0800 (PST) Message-ID: Date: Sun, 28 Dec 2008 00:51:28 -0200 From: "Carlos A. M. dos Santos" To: "lazaax -" In-Reply-To: <4374ff010812261128i4d723deby9aa1ed49eec1926c@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4374ff010812261128i4d723deby9aa1ed49eec1926c@mail.gmail.com> Cc: freebsd-hackers@freebsd.org Subject: Re: hack bios hp compaq v3618la X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Dec 2008 02:51:35 -0000 On Fri, Dec 26, 2008 at 5:28 PM, lazaax - wrote: > hi, people, anyone knows how to hack bios from v3618la or had a bios > hacked, i want to bypass whitelist wireless network, i put a atheros > on mi hp and cant supported, sorry my english is poor, i dont want ro > risk my computer... please help I attempted to replace the wireless card of a similar HP notebook and gave up. I suggest you to use a USB "dongle", instead. This is a cheap and much safer approach. Hacking the BIOS may lead to a broken notebook that can be hard to recover. Also, this kind of question should be sent to freebsd-questions, not to -hackers. -- cd /usr/ports/sysutils/life make clean From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 28 03:18:24 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5EEAB1065674 for ; Sun, 28 Dec 2008 03:18:24 +0000 (UTC) (envelope-from sam@freebsd.org) Received: from ebb.errno.com (ebb.errno.com [69.12.149.25]) by mx1.freebsd.org (Postfix) with ESMTP id 366558FC08 for ; Sun, 28 Dec 2008 03:18:24 +0000 (UTC) (envelope-from sam@freebsd.org) Received: from trouble.errno.com (trouble.errno.com [10.0.0.248]) (authenticated bits=0) by ebb.errno.com (8.13.6/8.12.6) with ESMTP id mBS3IMnp092448 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 27 Dec 2008 19:18:23 -0800 (PST) (envelope-from sam@freebsd.org) Message-ID: <4956EFFE.7070603@freebsd.org> Date: Sat, 27 Dec 2008 19:18:22 -0800 From: Sam Leffler Organization: FreeBSD Project User-Agent: Thunderbird 2.0.0.18 (X11/20081209) MIME-Version: 1.0 To: "Carlos A. M. dos Santos" References: <4374ff010812261128i4d723deby9aa1ed49eec1926c@mail.gmail.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DCC-sonic.net-Metrics: ebb.errno.com; whitelist Cc: freebsd-hackers@freebsd.org, lazaax - Subject: Re: hack bios hp compaq v3618la X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Dec 2008 03:18:24 -0000 Carlos A. M. dos Santos wrote: > On Fri, Dec 26, 2008 at 5:28 PM, lazaax - wrote: > >> hi, people, anyone knows how to hack bios from v3618la or had a bios >> hacked, i want to bypass whitelist wireless network, i put a atheros >> on mi hp and cant supported, sorry my english is poor, i dont want ro >> risk my computer... please help >> > > I attempted to replace the wireless card of a similar HP notebook and > gave up. I suggest you to use a USB "dongle", instead. This is a cheap > and much safer approach. Hacking the BIOS may lead to a broken > notebook that can be hard to recover. > > Also, this kind of question should be sent to freebsd-questions, not > to -hackers. > > The usual way to deal w/ this is to overwrite the pci dev id in the card's eeprom so it matches the bios whitelist. Then map that device id to something ath will attach to. I'm not allowed to give out the recipe for writing the eeprom but it's well-known and easily found with google. Sam From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 28 10:54:04 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7A7A81065670 for ; Sun, 28 Dec 2008 10:54:04 +0000 (UTC) (envelope-from Andre.Albsmeier@siemens.com) Received: from goliath.siemens.de (goliath.siemens.de [192.35.17.28]) by mx1.freebsd.org (Postfix) with ESMTP id 0ED128FC13 for ; Sun, 28 Dec 2008 10:54:03 +0000 (UTC) (envelope-from Andre.Albsmeier@siemens.com) Received: from mail3.siemens.de (localhost [127.0.0.1]) by goliath.siemens.de (8.12.11.20060308/8.12.11) with ESMTP id mBSAs1fU027357; Sun, 28 Dec 2008 11:54:02 +0100 Received: from curry.mchp.siemens.de (curry.mchp.siemens.de [139.25.40.130]) by mail3.siemens.de (8.12.11.20060308/8.12.11) with ESMTP id mBSAs16L027577; Sun, 28 Dec 2008 11:54:01 +0100 Received: (from localhost) by curry.mchp.siemens.de (8.14.3/8.14.3) id mBSAs1Vr082193; Date: Sun, 28 Dec 2008 11:54:01 +0100 From: Andre Albsmeier To: Rui Paulo Message-ID: <20081228105401.GA28657@curry.mchp.siemens.de> References: <20081225095333.GA12699@curry.mchp.siemens.de> <490EB16A-F626-4A40-9716-A02A32F04ED4@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <490EB16A-F626-4A40-9716-A02A32F04ED4@freebsd.org> X-Echelon: X-Advice: Drop that crappy M$-Outlook, I'm tired of your viruses! User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-hackers@freebsd.org, Andre Albsmeier Subject: Re: Writing device drivers: How to access a specific memory area? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Dec 2008 10:54:04 -0000 On Thu, 25-Dec-2008 at 13:57:00 +0000, Rui Paulo wrote: > > On 25 Dec 2008, at 09:53, Andre Albsmeier wrote: > > > Hello all, > > > > I am writing a driver which needs to access memory at a > > specific location. The location depends on what the BIOS > > has configured into the host bridge. For example, my > > current machine uses an Intel 975X chipset and the memory > > location I am interested in has been set to 0xFED14000 and > > is 16KB in size (this is MCHBAR of the 975X memory hub). > > You probably just need to do something like: > > rid = PCI_BAR(number); > res = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &rid, RF_ACTIVE); The problem was, that this memory belongs to acpi0 so I couldn't read it from my driver which has no resources assigned to it. But, I found a way, see below... > > And then, > bus_read_4(res, offset from specified PCI BAR); > > > > > > > I have no idea how to access this space from my driver. > > I have played around with bus_alloc_resource() but this > > only gives me back NULL. > > > > However, a devinfo -r gives me: > > > > nexus0 > > npx0 > > acpi0 > > Interrupt request lines: > > 9 > > I/O ports: > > 0x10-0x1f > > ... > > 0x800-0x87f > > I/O memory addresses: > > 0x0-0x9ffff > > 0xc0000-0xdffff > > 0xe0000-0xfffff > > 0x100000-0x7fffffff > > 0xf0000000-0xf3ffffff > > 0xfec00000-0xfec00fff > > 0xfed13000-0xfed19fff <--- > > 0xfed1c000-0xfed1ffff > > 0xfed20000-0xfed3ffff > > 0xfed50000-0xfed8ffff > > 0xfee00000-0xfee00fff > > 0xffb00000-0xffbfffff > > 0xfff00000-0xffffffff > > cpu0 > > ... > > > > The line marked with <--- shows the range which includes > > the location I am interested in. It is probably assigned > > to the acpi0 device. > > > > How do I proceed from this? Do I have to hack around in > > the ACPI-Code? I don't hope so ;-) > > You'll probably need to create a fake ACPI child driver to access it. > > > Create your identify routine with something like: > > static void mydriver_identify(driver_t *driver, device_t parent) > { > if (device_find_child(parent, "mydriver", -1) == NULL && > mydriver_match(parent)) > device_add_child(parent, "mydriver", -1); > } > > mydriver_match() should check if you were given the acpi0 device. Found something easier: I just do a bus_set_resource() followed by bus_alloc_resource_any(), access my host bridge registers, and free things with bus_release_resource() followed by bus_delete_resource(). > > > > > > > > I only need access to this memory location during the > > probe of my driver to read some configuration data. > > Is this pci configuration space ? If so, pci_read_config (man 9 pci) Unfortunately not. As I wrote, it is a 16k memory window where the 975x maps a part of its config registers. The address of this window can be read by pci_read_config( dev, 0x44, 4 ). Then you can access the bridge's cXdrcY registers through this memory window which I needed to determine the machines RAM layout. Thanks anyway for your help, -Andre From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 29 05:28:27 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F004B106564A for ; Mon, 29 Dec 2008 05:28:27 +0000 (UTC) (envelope-from yuri@rawbw.com) Received: from shell.rawbw.com (shell.rawbw.com [198.144.192.42]) by mx1.freebsd.org (Postfix) with ESMTP id CC38C8FC12 for ; Mon, 29 Dec 2008 05:28:27 +0000 (UTC) (envelope-from yuri@rawbw.com) Received: from eagle.syrec.org (c-24-6-210-197.hsd1.ca.comcast.net [24.6.210.197]) (authenticated bits=0) by shell.rawbw.com (8.13.6/8.13.6) with ESMTP id mBT5DQue018221 for ; Sun, 28 Dec 2008 21:13:26 -0800 (PST) Message-ID: <49585C75.80203@rawbw.com> Date: Sun, 28 Dec 2008 21:13:25 -0800 From: Yuri User-Agent: Thunderbird 2.0.0.18 (X11/20081127) MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: How process size is calculated? Is it always based on the current highest available address in memory space? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: yuri@rawbw.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Dec 2008 05:28:28 -0000 malloc(3) can be controlled by MALLOC_OPTIONS to use mmap-based allocation as opposed to sbrk-based allocation. But allocations/deallocations with mmaps can eventually lead to non-continuously mmapped memory (having some non-mmapped gaps). Are these gaps excluded from the process size or size is always linked to the current highest available address in memory space? Yuri From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 29 10:13:59 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 045371065674 for ; Mon, 29 Dec 2008 10:13:59 +0000 (UTC) (envelope-from laladelausanne@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.156]) by mx1.freebsd.org (Postfix) with ESMTP id 45D118FC14 for ; Mon, 29 Dec 2008 10:13:57 +0000 (UTC) (envelope-from laladelausanne@gmail.com) Received: by fg-out-1718.google.com with SMTP id l26so1540848fgb.35 for ; Mon, 29 Dec 2008 02:13:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:mime-version :subject:date:references:x-mailer; bh=aNSDsAPXYFNrKqJl5g1WZilCL/oKEaO5U1AUG6rOGUo=; b=HSJ3HZXOqyl8qg0xizsJManKoNL7jNnHr3t5FBX/2j+Jn++iy8b3oqv9y1DC37sGkS gjc32g6M1AFNWLN6nEcoJ/4Wd9sP/NAw8ZV1koXNPoZ7ekU5CbQU121Yj0TSxFGkTQ6H 2///vUeKMalSFIzx4+KAW666u5oR4w2CJzSW4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:from:to:in-reply-to:content-type :content-transfer-encoding:mime-version:subject:date:references :x-mailer; b=hHJd8Id6kaAHcGgzADNjKxZsoh7nMcLs1TIY2cZE5SHA4mQ/P3pW4iJQQURifnGeXy QVhvCZ5ucTFp5kDxK8+GOZ4xqL4500ajBelbQDkojZJnHQS/5Sjoz+04qhIXuVPLFkN6 mwF9TE0vCEodr2zAHVr9z61v+Dlr62e2okmVg= Received: by 10.86.4.2 with SMTP id 2mr8026430fgd.4.1230545636910; Mon, 29 Dec 2008 02:13:56 -0800 (PST) Received: from nslpc5.epfl.ch (nslpc5.epfl.ch [128.178.149.20]) by mx.google.com with ESMTPS id 12sm19221739fgg.36.2008.12.29.02.13.55 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 29 Dec 2008 02:13:55 -0800 (PST) Message-Id: From: =?ISO-8859-2?Q?Nikola_Kne=BEevi=E6?= To: freebsd-hackers@freebsd.org In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v930.3) Date: Mon, 29 Dec 2008 11:13:53 +0100 References: <623D1024-5DA0-4252-8DA9-FA86E0407DA8@gmail.com> X-Mailer: Apple Mail (2.930.3) Subject: Re: debugging mbuf allocation/dealocation X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Dec 2008 10:13:59 -0000 > > You can use the KTR(4) facility to trace memory allocations and > deallocations, logging them to memory, disk, etc. Unfortunately > interpreting the data can be fairly tricky, as network leaks tend to > happen over a long period of time, be stored in sockets, etc, but > it's definitely possible and has been done. :-) Processing the > results with a perl script goes a long way, as the allocated/freed > pointers are included, etc. Hi Robert, thanks for the pointers :) I discovered that root cause of the leakage - Click::Packet had a bad destructor, which wasn't free-ing mbufs all the time. Since there are some other forms of leaking in Click, I'll use KTR do discover them. Cheers, Nikola From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 29 16:16:03 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1C3EB106564A for ; Mon, 29 Dec 2008 16:16:03 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from email2.allantgroup.com (email2.emsphone.com [199.67.51.116]) by mx1.freebsd.org (Postfix) with ESMTP id CC5668FC08 for ; Mon, 29 Dec 2008 16:16:02 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (dan-a.emsphone.com [199.67.51.107]) by email2.allantgroup.com (8.14.0/8.14.0) with ESMTP id mBTFjjX2010457 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 29 Dec 2008 09:45:45 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.3/8.14.3) with ESMTP id mBTFjiES059514 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 29 Dec 2008 09:45:44 -0600 (CST) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.3/8.14.3/Submit) id mBTFji9K059513; Mon, 29 Dec 2008 09:45:44 -0600 (CST) (envelope-from dan) Date: Mon, 29 Dec 2008 09:45:44 -0600 From: Dan Nelson To: Yuri Message-ID: <20081229154544.GB21654@dan.emsphone.com> References: <49585C75.80203@rawbw.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <49585C75.80203@rawbw.com> X-OS: FreeBSD 7.1-PRERELEASE User-Agent: Mutt/1.5.18 (2008-05-17) X-Virus-Scanned: ClamAV version 0.94.2, clamav-milter version 0.94.2 on email2.allantgroup.com X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0.2 (email2.allantgroup.com [199.67.51.78]); Mon, 29 Dec 2008 09:45:45 -0600 (CST) X-Scanned-By: MIMEDefang 2.45 Cc: freebsd-hackers@freebsd.org Subject: Re: How process size is calculated? Is it always based on the current highest available address in memory space? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Dec 2008 16:16:03 -0000 In the last episode (Dec 28), Yuri said: > malloc(3) can be controlled by MALLOC_OPTIONS to use mmap-based > allocation as opposed to sbrk-based allocation. But > allocations/deallocations with mmaps can eventually lead to > non-continuously mmapped memory (having some non-mmapped gaps). > > Are these gaps excluded from the process size or size is always > linked to the current highest available address in memory space? It looks like only mapped memory is counted in process size. The test program below shows that the reported size goes down, even if a memory range inbetween two others is unmapped: $ ./mmap Before mmap: UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 1000 48058 62165 0 8 0 1928 824 wait S+ ph 0:00.01 ./mmap mmap 64MB A=0x28280000 UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 1000 48058 62165 0 8 0 67464 824 wait S+ ph 0:00.01 ./mmap mmap 64MB B=0x2c280000 UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 1000 48058 62165 0 8 0 133000 824 wait S+ ph 0:00.01 ./mmap mmap 64MB C=0x30280000 UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 1000 48058 62165 0 8 0 198536 824 wait S+ ph 0:00.01 ./mmap munmap B UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 1000 48058 62165 0 8 0 133000 824 wait S+ ph 0:00.01 ./mmap #include #include #include #include int main(void) { char *cmd; void *a, *b, *c; asprintf(&cmd, "ps axlp %d", getpid()); printf("Before mmap:\n"); system(cmd); a = mmap(NULL, 64 * 1024 * 1024, PROT_READ|PROT_WRITE, MAP_ANON, -1, 0); printf("mmap 64MB A=%p\n", a); system(cmd); b = mmap(NULL, 64 * 1024 * 1024, PROT_READ|PROT_WRITE, MAP_ANON, -1, 0); printf("mmap 64MB B=%p\n", b); system(cmd); c = mmap(NULL, 64 * 1024 * 1024, PROT_READ|PROT_WRITE, MAP_ANON, -1, 0); printf("mmap 64MB C=%p\n", c); system(cmd); printf("munmap B\n"); munmap(b, 64 * 1024 * 1024); system(cmd); return 0; } -- Dan Nelson dnelson@allantgroup.com From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 29 19:47:54 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9BFFB106564A for ; Mon, 29 Dec 2008 19:47:54 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 73EFB8FC13 for ; Mon, 29 Dec 2008 19:47:54 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTP id D9F9B46B17; Mon, 29 Dec 2008 14:47:53 -0500 (EST) Date: Mon, 29 Dec 2008 19:47:53 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Dan Nelson In-Reply-To: <20081229154544.GB21654@dan.emsphone.com> Message-ID: References: <49585C75.80203@rawbw.com> <20081229154544.GB21654@dan.emsphone.com> User-Agent: Alpine 1.10 (BSF 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Yuri , freebsd-hackers@freebsd.org Subject: Re: How process size is calculated? Is it always based on the current highest available address in memory space? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Dec 2008 19:47:54 -0000 On Mon, 29 Dec 2008, Dan Nelson wrote: > In the last episode (Dec 28), Yuri said: >> malloc(3) can be controlled by MALLOC_OPTIONS to use mmap-based allocation >> as opposed to sbrk-based allocation. But allocations/deallocations with >> mmaps can eventually lead to non-continuously mmapped memory (having some >> non-mmapped gaps). >> >> Are these gaps excluded from the process size or size is always linked to >> the current highest available address in memory space? > > It looks like only mapped memory is counted in process size. The test > program below shows that the reported size goes down, even if a memory range > inbetween two others is unmapped: You can use procstat(8), or on older versions of FreeBSD procfs, to explore the layout of process memory, which may also lend some insight to what's going on. Robert N M Watson Computer Laboratory University of Cambridge > > $ ./mmap > Before mmap: > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 1000 48058 62165 0 8 0 1928 824 wait S+ ph 0:00.01 ./mmap > mmap 64MB A=0x28280000 > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 1000 48058 62165 0 8 0 67464 824 wait S+ ph 0:00.01 ./mmap > mmap 64MB B=0x2c280000 > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 1000 48058 62165 0 8 0 133000 824 wait S+ ph 0:00.01 ./mmap > mmap 64MB C=0x30280000 > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 1000 48058 62165 0 8 0 198536 824 wait S+ ph 0:00.01 ./mmap > munmap B > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 1000 48058 62165 0 8 0 133000 824 wait S+ ph 0:00.01 ./mmap > > > #include > #include > #include > #include > > int main(void) > { > char *cmd; > void *a, *b, *c; > asprintf(&cmd, "ps axlp %d", getpid()); > printf("Before mmap:\n"); > system(cmd); > a = mmap(NULL, 64 * 1024 * 1024, PROT_READ|PROT_WRITE, MAP_ANON, -1, 0); > printf("mmap 64MB A=%p\n", a); > system(cmd); > b = mmap(NULL, 64 * 1024 * 1024, PROT_READ|PROT_WRITE, MAP_ANON, -1, 0); > printf("mmap 64MB B=%p\n", b); > system(cmd); > c = mmap(NULL, 64 * 1024 * 1024, PROT_READ|PROT_WRITE, MAP_ANON, -1, 0); > printf("mmap 64MB C=%p\n", c); > system(cmd); > printf("munmap B\n"); > munmap(b, 64 * 1024 * 1024); > system(cmd); > return 0; > } > > > -- > Dan Nelson > dnelson@allantgroup.com > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 29 21:58:30 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74656106568B for ; Mon, 29 Dec 2008 21:58:30 +0000 (UTC) (envelope-from mailnull@mips.inka.de) Received: from mail-in-11.arcor-online.net (mail-in-11.arcor-online.net [151.189.21.51]) by mx1.freebsd.org (Postfix) with ESMTP id F0CA68FC1A for ; Mon, 29 Dec 2008 21:58:29 +0000 (UTC) (envelope-from mailnull@mips.inka.de) Received: from mail-in-14-z2.arcor-online.net (mail-in-14-z2.arcor-online.net [151.189.8.31]) by mail-in-11.arcor-online.net (Postfix) with ESMTP id 8094F2091F7 for ; Mon, 29 Dec 2008 22:25:02 +0100 (CET) Received: from mail-in-16.arcor-online.net (mail-in-16.arcor-online.net [151.189.21.56]) by mail-in-14-z2.arcor-online.net (Postfix) with ESMTP id 4B06510107 for ; Mon, 29 Dec 2008 22:25:00 +0100 (CET) Received: from lorvorc.mips.inka.de (dslb-092-075-216-131.pools.arcor-ip.net [92.75.216.131]) by mail-in-16.arcor-online.net (Postfix) with ESMTP id 9661B236E46 for ; Mon, 29 Dec 2008 22:25:00 +0100 (CET) Received: from lorvorc.mips.inka.de (localhost [127.0.0.1]) by lorvorc.mips.inka.de (8.14.3/8.14.3) with ESMTP id mBTLP0pp017792 for ; Mon, 29 Dec 2008 22:25:00 +0100 (CET) (envelope-from mailnull@lorvorc.mips.inka.de) Received: (from mailnull@localhost) by lorvorc.mips.inka.de (8.14.3/8.14.3/Submit) id mBTLP0Kc017791 for freebsd-hackers@freebsd.org; Mon, 29 Dec 2008 22:25:00 +0100 (CET) (envelope-from mailnull) From: naddy@mips.inka.de (Christian Weisgerber) Date: Mon, 29 Dec 2008 21:25:00 +0000 (UTC) Message-ID: Originator: naddy@mips.inka.de (Christian Weisgerber) To: freebsd-hackers@freebsd.org X-Virus-Scanned: ClamAV 0.94.1/8810/Mon Dec 29 17:40:50 2008 on mail-in-16.arcor-online.net X-Virus-Status: Clean Subject: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Dec 2008 21:58:30 -0000 The archivers/lzo2 port runs a series of regression tests after the actual build. These tests show extremely divergent behavior on different machines. There are two types of machines: Type #1: Running the tests takes roughly the same time as configure and compile did, whether it's 30 seconds on a fast machine or 10 minutes on an old slow one. Type #2: Running the tests takes much, much, MUCH longer. I've tried this across alpha, amd64, i386, and sparc64, partially on FreeBSD, partially on OpenBSD. The operating system doesn't matter and there is no pattern related to endianness or 32/64 bits. You can find machines that are the same architecture (e.g. amd64) and are of similar overall speed (e.g. an Intel Xeon Xeon E5405 and an AMD Phenom 9350e) and one of these machines will be type #1 and the other will be #2 and take _a hundred_ times longer to run the tests. A hundred times. I have never seen anything like this before. On the "slow" machines, the tests also consume a lot of system time. I've seen figures from 20 to 50%. However, ktrace shows nothing out of the ordinary. My best guess at this time is that lzo2 somehow manages to induce crazy cache thrashing on some CPU models. Ideas and explanations welcome. -- Christian "naddy" Weisgerber naddy@mips.inka.de From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 29 23:17:49 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E2EE6106566C for ; Mon, 29 Dec 2008 23:17:49 +0000 (UTC) (envelope-from dimitry@andric.com) Received: from tensor.andric.com (cl-327.ede-01.nl.sixxs.net [IPv6:2001:7b8:2ff:146::2]) by mx1.freebsd.org (Postfix) with ESMTP id A6D068FC13 for ; Mon, 29 Dec 2008 23:17:49 +0000 (UTC) (envelope-from dimitry@andric.com) Received: from [IPv6:2001:7b8:3a7:0:98c5:4bd4:24a2:d38c] (unknown [IPv6:2001:7b8:3a7:0:98c5:4bd4:24a2:d38c]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 5A9D511F859; Tue, 30 Dec 2008 00:17:48 +0100 (CET) Message-ID: <49595A9C.6050809@andric.com> Date: Tue, 30 Dec 2008 00:17:48 +0100 From: Dimitry Andric User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.1b3pre) Gecko/20081229 Shredder/3.0b2pre MIME-Version: 1.0 To: Christian Weisgerber References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Dec 2008 23:17:50 -0000 On 2008-12-29 22:25, Christian Weisgerber wrote: > On the "slow" machines, the tests also consume a lot of system time. > I've seen figures from 20 to 50%. However, ktrace shows nothing > out of the ordinary. What's up with the memory on these machines? Lzo tends to take insane amounts.... From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 29 23:19:13 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3A0E81065675 for ; Mon, 29 Dec 2008 23:19:13 +0000 (UTC) (envelope-from dimitry@andric.com) Received: from tensor.andric.com (cl-327.ede-01.nl.sixxs.net [IPv6:2001:7b8:2ff:146::2]) by mx1.freebsd.org (Postfix) with ESMTP id F05248FC23 for ; Mon, 29 Dec 2008 23:19:12 +0000 (UTC) (envelope-from dimitry@andric.com) Received: from [IPv6:2001:7b8:3a7:0:98c5:4bd4:24a2:d38c] (unknown [IPv6:2001:7b8:3a7:0:98c5:4bd4:24a2:d38c]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 4FFE811F859; Tue, 30 Dec 2008 00:19:12 +0100 (CET) Message-ID: <49595AF2.30101@andric.com> Date: Tue, 30 Dec 2008 00:19:14 +0100 From: Dimitry Andric User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.1b3pre) Gecko/20081229 Shredder/3.0b2pre MIME-Version: 1.0 To: Christian Weisgerber References: <49595A9C.6050809@andric.com> In-Reply-To: <49595A9C.6050809@andric.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Dec 2008 23:19:13 -0000 On 2008-12-30 00:17, Dimitry Andric wrote: > What's up with the memory on these machines? Lzo tends to take insane amounts.... Duh, nevermind... I'm confusing this with lzma. :) Sorry for the noise. From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 29 23:25:07 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE6EB1065700 for ; Mon, 29 Dec 2008 23:25:07 +0000 (UTC) (envelope-from neldredge@math.ucsd.edu) Received: from euclid.ucsd.edu (euclid.ucsd.edu [132.239.145.52]) by mx1.freebsd.org (Postfix) with ESMTP id B5EBE8FC0C for ; Mon, 29 Dec 2008 23:25:07 +0000 (UTC) (envelope-from neldredge@math.ucsd.edu) Received: from zeno.ucsd.edu (zeno.ucsd.edu [132.239.145.22]) by euclid.ucsd.edu (8.11.7p3+Sun/8.11.7) with ESMTP id mBTNP7Z00019; Mon, 29 Dec 2008 15:25:07 -0800 (PST) Received: from localhost (neldredg@localhost) by zeno.ucsd.edu (8.11.7p3+Sun/8.11.7) with ESMTP id mBTNP3b08065; Mon, 29 Dec 2008 15:25:07 -0800 (PST) X-Authentication-Warning: zeno.ucsd.edu: neldredg owned process doing -bs Date: Mon, 29 Dec 2008 15:25:03 -0800 (PST) From: Nate Eldredge X-X-Sender: neldredg@zeno.ucsd.edu To: Christian Weisgerber In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org Subject: Re: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Dec 2008 23:25:08 -0000 On Mon, 29 Dec 2008, Christian Weisgerber wrote: > The archivers/lzo2 port runs a series of regression tests after the > actual build. These tests show extremely divergent behavior on > different machines. There are two types of machines: > > Type #1: > Running the tests takes roughly the same time as configure and > compile did, whether it's 30 seconds on a fast machine or 10 > minutes on an old slow one. > > Type #2: > Running the tests takes much, much, MUCH longer. > > I've tried this across alpha, amd64, i386, and sparc64, partially > on FreeBSD, partially on OpenBSD. The operating system doesn't > matter and there is no pattern related to endianness or 32/64 bits. > > You can find machines that are the same architecture (e.g. amd64) > and are of similar overall speed (e.g. an Intel Xeon Xeon E5405 and > an AMD Phenom 9350e) and one of these machines will be type #1 and > the other will be #2 and take _a hundred_ times longer to run the > tests. A hundred times. > > I have never seen anything like this before. It might be good first to rule out compiler / library differences. First, can you isolate a single lzo command / input combination whose time differs dramatically? This would simplify tests compared to running the whole test suite. (It should be easy because it looks like the test suite prints the time for each test.) It might also simplify things to work on one "fast" and one "slow" machine. Then try copying the lzo binary from the "fast" machine to the "slow" machine (and vice versa) and see if the same test speeds up with the copied binary. If not, try again with the binary statically linked. If still not, it would be good to have a copy of the binary made available, along with more information about the "fast" and "slow" machines (CPU, amount of memory, load on the machine, kernel version, disk, etc). If the copied binary isn't faster than the natively produced one, then it would be good to have information about the compiler options, versions, etc. -- Nate Eldredge neldredge@math.ucsd.edu From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 29 23:57:25 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C910B106564A for ; Mon, 29 Dec 2008 23:57:25 +0000 (UTC) (envelope-from yuri@rawbw.com) Received: from shell.rawbw.com (shell.rawbw.com [198.144.192.42]) by mx1.freebsd.org (Postfix) with ESMTP id 9E9CA8FC16 for ; Mon, 29 Dec 2008 23:57:25 +0000 (UTC) (envelope-from yuri@rawbw.com) Received: from eagle.syrec.org (c-24-6-210-197.hsd1.ca.comcast.net [24.6.210.197]) (authenticated bits=0) by shell.rawbw.com (8.13.6/8.13.6) with ESMTP id mBTNvO6N022007; Mon, 29 Dec 2008 15:57:25 -0800 (PST) Message-ID: <495963E3.8010509@rawbw.com> Date: Mon, 29 Dec 2008 15:57:23 -0800 From: Yuri User-Agent: Thunderbird 2.0.0.18 (X11/20081127) MIME-Version: 1.0 To: Christian Weisgerber References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: yuri@rawbw.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Dec 2008 23:57:26 -0000 Christian Weisgerber wrote: > > > My best guess at this time is that lzo2 somehow manages to induce > crazy cache thrashing on some CPU models. > > Ideas and explanations welcome. > Did you ask the author? He might be the best person to ask. Yuri From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 30 00:04:29 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D56891065672 for ; Tue, 30 Dec 2008 00:04:29 +0000 (UTC) (envelope-from fbsd.hackers@rachie.is-a-geek.net) Received: from mail.rachie.is-a-geek.net (rachie.is-a-geek.net [66.230.99.27]) by mx1.freebsd.org (Postfix) with ESMTP id 8CC148FC1A for ; Tue, 30 Dec 2008 00:04:29 +0000 (UTC) (envelope-from fbsd.hackers@rachie.is-a-geek.net) Received: from localhost (mail.rachie.is-a-geek.net [192.168.2.101]) by mail.rachie.is-a-geek.net (Postfix) with ESMTP id 77E86AFBC02; Mon, 29 Dec 2008 15:04:28 -0900 (AKST) From: Mel To: freebsd-hackers@freebsd.org Date: Mon, 29 Dec 2008 15:04:28 -0900 User-Agent: KMail/1.9.10 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-6" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200812291504.28106.fbsd.hackers@rachie.is-a-geek.net> Cc: Christian Weisgerber Subject: Re: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Dec 2008 00:04:30 -0000 On Monday 29 December 2008 12:25:00 Christian Weisgerber wrote: > On the "slow" machines, the tests also consume a lot of system time. > I've seen figures from 20 to 50%. =A0However, ktrace shows nothing > out of the ordinary. If the program itself doesn't directly cause the system time, do interrupt= =20 rates give any hint as to what does? And to rule out the obvious, you did check swapping? =2D-=20 Mel From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 30 00:16:45 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A4116106566C for ; Tue, 30 Dec 2008 00:16:45 +0000 (UTC) (envelope-from yuri@rawbw.com) Received: from shell.rawbw.com (shell.rawbw.com [198.144.192.42]) by mx1.freebsd.org (Postfix) with ESMTP id 79ABE8FC24 for ; Tue, 30 Dec 2008 00:16:45 +0000 (UTC) (envelope-from yuri@rawbw.com) Received: from eagle.syrec.org (c-24-6-210-197.hsd1.ca.comcast.net [24.6.210.197]) (authenticated bits=0) by shell.rawbw.com (8.13.6/8.13.6) with ESMTP id mBU0Gjn8026224; Mon, 29 Dec 2008 16:16:45 -0800 (PST) Message-ID: <4959686C.7010009@rawbw.com> Date: Mon, 29 Dec 2008 16:16:44 -0800 From: Yuri User-Agent: Thunderbird 2.0.0.18 (X11/20081127) MIME-Version: 1.0 To: Christian Weisgerber References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: yuri@rawbw.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Dec 2008 00:16:45 -0000 Christian Weisgerber wrote: > My best guess at this time is that lzo2 somehow manages to induce > crazy cache thrashing on some CPU models. > > Ideas and explanations welcome Try running single command that is different on different machines under valgrind (callgrind) on these machines and see that at least number of instructions executed is the same. Lzo2 documentation says that there are a lot of algorithms implemented. It might be choosing the algorithm based on the CPU and the choice it's making might be bad. Yuri From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 30 00:34:06 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0DD021065680 for ; Tue, 30 Dec 2008 00:34:06 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-ew0-f21.google.com (mail-ew0-f21.google.com [209.85.219.21]) by mx1.freebsd.org (Postfix) with ESMTP id 64A398FC22 for ; Tue, 30 Dec 2008 00:34:05 +0000 (UTC) (envelope-from artemb@gmail.com) Received: by ewy14 with SMTP id 14so6039068ewy.19 for ; Mon, 29 Dec 2008 16:34:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender :to:subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references :x-google-sender-auth; bh=J1G8RdZ8bnPPzuGJXVn47oJeYmGiRZs0/P9REn+yM2U=; b=mY5A77PE8W/LeZ/4YxgF6zKDEh4q5coQSKfassd/JDdmSlpu9VaUa3jTLfF+Vmv4mW wFGmUVQethztGADYr85bLWr3OwGfXja5sAd3yXGRlnejAJFdtF5A2NPC275/7tsgJwD9 MLJajrbr1YfQmi2s5WIxu9bST2gOsRfG1IWbw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references:x-google-sender-auth; b=dB48/uv7CDhCAsSHEg8ymDMVVvhK5YDl5CV6hmr/MtEell6WaiG5U2hc+G6VE1mtjt s1Ujuo5ZI9F1Nx5sL+9tmf3RMDGZpskSlJtlMvDS8hR6sT2ROjYlcB0KJ+3nVyL6qTi/ y4yRQOf/CNZ3U0rkiz1aXlqWS0Wxt/knZ76ZU= Received: by 10.210.109.10 with SMTP id h10mr16602011ebc.110.1230595908263; Mon, 29 Dec 2008 16:11:48 -0800 (PST) Received: by 10.210.87.12 with HTTP; Mon, 29 Dec 2008 16:11:48 -0800 (PST) Message-ID: Date: Mon, 29 Dec 2008 16:11:48 -0800 From: "Artem Belevich" Sender: artemb@gmail.com To: "Nate Eldredge" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: X-Google-Sender-Auth: bbff1c951aa786a9 Cc: freebsd-hackers@freebsd.org, Christian Weisgerber Subject: Re: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Dec 2008 00:34:06 -0000 I see this performance difference on my boxes. First one has Core2Duo(E5-something), 4GB and runs RELENG_7/i386. lzotest is very fast. Second box is Core2Quad (Q9450), 8GB RAM and runs -current as of about a week ago. lzo2 binary built from ports is *slow*. However, 32-bit binary from the first box runs very fast. The only interesting difference I can see in ktrace is that read and munmap take much much longer in case of 64-bit lzotest. Here are two excerpts from ktrace on the second box: ### 32-bit app - runs fast on both boxes. 59657 lzotest 0.000010 CALL open(0xffffd91b,O_RDONLY,0x1b6) 59657 lzotest 0.000007 NAMI "./src/lzo1_d.ch" 59657 lzotest 0.000012 RET open 3 59657 lzotest 0.000005 CALL fstat(0x3,0xffffd504) 59657 lzotest 0.000007 STRU struct stat {dev=102, ino=544718, mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=2169160, atime=1230595144, stime=1209559909, ctime=1230588212, birthtime=1209559909, size=4563, blksize=4096, blocks=12, flags=0x0 } 59657 lzotest 0.000005 RET fstat 0 59657 lzotest 0.000006 CALL lseek(0x3,0,SEEK_SET,0x1) 59657 lzotest 0.000005 RET lseek 0 59657 lzotest 0.000005 CALL lseek(0x3,0x4000000,SEEK_SET,0) 59657 lzotest 0.000005 RET lseek 67108864/0x4000000 59657 lzotest 0.000006 CALL lseek(0x3,0,SEEK_SET,0) 59657 lzotest 0.000005 RET lseek 0 59657 lzotest 0.000005 CALL mmap(0,0x4000000,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,0xffffffff,0,0) 59657 lzotest 0.000007 RET mmap 673185792/0x28200000 59657 lzotest 0.000006 CALL read(0x3,0x28196000,0x1000) 59657 lzotest 0.000010 GIO fd 3 read 4096 bytes 59657 lzotest 0.000029 RET read 4096/0x1000 59657 lzotest 0.000028 CALL read(0x3,0x28196000,0x1000) 59657 lzotest 0.000010 GIO fd 3 read 467 bytes 59657 lzotest 0.000005 RET read 467/0x1d3 59657 lzotest 0.000010 CALL read(0x3,0x28196000,0x1000) 59657 lzotest 0.000007 GIO fd 3 read 0 bytes 59657 lzotest 0.000006 RET read 0 59657 lzotest 0.000005 CALL close(0x3) 59657 lzotest 0.000010 RET close 0 59657 lzotest 0.000025 CALL getrusage(0,0xffffd60c) 59657 lzotest 0.000006 RET getrusage 0 59657 lzotest 0.000005 CALL getrusage(0,0xffffd628) 59657 lzotest 0.000006 RET getrusage 0 59657 lzotest 0.000005 CALL getrusage(0,0xffffd60c) 59657 lzotest 0.000006 RET getrusage 0 59657 lzotest 0.000064 CALL getrusage(0,0xffffd60c) 59657 lzotest 0.000006 RET getrusage 0 59657 lzotest 0.000005 CALL getrusage(0,0xffffd60c) 59657 lzotest 0.000006 RET getrusage 0 59657 lzotest 0.000029 CALL getrusage(0,0xffffd60c) 59657 lzotest 0.000006 RET getrusage 0 59657 lzotest 0.000012 CALL getrusage(0,0xffffd60c) 59657 lzotest 0.000036 RET getrusage 0 59657 lzotest 0.000010 CALL write(0x1,0x28194000,0x4f) 59657 lzotest 0.000010 GIO fd 1 wrote 79 bytes 59657 lzotest 0.000006 RET write 79/0x4f 59657 lzotest 0.000006 CALL munmap(0x28200000,0x4000000) 59657 lzotest 0.000017 RET munmap 0 ### same file. 64-bit app (slow). Look at read/munmap 59158 lzotest 0.000015 CALL open(0x7fffffffe760,O_RDONLY,0x1b6) 59158 lzotest 0.000014 NAMI "./src/lzo1_d.ch" 59158 lzotest 0.000024 RET open 3 59158 lzotest 0.000011 CALL fstat(0x3,0x7fffffffe2d0) 59158 lzotest 0.000011 STRU struct stat {dev=102, ino=544718, mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=2169160, atime=1230588427, stime=1209559909, ctime=1230588212, birthtime=1209559909, size=4563, blksize=4096, blocks=12, flags=0x0 } 59158 lzotest 0.000007 RET fstat 0 59158 lzotest 0.000015 CALL lseek(0x3,0,SEEK_CUR) 59158 lzotest 0.000007 RET lseek 0 59158 lzotest 0.000006 CALL lseek(0x3,0x4000000,SEEK_SET) 59158 lzotest 0.000007 RET lseek 67108864/0x4000000 59158 lzotest 0.000007 CALL lseek(0x3,0,SEEK_SET) 59158 lzotest 0.000006 RET lseek 0 59158 lzotest 0.000008 CALL mmap(0,0x4000000,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,0xffffffff,0) 59158 lzotest 0.000010 RET mmap 11534336/0x800b00000 59158 lzotest 0.074126 CALL read(0x3,0x800a9e000,0x1000) 59158 lzotest 0.000054 GIO fd 3 read 4096 bytes 59158 lzotest 0.000010 RET read 4096/0x1000 59158 lzotest 0.000007 CALL read(0x3,0x800a9e000,0x1000) 59158 lzotest 0.000012 GIO fd 3 read 467 bytes 59158 lzotest 0.000006 RET read 467/0x1d3 59158 lzotest 0.000007 CALL read(0x3,0x800a9e000,0x1000) 59158 lzotest 0.000009 GIO fd 3 read 0 bytes 59158 lzotest 0.000006 RET read 0 59158 lzotest 0.000008 CALL close(0x3) 59158 lzotest 0.000020 RET close 0 59158 lzotest 0.000029 CALL getrusage(0,0x7fffffffe3d0) 59158 lzotest 0.000010 RET getrusage 0 59158 lzotest 0.000007 CALL getrusage(0,0x7fffffffe3e0) 59158 lzotest 0.000007 RET getrusage 0 59158 lzotest 0.000007 CALL getrusage(0,0x7fffffffe3d0) 59158 lzotest 0.000007 RET getrusage 0 59158 lzotest 0.000069 CALL getrusage(0,0x7fffffffe3d0) 59158 lzotest 0.000007 RET getrusage 0 59158 lzotest 0.000006 CALL getrusage(0,0x7fffffffe3d0) 59158 lzotest 0.000007 RET getrusage 0 59158 lzotest 0.000030 CALL getrusage(0,0x7fffffffe3d0) 59158 lzotest 0.000006 RET getrusage 0 59158 lzotest 0.000014 CALL getrusage(0,0x7fffffffe3d0) 59158 lzotest 0.000006 RET getrusage 0 59158 lzotest 0.000024 CALL write(0x1,0x800a9c000,0x4f) 59158 lzotest 0.000030 GIO fd 1 wrote 79 bytes 59158 lzotest 0.000008 RET write 79/0x4f 59158 lzotest 0.015677 CALL munmap(0x800b00000,0x4000000) 59158 lzotest 0.005430 RET munmap 0 --Artem From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 30 01:09:07 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37283106564A for ; Tue, 30 Dec 2008 01:09:07 +0000 (UTC) (envelope-from mailnull@mips.inka.de) Received: from mail-in-05.arcor-online.net (mail-in-05.arcor-online.net [151.189.21.45]) by mx1.freebsd.org (Postfix) with ESMTP id E24448FC13 for ; Tue, 30 Dec 2008 01:09:06 +0000 (UTC) (envelope-from mailnull@mips.inka.de) Received: from mail-in-01-z2.arcor-online.net (mail-in-01-z2.arcor-online.net [151.189.8.13]) by mail-in-05.arcor-online.net (Postfix) with ESMTP id 394DC183661 for ; Tue, 30 Dec 2008 02:09:05 +0100 (CET) Received: from mail-in-12.arcor-online.net (mail-in-12.arcor-online.net [151.189.21.52]) by mail-in-01-z2.arcor-online.net (Postfix) with ESMTP id 29DA32BF5AC for ; Tue, 30 Dec 2008 02:09:05 +0100 (CET) Received: from lorvorc.mips.inka.de (dslb-092-075-216-131.pools.arcor-ip.net [92.75.216.131]) by mail-in-12.arcor-online.net (Postfix) with ESMTP id F25A38C462 for ; Tue, 30 Dec 2008 02:09:04 +0100 (CET) Received: from lorvorc.mips.inka.de (localhost [127.0.0.1]) by lorvorc.mips.inka.de (8.14.3/8.14.3) with ESMTP id mBU194Hs005661 for ; Tue, 30 Dec 2008 02:09:04 +0100 (CET) (envelope-from mailnull@lorvorc.mips.inka.de) Received: (from mailnull@localhost) by lorvorc.mips.inka.de (8.14.3/8.14.3/Submit) id mBU194Mv005660 for freebsd-hackers@freebsd.org; Tue, 30 Dec 2008 02:09:04 +0100 (CET) (envelope-from mailnull) From: naddy@mips.inka.de (Christian Weisgerber) Date: Tue, 30 Dec 2008 01:09:04 +0000 (UTC) Message-ID: References: <200812291504.28106.fbsd.hackers@rachie.is-a-geek.net> Originator: naddy@mips.inka.de (Christian Weisgerber) To: freebsd-hackers@freebsd.org X-Virus-Scanned: ClamAV 0.94.1/8811/Mon Dec 29 21:02:38 2008 on mail-in-12.arcor-online.net X-Virus-Status: Clean Subject: Re: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Dec 2008 01:09:07 -0000 Mel wrote: > If the program itself doesn't directly cause the system time, do interrupt > rates give any hint as to what does? systat -vmstat shows a conspicuously large number of traps, I think. (I'm short on comparable FreeBSD machines.) > And to rule out the obvious, you did check swapping? No swapping. -- Christian "naddy" Weisgerber naddy@mips.inka.de From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 30 01:20:00 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 260E2106564A for ; Tue, 30 Dec 2008 01:20:00 +0000 (UTC) (envelope-from naddy@mips.inka.de) Received: from mail-in-04.arcor-online.net (mail-in-04.arcor-online.net [151.189.21.44]) by mx1.freebsd.org (Postfix) with ESMTP id D0E4E8FC1C for ; Tue, 30 Dec 2008 01:19:59 +0000 (UTC) (envelope-from naddy@mips.inka.de) Received: from mail-in-07-z2.arcor-online.net (mail-in-07-z2.arcor-online.net [151.189.8.19]) by mail-in-04.arcor-online.net (Postfix) with ESMTP id 4BEC417F445; Tue, 30 Dec 2008 01:47:48 +0100 (CET) Received: from mail-in-07.arcor-online.net (mail-in-07.arcor-online.net [151.189.21.47]) by mail-in-07-z2.arcor-online.net (Postfix) with ESMTP id 3EE512C77C4; Tue, 30 Dec 2008 01:47:48 +0100 (CET) Received: from lorvorc.mips.inka.de (dslb-092-075-216-131.pools.arcor-ip.net [92.75.216.131]) by mail-in-07.arcor-online.net (Postfix) with ESMTP id 0B75B28ABA6; Tue, 30 Dec 2008 01:47:48 +0100 (CET) Received: from lorvorc.mips.inka.de (localhost [127.0.0.1]) by lorvorc.mips.inka.de (8.14.3/8.14.3) with ESMTP id mBU0ll2S073460; Tue, 30 Dec 2008 01:47:47 +0100 (CET) (envelope-from naddy@lorvorc.mips.inka.de) Received: (from naddy@localhost) by lorvorc.mips.inka.de (8.14.3/8.14.3/Submit) id mBU0llPG073459; Tue, 30 Dec 2008 01:47:47 +0100 (CET) (envelope-from naddy) Date: Tue, 30 Dec 2008 01:47:47 +0100 From: Christian Weisgerber To: Nate Eldredge Message-ID: <20081230004747.GA55542@lorvorc.mips.inka.de> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.18 (2008-05-17) X-Virus-Scanned: ClamAV 0.94.1/8811/Mon Dec 29 21:02:38 2008 on mail-in-07.arcor-online.net X-Virus-Status: Clean Cc: freebsd-hackers@freebsd.org Subject: Re: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Dec 2008 01:20:00 -0000 Nate Eldredge: > It might be good first to rule out compiler / library differences. Sure. Let's cut this short: "Slow" Athlon 64 X2 5200+ 2.6 GHz, FreeBSD 8.0-CURRENT amd64 ~60 min Phenom 9350e 2.0 GHz, OpenBSD 4.4-CURRENT amd64 ~80 min UltraSPARC-IIe 500 MHz (Blade 100), OpenBSD 4.4-CURRENT sparc64 10 h++ "Fast" Pentium 4 3.0 GHz, FreeBSD 6.4-RELEASE i386 36 s Xeon E5405 2.0 GHz (PowerEdge 1950), OpenBSD 4.4-CURRENT amd64 47 s Alpha 21164A 500 MHz (AlphaPC164), OpenBSD 4.4-CURRENT alpha 9 min Let me draw your attention to the fact that the two amd64 systems that run different operating systems are both slow, whereas the two amd64 systems that run the same operating system (compiler, libraries) diverge in speed. Oh, and everybody is invited to run $ cd /usr/ports/archivers/lzo2 && make and check for themselves. PS: The Blade 100 is still crunching as I write this... -- Christian "naddy" Weisgerber naddy@mips.inka.de From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 30 02:05:31 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB227106564A for ; Tue, 30 Dec 2008 02:05:31 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (brucec-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:c09::2]) by mx1.freebsd.org (Postfix) with ESMTP id 6ABCC8FC1C for ; Tue, 30 Dec 2008 02:05:31 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id 934C3192DB; Mon, 29 Dec 2008 21:05:29 -0500 (EST) X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on muon X-Spam-Level: X-Spam-Status: No, score=-2.5 required=8.0 tests=AWL,BAYES_00,NO_RELAYS autolearn=ham version=3.2.5 Received: from gluon (unknown [IPv6:2a01:348:10f:0:240:f4ff:fe57:9871]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA; Mon, 29 Dec 2008 21:05:29 -0500 (EST) Date: Tue, 30 Dec 2008 02:05:24 +0000 From: Bruce Cran To: Christian Weisgerber Message-ID: <20081230020524.2563a6ac@gluon> In-Reply-To: <20081230004747.GA55542@lorvorc.mips.inka.de> References: <20081230004747.GA55542@lorvorc.mips.inka.de> X-Mailer: Claws Mail 3.3.1 (GTK+ 2.12.9; i486-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: Nate Eldredge , freebsd-hackers@freebsd.org Subject: Re: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Dec 2008 02:05:31 -0000 On Tue, 30 Dec 2008 01:47:47 +0100 Christian Weisgerber wrote: > Nate Eldredge: > > > It might be good first to rule out compiler / library differences. > > Sure. Let's cut this short: > > "Slow" > Athlon 64 X2 5200+ 2.6 GHz, FreeBSD 8.0-CURRENT amd64 ~60 > min Phenom 9350e 2.0 GHz, OpenBSD 4.4-CURRENT amd64 > ~80 min UltraSPARC-IIe 500 MHz (Blade 100), OpenBSD 4.4-CURRENT > sparc64 10 h++ > > "Fast" > Pentium 4 3.0 GHz, FreeBSD 6.4-RELEASE i386 36 s > Xeon E5405 2.0 GHz (PowerEdge 1950), OpenBSD 4.4-CURRENT amd64 47 s > Alpha 21164A 500 MHz (AlphaPC164), OpenBSD 4.4-CURRENT alpha 9 > min > > Let me draw your attention to the fact that the two amd64 systems > that run different operating systems are both slow, whereas the two > amd64 systems that run the same operating system (compiler, libraries) > diverge in speed. > > > Oh, and everybody is invited to run > > $ cd /usr/ports/archivers/lzo2 && make I'm running 8.0-CURRENT amd64 here on a Turion64 X2 machine. Without malloc debugging (malloc.conf -> aj) 'make test' takes 25s; after removing malloc.conf thus turning on debugging, it takes over 10 minutes. -- Bruce Cran From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 30 02:14:10 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8CA2A106564A for ; Tue, 30 Dec 2008 02:14:10 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from tokyo01.jp.mail.your.org (tokyo01.jp.mail.your.org [204.9.54.5]) by mx1.freebsd.org (Postfix) with ESMTP id 4E98C8FC19 for ; Tue, 30 Dec 2008 02:14:10 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from tokyo01.jp.mail.your.org (localhost.your.org [127.0.0.1]) by tokyo01.jp.mail.your.org (Postfix) with ESMTP id 95E492AD5948; Tue, 30 Dec 2008 01:56:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=dragondata.com; h=cc :message-id:from:to:in-reply-to:content-type :content-transfer-encoding:mime-version:subject:date:references; s=selector1; bh=yZ4F9KJwFCqNU+fITlthGtsIwIE=; b=DH635XT34rd33zq wDYSbkwQtWt7HhmFlpA8Qzk0a3e0WTYFeLWJcPTrIybsidjDorooSzDMSJbYCpKj //5kDEymSU9coEodoQ+/nYfHdhAJrtZe18Ujy9vPp4eLANehYkhripUCplwj3e51 qWinp76YDNmWXJTv5xdyWX5UMXRs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=dragondata.com; h=cc:message-id :from:to:in-reply-to:content-type:content-transfer-encoding :mime-version:subject:date:references; q=dns; s=selector1; b=wjU wDxbJ1kkWo6SQLXUFRgcX9wW3iJtH4KABNi4pKc/wpu+n0PUns7grJjPaCpaKdHz QlDFf9W4xuGkfFERsAk0RFWrp/OkVlwf4aZSSzd+DZSARfPPogceshi3+ldpzn40 7BKqY9eqxcneUiZbCXh8qWS9AM7feOr8uKwn3mpA= Received: from mail.your.org (server3-a.your.org [64.202.112.67]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by tokyo01.jp.mail.your.org (Postfix) with ESMTPS id 6B6D12AD58EF; Tue, 30 Dec 2008 01:56:10 +0000 (UTC) Received: from [IPv6:2002:451f:630b:1::1] (unknown [IPv6:2002:451f:630b:1::1]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.your.org (Postfix) with ESMTPSA id BE532A0A414; Tue, 30 Dec 2008 01:56:56 +0000 (UTC) Message-Id: From: Kevin Day To: Christian Weisgerber In-Reply-To: <20081230004747.GA55542@lorvorc.mips.inka.de> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v930.3) Date: Mon, 29 Dec 2008 19:56:04 -0600 References: <20081230004747.GA55542@lorvorc.mips.inka.de> X-Mailer: Apple Mail (2.930.3) X-Mailman-Approved-At: Tue, 30 Dec 2008 02:28:20 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Dec 2008 02:14:10 -0000 > > > Oh, and everybody is invited to run > > $ cd /usr/ports/archivers/lzo2 && make > > and check for themselves. > > I've used lzo2 quite a bit in the past and never saw this, so I thought I'd try this on a few boxes we have... Output is from "make fetch ; time make" 8-core Opteron 2350 2.0ghz, 64GB RAM, FreeBSD 7.1-PRERELEASE (just before RC1 was tagged), amd64 41.464u 20.671s 1:02.04 100.1% 2430+1556k 0+0io 377pf+0w 4-core Opteron 280 2.4ghz, 4GB RAM, FreeBSD 7.0-RELEASE-p6, amd64 40.907u 18.638s 1:03.08 94.3% 2339+603k 182+91io 681pf+0w Dual Athlon MP 2100+ 1.73ghz, 1GB RAM, FreeBSD 6.3-RELEASE, i386 82.812u 44.963s 2:06.89 100.6% 959+37724k 32+82io 46pf+0w Dual P3 850mhz, 1GB RAM, FreeBSD 7.0-RELEASE-p4, i386 208.494u 84.935s 8:07.23 60.2% 2270+990k 17+87io 60pf+0w 4-core Opteron 2218 2.6ghz, 16GB RAM, FreeBSD 7.0-RELEASE-p4, amd64 38.893u 16.623s 0:55.53 99.9% 2290+591k 96+99io 48pf+0w Dual Xeon 3.06GHz, 4GB RAM, FreeBSD 7.0-RELEASE-p4, i386 60.910u 24.667s 1:22.54 103.6% 2143+988k 146+134io 105pf+0w Dual P3 866mhz, 2GB RAM, FreeBSD 7.0-RELEASE-p4, i386 169.135u 58.198s 3:52.71 97.6% 2443+1002k 160+99io 368pf+0w 2-core Core 2 Duo 2.33ghz, 2GB RAM, Mac OS X 10.5.6, i386 48.155u 29.896s 1:25.14 91.6% 0+0k 30+222io 1845pf+0w 4-core Xeon 2.66ghz, 6GB RAM, Mac OS X 10.5.6, i386 real 1m17.024s user 0m44.373s sys 0m34.249s None of these boxes were idle, so relative times are pretty useless, but i'm not seeing anything on the order of tens of minutes or hours. Is the source .tar.gz identical on all your systems? -- Kevin From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 30 03:03:38 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17B6E106564A for ; Tue, 30 Dec 2008 03:03:38 +0000 (UTC) (envelope-from kozlov@ravenloft.kiev.ua) Received: from istc.kiev.ua (wolf.istc.kiev.ua [193.108.236.1]) by mx1.freebsd.org (Postfix) with ESMTP id C73738FC16 for ; Tue, 30 Dec 2008 03:03:37 +0000 (UTC) (envelope-from kozlov@ravenloft.kiev.ua) Received: from [91.123.146.100] (helo=ravenloft.kiev.ua) by istc.kiev.ua with esmtp (Exim 4.52) id 1LHU83-00063B-2K; Tue, 30 Dec 2008 04:14:43 +0200 Date: Tue, 30 Dec 2008 04:14:42 +0200 From: Alex Kozlov To: Christian Weisgerber , freebsd-hackers@freebsd.org Message-ID: <20081230021442.GA11407@ravenloft.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spam-Score: 0.0 (/) X-Spam-Report: Content analysis detailz: (0.0 points, 10.0 required) Cc: Subject: Re: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Dec 2008 03:03:38 -0000 Christian Weisgerber wrote: > Oh, and everybody is invited to run > $ cd /usr/ports/archivers/lzo2 && make $cd /usr/ports/archivers/lzo2 && time sudo make [...] All tests passed. Now you are ready to install LZO. real 1m1.041s user 0m38.087s sys 0m17.613s This is Intel q6600. -- Adios From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 30 13:20:55 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E25AA106566C for ; Tue, 30 Dec 2008 13:20:55 +0000 (UTC) (envelope-from mailnull@mips.inka.de) Received: from mail-in-03.arcor-online.net (mail-in-03.arcor-online.net [151.189.21.43]) by mx1.freebsd.org (Postfix) with ESMTP id 934A48FC17 for ; Tue, 30 Dec 2008 13:20:55 +0000 (UTC) (envelope-from mailnull@mips.inka.de) Received: from mail-in-17-z2.arcor-online.net (mail-in-17-z2.arcor-online.net [151.189.8.34]) by mail-in-03.arcor-online.net (Postfix) with ESMTP id 783062CA9FC for ; Tue, 30 Dec 2008 14:20:54 +0100 (CET) Received: from mail-in-01.arcor-online.net (mail-in-01.arcor-online.net [151.189.21.41]) by mail-in-17-z2.arcor-online.net (Postfix) with ESMTP id 615CB45C0DF for ; Tue, 30 Dec 2008 14:20:54 +0100 (CET) Received: from lorvorc.mips.inka.de (dslb-092-075-107-001.pools.arcor-ip.net [92.75.107.1]) by mail-in-01.arcor-online.net (Postfix) with ESMTP id 3554419B327 for ; Tue, 30 Dec 2008 14:20:51 +0100 (CET) Received: from lorvorc.mips.inka.de (localhost [127.0.0.1]) by lorvorc.mips.inka.de (8.14.3/8.14.3) with ESMTP id mBUDKope051998 for ; Tue, 30 Dec 2008 14:20:50 +0100 (CET) (envelope-from mailnull@lorvorc.mips.inka.de) Received: (from mailnull@localhost) by lorvorc.mips.inka.de (8.14.3/8.14.3/Submit) id mBUDKom4051997 for freebsd-hackers@freebsd.org; Tue, 30 Dec 2008 14:20:50 +0100 (CET) (envelope-from mailnull) From: naddy@mips.inka.de (Christian Weisgerber) Date: Tue, 30 Dec 2008 13:20:50 +0000 (UTC) Message-ID: References: <20081230004747.GA55542@lorvorc.mips.inka.de> <20081230020524.2563a6ac@gluon> Originator: naddy@mips.inka.de (Christian Weisgerber) To: freebsd-hackers@freebsd.org X-Virus-Scanned: ClamAV 0.94.1/8814/Tue Dec 30 09:43:21 2008 on mail-in-01.arcor-online.net X-Virus-Status: Clean Subject: Re: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Dec 2008 13:20:56 -0000 Bruce Cran wrote: > I'm running 8.0-CURRENT amd64 here on a Turion64 X2 machine. Without > malloc debugging (malloc.conf -> aj) 'make test' takes 25s; after > removing malloc.conf thus turning on debugging, it takes over 10 > minutes. Wow! That. Is. It. Toggling malloc debugging option J makes the "slow" machines "fast" and vice versa. > > Athlon 64 X2 5200+ 2.6 GHz, FreeBSD 8.0-CURRENT amd64 ~60 min 19 seconds. I guess that falls under the "obvious" configuration differences to check, but since it usually doesn't cause a significant slowdown I completely forgot about it. Embarrassing. But still. Two orders of magnitude? That is a pathological case. -- Christian "naddy" Weisgerber naddy@mips.inka.de From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 30 14:07:01 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2305A10656C2 for ; Tue, 30 Dec 2008 14:07:01 +0000 (UTC) (envelope-from des@des.no) Received: from tim.des.no (tim.des.no [194.63.250.121]) by mx1.freebsd.org (Postfix) with ESMTP id D6B518FC1A for ; Tue, 30 Dec 2008 14:07:00 +0000 (UTC) (envelope-from des@des.no) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id A04876D449; Tue, 30 Dec 2008 14:06:59 +0000 (UTC) Received: by ds4.des.no (Postfix, from userid 1001) id 7F246844B0; Tue, 30 Dec 2008 15:06:59 +0100 (CET) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Christian Weisgerber References: <20081230004747.GA55542@lorvorc.mips.inka.de> Date: Tue, 30 Dec 2008 15:06:59 +0100 In-Reply-To: <20081230004747.GA55542@lorvorc.mips.inka.de> (Christian Weisgerber's message of "Tue, 30 Dec 2008 01:47:47 +0100") Message-ID: <86eizp4v0c.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.60 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Nate Eldredge , freebsd-hackers@freebsd.org Subject: Re: lzo2 shows insane speed gap X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Dec 2008 14:07:02 -0000 Christian Weisgerber writes: > Oh, and everybody is invited to run > > $ cd /usr/ports/archivers/lzo2 && make I assume you meant "time make". This is insane: 3108.27 real 1215.69 user 1888.06 sys on an E6600 with 4 GB RAM. What surprises me most is the high sys time. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-hackers@FreeBSD.ORG Wed Dec 31 16:02:12 2008 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 84918106564A; Wed, 31 Dec 2008 16:02:12 +0000 (UTC) (envelope-from so14k@valentine.liquidneon.com) Received: from valentine.liquidneon.com (valentine.liquidneon.com [216.87.78.132]) by mx1.freebsd.org (Postfix) with ESMTP id 6E0A88FC0C; Wed, 31 Dec 2008 16:02:12 +0000 (UTC) (envelope-from so14k@valentine.liquidneon.com) Received: by valentine.liquidneon.com (Postfix, from userid 1018) id F0FD38FDE9; Wed, 31 Dec 2008 08:33:27 -0700 (MST) Date: Wed, 31 Dec 2008 08:33:27 -0700 From: Brad Davis To: freebsd-current@FreeBSD.org, freebsd-hackers@FreeBSD.org Message-ID: <20081231153327.GD80570@valentine.liquidneon.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: Subject: FreeBSD Status Reports due January 14th, 2009 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Dec 2008 16:02:13 -0000 Hi Everyone, Happy New Year! We would like to remind everybody who has exciting news to share to write a report about their project. This is a good way to improve exposure of your work, receive feedback and help. We are looking forward to your reports. As always you can either use the template or the CGI generator and mail the output to monthly@ by Wednesday January 14th, 2009. http://www.freebsd.org/news/status/ http://www.freebsd.org/cgi/monthly.cgi http://www.freebsd.org/news/status/report-sample.xml Regards, Brad Davis