From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 19 09:43:23 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AAE4316A4B3; Sun, 19 Oct 2003 09:43:23 -0700 (PDT) Received: from sweeper.openet-telecom.com (mail.openet-telecom.com [62.17.151.60]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9AF3043FA3; Sun, 19 Oct 2003 09:43:21 -0700 (PDT) (envelope-from peter.edwards@openet-telecom.com) Received: from mail.openet-telecom.com (unverified) by sweeper.openet-telecom.com ; Sun, 19 Oct 2003 17:43:27 +0100 Received: from openet-telecom.com (10.0.1.153) by mail.openet-telecom.com (NPlex 6.5.027) (authenticated as peter.edwards@openet-telecom.com) id 3F6FE91700015F73; Sun, 19 Oct 2003 17:39:17 +0100 Message-ID: <3F92BF18.9000604@openet-telecom.com> Date: Sun, 19 Oct 2003 17:43:04 +0100 From: Peter Edwards User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.3.1) Gecko/20030524 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Dag-Erling@FreeBSD.ORG, =?us-ascii@FreeBSD.ORG:iso-8859-1?Q?Sm=F8rgrav?= References: <20031016060632.GA68007@VARK.homeunix.com> In-Reply-To: <20031016060632.GA68007@VARK.homeunix.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: David Schultz cc: hackers@FreeBSD.ORG Subject: Re: real vs. avail memory X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: pmedwards@eircom.net List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Oct 2003 16:43:23 -0000 David Schultz wrote: >On Sun, Oct 12, 2003, Dag-Erling Smrgrav wrote: > > >>I've gotten used to the fact that there is a small discrepancy between >>real and available memory, but I was surprised to see the following in >>dmesg on a new P4 system: >> >>real memory = 1073676288 (1023 MB) >>avail memory = 1037799424 (989 MB) >> >>That's a full 40 MB difference... where does that memory go? is it >>used for page maps or something like that? >> >> > >Unless this is related to Peter's recent machdep.c changes, the >difference is probably just random chunks of memory that the BIOS >decided to use. This could include a shadow copy of the BIOS, the >BIOS data segment, maybe a frame buffer for a cheap integrated >video card, etc. If you do a verbose boot, you'll get a list of >the chunks of memory that are taken according to the BIOS. > > > This piqued my interest, so I studied it a bit. (My 512MB machine was showing a 24MB difference) Most of the memory is swallowed by two functions called from vm_mem_init(): The kernel allocates a vm_page structure for every page of available memory early on. These are 72 bytes in size. (see vm_page_startup() in vm_page.c) Later, the pmap code preallocates a pv_entry structure for each vm_page (I assume it can probably allocate more later, but will need at least this many if each page is mapped once). These are 28 bytes each. (See pmap_init() in pmap.c) The kernel itself will come out of real memory too. The kernel is loaded into 4MB pages, and any residue in the last 4MB page appears to be lost at the moment. I'm not sure if this was always the case (well, since PSE was introduced) or if it's just an issue since Bosko's fix for the weird Pentium bugs was committed. If you look at the "phys_avail" array of a running system, you can see how much the "hole" for the kernel takes up: > petere@hippo# gdb -k /usr/src/sys/i386/compile/HIPPO/kernel.debug /dev/mem > GNU gdb 5.2.1 (FreeBSD) > [snip] > (kgdb) p phys_avail > $1 = {4096, 651264, 1048576, 4194304, 12738560, 526987264, 0, 0, 0, 0} > (kgdb) This array shows the spans of physical memory that were found at boot time. The even elements (counting from 0) are the start of the span, with odd elements being the end. My box shows a hole between 4194304 and 12738559: This is where the kernel was loaded. Assuming your phys_avail map had a similar 8MB hole for the kernel, then the minimum estimate of the amount of space consumed before printing the "avail memory" message can be calculated as follows: Available memory in pages = 107366288 / 4096 = 253369 Space Lost to struct "vm_page"s: 253369 * 72 = 18242568 Space Lost to struct "pv_entry"s: 253369 * 28 = 7094332 Space Lost to kernel "hole": 12738560 - 4194304 = 8544256 -------- 33881156 that's within 2MB of what you're seeing. You can probably take out some more space for preloaded kld admin overhead, and the message buffer, and probably plenty more I can't think of (BIOS and WITNESS SYSINITS at least happen between the final print out and the VM initialisation: that probably accounts for a lot of the remainder)