Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 16 Jun 2005 16:59:26 -0400
From:      Karim Fodil-Lemelin <kfl@xiphos.ca>
To:        dima <_pppp@mail.ru>
Cc:        dave baukus <dbaukus@chiaro.com>, freebsd-net@freebsd.org
Subject:   Re: (panic) Lots of network memory needed
Message-ID:  <42B1E82E.2050102@xiphos.ca>
In-Reply-To: <E1Diw5K-00004d-00._pppp-mail-ru@f13.mail.ru>
References:  <E1Diw5K-00004d-00._pppp-mail-ru@f13.mail.ru>

next in thread | previous in thread | raw e-mail | index | archive | help
I use FreeBSD 4.9. Actually I had to _reduce_ VM_KMEM_SIZE in order to 
be able to create that much mnbclusters ...

Somehow it seems that vm_kmem_size cannot go over 1GB without crashing 
the system so I had to reduce memory map from other subsystems in order 
to get what I needed for network memory. From:

	vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE;

Assuming vm_kmem_size has some sort of hard limit, if I reduce VM_KMEM_SIZE I can get more nmbclusters into the system.


dima wrote:

>>I'm fairly sure that you are out of kernel virtual memory.
>>Look at kern/kern_malloc.c kmeminit (); you can play w/ VM_KMEM_SIZE_MAX
>>or  TUNABLE_INT_FETCH("kern.vm.kmem.size", &vm_kmem_size);
>>
>>      /*
>>          * Try to auto-tune the kernel memory size, so that it is
>>          * more applicable for a wider range of machine sizes.
>>          * On an X86, a VM_KMEM_SIZE_SCALE value of 4 is good, while
>>          * a VM_KMEM_SIZE of 12MB is a fair compromise.  The
>>          * VM_KMEM_SIZE_MAX is dependent on the maximum KVA space
>>          * available, and on an X86 with a total KVA space of 256MB,
>>          * try to keep VM_KMEM_SIZE_MAX at 80MB or below.
>>          *
>>          * Note that the kmem_map is also used by the zone allocator,
>>          * so make sure that there is enough space.
>>          */
>>         vm_kmem_size = VM_KMEM_SIZE;
>>         mem_size = cnt.v_page_count * PAGE_SIZE;
>>    
>>
>Which version do you use? I see the following in my /sys/kern/kern_malloc.c:
>        vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE;
>....
>        /*
>         * Limit kmem virtual size to twice the physical memory.
>         * This allows for kmem map sparseness, but limits the size
>         * to something sane. Be careful to not overflow the 32bit
>         * ints while doing the check.
>         */
>        if (((vm_kmem_size / 2) / PAGE_SIZE) > cnt.v_page_count)
>                vm_kmem_size = 2 * cnt.v_page_count * PAGE_SIZE;
>It's 5.4-p2
>
>
>
>  
>
>>#if defined(VM_KMEM_SIZE_SCALE)
>>         if ((mem_size / VM_KMEM_SIZE_SCALE) > vm_kmem_size)
>>                 vm_kmem_size = mem_size / VM_KMEM_SIZE_SCALE;
>>#endif
>>
>>#if defined(VM_KMEM_SIZE_MAX)
>>         if (vm_kmem_size >= VM_KMEM_SIZE_MAX)
>>                 vm_kmem_size = VM_KMEM_SIZE_MAX;
>>#endif
>>
>>         /* Allow final override from the kernel environment */
>>         TUNABLE_INT_FETCH("kern.vm.kmem.size", &vm_kmem_size);
>>
>>
>>Karim Fodil-Lemelin wrote:
>>    
>>
>>>Thanks but the system still crashes (FreeBSD 4.9) with 131072. Here is a 
>>>backtrace showing just that:
>>>
>>>Debugger (msg=0xc02b6cdb "panic") at ../../i386/i386/db_interface.c:321
>>>321     }
>>>(kgdb) bt
>>>#0  Debugger (msg=0xc02b6cdb "panic") at ../../i386/i386/db_interface.c:321
>>>#1  0xc016a230 in panic (
>>>   fmt=0xc02ea380 "pmap_enter: invalid page directory pdir=%#llx, 
>>>va=%#x\n")
>>>   at ../../kern/kern_shutdown.c:593
>>>#2  0xc0283594 in pmap_enter (pmap=0xc0340460, va=4292141056, m=0xc24b2848,
>>>   prot=7 '\a', wired=1) at ../../i386/i386/pmap.c:1943
>>>#3  0xc023ddd0 in vm_fault (map=0xc033322c, vaddr=4292141056,
>>>   fault_type=7 '\a', fault_flags=1) at ../../vm/vm_fault.c:841
>>>#4  0xc023df0a in vm_fault_wire (map=0xc033322c, start=4292141056,
>>>   end=4292149248) at ../../vm/vm_fault.c:915
>>>#5  0xc0240831 in vm_map_pageable (map=0xc033322c, start=4292141056,
>>>   real_end=4292149248, new_pageable=0) at ../../vm/vm_map.c:1817
>>>#6  0xc023ea25 in kmem_alloc (map=0xc033322c, size=8192)
>>>   at ../../vm/vm_kern.c:213
>>>#7  0xc024a46f in _zget (z=0xdb5c6e80) at ../../vm/vm_zone.c:425
>>>#8  0xc024a269 in zalloc (z=0xdb5c6e80) at ../../vm/vm_zone.c:60
>>>#9  0xc0196106 in namei (ndp=0xfa489ef4) at ../../kern/vfs_lookup.c:104
>>>#10 0xc01614ca in execve (p=0xfa482e00, uap=0xfa489f90)
>>>   at ../../kern/kern_exec.c:165
>>>#11 0xc01590a1 in start_init (dummy=0x0) at ../../kern/init_main.c:543
>>>(kgdb) p nmbclusters
>>>$1 = 131072
>>>(kgdb)
>>>
>>>Goran Spirovski - MorEl On.net wrote:
>>>
>>>      
>>>
>>>>AFAIK the number of mbufs (and consequently nmbclusters) has to be a 
>>>>power
>>>>of 2, so you should set it to 131072
>>>>
>>>>MorEl
>>>>
>>>>----- Original Message ----- From: "Karim Fodil-Lemelin" <kfl@xiphos.ca>
>>>>To: <freebsd-net@freebsd.org>
>>>>Sent: Wednesday, June 15, 2005 6:08 PM
>>>>Subject: (panic) Lots of network memory needed
>>>>
>>>>
>>>> 
>>>>
>>>>        
>>>>
>>>>>Hello,
>>>>>
>>>>>From kernel tuning page
>>>>>  
>>>>>          
>>>>>
>>>>(http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-kern 
>>>>
>>>>el-limits.html)
>>>> 
>>>>
>>>>        
>>>>
>>>>>Some sentence about nmbclusters says:
>>>>>
>>>>>"Under no circumstances should you specify an arbitrarily high value for
>>>>>  
>>>>>          
>>>>>
>>>>this parameter as it could lead to a boot time crash."
>>>> 
>>>>
>>>>        
>>>>
>>>>>Now I want to push the limits where I need 4KB buffer for each of the
>>>>>  
>>>>>          
>>>>>
>>>>32000 connections I want this server to handle. If I do the math:
>>>> 
>>>>
>>>>        
>>>>
>>>>>(32000 (conns) * 4 (KB/buffer) * 2 (buffer/conn)) / 2048 (KB/cluster)
>>>>>
>>>>>= 128000 clusters
>>>>>
>>>>>So I set this arbitrary high value in loader.conf under
>>>>>  
>>>>>          
>>>>>
>>>>(kern.ipc.nmbclusters) and no surprises I get panic: pmap_enter 
>>>>invalid page
>>>>directory pdir=0x3cb063, va=0xfff800
>>>> 
>>>>
>>>>        
>>>>
>>>>>(va has a weird address here)
>>>>>
>>>>>I know I am pushing the limits here but I have plenty of memory (2GB) on
>>>>>  
>>>>>          
>>>>>
>>>>this system (after all its just 250MB for network memory ;) and this is
>>>>mainly just experimentation.
>>>> 
>>>>
>>>>        
>>>>
>>>>>I would like some pointers toward fixing this. Is there another variable
>>>>>  
>>>>>          
>>>>>
>>>>tied into this (I guess so)? Could anybody points me to a technical 
>>>>document
>>>>that would explain the relationship with that (those) other(s) presumed
>>>>variable(s)?
>>>> 
>>>>
>>>>        
>>>>
>>>>>Thank you,
>>>>>
>>>>>
>>>>>Karim
>>>>>
>>>>>
>>>>>_______________________________________________
>>>>>freebsd-net@freebsd.org mailing list
>>>>>http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>>>>>  
>>>>>          
>>>>>
>>>>
>>>>
>>>> 
>>>>
>>>>        
>>>>
>>-- 
>>Dave Baukus
>>dbaukus@chiaro.com			
>>	Chiaro Networks Ltd.
>>	Richardson, Texas
>>	USA
>>
>>_______________________________________________
>>freebsd-net@freebsd.org mailing list
>>http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>>
>>    
>>
>
>
>  
>

-- 
Karim Fodil-Lemelin
Lead Programmer

Xiphos Technologies Inc.
(514) 848-9640 x223
(514) 848-9644 fax
www.xiplink.com

--------------------------------------------------------------
The information transmitted is intended only for the
person or entity to which it is addressed and may contain
confidential and/or privileged material.  If you have
received this in error, please contact the sender and delete
this communication and any copy immediately. Thank you.





Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?42B1E82E.2050102>