From owner-freebsd-hackers@FreeBSD.ORG Thu Jan 31 23:04:59 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 703B616A469 for ; Thu, 31 Jan 2008 23:04:59 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from cmail.optima.ua (cmail.optima.ua [195.248.191.121]) by mx1.freebsd.org (Postfix) with ESMTP id F387113C469 for ; Thu, 31 Jan 2008 23:04:58 +0000 (UTC) (envelope-from mav@FreeBSD.org) X-Spam-Flag: SKIP X-Spam-Yversion: Spamooborona 1.7.0 Received: from [212.86.226.226] (account mav@alkar.net HELO [192.168.3.2]) by cmail.optima.ua (CommuniGate Pro SMTP 5.1.14) with ESMTPA id 71393496; Fri, 01 Feb 2008 01:04:57 +0200 Message-ID: <47A25412.3010301@FreeBSD.org> Date: Fri, 01 Feb 2008 01:04:50 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0 To: freebsd-performance@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Memory allocation performance X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jan 2008 23:04:59 -0000 Hi. While profiling netgraph operation on UP HEAD router I have found that huge amount of time it spent on memory allocation/deallocation: 0.14 0.05 132119/545292 ip_forward [12] 0.14 0.05 133127/545292 fxp_add_rfabuf [18] 0.27 0.10 266236/545292 ng_package_data [17] [9]14.1 0.56 0.21 545292 uma_zalloc_arg [9] 0.17 0.00 545292/1733401 critical_exit [98] 0.01 0.00 275941/679675 generic_bzero [68] 0.01 0.00 133127/133127 mb_ctor_pack [103] 0.15 0.06 133100/545266 mb_free_ext [22] 0.15 0.06 133121/545266 m_freem [15] 0.29 0.11 266236/545266 ng_free_item [16] [8]15.2 0.60 0.23 545266 uma_zfree_arg [8] 0.17 0.00 545266/1733401 critical_exit [98] 0.00 0.04 133100/133100 mb_dtor_pack [57] 0.00 0.00 134121/134121 mb_dtor_mbuf [111] I have already optimized all possible allocation calls and those that left are practically unavoidable. But even after this kgmon tells that 30% of CPU time consumed by memory management. So I have some questions: 1) Is it real situation or just profiler mistake? 2) If it is real then why UMA is so slow? I have tried to replace it in some places with preallocated TAILQ of required memory blocks protected by mutex and according to profiler I have got _much_ better results. Will it be a good practice to replace relatively small UMA zones with preallocated queue to avoid part of UMA calls? 3) I have seen that UMA does some kind of CPU cache affinity, but does it cost so much that it costs 30% CPU time on UP router? Thanks! -- Alexander Motin