From owner-freebsd-database Tue Oct 26 13:15:41 1999 Delivered-To: freebsd-database@freebsd.org Received: from gatekeeper.veriohosting.com (gatekeeper.veriohosting.com [192.41.0.2]) by hub.freebsd.org (Postfix) with ESMTP id C21BC14CC4; Tue, 26 Oct 1999 13:15:19 -0700 (PDT) (envelope-from steveb@veriohosting.com) Received: by gatekeeper.veriohosting.com; Tue, 26 Oct 1999 14:15:14 -0600 (MDT) Received: from unknown(192.168.1.7) by gatekeeper.veriohosting.com via smap (V3.1.1) id xma021951; Tue, 26 Oct 99 14:15:03 -0600 Received: from iserver.com (glacier.orem.iserver.com [192.168.1.111]) by orca.orem.veriohosting.com [Verio Web Hosting, Inc. 801.437.0200] (8.8.8) id OAA79261; Tue, 26 Oct 1999 14:16:18 -0600 (MDT) Message-ID: <38160B6A.5568F6D1@iserver.com> Date: Tue, 26 Oct 1999 14:13:31 -0600 From: Steve Bishop X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 2.2.7-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: freebsd-database@freebsd.org, freebsd-hackers@freebsd.org Cc: "Paul.Marquess@btinternet.com" Subject: mbuf problem (panic)--possibly related to Berkeley DB 2.7.7 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-database@FreeBSD.ORG Precedence: bulk I am using FreeBSD 3.3-RELEASE, and it is running on a single processor PII 400. At first, I thought the problem was due to the network driver, so I swapped network cards. But, the problem still continues to occur. At first, I used a DEC (de0) NIC. Then, I switched to an Intel EtherExpress Pro 10/100B Ethernet (fxp0). My machine dies when it runs out of mbufs. Normally, one only associates mbufs with network usage, and network (NIC) drivers. I have tried to investigate this possibility, as mentioned, but to no avail. The problem is either a kernel problem, or is related to the Berkeley DB, and/or perl module - BerkeleyDB-0.07.pm. I have written to Paul Marquess who is the author of this perl module. He did not respond, and may not have any idea. The latest release of this perl module is a significant update to the perl support for the latest releases of Berkeley DB. It provides for concurrency and database integrity (consistency) between processes sharing or using the same database simultaneously. There are a number of scripts, I have written, in the system that run (as processes) in parallel. This is why I am using the Concurrent Access Method (CAM) to allow multiple simultaneous readers along with sequential write (one writer at a time) access. Berkeley DB keeps everything consistent. The scripts are designed to use the database a lot, and they also use a significant amount of network resources. The scripts sometimes can have up to 900 open tcp connections, and consistently use almost 600. I have increased the number of mbuf clusters (NMBCLUSTERS) from 1024 to 4096. I have also increased maxusers to 64. The "panics" seem to occur when my scripts or processes prematurely exit without shutting down Berkeley DB in the standard, proper fashion. As I explained, I have upped the number of mbuf clusters from the default 1024, to 4096. The actual number probably wouldn't matter a whole lot as long as it is sufficient to meet the needs of the system were being met. What I'm trying to say, is that when the script(s) crash, the system then begins to use mbufs exponentially, until they're completely exhausted within minutes. The system panics, and reboots. I know the association is loose, but nothing else presents itself as the problem. Is there some kernel resource that Berkeley DB gets a hold of, and never properly releases due to an improper, fatal termination of my script, which could indirectly cause my system to run out of mbufs? I know that the close_everything routine in BerkeleyDB.pm still runs when perl exits, but perhaps, it doesn't work properly in this case? Any ideas or leads as to why this is occuring would be sincerely appreciated. Thanks, Steve Bishop Developer Verio, Inc. verio.com -- the new world of business PS Here's some more background info. In perl, to setup Berkeley DB, I use the shared memory pool. Here is how I setup the environment, and open my database. $env = new BerkeleyDB::Env -Flags => DB_CREATE| DB_INIT_MPOOL| DB_INIT_CDB, -ErrFile => "db-errors"; $DB = tie %db, 'BerkeleyDB::Btree', -Filename => 'servers.db', -Mode => 0600, -Flags => DB_CREATE, -Pagesize => 4096, # default = 8192 -Env => $env or die "Cannot open db: $! $BerkeleyDB::Error"; I think I have discovered an association between my system crashes ("out of mbufs"), and the usage of Berkeley DB. When my script prematurely crashes or exits without following the proper database shutdown procedure, as outlined below, it isn't very long before the machine crashes (usually a few minutes). undef($DB); undef($env); untie(%db); To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-database" in the body of the message From owner-freebsd-database Tue Oct 26 21:22: 9 1999 Delivered-To: freebsd-database@freebsd.org Received: from misery.sdf.com (misery.sdf.com [204.244.213.49]) by hub.freebsd.org (Postfix) with ESMTP id B50EB14A18; Tue, 26 Oct 1999 21:22:02 -0700 (PDT) (envelope-from tom@sdf.com) Received: from tom (helo=localhost) by misery.sdf.com with local-esmtp (Exim 2.12 #1) id 11gJ6Q-0006v8-00; Tue, 26 Oct 1999 19:46:18 -0700 Date: Tue, 26 Oct 1999 19:46:15 -0700 (PDT) From: Tom To: Steve Bishop Cc: freebsd-database@freebsd.org, freebsd-hackers@freebsd.org, "Paul.Marquess@btinternet.com" Subject: Re: mbuf problem (panic)--possibly related to Berkeley DB 2.7.7 In-Reply-To: <38160B6A.5568F6D1@iserver.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-database@FreeBSD.ORG Precedence: bulk On Tue, 26 Oct 1999, Steve Bishop wrote: > The scripts are designed to use the database a lot, and they also use a significant amount > of network resources. The scripts sometimes can have up to 900 open tcp connections, > and consistently use almost 600. I have increased the number of mbuf clusters (NMBCLUSTERS) > from 1024 to 4096. I have also increased maxusers to 64. 4096 isn't really that many mbufs. You should double or triple that. I think that large web hosting people use 20000 mbufs. That eats a LOT of RAM, but at least you can shrug off the heaviest DoS attack. In fact, if you don't specify a number of mbufs in your kernel file, MAXUSERS set to 64 will probably give you more than 4096 mbufs! Tom To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-database" in the body of the message From owner-freebsd-database Wed Oct 27 4:36:37 1999 Delivered-To: freebsd-database@freebsd.org Received: from babelbrox.axion.bt.co.uk (babelbrox.axion.bt.co.uk [132.146.16.6]) by hub.freebsd.org (Postfix) with ESMTP id 8DBE914F56; Wed, 27 Oct 1999 04:36:32 -0700 (PDT) (envelope-from paul.marquess@bt.com) Received: from cbtlipnt01.btlabs.bt.co.uk by babelbrox.axion.bt.co.uk (local) with ESMTP; Wed, 27 Oct 1999 12:35:28 +0100 Received: by cbtlipnt01.btlabs.bt.co.uk with Internet Mail Service (5.5.2448.0) id ; Wed, 27 Oct 1999 12:35:28 +0100 Message-ID: <5104D4DBC598D211B5FE0000F8FE7EB202D49CA4@mbtlipnt02.btlabs.bt.co.uk> From: paul.marquess@bt.com To: steveb@veriohosting.com, freebsd-database@freebsd.org, freebsd-hackers@freebsd.org Cc: Paul.Marquess@btinternet.com Subject: RE: mbuf problem (panic)--possibly related to Berkeley DB 2.7.7 Date: Wed, 27 Oct 1999 12:35:11 +0100 X-Mailer: Internet Mail Service (5.5.2448.0) MIME-version: 1.0 Content-type: text/plain; charset="iso-8859-1" Sender: owner-freebsd-database@FreeBSD.ORG Precedence: bulk From: steveb@veriohosting.com [mailto:steveb@veriohosting.com] > I am using FreeBSD 3.3-RELEASE, and it is running on a single processor PII 400. > > At first, I thought the problem was due to the network driver, so I swapped network > cards. But, the problem still continues to occur. At first, I used a DEC (de0) NIC. > Then, I switched to an Intel EtherExpress Pro 10/100B Ethernet (fxp0). > > My machine dies when it runs out of mbufs. Normally, one only associates mbufs with > network usage, and network (NIC) drivers. I have tried to investigate this possibility, > as mentioned, but to no avail. > > The problem is either a kernel problem, or is related to the Berkeley DB, and/or > perl module - BerkeleyDB-0.07.pm. I have written to Paul Marquess who is the > author of this perl module. He did not respond, and may not have any idea. I got your original message, but haven't had the time to investigate your problem. After reviewing it and the messages in this thread, as you suspected, I don't have any idea. Paul To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-database" in the body of the message From owner-freebsd-database Thu Oct 28 10:48:33 1999 Delivered-To: freebsd-database@freebsd.org Received: from gatekeeper.veriohosting.com (gatekeeper.veriohosting.com [192.41.0.2]) by hub.freebsd.org (Postfix) with ESMTP id EAD92151C6; Thu, 28 Oct 1999 10:48:28 -0700 (PDT) (envelope-from steveb@veriohosting.com) Received: by gatekeeper.veriohosting.com; Thu, 28 Oct 1999 11:48:25 -0600 (MDT) Received: from unknown(192.168.1.7) by gatekeeper.veriohosting.com via smap (V3.1.1) id xma012904; Thu, 28 Oct 99 11:48:07 -0600 Received: from iserver.com (glacier.orem.iserver.com [192.168.1.111]) by orca.orem.veriohosting.com [Verio Web Hosting, Inc. 801.437.0200] (8.8.8) id LAA59123; Thu, 28 Oct 1999 11:49:20 -0600 (MDT) Message-ID: <38188BEC.75ABC44D@iserver.com> Date: Thu, 28 Oct 1999 11:46:21 -0600 From: Steve Bishop X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 2.2.7-RELEASE i386) X-Accept-Language: en MIME-Version: 1.0 To: freebsd-database@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG Cc: "Paul.Marquess@btinternet.com" Subject: Re: mbuf problem (panic) SOLVED --possibly related to Berkeley DB 2.7.7 References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-database@FreeBSD.ORG Precedence: bulk I've increased maxusers to 512, and NMBCLUSTERS to 16384, and I haven't been able to reproduce the kernel panic problem anymore. So, my conclusion is that increasing maxusers solved the problem. I don't believe that raising NMBCLUSTERS by itself, or it indirectly being raised by maxusers, fixed the problem. The reason I think this is because I raised NMBCLUSTERS to 8192 before, and it had no effect whatsover in delaying or preventing the kernel panics I was experiencing. As I described before, right before the panic occurred, the OS began to exponentially use up all of the mbufs, and even if I had set NBMCLUSTERS to 30000 it would have used them up in short order. The typical mbuf usage for this system while running all of its software (scripts) is about 100 mbufs, and it has been running over a day now, and not exceeded 962. I've not seen anywhere near the kind of mbuf usage, that occurred when the machine panicked before. This leads me to the conclusion that maxusers, affects many other kernel parameters (which of course it does), and indirectly eliminated the mbuf problem by providing enough resources for the database in all situations, and most especially in cases when Berkeley DB wasn't shut down clean, or in its proper fashion. Although I don't know all of the kernel parameters affected by increasing maxusers, it seems to have solved my problem, and in a roundabout way prevented the machine going into its "spiral-of-death" mbuf problem. -Steve Developer Verio, Inc. verio.com -- the new world of business To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-database" in the body of the message