From owner-freebsd-questions@FreeBSD.ORG Tue Sep 23 10:33:36 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 879811065674 for ; Tue, 23 Sep 2008 10:33:36 +0000 (UTC) (envelope-from lists@servingpeace.com) Received: from smtp.servingpeace.com (servingpeace.com [209.172.55.71]) by mx1.freebsd.org (Postfix) with ESMTP id 609E88FC15 for ; Tue, 23 Sep 2008 10:33:35 +0000 (UTC) (envelope-from lists@servingpeace.com) Received: from [192.168.1.100] (adsl-68-122-10-218.dsl.pltn13.pacbell.net [68.122.10.218]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: sam@samnilsson.com) by smtp.servingpeace.com (Postfix) with ESMTP id ED7677D for ; Tue, 23 Sep 2008 03:13:59 -0700 (PDT) Message-ID: <48D8C166.1020601@servingpeace.com> Date: Tue, 23 Sep 2008 03:13:58 -0700 From: Sam Nilsson User-Agent: Thunderbird 2.0.0.16 (Macintosh/20080707) MIME-Version: 1.0 To: FreeBSD Questions Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: MySQL Error: Can't create a new thread (errno 35); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2008 10:33:36 -0000 Hello, The error message: Can't create a new thread (errno 35); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug Our website started getting this error several weeks ago (when we increased the number of application server machines connecting to the database from 5 to 7. It seems to happen anywhere from every 2 days to once a week, there does not seem to be a consistent pattern. Our CPU utilization, memory utilization, and number of HTTP Requests seem to be at "normal" (they are no where closed to being maxed out). When this error happens we CAN log onto the server with the MySQL database, but I CAN NOT use the mysql client to connect to the mysql database (localhost connection as root) - it reports the error mentioned above. The way we temporarily solve the problem is to restart one of our web servers. Here is our technology stack: Web Servers: 7, Each server has: 4 Gb of Memory on each server FreeBSD 6.2-RELEASE-p9 2.4 GHz CPU Apache 2.2 Webware for Python Python 2.5 DB Servers: One Master, Two Read Only (replication) 4 GB of Memory on each server FreeBSD 6.3-RELEASE-p3 FreeBSD 6.3-RELEASE-p3 MySQL 5.0.1 We have tried bumping up our number of max allowed connections (up to 1000) on MySQL and kern.ssize to 512Mb for FreeBSD. That did not help. The max connections usually hovers ~300, so it does not even get close to the 1000 we have set. Finally the last place that we have looked is the openfiles, it is set at 20,000 for the whole system, with 14K per process on the FreeBSD operating system. Here are some relevent items from my.cnf: - set-variable = max_connections=1000 - set-variable = key_buffer_size=384M - set-variable = read_buffer_size=64M - set-variable = read_rnd_buffer_size=32M - set-variable = thread_cache_size=20 Any help would be much appreciated. Thanks - Sam Nilsson