From owner-freebsd-database@FreeBSD.ORG Thu Nov 10 12:08:46 2005 Return-Path: X-Original-To: freebsd-database@freebsd.org Delivered-To: freebsd-database@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A6C8416A41F for ; Thu, 10 Nov 2005 12:08:46 +0000 (GMT) (envelope-from zparta@gmail.com) Received: from zproxy.gmail.com (zproxy.gmail.com [64.233.162.193]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7DCEA43D6E for ; Thu, 10 Nov 2005 12:08:23 +0000 (GMT) (envelope-from zparta@gmail.com) Received: by zproxy.gmail.com with SMTP id 9so373286nzo for ; Thu, 10 Nov 2005 04:08:22 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=rlAwKFqRuxoX139IgWu89kTm+0LgsI2m9Gi7A2IUeoHoFOtK19gFFVvpmjWHFHjEBLoA1ZzSCgcJ8FTXPHWQTCWNdSkrGZhfWuyqm26r9KnZ8XKR1d9WI/1nsNXHMu3jbHfvTQO8OXyCeSUbCH7d72auzfRCa62s+Jen3gvdmhQ= Received: by 10.36.100.4 with SMTP id x4mr464166nzb; Thu, 10 Nov 2005 04:08:22 -0800 (PST) Received: by 10.37.18.50 with HTTP; Thu, 10 Nov 2005 04:08:21 -0800 (PST) Message-ID: <3b41db850511100408i6d48b4a4v5ee4e41571fa2870@mail.gmail.com> Date: Thu, 10 Nov 2005 13:08:22 +0100 From: Jens Holmqvist To: Justin Bastedo In-Reply-To: <8a5255240511091416i449bc233s4706f26c29002284@mail.gmail.com> MIME-Version: 1.0 References: <20051013025632.GN49168@wantadilla.lemis.com> <20051013043115.4693.qmail@web32913.mail.mud.yahoo.com> <8a5255240511091416i449bc233s4706f26c29002284@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: FreeBSD-DataBase , Kris Kennaway , questions Subject: Re: Mysql server not able to stay running on anything but Linux? X-BeenThere: freebsd-database@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Database use and development under FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Nov 2005 12:08:46 -0000 On 11/9/05, Justin Bastedo wrote: > > Hey everyone, > Not trying to beat a dead horse or anything but was wondering if any > resolution was found for this on either end. We are about to launch > another site and can easily forsee my load getting close to this size > within 6 months. Just need to find out any concerns with growth we > might have and come up with options. > > Just trying to plan ahead and keep problems from happening, since I > hadn't seen any formal resolution on this thread figured I'd inquire. > > Thanks everyone > > Justin Bastedo > > On 10/12/05, NMH wrote: > > --- Greg 'groggy' Lehey wrote: > > > > > Note the From: address. > > > > > > On Wednesday, 12 October 2005 at 13:16:22 -0700, NMH > > > wrote: > > > > > > > > I am stuck with a delema and I feel like a damn > > > troll. But.. I have > > > > a Mysql Database that I posted about earlier. It > > > seems that it is > > > > only able to not die by running on BSD 4.11 with > > > Linux Threads. My > > > > boss is convinced this means that Linux is better > > > for MySQL and > > > > wants that installed now. > > > > > > > > We even got a support contact from Mysql that so > > > far has gotten us > > > > nothing for almost a month while our production > > > database server died > > > > up to 3 times a day. (and lots of we're looking > > > into it's) > > > > > > One of the reasons why you haven't got much more > > > than "we're looking > > > into it"s is because we haven't been able to > > > reproduce the problem; > > > you acknowledge this in follow-up mail quoted below. > > > > > > As you know from various threads on the FreeBSD > > > lists, including this > > > one, the typical answer is "works fine for me". > > > That doesn't mean > > > that we're not taking your problems seriously, but > > > we do have a > > > significant issue just reproducing the problem. We > > > have a number of > > > choices: > > > > > > 1. Try different hardware or a different version of > > > FreeBSD. It's > > > conceivable that there's something about your > > > specific hardware, > > > or about the combination of i386 kernel on amd64 > > > in general, that > > > triggers the problem. > > > > > > Yes possible. However the same hardware was used for > > the earlier version machine that worked fine. IE > > Freebsd 5.1-RELEASE-P11 and Mysql 4.0.18 worked on the > > same hardware. > > > > > 2. Do debugging on your production servers. This > > > isn't really a > > > choice at all: it would involve even more down > > > time. > > > > Yea not really an option. > > > > > 3. Get you to run a more stable version of FreeBSD > > > while we > > > investigate the problem. This is the method we > > > chose. I haven't > > > heard from you since the weekend, so I hope I'm > > > correct in > > > understanding that you currently don't have > > > stability problems. > > > On our side, we have installed FreeBSD 5.4 on > > > one of our internal > > > machines, and we're trying to reproduce the > > > problem there. > > > > So far we have had only one crash that seemed to have > > been SCSI related. So far it has not happend again. > > One problem with replication that was a coding issue. > > > > > > > > We were running fine but a little slow on FreeBSD > > > 5.1-P11 and MySQL > > > > 4.0.18.(apperantly before a big Lib change) We had > > > to move quicker > > > > than we wanted to a new server running FreeBSD 5.4 > > > and MySQL 4.11 > > > > (becouse of a dual HD death) Under production load > > > the new 5.4 > > > > server fell over regulary. It has only now become > > > stable by wiping > > > > it and running it on FreeBSD 4.11 with Linux > > > Threads. (it regularly > > > > has over 400+ threads) > > > > > > Kris obviously understood that by this statement you > > > meant a kernel > > > crash. My understanding is that only the mysqld > > > server is crashing. > > > Is this still correct? > > > > Yes only Mysqld would crash. Sometimes brb and > > autorestart with minimal damage. Other times it would > > die a horrible death and damage tons of data on its > > way out. > > > > > > I want to try FreeBSD 5.4 AMD64 (the machines are > > > Opteron) or 6.0 > > > > but my boss feels that would be a waste given that > > > MYSQL doesn't > > > > support Mysql on AMD64 well enough. > > > > > > I think it would be a good idea to try this. It's > > > one of the things > > > that we intend to do in-house as soon as we can > > > reproduce the problem > > > at all. > > > > Yes however as I pointed out.. Just Trying things on > > a production database is not desirable without some > > serious indicators that its worthwhile. Last you wrote > > you said you doubted that would do anything I believe. > > Also support for amd64 based mysql is listed as > > Limited. As you said reproducing the problem is the > > key. > > However as I suggested I would have thought that if > > mysql were really into solving the problem, someone > > would have requested a login on the box to look at our > > queries to see how they are. Are they 60% reads 40% > > writes, are they many divergent queries bundled > > together.. etc. IE come and see our production > > database in action to see what needs to be replicated. > > > > I haven't seen anything like this. Now I don't know > > much but to me if I can't replicate something it's > > becouse I don't know enough about it. > > > > > > Can anyone help or offer assistance to help track > > > this down? Perhaps > > > > also any annecdotes or examples I can show my boss > > > that other people > > > > have as busy MYSQL databases on BSD 5.X. We paid > > > 3K to Mysql for > > > > help and so far they have been unable to offer any > > > clues as to why > > > > ours will not stay stable on anything but Linux > > > threads. > > > > > > Have you had any kind of crash under 4.x? I don't > > > think that the > > > issue is so much linuxthreads as 5.x. > > > > Well. > > 5.1.. Old lib style No crashes > > 5.4 .. Crashes like mad > > 5.4 with libmap.conf to other threads works better.. > > 4.11 with Linux threads.. No Crashes. > > > > So yea.. But thats the other issue that makes it > > hard. Is it Mysql or is it Freebsd or the interaction > > and memory sharing going on. I hope that someone here > > who deals with these things might actually read these > > emails. Its just very frustrating. > > > > > > As I say I only manage the server, I don't program > > > the databases. Is > > > > there anything I should/could look for database > > > wise that could > > > > trigger such things? > > > > > > So far we've had the machine up in-house and have > > > not reproduced the > > > problem. If you have a spare machine that we could > > > run under more > > > typical conditions on your premises, this might > > > help. > > > > We have one machine still running 5.4. But what to do > > with it? I would love to convert it to amd64 but we > > need some machine to still 5.4 so we can have > > something to test. > > > > > On Wednesday, 12 October 2005 at 17:07:57 -0400, > > > Kris Kennaway wrote: > > > > > > > > Unfortunately you'll need to provide details of > > > how it "fell over" > > > > (e.g. panic messages + backtraces). > > > > I didn't see this email. > > We did supply all that to you (greg and Mysql) as far > > as I know. If you could share what is needed that > > would be appreciatted. > > > > > > > As I mention above, I think this is only a server > > > crash. I mentioned > > > this on the list a couple of weeks ago: all the > > > backtraces I have seen > > > have been a SIGSEGV out of mutex_unlock_common. > > > > Your opinion on this given what you have seen of our > > issue? > > > > > >> Can anyone help or offer assistance to help track > > > this down? > > > >> Perhaps also any annecdotes or examples I can > > > show my boss that > > > >> other people have as busy MYSQL databases on BSD > > > 5.X. We paid 3K to > > > >> Mysql for help and so far they have been unable > > > to offer any clues > > > >> as to why ours will not stay stable on anything > > > but Linux > > > >> threads. I feel really sad that so far no one has > > > responded to my > > > >> posts and it feels like a victory for linux. > > > > > > > > If I was your boss I'd be asking why mysql hasn't > > > delivered on their > > > > support contract. > > > > > > Indeed. I think we have, though. There's a certain > > > class of bugs > > > which are almost impossible to fix because they're > > > so hard to chase > > > down. This is one of them. > > > > Lucky us. :( > > But does that also mean that anyone else running 5.4 > > and Mysql will likely run into this when they reach > > our level of activity? > > > > > On Wednesday, 12 October 2005 at 14:41:54 -0700, NMH > > > wrote: > > > > --- Kris Kennaway wrote: > > > > > > > >> If I was your boss I'd be asking why mysql hasn't > > > delivered on > > > >> their support contract. > > > > > > > > > > > Well I think support has many meanings. I decided > > > to look at what he > > > > paid for and it says: > > > > We get "access" to the mysql devlopers... > > > > We get "access" to certified binaries. (none of > > > which > > > > are FreeBSD) > > > > > > > > So, sad as it may seem, if your running on > > > FreeBSD, $3000.00 buys > > > > you someone to talk to. It doesn't mean they have > > > to say anything > > > > meaningful back or within any reasonable time. :( > > > > > > We try to handle all problems within a reasonable > > > time. The fact that > > > you're running FreeBSD does mean that you don't get > > > certified > > > binaries, but that's the only drawback. And the > > > fact that the time > > > for this problem has been unreasonable has nothing > > > to do with the fact > > > you're running FreeBSD: it's because it's a bugger > > > to track down. > > > > Right. Which is why I am here. Hoping to widing the > > audience and ask for help rather than having to switch > > to Linux to get this solved. I didn't see you posting > > anything to ask others so I thought I would. As I said > > sometimes the gift and curse of FreeBSD is most of its > > users not needing or relying on commercial support for > > hardware and software things. So we can be overlooked > > as a market. > > > > Greg. if you wouldn't mind sharing the info with Kris > > I would appreciate it. If there is anything else > > needed just ask. > > > > > > Be well! > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" > if you are using mysql from the ports you might wanna try to compile it wit= h the option "WITH_LINUXTHREADS=3Dyes" and if you are already using that option try it without and we cant say for sure what the problem is if you dont give use an output of 'gdb' with 'backtrace' From owner-freebsd-database@FreeBSD.ORG Thu Nov 10 12:17:10 2005 Return-Path: X-Original-To: freebsd-database@freebsd.org Delivered-To: freebsd-database@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AACA016A41F for ; Thu, 10 Nov 2005 12:17:10 +0000 (GMT) (envelope-from zparta@gmail.com) Received: from zproxy.gmail.com (zproxy.gmail.com [64.233.162.194]) by mx1.FreeBSD.org (Postfix) with ESMTP id BFA2543D48 for ; Thu, 10 Nov 2005 12:17:08 +0000 (GMT) (envelope-from zparta@gmail.com) Received: by zproxy.gmail.com with SMTP id 9so374810nzo for ; Thu, 10 Nov 2005 04:17:08 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=IWsRtsawmVda+etYccr2F0AOxBDQ4OVkFSgMotn2zeHgAf9dX7+5FmsProSTlnsHB8SL5lE1zxDlnCbs6oiHu58gMgba7v6w3MmcTNJt8+IvKhh2c9x4YwBaIAIi3B8wxrtkzbmnaMQxOhnYXxOdbYgu+uHJn8AnmNqsfx/83kA= Received: by 10.36.221.32 with SMTP id t32mr457605nzg; Thu, 10 Nov 2005 04:17:08 -0800 (PST) Received: by 10.37.18.50 with HTTP; Thu, 10 Nov 2005 04:17:07 -0800 (PST) Message-ID: <3b41db850511100417u542c24e0q935af89f9ba78582@mail.gmail.com> Date: Thu, 10 Nov 2005 13:17:07 +0100 From: Jens Holmqvist To: Justin Bastedo In-Reply-To: <8a5255240511091416i449bc233s4706f26c29002284@mail.gmail.com> MIME-Version: 1.0 References: <20051013025632.GN49168@wantadilla.lemis.com> <20051013043115.4693.qmail@web32913.mail.mud.yahoo.com> <8a5255240511091416i449bc233s4706f26c29002284@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: FreeBSD-DataBase , questions Subject: Re: Mysql server not able to stay running on anything but Linux? X-BeenThere: freebsd-database@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Database use and development under FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Nov 2005 12:17:10 -0000 On 11/9/05, Justin Bastedo wrote: > > Hey everyone, > Not trying to beat a dead horse or anything but was wondering if any > resolution was found for this on either end. We are about to launch > another site and can easily forsee my load getting close to this size > within 6 months. Just need to find out any concerns with growth we > might have and come up with options. > > Just trying to plan ahead and keep problems from happening, since I > hadn't seen any formal resolution on this thread figured I'd inquire. > > Thanks everyone > > Justin Bastedo > > On 10/12/05, NMH wrote: > > --- Greg 'groggy' Lehey wrote: > > > > > Note the From: address. > > > > > > On Wednesday, 12 October 2005 at 13:16:22 -0700, NMH > > > wrote: > > > > > > > > I am stuck with a delema and I feel like a damn > > > troll. But.. I have > > > > a Mysql Database that I posted about earlier. It > > > seems that it is > > > > only able to not die by running on BSD 4.11 with > > > Linux Threads. My > > > > boss is convinced this means that Linux is better > > > for MySQL and > > > > wants that installed now. > > > > > > > > We even got a support contact from Mysql that so > > > far has gotten us > > > > nothing for almost a month while our production > > > database server died > > > > up to 3 times a day. (and lots of we're looking > > > into it's) > > > > > > One of the reasons why you haven't got much more > > > than "we're looking > > > into it"s is because we haven't been able to > > > reproduce the problem; > > > you acknowledge this in follow-up mail quoted below. > > > > > > As you know from various threads on the FreeBSD > > > lists, including this > > > one, the typical answer is "works fine for me". > > > That doesn't mean > > > that we're not taking your problems seriously, but > > > we do have a > > > significant issue just reproducing the problem. We > > > have a number of > > > choices: > > > > > > 1. Try different hardware or a different version of > > > FreeBSD. It's > > > conceivable that there's something about your > > > specific hardware, > > > or about the combination of i386 kernel on amd64 > > > in general, that > > > triggers the problem. > > > > > > Yes possible. However the same hardware was used for > > the earlier version machine that worked fine. IE > > Freebsd 5.1-RELEASE-P11 and Mysql 4.0.18 worked on the > > same hardware. > > > > > 2. Do debugging on your production servers. This > > > isn't really a > > > choice at all: it would involve even more down > > > time. > > > > Yea not really an option. > > > > > 3. Get you to run a more stable version of FreeBSD > > > while we > > > investigate the problem. This is the method we > > > chose. I haven't > > > heard from you since the weekend, so I hope I'm > > > correct in > > > understanding that you currently don't have > > > stability problems. > > > On our side, we have installed FreeBSD 5.4 on > > > one of our internal > > > machines, and we're trying to reproduce the > > > problem there. > > > > So far we have had only one crash that seemed to have > > been SCSI related. So far it has not happend again. > > One problem with replication that was a coding issue. > > > > > > > > We were running fine but a little slow on FreeBSD > > > 5.1-P11 and MySQL > > > > 4.0.18.(apperantly before a big Lib change) We had > > > to move quicker > > > > than we wanted to a new server running FreeBSD 5.4 > > > and MySQL 4.11 > > > > (becouse of a dual HD death) Under production load > > > the new 5.4 > > > > server fell over regulary. It has only now become > > > stable by wiping > > > > it and running it on FreeBSD 4.11 with Linux > > > Threads. (it regularly > > > > has over 400+ threads) > > > > > > Kris obviously understood that by this statement you > > > meant a kernel > > > crash. My understanding is that only the mysqld > > > server is crashing. > > > Is this still correct? > > > > Yes only Mysqld would crash. Sometimes brb and > > autorestart with minimal damage. Other times it would > > die a horrible death and damage tons of data on its > > way out. > > > > > > I want to try FreeBSD 5.4 AMD64 (the machines are > > > Opteron) or 6.0 > > > > but my boss feels that would be a waste given that > > > MYSQL doesn't > > > > support Mysql on AMD64 well enough. > > > > > > I think it would be a good idea to try this. It's > > > one of the things > > > that we intend to do in-house as soon as we can > > > reproduce the problem > > > at all. > > > > Yes however as I pointed out.. Just Trying things on > > a production database is not desirable without some > > serious indicators that its worthwhile. Last you wrote > > you said you doubted that would do anything I believe. > > Also support for amd64 based mysql is listed as > > Limited. As you said reproducing the problem is the > > key. > > However as I suggested I would have thought that if > > mysql were really into solving the problem, someone > > would have requested a login on the box to look at our > > queries to see how they are. Are they 60% reads 40% > > writes, are they many divergent queries bundled > > together.. etc. IE come and see our production > > database in action to see what needs to be replicated. > > > > I haven't seen anything like this. Now I don't know > > much but to me if I can't replicate something it's > > becouse I don't know enough about it. > > > > > > Can anyone help or offer assistance to help track > > > this down? Perhaps > > > > also any annecdotes or examples I can show my boss > > > that other people > > > > have as busy MYSQL databases on BSD 5.X. We paid > > > 3K to Mysql for > > > > help and so far they have been unable to offer any > > > clues as to why > > > > ours will not stay stable on anything but Linux > > > threads. > > > > > > Have you had any kind of crash under 4.x? I don't > > > think that the > > > issue is so much linuxthreads as 5.x. > > > > Well. > > 5.1.. Old lib style No crashes > > 5.4 .. Crashes like mad > > 5.4 with libmap.conf to other threads works better.. > > 4.11 with Linux threads.. No Crashes. > > > > So yea.. But thats the other issue that makes it > > hard. Is it Mysql or is it Freebsd or the interaction > > and memory sharing going on. I hope that someone here > > who deals with these things might actually read these > > emails. Its just very frustrating. > > > > > > As I say I only manage the server, I don't program > > > the databases. Is > > > > there anything I should/could look for database > > > wise that could > > > > trigger such things? > > > > > > So far we've had the machine up in-house and have > > > not reproduced the > > > problem. If you have a spare machine that we could > > > run under more > > > typical conditions on your premises, this might > > > help. > > > > We have one machine still running 5.4. But what to do > > with it? I would love to convert it to amd64 but we > > need some machine to still 5.4 so we can have > > something to test. > > > > > On Wednesday, 12 October 2005 at 17:07:57 -0400, > > > Kris Kennaway wrote: > > > > > > > > Unfortunately you'll need to provide details of > > > how it "fell over" > > > > (e.g. panic messages + backtraces). > > > > I didn't see this email. > > We did supply all that to you (greg and Mysql) as far > > as I know. If you could share what is needed that > > would be appreciatted. > > > > > > > As I mention above, I think this is only a server > > > crash. I mentioned > > > this on the list a couple of weeks ago: all the > > > backtraces I have seen > > > have been a SIGSEGV out of mutex_unlock_common. > > > > Your opinion on this given what you have seen of our > > issue? > > > > > >> Can anyone help or offer assistance to help track > > > this down? > > > >> Perhaps also any annecdotes or examples I can > > > show my boss that > > > >> other people have as busy MYSQL databases on BSD > > > 5.X. We paid 3K to > > > >> Mysql for help and so far they have been unable > > > to offer any clues > > > >> as to why ours will not stay stable on anything > > > but Linux > > > >> threads. I feel really sad that so far no one has > > > responded to my > > > >> posts and it feels like a victory for linux. > > > > > > > > If I was your boss I'd be asking why mysql hasn't > > > delivered on their > > > > support contract. > > > > > > Indeed. I think we have, though. There's a certain > > > class of bugs > > > which are almost impossible to fix because they're > > > so hard to chase > > > down. This is one of them. > > > > Lucky us. :( > > But does that also mean that anyone else running 5.4 > > and Mysql will likely run into this when they reach > > our level of activity? > > > > > On Wednesday, 12 October 2005 at 14:41:54 -0700, NMH > > > wrote: > > > > --- Kris Kennaway wrote: > > > > > > > >> If I was your boss I'd be asking why mysql hasn't > > > delivered on > > > >> their support contract. > > > > > > > > > > > Well I think support has many meanings. I decided > > > to look at what he > > > > paid for and it says: > > > > We get "access" to the mysql devlopers... > > > > We get "access" to certified binaries. (none of > > > which > > > > are FreeBSD) > > > > > > > > So, sad as it may seem, if your running on > > > FreeBSD, $3000.00 buys > > > > you someone to talk to. It doesn't mean they have > > > to say anything > > > > meaningful back or within any reasonable time. :( > > > > > > We try to handle all problems within a reasonable > > > time. The fact that > > > you're running FreeBSD does mean that you don't get > > > certified > > > binaries, but that's the only drawback. And the > > > fact that the time > > > for this problem has been unreasonable has nothing > > > to do with the fact > > > you're running FreeBSD: it's because it's a bugger > > > to track down. > > > > Right. Which is why I am here. Hoping to widing the > > audience and ask for help rather than having to switch > > to Linux to get this solved. I didn't see you posting > > anything to ask others so I thought I would. As I said > > sometimes the gift and curse of FreeBSD is most of its > > users not needing or relying on commercial support for > > hardware and software things. So we can be overlooked > > as a market. > > > > Greg. if you wouldn't mind sharing the info with Kris > > I would appreciate it. If there is anything else > > needed just ask. > > > > > > Be well! > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" if you are using mysql from the ports you might wanna try to compile it wit= h the option "WITH_LINUXTHREADS=3Dyes" and if you are already using that option try it without and we cant say for sure what the problem is if you dont give use an output of 'gdb' with 'backtrace' From owner-freebsd-database@FreeBSD.ORG Thu Nov 10 19:02:18 2005 Return-Path: X-Original-To: freebsd-database@freebsd.org Delivered-To: freebsd-database@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 314DD16A41F for ; Thu, 10 Nov 2005 19:02:18 +0000 (GMT) (envelope-from justin.bastedo@gmail.com) Received: from xproxy.gmail.com (xproxy.gmail.com [66.249.82.203]) by mx1.FreeBSD.org (Postfix) with ESMTP id 03AA643D68 for ; Thu, 10 Nov 2005 19:02:06 +0000 (GMT) (envelope-from justin.bastedo@gmail.com) Received: by xproxy.gmail.com with SMTP id t12so338837wxc for ; Thu, 10 Nov 2005 11:02:06 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=eR+JH2m6K101qa83cv3nm+fWwYtL4BFeFieuBMl1O0siq7ERI2EQLP7UurAtQSkJ2OeNPw0MYiT+EwHWF3DXKKp/87D7kZi/GoaZAqdMHwLTiBDvDjb0gpvTXKux0cgqWNUSv0HqoNs8f/lOjKKgm4ZaMa9JdEYJER0Gjh9eRuQ= Received: by 10.70.44.1 with SMTP id r1mr25797wxr; Wed, 09 Nov 2005 14:16:50 -0800 (PST) Received: by 10.70.36.9 with HTTP; Wed, 9 Nov 2005 14:16:50 -0800 (PST) Message-ID: <8a5255240511091416i449bc233s4706f26c29002284@mail.gmail.com> Date: Wed, 9 Nov 2005 14:16:50 -0800 From: Justin Bastedo To: NMH In-Reply-To: <20051013043115.4693.qmail@web32913.mail.mud.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <20051013025632.GN49168@wantadilla.lemis.com> <20051013043115.4693.qmail@web32913.mail.mud.yahoo.com> Cc: FreeBSD-DataBase , Kris Kennaway , questions Subject: Re: Mysql server not able to stay running on anything but Linux? X-BeenThere: freebsd-database@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Database use and development under FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Nov 2005 19:02:18 -0000 Hey everyone, Not trying to beat a dead horse or anything but was wondering if any resolution was found for this on either end. We are about to launch another site and can easily forsee my load getting close to this size within 6 months. Just need to find out any concerns with growth we might have and come up with options. Just trying to plan ahead and keep problems from happening, since I hadn't seen any formal resolution on this thread figured I'd inquire. Thanks everyone Justin Bastedo On 10/12/05, NMH wrote: > --- Greg 'groggy' Lehey wrote: > > > Note the From: address. > > > > On Wednesday, 12 October 2005 at 13:16:22 -0700, NMH > > wrote: > > > > > > I am stuck with a delema and I feel like a damn > > troll. But.. I have > > > a Mysql Database that I posted about earlier. It > > seems that it is > > > only able to not die by running on BSD 4.11 with > > Linux Threads. My > > > boss is convinced this means that Linux is better > > for MySQL and > > > wants that installed now. > > > > > > We even got a support contact from Mysql that so > > far has gotten us > > > nothing for almost a month while our production > > database server died > > > up to 3 times a day. (and lots of we're looking > > into it's) > > > > One of the reasons why you haven't got much more > > than "we're looking > > into it"s is because we haven't been able to > > reproduce the problem; > > you acknowledge this in follow-up mail quoted below. > > > > As you know from various threads on the FreeBSD > > lists, including this > > one, the typical answer is "works fine for me". > > That doesn't mean > > that we're not taking your problems seriously, but > > we do have a > > significant issue just reproducing the problem. We > > have a number of > > choices: > > > > 1. Try different hardware or a different version of > > FreeBSD. It's > > conceivable that there's something about your > > specific hardware, > > or about the combination of i386 kernel on amd64 > > in general, that > > triggers the problem. > > > Yes possible. However the same hardware was used for > the earlier version machine that worked fine. IE > Freebsd 5.1-RELEASE-P11 and Mysql 4.0.18 worked on the > same hardware. > > > 2. Do debugging on your production servers. This > > isn't really a > > choice at all: it would involve even more down > > time. > > Yea not really an option. > > > 3. Get you to run a more stable version of FreeBSD > > while we > > investigate the problem. This is the method we > > chose. I haven't > > heard from you since the weekend, so I hope I'm > > correct in > > understanding that you currently don't have > > stability problems. > > On our side, we have installed FreeBSD 5.4 on > > one of our internal > > machines, and we're trying to reproduce the > > problem there. > > So far we have had only one crash that seemed to have > been SCSI related. So far it has not happend again. > One problem with replication that was a coding issue. > > > > > We were running fine but a little slow on FreeBSD > > 5.1-P11 and MySQL > > > 4.0.18.(apperantly before a big Lib change) We had > > to move quicker > > > than we wanted to a new server running FreeBSD 5.4 > > and MySQL 4.11 > > > (becouse of a dual HD death) Under production load > > the new 5.4 > > > server fell over regulary. It has only now become > > stable by wiping > > > it and running it on FreeBSD 4.11 with Linux > > Threads. (it regularly > > > has over 400+ threads) > > > > Kris obviously understood that by this statement you > > meant a kernel > > crash. My understanding is that only the mysqld > > server is crashing. > > Is this still correct? > > Yes only Mysqld would crash. Sometimes brb and > autorestart with minimal damage. Other times it would > die a horrible death and damage tons of data on its > way out. > > > > I want to try FreeBSD 5.4 AMD64 (the machines are > > Opteron) or 6.0 > > > but my boss feels that would be a waste given that > > MYSQL doesn't > > > support Mysql on AMD64 well enough. > > > > I think it would be a good idea to try this. It's > > one of the things > > that we intend to do in-house as soon as we can > > reproduce the problem > > at all. > > Yes however as I pointed out.. Just Trying things on > a production database is not desirable without some > serious indicators that its worthwhile. Last you wrote > you said you doubted that would do anything I believe. > Also support for amd64 based mysql is listed as > Limited. As you said reproducing the problem is the > key. > However as I suggested I would have thought that if > mysql were really into solving the problem, someone > would have requested a login on the box to look at our > queries to see how they are. Are they 60% reads 40% > writes, are they many divergent queries bundled > together.. etc. IE come and see our production > database in action to see what needs to be replicated. > > I haven't seen anything like this. Now I don't know > much but to me if I can't replicate something it's > becouse I don't know enough about it. > > > > Can anyone help or offer assistance to help track > > this down? Perhaps > > > also any annecdotes or examples I can show my boss > > that other people > > > have as busy MYSQL databases on BSD 5.X. We paid > > 3K to Mysql for > > > help and so far they have been unable to offer any > > clues as to why > > > ours will not stay stable on anything but Linux > > threads. > > > > Have you had any kind of crash under 4.x? I don't > > think that the > > issue is so much linuxthreads as 5.x. > > Well. > 5.1.. Old lib style No crashes > 5.4 .. Crashes like mad > 5.4 with libmap.conf to other threads works better.. > 4.11 with Linux threads.. No Crashes. > > So yea.. But thats the other issue that makes it > hard. Is it Mysql or is it Freebsd or the interaction > and memory sharing going on. I hope that someone here > who deals with these things might actually read these > emails. Its just very frustrating. > > > > As I say I only manage the server, I don't program > > the databases. Is > > > there anything I should/could look for database > > wise that could > > > trigger such things? > > > > So far we've had the machine up in-house and have > > not reproduced the > > problem. If you have a spare machine that we could > > run under more > > typical conditions on your premises, this might > > help. > > We have one machine still running 5.4. But what to do > with it? I would love to convert it to amd64 but we > need some machine to still 5.4 so we can have > something to test. > > > On Wednesday, 12 October 2005 at 17:07:57 -0400, > > Kris Kennaway wrote: > > > > > > Unfortunately you'll need to provide details of > > how it "fell over" > > > (e.g. panic messages + backtraces). > > I didn't see this email. > We did supply all that to you (greg and Mysql) as far > as I know. If you could share what is needed that > would be appreciatted. > > > > As I mention above, I think this is only a server > > crash. I mentioned > > this on the list a couple of weeks ago: all the > > backtraces I have seen > > have been a SIGSEGV out of mutex_unlock_common. > > Your opinion on this given what you have seen of our > issue? > > > >> Can anyone help or offer assistance to help track > > this down? > > >> Perhaps also any annecdotes or examples I can > > show my boss that > > >> other people have as busy MYSQL databases on BSD > > 5.X. We paid 3K to > > >> Mysql for help and so far they have been unable > > to offer any clues > > >> as to why ours will not stay stable on anything > > but Linux > > >> threads. I feel really sad that so far no one has > > responded to my > > >> posts and it feels like a victory for linux. > > > > > > If I was your boss I'd be asking why mysql hasn't > > delivered on their > > > support contract. > > > > Indeed. I think we have, though. There's a certain > > class of bugs > > which are almost impossible to fix because they're > > so hard to chase > > down. This is one of them. > > Lucky us. :( > But does that also mean that anyone else running 5.4 > and Mysql will likely run into this when they reach > our level of activity? > > > On Wednesday, 12 October 2005 at 14:41:54 -0700, NMH > > wrote: > > > --- Kris Kennaway wrote: > > > > > >> If I was your boss I'd be asking why mysql hasn't > > delivered on > > >> their support contract. > > > > > > > > Well I think support has many meanings. I decided > > to look at what he > > > paid for and it says: > > > We get "access" to the mysql devlopers... > > > We get "access" to certified binaries. (none of > > which > > > are FreeBSD) > > > > > > So, sad as it may seem, if your running on > > FreeBSD, $3000.00 buys > > > you someone to talk to. It doesn't mean they have > > to say anything > > > meaningful back or within any reasonable time. :( > > > > We try to handle all problems within a reasonable > > time. The fact that > > you're running FreeBSD does mean that you don't get > > certified > > binaries, but that's the only drawback. And the > > fact that the time > > for this problem has been unreasonable has nothing > > to do with the fact > > you're running FreeBSD: it's because it's a bugger > > to track down. > > Right. Which is why I am here. Hoping to widing the > audience and ask for help rather than having to switch > to Linux to get this solved. I didn't see you posting > anything to ask others so I thought I would. As I said > sometimes the gift and curse of FreeBSD is most of its > users not needing or relying on commercial support for > hardware and software things. So we can be overlooked > as a market. > > Greg. if you wouldn't mind sharing the info with Kris > I would appreciate it. If there is anything else > needed just ask. > > > Be well! From owner-freebsd-database@FreeBSD.ORG Fri Nov 11 09:33:25 2005 Return-Path: X-Original-To: freebsd-database@freebsd.org Delivered-To: freebsd-database@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 29E5816A41F for ; Fri, 11 Nov 2005 09:33:25 +0000 (GMT) (envelope-from dryice@hotpop.com) Received: from smtp-out.hotpop.com (smtp-out.hotpop.com [38.113.3.61]) by mx1.FreeBSD.org (Postfix) with ESMTP id C930643D45 for ; Fri, 11 Nov 2005 09:33:24 +0000 (GMT) (envelope-from dryice@hotpop.com) Received: from hotpop.com (kubrick.hotpop.com [38.113.3.105]) by smtp-out.hotpop.com (Postfix) with SMTP id E38E41CCAE0F for ; Fri, 11 Nov 2005 09:33:14 +0000 (UTC) Received: from dryice.3322.org (unknown [221.0.236.221]) by smtp-2.hotpop.com (Postfix) with ESMTP id 73B481C8443B for ; Fri, 11 Nov 2005 09:26:34 +0000 (UTC) To: freebsd-database@freebsd.org From: Dryice Liu Date: Fri, 11 Nov 2005 17:26:05 +0800 Message-ID: <8664qzcz82.fsf@dryice.3322.org> User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/23.0.0 (berkeley-unix) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha1; protocol="application/pgp-signature" X-HotPOP: ----------------------------------------------- Sent By HotPOP.com FREE Email Get your FREE POP email at www.HotPOP.com ----------------------------------------------- Subject: suggested block size for a frequently updated postgresql? X-BeenThere: freebsd-database@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Database use and development under FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Nov 2005 09:33:25 -0000 --=-=-= Content-Transfer-Encoding: quoted-printable I'm planning on setting up a postgresql server, the database data files are on their own slice so I can tune the file system for pgsql. The database is pretty large. On my test server, some data files get larger than 1G and is splitted by pgsql. Also the database will be updated frequently. I'm planning on setting the slice with a bigger block size/fragment size but not sure if that's a good idea. I know the default on FreeBSD is 16K/2K, I'm planning on something like 1M/128K. The most concern is how the database files get updated? I heard that most database do update/delete by appending a mark and new record at the end of the data file but can't find any evidence for that. If that's true, then the thing looks good. But if the database update the data file by changing the original file block, setting a bigger block size may bring extra effort for the drive, especially when it's a RAID5 and a checksum build is required. Can someone shine some light on this? Thanks! =2D-=20 Dryice @ http://dryice.3322.org Please avoid sending me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/sylvester-response.html --=-=-= Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (FreeBSD) iD8DBQBDdGO3a1t4qHe2eHQRAkszAJ0bclnu1HRXwVrXmeeFulgjmMK99gCeK3Yl X2zpG6s0vLUKfrbLX5LjPEM= =UVJp -----END PGP SIGNATURE----- --=-=-=-- From owner-freebsd-database@FreeBSD.ORG Fri Nov 11 18:24:07 2005 Return-Path: X-Original-To: freebsd-database@FreeBSD.ORG Delivered-To: freebsd-database@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9C6FD16A41F for ; Fri, 11 Nov 2005 18:24:07 +0000 (GMT) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (lurza.secnetix.de [83.120.8.8]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6E14D43D49 for ; Fri, 11 Nov 2005 18:24:06 +0000 (GMT) (envelope-from olli@lurza.secnetix.de) Received: from lurza.secnetix.de (cdsdyb@localhost [127.0.0.1]) by lurza.secnetix.de (8.13.1/8.13.1) with ESMTP id jABIO4wW074289 for ; Fri, 11 Nov 2005 19:24:05 +0100 (CET) (envelope-from oliver.fromme@secnetix.de) Received: (from olli@localhost) by lurza.secnetix.de (8.13.1/8.13.1/Submit) id jABIO4Sh074288; Fri, 11 Nov 2005 19:24:04 +0100 (CET) (envelope-from olli) Date: Fri, 11 Nov 2005 19:24:04 +0100 (CET) Message-Id: <200511111824.jABIO4Sh074288@lurza.secnetix.de> From: Oliver Fromme To: freebsd-database@FreeBSD.ORG In-Reply-To: <8664qzcz82.fsf@dryice.3322.org> X-Newsgroups: list.freebsd-database User-Agent: tin/1.5.4-20000523 ("1959") (UNIX) (FreeBSD/4.11-RELEASE (i386)) Cc: Subject: Re: suggested block size for a frequently updated postgresql? X-BeenThere: freebsd-database@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-database@FreeBSD.ORG List-Id: Database use and development under FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Nov 2005 18:24:07 -0000 Dryice Liu wrote: > I'm planning on setting up a postgresql server, the database data > files are on their own slice so I can tune the file system for pgsql. > > The database is pretty large. On my test server, some data files get > larger than 1G and is splitted by pgsql. Also the database will be > updated frequently. > > I'm planning on setting the slice with a bigger block size/fragment > size but not sure if that's a good idea. I know the default on FreeBSD > is 16K/2K, I'm planning on something like 1M/128K. FreeBSD doesn't support bsize/fsize that large. In fact, I wouldn't trust anything other than 16K/2K and 8K/1K to work reliably under load. I also remember Matt Dillon mentioned that larger block sizes are less-than-optimal for the VM system. Besides, I don't think that PostgreSQL would benefit from such huge fragment and block sizes. Let me explain ... PostgreSQL does two things on the file system: First, it records all modifications to the WAL file. This is a sequential process, and I wouldn't expect the blocksize to have any significant effect. Second, it updates the actual table data in the data files. This consists of appending data to the files and random access inside the files, which would definitely not benefit from a larger block size. Also take into account that the regular "vacuum" process of postgresql is random access inside the data files. For more details on that I recommend you ask in the Post- greSQL list (pgsql-novice). There are very knowledgeable and responsive people there. So, the bottom line is: I recommend you leave the bsize and fsize at the default 16K/2K. However, you probably should reduce the inode density from the default, i.e. use the -i option with some high value such as 262144 (that's 2^18). But don't make this value too high either ... I remember someone tried to set it to 64 million or some- thing, which broke his FS. Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd Any opinions expressed in this message may be personal to the author and may not necessarily reflect the opinions of secnetix in any way. "With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead." -- RFC 1925 From owner-freebsd-database@FreeBSD.ORG Sat Nov 12 02:34:46 2005 Return-Path: X-Original-To: freebsd-database@freebsd.org Delivered-To: freebsd-database@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2410716A41F for ; Sat, 12 Nov 2005 02:34:46 +0000 (GMT) (envelope-from dryice@hotpop.com) Received: from smtp-out.hotpop.com (smtp-out.hotpop.com [38.113.3.71]) by mx1.FreeBSD.org (Postfix) with ESMTP id B7C0043D45 for ; Sat, 12 Nov 2005 02:34:45 +0000 (GMT) (envelope-from dryice@hotpop.com) Received: from hotpop.com (kubrick.hotpop.com [38.113.3.105]) by smtp-out.hotpop.com (Postfix) with SMTP id 8EF021A80201 for ; Sat, 12 Nov 2005 02:34:42 +0000 (UTC) Received: from dryice.3322.org (unknown [221.0.236.221]) by smtp-1.hotpop.com (Postfix) with ESMTP id ABBC01A0265 for ; Sat, 12 Nov 2005 02:34:35 +0000 (UTC) To: freebsd-database@freebsd.org References: <200511111824.jABIO4Sh074288@lurza.secnetix.de> From: Dryice Liu Date: Sat, 12 Nov 2005 10:32:52 +0800 In-Reply-To: <200511111824.jABIO4Sh074288@lurza.secnetix.de> (Oliver Fromme's message of "Fri, 11 Nov 2005 19:24:04 +0100 (CET)") Message-ID: <86ek5msii3.fsf@dryice.3322.org> User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/23.0.0 (berkeley-unix) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha1; protocol="application/pgp-signature" X-HotPOP: ----------------------------------------------- Sent By HotPOP.com FREE Email Get your FREE POP email at www.HotPOP.com ----------------------------------------------- Subject: Re: suggested block size for a frequently updated postgresql? X-BeenThere: freebsd-database@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Database use and development under FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Nov 2005 02:34:46 -0000 --=-=-= Content-Transfer-Encoding: quoted-printable Oliver Fromme wrote: > Dryice Liu wrote: > > I'm planning on setting up a postgresql server, the database data > > files are on their own slice so I can tune the file system for pgsql. > >=20 > > The database is pretty large. On my test server, some data files get > > larger than 1G and is splitted by pgsql. Also the database will be > > updated frequently. > >=20 > > I'm planning on setting the slice with a bigger block size/fragment > > size but not sure if that's a good idea. I know the default on FreeBSD > > is 16K/2K, I'm planning on something like 1M/128K. > > So, the bottom line is: I recommend you leave the bsize > and fsize at the default 16K/2K. However, you probably > should reduce the inode density from the default, i.e. use > the -i option with some high value such as 262144 (that's > 2^18). But don't make this value too high either ... > I remember someone tried to set it to 64 million or some- > thing, which broke his FS. Thanks for the detail explanation and suggestion! =2D-=20 Dryice @ http://dryice.3322.org Please avoid sending me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/sylvester-response.html --=-=-= Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (FreeBSD) iD8DBQBDdVSZa1t4qHe2eHQRAsy2AJ4hY0TgXjeO5tLQZERgnBtbBGuqQgCfT5+M RWfSgzQ6PhAdhJOS5rR8xog= =IH8u -----END PGP SIGNATURE----- --=-=-=--