Date: Wed, 22 Mar 2006 15:55:44 -0600 From: Glenn Nielsen <glenn@more.net> To: freebsd-database@freebsd.org Subject: MySQL 4.1 SIGBUS failiure with FreeBSD 6.0 Message-ID: <20060322215544.GA9720@gkar.earthdome.org>
next in thread | raw e-mail | index | archive | help
MySQL 4.1 SIGBUS failiure with FreeBSD 6.0 We recently put four FreeBSD 6.0 servers with MySQL 4.1 into production. MySQL on three of these servers is moderately busy. Each requiring between 60-200 threads to handle anywhere from 6-40 queries per second during business hours. One is a Master DB, the other two are RO Slaves. The fourth where we haven't seen any problems is a hotbackup slave. Since going into production three days ago the Master DB has failed twice. The RO Slaves are failing 1-4 times per hour. In all cases the failure is a signal 10 for a SIGBUS error. The mysqld_safe script successfully restarts mysqld after each failure. I am posting on this list first to see if anyone has any ideas of how to resolve this problem before submitting either a FreeBSD or MySQL bug report. I enabled core files with the "--core-file" argument to the start script. I have examined six core files now. All look very similar. Core was generated by `mysqld'. Program terminated with signal 10, Bus error. (gdb) bt #0 0x285292b7 in pthread_testcancel () from /usr/lib/libpthread.so.2 #1 0x285190a2 in sigaction () from /usr/lib/libpthread.so.2 #2 0x2851318d in pthread_kill () from /usr/lib/libpthread.so.2 #3 0x0815a1bf in ?? () #4 0x0841b000 in ?? () #5 0x0000000a in ?? () #6 0xbfbfdee8 in ?? () #7 0x0000000a in ?? () #8 0xbfbfe230 in ?? () #9 0x0000000a in ?? () #10 0xbfbfdef8 in ?? () #11 0x080aaf95 in ?? () #12 0x0000000a in ?? () #13 0x0835b8da in ?? () #14 0x000403fb in ?? () #15 0x00000000 in ?? () #16 0xbfbfdf30 in ?? () #17 0x2852c4b4 in ?? () from /usr/lib/libpthread.so.2 #18 0xbfbfdf28 in ?? () #19 0x28517252 in sigaction () from /usr/lib/libpthread.so.2 (gdb) info threads * 166 Thread 0x841b000 (LWP 100330) 0x285292b7 in pthread_testcancel () from /usr/lib/libpthread.so.2 165 Thread 0x841bc00 (LWP 100188) 0x28529277 in pthread_testcancel () from /usr/lib/libpthread.so.2 164 Thread 0x841be00 (LWP 100206) 0x28529337 in pthread_testcancel () from /usr/lib/libpthread.so.2 163 Thread 0xb22a400 (LWP 100205) 0x28573833 in read () from /lib/libc.so.6 162 Thread 0xb22a600 (LWP 100175) 0x28529277 in pthread_testcancel () from /usr/lib/libpthread.so.2 161 Thread 0xb22a800 (LWP 100211) 0x28529277 in pthread_testcancel () from /usr/lib/libpthread.so.2 160 Thread 0xb22aa00 (LWP 100204) 0x28573833 in read () from /lib/libc.so.6 159 Thread 0xb22ac00 (LWP 100257) 0x28573833 in read () from /lib/libc.so.6 158 Thread 0xb22ae00 (LWP 100258) 0x28573833 in read () from /lib/libc.so.6 157 Thread 0xb2eb000 (LWP 100259) 0x28573833 in read () from /lib/libc.so.6 ... In every case the SIGBUS error is occurring when mysql is killing off some threads. MySQL has a varible for setting the thread_cache_size. By default it is 0, meaning mysql creates threads as needed and kills off threads which are no longer needed almost immediately. We configured the thread_cache_size=50 so that mysql would maintain a pool of threads to reduce the frequency at which it did a thread kill. This has reduced the frequency of failures down to less than once an hour. We are using mysql-server-4.1.18_2 from ports with native threads. Here is some information on the system hardware and kernel config. vai and blackmore: Dell PowerEdge 1850 2x2.8 Ghz P4 1024 MB 2 x 36 GB disks (RAID1) FreeBSD vai.kinetic.more.net 6.0-RELEASE FreeBSD 6.0-RELEASE #0: Mon Jan 30 15:40:24 CST 2006 FreeBSD blackmore.kinetic.more.net 6.0-RELEASE-p5 FreeBSD 6.0-RELEASE-p5 #1: Tue Mar 21 13:44:15 CST 2006 hendrix and satriani: Dell PowerEdge 2650 2 x 2.8 Ghz P4 2048 MB 2 x 36 GB disks (RAID1) FreeBSD satriani.kinetic.more.net 6.0-RELEASE-p4 FreeBSD 6.0-RELEASE-p4 #0: Tue Mar 7 14:32:43 CST 2006 FreeBSD hendrix.kinetic.more.net 6.0-RELEASE-p5 FreeBSD 6.0-RELEASE-p5 #0: Wed Mar 15 10:54:10 CST 2006 The kernel on both platforms is the same (we include GENERIC and add a couple of options): # QUOTAS_SMP include GENERIC ident MOREnet-SMP maxusers 0 options IPFILTER #ipfilter support options IPFILTER_LOG #ipfilter logging # To make an SMP kernel, the next line is needed options SMP # Symmetric MultiProcessor Kernel options QUOTA Hyper Threading is disabled in the BIOS on all four servers. Thanks, Glenn Nielsen ---------------------------------------------------------------------- Glenn Nielsen glenn@more.net | /* Spelin donut madder | MOREnet System Programming | * if iz ina coment. | Missouri Research and Education Network | */ | ----------------------------------------------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060322215544.GA9720>