Date: Tue, 11 Jan 2000 04:08:22 +0200 (EET) From: Michael Widenius <monty@monty.pp.sci.fi> To: "Scott Hess" <scott@avantgo.com> Cc: <freebsd-hackers@freebsd.org>, <developer@lists.mysql.com> Subject: Concept check: iothreads addition to pthreads for MYSQL+FreeBSD. Message-ID: <14458.37014.846894.473677@monty.pp.sci.fi> In-Reply-To: <1a6101bf5bc1$4e364b20$1e80000a@avantgo.com> References: <1a6101bf5bc1$4e364b20$1e80000a@avantgo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
>>>>> "Scott" == Scott Hess <scott@avantgo.com> writes: Scott> Recently I was tasked to find a way to scale up our MYSQL server, running Scott> MYSQL3.22.15 on FreeBSD3.3. I've been testing a hardware RAID solution, Scott> and found that with 6 disks in a RAID5 configuration, the system was only Scott> perhaps 30% faster than when running on a single disk. [The 6 disks in the Scott> RAID5 are the same model as the single-disk test I was comparing against.] Scott> Experimentation determined that pthreads was the problem. FreeBSD's Scott> implementation of pthreads using a select() loop, and select() always says Scott> that disk I/O is ready to proceed, and disk I/O never return EWOULDBLOCK. Scott> Essentially, pthreads was serializing the MYSQL read() requests, and if the Scott> dataset exceeds memory size, performance becomes entirely seek bound. Scott> I've implemented a rough fix, which is to rfork() processes which I label Scott> "iothreads" to handle the disk I/O. The parent process running pthreads Scott> has a socketpair() to each of the iothreads. The iothreads wait for Scott> requests on the socketpair, and since socketpairs can block, pthreads can Scott> handle them efficiently. This essentially allows me to turn blocking disk Scott> I/O calls into non-blocking calls. Having multiple pending seeks turns out Scott> to be a huge win for MYSQL, allowing it to scale much better as disks are Scott> added to the RAID5 array. Scott> Unfortunately, I'm concerned about using this code in production, because Scott> it needs a fair amount of cleanup to handle signals and administrative Scott> functions correctly. For this reason and others, I'm starting a project to Scott> move it into the pthreads library itself. Before I embark on that effort, Scott> I have a couple questions: Scott> 1) Does this seem like a reasonable approach? [It _works_, and well. But Scott> it tastes strongly of hack.] If you can't get the freebsd team to add blocking reads, then this may one solution to speed this up. Scott> 2) Does anyone have suggestions for a solution that will be cleaner and Scott> won't take man-months to implement? [Which is the redeeming quality of Scott> what I've got - it took me two days to zero in on a very workable Scott> solution.] Everything depends of how neatly you can get this into the FreeBSD kernel and if you can get the FreeBSD developers to incorporate your patch in the offical kernel. If not, then this may be a problem. Scott> 3) Is anyone working on something I might leverage off of in this area? Scott> [For instance, a select()-based interface to async I/O? Or support for Scott> blocking disk I/O in select() and read()?] blocking reads would probably be the best option. Scott> 4) Is there anyone willing to commit to testing my modified pthreads Scott> library against MYSQL? [I'll be stress testing it quite heavily, of Scott> course. It would probably also be testable against Squid with async I/O Scott> and multithreaded Apache 2.0.] Sorry, but I don't have a FreeBSD system available that I can use for testing (and I don't have any real good application to use for testing this). Regards, Monty To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?14458.37014.846894.473677>