From owner-freebsd-hackers Mon Jan 10 16:25:55 2000 Delivered-To: freebsd-hackers@freebsd.org Received: from mycenae.ilion.eu.org (mycenae.ilion.eu.org [203.35.206.129]) by hub.freebsd.org (Postfix) with ESMTP id 05C7F14F05 for ; Mon, 10 Jan 2000 16:25:46 -0800 (PST) (envelope-from patrykz@ilion.eu.org) Received: from mycenae.ilion.eu.org (localhost [127.0.0.1]) by mycenae.ilion.eu.org (8.9.3/8.9.3) with ESMTP id LAA31411; Tue, 11 Jan 2000 11:25:37 +1100 (EST) (envelope-from patrykz@mycenae.ilion.eu.org) Message-Id: <200001110025.LAA31411@mycenae.ilion.eu.org> To: "Scott Hess" Cc: freebsd-hackers@FreeBSD.ORG, developer@lists.mysql.com Subject: Re: Concept check: iothreads addition to pthreads for MYSQL+FreeBSD. In-reply-to: Your message of "Mon, 10 Jan 2000 15:20:38 -0800." <1a6101bf5bc1$4e364b20$1e80000a@avantgo.com> Date: Tue, 11 Jan 2000 11:25:35 +1100 From: Patryk Zadarnowski Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > Recently I was tasked to find a way to scale up our MYSQL server, running > MYSQL3.22.15 on FreeBSD3.3. I've been testing a hardware RAID solution, > and found that with 6 disks in a RAID5 configuration, the system was only > perhaps 30% faster than when running on a single disk. [The 6 disks in the > RAID5 are the same model as the single-disk test I was comparing against.] > > Experimentation determined that pthreads was the problem. FreeBSD's > implementation of pthreads using a select() loop, and select() always says > that disk I/O is ready to proceed, and disk I/O never return EWOULDBLOCK. > Essentially, pthreads was serializing the MYSQL read() requests, and if the > dataset exceeds memory size, performance becomes entirely seek bound. > > I've implemented a rough fix, which is to rfork() processes which I label > "iothreads" to handle the disk I/O. The parent process running pthreads > has a socketpair() to each of the iothreads. The iothreads wait for > requests on the socketpair, and since socketpairs can block, pthreads can > handle them efficiently. This essentially allows me to turn blocking disk > I/O calls into non-blocking calls. Having multiple pending seeks turns out > to be a huge win for MYSQL, allowing it to scale much better as disks are > added to the RAID5 array. > > Unfortunately, I'm concerned about using this code in production, because > it needs a fair amount of cleanup to handle signals and administrative > functions correctly. For this reason and others, I'm starting a project to > move it into the pthreads library itself. Before I embark on that effort, > I have a couple questions: > > 1) Does this seem like a reasonable approach? [It _works_, and well. But > it tastes strongly of hack.] > > 2) Does anyone have suggestions for a solution that will be cleaner and > won't take man-months to implement? [Which is the redeeming quality of > what I've got - it took me two days to zero in on a very workable > solution.] > > 3) Is anyone working on something I might leverage off of in this area? > [For instance, a select()-based interface to async I/O? Or support for > blocking disk I/O in select() and read()?] > > 4) Is there anyone willing to commit to testing my modified pthreads > library against MYSQL? [I'll be stress testing it quite heavily, of > course. It would probably also be testable against Squid with async I/O > and multithreaded Apache 2.0.] I'm no expert on pthreads, but, if you decide to proceed with implementing a mixed user-land/rfork pthread implementation, may I suggest that you implement is through POSIX pthread_attr_setscope() interfaces instead of some local extension. pthread_attr_setscope() sounds like a portable interface to precisely what you're trying to achieve. Pat. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message