From owner-freebsd-stable@FreeBSD.ORG Tue May 29 19:34:21 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 21856106564A for ; Tue, 29 May 2012 19:34:21 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-qa0-f49.google.com (mail-qa0-f49.google.com [209.85.216.49]) by mx1.freebsd.org (Postfix) with ESMTP id CCC8B8FC16 for ; Tue, 29 May 2012 19:34:20 +0000 (UTC) Received: by qabj40 with SMTP id j40so1945606qab.15 for ; Tue, 29 May 2012 12:34:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=udPAnZ/ob/kYHli9J0xMiO/4ZR2vVLSvYgjte3MWyXU=; b=noahNHJb2N5wJ2PSn8JI1hK01wWIXe/GypXLpUWxFGTP5G+6rB/UIC6chadHqK18Hp oF25GBaTgsr9Fya0lS6R9n37m0gRzrfEqrm1N9UPyq/fvjkbPI8wqywf8PfNMdHog+oV oEaJXzbH+0ZVw3QheddL3MHGVrAmuiZpNWTvYxMX3ZFDrfOr+AYAuODkJpLYysudCFcb rOoIj+d46xUPVdL/4tFR9MeI8HwXL/SUMX3PhG8uPzQEIbl7HxEiz4KXNEVBJ6V06gSq TjX/kpJtLtP2jKW+ePfCtIh4mEZRXstzRKPSCDXpTBntTq/Q8HAKd/xhsWX+956/6Ux6 kCaw== MIME-Version: 1.0 Received: by 10.224.221.206 with SMTP id id14mr2884512qab.52.1338320060089; Tue, 29 May 2012 12:34:20 -0700 (PDT) Received: by 10.229.20.148 with HTTP; Tue, 29 May 2012 12:34:19 -0700 (PDT) In-Reply-To: References: Date: Tue, 29 May 2012 12:34:19 -0700 Message-ID: From: Freddie Cash To: Kees Jan Koster Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: FreeBSD 9.0 hangs on heavy I/O X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 19:34:21 -0000 On Tue, May 29, 2012 at 12:26 PM, Kees Jan Koster wrot= e: > I seem to have a problem where really heavy disk I/O is drowning my machi= ne. I see hangs in the shell where I am logged on using ssh. Network connec= tions get dropped for no apparent reason and some HTTP requests are served = really slowly. Profiling the app code shows that the hangs are in completel= y random places. Operations that are no more than a few lines of code apart= suddenly take seconds to complete. > > In my search I seem to find that my machine is quite slow on the disk. I = find that rather odd, given that the device in question is an SSD drive and= it is a good bit faster than the WD drive that used to carry the data set = that is accessed heavily. This drive is doing 1.5 times the throughput, but= the hangs have not gone away. > > To clarify, the data set used to live on ada2 (see the devlist below) whi= ch is a spinning disk. When I experienced intermittent hangs I plugged in a= n SSD drive (ada3 on the devlist) and moved the data there. This improved t= he MB's per second that are being written (it is mostly-write data) but has= not changed the hangs. If anything, they got worse since. > > Using gstat I notice that I/O service time is quite high. From the gstat = below you can see that it takes just over 2s to servr the requests. The L(q= ) seems to never drop far below 100 and %busy hovers around 100% all day lo= ng. Can someone please help me troubleshoot that further? What can I do to = make the underlying problem visible? > > I should mention all data is referenced through cross-mountpoint symlinks= , would that make a difference? Should I use canonical paths in the code in= stead? > > All file systems are mounted "noatime, soft-updates". You may want to play around with gshed, the GEOM Scheduler. Matt Dillon did a bunch of tests comparing FreeBSD+UFS to DragonflyBSD+HAMMER and found that FreeBSD starves read threads in order to satisfy write threads (or the other way around?). But, adding gsched into the mix helped things immensely, allowing mixed reads/writes to better shares disk I/O resources. I'll see if I can dig up a link to his testing e-mail messages. --=20 Freddie Cash fjwcash@gmail.com