From owner-freebsd-hackers@FreeBSD.ORG Sun Oct 6 07:30:51 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2FEAE256; Sun, 6 Oct 2013 07:30:51 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-x236.google.com (mail-ee0-x236.google.com [IPv6:2a00:1450:4013:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 46B442C09; Sun, 6 Oct 2013 07:30:50 +0000 (UTC) Received: by mail-ee0-f54.google.com with SMTP id e53so2611084eek.13 for ; Sun, 06 Oct 2013 00:30:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=aEAivdcImJx+yXZI5NuiWTYkGe49ecX2SIFKvkhmty4=; b=WTTPkWUGjjMcV/bRX+SexY5dQnvUyNu9KLmpcu9rMwlLyv1kv97pWT1OLOOKpOdTn4 QPTY14AsZt8qAXxVvWshXTTQtJ2Gyn5rd/HXhwUKjup6CKx9GqtA+/18QCAI5iNs5Ky+ q/kiOiqIwauSh6osx3JUc4J9LJKaVsa/LoTZhGhJNCnRKLDn/TYmnDYL+yz8/XRlNSXR FktfHaKSiaRKpV/UT7teb3HA7lYhgQ6dMXjHwLFPvIVUIDp2BX344a1irStjc/udZomn OaiagxOFiZyoO8vJOnHU9KOguY26lf2soDNBuj0SYZ+Mylm6O70Z2TN2SBe81iQ3jDMG X9VA== X-Received: by 10.15.43.13 with SMTP id w13mr37255953eev.37.1381044647533; Sun, 06 Oct 2013 00:30:47 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([46.211.77.102]) by mx.google.com with ESMTPSA id v8sm48252297eeo.12.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 06 Oct 2013 00:30:45 -0700 (PDT) Sender: Alexander Motin Message-ID: <525111A2.1020106@FreeBSD.org> Date: Sun, 06 Oct 2013 10:30:42 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130616 Thunderbird/17.0.6 MIME-Version: 1.0 To: John Baldwin Subject: Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking References: <5224511D.4090503@FreeBSD.org> <20130906230236.GI43281@caravan.chchile.org> <522AC88D.4070005@FreeBSD.org> <201310021330.23251.jhb@freebsd.org> In-Reply-To: <201310021330.23251.jhb@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-hackers@freebsd.org, freebsd-current@freebsd.org, freebsd-geom@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Oct 2013 07:30:51 -0000 On 02.10.2013 20:30, John Baldwin wrote: > On Saturday, September 07, 2013 2:32:45 am Alexander Motin wrote: >> On 07.09.2013 02:02, Jeremie Le Hen wrote: >>> On Fri, Sep 06, 2013 at 11:29:11AM +0300, Alexander Motin wrote: >>>> On 06.09.2013 11:06, Jeremie Le Hen wrote: >>>>> On Fri, Sep 06, 2013 at 12:46:27AM +0200, Olivier Cochard-Labbé wrote: >>>>>> On Thu, Sep 5, 2013 at 11:38 PM, Alexander Motin > wrote: >>>>>>> I've found and fixed possible double request completion, that could > cause >>>>>>> such symptoms if happened. Updated patch located as usual: >>>>>>> http://people.freebsd.org/~mav/camlock_patches/camlock_20130905.patch >>>>>>> >>>>> With this new one I cannot boot any more (I also updated the source >>>>> tree). This is a hand transcripted version: >>>>> >>>>> Trying to mount root from zfs:zroot/root []... >>>>> panic: Batch flag already set >>>>> cpuid = 1 >>>>> KDB: stack backtrace: >>>>> db_trace_self_wrapper() >>>>> kdb_backtrace() >>>>> vpanic() >>>>> kassert_panic() >>>>> xpt_batch_start() >>>>> ata_interrupt() >>>>> softclock_call_cc() >>>>> softclock() >>>>> ithread_loop() >>>>> fork_exit() >>>>> fork_trampoline() >>>> >>>> Thank you for the report. I see my fault. It is probably specific to >>>> ata(4) driver only. I've workarounded that in new patch version, but >>>> probably that area needs some rethinking. >>>> >>>> http://people.freebsd.org/~mav/camlock_patches/camlock_20130906.patch >>> >>> I'm not sure you needed a confirmation, but it boots. Thanks :). >>> >>> I didn't quite understand the thread; is direct dispatch enabled for >>> amd64? ISTR you said only i386 but someone else posted the macro for >>> amd64. >> >> Yes, it is enabled for amd64. I've said x86, meaning both i386 and amd64. > > FYI, I tested mfi with this patch set and mfid worked fine for handling g_up > directly: > > Index: dev/mfi/mfi_disk.c > =================================================================== > --- dev/mfi/mfi_disk.c (revision 257407) > +++ dev/mfi/mfi_disk.c (working copy) > @@ -162,6 +162,7 @@ > sc->ld_disk->d_unit = sc->ld_unit; > sc->ld_disk->d_sectorsize = secsize; > sc->ld_disk->d_mediasize = sectors * secsize; > + sc->ld_disk->d_flags = DISKFLAG_DIRECT_COMPLETION; > if (sc->ld_disk->d_mediasize >= (1 * 1024 * 1024)) { > sc->ld_disk->d_fwheads = 255; > sc->ld_disk->d_fwsectors = 63; > Thank you for the feedback. But looking on mfi driver sources I would say that it calls biodone() from mfi_disk_complete() from cm_complete() method, which is called while holding mfi_io_lock mutex. I guess that if on top of mfi device would be some GEOM class, supporting direct dispatch and sending new requests down on previous request completion (or retrying requests), that could cause recursive mfi_io_lock acquisition. That is exactly the cause why I've added this flag. May be it is a bit paranoid, but it is better to be safe then sorry. Another good reason to drop the lock before calling biodone() would be reducing the lock hold time. Otherwise it may just increase lock congestion there and destroy all benefits of the direct dispatch. -- Alexander Motin