From owner-freebsd-geom@freebsd.org Wed Apr 20 18:38:28 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4DE2AB16185; Wed, 20 Apr 2016 18:38:28 +0000 (UTC) (envelope-from rpokala@mac.com) Received: from mr11p00im-asmtp002.me.com (mr11p00im-asmtp002.me.com [17.110.69.253]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3D2531EA3; Wed, 20 Apr 2016 18:38:28 +0000 (UTC) (envelope-from rpokala@mac.com) Received: from [172.17.133.77] (dip-cali.panasas.com [64.80.217.3]) by mr11p00im-asmtp002.me.com (Oracle Communications Messaging Server 7.0.5.36.0 64bit (built Sep 8 2015)) with ESMTPSA id <0O5Y003Z63RXHW10@mr11p00im-asmtp002.me.com>; Wed, 20 Apr 2016 18:38:22 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-04-20_12:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 clxscore=1011 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1510270003 definitions=main-1604200295 User-Agent: Microsoft-MacOutlook/f.15.1.160411 Date: Wed, 20 Apr 2016 11:38:20 -0700 Subject: g_event and sysctllock From: Ravi Pokala To: "freebsd-hackers@freebsd.org" , "freebsd-geom@freebsd.org" , "Rai, Sushanth" Message-id: <1E22BEFB-5C0E-4DE2-91E1-FD5F8AEFD04A@mac.com> Thread-topic: g_event and sysctllock MIME-version: 1.0 Content-type: text/plain; charset=UTF-8 Content-transfer-encoding: 7bit X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Apr 2016 18:38:28 -0000 Hi folks, My colleague Sushanth (CCed) tried to send this to hackers@ yesterday, but it didn't go through for some reason. Resending on his behalf, adding geom@, and noting that while we saw this on 7-STABLE, it looks like it could still be an issue in -HEAD. -------------------------------- Hello, We have a home-grown geom driver that creates sysctl in the device creation path. Device creation is handled by the geom event thread. The call to SYSCTL_ADD_NODE() takes the sysctllock in exclusive mode. If the event thread is racing with another thread that calls sysctl_disks(), then you end up with a deadlock since sysctl_disks() tickles the event thread and goes to sleep while holding the sysctllock. It is expected to woken up by the event thread when the event of listing all the disks is processed. But the geom event is blocked waiting for the sysctllock. I did see g_disk_create() adds a sysctl variable in a similar fashion, hence the email. I'm thinking of fixing the sysctl_disks() to drop the sysctllock before going to sleep and reacquiring it after being woken up. Let me know your thoughts. Thanks, Sushanth -------------------------------- -Ravi (rpokala@) From owner-freebsd-geom@freebsd.org Wed Apr 20 19:24:16 2016 Return-Path: Delivered-To: freebsd-geom@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 23542B1520E; Wed, 20 Apr 2016 19:24:16 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pa0-x22b.google.com (mail-pa0-x22b.google.com [IPv6:2607:f8b0:400e:c03::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E8CE31DD3; Wed, 20 Apr 2016 19:24:15 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pa0-x22b.google.com with SMTP id r5so18696252pag.1; Wed, 20 Apr 2016 12:24:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=T7kZ1cJeJLAMT/osQKq2EoCoPL61egMEQfP1cqC4URc=; b=FN6bDvcEVE0AylzTo1GPUtHWyZ52rueLVT9SXs433KMPiTRBcDdVxqTw3n4/DHz0HZ 4J/M3pDrZpz/QN8RU4AoR2Wx9XhF2YaoGraa1tll2G7hQi2lu8K8U4MJJBGhSwNYPied KaoS79jNgk1DBHUzcG/omjDh4EoXekgNR1FOtFAbcczc4IdbUrHwPlHLdHXh3oLYFttv Yflr8PjQSQtlAXbFc3oo4x2osVPzIMcENjzgPvD8MZUBgo0TSZ6cdXhzDZ6VY7Xp2GR9 F1oDxDo84YvaA5K6BhKhZqISoa3qaUdy0txLECuEb5v6Jgh3B+85EyAkVJlI8m43AhVe xtRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=T7kZ1cJeJLAMT/osQKq2EoCoPL61egMEQfP1cqC4URc=; b=CtM0tNTCRKkb2EkMXbCMHSzm60w/3i3iLVLxB505ug8IZGbCBoJrPhhoxdgMuafYHC XBlGKDQVb0Fc1MGfAN1x2h9DYKVsoj2SmIussCjhdg1OpqxvsPOzZ5jqKPu8ppcC8RJG 2n2S2QCOcN8u0G4nL+p1/3rh5SElZSDVV5CdDExIy5+h54J3lkayICDSG5goGjQi/mE8 hBhlzudvyJny0KmanLT0OiNY41620eU89TLsDbSBPtjLljwLiP1wlUqKFKvmCtbzU8LV 038KF+gLfxGrZcoKjC6KRQgwe1F25TNSnnFQBtsb1xQQ5ERy5oWp7XTU0EwwOPb6yFf7 y84A== X-Gm-Message-State: AOPr4FV8HDPCm9C0PM8LKeqGR0ymGBxNgtveXYjNnod/DQ6fJaTDkJbiAPTXEJIr/JI0UQ== X-Received: by 10.66.251.10 with SMTP id zg10mr14628476pac.1.1461180255525; Wed, 20 Apr 2016 12:24:15 -0700 (PDT) Received: from wkstn-mjohnston.west.isilon.com (c-67-182-131-225.hsd1.wa.comcast.net. [67.182.131.225]) by smtp.gmail.com with ESMTPSA id h85sm100083737pfj.52.2016.04.20.12.24.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 20 Apr 2016 12:24:14 -0700 (PDT) Sender: Mark Johnston Date: Wed, 20 Apr 2016 12:27:16 -0700 From: Mark Johnston To: Ravi Pokala Cc: "freebsd-hackers@freebsd.org" , "freebsd-geom@freebsd.org" , "Rai, Sushanth" Subject: Re: g_event and sysctllock Message-ID: <20160420192716.GA16105@wkstn-mjohnston.west.isilon.com> References: <1E22BEFB-5C0E-4DE2-91E1-FD5F8AEFD04A@mac.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1E22BEFB-5C0E-4DE2-91E1-FD5F8AEFD04A@mac.com> User-Agent: Mutt/1.6.0 (2016-04-01) X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Apr 2016 19:24:16 -0000 On Wed, Apr 20, 2016 at 11:38:20AM -0700, Ravi Pokala wrote: > Hi folks, > > My colleague Sushanth (CCed) tried to send this to hackers@ yesterday, but it didn't go through for some reason. Resending on his behalf, adding geom@, and noting that while we saw this on 7-STABLE, it looks like it could still be an issue in -HEAD. > > -------------------------------- > > Hello, > > We have a home-grown geom driver that creates sysctl in the device creation path. Device creation is handled by the geom event thread. The call to SYSCTL_ADD_NODE() takes the sysctllock in exclusive mode. If the event thread is racing with another thread that calls sysctl_disks(), then you end up with a deadlock since sysctl_disks() tickles the event thread and goes to sleep while holding the sysctllock. It is expected to woken up by the event thread when the event of listing all the disks is processed. But the geom event is blocked waiting for the sysctllock. > > I did see g_disk_create() adds a sysctl variable in a similar fashion, hence the email. I'm thinking of fixing the sysctl_disks() to drop the sysctllock before going to sleep and reacquiring it after being woken up. Let me know your thoughts. > This is not an issue in head. r216060 modified the sysctl code so that the sysctl lock is dropped before handlers are called. So sysctl_disks() is no longer executed with the sysctl lock held, and this addresses the problem you described. This change was MFCed to stable/7 in r220012 though. Do you have that revision?