From owner-freebsd-current@freebsd.org Sun Dec 17 21:54:16 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B2938E973F2 for ; Sun, 17 Dec 2017 21:54:16 +0000 (UTC) (envelope-from dan@langille.org) Received: from clavin2.langille.org (clavin2.langille.org [199.233.228.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "clavin.langille.org", Issuer "BSD Cabal Headquarters" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7AE7880189 for ; Sun, 17 Dec 2017 21:54:16 +0000 (UTC) (envelope-from dan@langille.org) Received: from (clavin2.int.langille.org (clavin2.int.unixathome.org [10.4.7.7]) (Authenticated sender: hidden) with ESMTPSA id BB8B11FAB9 ; Sun, 17 Dec 2017 21:54:13 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: cannot access pass device from within jail From: Dan Langille In-Reply-To: Date: Sun, 17 Dec 2017 16:53:46 -0500 Cc: FreeBSD Current Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Warner Losh X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Dec 2017 21:54:16 -0000 > On Dec 17, 2017, at 4:48 PM, Warner Losh wrote: >=20 > Sorry to top post. The problem did turn out to be securelevel. We = found > this by doing >=20 > dtrace -n 'fbt::securelevel_gt:return {print(args[1];)}' >=20 > Though you could also replace securelevel_gt with passopen to see it = was in > passopen too... On the host we ran: [root@r710-01:~] # sudo dtrace -n 'fbt::securelevel_gt:return = {print(args[1]);}' dtrace: description 'fbt::securelevel_gt:return ' matched 1 probe CPU ID FUNCTION:NAME 21 50269 securelevel_gt:return int 0x1 In the jail: mtx -f /dev/pass7 status Based on the dtrace output, I again checked securelevel in the jail: [dan@bacula-sd-02] $ sysctl kern.securelevel kern.securelevel: 2 WTF? I'd already checked that as seen at = https://lists.freebsd.org/pipermail/freebsd-scsi/2017-December/007614.html= Oh wait. The above URL refers to bacula-sd-01, not bacula-sd-02. Ouch. = User error... me. After going back into the host, I set: $ sudo iocage set securelevel=3D1 bacula-sd-02 Property: securelevel has been updated to 1 After restarting the jail and getting back into it: [root@bacula-sd-02 ~]# sysctl kern.securelevel kern.securelevel: 1 [root@bacula-sd-02 ~]# mtx -f /dev/pass7 status=20 Storage Changer /dev/pass7:2 Drives, 47 Slots ( 0 Import/Export ) Data Transfer Element 0:Full (Storage Element 1 Loaded):VolumeTag =3D = 000001L4 =20 .... Success. Sorry for the noise which would have been reduced if I'd gotten my = sysctl on the right host. Thank you all=10. >=20 > Warner >=20 > On Sun, Dec 17, 2017 at 2:08 PM, Dan Langille = wrote: >=20 >>> On Dec 17, 2017, at 4:04 PM, Warner Losh wrote: >>>=20 >>> What's the permissions of /dev/xpt0 in the jail? If it's not there I = know >>> at least camcontrol won't work. I've not used mtx, so I can't say if = it's >>> affected too or not. >>=20 >> I have tried both with and without xpt0. When I tried, it was: >>=20 >> # ls -l /dev/xpt0 >> crw------- 1 root operator 0x4c Dec 16 21:52 /dev/xpt0 >>=20 >>>=20 >>> However, looking at the truss output: >>>=20 >>> openat(AT_FDCWD,"/dev/pass7",O_RDWR|O_EXCL,00) ERR#1 'Operation not >>> permitted' >>> suggests something other than the canonical xpt0 issue else is going = on. >> If >>> we look at passopen in cam, I can see two exit paths: >>>=20 >>> error =3D securelevel_gt(td->td_ucred, 1); if (error !=3D 0) = {... >>> return error; } >>> securelevel_gt is just "return (cr->cr_prison->pr_securelevel > = level ? >>> EPERM : 0);" which might be possible. What's the securelevel of the = jail? >>> Maybe this is going on somehow? >>=20 >>=20 >> On the host: >>=20 >> $ sysctl kern.securelevel >> kern.securelevel: -1 >>=20 >>=20 >> On the jail: >>=20 >> $ sysctl kern.securelevel >> kern.securelevel: -1 >>=20 >>>=20 >>> The second is basically >>> if (((flags & FWRITE) =3D=3D 0) || ((flags & FREAD) =3D=3D 0)) = {... return >>> EPERM; } >>> which isn't happening because of the O_RDWR in the truss output. >>>=20 >>> The other possibility is that something above the pass driver is = doing >> the >>> check. I've not looked at that code path yet, buy you can see if = it's >>> making it to passopen() with dtrace and checking its return value. I >> don't >>> see anything in how we register the device, though, that would = suggest >>> filtering it in jails. >>>=20 >>> Warner >>>=20 >>> On Sun, Dec 17, 2017 at 12:52 PM, Dan Langille = wrote: >>>=20 >>>> Hello, >>>>=20 >>>> What suggestions do you have for where I should look next? I'm = happy to >>>> start installing various builds of FreeBSD in order to track down = which >>>> commit caused this. >>>>=20 >>>> I'm trying to access a tape library from within a jail running on a >>>> FreeBSD 11.1 host. sa(4) devices are working (e.g. I can rewind = nsa0). >>>>=20 >>>> pass(4) devices (i.e. the tape changer ch0) are not working. This >> morning >>>> I posted to -scsi@: https://lists.freebsd.org/ >> pipermail/freebsd-scsi/2017- >>>> December/007608.html >>>>=20 >>>> The device appears in the jail and has appropriate permissions. = This >>>> access was granted >>>> via /etc/devfs.rules using the same approach I used for FreeBSD = 10.3 >>>>=20 >>>> The permissions in the jail: >>>>=20 >>>> [root@bacula-sd-02 ~]# ls -l /dev/pass7 >>>> crw------- 1 root operator 0x74 Dec 16 21:52 /dev/pass7 >>>>=20 >>>> The command in the jail: >>>>=20 >>>> [root@bacula-sd-02 ~]# mtx -f /dev/pass7 status >>>> cannot open SCSI device '/dev/pass7' - Operation not permitted >>>>=20 >>>> Here is the truss output of the command in question: >>>> https://gist.github.com/dlangille/b80ee804b8080e1cbf5b5ab67f0bdabe >>>>=20 >>>> Thank you. >>>>=20 >>>> -- >>>> Dan Langille - BSDCan / PGCon >>>> dan@langille.org >>>>=20 >>>>=20 >>>> _______________________________________________ >>>> freebsd-current@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-current >>>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@ >> freebsd.org" >>>>=20 >>> _______________________________________________ >>> freebsd-current@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-current >>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@ >> freebsd.org" >>=20 >>=20 > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to = "freebsd-current-unsubscribe@freebsd.org"