From owner-freebsd-current@freebsd.org Fri Jan 26 12:29:57 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A5E13ED4453 for ; Fri, 26 Jan 2018 12:29:57 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2F2927130C for ; Fri, 26 Jan 2018 12:29:57 +0000 (UTC) (envelope-from david@catwhisker.org) Received: by mailman.ysv.freebsd.org (Postfix) id DA63CED4452; Fri, 26 Jan 2018 12:29:56 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B6A15ED4451 for ; Fri, 26 Jan 2018 12:29:56 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from mx.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 08DB97130B for ; Fri, 26 Jan 2018 12:29:55 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.15.2/8.15.2) with ESMTP id w0QCTlkj072373 for ; Fri, 26 Jan 2018 12:29:47 GMT (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.15.2/8.15.2/Submit) id w0QCTlZU072372 for current@freebsd.org; Fri, 26 Jan 2018 04:29:47 -0800 (PST) (envelope-from david) Date: Fri, 26 Jan 2018 04:29:47 -0800 From: David Wolfskill To: current@freebsd.org Subject: Panic on shutdown @r328436: "Unholding 6 with cnt = -559038242" Message-ID: <20180126122947.GA1287@albert.catwhisker.org> Reply-To: current@freebsd.org Mail-Followup-To: current@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="7ULH7gtHJxM9ukDW" Content-Disposition: inline User-Agent: Mutt/1.9.2 (2017-12-15) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jan 2018 12:29:57 -0000 --7ULH7gtHJxM9ukDW Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable This is on my "build machine" (laptop is still building updated ports for today, so I don't know yet whether or not it encounters this.)=20 I had performed a source-based update from r328393 to r328436, rebooted, performed "make delete-old-libs", and all seemed well. I then issued "sudo shutdown -p now", and serial console shows: FreeBSD/amd64 (freebeast.catwhisker.org) (ttyu0) login: Jan 26 12:11:03 Stopping sshd. Waiting for PIDS: 681. Stopping rsyncd. Waiting for PIDS: 652. Stopping powerd. Waiting for PIDS: 636. Stopping ntpd. Waiting for PIDS: 633, 633. Stopping lpd. Waiting for PIDS: 610. Stopping lockd. WaitingWARNING: autofs_unmount: vflush failed with error 16 for PIDS: 592. Stopping statd. Waiting for PIDS: 589. Stopping nfsd. WaitinlJan 26 12:11:05 ock ordefreebeast syslogr reversal: 1st 0xfffff800692cd490 filed: exiting on sidesc structure (filedesc struct= ure) @ /usr/src/sys/kern/sys_generic.c:1567 2nd 0xfffff8006973f9a0 devfs (devfs) @ /usr/src/gnal 15 sys/kern/vfs_vnops.c:1526 stack backtrace: #0 0xffffffff80b2c2e3 at witness_debugger+0x73 #1 0xffffffff80b2c164 at witness_checkorder+0xe34 #2 0xffffffff80a9ca21 at lockmgr_lock_fast_path+0x1b1 #3 0xffffffff810f8ca9 at VOP_LOCK1_APV+0xd9 #4 0xffffffff80ba7746 at _vn_lock+0x66 #5 0xffffffff80ba654b at vn_poll+0x3b #6 0xffffffff80992f4d at devfs_poll_f+0xcd #7 0xffffffff80b33255 at kern_poll+0x385 #8 0xffffffff80b32ec0 at sys_poll+0x50 #9 0xffffffff80f7a37b at amd64_syscall+0x79b #10 0xffffffff80f569c8 at fast_syscall_common+0xfc Waiting (max 60 seconds) for system process `vnlru' to stop... done Waiting (max 60 seconds) for system process `bufdaemon' to stop... done Waiting (max 60 seconds) for system process `syncer' to stop...=20 Syncing disks, vnodes remaining... 5 5 lock order reversal: 1st 0xfffff80007927240 syncer (syncer) @ /usr/src/sys/kern/vfs_subr.c:2157 2nd 0xfffff80007570d50 devfs (devfs) @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1= 583 stack backtrace: #0 0xffffffff80b2c2e3 at witness_debugger+0x73 #1 0xffffffff80b2c164 at witness_checkorder+0xe34 #2 0xffffffff80a9ca21 at lockmgr_lock_fast_path+0x1b1 #3 0xffffffff810f8ca9 at VOP_LOCK1_APV+0xd9 #4 0xffffffff80ba7746 at _vn_lock+0x66 #5 0xffffffff80dc1dac at ffs_sync+0x2cc #6 0xffffffff80b9ce8f at sync_fsync+0xff #7 0xffffffff810f7c19 at VOP_FSYNC_APV+0xd9 #8 0xffffffff80b9acc4 at sched_sync+0x284 #9 0xffffffff80a88414 at fork_exit+0x84 #10 0xffffffff80f56e1e at fork_trampoline+0xe 5 3 2 2 1 1 1 1 1 0 0 0 0 0 done All buffers synced. lock order reversal: 1st 0xfffff800079c57c8 ufs (ufs) @ /usr/src/sys/kern/vfs_mount.c:1280 2nd 0xfffff8000795e418 devfs (devfs) @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1= 371 stack backtrace: #0 0xffffffff80b2c2e3 at witness_debugger+0x73 #1 0xffffffff80b2c164 at witness_checkorder+0xe34 #2 0xffffffff80a9ca21 at lockmgr_lock_fast_path+0x1b1 #3 0xffffffff810f8ca9 at VOP_LOCK1_APV+0xd9 #4 0xffffffff80ba7746 at _vn_lock+0x66 #5 0xffffffff80dbef63 at ffs_flushfiles+0x93 #6 0xffffffff80da24f2 at softdep_flushfiles+0x82 #7 0xffffffff80dc15f7 at ffs_unmount+0x77 #8 0xffffffff80b8e2c9 at dounmount+0x519 #9 0xffffffff80b9800b at vfs_unmountall+0x6b #10 0xffffffff80b73d25 at bufshutdown+0x3a5 #11 0xffffffff80ac754a at kern_reboot+0x1da #12 0xffffffff80ac7312 at sys_reboot+0x3c2 #13 0xffffffff80f7a37b at amd64_syscall+0x79b #14 0xffffffff80f569c8 at fast_syscall_common+0xfc Swap device [file] removed. Uptime: 1m44s (ada0:ahcich0:0:0:0): spin-down (ada1:ahcich2:0:0:0): spin-down (ada2:ahcich3:0:0:0): spin-down (ada3:ahcich4:0:0:0): spin-down panic: Unholding 6 with cnt =3D -559038242 cpuid =3D 3 time =3D 1516968697 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0000428= 8c0 vpanic() at vpanic+0x18d/frame 0xfffffe0000428920 panic() at panic+0x43/frame 0xfffffe0000428980 dadiskgonecb() at dadiskgonecb+0x42/frame 0xfffffe00004289a0 g_disk_providergone() at g_disk_providergone+0x25/frame 0xfffffe00004289d0 g_destroy_provider() at g_destroy_provider+0xae/frame 0xfffffe00004289f0 g_wither_washer() at g_wither_washer+0x87/frame 0xfffffe0000428a30 g_run_events() at g_run_events+0x3ca/frame 0xfffffe0000428a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000428ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000428ab0 --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- KDB: enter: panic [ thread pid 13 tid 100044 ] Stopped at kdb_enter+0x3b: movq $0,kdb_why db>=20 As noted, this is a build machine, and it was to be powered off for the rest of the day anyway, so I don't need to get it up & running immediately: I can poke at the ddb prompt, given some clues. When running head, the system does not use ZFS (only UFS2+SU -- not SUJ -- & tmpfs). Same system had completed a source-based update for stable/11 from r328392 to r328429 earlier today without incident (using a different slice of the boot drive). Peace, david --=20 David H. Wolfskill david@catwhisker.org "unfortunately, no trust!=E2=80=9D -- well, of course! You reap what you s= ow. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --7ULH7gtHJxM9ukDW Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGTBAEBCgB9FiEEzLfO+ReoAfQwZNd7FTnMQKBJ7hcFAlprHztfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEND QjdDRUY5MTdBODAxRjQzMDY0RDc3QjE1MzlDQzQwQTA0OUVFMTcACgkQFTnMQKBJ 7hfl0QgAkIRYZ8zKVQUwM80iXVpTV8FeAj6Y24EHtEEolm8IaAfrYEAqklibp84X izxlp9raYgLNZuPKBqFBgQFVCkiZMypGfOOzVfDuetkYHXhxzJg6SVmJW8hc3gvg Cj1IqcsbPEucmfBZJmmF6juDVsPw0zEQkLeVCp58PKmZ8W/ziy3GLOwfYZakqkCn 0zIxJZL/FwNESDVl532qVDlc9k8yQ0RbajnIlMAHsvHY0b5ROI5TJgBSeokj0QXi swr3wc05Gb35XTEG7UxzSGD/FvUainNL0opTFN649o7xickDFeO9XijzAYDzyJjr IzBAmfbeHPw/MhiQr7Z852e1qyDhLA== =4NS6 -----END PGP SIGNATURE----- --7ULH7gtHJxM9ukDW--