From owner-freebsd-scsi@FreeBSD.ORG Fri May 20 20:18:03 2011 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34CBC1065672 for ; Fri, 20 May 2011 20:18:03 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 48D328FC1A for ; Fri, 20 May 2011 20:18:01 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id p4KKHvei075952 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 20 May 2011 23:17:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id p4KKHvWt016801; Fri, 20 May 2011 23:17:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id p4KKHvK8016800; Fri, 20 May 2011 23:17:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 20 May 2011 23:17:57 +0300 From: Kostik Belousov To: Joerg Wunsch Message-ID: <20110520201757.GE48734@deviant.kiev.zoral.com.ua> References: <20110508085314.GA5364@uriah.heep.sax.de> <20110508094509.GT48734@deviant.kiev.zoral.com.ua> <20110508104543.GB5364@uriah.heep.sax.de> <20110518060429.GA4146@uriah.heep.sax.de> <20110518165120.GT48734@deviant.kiev.zoral.com.ua> <20110520083948.GA2277@uriah.heep.sax.de> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="K5FRBbPSksISZ1JK" Content-Disposition: inline In-Reply-To: <20110520083948.GA2277@uriah.heep.sax.de> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-scsi@freebsd.org Subject: Re: Panic when removing a SCSI device entry X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 May 2011 20:18:03 -0000 --K5FRBbPSksISZ1JK Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, May 20, 2011 at 10:39:48AM +0200, Joerg Wunsch wrote: > As Kostik Belousov wrote: >=20 > > Please do "p *(struct cdev_priv *)0xe98804f4" and > > "p *(struct cdev_priv *)0xce0dc900" from kgdb. >=20 > Well, that kernel unfortunately lacked debugging symbols, and while > I've still been thinking about the best way to recompile an exact > same kernel with them ... Yes, it would be quite interesting to see the data I asked for. I spent significant time trying to imagine a scenario where the reported panic could be possible, and did not end with anything. >=20 > > I am pretty much sure that INVARIANTS kernel would hit the assert > > about SI_NAMED flag being clear on destroy_devl() invocation. > > We would have catched the issue earlier, with less interesting data > > destroyed. >=20 > ... the now INVARIANTS kernel panicked again last night. This time, > I've got debugging symbols. So, as you expected, the corruption had > been caught earliere now. The panic message is: >=20 > (kgdb) p panicstr > $1 =3D 0xc088dca0 "Bad link elm 0xc81cc200 prev->next !=3D elm" >=20 > Here is the stack trace: >=20 > (kgdb) bt > #0 doadump () at pcpu.h:231 > #1 0xc057943e in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c= :419 > #2 0xc0579710 in panic (fmt=3DVariable "fmt" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:592 > #3 0xc044b707 in camperiphfree (periph=3D0xc81cc200) at /usr/src/sys/cam= /cam_periph.c:550 What is the exact revision of your sources ? I do not see ay list manipulation macros at the line 550 in cam_periph.c, both in HEAD and stable/8. There is one three lines earlier, and it could cause the panic shown. This looks like a CAM issue, which is out of my scope. Hope other subscribers will offer the help. I committed the devfs fix you tested, it should land into stable/8 in a wee= k. > #4 0xc044b8e5 in cam_periph_release_locked (periph=3D0xc81cc200) at /usr= /src/sys/cam/cam_periph.c:336 > #5 0xc044ba74 in cam_periph_release (periph=3D0xc81cc200) at /usr/src/sy= s/cam/cam_periph.c:352 > #6 0xc046eea2 in saopen (dev=3D0xc8d9ba00, flags=3D1, fmt=3D8192, td=3D0= xc93d15c0) > at /usr/src/sys/cam/scsi/scsi_sa.c:499 > #7 0xc053833e in giant_open (dev=3D0xc8d9ba00, oflags=3D1, devtype=3D819= 2, td=3D0xc93d15c0) > at /usr/src/sys/kern/kern_conf.c:361 > #8 0xc05177b2 in devfs_open (ap=3D0xe98b3b00) at /usr/src/sys/fs/devfs/d= evfs_vnops.c:992 > #9 0xc07b9a95 in VOP_OPEN_APV (vop=3D0xc08403c0, a=3D0xe98b3b00) at vnod= e_if.c:445 > #10 0xc06143d6 in vn_open_cred (ndp=3D0xe98b3b78, flagp=3D0xe98b3c2c, cmo= de=3D0, vn_open_flags=3D0,=20 > cred=3D0xc807a780, fp=3D0xc90f6d58) at vnode_if.h:196 > #11 0xc06144db in vn_open (ndp=3D0xe98b3b78, flagp=3D0xe98b3c2c, cmode=3D= 0, fp=3D0xc90f6d58) > at /usr/src/sys/kern/vfs_vnops.c:94 > #12 0xc06133fc in kern_openat (td=3D0xc93d15c0, fd=3D-100, path=3D0x804a0= bb
,=20 > pathseg=3DUIO_USERSPACE, flags=3D1, mode=3D6) at /usr/src/sys/kern/vf= s_syscalls.c:1083 > #13 0xc0613845 in kern_open (td=3D0xc93d15c0, path=3D0x804a0bb
,=20 > pathseg=3DUIO_USERSPACE, flags=3D0, mode=3D6) at /usr/src/sys/kern/vf= s_syscalls.c:1039 > #14 0xc06138c0 in open (td=3D0xc93d15c0, uap=3D0xe98b3cec) at /usr/src/sy= s/kern/vfs_syscalls.c:1015 > #15 0xc05b6276 in syscallenter (td=3D0xc93d15c0, sa=3D0xe98b3ce4) at /usr= /src/sys/kern/subr_trap.c:326 > #16 0xc0799b54 in syscall (frame=3D0xe98b3d28) at /usr/src/sys/i386/i386/= trap.c:1086 > #17 0xc077fd21 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception= .s:266 > #18 0x00000033 in ?? () > Previous frame inner to this frame (corrupt stack?) >=20 > The dev node in question (I think, it's the "dev" argument in stack As I said, devfs looks innocent in this backtrace. > frame #6) is: > (kgdb) p dev > $5 =3D (struct cdev *) 0xc8d9ba00 > (kgdb) p *dev > $6 =3D {__si_reserved =3D 0x0, si_flags =3D 4, si_atime =3D {tv_sec =3D 1= 305772805, tv_nsec =3D 0}, si_ctime =3D { > tv_sec =3D 1305773153, tv_nsec =3D 0}, si_mtime =3D {tv_sec =3D 13057= 73153, tv_nsec =3D 0}, si_uid =3D 0,=20 > si_gid =3D 5, si_mode =3D 432, si_cred =3D 0x0, si_drv0 =3D 1, si_refco= unt =3D 2, si_list =3D { > le_next =3D 0xc7f9ed00, le_prev =3D 0xc8d43538}, si_clone =3D {le_nex= t =3D 0x0, le_prev =3D 0x0},=20 > si_children =3D {lh_first =3D 0xc8d9f100}, si_siblings =3D {le_next =3D= 0x0, le_prev =3D 0x0}, si_parent =3D 0x0,=20 > si_name =3D 0xc8d9ba78 "nsa0.0", si_drv1 =3D 0xc81cc200, si_drv2 =3D 0x= 0, si_devsw =3D 0xc0833a40,=20 > si_iosize_max =3D 65536, si_usecount =3D 1, si_threadcount =3D 2, __si_= u =3D {__sid_snapdata =3D 0x0},=20 > __si_namebuf =3D "nsa0.0", '\0' } >=20 > The contents of the "periph" object as seen in the various CAM layer > functions is: >=20 > (kgdb) p *periph > $1 =3D {pinfo =3D {priority =3D 1, generation =3D 87063, index =3D -1}, p= eriph_start =3D 0xc046ba80 ,=20 > periph_oninval =3D 0xc046bcf0 , periph_dtor =3D 0xc046f= 2d0 ,=20 > periph_name =3D 0xc07e07bc "sa", path =3D 0xc8dd31b0, softc =3D 0xc8387= 800, sim =3D 0xc6e83c80,=20 > unit_number =3D 0, type =3D CAM_PERIPH_BIO, flags =3D 8, immediate_prio= rity =3D 4294967295, refcount =3D 0,=20 > ccb_list =3D {slh_first =3D 0x0}, periph_links =3D {sle_next =3D 0x0}, = unit_links =3D {tqe_next =3D 0x0,=20 > tqe_prev =3D 0xc0833b30}, deferred_callback =3D 0, deferred_ac =3D 0} >=20 > Finally, here's the "show cdev" command from DDB: >=20 > db> show cdev > geom.ctl 0xc6d1a100 > devctl 0xc6ccc700 > console 0xc6ccc600 > sndstat 0xc6ccc500 > ptmx 0xc6ccc400 > ctty 0xc6ccc300 > mem 0xc6ccc200 > kmem 0xc6db3800 > audit 0xc6db3700 > bpf 0xc6db3600 > bpf0 0xc6db3500 > null 0xc6db3400 > zero 0xc6db3300 > fd/0 0xc6db3200 > stdin 0xc6db3100 > fd/1 0xc6db3000 > stdout 0xc6db2e00 > fd/2 0xc6db2d00 > stderr 0xc6db2c00 > klog 0xc6db2b00 > pci 0xc6db2a00 > midistat 0xc6db2900 > kbdmux0 0xc6db2700 > kbd0 0xc6db2600 > random 0xc6db2400 > urandom 0xc6db2300 > sysmouse 0xc6db2200 > io 0xc6db2100 > speaker 0xc6db2000 > fido 0xc6d1be00 > ata 0xc6d1bd00 > acpi 0xc6d1b800 > ttyu2 0xc6e7dd00 > ttyu2.init 0xc6e7d800 > ttyu2.lock 0xc6e7d700 > cuau2 0xc6e7d600 > cuau2.init 0xc6e7d500 > cuau2.lock 0xc6e7d400 > ttyu3 0xc6e7d000 > ttyu3.init 0xc6e7ce00 > ttyu3.lock 0xc6e7cd00 > cuau3 0xc6e7cc00 > cuau3.init 0xc6e7cb00 > cuau3.lock 0xc6e7ca00 > ttyu4 0xc6e7c600 > ttyu4.init 0xc6e7c500 > ttyu4.lock 0xc6e7c800 > cuau4 0xc6e7c900 > cuau4.init 0xc6e7d200 > cuau4.lock 0xc6e7d300 > ttyu5 0xc6e7e400 > ttyu5.init 0xc6e7e500 > ttyu5.lock 0xc6e7e600 > cuau5 0xc6e7e700 > cuau5.init 0xc6e7e800 > cuau5.lock 0xc6e7e900 > ttyu6 0xc6e7e000 > ttyu6.init 0xc6f01e00 > ttyu6.lock 0xc6f01d00 > cuau6 0xc6f01c00 > cuau6.init 0xc6f01b00 > cuau6.lock 0xc6f01a00 > ttyu7 0xc6f01600 > ttyu7.init 0xc6f01500 > ttyu7.lock 0xc6f01400 > cuau7 0xc6f01300 > cuau7.init 0xc6f01200 > cuau7.lock 0xc6f01100 > ttyu8 0xc6f00c00 > ttyu8.init 0xc6f00b00 > ttyu8.lock 0xc6f00a00 > cuau8 0xc6f00900 > cuau8.init 0xc6f00800 > cuau8.lock 0xc6f00700 > ttyu9 0xc6f00300 > ttyu9.init 0xc6f00200 > ttyu9.lock 0xc6f00100 > cuau9 0xc6f00000 > cuau9.init 0xc6e7fe00 > cuau9.lock 0xc6e7fd00 > ttyv0 0xc6f01000 > ttyv1 0xc6f01800 > ttyv2 0xc6fa5d00 > ttyv3 0xc6fa5c00 > ttyv4 0xc6fa5b00 > ttyv5 0xc6fa5a00 > ttyv6 0xc6fa5900 > ttyv7 0xc6fa5800 > ttyv8 0xc6fa5700 > ttyv9 0xc6fa5600 > ttyva 0xc6fa5500 > ttyvb 0xc6fa5400 > ttyvc 0xc6fa5300 > ttyvd 0xc6fa5200 > ttyve 0xc6fa5100 > ttyvf 0xc6fa5000 > consolectl 0xc6fa4e00 > lpt0 0xc6fa4b00 > lpt0.ctl 0xc6fa4a00 > ppi0 0xc6fa4900 > ttyu0 0xc6fa4600 > ttyu0.init 0xc6fa4500 > ttyu0.lock 0xc6fa4400 > cuau0 0xc6fa4300 > cuau0.init 0xc6fa4200 > cuau0.lock 0xc6fa4100 > usbctl 0xc71d6d00 > mdctl 0xc71d6b00 > devstat 0xc71d6a00 > fd0 0xc71d6900 > usb/0.1.0 0xc71d6700 > ugen0.1 0xc71d6600 > usb/1.1.0 0xc71d6500 > ugen1.1 0xc71d6400 > usb/0.1.1 0xc71d6300 > usb/1.1.1 0xc71d5d00 > xpt0 0xc71d5800 > mixer0 0xc71d4a00 > mixer1 0xc71d4000 > mixer2 0xc7216a00 > acd0 0xc7216100 > ad4 0xc7216000 > ad4s1 0xc7215e00 > ad4s1b 0xc7215d00 > ad4s1h 0xc7215c00 > gvinum/ports 0xc728e100 > gvinum/src 0xc728e000 > gvinum/news 0xc728de00 > gvinum/distfiles 0xc728dd00 > gvinum/pdf 0xc728dc00 > gvinum/mysql 0xc728db00 > gvinum/upload 0xc728da00 > gvinum/obj 0xc728d900 > gvinum/root 0xc728d800 > gvinum/local 0xc728d700 > gvinum/usr 0xc728d600 > gvinum/var 0xc728d500 > gvinum/home_cvs 0xc728d400 > gvinum/home 0xc728d300 > gvinum/junk 0xc728d200 > gvinum/bacula_db 0xc728d100 > gvinum/dump 0xc728d000 > gvinum/tmp 0xc72a4400 > gvinum/camel 0xc72a4300 > gvinum/squid 0xc72a4200 > gvinum/sound 0xc72a4100 > usb/1.2.0 0xc72a2b00 > ugen1.2 0xc72a2a00 > cd0 0xc7290c00 > usb/1.2.1 0xc7290b00 > pass0 0xc7290400 > pass1 0xc7290300 > pass2 0xc7290200 > da0 0xc72e8800 > da0a 0xc72a2700 > da0h 0xc72a2800 > da1 0xc72a2e00 > ufsid/4856d98a00081994 0xc72a3000 > da1a 0xc72a3100 > da1h 0xc72a3200 > usb/0.2.0 0xc72e6900 > ugen0.2 0xc72e6a00 > usb/0.2.1 0xc72e6b00 > usb/0.3.0 0xc72e7400 > ugen0.3 0xc72e7500 > usb/0.3.1 0xc72e7600 > ukbd0 0xc72e7e00 > kbd1 0xc72e8000 > usb/0.4.0 0xc72e8100 > ugen0.4 0xc72e8200 > usb/0.4.1 0xc72e8300 > ums0 0xc72e7300 > usb/0.5.0 0xc72e6e00 > ugen0.5 0xc72e6c00 > usb/0.5.1 0xc72e6100 > usb/0.6.0 0xc72a4900 > ugen0.6 0xc72a4800 > usb/0.6.2 0xc72a4700 > pf 0xc7449700 > nfslock 0xc7e39100 > tap0 0xc7e95b00 > apm0 0xc7fa0100 > dsp2.0 0xc7446100 > dsp1.0 0xc7e37300 > dsp0.0 0xc7e37100 > pts/1 0xc8230600 > pts/2 0xc8230900 > ptyp0 0xc7e36d00 > ttyp0 0xc8230700 > ptyp1 0xc8230b00 > ttyp1 0xc82eb800 > pts/3 0xc8d9ae00 > pts/0 0xc8047800 > pts/4 0xc82c9400 > pts/5 0xc8da1400 > pts/6 0xc8e02700 > usb/0.6.0 0xc8497a00 > ugen0.6 0xc7e97500 > usb/0.6.2 0xc82ed400 > pass3 0xc8d9d700 > ch0 0xc8046200 > nsa0.0 0xc8d9ba00 > esa0.0 0xc8d43500 > nsa0 0xc8d9f100 > esa0 0xc8d43c00 > sa0.1 0xc8497e00 > nsa0.1 0xc8048800 > esa0.1 0xc8e00100 > sa0.2 0xc8e01b00 > nsa0.2 0xc8da0c00 > esa0.2 0xc8d9d900 > sa0.3 0xc8d9de00 > nsa0.3 0xc8d9f900 > esa0.3 0xc8d42600 > usb/0.7.0 0xcecdad00 > ugen0.7 0xcecd4a00 > usb/0.7.2 0xcecd7800 > pts/7 0xc8046100 > ptyp2 0xc822da00 > ttyp2 0xc8230300 > pass4 0xce291100 > sa0.ctl 0xcecfca00 > sa0.0 0xcecfcd00 > nsa0.0 0xc7f9e900 > esa0.0 0xcecf3000 > sa0 0xc8d42b00 > nsa0 0xcecd7b00 > esa0 0xc90a7000 > sa0.1 0xcbda4c00 > nsa0.1 0xcecfcc00 > esa0.1 0xce21a400 > sa0.2 0xceccb600 > nsa0.2 0xcece3b00 > esa0.2 0xc8e01d00 > sa0.3 0xcecd4600 > nsa0.3 0xceccb300 > esa0.3 0xcecfc500 >=20 > I think that's all I could tell by now ... > --=20 > cheers, J"org .-.-. --... ...-- -.. . DL8DTL >=20 > http://www.sax.de/~joerg/ NIC: JW11-RIPE > Never trust an operating system you don't have sources for. ;-) --K5FRBbPSksISZ1JK Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk3WzHQACgkQC3+MBN1Mb4gA2gCdH1IBuR7JGlSJP6CQA1r+WwDI uuQAoMTQvvRLUF0XHoxkJMa8qMaAxvqv =Hxmp -----END PGP SIGNATURE----- --K5FRBbPSksISZ1JK--