Date: Tue, 9 Sep 2014 12:18:58 -0700 From: Mike Carlson <mike@bayphoto.com> To: Steven Hartland <smh@freebsd.org> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS Panic (3rd time) Message-ID: <CA%2BKW7xSAnJRYdxGvJt4WNe%2BSVZXjWCwVpmieKU1wnfwyW7-LLg@mail.gmail.com> In-Reply-To: <55240AD4BA84415EBCE5B5B2F9DC22F3@multiplay.co.uk> References: <CA%2BKW7xQLnbcfZPWFyNAFnpQ7V-TVW=Z=M9c5GTBuuO6SKz9X7A@mail.gmail.com> <55240AD4BA84415EBCE5B5B2F9DC22F3@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Thanks Steve
That is troubling news!
zdb without parameters:
# zdb
data:
version: 5000
name: 'data'
state: 0
txg: 26
pool_guid: 17275286806962484556
hostid: 1566810261
hostname: 'zfs-2.discdrive.bayphoto.com'
vdev_children: 2
vdev_tree:
type: 'root'
id: 0
guid: 17275286806962484556
create_txg: 4
children[0]:
type: 'raidz'
id: 0
guid: 18387238254393289487
nparity: 2
metaslab_array: 33
metaslab_shift: 37
ashift: 12
asize: 26005123629056
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 12919111950950057118
path: '/dev/da0p1.nop'
phys_path: '/dev/da0p1.nop'
whole_disk: 1
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 6204755784205312226
path: '/dev/da1p1.nop'
phys_path: '/dev/da1p1.nop'
whole_disk: 1
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 2095825539678825225
path: '/dev/da2p1.nop'
phys_path: '/dev/da2p1.nop'
whole_disk: 1
create_txg: 4
children[3]:
type: 'disk'
id: 3
guid: 13288853293260483450
path: '/dev/da3p1.nop'
phys_path: '/dev/da3p1.nop'
whole_disk: 1
create_txg: 4
children[4]:
type: 'disk'
id: 4
guid: 17651239728056787408
path: '/dev/da4p1.nop'
phys_path: '/dev/da4p1.nop'
whole_disk: 1
create_txg: 4
children[5]:
type: 'disk'
id: 5
guid: 10121035556839569555
path: '/dev/da5p1.nop'
phys_path: '/dev/da5p1.nop'
whole_disk: 1
create_txg: 4
children[6]:
type: 'disk'
id: 6
guid: 8141814630245447282
path: '/dev/da6p1.nop'
phys_path: '/dev/da6p1.nop'
whole_disk: 1
create_txg: 4
children[7]:
type: 'disk'
id: 7
guid: 12107673881548157163
path: '/dev/da7p1.nop'
phys_path: '/dev/da7p1.nop'
whole_disk: 1
create_txg: 4
children[8]:
type: 'disk'
id: 8
guid: 13722803165058102841
path: '/dev/da8p1.nop'
phys_path: '/dev/da8p1.nop'
whole_disk: 1
create_txg: 4
children[9]:
type: 'disk'
id: 9
guid: 3812972074943976242
path: '/dev/da9p1.nop'
phys_path: '/dev/da9p1.nop'
whole_disk: 1
create_txg: 4
children[10]:
type: 'disk'
id: 10
guid: 3387379670710299146
path: '/dev/da10p1.nop'
phys_path: '/dev/da10p1.nop'
whole_disk: 1
create_txg: 4
children[11]:
type: 'disk'
id: 11
guid: 17209778087441255883
path: '/dev/da11p1.nop'
phys_path: '/dev/da11p1.nop'
whole_disk: 1
create_txg: 4
children[12]:
type: 'disk'
id: 12
guid: 14155860984589567095
path: '/dev/da12p1.nop'
phys_path: '/dev/da12p1.nop'
whole_disk: 1
create_txg: 4
children[1]:
type: 'raidz'
id: 1
guid: 7358812460992449364
nparity: 2
metaslab_array: 37
metaslab_shift: 37
ashift: 12
asize: 24004729503744
is_log: 0
create_txg: 16
children[0]:
type: 'disk'
id: 0
guid: 16184992168562751178
path: '/dev/da13p1.nop'
phys_path: '/dev/da13p1.nop'
whole_disk: 1
create_txg: 16
children[1]:
type: 'disk'
id: 1
guid: 17273520967287205930
path: '/dev/da14p1.nop'
phys_path: '/dev/da14p1.nop'
whole_disk: 1
create_txg: 16
children[2]:
type: 'disk'
id: 2
guid: 17965068062958146105
path: '/dev/da15p1.nop'
phys_path: '/dev/da15p1.nop'
whole_disk: 1
create_txg: 16
children[3]:
type: 'disk'
id: 3
guid: 6440721779503392985
path: '/dev/da16p1.nop'
phys_path: '/dev/da16p1.nop'
whole_disk: 1
create_txg: 16
children[4]:
type: 'disk'
id: 4
guid: 5129596340557895557
path: '/dev/da17p1.nop'
phys_path: '/dev/da17p1.nop'
whole_disk: 1
create_txg: 16
children[5]:
type: 'disk'
id: 5
guid: 13197465381631225536
path: '/dev/da18p1.nop'
phys_path: '/dev/da18p1.nop'
whole_disk: 1
create_txg: 16
children[6]:
type: 'disk'
id: 6
guid: 13521709969101776408
path: '/dev/da19p1.nop'
phys_path: '/dev/da19p1.nop'
whole_disk: 1
create_txg: 16
children[7]:
type: 'disk'
id: 7
guid: 7379733698654539430
path: '/dev/da20p1.nop'
phys_path: '/dev/da20p1.nop'
whole_disk: 1
create_txg: 16
children[8]:
type: 'disk'
id: 8
guid: 10498685535964391283
path: '/dev/da21p1.nop'
phys_path: '/dev/da21p1.nop'
whole_disk: 1
create_txg: 16
children[9]:
type: 'disk'
id: 9
guid: 12185894059804382853
path: '/dev/da22p1.nop'
phys_path: '/dev/da22p1.nop'
whole_disk: 1
create_txg: 16
children[10]:
type: 'disk'
id: 10
guid: 6545374147807002239
path: '/dev/da23p1.nop'
phys_path: '/dev/da23p1.nop'
whole_disk: 1
create_txg: 16
children[11]:
type: 'disk'
id: 11
guid: 1183756296391348826
path: '/dev/da24p1.nop'
phys_path: '/dev/da24p1.nop'
whole_disk: 1
create_txg: 16
features_for_read:
Is there a way to "rollback" a transaction?
The behavior that I can see at the moment is any zpool or zfs command that
touches 'data' causes all of the drives to "scan", and then after a minute
the system panics.
On Tue, Sep 9, 2014 at 12:04 PM, Steven Hartland <smh@freebsd.org> wrote:
> You panic is being cause by the dereference of a null vdev
> in vdev_rele but the issue seems to start earlier as the
> zio value passed to vdev_mirror_scrub_done also looks invalid.
>
> The call point for this should be in zio_done specifically
> if (zio->io_done)
> zio->io_done(zio);
>
> So if zio is actually invalid then something is really wrong
> which is unfortunatelly not much help :(
>
> With regards zdb try not passing a pool.
>
> ----- Original Message ----- From: "Mike Carlson" <mike@bayphoto.com>
>
> snip...
>
>>
>> #7 0xffffffff81860336 in vdev_rele (vd=0x0) at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/vdev.c:1556
>> #8 0xffffffff81866800 in vdev_mirror_scrub_done (zio=0x3) at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/vdev_mirror.c:193
>> #9 0xffffffff81866344 in vdev_mirror_io_start (zio=0xfffff80142733d00)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/vdev_mirror.c:286
>> #10 0xffffffff818803c4 in zio_vdev_io_start (zio=0xfffff8013eb20b10)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/zio.c:2571
>> #11 0xffffffff8187d796 in zio_suspend (spa=0xfffff8000e122000,
>> zio=0xfffff8013eb20b10)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/zio.c:1456
>> #12 0xffffffff8180ddec in arc_read (pio=0x0, spa=0xfffff8001e37d000,
>> bp=<value optimized out>, done=0x2, private=0x0, priority=6, zio_flags=0,
>> arc_flags=<value optimized out>, zb=0xfffff8001ed06558) at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/arc.c:3371
>> #13 0xffffffff818268f1 in traverse_prefetcher (spa=0xfffff8001e37d000,
>> zilog=0xf01ff, bp=<value optimized out>, zb=<value optimized out>,
>> dnp=0xfffff80142733d00, arg=<value optimized out>) at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/dmu_traverse.c:451
>> #14 0xffffffff81825d14 in traverse_visitbp (td=0xfffffe104c763900,
>> dnp=0xfffffe000fe64800, bp=0xfffffe000fe64980, zb=0xfffffe104c762e88)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/dmu_traverse.c:250
>> #15 0xffffffff8182677f in traverse_dnode (td=0xfffffe104c763900,
>> dnp=0xfffffe000fe64800, objset=110, object=26823324)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/dmu_traverse.c:417
>> #16 0xffffffff81826487 in traverse_visitbp (td=0xfffffe104c763900,
>> dnp=0xfffffe000fe61000, bp=0xfffffe001285ea00, zb=0xfffffe104c7630a8)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/dmu_traverse.c:309
>> #17 0xffffffff81825ee3 in traverse_visitbp (td=0xfffffe104c763900,
>> dnp=0xfffff8013ed8f000, bp=0xfffffe0012867a00, zb=0xfffffe104c7631d8)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/dmu_traverse.c:284
>> #18 0xffffffff81825ee3 in traverse_visitbp (td=0xfffffe104c763900,
>> dnp=0xfffff8013ed8f000, bp=0xfffffe0012842980, zb=0xfffffe104c763308)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/dmu_traverse.c:284
>> #19 0xffffffff81825ee3 in traverse_visitbp (td=0xfffffe104c763900,
>> dnp=0xfffff8013ed8f000, bp=0xfffffe0012848000, zb=0xfffffe104c763438)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/dmu_traverse.c:284
>> #20 0xffffffff81825ee3 in traverse_visitbp (td=0xfffffe104c763900,
>> dnp=0xfffff8013ed8f000, bp=0xfffffe000fe6d000, zb=0xfffffe104c763568)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/dmu_traverse.c:284
>> #21 0xffffffff81825ee3 in traverse_visitbp (td=0xfffffe104c763900,
>> dnp=0xfffff8013ed8f000, bp=0xfffffe001282d000, zb=0xfffffe104c763698)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/dmu_traverse.c:284
>> #22 0xffffffff81825ee3 in traverse_visitbp (td=0xfffffe104c763900,
>> dnp=0xfffff8013ed8f000, bp=0xfffff8013ed8f040, zb=0xfffffe104c763758)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/dmu_traverse.c:284
>> #23 0xffffffff81826714 in traverse_dnode (td=0xfffffe104c763900,
>> dnp=0xfffff8013ed8f000, objset=110, object=0)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/dmu_traverse.c:407
>> #24 0xffffffff81826190 in traverse_visitbp (td=0xfffffe104c763900,
>> dnp=0x0,
>> bp=0xfffff8013ed7ea80, zb=0xfffffe104c7638e0)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/dmu_traverse.c:338
>> #25 0xffffffff81825af6 in traverse_prefetch_thread
>> (arg=0xfffffe104cd2f0e0)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/
>> common/fs/zfs/dmu_traverse.c:470
>> #26 0xffffffff817fcc00 in taskq_run (arg=0xfffff801445bef30,
>> pending=983551)
>> at
>> /usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/
>> opensolaris_taskq.c:109
>> #27 0xffffffff808f5c26 in taskqueue_run_locked (queue=0xfffff8000e0eba00)
>> at /usr/src/sys/kern/subr_taskqueue.c:333
>> #28 0xffffffff808f64a8 in taskqueue_thread_loop (arg=<value optimized
>> out>)
>> at /usr/src/sys/kern/subr_taskqueue.c:535
>> #29 0xffffffff80881a4a in fork_exit (callout=0xffffffff808f6400
>> <taskqueue_thread_loop>, arg=0xfffff8000e10aac0, frame=0xfffffe104c763a40)
>> at /usr/src/sys/kern/kern_fork.c:995
>> #30 0xffffffff80c75a6e in fork_trampoline () at
>> /usr/src/sys/amd64/amd64/exception.S:606
>> #31 0x0000000000000000 in ?? ()
>> Current language: auto; currently minimal
>> _______________________________________________
>> freebsd-fs@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
>>
>>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BKW7xSAnJRYdxGvJt4WNe%2BSVZXjWCwVpmieKU1wnfwyW7-LLg>