Date: Tue, 15 Apr 2014 12:15:18 +0300 From: Andriy Gapon <avg@FreeBSD.org> To: Phil Murray <pmurray@nevada.net.nz> Cc: freebsd-fs@FreeBSD.org, stable@FreeBSD.org, Matthew Ahrens <mahrens@delphix.com> Subject: Re: Panic in ZFS, solaris assert: sa.sa_magic == 0x2F505A Message-ID: <534CF8A6.1050205@FreeBSD.org> In-Reply-To: <E44610BC-B49E-4320-B2A9-FAC1A6854FBE@nevada.net.nz> References: <A746AB69-2EB9-47C6-958D-965DF5DF8AA6@nevada.net.nz> <5347C5A0.3030806@FreeBSD.org> <E44610BC-B49E-4320-B2A9-FAC1A6854FBE@nevada.net.nz>
next in thread | previous in thread | raw e-mail | index | archive | help
on 15/04/2014 08:39 Phil Murray said the following: > > On 11/04/2014, at 10:36 pm, Andriy Gapon <avg@FreeBSD.org> wrote: > >> on 11/04/2014 11:02 Phil Murray said the following: >>> Hi there, >>> >>> I’ve recently experienced two kernel panics on 8.4-RELEASE (within 2 days of each other, and both around the same time of day oddly) with ZFS. Sorry no dump available, but panic below. >>> >>> Any ideas where to start solving this? Will upgrading to 9 (or 10) solve it? >> >> By chance, could the system be running zfs recv at the times when the panics >> happened? > > I think it might be related to this bug reported on ZFS-on-linux when upgrading from v3 -> v5, which is exactly what I’ve done on this machine: > > https://github.com/zfsonlinux/zfs/issues/2025 > > In my case, the bogus sa.sa_magic value looks like this: > > panic:solaris asset: sa.sa_magic == 0x2F505A (0x5112fb3d == 0x2f505a), file: > > $ date -r 0x5112fb3d > Thu Feb 7 13:54:21 NZDT 2013 Great job finding that ZoL bug report! And very good job done by people who analyzed the problem. Below is my guess about what could be wrong. A thread is changing file attributes and it could end up calling zfs_sa_upgrade() to convert file's bonus from DMU_OT_ZNODE to DMU_OT_SA. The conversion is achieved in two steps: - dmu_set_bonustype() to change the bonus type in the dnode - sa_replace_all_by_template_locked() to re-populate the bonus data dmu_set_bonustype() calls dnode_setbonus_type() which does the following: dn->dn_bonustype = newtype; dn->dn_next_bonustype[tx->tx_txg & TXG_MASK] = dn->dn_bonustype; Concurrently, the sync thread can run into the dnode if it was dirtied in an earlier txg. The sync thread calls dmu_objset_userquota_get_ids() via dnode_sync(). dmu_objset_userquota_get_ids() uses dn_bonustype that has the new value, but the data corresponding to the txg being sync-ed is still in the old format. As I understand, dmu_objset_userquota_get_ids() already uses dmu_objset_userquota_find_data() when before == B_FALSE to find a proper copy of the data corresponding to the txg being sync-ed. So, I think that in that case dmu_objset_userquota_get_ids() should also use values of dn_bonustype and dn_bonuslen that correspond to the txg. If I am not mistaken, those values could be deduced from dn_next_bonustype[tx->tx_txg & TXG_MASK] plus dn_phys->dn_bonustype and dn_next_bonuslen[tx->tx_txg & TXG_MASK] plus dn_phys->dn_bonuslen. -- Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?534CF8A6.1050205>