Date: Mon, 31 May 2021 17:43:56 +0200 From: Dimitry Andric <dim@FreeBSD.org> To: Mateusz Guzik <mjguzik@gmail.com> Cc: FreeBSD Current <freebsd-current@freebsd.org>, Rick Macklem <rmacklem@freebsd.org> Subject: Re: Panics in recent NFS server Message-ID: <5A985368-32EF-4FD1-A325-0CC1600E90E0@FreeBSD.org> In-Reply-To: <CAGudoHGsnMtwDRKoCYLio_SCm3HZdYU7qF=uCHEb0y_HS5m-Ng@mail.gmail.com> References: <EA017AF6-25C8-48FC-B52F-D83D9DD314AA@FreeBSD.org> <CAGudoHHd3QVT6mefMcELRKa7nPptjVuc=D-agoHUNY4pfOz8tg@mail.gmail.com> <CAGudoHGk0mrmiLHqhfvCA596kxZa8vD1ex%2BQOgLBE0vt%2B7OJrA@mail.gmail.com> <CAGudoHGsnMtwDRKoCYLio_SCm3HZdYU7qF=uCHEb0y_HS5m-Ng@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_160E9DA0-7FF3-42C8-A506-B1D6E015EE57
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii
Yes, this works for me too. I first tried reverting
d81aefa8b7dd8cbeffeda541fca9962802404983, but this looked a bit more
sane. :)
-Dimitry
> On 31 May 2021, at 17:03, Mateusz Guzik <mjguzik@gmail.com> wrote:
>=20
> I reproduced the panic, things work for me with the patch below.
> However, there may be more to it so I'm going to ask Rick to weigh in.
> but short version is that length returned by nfsrv_parsename is off by
> one compared to copyinstr.
>=20
> diff --git a/sys/fs/nfsserver/nfs_nfsdsubs.c =
b/sys/fs/nfsserver/nfs_nfsdsubs.c
> index 2b6e17752544..8c7db36bbd05 100644
> --- a/sys/fs/nfsserver/nfs_nfsdsubs.c
> +++ b/sys/fs/nfsserver/nfs_nfsdsubs.c
> @@ -2065,7 +2065,7 @@ nfsrv_parsename(struct nfsrv_descript *nd, char
> *bufp, u_long *hashp,
> }
> }
> *tocp =3D '\0';
> - *outlenp =3D (size_t)outlen;
> + *outlenp =3D (size_t)outlen + 1;
> if (hashp !=3D NULL)
> *hashp =3D hash;
> nfsmout:
>=20
>=20
> On 5/31/21, Mateusz Guzik <mjguzik@gmail.com> wrote:
>> On 5/31/21, Mateusz Guzik <mjguzik@gmail.com> wrote:
>>> It's probably my commit d81aefa8b7dd8cbeffeda541fca9962802404983 ,
>>> I'll look at this later.
>>=20
>> Well let me rephrase. While the panic was added in said commit, I
>> suspect the bug is on nfs side -- it has its own namei variant which =
I
>> suspect is managing ni_pathlen in a manner different than the
>> original, it just happens to not panic on kernels prior to the above
>> change.
>>=20
>>>=20
>>> On 5/31/21, Dimitry Andric <dim@freebsd.org> wrote:
>>>> Hi,
>>>>=20
>>>> I recently upgraded a -CURRENT NFS server from 2021-05-12 to today
>>>> (2021-05-31), and when the first NFS client attempted to connect, I =
got
>>>> this
>>>> panic:
>>>>=20
>>>> panic: lookup: expected nul at 0xfffff800104b3002; string [dim]
>>>>=20
>>>> cpuid =3D 0
>>>> time =3D 1622463863
>>>> KDB: stack backtrace:
>>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>>>> 0xfffffe00747e89b0
>>>> vpanic() at vpanic+0x187/frame 0xfffffe00747e8a10
>>>> panic() at panic+0x43/frame 0xfffffe00747e8a70
>>>> lookup() at lookup+0xad2/frame 0xfffffe00747e8b10
>>>> nfsvno_namei() at nfsvno_namei+0x1a4/frame 0xfffffe00747e8bc0
>>>> nfsrvd_lookup() at nfsrvd_lookup+0x191/frame 0xfffffe00747e8eb0
>>>> nfsrvd_dorpc() at nfsrvd_dorpc+0xfab/frame 0xfffffe00747e90c0
>>>> nfssvc_program() at nfssvc_program+0x604/frame 0xfffffe00747e92a0
>>>> svc_run_internal() at svc_run_internal+0xa72/frame =
0xfffffe00747e93d0
>>>> svc_run() at svc_run+0x250/frame 0xfffffe00747e9430
>>>> nfsrvd_nfsd() at nfsrvd_nfsd+0x33c/frame 0xfffffe00747e9590
>>>> nfssvc_nfsd() at nfssvc_nfsd+0x473/frame 0xfffffe00747e9aa0
>>>> sys_nfssvc() at sys_nfssvc+0xc7/frame 0xfffffe00747e9ac0
>>>> amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe00747e9bf0
>>>> fast_syscall_common() at fast_syscall_common+0xf8/frame
>>>> 0xfffffe00747e9bf0
>>>> --- syscall (155, FreeBSD ELF64, sys_nfssvc), rip =3D 0x8011aa59a, =
rsp =3D
>>>> 0x7fffffffe4e8, rbp =3D 0x7fffffffe780 ---
>>>> KDB: enter: panic
>>>>=20
>>>> __curthread ()
>>>> at =
/share/dim/src/freebsd/src-dim/sys/amd64/include/pcpu_aux.h:55
>>>> 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" =
(offsetof(struct pcpu,
>>>> (kgdb) #0 __curthread ()
>>>> at =
/share/dim/src/freebsd/src-dim/sys/amd64/include/pcpu_aux.h:55
>>>> #1 doadump (textdump=3Dtextdump@entry=3D0)
>>>> at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:399
>>>> #2 0xffffffff804cca5a in db_dump (dummy=3D<optimized out>,
>>>> dummy2=3D<unavailable>, dummy3=3D<unavailable>, =
dummy4=3D<unavailable>)
>>>> at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:575
>>>> #3 0xffffffff804cc912 in db_command (last_cmdp=3D<optimized out>,
>>>> cmd_table=3D<optimized out>, dopager=3Ddopager@entry=3D1)
>>>> at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:482
>>>> #4 0xffffffff804cc58d in db_command_loop ()
>>>> at /share/dim/src/freebsd/src-dim/sys/ddb/db_command.c:535
>>>> #5 0xffffffff804cfd06 in db_trap (type=3D<optimized out>, =
code=3D<optimized
>>>> out>)
>>>> at /share/dim/src/freebsd/src-dim/sys/ddb/db_main.c:270
>>>> #6 0xffffffff80c69f17 in kdb_trap (type=3Dtype@entry=3D3,
>>>> code=3Dcode@entry=3D0,
>>>> tf=3Dtf@entry=3D0xfffffe00747e88e0)
>>>> at /share/dim/src/freebsd/src-dim/sys/kern/subr_kdb.c:727
>>>> #7 0xffffffff810d7aee in trap (frame=3D0xfffffe00747e88e0)
>>>> at /share/dim/src/freebsd/src-dim/sys/amd64/amd64/trap.c:576
>>>> #8 <signal handler called>
>>>> #9 kdb_enter (why=3D0xffffffff812d3d27 "panic", msg=3D<optimized =
out>)
>>>> at /share/dim/src/freebsd/src-dim/sys/kern/subr_kdb.c:506
>>>> #10 0xffffffff80c1d248 in vpanic (
>>>> fmt=3D0xffffffff8129dfef "%s: expected nul at %p; string =
[%s]\n",
>>>> ap=3D<optimized out>, ap@entry=3D0xfffffe00747e8a50)
>>>> at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:907
>>>> #11 0xffffffff80c1cfd3 in panic (
>>>> fmt=3D0xffffffff81e9b9c8 <cnputs_mtx> =
"=3D\t)\201\377\377\377\377")
>>>> at /share/dim/src/freebsd/src-dim/sys/kern/kern_shutdown.c:843
>>>> #12 0xffffffff80cfa992 in lookup (ndp=3Dndp@entry=3D0xfffffe00747e8d9=
0)
>>>> at /share/dim/src/freebsd/src-dim/sys/kern/vfs_lookup.c:919
>>>> #13 0xffffffff80b33f84 in nfsvno_namei =
(nd=3Dnd@entry=3D0xfffffe00747e9100,
>>>> ndp=3Dndp@entry=3D0xfffffe00747e8d90, dp=3D<optimized out>,
>>>> dp@entry=3D0xfffff80010544380, islocked=3D<optimized out>,
>>>> islocked@entry=3D0,
>>>> exp=3Dexp@entry=3D0xfffffe00747e8fd8, =
p=3Dp@entry=3D0xfffffe00bbfa3000,
>>>> retdirp=3D0xfffffe00747e8e78)
>>>> at
>>>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdport.c:597
>>>> #14 0xffffffff80b266a1 in nfsrvd_lookup (nd=3D0xfffffe00747e9100,
>>>> isdgram=3D<optimized out>, dp=3D0xfffff80010544380,
>>>> vpp=3D0xfffffe00747e9010,
>>>> fhp=3D0xfffffe00747e9074, exp=3D0xfffffe00747e8fd8)
>>>> at
>>>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdserv.c:607
>>>> #15 0xffffffff80b1073b in nfsrvd_compound (nd=3D0xfffffe00747e9100,
>>>> isdgram=3D0,
>>>> tag=3D0xf <error: Cannot access memory at address 0xf>, =
taglen=3D6,
>>>> minorvers=3D4294967294)
>>>> at
>>>> =
/share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdsocket.c:1098
>>>> #16 nfsrvd_dorpc (nd=3Dnd@entry=3D0xfffffe00747e9100,
>>>> isdgram=3Disdgram@entry=3D0,
>>>> tag=3D0xf <error: Cannot access memory at address 0xf>, =
taglen=3D6,
>>>> minorvers=3D4294967294)
>>>> at
>>>> =
/share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdsocket.c:626
>>>> #17 0xffffffff80b24c44 in nfs_proc (nd=3D0xfffffe00747e9100,
>>>> xid=3D<optimized out>, xprt=3D0xfffff80003a14e00, rpp=3D<optimized=
out>)
>>>> at
>>>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:402
>>>> #18 nfssvc_program (rqst=3D0xfffff80010455800, =
xprt=3D0xfffff80003a14e00)
>>>> at
>>>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:288
>>>> #19 0xffffffff80edead2 in svc_executereq (rqstp=3D0xfffff80010455800)=
>>>> at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1037
>>>> #20 svc_run_internal (grp=3D<optimized out>, =
grp@entry=3D0xfffff800100e6100,
>>>> ismaster=3Dismaster@entry=3D1)
>>>> at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1313
>>>> #21 0xffffffff80eddf80 in svc_run (pool=3D<optimized out>)
>>>> at /share/dim/src/freebsd/src-dim/sys/rpc/svc.c:1392
>>>> #22 0xffffffff80b251ec in nfsrvd_nfsd (td=3D<optimized out>,
>>>> td@entry=3D0xfffffe00bbfa3000, =
args=3Dargs@entry=3D0xfffffe00747e9660)
>>>> at
>>>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdkrpc.c:561
>>>> #23 0xffffffff80b3ec93 in nfssvc_nfsd (td=3D0xfffffe00bbfa3000,
>>>> uap=3D<optimized out>)
>>>> at
>>>> /share/dim/src/freebsd/src-dim/sys/fs/nfsserver/nfs_nfsdport.c:3714
>>>> #24 0xffffffff80e6f647 in sys_nfssvc (td=3D0xfffffe00bbfa3000,
>>>> uap=3D0xfffffe00bbfa33e8)
>>>> at /share/dim/src/freebsd/src-dim/sys/nfs/nfs_nfssvc.c:111
>>>> #25 0xffffffff810d891e in syscallenter (td=3D<optimized out>)
>>>> at
>>>> =
/share/dim/src/freebsd/src-dim/sys/amd64/amd64/../../kern/subr_syscall.c:1=
89
>>>> #26 amd64_syscall (td=3D0xfffffe00bbfa3000, traced=3D0)
>>>> at /share/dim/src/freebsd/src-dim/sys/amd64/amd64/trap.c:1156
>>>> #27 <signal handler called>
>>>> #28 0x00000008011aa59a in ?? ()
>>>>=20
>>>> Is anybody seeing this too? :)
>>>>=20
>>>> I can probably bisect, but it'll take quite a while.
>>>>=20
>>>> -Dimitry
>>>>=20
>>>>=20
>>>=20
>>>=20
>>> --
>>> Mateusz Guzik <mjguzik gmail.com>
>>>=20
>>=20
>>=20
>> --
>> Mateusz Guzik <mjguzik gmail.com>
>>=20
>=20
>=20
> --
> Mateusz Guzik <mjguzik gmail.com>
--Apple-Mail=_160E9DA0-7FF3-42C8-A506-B1D6E015EE57
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=signature.asc
Content-Type: application/pgp-signature;
name=signature.asc
Content-Description: Message signed with OpenPGP
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.2
iF0EARECAB0WIQR6tGLSzjX8bUI5T82wXqMKLiCWowUCYLUEPQAKCRCwXqMKLiCW
o014AKCELs2nS5ckmCT+1XWWu70ye8xlAgCg7yVROdaFZf0XF92USYltTe2qrp4=
=8L0A
-----END PGP SIGNATURE-----
--Apple-Mail=_160E9DA0-7FF3-42C8-A506-B1D6E015EE57--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5A985368-32EF-4FD1-A325-0CC1600E90E0>