Date: Mon, 27 Jan 2020 13:43:11 -0600 From: Valeri Galtsev <galtsev@kicp.uchicago.edu> To: hartzell@alerce.com Cc: freebsd-questions <freebsd-questions@freebsd.org> Subject: Re: How can I get useful debugging info from rust app's core files? Message-ID: <BE6478A6-6B8B-4ABC-9820-79B4183B324F@kicp.uchicago.edu> In-Reply-To: <24111.15366.843761.242233@alice.local> References: <24111.15366.843761.242233@alice.local>
next in thread | previous in thread | raw e-mail | index | archive | help
> On Jan 27, 2020, at 1:37 PM, George Hartzell <hartzell@alerce.com> =
wrote:
>=20
>=20
> Hi Folks,
>=20
> TL;DR
This remindes me the times of old radio amateur communications. When =
Morse code was used. There were many =E2=80=9Cstandard=E2=80=9D =
abbrebiations then. But these days, when we are typing so fast (excepl =
slowing down when doung actual system administration), this kind of =
looks out of place for me.
Valeri
> How can I build a rust application on an amd64 FreeBSD system so that
> I can use a debugger to examine its core files?
>=20
> Using a "dev" build, rust@1.38.0, `rust-gdb` and gdb@8.3.1 don't seem
> to get me there.
>=20
> ---
>=20
> I'm hoping for feedback on, or other places to ask, a question about
> getting debugging info from core dumps from a rust application on
> FreeBSD.
>=20
> Here's the scenario: I listen to music using the Logitech Media Server
> ("Slimdevices") ecosystem
> [https://en.wikipedia.org/wiki/Logitech_Media_Server]. There's a
> plugin [https://github.com/michaelherger/spotty] that lets me listen
> to spotify, via the librespot library. The application periodically
> crashes due to a SIGABRT. I'm unable to get debug info from the
> resulting core files.
>=20
> I'm working on an amd64 FreeNAS system, which runs FreeBSD 11.2 under
> the covers. I've installed "everything" from packages; in particular
>=20
> - `rustc --version` says: `rustc 1.38.0`
> - the core gdb is 6.1.1
> - the gdb from packages is 8.3.1.
>=20
> I've cloned the application and built it with `cargo build`, which
> seems to give me a "debugging binary".
>=20
> ```
> root@spotty-build:~/spotty/target/debug # file ./spotty
> ./spotty: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), =
dynamically
> linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.2, =
FreeBSD-style, with
> debug_info, not stripped
> root@spotty-build:~/spotty/target/debug #
> ```
>=20
> I ran it by hand in the background and killed it via `kill -ABRT
> <pid>` to generate a trivial core file so that I can figure out how to
> get debugging symbols in the deployed application. I think that this
> is rust-related, but perhaps my gdb skills are just rusty.
>=20
> - If I use the system gdb it seems to have trouble with version of the
> Dwarf debugging info:
>=20
> ```
> root@spotty-build:~/spotty/target/debug # gdb spotty =
/var/db/system/cores/spotty.core
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, =
and you are
> welcome to change it and/or distribute copies of it under certain =
conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for =
details.
> This GDB was configured as "amd64-marcel-freebsd"...Dwarf Error: =
wrong version in compilation unit header (is 4, should be 2) [in module =
/root/spotty/target/debug/spotty]
>=20
> Core was generated by `./spotty -n poodle'.
> Program terminated with signal 6, Aborted.
> #0 0x0000000803182b2c in ?? ()
> (gdb) where
> #0 0x0000000803182b2c in ?? ()
> #1 0x0000000803176bc1 in ?? ()
> #2 0x00007fffffffad60 in ?? ()
> #3 0x00000008031797cc in ?? ()
> #4 0x000000000001894b in ?? ()
> #5 0x0000000000000001 in ?? ()
> #6 0x00007fffffffadd0 in ?? ()
> #7 0x0000000803180858 in ?? ()
> #8 0x00007fffffffad90 in ?? ()
> #9 0x00000008040162b8 in ?? ()
> #10 0x0000000000000000 in ?? ()
> (gdb)
> ```
>=20
> - gdb@8.3.1 (from packages) seems to handle the DWARF debugging info
> version but still no joy:
>=20
> ```
> root@spotty-build:~/spotty/target/debug # /usr/local/bin/gdb =
--version
> GNU gdb (GDB) 8.3.1 [GDB v8.3.1 for FreeBSD]
> Copyright (C) 2019 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later =
<http://gnu.org/licenses/gpl.html>;
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> root@spotty-build:~/spotty/target/debug # /usr/local/bin/gdb spotty =
/var/db/system/cores/spotty.core
> GNU gdb (GDB) 8.3.1 [GDB v8.3.1 for FreeBSD]
> Copyright (C) 2019 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later =
<http://gnu.org/licenses/gpl.html>;
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "x86_64-portbld-freebsd11.3".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
>=20
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from spotty...
> [New LWP 100683]
> [New LWP 100623]
> [New LWP 100649]
>=20
> warning: Unexpected size of section `.reg-xstate/100683' in core =
file.
> Core was generated by `./spotty -n poodle'.
> Program terminated with signal SIGABRT, Aborted.
>=20
> warning: Unexpected size of section `.reg-xstate/100683' in core =
file.
> #0 0x0000000803182b2c in ?? ()
> [Current thread is 1 (LWP 100683)]
> warning: Missing auto-load script at offset 0 in section =
.debug_gdb_scripts
> of file /root/spotty/target/debug/spotty.
> Use `info auto-load python-scripts [REGEXP]' to list them.
> (gdb) where
> #0 0x0000000803182b2c in ?? ()
> #1 0x0000000803176bc1 in ?? ()
> #2 0x00007fffffffad60 in ?? ()
> #3 0x00000008031797cc in ?? ()
> #4 0x000000000001894b in ?? ()
> #5 0x0000000000000001 in ?? ()
> #6 0x00007fffffffadd0 in ?? ()
> #7 0x0000000803180858 in ?? ()
> #8 0x00007fffffffad90 in ?? ()
> #9 0x00000008040162b8 in ?? ()
> #10 0x0000000000000000 in ?? ()
> (gdb) quit
> root@spotty-build:~/spotty/target/debug #
> ```
>=20
> - using `rust-gdb` (and setting the path so that it picks up the newer
> gdb) clears up the debug script issue, but I'm still not getting
> useful line numbers:
>=20
> ```
> root@spotty-build:~/spotty # rust-gdb target/debug/spotty =
/var/db/system/cores/spotty.core
> GNU gdb (GDB) 8.3.1 [GDB v8.3.1 for FreeBSD]
> Copyright (C) 2019 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later =
<http://gnu.org/licenses/gpl.html>;
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "x86_64-portbld-freebsd11.3".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
>=20
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from target/debug/spotty...
> [New LWP 100683]
> [New LWP 100623]
> [New LWP 100649]
>=20
> warning: Unexpected size of section `.reg-xstate/100683' in core =
file.
> Core was generated by `./spotty -n poodle'.
> Program terminated with signal SIGABRT, Aborted.
>=20
> warning: Unexpected size of section `.reg-xstate/100683' in core =
file.
> #0 0x0000000803182b2c in ?? ()
> [Current thread is 1 (LWP 100683)]
> (gdb) where
> #0 0x0000000803182b2c in ?? ()
> #1 0x0000000803176bc1 in ?? ()
> #2 0x00007fffffffad60 in ?? ()
> #3 0x00000008031797cc in ?? ()
> #4 0x000000000001894b in ?? ()
> #5 0x0000000000000001 in ?? ()
> #6 0x00007fffffffadd0 in ?? ()
> #7 0x0000000803180858 in ?? ()
> #8 0x00007fffffffad90 in ?? ()
> #9 0x00000008040162b8 in ?? ()
> #10 0x0000000000000000 in ?? ()
> (gdb)
> ```
>=20
>=20
> I'd appreciate any help, suggestions or ... on how to get useful
> information about crashed from these core files.
>=20
> Thanks!
>=20
> g.
> _______________________________________________
> freebsd-questions@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to =
"freebsd-questions-unsubscribe@freebsd.org"
++++++++++++++++++++++++++++++++++++++++
Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247
++++++++++++++++++++++++++++++++++++++++
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BE6478A6-6B8B-4ABC-9820-79B4183B324F>