gcore.py generates coredumps with broken thread stacks in gdb

Let's say I run glxgears, and wanted to take a core dump of it with drgn/contrib/gcore.py. (The actual process I'm trying to backtrace is in uninterruptible sleep, which the standard gcore cannot handle). ## Steps I generate two coredumps using gcore and drgn (to compare them). ```sh gcore -o glxgears (pgrep glxgears) # note, parentheses in fish shell are like $() or backticks /usr/share/drgn/contrib/gcore.py (pgrep glxgears) glxgears.core ``` ## Results Now if I run `gdb /bin/glxgears glxgears.core` on drgn's core dump, I can see five threads but cannot take a backtrace: ``` (gdb) bt #0 0x0000000000000246 in ?? () Backtrace stopped: Cannot access memory at address 0x0 (gdb) info thread Id Target Id Frame * 1 Thread 0x7f406beb3e00 (LWP 47770) 0x0000000000000246 in ?? () 2 Thread 0x7f405f9606c0 (LWP 47771) 0x0000000000000246 in ?? () 3 Thread 0x7f405f15f6c0 (LWP 47772) 0x0000000000000246 in ?? () 4 Thread 0x7f405e95e6c0 (LWP 47773) 0x0000000000000246 in ?? () 5 Thread 0x7f405e15d6c0 (LWP 47774) 0x0000000000000246 in ?? () ``` Running a backtrace on the standard gcore dump works fine: ``` nyanpasu64@ivy-fedora ~> gdb /bin/glxgears glxgears.47770 (gdb) bt #0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 #1 0x00007f406c011c3c in __internal_syscall_cancel (a1=a1@entry=140734148891400, a2=a2@entry=1, a3=a3@entry=-1, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=0, nr=7) at cancellation.c:49 #2 0x00007f406c011c84 in __syscall_cancel (a1=a1@entry=140734148891400, a2=a2@entry=1, a3=a3@entry=-1, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=0, nr=7) at cancellation.c:75 #3 0x00007f406c08b19e in __GI___poll (fds=fds@entry=0x7fff38f3d308, nfds=nfds@entry=1, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #4 0x00007f406bebc616 in poll (__fds=0x7fff38f3d308, __nfds=1, __timeout=-1) at /usr/include/bits/poll2.h:44 ... #21 draw_frame (dpy=<optimized out>, win=<optimized out>) at ../src/xdemos/glxgears.c:344 #22 event_loop (dpy=0x560f28dd1e10, win=46137346) at ../src/xdemos/glxgears.c:771 #23 main (argc=<optimized out>, argv=<optimized out>) at ../src/xdemos/glxgears.c:875 (gdb) info thread Id Target Id Frame * 1 LWP 47770 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 2 LWP 47774 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 3 LWP 47773 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 4 LWP 47772 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 5 LWP 47771 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 ``` ### Debugging I do think the memory (heap? stack?) is intact. In the working gcore dump I can view the contents of the dpy pointer: ``` (gdb) f 22 (gdb) p dpy $2 = (Display *) 0x560f28dd1e10 (gdb) p *dpy $3 = {ext_data = 0x0, free_funcs = 0x560f28dd4480, fd = 3, conn_checker = 0, proto_major_version = 11, proto_minor_version = 0, vendor = 0x560f28dd3410 "The X.Org Foundation", resource_base = 46137344, resource_mask = 2097151, resource_id = 0, resource_shift = 0, resource_alloc = 0x7f406c2b5ec0 <_XAllocID>, byte_order = 0, bitmap_unit = 32, bitmap_pad = 32, bitmap_bit_order = 0, nformats = 7, pixmap_format = 0x560f28dd3630, vnumber = 11, release = 12401009, head = 0x0, tail = 0x0, qlen = 0, last_request_read = 183, request = 2922, last_req = 0x7f406c34bb92 <dummy_request> "", buffer = 0x560f28ddb7f0 "\207\001\005", bufptr = 0x560f28ddb7f0 "\207\001\005", bufmax = 0x560f28ddf7f0 "", max_request_size = 65535, db = 0x0, synchandler = 0x0, display_name = 0x560f28dd3090 ":0", default_screen = 0, nscreens = 1, screens = 0x560f28dd3180, motion_buffer = 256, flags = 0, min_keycode = 8, max_keycode = 255, keysyms = 0x0, modifiermap = 0x0, keysyms_per_keycode = 0, xdefaults = 0x560f28dd3d10 "Xcursor.size:\t24\nXcursor.theme:\tbreeze_cursors\nXft.antialias:\t1\nXft.dpi:\t96\nXft.hinting:\t1\nXft.hintstyle:\thintslight\nXft.rgba:\trgb\n", scratch_buffer = 0x0, scratch_length = 0, ext_number = 3, ext_procs = 0x560f28e4b360, event_vec = {0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b8000 <_XWireToEvent> <repeats 34 times>, 0x7f406c2b3ce0 <_XUnknownWireEvent> <repeats 49 times>, 0x7f406c3156d0 <wire_to_event>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406be87260 <__glXWireToEvent> <repeats 17 times>, 0x7f406c2b3ce0 <_XUnknownWireEvent> <repeats 17 times>}, wire_vec = {0x7f406c2b3d10 <_XUnknownNativeEvent>, 0x7f406c2b3d10 <_XUnknownNativeEvent>, 0x0 <repeats 34 times>, 0x7f406c2b3d10 <_XUnknownNativeEvent> <repeats 58 times>, 0x7f406be87240 <__glXEventToWire> <repeats 17 times>, 0x7f406c2b3d10 <_XUnknownNativeEvent> <repeats 17 times>}, lock_meaning = 0, lock = 0x560f28dd34b0, async_handlers = 0x0, bigreq_size = 4194303, lock_fns = 0x560f28dd3130, idlist_alloc = 0x7f406c2b5a00 <_XAllocIDs>, key_bindings = 0x0, cursor_font = 0, atoms = 0x0, mode_switch = 0, num_lock = 0, context_db = 0x0, error_vec = 0x0, cms = {defaultCCCs = 0x0, clientCmaps = 0x560f28ea45c0 "\001", perVisualIntensityMaps = 0x0}, im_filters = 0x0, qfree = 0x560f28fb6db0, next_event_serial_num = 11, flushes = 0x0, im_fd_info = 0x0, im_fd_length = 0, conn_watchers = 0x0, watcher_count = 0, filedes = 0x560f28dd3600 "\003", savedsynchandler = 0x0, resource_max = 2097146, xcmisc_opcode = 0, xkb_info = 0x560f28dd3da0, trans_conn = 0x0, xcb = 0x560f28dd30b0, next_cookie = 0, generic_event_vec = {0x0 <repeats 128 times>}, generic_event_copy_vec = {0x0 <repeats 128 times>}, cookiejar = 0x0, error_threads = 0x0, exit_handler = 0x7f406c2b5f30 <_XDefaultIOErrorExit>, exit_handler_data = 0x0, in_ifevent = 0, ifevent_thread = 0} ``` I cannot view the stack in the gcore.py dump, but can still print memory: ``` (gdb) p *(Display *) 0x560f28dd1e10 $1 = {ext_data = 0x0, free_funcs = 0x560f28dd4480, fd = 3, conn_checker = 0, proto_major_version = 11, proto_minor_version = 0, vendor = 0x560f28dd3410 "The X.Org Foundation", resource_base = 46137344, resource_mask = 2097151, resource_id = 0, resource_shift = 0, resource_alloc = 0x7f406c2b5ec0 <_XAllocID>, byte_order = 0, bitmap_unit = 32, bitmap_pad = 32, bitmap_bit_order = 0, nformats = 7, pixmap_format = 0x560f28dd3630, vnumber = 11, release = 12401009, head = 0x0, tail = 0x0, qlen = 0, last_request_read = 183, request = 1067, last_req = 0x7f406c34bb92 <dummy_request> "", buffer = 0x560f28ddb7f0 "\207\001\005", bufptr = 0x560f28ddb7f0 "\207\001\005", bufmax = 0x560f28ddf7f0 "", max_request_size = 65535, db = 0x0, synchandler = 0x0, display_name = 0x560f28dd3090 ":0", default_screen = 0, nscreens = 1, screens = 0x560f28dd3180, motion_buffer = 256, flags = 0, min_keycode = 8, max_keycode = 255, keysyms = 0x0, modifiermap = 0x0, keysyms_per_keycode = 0, xdefaults = 0x560f28dd3d10 "Xcursor.size:\t24\nXcursor.theme:\tbreeze_cursors\nXft.antialias:\t1\nXft.dpi:\t96\nXft.hinting:\t1\nXft.hintstyle:\thintslight\nXft.rgba:\trgb\n", scratch_buffer = 0x0, scratch_length = 0, ext_number = 3, ext_procs = 0x560f28e4b360, event_vec = {0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b8000 <_XWireToEvent> <repeats 34 times>, 0x7f406c2b3ce0 <_XUnknownWireEvent> <repeats 49 times>, 0x7f406c3156d0 <wire_to_event>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406c2b3ce0 <_XUnknownWireEvent>, 0x7f406be87260 <__glXWireToEvent> <repeats 17 times>, 0x7f406c2b3ce0 <_XUnknownWireEvent> <repeats 17 times>}, wire_vec = {0x7f406c2b3d10 <_XUnknownNativeEvent>, 0x7f406c2b3d10 <_XUnknownNativeEvent>, 0x0 <repeats 34 times>, 0x7f406c2b3d10 <_XUnknownNativeEvent> <repeats 58 times>, 0x7f406be87240 <__glXEventToWire> <repeats 17 times>, 0x7f406c2b3d10 <_XUnknownNativeEvent> <repeats 17 times>}, lock_meaning = 0, lock = 0x560f28dd34b0, async_handlers = 0x0, bigreq_size = 4194303, lock_fns = 0x560f28dd3130, idlist_alloc = 0x7f406c2b5a00 <_XAllocIDs>, key_bindings = 0x0, cursor_font = 0, atoms = 0x0, mode_switch = 0, num_lock = 0, context_db = 0x0, error_vec = 0x0, cms = {defaultCCCs = 0x0, clientCmaps = 0x560f28ea45c0 "\001", perVisualIntensityMaps = 0x0}, im_filters = 0x0, qfree = 0x560f28fb6db0, next_event_serial_num = 11, flushes = 0x0, im_fd_info = 0x0, im_fd_length = 0, conn_watchers = 0x0, watcher_count = 0, filedes = 0x560f28dd3600 "\003", savedsynchandler = 0x0, resource_max = 2097146, xcmisc_opcode = 0, xkb_info = 0x560f28dd3da0, trans_conn = 0x0, xcb = 0x560f28dd30b0, next_cookie = 0, generic_event_vec = {0x0 <repeats 128 times>}, generic_event_copy_vec = {0x0 <repeats 128 times>}, cookiejar = 0x0, error_threads = 0x0, exit_handler = 0x7f406c2b5f30 <_XDefaultIOErrorExit>, exit_handler_data = 0x0, in_ifevent = 0, ifevent_thread = 0} ``` ---- I had thought the *list* of threads (`info thread`) was corrupted. I put Dolphin into uninterruptible sleep (ctrl+alt+c on `/run/user/1000/doc`, which is serviced by a FUSE filesystem from a hung `xdg-document-portal` process itself blocked on a FUSE operation). Then I ran gcore.py and loaded the core dump in gdb. There, `info thread` revealed only one thread, unlike Dolphin's usual zoo of worker threads. However I believe this to be actually accurate, since in htop, if I show threads with `H`, Dolphin does *not* have any child threads under its PID. The workers must've died at some point, leaving the main thread in uninterruptible sleep hell. ## System info Operating System: Fedora Linux 43 Kernel Version: 6.19.0-0.rc1.251219.dd9b004b.317.vanilla.fc43.x86_64 (64-bit) drgn installed from `dnf install drgn`.