Reverse Debugging Rust

[DrRuhe]

I am stuck getting codeLLDB to connect to a rr replay session.

Currently I have this launch.json:

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Replay",
            "type": "lldb",
            "request": "custom",
            "processCreateCommands": ["gdb-remote 127.0.0.1:1234"],
            "reverseDebugging": true,
            "preLaunchTask": "replay-latest-rr-trace"
        }
    ]
}

With the following tasks.json:

{
    "version": "2.0.0",
    "tasks": [{
        "label": "replay-latest-rr-trace",
        "type": "shell",
        "command": "rr replay -s 1234 -d rust-gdb ./target/rr/$(cat ./target/rr/latest)",
        "group": {
            "kind": "build",
            "isDefault": true
        }
    }]
}

I record the traces using cargo rr and I would like to have a debug configuration for debugging the latest trace.

I took the reverse debugging example and tried to make it work, but so far it has been unsuccessful.

I removed the targetCreateCommands": ["target create ${workspaceFolder}/build/debuggee"], section, since I thought rr replay has all the necessary info and does not need to create a target.

Any Ideas?

[vadimcn]

Um, it would be helpful to qualify "has been unsuccessful". What actually happens? Do you see any errors?

If I had to guess looking at your setup, the debug session never starts because VSCode is waiting for preLaunchTask to complete (it's meant to be used for building). This can be worked around by using a background task, but it's tricky to do robustly, because you'd want to wait until rr replay ... fully spins up and opens the port before continuing. I would recommend running rr replay in a separate terminal.

[vadimcn]

Or is there a way to automatically get the current target to load from the trace that I am just not aware of?

I don't know. Perhaps rr simply does not implement the requisite query command.

Stepping for/backward went straight back to disassembly view.

This is expected, LLDB does not know to line-step in reverse. Just set a breakpoint on the line you want to return to and hit the "reverse run" button.

For better visualization I also tried using the "sourceLanguages": ["rust"] option which was met with a Property sourceLanguages is not allowed. error. Can this not be used with reverse debugging?

Should be. If the error comes from VSCode schema validation, try just ignoring it.

[DrRuhe]

So I tried using the "Reverse" Button to get to a previous breakpoint. This is the list of locations I was placed before actually arriving at the previous breakpoint:

• some internal syscall hook with this callstack:

70000002 (@70000002..70000037:3)
syscall_hook_internal (@syscall_hook_internal:304)
syscall_hook (@syscall_hook:30)

• a traced raw syscall with this callstack:

_raw_syscall (@_raw_syscall:16)
traced_raw_syscall (@traced_raw_syscall:49)
syscall_hook_internal (@syscall_hook_internal:304)
syscall_hook (@syscall_hook:30)
_syscall_hook_trampoline (@_syscall_hook_trampoline:26)
__morestack (@__morestack:4)
_syscall_hook_trampoline_48_3d_00_f0_ff_ff (@_syscall_hook_trampoline_48_3d_00_f0_ff_ff:4)
wait4 (@wait4:13)
wait4 (@wait4:13)

• another internal syscall hook
• some disassembly of the launch of a different process (this one has a few rust specific things in the callstack):

__spawnix (@__spawnix:106)
__spawni (@__spawni:13)
posix_spawnp@@GLIBC_2.15 (@posix_spawnp@@GLIBC_2.15:7)
std::sys::unix::process::process_inner::<impl std::sys::unix::process::process_common::Command>::posix_spawn (@std::sys::unix::process::process_inner::<impl std::sys::unix::process::process_common::Command>::spawn:505)
std::sys::unix::process::process_inner::<impl std::sys::unix::process::process_common::Command>::spawn (@std::sys::unix::process::process_inner::<impl std::sys::unix::process::process_common::Command>::spawn:472)
std::process::Command::spawn (@std::process::Command::spawn:12)
waila::Rule::check (/home/caspar/Documents/waila/src/main.rs:58)
waila::Config::valid_rules::{{closure}} (/home/caspar/Documents/waila/src/main.rs:180)
waila::graph::topological_filter_map (/home/caspar/Documents/waila/src/graph.rs:96)
waila::Config::valid_rules (/home/caspar/Documents/waila/src/main.rs:177)
waila::Config::show_summary (/home/caspar/Documents/waila/src/main.rs:204)
waila::main (/home/caspar/Documents/waila/src/main.rs:235)
core::ops::function::FnOnce::call_once (@core::ops::function::FnOnce::call_once:6)
std::sys_common::backtrace::__rust_begin_short_backtrace (@std::sys_common::backtrace::__rust_begin_short_backtrace:6)
std::rt::lang_start::{{closure}} (@std::rt::lang_start::{{closure}}:7)
core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once (@std::rt::lang_start_internal:256)
std::panicking::try::do_call (@std::rt::lang_start_internal:255)
std::panicking::try (@std::rt::lang_start_internal:255)
std::panic::catch_unwind (@std::rt::lang_start_internal:255)
std::rt::lang_start_internal::{{closure}} (@std::rt::lang_start_internal:255)

• And 5 times more internal syscall hook and raw traced syscalls.

I have not set breakpoints at syscalls. Is this expected behavior?

[DrRuhe]

I don't know. Perhaps rr simply does not implement the requisite query command.

I am a bit clueless as to how I can test this. Do I have to call this myself or does CodeLLDB do this automatically? Do I need the rr script from the wiki to send these commands? Can this even be sent through the debugging console?

[vadimcn]

I have not set breakpoints at syscalls. Is this expected behavior?

No. Sorry, I don't know what's going on. When I tested it a few years ago, it worked as expected.
As I said, something might have changed in rr or in LLDB or both. Given that authors of rr probably test it only with GDB, this would not be surprising.

[vogelsgesang]

Or is there a way to automatically get the current target to load from the trace that I am just not aware of?

I don't know. Perhaps rr simply does not implement the requisite query command.

For what it's worth: rr does implement this command (https://github.com/rr-debugger/rr/blob/05b8393c3eefef3dba96c42915d442a47a1d1308/src/GdbServer.cc#L426, https://github.com/rr-debugger/rr/blob/6bedb9a1d202a1ce6529150b07879beda7684415/src/GdbConnection.cc). It seems to be lldb which doesn't use it.