Remote debug an android app that is on a truly remote android device

[mark2185]

I want to be able to run a process on a truly remote android (connected to a remote server), connect to it via lldb and debug C++ code.

I've tried a few different things to do it manually and in the end I managed to do it with lldb-server gdbserver :<port> and then connecting to that via gdb-remote.

As for connecting to a locally plugged in android device I used platform select remote-android and platform connect, which enables me to do everything with the debugger one would expect (breakpoints, stacks, registers).

The gdb-remote approach works from vscode as well, however I am unable to hit any breakpoints so I assume platform connect does some other things in the background. This is my configuration:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Remote launch",
            "type": "lldb",
            "request": "custom",
            "processCreateCommands": [
                "gdb-remote localhost:6666",
                "attach 14036"
            ]
        }
    ]
}

I've seen the troubleshooting instructions and found out that there are no modules in the current debuggee, but that's not why I'm opening this discussion.

My question is this - have you had any experience with platform connect to a truly remote android device? Or some other approach? I've been digging this rabbit hole for a while now so I'm open to all suggestions.

[mark2185]

Alright, platform connect is no-go since it takes up a random port.

So, new question - how do I give lldb debug symbols when attaching to an android process through gdb-remote?

If I try target create <the-apk-that-I-install-on-the-android>.apk it says that it doesn't support the architecture,
if I try lldb --arch <armv7|armv8|aarch64> <the-apk-that-I-install-on-the-android>.apk it says the architecture does not match.

My question is probably better worded as "how to debug native code on an android, without using android studio, using lldb". Do you have any hints or tips?

[vadimcn]

Ah, wait, you are not specifying where to find the local copy of the debuggee. Symbol parsing happens on the local machine. Use "file" command for that.

[mark2185]

Hm, now we're back to the start, since the debuggee is an .apk? It does contain a .so where the code resides, and target create <path-to-that.so> does load it, and breakpoints are red, but they are still not being hit. I'll try poking some more.

An .apk is a package, I would expect that lldb wants the actual executable binary. I'd try installing the apk via adb, then pointing lldb directly to the binary in the installation folder. You might need to tell lldb where to find files locally as described here.

Since we're back here, the only target that the platform connect lists is this:

$ (lldb) > target list
Current targets:
* target #0: /home/mark/.lldb/module_cache/remote-android/.cache/A1AB812B-2621-21CB-66F7-CBE228DC9674/app_process64 ( arch=aarch64-unknown-linux-android, platform=remote-android, pid=15380, state=running )

But that's not the .so, it's much much smaller and running file on it returns this:

app_process64: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /system/bin/linker64, BuildID[md5/uuid]=a1ab812b262121cb66f7cbe228dc9674, stripped

After log enable lldb module I can see that a bunch of modules are loaded, along with the lib.so that I am trying to debug. But only if I use platform connect through codelldb, not if I connect to it through a manual lldb session via platform connect.

And after log enable lldb dyld, I can see that after attaching to the given PID, it loads all current modules of the process:

Executing script: initCommands
  Platform: remote-android
 Connected: no

  Platform: remote-android
    Triple: aarch64-unknown-linux-android
OS Version: 33 (4.14.276-ge333cb8619d0-ab8811257)
  Hostname: localhost
 Connected: yes
WorkingDir: /data/user/0/com.company.exerunner.CoreUtilsTest
    Kernel: #1 SMP PREEMPT Fri Jul 8 12:03:57 UTC 2022

intern-state     DynamicLoaderDarwin::UseDYLDSPI: Use old DynamicLoader plugin
intern-state     DynamicLoaderDarwin::UseDYLDSPI: Use old DynamicLoader plugin
intern-state     DYLDRendezvous::UpdateExecutablePath cannot cache exe module path: null executable module pointer
intern-state     DynamicLoaderPOSIXDYLD::DidAttach() pid 9156
intern-state     DynamicLoaderPOSIXDYLD::DidAttach pid 9156 reloaded auxv data
intern-state     DynamicLoaderPOSIXDYLD::ResolveExecutableModule - got executable by pid 9156: /system/bin/app_process64
intern-state     DYLDRendezvous::UpdateExecutablePath exe module executable path set: '/system/bin/app_process64'
intern-state     DynamicLoaderPOSIXDYLD::DidAttach pid 9156 executable '/home/mark/.lldb/module_cache/remote-android/.cache/A1AB812B-2621-21CB-66F7-CBE228DC9674/app_process64', load_offset 0x6423aa3000
intern-state     DynamicLoaderPOSIXDYLD::DidAttach pid 9156 added executable '/home/mark/.lldb/module_cache/remote-android/.cache/A1AB812B-2621-21CB-66F7-CBE228DC9674/app_process64' to module load list
intern-state     DYLDRendezvous::Resolve address size: 8, padding 4
intern-state     ResolveRendezvousAddress info_location = 0xffffffffffffffff
intern-state     ResolveRendezvousAddress resolved via direct object file approach to 0x6423aa61b8
intern-state     ResolveRendezvousAddress reading pointer (8 bytes) from 0x6423aa61b8
intern-state     DYLDRendezvous::Resolve cursor = 0x742c1d5610
intern-state     LoadAllCurrentModules loading module: linker64
intern-state     LoadAllCurrentModules loading module: [vdso]
intern-state     LoadAllCurrentModules loading module: libandroid_runtime.so
...
intern-state     LoadAllCurrentModules loading module: libnpt.so
intern-state     LoadAllCurrentModules loading module: libdt_fd_forward.so
intern-state     LoadAllCurrentModules loading module: libframework-connectivity-jni.so
intern-state     LoadAllCurrentModules loading module: libCoreUtilsTest.so                    <------------- the one I'm interested in
intern-state     LoadAllCurrentModules loading module: gralloc.sm6150.so
intern-state     DynamicLoaderPOSIXDYLD::DidAttach told the target about the modules that loaded:
intern-state     -- [module] /home/mark/.lldb/module_cache/remote-android/.cache/A1AB812B-2621-21CB-66F7-CBE228DC9674/app_process64 (pid 9156)
intern-state     Setting rendezvous break address for pid 9156 at 0x742c1049b0
intern-state     Successfully set rendezvous breakpoint at address 0x742c1049b0 for pid 9156
Attached to process 9156

So, something in codelldb is loading all the current modules of the process.

If I were to do it (whatever it is) manually, based on the MANUAL.md:

Please note that depending on protocol features implemented by the remote stub, there may be more setup needed.
For example, in the case of "bare-metal" debugging (OpenOCD), the debugger may not be aware of memory locations
of the debuggee modules; you may need to specify this manually:

target modules load --file ${workspaceFolder}/build/debuggee -s <base load address>`

How can I find the <base load address> on an android? Since codelldb can do it, I'm hoping there's a lldb cli available API for that.

[mark2185]

I managed to platform connect to the remote machine (where the android is plugged into) by patching lldb, now it successfully attaches, but it fails to load up the modules:

codelldb         ProcessGDBRemote::DoConnectRemote pid 0: normalizing target architecture initial triple:  (GetTarget().GetArchitecture().IsValid() false, m_gdb_comm.GetHostArchitecture().IsValid(): true)
codelldb         ProcessGDBRemote::DoConnectRemote pid 0: normalized target architecture triple: arm-unknown-linux-android
codelldb         ProcessGDBRemote::DoAttachToProcessWithID()
b-remote.async>  ProcessGDBRemote::AsyncThread(pid = 15337) Got an event of type: 1...
b-remote.async>  ProcessGDBRemote::AsyncThread(pid = 15337) got eBroadcastBitAsyncContinue: vAttach;3be9
b-remote.async>  GDBRemoteClientBase::ContinueLock::lock() resuming with vAttach;3be9
b-remote.async>  GDBRemoteClientBase::SendContinuePacketAndWaitForResponse () got packet: T13thread:3be9;name:r.CoreUtilsTest;threads:3be9, ... a lot of hashes...;reason:signal;
b-remote.async>  ProcessGDBRemote::AsyncThread(pid = 15337) listener.WaitForEvent (NULL, event_sp)...
intern-state     gdb-remote had process architecture, using arm arm-unknown-linux-android
intern-state     analyzing target arch, currently arm arm-unknown-linux-android
intern-state     final target arch after adjustments for remote architecture: arm arm-unknown-linux-android
intern-state     PlatformRemoteGDBServer::GetModuleSpec - got module info for (/system/bin/app_process32:arm-unknown-linux-android) : file = '/system/bin/app_process32', arch = arm-unknown-linux-android, uuid = 92E3A4F0-6425-5DE4-6F36-74695BA5E7ED, object size = 22112
intern-state     PlatformRemoteGDBServer::GetModuleSpec - failed to get module info for com.microblink.exerunner.CoreUtilsTest:arm-unknown-linux-android
Attached to process 15337

Now I need to figure out why, does it open up a new port for that, or does it fail for some other reason? EDIT: Huh, managed to load more modules on an armv8 android, the one where my code resides loads only sometimes, but even those times the breakpoints are still not getting hit. I can see in the logs that the gdb-remote does receive them, but can't seem to break on them.

Also I still need to figure out what exactly in codelldb triggers the reading of the libraries.

[vadimcn]

How can I find the <base load address> on an android

Explicit module loading should be required only on bare-metal platforms, which Android is not.

Now I need to figure out why, does it open up a new port for that, or does it fail for some other reason?

As far as I know, lldb-server is platform mode does indeed open a new connection for each new debug session: https://lldb.llvm.org/man/lldb-server.html#gdb-server-connections

Also I still need to figure out what exactly in codelldb triggers the reading of the libraries.

In theory, lldb should work the same inside CodeLLDB as it does in cli mode. Of course, there is a chance that something works slightly differently when driven programmatically via the API, but so far I haven't noticed any of that.

As for the rest of it, sorry I've no clue.

[mark2185]

In theory, lldb should work the same inside CodeLLDB as it does in cli mode.

You're right, it does work the same. I managed to patch up lldb and connect to a remote android device using platform connect, but now I would like to see if it's possible to do using only gdbserver. I can connect and attach, but no libraries are loaded so I cannot put down breakpoints.

In any case, it's not a vscode-lldb issue, but a lldb mystery, so I think we can close this discussion up if you think it's too off-topic. Also if you have any advice for making it work through gdbserver directly, I'm all ears! :)

[mark2185]

Forgot to mention that the issue was solved, the local ports for platform connect weren't configurable. Here's the entire discussion.

Thanks for all your help! :)