Initial strawperson proposal for debugging modules

[fitzgen]

Disclaimer: This is very much a work-in-progress and nothing I've written up is set stone! My hope is that we can merge this PR and continue design in the form of follow up PRs, issue discussions, and video meetings.

Rendered

[billti]

The term WasmDebugger in the prior bullet refers to an interface. Should this be Debugging module as defined in the 2nd bullet?

[fitzgen]

Yeah, I suppose it is wrapping both the WasmDebugger (since it is translating methods from source-level into Wasm-level method calls on its wrapped WasmDebugger) and internal tables/routines of the debugging module (which are used for that translation).

Do you have any suggestions on wording or clarifications we can make?

[billti]

I'd assume this should be a collection of SourceDebuggers if there is one per Debugee Module. For example, I can imagine common headers with inline functions (e.g. <string>), would result in the same source running in multiple modules.

[fitzgen]

Great point! This should work out since even if we end up setting many breakpoints in the "same" source that end up in different debuggee modules, only one Wasm breakpoint is ever hit at a time.

I'll make an update in a little bit.

[billti]

I believe unsigned long is 32-bits in WebIDL https://www.w3.org/TR/WebIDL-1/#idl-unsigned-long. Is this specific to Wasm32?

[fitzgen]

It's true, I wasn't thinking of a future wasm 64. Even with wasm 64, I'd be a bit surprised if a code section was larger than 2³² - 1 bytes, but I suppose we shouldn't artificially limit ourselves in that situation.

It isn't clear to me whether we want to eagerly make this a 64 bit integer or not before wasm 64 is a thing, however.

[aardappel]

I'm sure you could use an existing serialization format (Protobuf, FlatBuffers, Cap'n Proto..) that is already future extensible?

[aardappel]

That seems entirely a question of perspective. I could say, "why couple this tightly to a specific API that needs to be explicitly forwarded and implemented, when you could simply have a protocol message that is trivially forwardable, inspectable, loggable, forwards and backwards compatible, extensible and can be interfaced with in a wide range of languages with already available code generators?" :)

[aardappel]

It could be more clearly noted earlier in the document that this is not meant to be specific to an embedding that uses JS, as that is the impression I got from this and earlier documents until I saw this.

[billti]

Out of interest... was there any "prior art" that went into this, or was researched for comparison? For example, VS Code based their debug adapters on, and documented, the https://microsoft.github.io/debug-adapter-protocol/overview, which is pretty well battle tested at this point as an interface/protocol for various languages to engines. There may be some hard learned lessons in there we can leverage. (Or just use it as a base, which would make integration with a few IDEs relatively easy).

[fitzgen]

@billti

was there any "prior art" that went into this, or was researched for comparison?

As mentioned in the subgroup meeting today, it is based on my experience with DWARF, source maps, and SpiderMonkey's debugging API.

I agree that it is worth looking into how VS code solves cross-language support problems and taking inspiration from there.

---

@aardappel

I'm sure you could use an existing serialization format (Protobuf, FlatBuffers, Cap'n Proto..) that is already future extensible?

We discussed this a little in the subgroup meeting today.

Even if we use an existing serialization format, we would need some sort of calling convention describing how to get the serialized buffer into the debugging module's memory, as well as taking new serialized messages out of it. This would require understanding malloc vs pre-allocated space, ownership, etc... and it turns out this is exactly the type of thing that WebAssembly Interface Types is already solving for us. So why not fully leverage WebAssembly Interface Types directly?

---

@paolosevMSFT

Do we also need to ... register a callback that the debugger module will use to communicate with the WebAssembly engine it is debugging?

I was imagining that when the embedder uses a debugging module to create a SourceDebugger, that it would be the embedder's responsibility to automatically register the resulting SourceDebugger and call its onBreak/onStep hooks at the appropriate time.

This is definitely something that we should clarify moving forward.

[fitzgen]

Oh also! As far as representing debug info programmatically, Norman Ramsay's ldb is definitely something to read up on. His goal with ldb was easily retargeting the debugger across different targets. While we only have to deal with Wasm as a target, I think there is inspiration to be gained from ldb, especially in terms of simplifying the implementation of the debugger (i.e. moving duplicated work away from devtools and into shared debugging modules).

[aardappel]

@fitzgen

as well as taking new serialized messages out of it. This would require understanding malloc vs pre-allocated space, ownership, etc

Both FlatBuffers and Cap'n Proto allow accessing serialized data in place, so beyond the initial buffer, there is no allocating/owning etc going on.

So why not fully leverage WebAssembly Interface Types directly?

It will be a while before these can express data structures as rich as what these serializers can do, besides all the other advantages I mentioned.

[fitzgen]

Both FlatBuffers and Cap'n Proto allow accessing serialized data in place, so beyond the initial buffer, there is no allocating/owning etc going on.

Once a Wasm module has access to the serialized data in its memory, yes it can deserialize it in place in a zero-copy fashion. But how does it get that initial access to the serialized data? That requires some sort of copy into some region of linear memory, and doing that requires understanding malloc/ownership/etc.

[aardappel]

Once a Wasm module has access to the serialized data in its memory, yes it can deserialize it in place in a zero-copy fashion. But how does it get that initial access to the serialized data? That requires some sort of copy into some region of linear memory, and doing that requires understanding malloc/ownership/etc.

Yes, and why is that a problem? Surely there's plenty of Wasm modules out there that exchange buffers with the outside world just fine? I can't imagine this would be a major deciding factor in deciding between using a protocol/serialized format or a set of API calls.

[paolosevMSFT]

Regarding the need to register a callback function that the debugger module should use to talk with the debuggee... I agree that we should clarify this moving forward, but this is quite important in my opinion because it really impacts the design of the architecture.

The idea is that the debugging module (DM) would expose WebIDL interfaces (like SourceDebugger) to the debugger/devtools, and this is very clear. Then, the wasm engine would expose WebIDL interfaces (like WasmDebugger) to the DM, but I think we need to clarify how the DM will call this interface.

It could be the embedder's responsibility to register somehow the WasmDebugger interface to the DM, but how? The wasm engine runs in a different process from the DM, and possibly even in a different machine. Would that mean that the embedder also needs to implement the same WasmDebugger interface, register it to the DM, and then forward every call remotely to the actual wasm engine, which also implements the same interface?

The mechanism that the embedder/debugger will use to communicate with the wasm engine could vary considerably; if the debugger is in the browser DevTools, it already has its own channel to communicate with the script engine running in the debuggee webpage, but if the debugger is a standalone app like LLDB we could use to debug wasmtime, we would need to introduce a new mechanism to communicate with the wasm engine.

This is why I proposed that we should at least define, as part of the DM interface, a function that the embedder would use to register a callback function that the DM will invoke to send messages to the debuggee engine, so abstracting the concrete communication channel. It would then be up to the embedder to actually send the messages to the debuggee using its own channels.

For these reasons, I am totally fine with the SourceDebugger interfaces exposed by the DM and consumed by the debugger/embedder, but I am not so sure about the WasmDebugger interface that should be exposed by the engine.
I wonder whether we should consider instead using a lower-level remote debugging protocol (like gdb-remote) for the communication between the DM and the engine. I wrote a few notes in this doc:
https://docs.google.com/document/d/1wPUL0rzojvi7Pl0ifPM2CDAkdTsSDWcKIKhZfPtyvn8/edit#... could an architecture with an high-level debugging interface provided by the DM and a low-level remote debugging protocol managed by the wasm engines be a reasonable alternative in your opinion?

[fitzgen]

I pushed two more commits, notably 6d74a42 which expands on the "why not a protocol?" FAQ question with this additional text:

One might might be tempted to use a protocol to avoid an inter-standards dependency on Wasm interface types. A protocol requires passing the serialized data into and out of the debugging module. Passing that data in or out requires knowledge of calling conventions and memory ownership (who mallocs and who frees). This is a problem that Wasm interface types are already standardizing a solution for, and which engines already intend to support. Duplicating standards work done by another subgroup is far from ideal: it leads to more implementation work for both toolchains and engines.

The final thing to consider is the code size impact that using a protocol implies. Incoming messages must be deserialized and outgoing messages must be serialized, and both those things require non-trivial amounts of code. On the other hand, with Wasm interface types most of the functionality is implemented once in the Wasm engine, and doesn't bloat every module's code size.

[fitzgen]

@paolosevMSFT, thanks for your patience while waiting for my response. Replies inline below.

It could be the embedder's responsibility to register somehow the WasmDebugger interface to the DM, but how?

The embedder is creating the WasmDebugger object and it is instantiating the debugging module, and passing it into the debugging module's SourceDebugger constructor. The embedder has everything it needs to maintain a bidirectional map between a WasmDebugger and a SourceDebugger.

The wasm engine runs in a different process from the DM, and possibly even in a different machine.

Yes, these are things that we want to ensure are possible for implementations to do, but also aren't required of all implementations.

Would that mean that the embedder also needs to implement the same WasmDebugger interface, register it to the DM, and then forward every call remotely to the actual wasm engine, which also implements the same interface? The mechanism that the embedder/debugger will use to communicate with the wasm engine could vary considerably; if the debugger is in the browser DevTools, it already has its own channel to communicate with the script engine running in the debuggee webpage,

Yes, if an implementation is running the debugging module inside the browser devtools frontend, it will need to proxy calls over the browser's internal remote debugging protocol, similar to how existing JS debugging calls are proxied from the devtools frontend to the JS engine's debugging APIs.

but if the debugger is a standalone app like LLDB we could use to debug wasmtime, we would need to introduce a new mechanism to communicate with the wasm engine.

This proposal is explicitly not attempting to support the use case of "just connect LLDB to the Wasm engine's generated native code". See the FAQ item about AOT compilation.

It is true that if wasmtime wants to provide its own custom debugging tools, it would want to get source-level debug info from debugging modules, and it would require implementation work to build out debugging and reflection APIs.

This is why I proposed that we should at least define, as part of the DM interface, a function that the embedder would use to register a callback function that the DM will invoke to send messages to the debuggee engine, so abstracting the concrete communication channel. It would then be up to the embedder to actually send the messages to the debuggee using its own channels.

For any event that originates within the debugging module itself, yes, we would need a way for the debugging module to notify the embedder of the event (such as giving it a callback).

But what sorts of events originate in the debugging module, as opposed to originating in the debuggee or being explicitly requested by the embedder?

I wonder whether we should consider instead using a lower-level remote debugging protocol (like gdb-remote) for the communication between the DM and the engine. I wrote a few notes in this doc: https://docs.google.com/document/d/1wPUL0rzojvi7Pl0ifPM2CDAkdTsSDWcKIKhZfPtyvn8/edit#... could an architecture with an high-level debugging interface provided by the DM and a low-level remote debugging protocol managed by the wasm engines be a reasonable alternative in your opinion?

The gdb remote protocol is not a specified standard in any meaningful sense. It is more of a documented implementation. Additionally, in the same way that DWARF won't work off the shelf with Wasm, and requires extensions to model the Wasm execution stack, locals, and globals, the gdb remote protocol would also need to be modified.

I remain unconvinced that standardizing a protocol is the right choice, let alone any existing native debugger's protocol.

[paolosevMSFT]

@fitzgen
Thank you for your detailed reply! A few comments:

The embedder is creating the WasmDebugger object and it is instantiating the debugging module, and passing it into the debugging module's SourceDebugger constructor. The embedder has everything it needs to maintain a bidirectional map between a WasmDebugger and a SourceDebugger.

I am not sure what the WasmDebugger object is in this context. The document describes WasmDebugger as an interface that:

provides raw, Wasm-level debugging APIs for inspecting the debuggee Wasm module. It is implemented by the Wasm engine and given to a debugging module's SourceDebugger interface.
so I thought it was always part of the engine, and not created by the embedder.

Also:

The wasm engine runs in a different process from the DM, and possibly even in a different machine.

Yes, these are things that we want to ensure are possible for implementations to do, but also aren't required of all implementations.

Maybe I am missing something here, but even if we ignore the case of remote debugging, I don't think there will ever be any implementation where the SourceDebugger and the WasmDebugger are in the same process. We say that the debugging module is used by the engine's developer tools, so it runs in the debugger. But then WasmDebugger will be in a different process and the embedder will not be able to directly register a WasmDebugger with the DM, and there will always be the need to proxy the calls through the browser's internal remote debugging protocol.

But if this is the case, shouldn't the design anticipate the need that the communication between the DM and the debuggee always needs to be proxied? The communication channel between DM and debuggee should be a central aspect of this spec, since we need to cover very different debugging scenarios.

At the very least, we should mention that the WasmDebugger passed to the DM will always be implemented by a proxy that forwards the requests to the Debuggee, which concretely implements the same WasmDebugger interface.
But then maybe we should also standardize the mechanism we will use to do this proxying. It does not have to be a protocol like gdb-remote, but we need to have a clear communication mechanism in place.

But what sorts of events originate in the debugging module, as opposed to originating in the debuggee or being explicitly requested by the embedder?

There are several requests made by the embedder that the debugging module can satisfy only querying the state of the wasm engine, so originating query events.
In particular, if the embedder requests the value of a source-level variable in a scope, the DM might need to inspect the Wasm engine state calling functions like these (in pseudocode):

byte[] getWasmGlobal(moduleId, threadId, index)
byte[] getWasmLocal(moduleId, threadId, frameId, index)
byte[] getWasmMemory(instanceId, address, length)
addr_t[] getWasmCallStack()

Also, to support source-level stepping (which is fairly complicated) the DM could need to have access to the whole module bytecode, to be able to disassemble instructions in the current function. It could then make requests to add (or remove) breakpoints at specific locations in the Wasm module:

byte[] getWasmBytecode(moduleId, offset, length)
BreakpointId addBreakpoint(moduleId, offset)
bool removeBreakpoint(BreakpointId)

All these requests originate in the debugging module. This is why my proposal was be that while the SourceDebugger API should be defined with a WebIDL interface, to be consumed by the debugger, the WasmDebugger API that always needs to be "remoted" could instead be a protocol.

The gdb remote protocol is not a specified standard in any meaningful sense. It is more of a documented implementation. Additionally, in the same way that DWARF won't work off the shelf with Wasm, and requires extensions to model the Wasm execution stack, locals, and globals, the gdb remote protocol would also need to be modified.

It is true that DWARF won't work off the shelf with Wasm, but the changes required are very small, and even if the proposal wisely hides the debug information details behind a clean API, I don't think that realistically we will have other options if we want to debug Clang-generated code.
You are correct that also gdb-remote would have to be extended to support Wasm; it does provide mechanisms to add custom query packets, that would be ignored by existing implementations. As you say gdb-remote is not a standard protocol but it would have the advantage that it could also open the way to support debuggers like LLDB (and not necessarily only for AOT), even though I understand this is out of the scope of this proposal.

To summarize, my main concern is that we should define more clearly the communication mechanism between the debugging module and the debuggee engine.

[paolosevMSFT]

It is not possible to implement "step-over" just by setting breakpoints, because the debugger does not know what is the next source line that will be executed. It might not be the one directly succeeding the current line, as we could be in a loop, or some conditional construct.
Debuggers can examine what instructions are being executed and work out all of the possible branch targets, setting breakpoints on all of them. To do this, the debugger module should be able to disassemble the wasm module bytecode (which means that they also need access to that bytecode).
A simpler (but very inefficient) alternative can be to send a number of instruction-level 'step' requests to the engine, until a different line is reached.

[paolosevMSFT]

I think that implementations will always need to proxy the interface method calls across a protocol, because the debugger and debuggee will always be in different processes.
This does not mean that we necessarily need to add a specific wire protocol into the standard, but the standard should mention how the Debugger Module will communicate with the debuggee.

[paolosevMSFT]

Could the requirement that a debugging module needs to be a Wasm module be problematic? When we'll start implementing debugging modules for DWARF, for example, we could leverage existing code from LLDB/LLVM that will give us the ability of parsing DWARF files, decode DWARF information, un-mangle names according to the source language specified, supporting multiple languages, evaluate expressions to determine the value of source-level variables, and so on.
Modifying all this existing code so that it compiles to WASM could be a daunting task. It would be much simpler to have 'native' debugging module and define a (websocket-based?) debugging protocol to communicate with them.

[duongnhn]

+1 to this. Probably we should just leave it to implementation detail.