What Could Testing Look Like For Web?

[Stringer90]

This discussion was created by myself and @vt37

I'm aware that multithreading can't be used due to WebAssembly, so perhaps the web app to be tested can be run on an HTTP server (as it already is), with another program/process running the tests. This testing process could run tests similar to those in the toga/testbed/tests directory, and the (to-be-implemented) probes in the tests_backend of the web directory could use Playwright to "look at" and interact with the DOM of the web app.

Communication Between Web App and Test Process

As for a communication channel between the web app and testing program, we could use Pyodide to run Python code remotely. Once a Playwright Page object navigates to the URL (e.g., page.goto("http://localhost:8000")), it could then execute:

page.evaluate("() => pyodide.runPythonAsync('button.text = \"Click Me!\"')")

This method could also be used to return data from the widget objects directly, in addition to inspecting the rendered DOM widget.

Example Use Case

Testing a button’s text currently requires:

widget.text = text
assert widget.text == expected
assert probe.text == expected

• widget.text = text could be run via Pyodide.
• The widget fixture/object could retrieve the text using a similar Pyodide call.
• The probe fixture/object would use Playwright to get the DOM representation of the button’s text.

Potential Considerations

• Ensure the page is fully loaded and Pyodide is initialized before interaction.
• Scope could be an issue, as widget objects may need to be global to be accessed by Pyodide.
• This may require rebuilding much of the testbed directory or creating a new one entirely, where the Pytest fixtures may not even be needed.

---

What do you think about this idea? Is it feasible? I'm aware it's fairly high-level, so I’m sure there are aspects I haven’t considered yet. Any thoughts or feedback are much appreciated!

[freakboy3742]

Thanks for kicking off this discussion. There's clearly some technical issues to resolve here.

I'm aware that multithreading can't be used due to WebAssembly, so perhaps the web app to be tested can be run on an HTTP server (as it already is), with another program/process running the tests.

One detail that is worth being explicit about here (for the benefit of anyone else reading): when you say "another program/process", we're referring to the Pytest suite. We're essentially looking at a situation where the pytest suite runs outside the context of the "app". In every other platform to date (macOS, Android etc), the test suite is a second thread inside the testbed app.

Communication Between Web App and Test Process

As for a communication channel between the web app and testing program, we could use Pyodide to run Python code remotely. Once a Playwright Page object navigates to the URL (e.g., page.goto("http://localhost:8000")), it could then execute:

page.evaluate("() => pyodide.runPythonAsync('button.text = \"Click Me!\"')")

Pyodide APIs will definitely be available in this way - but then, so should any Javascript API, including PyScript APIs. If Pyodide is the easiest/best way to perform a given action, that's fine - but it's also worth keeping in mind that both higher and lower level APIs may also be an option.

This method could also be used to return data from the widget objects directly, in addition to inspecting the rendered DOM widget.

The rendered DOM widget should be accessible through the Playwright API - I'm not sure I see why there would be a benefit (or need) to use "Playwright running pyodide running Python" to obtain DOM state. Am I missing something about what you're describing here?

widget.text = text
assert widget.text == expected
assert probe.text == expected

• widget.text = text could be run via Pyodide.
• The widget fixture/object could retrieve the text using a similar Pyodide call.
• The probe fixture/object would use Playwright to get the DOM representation of the button’s text.

That looks like a good summary of the types of operations that need to be performed.

To put this another way:

• The probe will be operating at the level of the DOM - it will be interrogating DOM state, looking for attributes, style arguments, tag names and hierarchy, and so on.
• The widget is operating in the browser. There is a need to retrieve and set attributes, to invoke methods, and retrieve the results of those methods - but perform this on the object in the Python context in the browser.

The latter will be the harder of the two - essentially, you need to be able to get a handle to a proxy for the object inside the browser. When you invoke widget.text, you won't be touching the actual widget- you'll be calling a __getattr__ method on a proxy object that knows how to use Playwright to poke Pyscript/pyodide to retrieve the same attribute on the object in the browser.

Potential Considerations

• Ensure the page is fully loaded and Pyodide is initialized before interaction.

True - but that should be something Playwright is able to manage - or if it isn't, we can put a "ready" marker into the startup sequence of the testbed app itself.

• Scope could be an issue, as widget objects may need to be global to be accessed by Pyodide.

This is a non-starter. We can't make widget objects global.

However, a Toga app has an API for doing a DOM-like "lookup by ID". toga.App.widget["some-id"] will return a specific widget instance. That lookup process could be part of the proxy mechanism used to access/invoke methods.

• This may require rebuilding much of the testbed directory or creating a new one entirely, where the Pytest fixtures may not even be needed.

I'm not sure you've explained why this is the case. At a high level, the current testbed test suite is expressed in terms of high level functional behaviors that a widget should have, with the probe being used as a mechanism for evaluating if those functional behaviors have actually occurred (e.g., you can set the text color of a label - and sure, toga's API will return the new style, but does the actual underlying widget toolkit agree that the color has changed?)

There's definitely a new way to start the testbed test suite, and there definitely needs to be a web probe backend; but why does the entire testbed directory need to be rebuilt?

[Stringer90]

Thanks for the feedback and extra information!

Sorry that I wasn't clear about some of the things I talked about, and I now know the entire testbed directory doesn't need to be rebuilt, partly due to you letting me know about the proxy method, which I have looked into.

Regarding the APIs for communication between the processes, I believe Pyodide is best, at least for now, but I'll have a look at others another time.

It should also be noted that Pyodide will have to be exposed to the JavaScript environment. This should be able to be done by running something similar to:

js.window.app = self.interface

in the initialisation method of the App class in app.py of toga_web. We would only do this in test mode, so we will need to check for this before Pyodide is exposed like this. This should then allow us access to main_window and its content.

This would allow us to modify widget objects directly, and then Playwright can separately inspect the DOM state of the app and its widgets in the probes.

Sorry that my previous explanation came across as saying "Playwright running Pyodide running Python to obtain DOM state". Pyodide will not be used for the DOM, but for interacting with objects in the Python context in the browser, such as the widget objects.

The other issue is that the test suite can't access widget objects or the app singleton directly as they are not part of the same process.

In the conftest.py file of the test suite:

@fixture(scope="session")
def app():
    return toga.App.app

This will be used to get the main_window fixture that is then used to create probe fixtures.

This also assumes the App singleton is already set, which usually occurs in the app.py of toga/core:

# Keep an accessible copy of the app singleton instance
App.app = self

This won't be possible since the test suite is running outside the context of the app.

A solution is, like you mentioned, to use proxies. I had a look into proxies, and we could set toga.App.app to an app proxy object. We could then do the same for main_window and then widgets.

Setting app to the proxy object will have to be run before the app fixture is defined, so perhaps it could be set in __init__.py of the tests_backend directory, and then add:

import_module("tests_backend")

to the top of the conftest.py file in the test suite to ensure it runs before the app fixture is defined?

As for the widgets, proxies for them should be able to be set up with __getattr__ and __setattr__ methods like you mentioned. These methods would use Pyodide to get/set attributes of the actual widget objects in the app context.

We will have to somehow override the widget fixture so that the proxy object is returned instead. I'm also not sure how the current probe fixture could be run successfully if this is the case though. Perhaps it can be re-defined to take into account the widget variable/fixture not being a Toga widget object, but a proxy object?

As can be seen I'll have to investigate a few things and hopefully prototype parts sometime in the future.

Please let me know if there’s anything else I should be aware of, or if I have misunderstood anything.

[freakboy3742]

It should also be noted that Pyodide will have to be exposed to the JavaScript environment. This should be able to be done by running something similar to:

js.window.app = self.interface

in the initialisation method of the App class in app.py of toga_web. We would only do this in test mode, so we will need to check for this before Pyodide is exposed like this. This should then allow us access to main_window and its content.

A macOS/Cocoa widget will have a reference to the _impl and interface layers, for example. So, exact naming notwithstanding, as long as the Javascript widget allows this, we have precedent for doing this sort of thing.

The other issue is that the test suite can't access widget objects or the app singleton directly as they are not part of the same process.

Correct - but that then becomes the challenge, because you must have an ability to do this. A test must be able to call methods on a widget, and access properties on a widget - otherwise, you're not testing the API.

A solution is, like you mentioned, to use proxies. I had a look into proxies, and we could set toga.App.app to an app proxy object. We could then do the same for main_window and then widgets.

I'm not entirely sure I follow what you're describing here. You'll need to have a proxy for at least app - but that proxy should be returned by the fixture - as would any widget that is created. I'm not sure I see how "setting toga.App.app would help. The test shouldn't ever have an actual reference to toga.App - it should be poking the browser to create/retrieve an App instance, and then constructing a Proxy instance wrapping that object.

We will have to somehow override the widget fixture so that the proxy object is returned instead. I'm also not sure how the current probe fixture could be run successfully if this is the case though. Perhaps it can be re-defined to take into account the widget variable/fixture not being a Toga widget object, but a proxy object?

My guess would be that one of two things needs to happen:

1. the actual object creation would need to be pushed down the probe layer somehow. Instead of a direct call to toga.Button(), the probe is invoked with a app_probe.create_widget("Button", *args, **kwargs) method. On most backends, that would be a direct call to getattr(toga, "Button")(*args, **kwargs); but the web probe would go through the proxy-generation process.
2. We add a new backend, similar to toga-dummy - say, toga-proxy - which implements all the Toga widget interfaces, but implements them as a remote control interface. Those "widgets" wouldn't actually require full implementations - they'd all be proxy objects; it would be mostly a mechanism for having a namespace of widgets so that toga.Button actually resolves to something.

[Stringer90]

I'm going to have a go at explaining things better, including a new plan. Note that these are only ideas currently. I will be using the Button widget and it's test file test_button.py, as well as test_base.py for examples.

As mentioned, the main problem is that for web, the testbed app and the test suite can't run multi-threaded in the same process and must run separately. In the test suite, any toga element created is in the same memory space as the testbed app, and so can be easily referenced, modified and added to main_window when probes are created. So the core problem is that the test suite assumes that it is running in the same process as the testbed app, which it can't for web.

Currently, in Pytest fixtures such as

@fixture 
async def widget(): 
	return toga.Button("Hello")

in test_button.py, or

@pytest.fixture
async def widget():
    return toga.Box(style=Pack(width=100, height=200, background_color=RED))

in test_base.py.

The Button and Box widgets are created in the same memory space as the testbed app. So they can be easily modified and added to the main_window, performing actions such as:

widget.text = text

child = toga.Box(style=Pack(width=75, height=100, background_color=GREEN)) 
grandchild = toga.Button("Hello") 
child.add(grandchild) 
widget.add(child)

old_content = main_window.content 
box = toga.Box(children=[widget]) 
main_window.content = box 
main_window.content = old_content

The only idea that I have is to do some aliasing, aliasing these toga classes with proxy/remote classes behind the scene. These alias proxy classes will act like a wrapper that will 'forward' any commands or changes to the real ones in the testbed app, so that they can be executed there, as if they are in the same process as the test suite expects.

To get this set up, we can run the following, which will only be executed if the test suite is testing the web platform. Something like:

@pytest.fixture(autouse=True, scope="session") 
def patch_toga_classes_for_web(): 
	if toga.platform.current_platform == "web": 
		toga.Button = RemoteButton 
		toga.Box = RemoteBox # or RemoteBoxProxy or BoxProxy 

These will act like aliases (but only for web testing) so that when toga.Button("Hello") or toga.Box(style=Pack(...)) is run, it creates/returns RemoteButton or RemoteBox instead.

In the RemoteButton class, in the __init__ method, a command will be sent to the testbed app that actually creates the Toga Button widget through a remote code execution method (like through the exposed Pyodide discussed earlier). So the result will be the app executing something like new_button = toga.Button("Hello"). This should be added to the widget 'registry' of the app (toga.App.widget["some-id"]). We can then have its id returned to RemoteButton instance to be stored, so that the Button instance in the testbed app can be accessed easily through the widget 'registry' in the future. We have to make sure we can access this as well.

After this, methods can be created in RemoteButton that get and set attributes (__getattr__ and __setattr__), forwarding commands to the testbed app that are used to modify its state, and even to return data. The details of how this can be implemented are yet to be ironed-out, as it would involve building command strings that would be sent to/executed in the testbed app's Pyodide.

For example, when the test suite runs widget.text = text, what will effectively happen is the testbed app will execute something like:

toga.App.widget["button-id"].text = "new text"

I believe this is would effectively solve the problem that you explained earlier:

The latter will be the harder of the two - essentially, you need to be able to get a handle to a proxy for the object inside the browser. When you invoke widget.text, you won't be touching the actual widget - you'll be calling a __getattr__ method on a proxy object that knows how to use Playwright to poke Pyscript/Pyodide to retrieve the same attribute on the object in the browser.

Where RemoteButton is acting as the proxy, forwarding commands to the testbed app that will act on the widget instance in the browser.

Obviously, the probes will be using Playwright to interrogate the DOM state and the rendered widgets in the DOM. The above proxy stuff just ensures that the test suite can get/set attributes and invoke methods as if it is running in the same process as the testbed app, with the proxies just forwarding commands to be executed in the testbed app context.

Some other areas that need looking at in more detail (other than command string building) is to do with app, main_window, their fixtures/proxies/other interactions and other non-widget Toga elements. Part of this is ensuring that the exposed Pyodide can access/expose these levels.
    We would need the app fixture to return a RemoteAppProxy object instance or similar, like the ones above somehow, so that test_app can be run and also help create the main_window fixture.
    Another thing is how Probe instances can get a hold of the id of the element it will be interrogating, but this should be possible.
    Another is whether we have to name the widgets/elements created in the testbed app, or if we have to remotely delete them, such as when all tests in a test file have run, when a test method has finished execution or when a widget/probe fixture is no longer being used.
    Another is handling commands like box = toga.Box(children=[widget]), which should be possible by getting the id of the widget and creating a command string from that.
    One more thing to be figured out is how these widget proxies can be built so that they can work with probes, such as SimpleProbe in cocoa/tests_backend/widgets/base.py, specifically handling what goes on in the __init__ method.

Briefcase would also need to know how to run the testbed app and the test suite as separate processes when a web app is run with the test flag. I'm not sure how this could be done however, as I'm not familiar with Briefcase. Also, at some level, either in core, toga_web/app.py or other, will need to effectively expose Pyodide for the RCE discussed previously.

Once some things have been figured out, some proof-of-concepts or prototypes could be developed.

I have thought a lot about this and this plan should be feasible, the proxies/communication bridge between the test suite and testbed app that forwards commands/data is the stuff that needs to be figured out fully. There's also the other things discussed above (proxies, probes, command string building, etc). The proxy class/alias stuff should definitely work though (I hope) and basic tests (i.e. testing text changes) should be able to be added relatively easily once the architecture is figured out and is actually implemented.

If this ends up not being feasible, I'm not really sure how it can be done at the moment (maybe somehow with web sockets or an API, but the RCE should be adequate, or modifying the test suite differently).

[freakboy3742]

These will act like aliases (but only for web testing) so that when toga.Button("Hello") or toga.Box(style=Pack(...)) is run, it creates/returns RemoteButton or RemoteBox instead.

This sounds like you're on the right track with this design. What you've describe here is broadly what I had in mind; the only question/complication is exactly how the "remote" objects are created in the first place.

The approach you've described of a test session-level replacement of objects in the Toga namespace could definitely work, and might even be the easiest way to get a "first pass" API - but long term, that sort of monkeypatching is at least a little risky.

My preference would be for that interface to be an intentional API - either something where we ask the probe to create a widget (so we don't ever directly invoke toga.Button in the testbed suite), or we have a different package that has the same public API surface as toga-core, but returns Proxy/Remote objects. The latter would essentially be new top-level package (pip install toga-remote, which exposes the toga namespace so that invoking toga.Button returns a RemoteButton.

After this, methods can be created in RemoteButton that get and set attributes (__getattr__ and __setattr__), forwarding commands to the testbed app that are used to modify its state, and even to return data. The details of how this can be implemented are yet to be ironed-out, as it would involve building command strings that would be sent to/executed in the testbed app's Pyodide.

Step 2: Draw the rest of the owl 😝

But yes - I agree this general approach should be possible.

Briefcase would also need to know how to run the testbed app and the test suite as separate processes when a web app is run with the test flag.

Yes - there's a whole separate piece required to allow Briefcase to invoke this new, "remote-style testbed" app.

The current Briefcase testing interface involves starting an app in "test mode" - it's starting the bundled app, but passing in arguments at runtime so that the startup code for the app is slightly different (and runs a test suite).

My expectation is that we will ultimately require the ability to configure Briefcase so that it will run a "normal" pytest/unittest test suite on the local machine, but also start the app - but do so in such a way that the app under test knows it needs to "call home" to establish a testing session with the test suite, or enable some proxy/remote interface during startup so that the pytest suite can pass in the RemoteButton et al API calls described above.

I'm not sure how this could be done however, as I'm not familiar with Briefcase. Also, at some level, either in core, toga_web/app.py or other, will need to effectively expose Pyodide for the RCE discussed previously.

Once some things have been figured out, some proof-of-concepts or prototypes could be developed.

I completely agree that this is a Skunkworks project. Unlike the other Capstone projects, the goal for this piece isn't to build and deliver a mergeable PR, but to prove out a general approach and establish some specifications and guidelines for future pieces of work building on the research you do here.

I'm not expecting that the outcome of this project will include running the test suite through Briefcase. However, I would expect that we would be in a position to specify the sequence of calls that a prospective "remote testing" Briefcase PR would need to implement. Briefcase is, ultimately, a set of wrappers around commands that you could run yourself; so I'd expect the deliverable from this project to define what the manual process for running a "remote" test suite is, so that behavior could be encoded in Briefcase.

[Stringer90]

Regarding setting up/patching aliases for toga classes (essentially being its own remote backend/package as you said), the test suite will need to be run with the web platform/backend, so that the test suite can detect when to set up these aliases. This would be a shorter-term solution, but as you mentioned, in the future we could create a new remote backend/package specifically for remote web testing that has the same public API as toga-core, but returns the proxy objects that forward commands to the remote web testbed app. This would also include somehow running the test suite with this specified (future) remote backend/package after starting the testbed app with the normal web backend. Installing another toga-remote package just for web testing would complicate setting-up/running the test suite as only the test suite and not the testbed app would have to be exposed to it, as that would still need the normal toga package, so there may need to be different virtual environments or something? As a much simpler alternative, I think the patching of toga objects could work long-term though. The test suite would just have to detect if the backend is web, and if so, import a module (new file) that automatically patches the toga objects to be the remote proxy versions. This would require only a minimal modification to the existing test suite.

I had a look at how the testbed app and test suite is currently run, as in how it sets up code coverage tracking and reporting, runs the test suite in a thread, etc, and how it requires a return code from the test suite. This will probably be a separate issue, but for web testing, different methods/modified methods will probably have to be run. This is because the test suite is run as a second thread, and coverage reporting/test suite running/app return code is handled in the same method with this assumption.

For example with web, setting up all the coverage tracking and reporting (among other things) when the app runs will have to be handled separately and run in its own process/sub-process from a new/modified method that starts it as a web app. The test suite will then have to be run in another process/sub-process and provide a return code, all from the new/modified method (telling it to use the remote-backend instead of web, in the future). After the test suite provides the return code, the process that is running the web testbed app can be terminated and results can be collected and handled.

As mentioned, this is probably a separate issue which would require a new, but similar file to be run specifically when testing web. Maybe having a remoteweb-testbed.py file be run instead of testbed.py? With this proposed file, the briefcase command that starts web testing should only have to start this file. But for prototyping web testing, setting up some local environment should be good enough in the short-term with the shorter-term object patching.

[freakboy3742]

Regarding setting up/patching aliases for toga classes (essentially being its own remote backend/package as you said), the test suite will need to be run with the web platform/backend, so that the test suite can detect when to set up these aliases.

To be clear: the app in browser needs to use the toga-web backend. The environment where the test suite runs - the laptop running Briefcase - doesn't need to have toga-web installed. In fact, if it tries to invoke toga-web APIs that should, by definition, be an error.

This would be a shorter-term solution, but as you mentioned, in the future we could create a new remote backend/package specifically for remote web testing that has the same public API as toga-core, but returns the proxy objects that forward commands to the remote web testbed app.

Sure - as "get it working" solution, monkey-patching the toga module is fine. However, if we get to an initial proof of concept with monkey patching, investigating a more permanent solution will be high on the list of priorities.

I had a look at how the testbed app and test suite is currently run, as in how it sets up code coverage tracking and reporting, runs the test suite in a thread, etc, and how it requires a return code from the test suite. This will probably be a separate issue, but for web testing, different methods/modified methods will probably have to be run. This is because the test suite is run as a second thread, and coverage reporting/test suite running/app return code is handled in the same method with this assumption.

It's true that the testbed's entry point is very custom and based on threading; however, the thing that the "test thread" is actually running is a programmatic version of pytest tests. That's something you should be able to invoke directly from the command line; the trick will be getting all the other pre-conditions in place (like having and connecting to the running web version of the testbed app that is being tested)

For example with web, setting up all the coverage tracking and reporting (among other things) when the app runs will have to be handled separately and run in its own process/sub-process from a new/modified method that starts it as a web app.

Agreed that coverage tracking will be difficult. Coverage data would need to be captured in the browser, and then exfiltrated to the main machine. I'm happy to defer this to a "v2" of the proof of concept, if necessary.

As mentioned, this is probably a separate issue which would require a new, but similar file to be run specifically when testing web. Maybe having a remoteweb-testbed.py file be run instead of testbed.py? With this proposed file, the briefcase command that starts web testing should only have to start this file. But for prototyping web testing, setting up some local environment should be good enough in the short-term with the shorter-term object patching.

I'm not convinced a new entry point is needed - as noted previously, it should just be possible to run the test suite by invoking pytest test in a known environment (where "known" includes the app running and being contactable by the web site's remote proxy handling). However, I'm willing to be proven wrong on that.