-
-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Random tests #256
Labels
enhancement
New feature or request
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem
We currently have very few tests, and these tests are very low-level. They are definitely not enough to catch corner cases that happen infrequently. Bugs are being reported but happen under heavy load and are difficult to reproduce. Testing RTC e.g. in JupyterLab meetings on a Binder instance is not ideal, or at least not systematic.
Proposed Solution
One way to stress-test RTC at a high level in an automatic way could be through random tests. They are a good solution when it comes to exploring complex states that would otherwise be very difficult to imagine or reproduce. One might say that they are not very reproducible by design, but another way to look at them is that they explore more space at each run, which I see as a way to increase our confidence in the robustness of RTC over time. When failing, a test could write the YStore as an artifact in the CI, which could be used to reproduce the bug locally.
Tests could use Python web clients. This has the drawback of not testing JupyterLab's frontend (JavaScript) code, but most of the complexity lies in the backend anyway (in out-of-band change detection for instance). Writing tests in pure Python would be easier and would probably allow to test more scenarios. What I have in mind is the following:
Thanks to the guarantees provided by CRDTs, the fact that changes to the reference are not applied in the same order as for the shared models in the server and in the clients would not matter. The checkpoints would allow some time for conflicts to resolve before comparing states.
Additional context
See jupyterlab/jupyterlab#14532.
The text was updated successfully, but these errors were encountered: