Cannot install some Wally packages on Linux

[SolarHorizon]

When using pesde with Wally packages on Linux, dependency resolution will sometimes hang infinitely. I don't have this issue on Windows and I don't have a MacOS machine to test on.

I am able to consistently reproduce this issue when trying to install wally#jsdotlua/react-roblox on Linux.

For a minimal repro:

name = "test/dependency_resolution"
version = "0.1.0"

[target]
environment = "roblox"

[indices]
default = "https://github.com/pesde-pkg/index"

[wally_indices]
default = "https://github.com/UpliftGames/wally-index"

[scripts]
roblox_sync_config_generator = ".pesde/scripts/roblox_sync_config_generator.luau"
sourcemap_generator = ".pesde/scripts/sourcemap_generator.luau"

[dev_dependencies]
scripts = { name = "pesde/scripts_rojo", version = "^0.1.0", target = "lune" }
rojo = { name = "pesde/rojo", version = "^7.4.4", target = "lune" }

[dependencies]
react-roblox = { wally = "wally#jsdotlua/react-roblox", version = "^17.2.1" }

[daimond113]

Thank you for opening an issue. Could you tell me some information about your system, such as the distribution and its version?

[SolarHorizon]

Sure. I've tried this on both of my VPS instances which run Ubuntu. I've also tried via Ubuntu on WSL and Arch through WSL. I've seen the same behavior across all of them.

[SolarHorizon]

Sorry, forgot the versions. I tested on the following:

• Ubuntu 22.04.4 LTS
• Ubuntu 24.04.1 LTS (this is via WSL)

Arch is rolling release so I can't provide a version for you.

[daimond113]

From my testing, this seems to be either an issue with Lune or with the individual tooling packages. Does the problem persist if you install Rojo from another source (manually, Rokit, Aftman, etc.) and put that bin directory before the pesde bin directory in your PATH?

[SolarHorizon]

The problem does not persist when using a non-pesde managed Rojo installation.

[daimond113]

In that case, I'll transfer this issue to the tooling repository.

[CompeyDev]

It is hard for me to reliably reproduce this, but I think this might have to do with how we decompress ZIP archives.

I am currently already working on a pure Luau implementation for ZIP format extraction, so I will see if that fixes this.

[CompeyDev]

Whoops, looks like the issue got linked to the PR, and got closed with it. Reopening.

[CompeyDev]

In my testing, I don't see this happen anymore, I think it might have been fixed. Can you try on your end again?

[SolarHorizon]

I'm still not able to install the package (jsdotlua/react, specifically). I don't think the issue was related to compression in the first place because Rojo & other pesde managed tools were working fine when running them on their own, they just don't seem to work properly when pesde tries to run them through a script, like the sourcemap generator script.

When running pesde install using Rojo managed by pesde, even after verifying that Rojo works properly on its own (i.e. when running it myself), my CPU jumps to 100% usage and there are tons of processes left running. All of these processes are from pesde trying to run the sourcemap generator script, but it gets stuck at some point and never finishes, causing the install command to hang indefinitely.

[CompeyDev]

I don't think the issue was related to compression in the first place because Rojo & other pesde managed tools were working fine when running them on their own, they just don't seem to work properly when pesde tries to run them through a script, like the sourcemap generator script.

I believed so because this seems to occur when we have multiple simultaneous versions of pesde managed Rojo that are spawned, causing multiple unzip commands trying to install / write to the same path where some might succeed before others, leading to a race condition.

I'll try to debug further, this is a bit odd since I wasn't able to reproduce this after swapping over to the library.

EDIT: I'm now able to reproduce this, seems like the generator outputs to stdout correctly (because lune is forwarding the rojo output) but the function never returns and thus does not move on to exiting. Not really sure why this occurs.

EDIT: This has something to do with concurrency. Modifying pesde to only spawn scripts sequentially works just fine. The concurrency issue does not have anything to do with the scripts library, because using a non pesde managed rojo installation works fine.

[CompeyDev]

Someone else also ran into the same issue and I was able to further debug it -- it looks like a race condition to do with concurrent resource accesses when multiple processes of uncached tooling packages try to install at the same time.

To attempt to fix this, I have added a locking mechanism to the installation process, starting at commit 2950231, spanning over the next few commits further refining its implementation.

I will also attempt to try to reproduce this bug with these changes.

[CompeyDev]

These changes have been published as a part of v0.1.12, and with testing on my end, seems to have been fixed on Linux. Please let me know if this has been fixed for you too on Linux and Windows.

Run pesde update to get the latest version of toolchainlib in your project.

[SolarHorizon]

I'm still having the same issue, using the same pesde.toml from the original post. The pesde version probably shouldn't matter, but I tried with both pesde 0.5.3 and the most recent pesde 0.6 release candidate.

When using 0.6, I got this:

And then the process hangs indefinitely (as you can see in the image, this has been running for over 7 minutes without any progress)

When running pesde install using Rojo managed by pesde, even after verifying that Rojo works properly on its own (i.e. when running it myself), my CPU jumps to 100% usage and there are tons of processes left running. All of these processes are from pesde trying to run the sourcemap generator script, but it gets stuck at some point and never finishes, causing the install command to hang indefinitely.

This behavior is still occurring as well.