Add an ability to preload a sound file without decoding

[dos1]

PR #2006 made the preloaded sounds be decoded right away, bringing back old memory consumption issues from before #431. For games that want to download all their assets during the loading screen, but don't want to actually decode all sounds right away because of memory consumption concerns, being able to dynamically load/unload Howler objects via events is not enough, as the application may already go out-of-memory during loading, and not doing that will cause the game to download additional assets during gameplay. This change adds a new property, "preloadInCache", which makes GDJS request the sound asset via XHR. This puts the file into browser cache, so it can be reasonably expected to be decoded later without issuing network requests.

While at it, add user visible descriptions for all preload options.

[dos1]

In the future, it would be nice to extend this to actually act as a proper cache for pre-downloaded sounds that then get fed into Howler, so we wouldn't have to rely on browser cache at all.

Alternatively - instead of introducing "preload as file", we could make sure that "preload as music" actually ensures that the whole file is downloaded instead of calling onLoad as soon as the stream becomes playable. Thinking about it now, this may actually be a preferable option to me, but I haven't looked into how to implement it yet, and it would change the current semantics of this option, so I'd like to hear some opinions first before implementing it.

[arthuro555]

PR #2006 effectively reverted the behavior of preloaded sounds to state from before #431

That is a big misunderstanding of what #2006 does 💀 It is not comparable to before #431

The state before #431:

• In preloading, a howl is created, downloading and decoding the audio for nothing, leaving it immediately discarded, in hope that browser cache would
• Playing a sound recreated a new howl, redecoding it each time (and redownloading it if the browser cache cannot be relied on this context)
• It was 4 years ago, many older devices were supported

What #431 changed:

• Sounds were properly destroyed when their usage was finished
• The preloading that was done with howler was no different from prefetching a data with an XmlHttpRequest, so it did so instead.

What #2006 did was not mainly preloading: it was reuse of Howls. Once a howl is loaded, instead of reloading the whole file and redecoding it each time you want to play a sound, the same howl instance is reused. This spares memory, CPU usage, and in cases where there is no caching in place, bandwidth, when playing the same sound multiple times. This also made the playing of sounds whose Howl already loaded instant: the sound is already downloaded and decoded ready to start playing the moment you tell it to. This actually caches more than the XmlHttpRequest, instead of being equivalent to it like before. The XmlHttpRequest took much time to load, blocking the loading screen, and had in the end little effect in accelerating the time between playing a sound/music in actions and it actually starting to play. In fact, it was so useless that I made a browser extension to money patch GDJS and remove this preloading step, which didn't impact the games I played in any other way than making the initial loading screen much faster 💀

Preloading doesn't use a howl to essentially fetch the data and discard the howl: the howl is downloaded and decoded once, then never again. This behavior is opt-in too: by default, all files aren't preloaded, and it waits for you to play them to load them.

Sure, this is a matter of tradeoffs: if you are targetting older low end devices, you may not want to keep all the sounds loaded in memory, and that is also why you can unload the cache from events and opt out of preloading sounds.

But outright removing the option of preloading sounds/musics is not a good idea, most devices and users will not care for a few megabytes of RAM being used permanently as a cache for getting less memory spikes, potentially less bandwidth costs, and most importantly a sound that plays immediately, and if you don't agree well it is designed in a way you aren't forced to use it already.

[dos1]

It is not comparable to before #431

I used wrong words, sorry - it absolutely isn't the same and I didn't mean to imply that; it's just a single aspect that I happen to care about that got effectively reverted to the state from before #431 (meaning that a game that preloads lots of long sounds uses tons of memory). I'm talking about hundreds of megabytes that can make some mobile phones outright crash the browser tab. Compressed assets are small enough that they are fine to preload, but decompressing them as Howler.js does causes a huge spike of memory consumption.

In my use case:

• all assets should finish downloading before the loading screen disappears (otherwise what's the point of preloading?)
• long sound assets should be decoded opportunistically to keep memory usage at bay
• some short decoding delay is tolerable for those long background tracks (while it should be minimized for others)

Because Music assets don't get fully downloaded but instead report themselves as loaded as soon as the stream becomes playable, I have avoided them and used Sounds everywhere because of the first requirement. With this PR I hoped to be able to disable preloadAsSound for the longest sounds and use preloadAsFile for them instead, while keeping preloadAsSound for short sounds that should play immediately.

I have actually changed my approach now and worked on making it possible to use HTML5 streaming in Howler in a way that ensures that the whole file gets downloaded first: goldfire/howler.js#1604. With this, preloadAsMusic could actually be used to download the file during the loading screen - right now it only creates a Howl and waits until there's enough data to start playing the stream, but it doesn't wait for it to be actually downloaded - and therefore memory consumption concerns could be avoided too as HTML5 method doesn't decode the whole sound right away into memory (which also makes the decoding delay go away).

[dos1]

...although on closer inspection, using Music won't cut it on mobile either in my case for other reasons :( So, this PR stays relevant as a way to have some sounds be downloaded during the loading screen without decoding them right away.

[arthuro555]

Ah sorry, I missunderstood you and looked at the code too quickly, I thought you were suggesting to replace the current preloading methods completely with XmlHttpRequests.
Something like this, while understandable in your use case still bothers me a bit, since such an option would be confusing to most users, and end up missused in situations where it is not needed. Have you considered - if your target devices support recent enough browser technology - to use a service worker to preload and cache your resources, with tools like workbox-precache? That should be more flexible for you, allow you to ensure that caching is done properly in all cases and how you like, allow the cache to persist across multiple runs of the game, thereby lower the time spent on the loading screen, allow the game to run offline as a PWA... And allow us to not have to support such a preloading option that will very likely end up misunderstood and missused by most our users 😬

To make sure all preloading by the service worker is done, you could modify your GDevelop exports to make the sound loader wait for a message from the service worker that says it is done with the preloading.
To modify your exports pretty easily you can use my GDevelop exporter CLI, it has a plugins feature to modify your export. I already made a plugin to make an automatically generated workbox service worker for a game.

A few links:
https://developer.chrome.com/docs/workbox/modules/workbox-precaching/
https://github.com/arthuro555/gdexporter
https://github.com/arthuro555/gdexporter-plugins
https://github.com/arthuro555/gdexporter-plugins/tree/master/packages/plugin-offline
https://github.com/Gdevelop-game/GDevelop-Open-Game/tree/master/build

[4ian]

Overall, I think it's fair to add this additional optional behavior, brought by the PR, without harming anything.

If my understanding is correct, here are the steps to play a sound or a music:

• Sound (aka WebAudio):
  • Fetch the file > Decode the file (when creating the Howl) > it's ready to play anytime now!
• Music (aka HTML5 audio):
  • Stream the file and stream decode it > it can be played but you need to keep being able to fetch the file (it's not preloaded)

When we preload as...:

• ...a sound: we do fetch and do decode the file. Advantage: it's ready to play anytime! Disadvantage: you had to decode the file.
• ...a music: well I'm not sure we preload anything because of the streaming nature of HTML5 audio? Please correct me here.

When we "preload as a file", we do a browser request so that we fetch the file, and we rely on the browser cache to then provide the file instantly when a sound is needed (so that the decoding step can happen directly) or when a music is needed (so that streaming can start a tiny bit faster I guess?).

To be confident with this, I think we would need:

• To name the option maybe as "Preload file in cache" (and maybe add a description in the interface "The file will be loaded in cache, but not decoded in memory").
• We could probably write about this in the wiki.

a service worker to preload and cache your resources, with tools like workbox-precache

@dos1 can correct me, but I think we're also talking about Android exports too? (in the case of Karambola).
In which case you may want to get files ready in cache but not decode them... or is this not an issue because fetching the file is fast anyway on an Android export (because it's coming from the disk, not from the network)?

using Music won't cut it on mobile either in my case for other reasons :(

Interesting, I'm curious to know the reasons! :)

[dos1]

since such an option would be confusing to most users, and end up missused in situations where it is not needed

I understand the concern, although I'm not sure how much harm could it actually do - the code already avoids using XHR if preloadAsSound is selected too, as it would be redundant. In other cases, the only effect of this option is to make the loading screen wait until the file is downloaded, which I believe may often be actually desired.

To make sure all preloading by the service worker is done, you could modify your GDevelop exports to make the sound loader wait for a message from the service worker that says it is done with the preloading.

That's an interesting idea, although I already have this working with my own builds, so whether I'll have enough motivation to look at it closer remains to be seen 😆

...a music: well I'm not sure we preload anything because of the streaming nature of HTML5 audio? Please correct me here.

It is being preloaded just enough to be able to start playing without delays, but the rest of the file is still being downloaded, so if your network connection slows down or drops it may start buffering or even stop playing altogether.

or when a music is needed (so that streaming can start a tiny bit faster I guess?).

So that the file can be fully played regardless of network conditions. It's still streaming the decoding, but it streams from memory instead of network.

To name the option maybe as "Preload file in cache" (and maybe add a description in the interface "The file will be loaded in cache, but not decoded in memory").

Makes sense 👍

and we rely on the browser cache to then provide the file instantly when a sound is needed

In its current shape, yes, the code from this PR relies on the browser cache. This could be avoided by caching XHR response and giving it to Howler in similar way goldfire/howler.js#1604 does internally, but doing this outside of Howler will require some special care around format detection (and would break some of my other hacks :D), so I left it for later.

or is this not an issue because fetching the file is fast anyway on an Android export (because it's coming from the disk, not from the network)?

Yeah, it's mostly mobile browsers I'm concerned about, for local builds it would be enough to simply disable preloading altogether for those assets.

Interesting, I'm curious to know the reasons! :)

Playing simultaneous Music files can be problematic on mobile platforms, it comes with excessive UI for handling music in some OSes, and of course - browsers blocking autoplay are a PITA, which turns out to be even more troublesome than I initially thought (you would think that already having a playing AudioContext would be a good enough reason to allow Audio objects to play as well, but some browsers don't seem to agree :P)

[dos1]

Updated the name and added descriptions for all preload properties.

[4ian]

Wow I just realised we never merged this.
As I said, I think it's fine to add this to add a bit more control (and this is validated by a real world use case). In the worst case, this is an option we can remove later if we find better solution/smarter audio management!