[PAID BOUNTY] [LOCKED] Switch the console runtime from BRUTE to NativeAOT

[mrhelmut]

The MonoGame Foundation wishes to incentivize the deprecation of the current console .NET runtime (BRUTE) in favor of switching to NativeAOT. This is a paid bounty.

📃Full details

The .NET console runtime of MonoGame needs to be changed to NativeAOT (publicly known as the PublishAot build option since .NET 7, see documentation).

Related issue on the MonoGame issue tracker for more context: #7873

The MonoGame Foundation wishes to accelerate this transition by offering a paid opportunity to undertake this task.

Read this page to check how to apply and the requirements.

🔒 Bounty status

Locked for @tomspilman to complete. 🔒

💰 Bounty

$5.000,00 USD (Five thousands US dollars).

✋ Requirements to apply

In addition to our generic bounty requirements, here are requirements specific to this bounty.

This bounty requires to be a registered developer with all of the console platforms (Sony, Microsoft, and Nintendo). We would like that this task is undertaken by a single person/entity, and you would therefore have to be cleared with all manufacturers. We need the implementation to have the same structure and user experience throughout all platforms, hence it would be more complex to manage if we were to accept per-platform applications. If need be, we can help with getting registered with either platforms.

📦 Expected delivery content

• All console platforms working with NativeAOT with no regression in regard to current MonoGame features (we can however expect runtime regressions since BRUTE and NativeAOT might have big differences).
• A smooth "build & run/debug" user experience, and if it can't be achieved a good documentation to get a project running with NativeAOT on consoles.

📅 Delivery timeline

As soon as possible.

📚 Historical context

Long story: #7873

Back when MonoGame was initially ported to game console hardware, the most recent .NET version was .NET 4.5, associated with C# 5.0. At that time, .NET was a closed source technology and there was no runtime available to run .NET applications on game consoles.

Mono (the open-source reimplementation of the .NET runtime) could be ported to consoles (which Xamarin did to some extent) but this didn't cover all consoles and in many scenario Mono was too slow for games on consoles. Also, it was uncertain if Xamarin would keep putting efforts into bringing Mono to new consoles. Therefore, Sickhead Games came up with a plan: create and maintain their own .NET runtime, specifically crafted for performance, and ease to implement new consoles. That runtime is BRUTE (and only available on demand).

BRUTE doesn't work like any .NET runtime. It works by decompiling already compiled .NET applications into C++ source code, and developers can then compile this C++ code using console compilers directly into native code. Along the way, it plugs the C++ code with a garbage collector to manage memory, and also strips out many .NET features useless for games to boost the performances. In many cases, BRUTE is 10 to 100 times faster than Mono, and its C++ code is easily portable to any platform with a standard compiler available.

For the most part, BRUTE worked just well, down to limited hardware such as the PS Vita and the Wii U.

The problem

The main issue with BRUTE is that it has been started at a time at which the bleeding-edge version of .NET was .NET 4.5 and the latest language version was C# 5.0. For a long time, this was just fine, and it was assumed that it wouldn't be too complex to upgrade BRUTE to newer .NET standards.

In the end, BRUTE grew to be a very big project with now 10 years of history. The fast evolutions of .NET and breaking changes within dependencies of BRUTE made upgrading BRUTE very complex. Supporting current .NET basically requires the whole tool to be rewritten from scratch, which is months of work introducing possible regressions.

The major impact is that this limitation forces the main public repository of MonoGame to enforce a strict compatibility with .NET 4.5 and C# 5.0. This only applies to MonoGame itself, games made with MonoGame don't have to abide to this (unless they want to work on consoles). This limitation drives innovation backward and induces frustration within the community.

The solution

In the recent history of .NET, Microsoft introduced a new runtime in the name of NativeAOT (which is hidden behind the PublishAot build parameter since .NET 7.0). This new build option takes the compiled IL code and translates it directly into native code (without an intermediate language such as C++), and packages the .NET runtime with it. This produces a monolithic executable that is entirely native code. This runtime became quickly popular among game developers for requiring no runtime installation, having no JIT jitters, obfuscating code, and producing a clean single output (that is not an hidden zip file like PublishSingleFile does). Games like Streets of Rage 4, Carrion, or ScourgeBringer were published successfully to millions of players with NativeAOT on PC platforms.

Lately, the FNA team and Cellar Door Games ported NativeAOT to gaming consoles hardware, making the latest advancements in .NET available on consoles. Despite being less performant than BRUTE, NativeAOT is much more convenient to use and more featureful. It also has the advantage of being backed by Microsoft which guarantees its future within the .NET ecosystem.

The MonoGame Foundation has therefore voted to switch to NativeAOT and join the maintenance effort of its console ports along with the FNA team. The goal is to deprecate BRUTE.

The technical details

The details vary depending on the platform:

• PlayStation and Xbox platforms: those have their platform code (e.g. low level graphics, audio, and I/O) into a native C++ library that MonoGame then calls through DllImport. Switching to NativeAOT shouldn't be too complex and should theoretically involve restructuring the project template to work with NativeAOT. It is not expected to require to rewrite platform code for this to work (but we should expect the transition to not be that magical).
• Nintendo platform: this platform is more tricky because it uses a feature unique to BRUTE to avoid having a native library wrapped around DllImport. To put it bluntly, BRUTE has a feature that allows to write C++ code directly into empty C# methods. This code is then statically linked at compile time without having to do any kind of glue. Migrating to NativeAOT would require to extract all this C++ code into a native library (like the other platforms do), and replace those magic methods with DllImport. This is the platform that is expected to be the most time consuming to migrate. It is not expected that the C++ code should be rewritten (and it shouldn't to avoid regressions), but it is a lot of copy/pasting/testing while restructuring this code into a native library.

[Jure-BB]

Hello. Just a little nitpick. I think that the new LibraryImport is probably more appropriate than the DllImport, since NativeAOT is the target.

Using DllImport isn't an option for platforms that require full Native AOT scenarios and therefore using other approaches (for example, source generation) is more appropriate.
...
The P/Invoke source generator, included with the .NET 7 SDK and enabled by default, looks for LibraryImportAttribute on a static and partial method to trigger compile-time source generation of marshalling code, removing the need for the generation of an IL stub at run time and allowing the P/Invoke to be inlined.

https://learn.microsoft.com/en-us/dotnet/standard/native-interop/pinvoke-source-generation

[mrhelmut]

DllImport works just fine with NativeAOT. Maybe some other ahead-of-time environments don't, but we already shipped games using DllImport and NativeAOT so we should be all fine.

(NativeAOT has modes which are stricter and might require that, but it is not required to go as far as that on consoles with MonoGame.)

[Jure-BB]

You're right. DllImport doesn't work only in full NativeAOT.

[mrhelmut]

Following the foundation meeting of March 14, the applications have been reviewed and this bounty has been locked for @tomspilman to complete.