Welcome to Metalang99 Discussions!

[Hirrolot]

Feel free to ask any questions here!

[emdash]

Not a question, but I just stumbled across this via a reddit thread involving Interface99. I have written some macro-heavy C and C++ code over the years (I invite you to have a look at https://github.com/emdash/goof).

Long story short, I'm familiar with the macro patterns you are using here, and I would love to see some kind of community coalesce around this kind of thing, because I think it might provide a useful bridge between different high-level languages.

Looking forward to learning more about Metalang99!

[Hirrolot]

Thanks for digging into Metalang99!

Are you talking about something like Metacall, a language interoperability system? I had a discussion with the author of Metacall, and he is eager to try Metalang99-related stuff. But I haven't yet found spare time to investigate Metacall further. Personally, I do also think that Metalang99 and its sibling projects have a chance to be useful in the programming language design community.

I've added your Goof to my awesome-c-preprocessor, thanks for sharing!

[emdash]

I am not familiar with Metacall, so I will take a look at it later when I have some free time. At first glance, it seems to be mainly focused on scripting languages. Forgive me if I am mistaken.

I was just speaking generically about systems languages which on the one hand support high-order programming, but can't expose any advanced features in shared library interfaces.

Devs have been stuck with an impoverished ABI that inherits all the limitations of the platform's C tool-chain for so long that we just accept it. Hence the expanding popularity of "header-only" libraries within the C++ community. If one could lift the level of abstraction exposed by "plain C" libraries, then it might break this logjam.

The other motivation is with embedded code, where I would like type safety, but my vendor toolchain only supports C99 (C11 if you're lucky) with maybe partial support for C++11. I would love to just use Rust for embedded code, but it's still too bleeding edge for a lot of hardware and a lot of companies. Not to mention, llvm does still not target a lot of popular embedded micros, and may never target chips like ESP32.

So when I say "bridge", I mean any scenario where you have different "high-level" or functional languages whose implementations either treat C as compilation target, or else produce object code that is compatible with the platform linker. The historical choice is some kind of "managed" runtime, which certainly works but comes with its own set of trade-offs and compromises.

It seems somewhat analagous to asm.js. It took asm.js to show what life would be like if you could ship low-level byte-code directly to a browser. Now we have WebAssembly, which makes asm.js unecessary. But it took asm.js to point the way. Earlier efforts failed because they were too much of a clean break with existing web standards. So ML99 could point the way towards a richer ABI for native code generation.

P.S., thanks for looking at Goof, that was a fun little project.

[emdash]

BTW, this reminded me I needed to switch the default branch of Goof to the no_commas branch, since that one actually reflects the final state of the project. If you visited the emdash/goof repo previously, the default branch was main.

[emdash]

Okay, as promised, had a look over Metacall, and it is a way more expansive project than it first seemed. I would definitely like to know more about what its author had to say. Seems like a lot of lessons to be learned in there!

[Hirrolot]

Yeah, Metacall is now a fairly advanced language interoperability system -- the author has been working on it for several years. If I understand it correctly, it works by translating language constructions into C ABI and translating C ABI back to another (high-level) language. However, it requires a runtime system to do so (I guess it's the only way to cooperate between two scripting languages). It was started as an author's university project for his game engine architecture.

Regarding your suggestion, I think it might be interesting to compile high-level language constructs into Interface99/Datatype99 code so that they could be exposed in a standard C ABI. Maybe it's possible to do as a part of Metacall (I'm afraid to start a complex language interoperability system on my own). I'll definitely investigate this possibility when I have some time, thank you for your observations. BTW, GObject does pretty much the same thing except that it's pure OOP stuff. I've heard that GObject was designed with language cooperation in mind, hence we have GTK wrappers for almost any high-level language.

[emdash]

Yeah, in the short term, ML99 might have a role to play in the "C as an intermediate language" space. But I can also see (in the embedded systems use case) wanting to write ML99 directly, possibly with the help of additional ML-99-aware tooling.

BTW, I have worked extensively with GObject and its related projects over the years. It's been a little while, but I was working with GStreamer and GTK pretty heavily at the time I wrote Goof.

GObject is 100% about writing C libraries that can be used from any other language. It is basically a "managed runtime". If you write your library against GObject, you get bindings and run-time reflection for any supported language "for free". GObject is also "reactive" via "signals" and "properties". From the perspective of an application developer who exclusively works exclusively from Python or JavaScript, it all works great. And you really can use the same libraries from Lua, Python, JavaScript, Scheme, etc...

Meanwhile, the C library author is living in the worst of both worlds between C and scripting languages: having on the one hand to deal with all the pitfalls and shortcomings of C as an implementation language, plus the boilerplate imposed by GType, and now all fun run-time chaos that dynamic typing with both early and late binding: : memory leaks, reference cycles, monkey patching of virtual methods, callback hell, etc.

Maybe the situation has improved since I was actively working with it. I could also see Rust being a much better implementation language for GObject libraries, though I don't know if the official crate is taking full advantage.

[Hirrolot]

I'm currently writing software for @OpenIPC, an IP cameras startup. We use GNU C99 as a systems implementation language, so I've decided to try Interface99 & Datatype99 out there. They've been working pretty cool for several months and did a huge improvement in the overall architecture. However, we don't use Metalang99 directly -- I just see no real use case for it except metaprograms such as the aforementioned DT99 & IF99. In general, if you don't need to extend C with new language constructs, but you need to only generate something auxiliary, you can use M4 or other code generators that can provide better tooling.

It was interesting to read about your experience with GObject, didn't knew that developers have to deal with reference cycles, monkey patching, etc. under the hood. I also heard about Vala, a language aimed at writing "cool GObject code".

[emdash]

If I am starting a green-field project on hardware I have control over, I would choose embedded Rust over C99, even though it's still early days. I already think it's a huge improvement over existing C/C++ tool-chains, especially the HAL layer which is all based on traits. I only need to write a driver for a SPI device once, and then it will work anywhere. And all the traits will be de-virtualized in the production build.

If I am stuck with C, I'd at least see if I can bump it to C11 for the few nice features it offers. But C99 is alright. The main feature I would add to C, if I could wave a magic wand, is namespacing. Having to manually prefix identifiers to avoid name collisions might just be the biggest pain point in the language.

When I was working on firmware, I was "manually monomorphising" Rust-style "Result" types, and this really did seem to improve the readability and maintainability of the state machine and main loop, but it was so tedious that I did start factoring code into macros that bear at least some resemblance to IF99.

I ended up writing meta-programming macros that generate enums with a _from_string and _to_string method. But of course I was using my own conventions, etc. This is one of those things where, because it's not baked into the language, and because there's no de-facto standard, we all suffer from rolling our own structurally similar but superficially incompatible solutions.

I understand what you're saying about M4, but the thing is, it's also additional tooling. With different environments, especially some of the commercial embedded ones, it won't play well with the debugger or the IDE's project files either. But then you must run M4 to generate valid C, whereas an external linter can run in an entirely different environment and still give me useful feedback about my code.

The other thing I would say is, make sure you compile your code on different compilers. They all output different warnings and choke on different language features, even when operating in "strict" standard modes, though it's gotten a lot better in recent years.

[Hirrolot]

When I do need some mundane metaprogramming, I usually just use X-Macro. _from_string and _to_string are perfect examples for this idiom. The thing I don't see mentioned elsewhere is that X-Macros can be parameters, i.e.:

#define ENUM_VARIANTS(X) X(A) X(B) X(C)

Which makes them far more powerful since now we can invoke ENUM_VARIANTS in some macro FOO two times with different predefined X-Macros.

Yeah, I understand that M4 can be less convenient since it's external tooling. This is especially the case with DT99 & IF99; I covered this in my blog post What’s the Point of the C Preprocessor, Actually?. However, usually, when you use macros, you are "run out of your language". This is not good because macros generally have quite vague semantics in comparison with ordinary functions and etc. I'd suggest understanding which abstraction you're missing in your language and reify it into a separate macro library -- this is what I've done with DT99 & IF99. In my opinion, writing complex macros in a user codebase is quite unfortunate; if you just need to maintain parallel lists, there are cute X-Macros, otherwise, you're doing something unusual that should be moved out into a library.

The thing with different compilers is absolutely valid. I test ML99 and the sibling project on MSVC, TCC, GCC, and Clang, under different operating systems. When I added Windows 10 to the build, it turned out that aux.h is a reserved filename! I changed it to util.h then. Other environments didn't complain about aux.h.

[Hirrolot]

Basically, I conceived Metalang99 as a means for building those libraries. I thought it may be good if we had some of the high-level abstractions in pure C code, such as ADTs or interfaces, in case we cannot afford Rust or C++. Explicitly stating out the memory layout of the generated constructions was super nice: now they are ABI-friendly. We can take advantage of it and use ML99-related stuff not only in private code but also exhibit it in a public interface and probably use it as an inter-language glue (similar to GObject). Though I need some more time to think about this.

[emdash]

Thanks, I'll take a look at that blog post!

[Hirrolot]

By the way, recently found this crate, you may be interested in it: abi_stable_crates.

For Rust-to-Rust ffi, with a focus on creating libraries loaded at program startup, and with load-time type-checking.