string.byte() as a companion/inverse to string.char()

[tcoram]

Hi,
I've been playing around with converting my binary protocol packer/unpacker, from plain Lua 5.4 to Pallene (using string.char() as the method for packing into a table that I later table.concat -- outside of Pallene). With Palene, my protocol packing is now over 2x faster!

But now that I am working on unpacking, I find that I am missing string.byte()! That is, I want to be able to do (contrived example here):

function u.test () : integer
   local orig_num : integer = 167
   local bin_str : string = string.char(orig_num)
   return string.byte(bin_str, 1)
end

I really like that Pallene doesn't have everything thrown in, but to me, string.char() begs for a string.byte() companion.
I've made local mods to the Pallene source that seems to do the job, but certainly not fully thought out or tested well enough to request a pull.

I know that I can pull over the Lua string.byte() with a function prototype, but Pallene's native string.sub(), string.char() does most of what I want... I just need string.byte()...

Thoughts?

[hugomg]

We would love to eventually support the entire Lua standard library. The main catch has been:

1. We haven't taught Pallene how to deal with optional parameters or variadic parameters.
2. Due to an excessive focus on benchmarks, we reimplemented all of our standard library functions instead of just calling the ones from the Lua standard library. And we never finished the whole library... We should probably implement a way to grab the functions from the Lua stdlib instead of having to reimplement everything. See also: Allow using builtins as higher-order functions #337

[tcoram]

I actually "like" that the entire standard library hasn't been pulled in. Variadic/optional parameters, associative arrays, etc all make it more difficult for me to reason about performance and bottlenecks. (It reduces the temptation to do "too much" in Pallene).
In my case, a networked application, I do the protocol packing/unpacking as well as packet inspection/validation in Pallene and keep the socket/network facing code in Lua...

Having a way to grab a particular standard library (low level) call would be nice (for "os", etc). Right now I do wrapping and type aliases to bring Lua functions in...

But, thanks for the response. Looking forward to how you guys decide to approach this.

[hugomg]

Would you be interested in implementing the functionality for grabbing those missing builtins? I expect that it might be a good starting point for finding your way around the Pallene compiler; I would be more than willing to guide you along the way.

[tcoram]

Yes. I would be interested. I would rather work in the same direction as the Pallene team rather than fork and make hacks that just meets my needs. Or, at least, I would like to try to the best of my abilities to help implement...

Where do I start looking?

[hugomg]

Thanks! First of all, we should probably move this discussion into a Github issue (please create one).

Next, some starting points:

• builtins.lua specifies which builtins are available, and their types
• typechecker.lua needs to know the builtins list, to properly typecheck the file
• ir.lua currently specifies each builtin as a special instruction. For the new builtins, we should probably specify them as upvalues, containing references to function objects.
• to_ir.lua compiles the syntax tree down to intermediate representation. It should know which builtins are special instructions and which are just regular functions.
• coder.lua generates the final C code. Here we will have to add the code to grab the necessary builtin functions at module-initialization time (inside the luaopen_modname).

When we get to the coder parts, a good strategy is first play with the generated C files by hand, before properly implementing the feature in the coder.lua