How can i use Lemp in my own compiler?

[furesoft]

[qwertie]

It depends. Are you writing a compiler in .NET? Do you intend to write your own parser, or can your language be based on LES syntax (or both)?

[furesoft]

Im writing a compiler in .Net. I write my own Parser with custom Syntax

[qwertie]

In order to use LeMP, your parser needs to produce a Loyc tree for LeMP to process.

You should also write a class for macros you want to always be available in your language. You may want to write this class in Enhanced C# so that you can transform syntax trees easily:

#ecs;
using System;
using Loyc;
using Loyc.Syntax;
using Loyc.Collections;

namespace YourLanguage {
    [ContainsMacros]
    public static partial class Macros {
        // This is an example macro that transforms the "x...y" operator.
        [LexicalMacro("lo...hi", "Given `lo...hi, produces `Range.Inclusive(lo, hi)", "'...")]
        public static LNode RangeIncl(LNode node, IMacroContext context)
        {
            matchCode(node) {
                case $_($lo, $hi): // expect a call with two arguments
                    return quote(Range.Inclusive($lo, $hi));
                default:
                    return null;
            }
        }
    }
}

Then you'll create an instance of the macro processor that uses these macros:

var lmp = new LeMP.MacroProcessor(ConsoleMessageSink.Value, typeof(YourLanguage.Macros))
lmp.PreOpenedNamespaces.Add((Symbol) nameof(YourLanguage));

The simplest way to use LeMP is to call ProcessSynchronously:

var filename = "/path/source-file";
var chars = new StreamCharSource(new FileStream(filename, FileMode.Open));
// replace this with your own parser
var code = Loyc.Syntax.Les.Les3LanguageService.Value.Parse(chars, filename, ConsoleMessageSink.Value, ParsingMode.File, true);
VList<LNode> output = lmp.ProcessSynchronously(LNode.List(code));

If you want to be able to process all source files in parallel, LeMP currently wants you to write a class that implements IParsingService for your language. But you don't have to implement Tokenize or the Parse methods that deal with tokens. So, something like this:

public class YourParsingService : IParsingService
{
    public static YourParsingService Value = new YourParsingService();

    public IEnumerable<string> FileExtensions => new[] { "yourext" };
    public bool HasTokenizer => false;
    public bool CanPreserveComments => false;

    public IListSource<LNode> Parse(ICharSource text, string fileName, IMessageSink msgs = null, ParsingMode mode = null,
        bool preserveComments = true)
    {
        // Replace this with your parser
        return Syntax.Les.Les3LanguageService.Value.Parse(text, fileName, msgs, mode, preserveComments);
    }

    public IListSource<LNode> Parse(ILexer<Token> input, IMessageSink msgs = null, ParsingMode mode = null, bool preserveComments = true)
    {
        throw new NotImplementedException();
    }
    public IListSource<LNode> Parse(IListSource<Token> tokens, ISourceFile file, IMessageSink msgs, ParsingMode inputType)
    {
        throw new NotImplementedException();
    }
    public ILexer<Token> Tokenize(ICharSource text, string fileName, IMessageSink msgs)
    {
        throw new NotImplementedException();
    }
}

You can officially register the parsing service like this:

ParsingService.Register(YourParsingService.Value);

Next, create an InputOutput object for each source file:

var fileList = new List<InputOutput>();
var filename = "/path/source-file";
var stream = new FileStream(filename, FileMode.Open);
// Edit: choose parser by file extension, to support other syntaxes (such as LES)
var ext = Path.GetExtension("foo").Slice(1).ToString();
var parser = ParsingService.RegisteredLanguages.TryGetValue(ext).Or(YourParsingService.Value);
fileList.Add(new LeMP.InputOutput(new StreamCharSource(stream), filename, parser));

Then run the macro processor:

lmp.ProcessParallel(fileList, (InputOutput io) => {});

The lambda will be called after each file is done processing, and io.Output contains the output node list (VList<LNode>).

Edit: if you want to support LeMP's command-line options, set up LeMP with the LeMP.Compiler class instead.

[qwertie]

All of this will work best if your Loyc trees resemble the ones produced by Enhanced C#. For example, the names of operators should start with a single quote ( `'+`(x, y) meaning x+y), if-statements can be represented by #if(condition, trueBranch, falseBranch), etc.

[furesoft]

ive startet the parser, it produces already an loyc tree. i think your tree is very good and flexible.
is there a tutorial for writing a printer for my custom language?

[qwertie]

It has been on my to-do list for a long time to figure out how to write an (efficient) printer more easily. Maybe I will have time to figure that out on Sunday and give you suggestions.

[furesoft]

that would be nice.

[qwertie]

Sorry, I didn't have time today. Hopefully later..

[qwertie]

I still haven't gotten around to working out how to write printers efficiently. You could look at an existing one like Les2Printer.cs, but existing code is not the ideal guidebook.

Probably the best way involves writing a macro for this purpose, which is now easy in version 29 (aka 2.9) thanks to the new macro macro which allows you to compile macros written in Enhanced C# at compile-time. So, I just sketched out a printer for a hypothetical programming language sort of like Ruby/Python:

macro DefinePrinter($(...args)) {
    return quote(----TODO----);
}
macro write($(...args)) {
    return quote(----TODO----);
}
macro infix($precedence, $operatorName) {
    return quote(----TODO----);
}
macro infix($precedence) {
    // Auto-detect operator name
    return quote(infix($precedence, $(LNode.Literal(node.Target.Name.Name))));
}

/// Precedence table
public static class Precedences {
    public static readonly Precedence Primary      = new Precedence(100);          // . :: x() x[] x++ x--
    public static readonly Precedence Multiply     = new Precedence(80);           // * / %
    public static readonly Precedence Shift        = new Precedence(70);           // >> <<
    public static readonly Precedence Add          = new Precedence(60);           // + -
    public static readonly Precedence Compare      = new Precedence(40);           // == != > < >= <=
    public static readonly Precedence And          = new Precedence(25);           // &&    
    public static readonly Precedence Or           = new Precedence(20);           // || ^^
    public static readonly Precedence IfElse       = new Precedence(15);           // if-else operator
    public static readonly Precedence Assign       = new Precedence(28,10,10,10);  // = (different precedence on each side)
    public static readonly Precedence Top          = new Precedence(-1);
}

/// Printer (outline)
public class PrinterForMyLanguage {
    StringBuilder _sb = new StringBuilder();
    
    private void Write(string s) { _sb.Append(s); }
    private void WriteLiteral(LNode literal) { TODO; }
    private void WriteIdentifier(LNode identifier) { TODO; }
    private void StatementOnNewLine(LNode stmt) { TODO; }
    private void Newline() { _sb.Append("\n"); }
    private void Name(LNode node) { Expression(node, Precedences.Top); }
    
    DefinePrinter {
        Expression(LNode node, Precedence prec) {
            if (node.HasAttrs) { 
                // special logic for expressions that have attributes
            }
            case $a.$b:   infix(Precedences.Primary);
            case $a * $b: infix(Precedences.Multiply);
            case $a / $b: infix(Precedences.Multiply);
            case $a + $b: infix(Precedences.Add);
            case $a - $b: infix(Precedences.Add);
            case $a << $b: infix(Precedences.Shift);
            case $a >> $b: infix(Precedences.Shift);
            case $a == $b: infix(Precedences.Compare);
            case $a != $b: op(Precedences.Compare, "<>");
            case $a > $b: infix(Precedences.Compare);
            case $a < $b: infix(Precedences.Compare);
            case $a >= $b: infix(Precedences.Compare);
            case $a <= $b: infix(Precedences.Compare);
            case $a && $b: infix(Precedences.And, "and");
            case $a || $b: infix(Precedences.Or, "or");
            case $a = $b: infix(Precedences.Assign);
            case ($c ? $a : $b): // $a if $c else $b
                write(Expression(a, Precedences.IfElse.LeftContext(prec)),
                      "if", Expression(c, Precedences.Top),
                      "else", Expression(b, Precedences.IfElse.RightContext(prec)));
            case { if ($c) $a; else $b; }: // if $c: $a else: $b
                write("if", Expression(c, Precedences.Top), 
                      ":", StatementOnNewLine(a), writeln("else:"), StatementOnNewLine(b));
            case #while($cond, $body): // while $cond: $body
                write("while", Expression(cond, Precedence.MinValue),
                      ":", StatementOnNewLine(body));
            case { { $(..statements); } }: // begin $(..statements) end
                write(writeln("begin"), list(statements, s => Statement(s)), writeln("end"));
            default:
                if (node.IsCall) {
                    Expression(node.Target, Precedence.Primary);
                    write("(", list(node.Args, arg => Expression(arg, Precedences.Top), ","), ")");
                } else if (node.IsLiteral) {
                    WriteLiteral(node);
                } else if (node.IsId) {
                    WriteIdentifier(node);
                }
        }
        Statement(LNode node) {
            if (node.HasAttrs) { 
                // special logic for statements that have attributes
            }
            case #namespace($namespaceName): // package package.name
                write("package ", Name($namespaceName), Newline());
            case #import($(...namespaces)): // import package1, package2, package3
                write("import ", list(namespaces, ",", ns => Name(ns)), Newline());
            case { class $name : $(...baseClasses) { $(...statements); } }:
                // class $name inherits $(...baseClasses): $(...statements) end
                write("class ", Name(name));
                if (baseClasses.Count != 0)
                    write(" inherits ", list(baseClasses, bc => Name(bc), ","));
                write(writeln(":"), list(statements, s => Statement(s)), writeln("end"));
            case { $retType $methodName($(...args)) => $body; }:
                // fn Name(argument list): $body end
                write("fn ", Name(methodName), 
                    "(", list(args, arg => Expression(arg, Precedences.Top), ","), ")");
                if (body.Calls(S.Braces) || body.Calls(S.Splice))
                    write(writeln(":"), list(body, stmt => Statement(body.Args)));
                else
                    write(": ", Statement(body));
                writeln("end");
        }
    }
}

The idea is to write some macros to help accelerate the process of writing a printer. I haven't written the necessary macros, but I hope this could help someone get started.

[qwertie]

Related to this, in a few days I intend to release v29 which will include StandardLiteralHandlers, which helps you parse and print literals - especially numeric literals - in standard formats, to minimize the amount of parsing/printing code you write yourself and to make it easy to support the new unified literal system (ask me about it). It doesn't do string escaping/unescaping, but functions for that are already provided in classes like ParseHelpers, PrintHelpers and the Les3Lexer.UnescapeQuotedString method.