[DOC] Red compile process and build its runtime

Compile Red to Red/System

If we have a Red script named test.red:

Red [ ]

; numbers
a: 1
b: 2 + a
print b

; strings
s: "hello"
prin s
print " world"

; function and call
inc: func [n][n + 1]
b: inc b
print b

We use rebol or red-063.exe to compile test.red and generate an executable:

; red.r is from https://github.com/red/red/blob/master/red.r,
; you should download the 0.6.3 source code.

rebol>> do/args %red.r "--release %test.red"

or

red-063.exe --release -v 3 test.red > out.reds

We get the output Red/System code as out.reds:

with red [
    root-base: redbin/boot-load system/boot-data yes 
    exec: context [
        ------------| "Symbols" 
        ~datatype!: word/load "datatype!" 
        ~make: word/load "make" 
        ~unset!: word/load "unset!" 
        ~none!: word/load "none!" 
        ~logic!: word/load "logic!" 
        ~block!: word/load "block!" 
        ~string!: word/load "string!" 
        ~integer!: word/load "integer!" 
...省略...
        ~ctx348~fetch-options: word/load "ctx348~fetch-options" 
        ~ctx348~make-actor: word/load "ctx348~make-actor" 
        ~all?: word/load "all?" 

; 下面 3 个是 test.red 中声明的 int 变量和函数，但是没有字符串变量 s
        ~a435: word/load "a" 
        ~b436: word/load "b" 
        ~inc: word/load "inc" 
        ------------| "Literals" 
        ctx45: get-root-node2 89 
        ctx46: get-root-node2 92
...省略...
        ctx431: get-root-node2 1433 
        ctx437: get-root-node 2 
        ts438: as red-typeset! get-root 5 
        ------------| "Declarations" 
        ------------| "Functions" 
        f_inc: func [/local ctx saved ~n] [
            ctx: TO_CTX (ctx437) 
            saved: ctx/values 
            ctx/values: as node! stack/arguments 
            ~n: type-check ts438 0 stack/arguments 
            stack/mark-func-body words/_body 
            stack/reset 
            stack/mark-native ~+ 
            stack/push ~n 
            integer/push 1 
            actions/add* 
            stack/unwind 
            stack/reset ------------| 
            "n + 1" 
            stack/unwind-last 
            ctx/values: saved
        ] 
        ------------| "Main program" ; test.red 的主程序
        stack/mark-native ~set 
        word/push ~datatype! 
        datatype/push TYPE_DATATYPE 
        word/set 
        stack/unwind 
        stack/reset 
        #script %/Volumes/Red/samples/test/test.red 
        #either type = 'exe [stack/push get-cmdline-args] [none/push] 
        #script %/Volumes/Red/samples/test/test.red 
        either all [
            object/unchanged? ~system 203 
            object/unchanged2? ctx202 12 244
        ] [word/set-in-ctx ctx243 4] [
            stack/mark-func ~eval-set-path 
            if stack/arguments > stack/bottom [stack/push stack/arguments - 1] 
            set-path* eval-path _context/get ~system as cell! ~script as cell! ~args 
            stack/unwind
        ] 
        stack/reset ------------| {system/script/args: #system [ #either type = 'exe ...} 
        stack/mark-func ~extract-boot-args 
        f_extract-boot-args 
        stack/unwind ------------| "extract-boot-args" #user-code 
        stack/mark-native ~set 
        word/push ~a435 ----->变量 a
        integer/push 1 
        word/set 
        stack/unwind 
        stack/reset ------------| "a: 1" 
        stack/mark-native ~set 
        word/push ~b436 ----->变量 b
        stack/mark-native ~+ 
        integer/push 2 
        word/get ~a435 
        actions/add* 
        stack/unwind 
        word/set 
        stack/unwind 
        stack/reset ------------| "b: 2 + a" 
        stack/mark-native ~print 
        word/get ~b436 
        natives/print* true 
        stack/unwind 
        stack/reset ------------| "print b" 
        stack/mark-native ~set 
        word/push ~s -----------> 字符串变量 s ？
        string/push as red-string! get-root 0 
        word/set 
        stack/unwind 
        stack/reset ------------| {s: "hello"} 
        stack/mark-native ~prin 
        word/get ~s 
        natives/prin* true 
        stack/unwind 
        stack/reset ------------| "prin s" 
        stack/mark-native ~print 
        string/push as red-string! get-root 1 
        natives/print* true 
        stack/unwind 
        stack/reset ------------| {print " world"} 
        stack/mark-native ~set 
        word/push ~inc 
        _function/push get-root 3 get-root 4 ctx437 as integer! :f_inc null 
        word/set 
        stack/unwind 
        stack/reset ------------| "inc: func [n] [n + 1]" 
        stack/mark-native ~set 
        word/push ~b436 
        stack/mark-func ~inc 
        word/get ~b436 
        f_inc 
        stack/unwind 
        word/set 
        stack/unwind 
        stack/reset ------------| "b: inc b" 
        stack/mark-native ~print 
        word/get ~b436 
        natives/print* true 
        stack/unwind ------------| "print b" #user-code
        ]
    ]
]
...compilation time : 47 ms

What's going on about the compile process?

• How did the test.red compiled to out.reds?
• How did the Red runtime built in to the final executable test.exe?
• Where is the Red runtime starts?

I just make a sequence diagram for this compile process, using PlantUML, in order to see how Red code compiled to Red/System and where Red runtime starts.

From the compiling process, I found the test.red compiled to standard Red/System:

Red/System [origin: 'Red] 

red: context [...] ; The Red runtime context

red/init ; Initial Red runtime (allocate memory and other things).

with red [
    exec: context [
        ------------| "Symbols" 
        ------------| "Literals"
        ------------| "Declarations"
        ------------| "Functions"
        ------------| "Main program"
    ]
]

Now I know the Red runtime is stars from runtime/red.reds, and it's memory model is here runtime/allocator.reds.

Go Red!

Here is the PlantUML source to generate sequence diagram, it must exists many mistakes. Hope you could correct my mistakes, thanks!

You can change the source below and copy to PlantUML online server to regenerate a new diagram.

sequence diagram(click here to see origin SVG image)

PlantUML source code

@startuml
skinparam titleBorderRoundCorner 15
skinparam titleBorderThickness 2
skinparam titleBorderColor red
skinparam titleBackgroundColor Aqua-CadetBlue
title Red compile process and runtime initial\n\nBase on Red 0.6.3(https://github.com/red/red/releases)\n\nSee: http://www.red-lang.org/2011/05/redsystem-compiler-overview.html

skinparam ParticipantPadding 20
skinparam BoxPadding 10

actor Reducer

box "Red command-line front-end" #LightBlue
    participant "red.r\n\nredc: context" as red.r
end box

box "Red compiler"
    participant "compiler.r\n\nred: context" as redcompiler 
end box

box "Red/System compiler" #DarkSalmon
    participant "system/compiler.r\n\nsystem-dialect: context" as rscompiler    
    participant "system/linker.r\n\nlinker: context" as linker
    participant "system/emitter.r\n\nemitter: context" as emitter
    participant "system/loader.r\n\nloader: context" as loader
    participant "system/lexer.r\n\nlexer: context" as lexer
    participant "system/utils/libRedRT.r\n\nlibRedRT: context" as libRedRT.r
end box

box "Red runtime initial" #FFBBBB
    participant "runtime/red.reds\n\nred: context" as redrt
    participant "runtime/allocator.reds" as redalloc
end box

autonumber "<font color=red><b>Step-0  "
Reducer -> red.r : rebol>> do/args %red.r "--release %tests/hello.red"

||45||
== Load compiler toolchain(1): compiler / linker / emitter ==

red.r -> redcompiler : @17> do-cache %compiler.r
redcompiler -> rscompiler : @10> do-cache %system/compiler.r
rscompiler -> linker : @19> do-cache %system/linker.r
linker -[#0000FF]-> rscompiler : return linker: context
rscompiler -> emitter : @20> do-cache %system/emitter.r
emitter -[#0000FF]-> rscompiler : return emitter: context

||45||
== Prepare Red runtime includes ==

rscompiler -> libRedRT.r : @21> do-cache %system/utils/libRedRT.r
libRedRT.r -> libRedRT.r : @13> set [funcs vars] load-cache %system/utils/libRedRT-exports.r\ninclude red/(boot & actions & natives & stack...)
libRedRT.r -[#0000FF]-> rscompiler : return libRedRT: context

||45||
== Load compiler toolchain(2): loader / lexer ==

rscompiler -> loader : @29> loader: do bind load-cache %system/loader.r 'self
loader -> lexer : @10> do-cache %lexer.r
lexer -[#0000FF]-> loader : return lexer: context
loader -[#0000FF]-> rscompiler : return loader: context
rscompiler -[#0000FF]-> redcompiler : return system-dialect: context

||45||
== Initial Red compiler options ==

redcompiler -> redcompiler : load other things
note over redcompiler
	"**Red compiler load other things**"
	lexer: do bind load-cache %lexer.r 'self
	extracts: do bind load-cache %utils/extractor.r 'self
	redbin:	do bind load-cache %utils/redbin.r 'self
	preprocessor: do-cache file: %utils/preprocessor.r
	preprocessor: do preprocessor/expand/clean load-cache file none
end note

redcompiler -[#0000FF]-> red.r : return red: context

||45||
== Compile Red to Red/System ==

red.r -> red.r : @870> redc/main\n@852> if result: compile src opts\n@797> if needs-libRedRT? opts [build-libRedRT opts]
red.r -> redcompiler : @518> result: red/compile script opts
redcompiler -> rscompiler : @4673> if file? file [system-dialect/collect-resources src/1 resources file]
rscompiler -> redcompiler : return %system/assets/red.ico image
redcompiler -> redcompiler : @4691: comp-as-exe src
note over redcompiler
    **`comp-as-exe` return a Red/System code template and its Redbin**
Red/System [origin: 'Red] 
red/init ;**initial Red runtime**
with red [ ;**but WHERE is the `red context?**
    exec: context [
        ------------| "Symbols" 
        ------------| "Literals"
        ------------| "Declarations"
        ------------| "Functions"
        ------------| "Main program"
    ]
]
end note
redcompiler -[#0000FF]-> red.r : return Red/System code template above


||45||
== Load and compile Red runtime, generate a executable ==

red.r -> rscompiler : @820> system-dialect/compile/options/loaded src opts result
rscompiler -> rscompiler : @3871> comp-runtime-prolog to logic! loaded all [loaded job-data/3]
rscompiler -> rscompiler : @3716> script: pick [%red.reds %../runtime/red.reds] encap?
rscompiler -> loader : @3717> script: job loader/process/own script script
loader -> redrt : load many things Red runtime needs
redrt -> redrt : include many things Red runtime needs
note over redrt 
#include %definitions.reds
#include %macros.reds
#include %platform/win32.reds
#include %allocator.reds
#include %datatypes/structures.reds
#include %datatypes/datatype.reds
#include %actions.reds
#include %natives.reds
#include %stack.reds
#include ...
end note

redrt --> loader : return Red runtime files
loader --> rscompiler : return red: context

rscompiler -> rscompiler : @3717> compiler/run job loader/process/own script script
rscompiler -> rscompiler : @3582> comp-dialect
note over rscompiler
emit %runtime/common.reds
emit %red.reds
emit %hello.reds
--->
---> In other words, this step will compile, load, emit below Red/System code
Red/System [origin: 'Red] 
red: context [...] ;**Now we have the `red context**
red/init
with red [
    exec: context [
        ------------| "Symbols" 
        ------------| "Literals"
        ------------| "Declarations"
        ------------| "Functions"
        ------------| "Main program"
    ]
]
end note

rscompiler --> red.r : generate a executable hello.exe
red.r --> Reducer : return a executable hello.exe


||45||
== User run the executable hello.exe ==

Reducer -> redrt : run the executable hello.exe
redrt -> redalloc : red/init
redalloc --> redrt : allocate Red runtime memory
note over redalloc
memory: declare struct! [					; TBD: instanciate this structure per OS thread
	total	 [integer!]						;-- total memory size allocated (in bytes)
	n-head	 [node-frame!]					;-- head of node frames list
	n-active [node-frame!]					;-- actively used node frame
	n-tail	 [node-frame!]					;-- tail of node frames list
	s-head	 [series-frame!]				;-- head of series frames list
	s-active [series-frame!]				;-- actively used series frame
	s-tail	 [series-frame!]				;-- tail of series frames list
	s-start	 [integer!]						;-- start size for new series frame		(1)
	s-size	 [integer!]						;-- current size for new series frame	(1)
	s-max	 [integer!]						;-- max size for new series frames		(1)
	b-head	 [big-frame!]					;-- head of big frames list
]

init-mem: does [
	memory/total: 	0
	memory/s-start: _1MB
	memory/s-max: 	_2MB
	memory/s-size: 	memory/s-start
]
end note

@enduml