Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Occasional Errors When Generating Large Diagrams #33

Open
jamestiotio opened this issue Feb 20, 2023 · 14 comments
Open

Occasional Errors When Generating Large Diagrams #33

jamestiotio opened this issue Feb 20, 2023 · 14 comments

Comments

@jamestiotio
Copy link

jamestiotio commented Feb 20, 2023

On top of the err: Timed out: EdgeRouting error specified in #20, experimenting with different seeds to generate a fairly large diagram would also sometimes produce either of the following errors:

  1. failed to compile: exit status 2
    stderr:
    ...
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal 0xc0000005 code=0x0 addr=0x70 pc=0x11116ba]
    
  2. failed to compile: exit status 1
    stderr:
    ...
    err: Reached a bad state: Dimensions w:13458, h:20121 reached after stage GapNormalization
    

The larger the diagram, the higher the chances of these errors occurring (and the lower the chances of successfully generating a diagram).

The diagram does not have to contain $>= 100$ nodes for these errors to start cropping up. I encountered the above issues on a moderately-sized diagram of around 50-60 nodes/shapes (with a maximum of 2 hierarchy levels), 90-100 connections, and a decent amount of text (around 13000 characters including shape labels) sprinkled across the entire diagram.

@jamestiotio jamestiotio changed the title Occasional Errors when Generating Large Diagrams Occasional Errors When Generating Large Diagrams Feb 20, 2023
@alixander
Copy link
Contributor

@jamestiotio sorry for the late reply, been busy getting new TALA release out. Can you try again with the diagram you tried with? it should be faster now. also if you get a panic, would love to get a few more lines of the stack trace (to get to a line number).

@jamestiotio
Copy link
Author

Sure, I have attached this file that contains the complete logs when I encountered the panic. I managed to reproduce the panic using d2 v0.2.1 and TALA v0.3.0.

It seems that only the last part of the logs is relevant:

err: 2023-02-25 06:32:35.536 [INFO]     (default)       <./tala\pipeline.go:537>        (*Pipeline).RunAllStages        finished stage  {"stage": "AlignAxes", "time_ms": 249}
err: 2023-02-25 06:32:35.536 [WARN]     (default)       <./..\..\..\..\oss.terrastruct.com\d2\lib\log\log.go:31>        from    missing slog.Logger in context, see lib/log.With ...
err:   "stack": goroutine 10 [running]:
err:            runtime/debug.Stack()
err:                    runtime/debug/stack.go:24 +0x65
err:            oss.terrastruct.com/d2/lib/log.from({0x1f31680, 0xc0001dac00})
err:                    oss.terrastruct.com/[email protected]/lib/log/log.go:31 +0x128
err:            oss.terrastruct.com/d2/lib/log.Debug({0x1f31680, 0xc0001dac00}, {0x1411ffe, 0x11}, {0xc0005b9a28, 0x1, 0x1})
err:                    oss.terrastruct.com/[email protected]/lib/log/log.go:52 +0x7b
err:            github.com/terrastruct/src/backend/tala.(*Pipeline).RunAllStages(0xc0000b6c30, {0x1f31680, 0xc0001dac00})
err:                    github.com/terrastruct/src/backend/tala/pipeline.go:507 +0x21b
err:            github.com/terrastruct/src/backend/tala.Autolayout({0x1f31680, 0xc0001dac00}, 0x118f7dd?, 0xc0004a5ef0?, 0x0?, 0x39?, 0x68?)    
err:                    github.com/terrastruct/src/backend/tala/pipeline.go:558 +0x11b
err:            github.com/terrastruct/src/backend/tala.AutolayoutLocal({0x1f31680, 0xc0001dac00}, 0x13f75c6?, 0x4?, 0x0?, 0xc0005b5d28?)       
err:                    github.com/terrastruct/src/backend/tala/pipeline.go:603 +0x1d5
err:            github.com/terrastruct/src/backend/tala/transpilers/d2transpiler.Layout({0x1f31610, 0xc000d4b600}, 0xc0005b5db0?, 0x1?)
err:                    github.com/terrastruct/src/backend/tala/transpilers/d2transpiler/d2transpiler.go:162 +0x376
err:            main.talaPlugin.Layout({0xc000e411d0?, 0xc000d48688?}, {0x1f31610?, 0xc000d4b600?}, 0x0?)
err:                    github.com/terrastruct/src/backend/tala/cmd/d2plugin-tala/plugin_tala.go:161 +0x12c
err:            oss.terrastruct.com/d2/d2plugin.layout({0x1f31610, 0xc000d4b600}, {0x1f32f40, 0xc000e6cdb0}, 0xc000086480)
err:                    oss.terrastruct.com/[email protected]/d2plugin/serve.go:106 +0xb5
err:            oss.terrastruct.com/d2/d2plugin.Serve.func1({0x1f31610, 0xc000d4b600}, 0xc000086480)
err:                    oss.terrastruct.com/[email protected]/d2plugin/serve.go:56 +0x2e5
err:            oss.terrastruct.com/util-go/xmain.(*State).Main.func1()
err:                    oss.terrastruct.com/[email protected]/xmain/xmain.go:91 +0x71
err:            created by oss.terrastruct.com/util-go/xmain.(*State).Main
err:                    oss.terrastruct.com/[email protected]/xmain/xmain.go:89 +0x155
err: panic: runtime error: invalid memory address or nil pointer dereference
err: [signal 0xc0000005 code=0x0 addr=0x78 pc=0x117c617]
err:
err: goroutine 10 [running]:
err: github.com/terrastruct/src/backend/tala.(*Node).getContainer(...)
err:    github.com/terrastruct/src/backend/tala/node.go:3432
err: github.com/terrastruct/src/backend/tala.(*Node).reduceGapToNeighbors(0xc000926e40, {0x1f31680?, 0xc0001dac00}, 0x1, 0x0, 0x1)
err:    github.com/terrastruct/src/backend/tala/gapreduction.go:332 +0x10d7
err: github.com/terrastruct/src/backend/tala.Nodes.GapNormalization.func2()
err:    github.com/terrastruct/src/backend/tala/gapreduction.go:39 +0x36
err: github.com/terrastruct/src/backend/tala.(*Transaction).Commit(0xc0005d34a0, {0x280fd20?, 0xc000926e40?})
err:    github.com/terrastruct/src/backend/tala/transaction.go:237 +0x97
err: github.com/terrastruct/src/backend/tala.Nodes.GapNormalization({0xc000b725a0?, 0x2, 0xc001006948?}, {0x1f31680?, 0xc0001dac00?}, 0x1, 0x0, 
0xc0002426c0)
err:    github.com/terrastruct/src/backend/tala/gapreduction.go:42 +0x2d2
err: github.com/terrastruct/src/backend/tala.(*Pipeline).GapNormalizationStage.func1({0xc000b725a0, 0x2, 0x2}, 0xd0?)
err:    github.com/terrastruct/src/backend/tala/pipeline.go:199 +0x8a
err: github.com/terrastruct/src/backend/tala.(*Pipeline).GapNormalizationStage(0xc0000b6c30, {0x1f31680?, 0xc0001dac00?})
err:    github.com/terrastruct/src/backend/tala/pipeline.go:208 +0x18f
err: github.com/terrastruct/src/backend/tala.(*Pipeline).RunAllStages(0xc0000b6c30, {0x1f31680, 0xc0001dac00})
err:    github.com/terrastruct/src/backend/tala/pipeline.go:510 +0x25f
err: github.com/terrastruct/src/backend/tala.Autolayout({0x1f31680, 0xc0001dac00}, 0x118f7dd?, 0xc0004a5ef0?, 0x0?, 0x39?, 0x68?)
err:    github.com/terrastruct/src/backend/tala/pipeline.go:558 +0x11b
err: github.com/terrastruct/src/backend/tala.AutolayoutLocal({0x1f31680, 0xc0001dac00}, 0x13f75c6?, 0x4?, 0x0?, 0xc0005b5d28?)
err:    github.com/terrastruct/src/backend/tala/pipeline.go:603 +0x1d5
err: github.com/terrastruct/src/backend/tala/transpilers/d2transpiler.Layout({0x1f31610, 0xc000d4b600}, 0xc0005b5db0?, 0x1?)
err:    github.com/terrastruct/src/backend/tala/transpilers/d2transpiler/d2transpiler.go:162 +0x376
err: main.talaPlugin.Layout({0xc000e411d0?, 0xc000d48688?}, {0x1f31610?, 0xc000d4b600?}, 0x0?)
err:    github.com/terrastruct/src/backend/tala/cmd/d2plugin-tala/plugin_tala.go:161 +0x12c
err: oss.terrastruct.com/d2/d2plugin.layout({0x1f31610, 0xc000d4b600}, {0x1f32f40, 0xc000e6cdb0}, 0xc000086480)
err:    oss.terrastruct.com/[email protected]/d2plugin/serve.go:106 +0xb5
err: oss.terrastruct.com/d2/d2plugin.Serve.func1({0x1f31610, 0xc000d4b600}, 0xc000086480)
err:    oss.terrastruct.com/[email protected]/d2plugin/serve.go:56 +0x2e5
err: oss.terrastruct.com/util-go/xmain.(*State).Main.func1()
err:    oss.terrastruct.com/[email protected]/xmain/xmain.go:91 +0x71
err: created by oss.terrastruct.com/util-go/xmain.(*State).Main
err:    oss.terrastruct.com/[email protected]/xmain/xmain.go:89 +0x155

@jamestiotio
Copy link
Author

jamestiotio commented Feb 25, 2023

Additionally, it seems that a diagram that was successfully generated on older TALA versions (tested on v0.2.15) can fail to compile on v0.3.0 of TALA with the following error:

err: failed to compile: Connection "empty.(not_present -> not_present)[0]" is a self loop on a container, but layout engine "tala" does not support this.

The empty.not_present node and self-connection are declared as such:

empty.not_present: not_present {
  ex: |`txt
    Mucho texto.
  `|
}

empty.not_present -> empty.not_present: |`md
  Stay in this state if blablabla...
`|

This is strange since using only the above declaration works (as demonstrated here), but in the context of additional nodes and connections present in a large diagram such as mine, it fails.

@alixander
Copy link
Contributor

ah shoot, someone added that feature flag and forgot to add it to TALA plugin. i'm going to release a hotfix for that. thank you for surfacing. cc @gavin-ts

@jamestiotio
Copy link
Author

i'm going to release a hotfix for that. thank you for surfacing.

No problem.

Testing with TALA v0.3.1 (which includes the hotfix for the container self-loop edges) seems to indicate that it is much more stable now. I have not encountered any Timed out: EdgeRouting or Reached a bad state errors so far. I do still occasionally encounter the panic issue. Once the panic is resolved, I think we can close this issue.

@orange-buffalo
Copy link

With d2 v0.4.2 and TALA v0.3.7 I am facing a similar issue on a fairly complex graph. The output and the failure is:

info: compiling & running layout algorithms...
info: compiling & running layout algorithms...
info: compiling & running layout algorithms...
info: compiling & running layout algorithms...
err: failed to compile: exit status 1
err: stderr:
err: 2023-05-22 07:22:22.496 [WARN]     (default)       <./../../../../oss.terrastruct.com/d2/lib/log/log.go:31>        from    missing slog.Logger in context, see lib/log.With ...
...
err: 2023-05-22 07:22:22.496 [INFO]     (default)       <./tala/pipeline.go:548>        (*Pipeline).RunAllStages        finished stage  {"stage": "Prescale", "time_ms": 0}
err: 2023-05-22 07:22:22.496 [INFO]     (default)       <./tala/pipeline.go:548>        (*Pipeline).RunAllStages        finished stage  {"stage": "Prescale", "time_ms": 0}
err: 2023-05-22 07:22:22.496 [INFO]     (default)       <./tala/pipeline.go:548>        (*Pipeline).RunAllStages        finished stage  {"stage": "Prescale", "time_ms": 0}
err: 2023-05-22 07:22:22.497 [INFO]     (default)       <./tala/pipeline.go:548>        (*Pipeline).RunAllStages        finished stage  {"stage": "PreprocessSequences", "time_ms": 0}
err: 2023-05-22 07:22:41.373 [INFO]     (default)       <./tala/pipeline.go:548>        (*Pipeline).RunAllStages        finished stage  {"stage": "NodePlacement", "time_ms": 18850}
err: 2023-05-22 07:22:41.569 [INFO]     (default)       <./tala/pipeline.go:548>        (*Pipeline).RunAllStages        finished stage  {"stage": "NodePlacement", "time_ms": 19046}
err:   "stack": goroutine 49 [running]:
err:            runtime/debug.Stack()
err:                    runtime/debug/stack.go:24 +0x65
err:            oss.terrastruct.com/d2/lib/log.from({0x2492210, 0xc0015bb440})
err:                    oss.terrastruct.com/[email protected]/lib/log/log.go:31 +0x108
err:            oss.terrastruct.com/d2/lib/log.Debug({0x2492210, 0xc0015bb440}, {0x16b70c2, 0x11}, {0xc0008d9a78, 0x1, 0x1})
err:                    oss.terrastruct.com/[email protected]/lib/log/log.go:52 +0x7b
err:            github.com/terrastruct/src/backend/tala.(*Pipeline).RunAllStages(0xc0000aa050, {0x2492210, 0xc0015bb440})
err:                    github.com/terrastruct/src/backend/tala/pipeline.go:518 +0x21b
err:            github.com/terrastruct/src/backend/tala.Autolayout({0x2492210, 0xc0015bb440}, 0x1630340?, 0xa9f7?, 0x98?, 0x89?, 0x9a?)
err:                    github.com/terrastruct/src/backend/tala/pipeline.go:569 +0x11b
err:            github.com/terrastruct/src/backend/tala.AutolayoutLocal({0x2492210, 0xc0015bb440}, 0x0?, 0x0?, 0x0?, 0x0?)
err:                    github.com/terrastruct/src/backend/tala/pipeline.go:614 +0x1d5
err:            github.com/terrastruct/src/backend/tala/transpilers/d2transpiler.RaceSeeds.func1(0x2, 0x3)
err:                    github.com/terrastruct/src/backend/tala/transpilers/d2transpiler/d2transpiler.go:287 +0x5d6
err:            created by github.com/terrastruct/src/backend/tala/transpilers/d2transpiler.RaceSeeds
err:                    github.com/terrastruct/src/backend/tala/transpilers/d2transpiler/d2transpiler.go:257 +0x28f
err: err: Timed out: Autolayout

@alixander
Copy link
Contributor

@orange-buffalo would you mind sending me the D2 for that for us to investigate? [email protected] if you prefer email

@orange-buffalo
Copy link

@alixander, sent via email, hopefully it does not end up in the spam folder :)

@orange-buffalo
Copy link

@alixander, is there anything else we can provide to help with the investigation? We are currently evaluating TALA to support our architecture diagrams, and this issue is a blocker for us.

@alixander
Copy link
Contributor

alixander commented Jun 2, 2023

oh we actually did fix this but just haven't done a release. will cut one today and email you when it's out @orange-buffalo

@orange-buffalo
Copy link

Thanks @alixander! I've tested v0.3.8, but it fails with the same Timed out: Autolayout exception. The diagram has evolved since the last two weeks, I am sending a new version via email.

@orange-buffalo
Copy link

@alixander, with v0.3.9 we still observe timeouts on our diagram. Any chance to have a timeout configuration so we could set it is get the diagram generated, even if it takes more time?

@alixander
Copy link
Contributor

alixander commented Jul 31, 2023

@orange-buffalo just did a new release! should be able to set timeout now with the latest D2 (might have to wait a bit for package managers like homebrew to pick it up) and latest TALA

d2 --timeout 200 large.d2 to increase from 120s to 200s

@orange-buffalo
Copy link

Thank you @alixander! We now have it generated successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants