🐞 Dagger Fails due to ci/internal/telemetry error

[levlaz]

What is the issue?

Users are reporting intermittent errors where dagger starts to fail with this error:

cannot use proxySpan{…} (value of type proxySpan)

Dagger version

dagger v0.11.2 (registry.dagger.io/engine) darwin/arm64

Steps to reproduce

No response

Log output

✘ initialize 2.4s
! input: module.withSource.initialize resolve: failed to initialize module: failed to call module "ci" to get functions: call constructor: process "go build -o /runtime ." did not complete successfully: exit 
code: 1

Stderr:
# ci/internal/telemetry
internal/telemetry/proxy.go:50:14: cannot use proxySpan{…} (value of type proxySpan) as "go.opentelemetry.io/otel/trace".Span value in return statement: proxySpan does not implement "go.opentelemetry.io/otel/
trace".Span (missing method AddLink)
internal/telemetry/proxy.go:59:20: cannot use proxySpan{} (value of type proxySpan) as "go.opentelemetry.io/otel/trace".Span value in variable declaration: proxySpan does not implement "go.opentelemetry.io/ot
el/trace".Span (missing method AddLink)
  ✔ ModuleSource.asModule: Module! 2.1s
  ✘ Module.initialize: Module! 0.3s
  ! failed to initialize module: failed to call module "ci" to get functions: call constructor: process "go build -o /runtime ." did not complete successfully: exit code: 1
    ✘ exec go build -o /runtime . 0.3s
    ! process "go build -o /runtime ." did not complete successfully: exit code: 1
    ┃ # ci/internal/telemetry                                                                                                                                                                                   
    ┃ internal/telemetry/proxy.go:50:14: cannot use proxySpan{…} (value of type proxySpan) as "go.opentelemetry.io/otel/trace".Span value in return statement: proxySpan does not implement "go.opentelemetry.io
    ┃ trace".Span (missing method AddLink)                                                                                                                                                                      
    ┃ internal/telemetry/proxy.go:59:20: cannot use proxySpan{} (value of type proxySpan) as "go.opentelemetry.io/otel/trace".Span value in variable declaration: proxySpan does not implement "go.opentelemetry
    ┃ el/trace".Span (missing method AddLink)                                                                                                                                                                   

Error: input: module.withSource.initialize resolve: failed to initialize module: failed to call module "ci" to get functions: call constructor: process "go build -o /runtime ." did not complete successfully: 
exit code: 1

Stderr:
# ci/internal/telemetry
internal/telemetry/proxy.go:50:14: cannot use proxySpan{…} (value of type proxySpan) as "go.opentelemetry.io/otel/trace".Span value in return statement: proxySpan does not implement "go.opentelemetry.io/otel/
trace".Span (missing method AddLink)
internal/telemetry/proxy.go:59:20: cannot use proxySpan{} (value of type proxySpan) as "go.opentelemetry.io/otel/trace".Span value in variable declaration: proxySpan does not implement "go.opentelemetry.io/ot
el/trace".Span (missing method AddLink)

[levlaz]

Workaround (thank you @sagikazarmark)

Adding this to go.mod fixes it:

replace go.opentelemetry.io/otel/trace => go.opentelemetry.io/otel/trace v1.24.0

Parent project is at 1.26.0

[sagikazarmark]

I figured out I added a go.work file to the parent project (the Dagger module is in a ci subdirectory). I guess everything gets mounted (hopefully contextual modules will solve this?).

In the meantime, this might be a better workaround in dagger.json:

  "exclude": [
    "go.work",
    "go.work.sum"
  ],

[jedevc]

Aha, I think this might be solved by #7194 already - which will be in v0.11.3 (to be released this week).

Essentially, the behavior now is to always ensure that the go.mod provided files are always respected.

[sagikazarmark]

Essentially, the behavior now is to always ensure that the go.mod provided files are always respected.

What does that mean exactly? In this case, a go.work file in the parent should clearly not be respected.

(I'm not even sure why the parent is mounted)

[skycaptain]

Encountered the issue when switching from v0.11.2 to v0.11.4. Since I had no additional dependencies other than dagger, I removed go.sum and go.mod and ran dagger develop to fix the issue locally.

[jedevc]

Aha right, okay, this will be solved in #7264.

The problem that seems to be happening is that the ci subdirectory depends on the parent - the parent requires OTEL v1.26.0. So when go mod tidy is run, the child pulls in the v1.26.0 dependency.

This is really annoying: IMO, a minor version bump shouldn't add more methods to interfaces, which then means that this is a backwards incompatible change. But anyways, they did it. I think the right approach is to pin it using a replace directive, or some other go.work thing.

Not quite sure what the right call is long term here, will have a bit of a think.