6.3.16/6.3.17 cause a consistent exit code 139 on sync

[JacobWeyer]

What happened?
Whenever my developers try to connect to our cluster using 6.3.16; they see devspace start to sync and then throw a sync error, connection lost exit code 139

What did you expect to happen instead?
We expect the standard connection

How can we reproduce the bug? (as minimally and precisely as possible)
Unsure what the specifics are causing this breakage

My devspace.yaml:

version: v2beta1
name: product-api
pipelines:
  dev:
    run: start_dev product-api
  # devspace run-pipeline temporal
  temporal: |-
    start_dev product-api-worker
  both: |-
    start_dev product-api
    start_dev product-api-worker
dev:
 product-api:
    labelSelector:
      app.kubernetes.io/name: product-api-backend
      app.kubernetes.io/instance: product-api
    container: backend
    devImage: 11111.dkr.ecr.us-east-1.amazonaws.com/product-api:latest
    env:
      - name: HUSKY
        value: '0'
      - name: NODE_ENV
        value: 'development'
      - name: APP_SECRET_KEY
        value: 'asdf'
    command:
      - npm
    args:
      - run
      - start-dev
    patches:
      - op: replace
        path: metadata.annotations['karpenter.sh/do-not-disrupt']
        value: 'true'
    terminal:
      command: ./devspace_start.sh
      disableReplace: true
    sync:
      - path: ./
        startContainer: true
        printLogs: true
        excludePaths:
          - node_modules/*
          - dist/*
        onUpload:
          exec:
            - command: npm install --include=dev
              onChange:
                - ./package.json
                - ./package-lock.json
              name: npm install
    ports:
      - port: '9230'
    # Node debug and be a memory hog so we can increase the memory
    # This may need tweaked if you keep running out of memory
    resources:
      limits:
        memory: 3Gi
      requests:
        cpu: 0.5
        memory: 3Gi

 product-api-worker:
    labelSelector:
      app.kubernetes.io/name: product-api-worker
      app.kubernetes.io/instance: product-api
    container: worker
    devImage: 1111.dkr.ecr.us-east-1.amazonaws.com-product-api:latest
    env:
      - name: HUSKY
        value: '0'
      - name: NODE_ENV
        value: 'development'
      - name: APP_SECRET_KEY
        value: 'dfasdf'
      - name: USE_ENV
        value: 'true'
    command:
      - npm
    args:
      - run
      - start-worker:k8s
    patches:
      - op: replace
        path: metadata.annotations['karpenter.sh/do-not-disrupt']
        value: 'true'
    terminal:
      command: ./devspace_start.sh
      disableReplace: true
    sync:
      - path: ./
        startContainer: true
        printLogs: true
        excludePaths:
          - node_modules/*
          - dist/*
        onUpload:
          exec:
            - command: npm install --include=dev
              onChange:
                - ./package.json
                - ./package-lock.json
              name: npm install
    ports:
      - port: '9230'
    # Node debug and be a memory hog so we can increase the memory
    # This may need tweaked if you keep running out of memory
    resources:
      limits:
        memory: 3Gi
      requests:
        cpu: 0.5
        memory: 3Gi

Local Environment:

• DevSpace Version: 6.3.16
• Operating System: mac
• ARCH of the OS: ARM64

Kubernetes Cluster:

• Cloud Provider: aws
• Kubernetes Version: [use kubectl version]
  Client Version: v1.33.3
  Kustomize Version: v5.6.0
  Server Version: v1.31.10-eks-931bdca

Anything else we need to know?

user@JYJL7XX1K4 product-api % devspace dev
info Using namespace 'user'
info Using kube context 'workstation'
dev:product-api Waiting for pod to become ready...
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:0/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:1/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: Init:1/2
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: PodInitializing
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: PodInitializing
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: PodInitializing
dev:product-api DevSpace is waiting, because Pod product-api-backend-devspace-586cf8bbfc-p68b4 has status: PodInitializing
dev:product-api Selected pod product-api-backend-devspace-586cf8bbfc-p68b4
dev:product-api ports Port forwarding started on: 9230 -> 9230
dev:product-api sync  Inject devspacehelper...
dev:product-api sync  Start syncing
dev:product-api sync  Sync started on: ./ <-> ./
dev:product-api sync  Waiting for initial sync to complete
dev:product-api sync  Sync Error on /Users/user/Project/product-api: Sync - connection lost to pod user/product-api-backend-devspace-586cf8bbfc-p68b4:  command terminated with exit code 139
dev:product-api sync  Initial sync completed
start_dev: initial sync: Sync - connection lost to pod user/product-api-backend-devspace-586cf8bbfc-p68b4:  command terminated with exit code 139
fatal exit status 1

On 6.3.17 the error is now initial sync: Upstream Sync - connection lost to pod

[JacobWeyer]

#2993 did not fix the problems from #2989

[Mins]

The fix in #2993 works if devspace and devspacehelper is manually compiled from the v6.3.17 release.

I have a feeling that the artifacts for v6.3.17 is accidentally published from a testing branch, instead of the release branch.

Here's the debugging info after devspace upgrade

% devspace version                                                          
DevSpace version : 6.3.17

% devspace dev
...
command terminated with exit code 139

Confirming that the pod's devspacehelper is the same version

kubectl exec the-pod-name -- bash -c "cd /tmp && ./devspacehelper version"
v6.3.17

The devspacehelper is no longer fetched to local. Guessing that this is some WIP code that pulls the binary directly from the pod?

% ls -lah ~/.devspace/devspacehelper/                  
total 0
drwxr-xr-x  4 user  group   128B 22 Aug 10:24 .
drwxr-xr-x  5 user  group   160B 22 Aug 10:23 ..
drwxr--r--  3 user  group    96B 22 Aug 10:24 .devspace
drwxr-xr-x  2 user  group    64B 22 Aug 10:29 latest

% ls -lah ~/.devspace/devspacehelper/latest
total 0
drwxr-xr-x  2 user  group    64B 22 Aug 10:29 .
drwxr-xr-x  4 user  group   128B 22 Aug 10:24 ..

[JacobWeyer]

@zerbitx any way you can verify what @Mins posted above?

[zerbitx]

@JacobWeyer I can confirm that the artifacts in the release are from the release tag. That was automated when I drafted a new release in GitHub. It created the tag and release notes. From there an action built all the artifacts based on the tag.

As for the possibility of local build working, I can't yet say. I was never able to reproduce the issue myself. It was only in working with @Mins that I had any confirmation that the changes that went into 6.3.17 fixed the problem for them.

If you've confirmed you have the v6.3.17 devspacehelper in /tmp of the pod as discussed above and would like to try building locally there are instructions in the Contributing Guide. My first guess as to why this might still be an issue is in this branch, but I'll still have to be able to reproduce it to confirm.

A minimal example, you or anyone following this issue, could share would also be greatly appreciated.

[Mins]

Here's the steps for the manual compilation, if others are interested to repro

git checkout v6.3.17
go build -o /tmp/devspace-6.3.17 main.go && chmod +x /tmp/devspace-6.3.17

# Backup the existing "latest/devspacehelper" if necessary
GOOS=linux GOARCH=amd64 go build -C helper -o ~/.devspace/devspacehelper/latest/devspacehelper

# Run main CLI. It should sync without failures
./tmp/devspace-6.3.17

# Verify that the injected helper is the manually compiled version
kubectl exec <podname> -- ls -lah /tmp/devspacehelper
=> -rwxrwxrwx 1 root root 31M Sep 11 10:35 /tmp/devspacehelper

kubectl exec <podname> -- /tmp/devspacehelper version
=> latest

[Mins]

I just realized there is a feature to inline (assets.go) the helper and UI.

The problem seems to be caused by upx compression.
https://github.com/devspace-sh/devspace/blob/v6.3.17/hack/build-all.bash#L60

Once compressed, I am able to reproduce the segfault with the manually compiled devspacehelper.

@zerbitx can you publish an alpha version of the CLI without upx on devspacehelper for us to do further testing on?

[zerbitx]

Automated tools closed this issue on my behalf, so I'm reopening until we have some verification things are working. An alpha release with upx removed is here which obviously increases the size of the devspacehelper binary. Please try it out at your convenience @Mins and @JacobWeyer with devspace upgrade --version v6.3.18-alpha.0. It might be good to remove or rename ~/.devspace to get a clean starting state.

[Mins]

Thanks for the alpha release @zerbitx! I can confirm that v6.3.18-alpha.0 works for me

% devspace version
DevSpace version : 6.3.18-alpha.0

% kubectl exec <pod> -- /tmp/devspacehelper version
v6.3.18-alpha.0

[zerbitx]

@JacobWeyer Can you confirm this alpha is working for you as well before I create a new release?

[dusanradojcic00]

Had the same issue with 6.3.17. After upgrading to 6.3.18-alpha.0 the issue is gone! Thanks!

[jessebye]

Same for us - alpha resolves the issue.