The scan status displays 'Scanning,' even though the job has reached the specified backoff limit

[Alon-Katz]

🐞 Bug report

Describe the bug

The scan job created by the scan CRD is in a failed state with the event: 'Job has reached the specified backoff limit.' However, the scan CRD continues to display a 'Scanning' status and does not update to 'Failed.'
The latest version included a bug fix intended to address the issue 'Fixed Scans being marked as Failed after the first job has failed,' which may be relevant.
Unfortunately, this fix seems to have exacerbated the problem, as now scans are incorrectly showing a 'Scanning' status when they have actually failed.
ref to the bug fix: #2205

Steps To Reproduce

Run a scan that is failing, look at the job events, view the scan status

Expected behavior

If the job has reached the maximum backoff limit, the scan status should be "Failed" and not "Scanning"

System (please complete the following information):

• secureCodeBox Version/Release:
  v4.4.1
• Kubernetes Version [command: kubectl version]:
  Client Version: v1.29.2
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.28.5-eks-5e0fdde

Screenshots / Logs

Job output:

Name:             scan-2f1cf8e1-8bf5-4e7e-a0de-93fe6ac44005-rp8mw
Namespace:        scb-scanners
Selector:         batch.kubernetes.io/controller-uid=2ff42089-79a1-4620-a58d-e0175702153e
Labels:           securecodebox.io/job-type=scanner
Annotations:      <none>
Controlled By:    Scan/2f1cf8e1-8bf5-4e7e-a0de-93fe6ac44005
Parallelism:      1
Completions:      1
Completion Mode:  NonIndexed
Start Time:       Sun, 03 Mar 2024 10:33:15 +0200
Pods Statuses:    0 Active (0 Ready) / 0 Succeeded / 1 Failed
Pod Template:
  Labels:           app.kubernetes.io/managed-by=securecodebox
                    batch.kubernetes.io/controller-uid=2ff42089-79a1-4620-a58d-e0175702153e
                    batch.kubernetes.io/job-name=scan-2f1cf8e1-8bf5-4e7e-a0de-93fe6ac44005-rp8mw
                    controller-uid=2ff42089-79a1-4620-a58d-e0175702153e
                    job-name=scan-2f1cf8e1-8bf5-4e7e-a0de-93fe6ac44005-rp8mw
  Annotations:      auto-discovery.securecodebox.io/ignore: true
                    sidecar.istio.io/inject: false
  Service Account:  lurker
  Containers:
   gitleaks:
    Image:      docker.io/zricethezav/gitleaks:v8.18.2
    Port:       <none>
    Host Port:  <none>
    Command:
      gitleaks
      --verbose
      --report-format
      sarif
      --report-path
      /home/securecodebox/report.json
      --exit-code
      0
      detect
      detect
      --source
      /f14b0104-c583-45ae-bba6-d7eb8df767d0/48ec99a2-61f7-4a07-992b-cdb2bc7a0467/379931510/efea2f02da73197dd1f5faeb174fc6cddd36e459
      --log-opts
      efea2f02da73197dd1f5faeb174fc6cddd36e459
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:     500m
      memory:  1Gi
    Environment:
      STAGE:  prod
    Mounts:
      /f14b0104-c583-45ae-bba6-d7eb8df767d0/48ec99a2-61f7-4a07-992b-cdb2bc7a0467/379931510/efea2f02da73197dd1f5faeb174fc6cddd36e459 from efs (ro,path="f14b0104-c583-45ae-bba6-d7eb8df767d0/48ec99a2-61f7-4a07-992b-cdb2bc7a0467/379931510/efea2f02da73197dd1f5faeb174fc6cddd36e459")
      /home/securecodebox/ from scan-results (rw)
   lurker:
    Image:      docker.io/securecodebox/lurker:4.4.1
    Port:       <none>
    Host Port:  <none>
    Args:
      --container
      gitleaks
      --file
      /home/securecodebox/report.json
      --url
      ...
    Limits:
      cpu:     100m
      memory:  100Mi
    Requests:
      cpu:     20m
      memory:  20Mi
    Environment:
      NAMESPACE:   (v1:metadata.namespace)
    Mounts:
      /home/securecodebox/ from scan-results (ro)
  Volumes:
   scan-results:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
   efs:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  scb-scanners-pvc
    ReadOnly:   true
Events:
  Type     Reason                Age   From            Message
  ----     ------                ----  ----            -------
  Normal   SuccessfulCreate      12m   job-controller  Created pod: scan-2f1cf8e1-8bf5-4e7e-a0de-93fe6ac44005-rp8mw-5rh4z
  Normal   SuccessfulDelete      10m   job-controller  Deleted pod: scan-2f1cf8e1-8bf5-4e7e-a0de-93fe6ac44005-rp8mw-5rh4z
  Warning  BackoffLimitExceeded  10m   job-controller  Job has reached the specified backoff limit

Scan output:

Name:         2f1cf8e1-8bf5-4e7e-a0de-93fe6ac44005
Namespace:    scb-scanners
Labels:       <none>
Annotations: <none>
API Version:  execution.securecodebox.io/v1
Kind:         Scan
Metadata:
  Creation Timestamp:  2024-03-03T08:33:15Z
  Finalizers:
    s3.storage.securecodebox.io
  Generation:        2
  Resource Version:  77260138
  UID:               dcee8417-87ae-4cc7-8bc7-add3ef829d93
Spec:
  Parameters:
    detect
    --source
    /f14b0104-c583-45ae-bba6-d7eb8df767d0/48ec99a2-61f7-4a07-992b-cdb2bc7a0467/379931510/efea2f02da73197dd1f5faeb174fc6cddd36e459
    --log-opts
    efea2f02da73197dd1f5faeb174fc6cddd36e459
  Resource Mode:  namespaceLocal
  Resources:
  Scan Type:  gitleaks
  Volume Mounts:
    Mount Path:  /f14b0104-c583-45ae-bba6-d7eb8df767d0/48ec99a2-61f7-4a07-992b-cdb2bc7a0467/379931510/efea2f02da73197dd1f5faeb174fc6cddd36e459
    Name:        efs
    Read Only:   true
    Sub Path:    f14b0104-c583-45ae-bba6-d7eb8df767d0/48ec99a2-61f7-4a07-992b-cdb2bc7a0467/379931510/efea2f02da73197dd1f5faeb174fc6cddd36e459
  Volumes:
    Name:  efs
    Persistent Volume Claim:
      Claim Name:  scb-scanners-pvc
      Read Only:   true
Status:
  Finding Download Link:  ....
  Raw Result Type:           gitleaks-json
  State:                     Scanning
Events:                      <none>

[J12934]

Oh mh interessting, we'll have a look.
Thank you for the detailed report 🙌