CredScan intermittently failing for large SBOM files

[mthalman]

The Guardian: CredScan (Binary) step is having trouble with the SBOM files that are now being generated after the changes in #1787. This only happens for the very large SBOMs that get generated by the https://github.com/dotnet/dotnet-buildtools-prereqs-docker repo. And it happens intermittently, but more often than not.

------------------------------------------------------------------------------
Running Credential Scanner 2.3.12.23
------------------------------------------------------------------------------
/mnt/vss/_work/_gdn/packages/nuget/Microsoft.Security.CredScan.Client.linux-x64.2.3.12.23/lib/net6.0/CredentialScanner -I /mnt/vss/_work/1/a/sbom -f pre -e False -S /mnt/vss/_work/_gdn/packages/nuget/Microsoft.Security.CredScan.Client.linux-x64.2.3.12.23/lib/net6.0/ConfigFiles/FileSystemProvider.csk -b 7 -O /mnt/vss/_work/1/.gdn/.r/credscan/002/credscan
Start credential scanner
Initializing...
Initialization completed.
Start file scanning...
[WARNING] Failed to parse json file (/mnt/vss/_work/1/a/sbom/dotnet-buildtools_prereqs@sha256_b310f4d427a2c94ccbf1b22106a553f3a59417646eaad16e23f6bf3c72609cb7/manifest.spdx.json). Skipping advanced JsonScanner even though the file has a .json extension; Resuming full text scanning.
Unexpected end of content while loading JObject. Path 'files[1765].checksums[0]', line 47962, position 9.

[WARNING] Failed to parse json file (/mnt/vss/_work/1/a/sbom/dotnet-buildtools_prereqs@sha256_b310f4d427a2c94ccbf1b22106a553f3a59417646eaad16e23f6bf3c72609cb7/manifest.spdx.json). Skipping advanced JsonScanner even though the file has a .json extension; Resuming full text scanning.
Additional text encountered after finished reading JSON content: :. Path '', line 1, position 12.

[WARNING] Failed to parse json file (/mnt/vss/_work/1/a/sbom/dotnet-buildtools_prereqs@sha256_b310f4d427a2c94ccbf1b22106a553f3a59417646eaad16e23f6bf3c72609cb7/manifest.spdx.json). Skipping advanced JsonScanner even though the file has a .json extension; Resuming full text scanning.
Unexpected character encountered while parsing value: a. Path '', line 0, position 0.

[WARNING] Failed to parse json file (/mnt/vss/_work/1/a/sbom/dotnet-buildtools_prereqs@sha256_b310f4d427a2c94ccbf1b22106a553f3a59417646eaad16e23f6bf3c72609cb7/manifest.spdx.json). Skipping advanced JsonScanner even though the file has a .json extension; Resuming full text scanning.
JsonToken EndArray is not valid for closing JsonType None. Path '', line 2, position 7.

[WARNING] Failed to parse json file (/mnt/vss/_work/1/a/sbom/dotnet-buildtools_prereqs@sha256_b310f4d427a2c94ccbf1b22106a553f3a59417646eaad16e23f6bf3c72609cb7/manifest.spdx.json). Skipping advanced JsonScanner even though the file has a .json extension; Resuming full text scanning.
JsonToken EndArray is not valid for closing JsonType None. Path '', line 1, position 2.

[WARNING] Failed to parse json file (/mnt/vss/_work/1/a/sbom/dotnet-buildtools_prereqs@sha256_b310f4d427a2c94ccbf1b22106a553f3a59417646eaad16e23f6bf3c72609cb7/manifest.spdx.json). Skipping advanced JsonScanner even though the file has a .json extension; Resuming full text scanning.
Unexpected character encountered while parsing value: r. Path '', line 0, position 0.

[WARNING] Failed to parse json file (/mnt/vss/_work/1/a/sbom/dotnet-buildtools_prereqs@sha256_b310f4d427a2c94ccbf1b22106a553f3a59417646eaad16e23f6bf3c72609cb7/manifest.spdx.json). Skipping advanced JsonScanner even though the file has a .json extension; Resuming full text scanning.
Unexpected character encountered while parsing number: m. Path '', line 1, position 2.

[WARNING] Failed to parse json file (/mnt/vss/_work/1/a/sbom/dotnet-buildtools_prereqs@sha256_b310f4d427a2c94ccbf1b22106a553f3a59417646eaad16e23f6bf3c72609cb7/manifest.spdx.json). Skipping advanced JsonScanner even though the file has a .json extension; Resuming full text scanning.
Additional text encountered after finished reading JSON content: ,. Path '', line 1, position 0.

[WARNING] Failed to parse json file (/mnt/vss/_work/1/a/sbom/dotnet-buildtools_prereqs@sha256_b310f4d427a2c94ccbf1b22106a553f3a59417646eaad16e23f6bf3c72609cb7/manifest.spdx.json). Skipping advanced JsonScanner even though the file has a .json extension; Resuming full text scanning.
Additional text encountered after finished reading JSON content: ,. Path '', line 6, position 5.

[WARNING] Failed to parse json file (/mnt/vss/_work/1/a/sbom/dotnet-buildtools_prereqs@sha256_b310f4d427a2c94ccbf1b22106a553f3a59417646eaad16e23f6bf3c72609cb7/manifest.spdx.json). Skipping advanced JsonScanner even though the file has a .json extension; Resuming full text scanning.
Unexpected character encountered while parsing value: r. Path '', line 0, position 0.

[WARNING] Failed to parse json file (/mnt/vss/_work/1/a/sbom/dotnet-buildtools_prereqs@sha256_33984bb4dfb7a58ea7ac421aab05e734b7a86dab191f01ab79e8e1019a5271f6/manifest.spdx.json). Skipping advanced JsonScanner even though the file has a .json extension; Resuming full text scanning.
Unterminated string. Expected delimiter: ". Path 'files[1763].SPDXID', line 107793, position 58.

##[error]malloc_consolidate(): unaligned fastbin chunk detected

I've examined the JSON files and they are syntactically correct. The error indicated at the end seems to be the issue: a memory problem.

As a first step, we should compare the state of the SBOM now from what it was before the changes from #1787 to understand what difference exists now that would cause this to fail. Based on the logs, there is some big difference in what CredScanner is scanning for content because prior to the SBOM changes it was scanning about 6 MBs of content in one job and now is scanning 900+ MBs.

[mthalman]

As a first step, we should compare the state of the SBOM now from what it was before the changes from #1787 to understand what difference exists now that would cause this to fail. Based on the logs, there is some big difference in what CredScanner is scanning for content because prior to the SBOM changes it was scanning about 6 MBs of content in one job and now is scanning 900+ MBs.

It doesn't look like the old SBOMs were correct to begin with. They seem to be generated, at least partly, by scanning the contents of the host machine, not just the image. This is evidenced by references to image info files that do not exist within the image but do exist on the host.

e.g.

    {
      "fileName": "./imageInfo/linuxamd64src-azurelinux-3.0-net9.0-crossdeps-graph-image-info.json",
      "SPDXID": "SPDXRef-File--imageInfo-linuxamd64src-azurelinux-3.0-net9.0-crossdeps-graph-image-info.json-A1D5E50EE9C717CDB6E3BF9C4CDE52660FEEC8FF",
      "checksums": [
        {
          "algorithm": "SHA256",
          "checksumValue": "828b4031e3823542b8900647b475cbcadbe2d7ce9c7031a811cdf57a0b131d2a"
        },
        {
          "algorithm": "SHA1",
          "checksumValue": "a1d5e50ee9c717cdb6e3bf9c4cde52660feec8ff"
        }
      ],
      "licenseConcluded": "NOASSERTION",
      "licenseInfoInFiles": [
        "NOASSERTION"
      ],
      "copyrightText": "NOASSERTION"
    },

So it would seem that we're now generating correct, albeit large, SBOMs.

[mthalman]

Here's an example SBOM that the tool can fail on: manifest.spdx.zip

[lbussell]

Here's an example SBOM that the tool can fail on: manifest.spdx.zip

For reference, that's 1,265,840 lines of formatted JSON.

That seems excessive. I'm sure it's comprehensive, but maybe this has something to do with us making syft scan a directory instead of an image. For example, what's the diff between this and syft's scan of the image? (I mean scanning it the "right way", locally, as an image. Not necessarily however our pipeline was doing it with the manifest generator tool)

[mthalman]

Sure, I just executed it with syft <image-name> -o spdx-json=manifest.spdx.json and it also produces 1+ million line JSON output for these images.

[mthalman]

I have an ICM logged for this issue.