Scan GitHub and GitLab refs that aren't cloned by default

[rgmz]

Description:

This fixes #1588.

In my experience, this find significantly more secrets with a negligible performance impact.

The only issue is that these secrets are technically not a part of the repository, so refactoring may be necessary to indicate that a result comes from a historical PR/MR branch. It now outputs the source pull/merge request (based on git log --source), in case the commit only exists in the PR history and not the actual repo history, which can happen when PRs are squashed.

✅ Found verified result 🐷🔑
Detector Type: AWS
Decoder Type: PLAIN
Raw result: AKIAXYZDQCEN4B6JSJQI
Resource_type: Access key
Account: 534261010715
Is_canary: true
Message: This is an AWS canary token generated at canarytokens.org, and was not set off; learn more here: https://trufflesecurity.com/canaries
Arn: arn:aws:iam::534261010715:user/canarytokens.com@@88uc3ciwodujg5f18inco2yvu
Pull Request: 2387
Commit: e47780e2e4d2dbaa3d1e63bdfe1cf00eb2c5681b
Email: Ahrav Dutta <[email protected]>
File: pkg/gitparse/gitparse_test.go
Line: 715
Link: https://github.com/trufflesecurity/trufflehog/blob/e47780e2e4d2dbaa3d1e63bdfe1cf00eb2c5681b/pkg/gitparse/gitparse_test.go#L715
Repository: https://github.com/trufflesecurity/trufflehog.git
Timestamp: 2024-02-05 17:37:58 +0000

Checklist:

• Tests passing (make test-community)?
• Lint passing (make lint this requires golangci-lint)?

[CLAassistant]

All committers have signed the CLA.

[rosecodym]

This is awesome! Just to make sure I understand: This PR has two discrete changes, right? (Pulling down all the refs and printing the source ref of found secrets.)

[rgmz]

I've made some progress towards using --mirror. The GitHub source has been updated to clone and open the repo as bare, however, it seems like every Git-related source will need to be changed.

...

Additionally, getting binary files currently doesn't work. I can fix this, but I am not able to do the above.

2024-06-09T11:31:08-04:00       error   trufflehog      waiting for command failed      {"source_manager_worker_id": "jGSti", "repo": "https://github.com/trufflesecurity/trufflehog.git", "error": "error waiting for command: command=/usr/bin/git -C /tmp/trufflehog-1690212-1041037951/.git cat-file blob 2db06f05767dd8475df2f071785d9775144ee549:pkg/handlers/testdata/testdir.zip, stderr=fatal: cannot change to '/tmp/trufflehog-1690212-1041037951/.git': No such file or directory\n, commit=2db06f05767dd8475df2f071785d9775144ee549: exit status 128"}

[rosecodym]

@rgmz is it theoretically possible to split this work up into two separate changes: One that clones using --mirror and one that reports the provenance of detected secrets? I'm trying to think of ways to minimize the change risk.

[rgmz]

It is; I don't think it would accomplish much, as I'd consider the gitparse changes to be negligible compared to everything else.

[zricethezav]

Thanks for the useful information @bplaxco!

It is; I don't think it would accomplish much, as I'd consider the gitparse changes to be negligible compared to everything else.

It should be separated to keep the PR focused.

I did some toying around with getting --mirrored to work but was unable to get the shelled out git log -p cmd to generate any output... Ideally this change would just be

1. add --mirrored to git clone
2. add --no-replace-objects to git log
3. force the bare config option so we aren't looking for a .git folder

[rgmz]

I did some toying around with getting --mirrored to work but was unable to get the shelled out git log -p cmd to generate any output...

I'm not sure what you mean by this. From this PR, or your own experimentation?

Git itself is challenging because it's a mix of confusing, duplicated, and black-box logic. In particular, it seems to accept both a list of directories and repositories (sources.proto), along with separate scanRepos and scanDirs functions, but it the OSS code only accepts a single URI.

... #1918 (comment)

Any insight into this?

[zricethezav]

I'm not sure what you mean by this. From this PR, or your own experimentation?

From this PR

[rgmz]

I haven't encountered any issues, so I'm not sure what the cause could be.

If you can provide more detailed info here, or in Slack/Discord, I can try to troubleshoot.

[zricethezav]

@rgmz

~/code/trufflesecurity/trufflehog (feat/additional-refspecs) go build && ./trufflehog git https://github.com/leaktk/fake-leaks.git
🐷🔑🐷  TruffleHog. Unearth your secrets. 🐷🔑🐷

2024-06-14T10:00:57-04:00       info-0  trufflehog      prepareRepoSinceCommit  {"commit": ""}
2024-06-14T10:00:57-04:00       info-0  trufflehog      running source  {"source_manager_worker_id": "MQh7O", "with_units": true}
2024-06-14T10:00:57-04:00       info-0  trufflehog      enumerating dirs        {"source_manager_worker_id": "MQh7O", "repo": "/var/folders/qc/__3s_93j7tlg3qrj7zxsbjwr0000gn/T/trufflehog-49059-1392975446"}
2024-06-14T10:00:57-04:00       info-0  trufflehog      scanDir {"source_manager_worker_id": "MQh7O", "unit": "/var/folders/qc/__3s_93j7tlg3qrj7zxsbjwr0000gn/T/trufflehog-49059-1392975446", "unit_kind": "dir", "dir": "/var/folders/qc/__3s_93j7tlg3qrj7zxsbjwr0000gn/T/trufflehog-49059-1392975446"}
2024-06-14T10:00:57-04:00       error   trufflehog      error getting RepoFromPath      {"source_manager_worker_id": "MQh7O", "unit": "/var/folders/qc/__3s_93j7tlg3qrj7zxsbjwr0000gn/T/trufflehog-49059-1392975446", "unit_kind": "dir", "dir": "/var/folders/qc/__3s_93j7tlg3qrj7zxsbjwr0000gn/T/trufflehog-49059-1392975446", "error": "repository does not exist"}
2024-06-14T10:00:57-04:00       info-0  trufflehog      finished scanning       {"chunks": 0, "bytes": 0, "verified_secrets": 0, "unverified_secrets": 0, "scan_duration": "430.436792ms", "trufflehog_version": "dev"}

How were you testing remote repos?

[rgmz]

Ah. Only GitHub and GitLab sources work right now. I have not made the necessary changes to Git because I don't know what a "repo" is vs. a "directory", or how it's called in Enterprise vs the OSS CLI.

See #1918 (comment)

[zricethezav]

@rgmz @bplaxco alright after banging my head against this git source I think it's better if we use "remote.origin.fetch=+refs/:refs/remotes/origin/". Introducing --mirror has a large blast radius. We should just be able to append -c remote.origin.fetch=+refs/*:refs/remotes/origin/* to the clone command and call it a day.

[rgmz]

@rgmz is it theoretically possible to split this work up into two separate changes: One that clones using --mirror and one that reports the provenance of detected secrets? I'm trying to think of ways to minimize the change risk.

I've rebased this onto #2988. This now only has changes related to reporting ref provenance.