Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support scanning bare repos #12

Open
tarkatronic opened this issue Nov 6, 2019 · 2 comments
Open

Support scanning bare repos #12

tarkatronic opened this issue Nov 6, 2019 · 2 comments
Labels
enhancement New feature or request Hacktoberfest help wanted Extra attention is needed
Milestone

Comments

@tarkatronic
Copy link
Contributor

Currently, attempting to scan a bare git repo, using --repo_path, produces an error along the lines of:

Traceback (most recent call last):
  File "/home/jwilhelm/Documents/workspace/tartufo/.venv/bin/tartufo", line 11, in <module>
    load_entry_point('tartufo', 'console_scripts', 'tartufo')()
  File "/home/jwilhelm/Documents/workspace/tartufo/tartufo/cli.py", line 58, in main
    path_exclusions=path_exclusions,
  File "/home/jwilhelm/Documents/workspace/tartufo/tartufo/scanner.py", line 287, in find_strings
    for curr_commit in repo.iter_commits(branch_name, max_count=max_depth):
  File "/home/jwilhelm/Documents/workspace/tartufo/.venv/lib64/python3.7/site-packages/git/objects/commit.py", line 278, in _iter_from_process_or_stream
    finalize_process(proc_or_stream)
  File "/home/jwilhelm/Documents/workspace/tartufo/.venv/lib64/python3.7/site-packages/git/util.py", line 332, in finalize_process
    proc.wait(**kwargs)
  File "/home/jwilhelm/Documents/workspace/tartufo/.venv/lib64/python3.7/site-packages/git/cmd.py", line 414, in wait
    raise GitCommandError(self.args, status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git rev-list --max-count=1000000 1/head --
  stderr: 'fatal: bad revision '1/head'

This is because a bare repo is a wholly different structure from a normal clone, and produces different results from git operations. The specific problem causing the error here is this:

>>> repo = git.Repo('tartufo.git')
>>> for branch in repo.remotes.origin.fetch():
...     print(branch.name)
... 
master
split_tests
1/head
10/head
11/head
16/head
19/head
2/head
20/head
21/head
22/head
23/head
24/head
25/head
3/head
8/head
9/head
v0.0.1
v0.0.2
>>> repo = git.Repo('../tartufo')
>>> for branch in repo.remotes.origin.fetch():
...     print(branch.name)
... 
origin/master
origin/split_tests
>>>

All of the X/head references are not actual valid git revisions, and so tartufo chokes on them.

We should figure out a way to either scan all refs, or only scan actual branches.

@tarkatronic tarkatronic transferred this issue from another repository Nov 7, 2019
@tarkatronic tarkatronic added enhancement New feature or request help wanted Extra attention is needed labels Nov 7, 2019
@tarkatronic tarkatronic added this to the Version 2.1 milestone Sep 8, 2020
@sjacoby-godaddy
Copy link
Contributor

Just a note, processing --mirror repos seems to work
ex:

git clone --mirror [email protected]:username/myrepo.git
tartufo --regex --json --cleanup myrepo.git

@jolinger-godaddy
Copy link

Just to note @sjacoby-godaddy suggestion did not work in my case and this still fails even on mirror repos.

git clone --mirror [email protected]:org/repo.git

tartufo --regex --json scan-local-repo ${GITHUBREPO}.git | \    
          jq -r '.found_issues[].matched_string' | \
          sort -u > ${GITHUBREPO}-remove.txt

Fatal error

Traceback (most recent call last):
  File "/usr/local/bin/tartufo", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/tartufo/commands/scan_local_repo.py", line 39, in main
    issues = scanner.scan()
  File "/usr/local/lib/python3.8/site-packages/tartufo/scanner.py", line 258, in scan
    for chunk in self.chunks:
  File "/usr/local/lib/python3.8/site-packages/tartufo/scanner.py", line 477, in chunks
    for curr_commit, prev_commit in self._iter_branch_commits(
  File "/usr/local/lib/python3.8/site-packages/tartufo/scanner.py", line 438, in _iter_branch_commits
    for curr_commit in repo.iter_commits(
  File "/usr/local/lib/python3.8/site-packages/git/objects/commit.py", line 277, in _iter_from_process_or_stream
    finalize_process(proc_or_stream)
  File "/usr/local/lib/python3.8/site-packages/git/util.py", line 329, in finalize_process
    proc.wait(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/git/cmd.py", line 408, in wait
    raise GitCommandError(self.args, status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git rev-list --max-count=1000000 1/head --
  stderr: 'fatal: bad revision '1/head'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Hacktoberfest help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

3 participants