Failing to filtering out symlinks (using the clean-ignore example)

[PMPP]

Hi @newren ! Great work, I came across your tool, just after ignoring the git's suggestion to use, and after spending over 10hours waiting for filter-branch to finish what I wanted it to do, here I am!

What I am trying to achieve:

• Big monorepo
• Extract series of folders and root files into a new repo, keep its history

From:

/folder-1/folder-a
/folder-1/folder-c
/folder-2/folder-aa
/folder-2/folder-cc
file-a

To:

/folder-1/folder-a
/folder-2/folder-aa
file-a

I know the current repo used to have a folder-b and folder-bb inside the folders that I will keep, but I don't want to bring them along in the history.

with filter-branch I just git removed everything I didn't need then git ls-files > keep-these-ones , and tried to remove anything that was not in that list. For my case this gives me enough history, to make sense of past choices.

The example for clean-ignore roughly fits my case, or I wanted to believe it, so I could move on with my day.
.gitignore all the folders I don't want, add a couple of old folders that I know they used to exist, and we removed them and just run clean-ignore.

curl 'https://raw.githubusercontent.com/newren/git-filter-repo/main/git-filter-repo' > git-filter-repo
cp git-filter-repo git_filter_repo.py
curl 'https://raw.githubusercontent.com/newren/git-filter-repo/refs/heads/main/contrib/filter-repo-demos/clean-ignore' > clean-ignore
python3 clean-ignore --force

The issue I am facing, "seems" to happen due to symlinks that exist in the repo, where perhaps clean-ignore attempts to remove the symlink, but the target of the symlink is something that is not being .gitignore, resulting in:

Parsed 1655 commitsfatal: pathspec 'frontend/some-app/app/Resources/translations/messages.ms.po' is beyond a symbolic link
Traceback (most recent call last):
  File "/home/pmpp/please-god-make-it-work/my-repo/clean-ignore", line 81, in <module>
    main()
  File "/home/pmpp/please-god-make-it-work/my-repo/clean-ignore", line 77, in main
    filter.run()
  File "/home/pmpp/please-god-make-it-work/my-repo/git_filter_repo.py", line 4892, in run
    self._parser.run(self._input, self._output)
  File "/home/pmpp/please-god-make-it-work/my-repo/git_filter_repo.py", line 1527, in run
    self._parse_commit()
  File "/home/pmpp/please-god-make-it-work/my-repo/git_filter_repo.py", line 1378, in _parse_commit
    self._commit_callback(commit, aux_info)
  File "/home/pmpp/please-god-make-it-work/my-repo/git_filter_repo.py", line 4108, in _tweak_commit
    self._commit_callback(commit, self.callback_metadata(aux_info))
  File "/home/pmpp/please-god-make-it-work/my-repo/clean-ignore", line 68, in skip_ignores
    bad = self.get_ignored(filenames)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pmpp/please-god-make-it-work/my-repo/clean-ignore", line 52, in get_ignored
    (source, linenum, pattern, pathname) = result.split(b"\0")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 4, got 1)
fatal: stream ends early
fast-import: dumping crash report to .git/fast_import_crash_84569

And the dump cat .git/fast_import_crash_84569:

fast-import crash report:
    fast-import process: 84569
    parent process     : 84561
    at 2025-01-29 03:55:56 +0000

fatal: stream ends early

Most Recent Commands Before Crash
---------------------------------

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000

  reset refs/tags/v1.1.4
  from 0000000000000000000000000000000000000000
*

Active Branch LRU
-----------------
    active_branches = 1 cur, 5 max

  pos  clock name
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   1)      0 refs/tags/v1.1.4

Inactive Branches
-----------------
refs/tags/v1.1.4:
  status      : active dirty
  tip commit  : 0000000000000000000000000000000000000000
  old tree    : 0000000000000000000000000000000000000000
  cur tree    : 0000000000000000000000000000000000000000
  commit clock: 0
  last pack   :


Marks
-----
:1 1a8f11f2d9485a41810124eb52b2f6d730667b0a

-------------------
END OF CRASH REPORT

I didn't manage to find any issues related to symlinks, nor reference in the docs to address them, so here to inform this happens (perhaps its expected and I don't know better) and ... always open to suggestions if there is a better way to achieve my goal :)

Thank you for giving such a great tool to the community!

[newren]

Let's say we wanted to test whether the path "foo/bar" was ignored:

$ echo "foo/bar" | git check-ignore --verbose --stdin --non-match
::	foo/bar

Fair enough. It's not ignored at least in the current repository I was checking from. What if I create foo as a symlink, though?

ln -s /dev/null foo
$ echo "foo/bar" | git check-ignore --verbose --stdin --non-match
fatal: pathspec 'foo/bar' is beyond a symbolic link

So, git check-ignore, upon which git-filter-repo's clean-ignore script is based, will die on us if any leading portion of any paths that we attempt to check happen to be symlinks, regardless of what they were in history. That's not very helpful for our purposes, but git check-ignore does not provide a way to override this.

Out of curiosity, if you run

find . -type l | xargs rm

and then run clean-ignore, does everything work for you? (I think filter-repo's embedded git reset --hard at the end of the run should restore the symlinks for you.)

[newren]

Oh, and I love your directory name under which you place your repositories. Very fun. :-)