Several test failures

[musicinmybrain]

I’m trying to package release 0.9.8 for Fedora Linux, and I’m encountering several test failures, all of which I am able to reproduce in a plain virtualenv with current main, 4c2046e.

$ python3 -V
Python 3.12.2
$ python3 -m venv _e
$ . _e/bin/activate
(_e) $ pip install -e '.[tests,all]'
(_e) $ python -m pytest -v

============================================== short test summary info ===============================================
SKIPPED [1] tests/test_fsspec.py:849: the test server does not work without content-length :(
FAILED tests/test_cli.py::test_sync_cli_local_to_remote - AssertionError: assert {'/data/./bar.../foo': b'foo'} == {'/data/bar':.../foo': b'foo'}
FAILED tests/test_cli.py::test_sync_remote_to_local - AssertionError: assert {'data': {'da...foo': 'foo'}}} == {'data': {'foo': 'foo'}}
FAILED tests/test_cli.py::test_cp_cli - AssertionError: assert False
FAILED tests/test_cli.py::test_mv_cli - AssertionError: assert {'dir': {'ips...m': 'lorem'}}} == {'dir': {'ips...em': 'lorem'}}
FAILED tests/test_cli.py::test_sync_remote_to_remote - FileNotFoundError: /data/.
FAILED tests/test_cli.py::test_ls_cli - AssertionError: assert {Row(date='-'... isdir=False)} == {Row(date='-'... isdir=False)}
============================== 6 failed, 330 passed, 1 skipped, 2621 warnings in 9.60s ===============================

I will post the full output from pytest -vv for these six failures in a comment.

I haven’t tried to dig into the root causes. Is there something that needs to be updated for current dependency versions?

[musicinmybrain]

====================================================== FAILURES ======================================================
____________________________________________________ test_cp_cli _____________________________________________________
[gw9] linux -- Python 3.12.2 /home/ben/src/forks/webdav4/_e/bin/python

storage_dir = TmpDir('/tmp/pytest-of-ben/pytest-3/popen-gw9/webdav0')
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fad7e42e810>

    def test_cp_cli(storage_dir: TmpDir, monkeypatch: MonkeyPatch):
        """Test cp command."""
        memfs = MemoryFileSystem()
    
        memfs.mkdir("data1")
        memfs.pipe({"data1/foo": b"foo", "data1/bar": b"bar"})
    
        ns = Namespace(
            path1="memory://data1", path2="memory://data2", recursive=False
        )
        CommandCopy(ns, memfs).run()
    
        assert memfs.cat("data1", recursive=True, on_error="ignore") == {
            "/data1/bar": b"bar",
            "/data1/foo": b"foo",
        }
>       assert memfs.isdir("data2")
E       AssertionError: assert False
E        +  where False = <bound method AbstractFileSystem.isdir of <tests.test_cli.MemoryFileSystem object at 0x7fad7e42e7e0>>('data2')
E        +    where <bound method AbstractFileSystem.isdir of <tests.test_cli.MemoryFileSystem object at 0x7fad7e42e7e0>> = <tests.test_cli.MemoryFileSystem object at 0x7fad7e42e7e0>.isdir

tests/test_cli.py:306: AssertionError
___________________________________________ test_sync_cli_local_to_remote ____________________________________________
[gw6] linux -- Python 3.12.2 /home/ben/src/forks/webdav4/_e/bin/python

storage_dir = TmpDir('/tmp/pytest-of-ben/pytest-3/popen-gw6/webdav0')
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f5dd695a9f0>

    def test_sync_cli_local_to_remote(
        storage_dir: TmpDir, monkeypatch: MonkeyPatch
    ):
        """Test syncing between local to remote filesystem."""
        storage_dir.gen({"data": {"foo": "foo"}})
        memfs = MemoryFileSystem()
    
        monkeypatch.chdir(storage_dir)
        ns = Namespace(path1="data", path2="memory://data", delete=False)
    
        cmd = CommandSync(ns, memfs)
        cmd.run()
        assert memfs.cat("data", True, "ignore") == {"/data/foo": b"foo"}
    
        (storage_dir / "data" / "bar").write_text("bar")
        cmd.run()
>       assert memfs.cat("data", True, "ignore") == {
            "/data/foo": b"foo",
            "/data/bar": b"bar",
        }
E       AssertionError: assert {'/data/./bar': b'bar', '/data/./foo': b'foo', '/data/bar': b'bar', '/data/foo': b'foo'} == {'/data/foo': b'foo', '/data/bar': b'bar'}
E         
E         Common items:
E         {'/data/bar': b'bar', '/data/foo': b'foo'}
E         Left contains 2 more items:
E         {'/data/./bar': b'bar', '/data/./foo': b'foo'}
E         
E         Full diff:
E           {
E         +     '/data/./bar': b'bar',
E         +     '/data/./foo': b'foo',
E               '/data/bar': b'bar',
E               '/data/foo': b'foo',
E           }

/home/ben/src/forks/webdav4/tests/test_cli.py:537: AssertionError
------------------------------------------------ Captured stdout call ------------------------------------------------
copy: /tmp/pytest-of-ben/pytest-3/popen-gw6/webdav0/data to /data
copy: /tmp/pytest-of-ben/pytest-3/popen-gw6/webdav0/data/. to /data/.
copy: /tmp/pytest-of-ben/pytest-3/popen-gw6/webdav0/data/bar to /data/bar
_____________________________________________ test_sync_remote_to_local ______________________________________________
[gw6] linux -- Python 3.12.2 /home/ben/src/forks/webdav4/_e/bin/python

storage_dir = TmpDir('/tmp/pytest-of-ben/pytest-3/popen-gw6/webdav1')
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f5dd67ad2b0>

    def test_sync_remote_to_local(storage_dir: TmpDir, monkeypatch: MonkeyPatch):
        """Test syncing between remote to local filesystem."""
        memfs = MemoryFileSystem()
        memfs.mkdir("data")
        memfs.pipe({"data/foo": b"foo"})
    
        monkeypatch.chdir(storage_dir)
        ns = Namespace(path1="memory://data", path2="data", delete=False)
    
        cmd = CommandSync(ns, memfs)
        cmd.run()
>       assert storage_dir.cat() == {"data": {"foo": "foo"}}
E       AssertionError: assert {'data': {'data': {'foo': 'foo'}}} == {'data': {'foo': 'foo'}}
E         
E         Differing items:
E         {'data': {'data': {'foo': 'foo'}}} != {'data': {'foo': 'foo'}}
E         
E         Full diff:
E           {
E               'data': {
E         +         'data': {
E         -         'foo': 'foo',
E         +             'foo': 'foo',
E         ? ++++
E         +         },
E               },
E           }

/home/ben/src/forks/webdav4/tests/test_cli.py:591: AssertionError
------------------------------------------------ Captured stdout call ------------------------------------------------
copy: /data to /tmp/pytest-of-ben/pytest-3/popen-gw6/webdav1/data
____________________________________________________ test_mv_cli _____________________________________________________
[gw9] linux -- Python 3.12.2 /home/ben/src/forks/webdav4/_e/bin/python

storage_dir = TmpDir('/tmp/pytest-of-ben/pytest-3/popen-gw9/webdav1')
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fad7e42e570>

    def test_mv_cli(storage_dir: TmpDir, monkeypatch: MonkeyPatch):
        """Test mv command."""
        memfs = MemoryFileSystem()
    
        memfs.mkdir("data1")
        memfs.pipe({"data1/foo": b"foo", "data1/bar": b"bar"})
    
        ns = Namespace(
            path1="memory://data1", path2="memory://data2", recursive=True
        )
        CommandMove(ns, memfs).run()
        assert not memfs.isdir("data1")
        assert memfs.cat("data2", recursive=True, on_error="ignore") == {
            "/data2/bar": b"bar",
            "/data2/foo": b"foo",
        }
        assert memfs.isdir("data2")
    
        ns = Namespace(
            path1="memory://data2/foo",
            path2="memory://data2/foobar",
            recursive=True,
        )
        CommandMove(ns, memfs).run()
        assert memfs.cat_file("data2/foobar") == b"foo"
        assert not memfs.exists("memory://data2/foo")
    
        monkeypatch.chdir(storage_dir)
    
        d = {"lorem": "lorem", "ipsum": "ipsum"}
        storage_dir.gen({"dir": d})
        ns = Namespace(path1="dir", path2="memory://dir", recursive=True)
        CommandMove(ns, memfs).run()
        assert memfs.cat("dir", True, "ignore") == {
            "/dir/ipsum": b"ipsum",
            "/dir/lorem": b"lorem",
        }
    
        ns = Namespace(path1="memory://dir", path2="dir2", recursive=True)
        CommandMove(ns, memfs).run()
>       assert storage_dir.cat() == {"dir": d, "dir2": d}
E       AssertionError: assert {'dir': {'lorem': 'lorem', 'ipsum': 'ipsum'}, 'dir2': {'dir': {'ipsum': 'ipsum', 'lorem': 'lorem'}}} == {'dir': {'lorem': 'lorem', 'ipsum': 'ipsum'}, 'dir2': {'lorem': 'lorem', 'ipsum': 'ipsum'}}
E         
E         Common items:
E         {'dir': {'ipsum': 'ipsum', 'lorem': 'lorem'}}
E         Differing items:
E         {'dir2': {'dir': {'ipsum': 'ipsum', 'lorem': 'lorem'}}} != {'dir2': {'ipsum': 'ipsum', 'lorem': 'lorem'}}
E         
E         Full diff:
E           {
E               'dir': {
E                   'ipsum': 'ipsum',
E                   'lorem': 'lorem',
E               },
E               'dir2': {
E         +         'dir': {
E         -         'ipsum': 'ipsum',
E         +             'ipsum': 'ipsum',
E         ? ++++
E         -         'lorem': 'lorem',
E         +             'lorem': 'lorem',
E         ? ++++
E         +         },
E               },
E           }

/home/ben/src/forks/webdav4/tests/test_cli.py:427: AssertionError
____________________________________________________ test_ls_cli _____________________________________________________
[gw9] linux -- Python 3.12.2 /home/ben/src/forks/webdav4/_e/bin/python

capsys = <_pytest.capture.CaptureFixture object at 0x7fad7e499910>

    def test_ls_cli(capsys: CaptureFixture):
        """Test ls command."""
        mfs = MemoryFileSystem()
    
        mfs.mkdir("data")
        mfs.mkdir("dir")
        mfs.pipe(
            {
                "data/foo": b"foo",
                "data/bar": b"bar",
                "dir/nested/foo": b"dirfoo" * 1000,
            }
        )
    
        ns = Namespace(path="data", recursive=False, level=None, full_path=False)
>       assert set(CommandLS(ns, mfs).ls()) == {
            Row("-", Size("3"), "foo"),
            Row("-", Size("3"), "bar"),
        }
E       AssertionError: assert {Row(date='-', size=Size(nbytes='0', suff=''), file='data', isdir=True), Row(date='-', size=Size(nbytes='3', suff=''), file='foo', isdir=False), Row(date='-', size=Size(nbytes='3', suff=''), file='bar', isdir=False)} == {Row(date='-', size=Size(nbytes='3', suff=''), file='bar', isdir=False), Row(date='-', size=Size(nbytes='3', suff=''), file='foo', isdir=False)}
E         
E         Extra items in the left set:
E         Row(date='-', size=Size(nbytes='0', suff=''), file='data', isdir=True)
E         
E         Full diff:
E           {
E         +     Row(date='-', size=Size(nbytes='0', suff=''), file='data', isdir=True),
E               Row(date='-', size=Size(nbytes='3', suff=''), file='bar', isdir=False),
E               Row(date='-', size=Size(nbytes='3', suff=''), file='foo', isdir=False),
E           }

tests/test_cli.py:445: AssertionError
_____________________________________________ test_sync_remote_to_remote _____________________________________________
[gw6] linux -- Python 3.12.2 /home/ben/src/forks/webdav4/_e/bin/python

    def test_sync_remote_to_remote():
        """Test syncing between remote to remote filesystem."""
        memfs = MemoryFileSystem()
    
        memfs.mkdir("data")
        memfs.pipe({"data/foo": b"foo"})
    
        ns = Namespace(path1="memory://data", path2="memory://data2", delete=False)
    
        cmd = CommandSync(ns, memfs)
        cmd.run()
        assert memfs.cat("data2", True, "ignore") == {"/data2/foo": b"foo"}
    
        memfs.pipe({"data/bar": b"bar"})
>       cmd.run()

tests/test_cli.py:635: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/webdav4/cli.py:906: in run
    return self.sync(src, dest, src_fs, dest_fs)
src/webdav4/cli.py:879: in sync
    self.sync(new_src, new_dest, src_fs, dest_fs)
src/webdav4/cli.py:848: in sync
    details_src = src_fs.info(src)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tests.test_cli.MemoryFileSystem object at 0x7f5dd67addc0>, path = '/data/.', kwargs = {}

    def info(self, path, **kwargs):
        path = self._strip_protocol(path)
        if path in self.pseudo_dirs or any(
            p.startswith(path + "/") for p in list(self.store) + self.pseudo_dirs
        ):
            return {
                "name": path,
                "size": 0,
                "type": "directory",
            }
        elif path in self.store:
            filelike = self.store[path]
            return {
                "name": path,
                "size": filelike.size,
                "type": "file",
                "created": getattr(filelike, "created", None),
            }
        else:
>           raise FileNotFoundError(path)
E           FileNotFoundError: /data/.

_e/lib64/python3.12/site-packages/fsspec/implementations/memory.py:159: FileNotFoundError
------------------------------------------------ Captured stdout call ------------------------------------------------
copy: /data to /data2

[skshetry]

There were some changes in MemoryFileSystem in fsspec that are breaking cli tests.

I'll try to look at it over the weekend. Thank you for creating this issue.

[musicinmybrain]

Thanks! Please let me know if there’s anything I can do to help.

[musicinmybrain]

I was looking back over the Snakemake 8 plugins that I haven’t yet packaged for Fedora, and noticed I still need webdav4 in order to package https://github.com/snakemake/snakemake-storage-plugin-webdav.

Did you happen to have any luck looking into the fsspec changes?

I suppose that I could package the webdav4 library without the dav command-line tool for now, since it looks like any issues are contained to the CLI…