Can't write xml files via "lxml" package when using pyfakefs

[buhtz]

I'm really sorry for such a broad and unspecific bug report. But I could break it down to the fact that the problem occurs only when I use pyfakefs but if I use a real fileystem for (nearly) the same test everything is fine.

Do you have any idea what could cause this side effect or how I could go on with my investigation?

Maybe pyfakefs doesn't write a real file to the fake filesystem? Can I checkt that somehow?

The last lines of the raised error

The raised errors seems to have nothing to do with pyfakefs or my own package.

  File "/usr/lib/python3/dist-packages/openpyxl/reader/excel.py", line 219, in read_worksheets
    ws = ReadOnlyWorksheet(self.wb, sheet.name, rel.target, self.shared_strings)
  File "/usr/lib/python3/dist-packages/openpyxl/worksheet/_read_only.py", line 40, in __init__
    self._get_size()
  File "/usr/lib/python3/dist-packages/openpyxl/worksheet/_read_only.py", line 46, in _get_size
    dimensions = parser.parse_dimensions()
  File "/usr/lib/python3/dist-packages/openpyxl/worksheet/_reader.py", line 164, in parse_dimensions
    for _event, element in it:
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1260, in iterator
    root = pullparser._close_and_return_root()
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1307, in _close_and_return_root
    root = self._parser.close()
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1729, in close
    self._raiseerror(v)
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1629, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: no element found: line 1, column 0

----------------------------------------------------------------------
Ran 1 test in 0.691s

FAILED (errors=1)

Description

The unittest checks if a excel file can be read. The test does this steps

1. Create a pandas.DataFrame.
2. Store it as an excel file (via pandas.DataFrame.to_excel())
3. Read the excel file into (via pandas.read_excel()).
4. Compare the initial and the returned data frame.

Of course in the real tests there happens a lot more between 2. and 3. I have a wrapper around pandas.DataFrame.to_excel().

The unittests

import unittest
import pandas
import pyfakefs.fake_filesystem_unittest as pyfakefs_ut

class Works(unittest.TestCase):

    def test_simple(self):
        """Simple excel."""
        excel_path = pathlib.Path('foobar.xlsx')
        if excel_path.exists():
            excel_path.unlink()

        df_init = pandas.DataFrame({'FOO': range(3), 'BAR': list('ABC')})
        df_init.to_excel(excel_path)

        self.assertTrue(excel_path.exists())

        df = pandas.read_excel(excel_path)

        self.assertEqual(df.shape, (3, 3))


class Problem(pyfakefs_ut.TestCase):

    def setUp(self):
        self.setUpPyfakefs(allow_root_user=False)

    def test_simple(self):
        excel_path = pathlib.Path('foobar.xlsx')
        df_init = pandas.DataFrame({'FOO': range(3), 'BAR': list('ABC')})
        df_init.to_excel(excel_path)

        self.assertTrue(excel_path.exists())

        # HERE comes the ERROR
        df = pandas.read_excel(excel_path)

        self.assertEqual(df.shape, (3, 3))

Environment

In the beginning this problem occur in older versions with Pandas (1.3.5), Numpy and openpyxl. Just for that bug report I updated everything possible (except my operating system and the python interpreter) to the current available stable release version. But the error is still there.

• Debian 11 (arm)
• Python 3.9.2 (via debian repo)
• Pandas 1.4.4 (via pip)
• Numpy 1.19.5 (via pip)
• openpyxl 3.0.10 (via pip)

Full error output

python3 -m unittest tests.test_bandas.Problem
E
======================================================================
ERROR: test_simple (tests.test_bandas.Problem)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1727, in close
    self.parser.Parse(b"", True) # end of data
xml.parsers.expat.ExpatError: no element found: line 1, column 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/ownCloud/my.work/buhtzology/tests/test_bandas.py", line 1139, in test_simple
    df = pandas.read_excel(excel_path)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/excel/_base.py", line 457, in read_excel
    io = ExcelFile(io, storage_options=storage_options, engine=engine)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/excel/_base.py", line 1419, in __init__
    self._reader = self._engines[engine](self._io, storage_options=storage_options)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/excel/_openpyxl.py", line 525, in __init__
    super().__init__(filepath_or_buffer, storage_options=storage_options)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/excel/_base.py", line 518, in __init__
    self.book = self.load_workbook(self.handles.handle)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/excel/_openpyxl.py", line 536, in load_workbook
    return load_workbook(
  File "/home/user/.local/lib/python3.9/site-packages/openpyxl/reader/excel.py", line 317, in load_workbook
    reader.read()
  File "/home/user/.local/lib/python3.9/site-packages/openpyxl/reader/excel.py", line 282, in read
    self.read_worksheets()
  File "/home/user/.local/lib/python3.9/site-packages/openpyxl/reader/excel.py", line 219, in read_worksheets
    ws = ReadOnlyWorksheet(self.wb, sheet.name, rel.target, self.shared_strings)
  File "/home/user/.local/lib/python3.9/site-packages/openpyxl/worksheet/_read_only.py", line 41, in __init__
    self._get_size()
  File "/home/user/.local/lib/python3.9/site-packages/openpyxl/worksheet/_read_only.py", line 47, in _get_size
    dimensions = parser.parse_dimensions()
  File "/home/user/.local/lib/python3.9/site-packages/openpyxl/worksheet/_reader.py", line 166, in parse_dimensions
    for _event, element in it:
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1260, in iterator
    root = pullparser._close_and_return_root()
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1307, in _close_and_return_root
    root = self._parser.close()
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1729, in close
    self._raiseerror(v)
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1629, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: no element found: line 1, column 0

----------------------------------------------------------------------
Ran 1 test in 0.699s

FAILED (errors=1)

Misc

Referenced by https://codeberg.org/buhtz/buhtzology/issues/28

[mrbean-bremen]

Thanks for the report! I just had a quick local test under Windows, and the problem did not occur, so I have to make further tests under Linux. I don't have the time today (and probably not in the next few days), but I will see what I can find - right now I have no idea. From the callstack it looks like the excel file had been written in the fake fs, but the contents are incorrect. I will let you know if I find the problem, and hopefully a solution.

[mrbean-bremen]

In your related issue you write that this only happens on an ARM system. I have no possibility to test under such a system, so this may be a bit tricky...

[buhtz]

Sidenote: Different from my original report I updated Numpy to 1.23.2 which is the current stable. But nothing changed.

This becomes more wired now.

1. I'm also not able to reproduce the problem on my usual home PC running with Debian 11 (stable, amd64).
2. But I'm able to reproduce this on a RaspberyPi4 with Debian 11 (stable; ARM (arm64)).

I'm sorry. ;)

This seems not to be urgent. I assume the cause isn't pyfakefs. It must be something behind. I have to think about further tests, maybe with xmltree or openpyxl itself.

[mrbean-bremen]

Ok, thanks - in this case, I'll probably leave this alone for the time being, maybe you will find something out.

[buhtz]

Can you please have a short look on this "class".
https://github.com/lxml/lxml/blob/1e3666018329cadf8e147607824614aebf7e2099/src/lxml/serializer.pxi#L1294

My expertise end here. The file suffix is pxi and there are un-pythonic keywords like cdef. But beside of it it seems that this class tries to write files "asynchrone".

Does this give you any idea about a possible problem?

Now I have better and easier code to reproduce but I have to test it on my home machine this night first.

[mrbean-bremen]

What you are showing is lxml, a Python libray written in C, which is not compatible with pyfakefs. The keyword cdef means a function that can only be called from C code and is not visible to Python code.
I haven't seen that lxml is used in your code (xml.etree is a standard Python module and should not be a problem, and has almost the same interface as lxml), but maybe there is some OS- or hardware-specific optimization that uses it.

If lxml is used, it has to be patched away if possible, for example by replacing it with xml.etree.

[buhtz]

I haven't seen that lxml is used in your code
You are right. Iinitialy it was pandas which use openpyxl to create Excel files. And openpyxl is using lxml. ;)

[mrbean-bremen]

Hm, this still doesn't explain why it works on most systems...
The CI tests use openpyxl 3.0.10 without problems (and we don't patch it), but these tests all run under amd64 hardware (with different Linux distributions), so no surprise here.

[buhtz]

Now I can reproduce it with my home PC (amd64) and with the Pi4 (arm64).

Using lxml.etree to create and write an XML file doesn't result in a file when using a pyfakefs filesystem.

The formally used openpyxl seems to use different xml libraries if needed. See here or here. That would explain why I couldn't reproduce the problem on both machines before.

Now the interesting question is if you can reproduce it either. ;)

Code to reproduce

import unittest
import pathlib
import lxml
import lxml.etree
import pyfakefs
import pyfakefs.fake_filesystem_unittest as pyfakefs_ut


def create_file(path):
    create_xml_file(path)


def create_xml_file(path):
    with lxml.etree.xmlfile(str(path), encoding='utf-8', close=True) as xf:
        with xf.element('root'):
            xf.write(lxml.etree.Element('test'))


def read_file(path):
    read_file_xml(path)


def read_file_xml(path):
    with path.open('r') as handle:
        print(handle.read())


class Problem(pyfakefs_ut.TestCase):
    """See https://github.com/jmcgeheeiv/pyfakefs/issues/713
    """

    def setUp(self):
        self.setUpPyfakefs(allow_root_user=False)

    def test_simple(self):
        path = pathlib.Path('foobar.xlsx')
        create_file(path)

        read_file(path)

class Works(unittest.TestCase):

    def test_simple(self):
        path = pathlib.Path('foobar.xlsx')
        create_file(path)

        read_file(path)

        path.unlink()

Output of test "Works"

<root><test/></root>

Output of test "Problem"

Traceback (most recent call last):
  File "/home/user/ownCloud/my.work/pyfakefs_problem/foo.py", line 39, in test_simple
    read_file(path)
  File "/home/user/ownCloud/my.work/pyfakefs_problem/foo.py", line 20, in read_file
    read_file_xml(path)
  File "/home/user/ownCloud/my.work/pyfakefs_problem/foo.py", line 24, in read_file_xml
    with path.open('r') as handle:
  File "/home/user/.local/lib/python3.9/site-packages/pyfakefs/fake_pathlib.py", line 571, in open
    return FakeFileOpen(self.filesystem)(
  File "/home/user/.local/lib/python3.9/site-packages/pyfakefs/fake_filesystem.py", line 5721, in __call__
    return self.call(*args, **kwargs)
  File "/home/user/.local/lib/python3.9/site-packages/pyfakefs/fake_filesystem.py", line 5806, in call
    file_object = self._init_file_object(file_object,
  File "/home/user/.local/lib/python3.9/site-packages/pyfakefs/fake_filesystem.py", line 5885, in _init_file_object
    self.filesystem.raise_os_error(errno.ENOENT, file_path)
  File "/home/user/.local/lib/python3.9/site-packages/pyfakefs/fake_filesystem.py", line 1122, in raise_os_error
    raise OSError(err_no, message, filename)
FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: 'foobar.xlsx'

[mrbean-bremen]

Yes, I can reproduce this. As I wrote, lxml as a C module using C functions for file system access is not compatible with pyfakefs. This (incompatibility with Python modules using C libraries for fs access) is actually the biggest limitation of pyfakefs. From the documentation:

A module may not work with pyfakefs because of one of the following reasons:
...

• It uses C libraries to access the file system. There is no way no make such a module work with pyfakefs–if you want to use it, you have to patch the whole module. In some cases, a library implemented in Python with a similar interface already exists. An example is lxml, which can be substituted with ElementTree in most cases for testing.

[mrbean-bremen]

See also #180.

[mrbean-bremen]

Unfortunately, xml.etree.ElementTree does not have xmlfile, so substituing it won't work in your case.

[mrbean-bremen]

Ah, I missed your link to the openpyxl code that shows that it checks for availability of lxml. This is actually something we might be able to patch - just let openpyxl think that lxml is not available, so that it will use xml instead. I didn't have the time for a closer look yet (probably at the weekend), but your findings are really helpful!

[buhtz]

Thanks a lot for explaining.

I'm not sure if pyfakefs should patch something like this. Isn't openpyxl/lxml an edge case? As a user I wouldn't expect that magic from pyfakefs. IMHO this is my own responsibility to handle that. But before that the user should know the reason.

What would help more is if pyfakefs would be able to detect a situation (C library using python package) like this and then give a warning about that.

I'm not good at this. But isn't it possible to look (e.g. via sys or inspect) which packages are loaded? And then pyfakefs should check that list with a "bad-list" and give a warning about it. Or you can inspect if a package use a C library or not.

[mrbean-bremen]

I'm not sure if pyfakefs should patch something like this.

In principle you are right, but for some often-used cases it may make sense to do the patching in pyfakefs. This can also be switched off if not wanted. pandas is such a case, and openpyxl is one the libraries that can be used by it. We already patch xlrd, which can be used instead of openpyxl.

But isn't it possible to look (e.g. via sys or inspect) which packages are loaded? And then pyfakefs should check that list with a "bad-list" and give a warning about it. Or you can inspect if a package use a C library or not.

Most packages that use C libraries are not problematic as they don't use fs functions (NumPy for example), and even if a loaded package can use file system fumctions, it may not use them in the tested code. So I would rather not try this...

[buhtz]

When there is so much patching I wonder if my tests are still "real".

Technically your fake-fs is in RAM, right? Or more precise: In the "RAM scope" of the current running Python interpreter?

[mrbean-bremen]

When there is so much patching I wonder if my tests are still "real".

Well, if are going to patch the file system access, this is somewhat unavoidable. As I wrote, patching the fs access of some common libraries is a convenience, as most users won't have the time or patience to do it themselves, but this is currently only done for pandas and the related xlrd, and django.

In the case of pandas, xlrd (and probably openpyxl) the patch just ensures that the library uses Python for filesystem access, which it would do anyway in some environments. This way, you won't catch some potential bugs in pandas or related libraries implementation that would only occur if using the C libraries, but that shouldn't be the scope of tests. If you need completely realistic tests, you have to check in the real fs, a RAM disk could help to that faster.

Technically your fake-fs is in RAM, right?

Yes, it is.

[buhtz]

Thanks a lot for spending time onto this.

[mrbean-bremen]

I had a look at the source code of openpyxl (where your link pointed), and it turns out that lxml is not used if the environment variable OPENPYXL_LXML is set to anything but True. This is also an easy way to fix this in your tests.

[buhtz]

I'm not able to reproduce this on my PC when I use openpyxl no matter if lxml is installed or not.
I am sure I misunderstood something.

The current code I'm using.

import unittest
import pathlib
import openpyxl
import pyfakefs
import pyfakefs.fake_filesystem_unittest as pyfakefs_ut


class Problem(pyfakefs_ut.TestCase):
    """See https://github.com/jmcgeheeiv/pyfakefs/issues/713
    """

    def setUp(self):
        self.setUpPyfakefs(allow_root_user=False)

        try:
            import lxml
            print('I can import "lxml".')
        except ImportError:
            print('No "lxml" importable.')

    def test_pandas(self):
        excel_path = pathlib.Path('foobar.xlsx')
        
        wb = openpyxl.Workbook()
        s = wb.create_sheet('foobar')
        s['A1'] = 'foobar'
        wb.save('foobar.xlsx')

        self.assertTrue(excel_path.exists())

I also added some debug-print-statements in the openpyxl code. I'm pretty sure when openpyxl does use Pythons inbuild xml.etree or lxml. When using lxml (installed via pip) the error does not occur. But it should?

[mrbean-bremen]

Well, it depends what calls are used in the code. In this example, the lxml filesystem functions are either not used, or used in a way that are not problematic. The problem seems to occur in reading the excel file, not in writing it (as shown in your other example).
I can reproduce the error with your first example (where it happens on reading the file), and not on your second, so this is consistent. I can prevent the error in the first example by setting OPENPYXL_LXML=False before running the tests.

[buhtz]

With writing I can reproduce the problem. Sorry, my mistake.

How do you set OPENPYXL_LXML=False?

OPENPYXL_LXML=False && python3 -m unittest foo

This doesn't work in my system. The goal is to prevent openpyxl to import lxml even if it is installed, right?

[mrbean-bremen]

You have to set the environment variable, probably export OPENPYXL_LXML=False, depending on your system.
And yes, this is the goal.

[buhtz]

Sorry, now it works. But this is no solution for unittesting.

So I did it in the code like this

import os
os.environ['OPENPYXL_LXML'] = 'False'
import openpyxl

It seems to me that I have to do this before importing openpyxl. That break PEP8 rules in the code. I opened an Issue at openpyxl upstream.

[mrbean-bremen]

Yeah, not the best solution, but at least a workaround. I haven't decided yet if and how to handle this in pyfakefs, didn't find a good solution yet.

[mrbean-bremen]

For your information: I have played around with patching openpyxl in pyfakefs, but it turned out to be a bit ugly, as there is a lot of internal modules that have to be patched or reloaded, so I don't feel that this is something we should do. It would be too fragile concerning any changes in the implementation, or any side effects.
Instead, I will mention the workaround (setting the environment variable before the test) in the documentation. I think that this is still the best way to handle this.

[mrbean-bremen]

I'm moving this to discussions, as it may be helpful to others.