Use of `MmapedDict` results in fd leaks (resource leaks)

[mgorny]

When running the test suite with -Wdefault, I get multiple reports of resource leaks, plus one of the tests is flaky and sometimes fails because of ResourceWarning being emitted where UserWarning was expected first:

$ python -m pytest -Wdefault
========================================================= test session starts =========================================================
platform linux -- Python 3.11.8, pytest-8.0.0, pluggy-1.4.0
rootdir: /tmp/client_python
collected 309 items                                                                                                                   

tests/openmetrics/test_exposition.py ....................                                                                       [  6%]
tests/openmetrics/test_parser.py .............................................                                                  [ 21%]
tests/test_asgi.py ssssssss                                                                                                     [ 23%]
tests/test_core.py ....................................................................................................         [ 55%]
tests/test_exposition.py ..........................................................                                             [ 74%]
tests/test_gc_collector.py ..                                                                                                   [ 75%]
tests/test_graphite_bridge.py .......                                                                                           [ 77%]
tests/test_multiprocess.py .....................F.......                                                                        [ 87%]
tests/test_parser.py ........................                                                                                   [ 94%]
tests/test_platform_collector.py ..                                                                                             [ 95%]
tests/test_process_collector.py ....                                                                                            [ 96%]
tests/test_samples.py ..                                                                                                        [ 97%]
tests/test_twisted.py s                                                                                                         [ 97%]
tests/test_wsgi.py .......                                                                                                      [100%]

============================================================== FAILURES ===============================================================
_____________________________________________ TestMultiProcess.test_remove_clear_warning ______________________________________________

self = <tests.test_multiprocess.TestMultiProcess testMethod=test_remove_clear_warning>

    def test_remove_clear_warning(self):
        os.environ['PROMETHEUS_MULTIPROC_DIR'] = self.tempdir
        with warnings.catch_warnings(record=True) as w:
            values.ValueClass = get_value_class()
            registry = CollectorRegistry()
            collector = MultiProcessCollector(registry)
            counter = Counter('c', 'help', labelnames=['label'], registry=None)
            counter.labels('label').inc()
            counter.remove('label')
            counter.clear()
            assert os.environ['PROMETHEUS_MULTIPROC_DIR'] == self.tempdir
>           assert issubclass(w[0].category, UserWarning)
E           AssertionError: assert False
E            +  where False = issubclass(<class 'ResourceWarning'>, UserWarning)
E            +    where <class 'ResourceWarning'> = <warnings.WarningMessage object at 0x7f22956b82d0>.category

tests/test_multiprocess.py:395: AssertionError
========================================================== warnings summary ===========================================================
tests/test_exposition.py::TestPushGateway::test_push_with_tls_auth_handler
  /usr/lib/python3.11/urllib/parse.py:480: ResourceWarning: unclosed <socket.socket fd=37, family=2, type=1, proto=0, laddr=('127.0.0.1', 40145)>
    for b in _UNSAFE_URL_BYTES_TO_REMOVE:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_exposition.py::TestPushGateway::test_push_with_tls_auth_handler
  /usr/lib/python3.11/urllib/parse.py:480: ResourceWarning: unclosed <socket.socket fd=38, family=2, type=1, proto=0, laddr=('127.0.0.1', 46167)>
    for b in _UNSAFE_URL_BYTES_TO_REMOVE:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_collect
  <frozen os>:676: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmp9on9bz25/counter_1334413.db' mode='ab+' closefd=True>
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_all
  /usr/lib/python3.11/glob.py:176: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpp8qzgxdb/counter_1.db' mode='ab+' closefd=True>
    with contextlib.closing(_iterdir(dirname, dir_fd, dironly)) as it:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_all
  /usr/lib/python3.11/glob.py:176: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpp8qzgxdb/gauge_all_1.db' mode='ab+' closefd=True>
    with contextlib.closing(_iterdir(dirname, dir_fd, dironly)) as it:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_all
  /usr/lib/python3.11/glob.py:176: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpp8qzgxdb/histogram_1.db' mode='ab+' closefd=True>
    with contextlib.closing(_iterdir(dirname, dir_fd, dironly)) as it:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_all
  /usr/lib/python3.11/glob.py:176: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpac3ow5d8/counter_1.db' mode='ab+' closefd=True>
    with contextlib.closing(_iterdir(dirname, dir_fd, dironly)) as it:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_all
  /usr/lib/python3.11/glob.py:176: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpac3ow5d8/gauge_all_1.db' mode='ab+' closefd=True>
    with contextlib.closing(_iterdir(dirname, dir_fd, dironly)) as it:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_all
  /usr/lib/python3.11/glob.py:176: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpac3ow5d8/histogram_1.db' mode='ab+' closefd=True>
    with contextlib.closing(_iterdir(dirname, dir_fd, dironly)) as it:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_all
  /usr/lib/python3.11/glob.py:176: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpx3sjgfk6/counter_1.db' mode='ab+' closefd=True>
    with contextlib.closing(_iterdir(dirname, dir_fd, dironly)) as it:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_all
  /usr/lib/python3.11/glob.py:176: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpq0v2xj70/counter_123.db' mode='ab+' closefd=True>
    with contextlib.closing(_iterdir(dirname, dir_fd, dironly)) as it:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_all
  /usr/lib/python3.11/glob.py:176: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpq0v2xj70/counter_456.db' mode='ab+' closefd=True>
    with contextlib.closing(_iterdir(dirname, dir_fd, dironly)) as it:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_mostrecent
  /tmp/client_python/prometheus_client/mmap_dict.py:86: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpoa1u3zei/gauge_liveall_123.db' mode='ab+' closefd=True>
    with open(filename, 'rb') as infp:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_mostrecent
  /tmp/client_python/prometheus_client/mmap_dict.py:86: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpoa1u3zei/gauge_liveall_456.db' mode='ab+' closefd=True>
    with open(filename, 'rb') as infp:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_mostrecent
  /tmp/client_python/prometheus_client/mmap_dict.py:86: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpkwjjpl18/gauge_livemax_123.db' mode='ab+' closefd=True>
    with open(filename, 'rb') as infp:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_mostrecent
  /tmp/client_python/prometheus_client/mmap_dict.py:86: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpkwjjpl18/gauge_livemax_456.db' mode='ab+' closefd=True>
    with open(filename, 'rb') as infp:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_mostrecent
  /tmp/client_python/prometheus_client/mmap_dict.py:86: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmp02r2ouao/gauge_livemin_123.db' mode='ab+' closefd=True>
    with open(filename, 'rb') as infp:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_mostrecent
  /tmp/client_python/prometheus_client/mmap_dict.py:86: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmp02r2ouao/gauge_livemin_456.db' mode='ab+' closefd=True>
    with open(filename, 'rb') as infp:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_mostrecent
  /tmp/client_python/prometheus_client/mmap_dict.py:86: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpryrzvlpo/gauge_livemostrecent_123.db' mode='ab+' closefd=True>
    with open(filename, 'rb') as infp:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_mostrecent
  /tmp/client_python/prometheus_client/mmap_dict.py:86: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpryrzvlpo/gauge_livemostrecent_456.db' mode='ab+' closefd=True>
    with open(filename, 'rb') as infp:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_mostrecent
  /tmp/client_python/prometheus_client/mmap_dict.py:86: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpribv4xj9/gauge_livesum_123.db' mode='ab+' closefd=True>
    with open(filename, 'rb') as infp:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_mostrecent
  /tmp/client_python/prometheus_client/mmap_dict.py:86: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpribv4xj9/gauge_livesum_456.db' mode='ab+' closefd=True>
    with open(filename, 'rb') as infp:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_mostrecent
  /tmp/client_python/prometheus_client/mmap_dict.py:86: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmp22gzxk2n/gauge_max_123.db' mode='ab+' closefd=True>
    with open(filename, 'rb') as infp:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_mostrecent
  /tmp/client_python/prometheus_client/mmap_dict.py:86: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmp22gzxk2n/gauge_max_456.db' mode='ab+' closefd=True>
    with open(filename, 'rb') as infp:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_mostrecent
  /tmp/client_python/prometheus_client/mmap_dict.py:86: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmp6n8vag2o/gauge_min_123.db' mode='ab+' closefd=True>
    with open(filename, 'rb') as infp:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMultiProcess::test_gauge_mostrecent
  /tmp/client_python/prometheus_client/mmap_dict.py:86: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmp6n8vag2o/gauge_min_456.db' mode='ab+' closefd=True>
    with open(filename, 'rb') as infp:
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMmapedDict::test_expansion
  /tmp/client_python/.tox/py311/lib/python3.11/site-packages/_pytest/unittest.py:211: ResourceWarning: unclosed file <_io.BufferedRandom name='/tmp/tmptd9lerf6'>
    self._obj = None
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMmapedDict::test_multi_expansion
  /tmp/client_python/.tox/py311/lib/python3.11/site-packages/_pytest/unittest.py:211: ResourceWarning: unclosed file <_io.BufferedRandom name='/tmp/tmp4ho56w3t'>
    self._obj = None
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMmapedDict::test_process_restart
  /tmp/client_python/.tox/py311/lib/python3.11/site-packages/_pytest/unittest.py:211: ResourceWarning: unclosed file <_io.BufferedRandom name='/tmp/tmpp3siphvy'>
    self._obj = None
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_multiprocess.py::TestMmapedDict::test_process_restart
  /tmp/client_python/.tox/py311/lib/python3.11/site-packages/_pytest/runner.py:546: ResourceWarning: unclosed file <_io.BufferedRandom name='/tmp/tmpqi5kklpc'>
    fin()
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_parser.py::TestParse::test_untyped
  /tmp/client_python/tests/test_parser.py:18: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmp8lpdamo1/summary_123.db' mode='ab+' closefd=True>
    for sa, sb in zip(a.samples, b.samples):
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

tests/test_parser.py::TestParse::test_untyped
  /tmp/client_python/tests/test_parser.py:18: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmp8lpdamo1/summary_456.db' mode='ab+' closefd=True>
    for sa, sb in zip(a.samples, b.samples):
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================= short test summary info =======================================================
FAILED tests/test_multiprocess.py::TestMultiProcess::test_remove_clear_warning - AssertionError: assert False
======================================== 1 failed, 299 passed, 9 skipped, 32 warnings in 4.10s ========================================

An example tracemalloc output:

$ PYTHONTRACEMALLOC=20 python -m pytest -Wdefault
[…]
tests/test_parser.py::TestParse::test_untyped
  /usr/lib/python3.11/typing.py:362: ResourceWarning: unclosed file <_io.FileIO name='/tmp/tmpyaxyl9i6/summary_456.db' mode='ab+' closefd=True>
    return cached(*args, **kwds)
  
  Object allocated at:
    File "/tmp/client_python/.tox/py311/lib/python3.11/site-packages/_pytest/runner.py", line 114
      runtestprotocol(item, nextitem=nextitem)
    File "/tmp/client_python/.tox/py311/lib/python3.11/site-packages/_pytest/runner.py", line 133
      reports.append(call_and_report(item, "call", log))
    File "/tmp/client_python/.tox/py311/lib/python3.11/site-packages/_pytest/runner.py", line 226
      call = call_runtest_hook(item, when, **kwds)
    File "/tmp/client_python/.tox/py311/lib/python3.11/site-packages/_pytest/runner.py", line 265
      return CallInfo.from_call(
    File "/tmp/client_python/.tox/py311/lib/python3.11/site-packages/_pytest/runner.py", line 345
      result: Optional[TResult] = func()
    File "/tmp/client_python/.tox/py311/lib/python3.11/site-packages/_pytest/runner.py", line 266
      lambda: ihook(item=item, **kwds), when=when, reraise=reraise
    File "/tmp/client_python/.tox/py311/lib/python3.11/site-packages/pluggy/_hooks.py", line 501
      return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
    File "/tmp/client_python/.tox/py311/lib/python3.11/site-packages/pluggy/_manager.py", line 119
      return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
    File "/tmp/client_python/.tox/py311/lib/python3.11/site-packages/pluggy/_callers.py", line 102
      res = hook_impl.function(*args)
    File "/tmp/client_python/.tox/py311/lib/python3.11/site-packages/_pytest/runner.py", line 173
      item.runtest()
    File "/tmp/client_python/.tox/py311/lib/python3.11/site-packages/_pytest/unittest.py", line 333
      self._testcase(result=self)  # type: ignore[arg-type]
    File "/usr/lib/python3.11/unittest/case.py", line 678
      return self.run(*args, **kwds)
    File "/usr/lib/python3.11/unittest/case.py", line 623
      self._callTestMethod(testMethod)
    File "/usr/lib/python3.11/unittest/case.py", line 579
      if method() is not None:
    File "/tmp/client_python/tests/test_multiprocess.py", line 79
      s2 = Summary('s', 'help', registry=None)
    File "/tmp/client_python/prometheus_client/metrics.py", line 151
      self._metric_init()
    File "/tmp/client_python/prometheus_client/metrics.py", line 513
      self._count = values.ValueClass(self._type, self._name, self._name + '_count', self._labelnames,
    File "/tmp/client_python/prometheus_client/values.py", line 68
      self.__reset()
    File "/tmp/client_python/prometheus_client/values.py", line 82
      files[file_prefix] = MmapedDict(filename)
    File "/tmp/client_python/prometheus_client/mmap_dict.py", line 64
      self._f = open(filename, 'rb' if read_mode else 'a+b')

[mgorny]

This is with 0.20.0 (7a80f00).

[claws]

This seems to be because the MmapedDict objects in the 'files' attribute are not having '.close()' called on them. As a workaround, in my unittests I monkey patched a del function onto them that calls the '.close()' function when they are garbage collected and this stopped the resource warnings. This is obviously a hack but was the best I could do at the time. The MmapedDict class wrapped in a function using the 'files' closure attribute makes it hard to nicely clean up.

For my own future reference, this is is how I silenced the ResourceWarning warnings:

def reset_prometheus_client_registry(recreate_default_collectors: bool  = True):
    """ Clear prometheus_client REGISTRY and rebuild it """
    prometheus_client.REGISTRY._names_to_collectors.clear()
    prometheus_client.REGISTRY._collector_to_names.clear()

    if recreate_default_collectors:
        # By default the prometheus_client package creates these collectors
        # and adds them to the REGISTRY.
        prometheus_client.platform_collector.PlatformCollector()
        prometheus_client.process_collector.ProcessCollector()
        prometheus_client.gc_collector.GCCollector()


class SomeTestCase(unittest.TestCase):
    def setUp(self):

        ...

        # The prometheus_client package decides whether metrics will support
        # multiprocess behaviour based on environment settings defined when
        # the package is initially imported. Such configuration settings are
        # not typically present when this unit test is run so they are manually
        # configured to be in the required state.
        prometheus_client.values.ValueClass = (
            prometheus_client.values.MultiProcessValue()
        )

        # Address https://github.com/prometheus/client_python/issues/1008
        # by monkey-patching a __dell__ function onto MmappedValue class,
        # returned from prometheus_client.values.MultiProcessValue function,
        # to close the file descriptor.
        def close_MmappedDict_file_descriptor(self):
            self._file.close()

        prometheus_client.values.ValueClass.__del__ = close_MmappedDict_file_descriptor

    def tearDown(self):
        # Return prometheus_client package settings to default
        prometheus_client.values.ValueClass = prometheus_client.values.MutexValue

        # As the unit tests do not make use of any of the default collectors
        # created by prometheus_client there is no need to recreate them when
        # resetting the registry.
        reset_prometheus_client_registry(recreate_default_collectors=False)