MultiCallbackTest.test_easy_pause_unpause fails on macOS

[swt2c]

MultiCallbackTest.test_easy_pause_unpause fails on macOS. It's unclear whether this is a bug with the test or with the pause/unpause code itself. This is on macOS 12.1 with Secure Transport backend.

CC: @fsbs

__________________ MultiCallbackTest.test_easy_pause_unpause ___________________

self = <tests.multi_callback_test.MultiCallbackTest testMethod=test_easy_pause_unpause>

    def test_easy_pause_unpause(self):
        self.multi.add_handle(self.easy)
        while self.multi.socket_action(*self.socket_action)[1]:
            if self.socket_result:
                # libcurl informed us about new sockets
                break
            # Without real event loop we just use blocking select call instead
            self.multi.select(0.1)
        self.socket_result = None
        # libcurl will now inform us that we should remove those sockets
        self.easy.pause(pycurl.PAUSE_ALL)
>       assert self.socket_result is not None
E       AssertionError: assert None is not None
E        +  where None = <tests.multi_callback_test.MultiCallbackTest testMethod=test_easy_pause_unpause>.socket_result

tests/multi_callback_test.py:87: AssertionError

[fsbs]

In those macOs builds test_easy_pause_unpause is consistently failing on the same assert that test_easy_close always succeeds on. The two tests up to that point are exactly the same, the only difference is between easy.pause() and easy.close() right before the assert. Both tests depend on the same callback fix (#708). If libcurl's curl_easy_pause() was indeed invoking pycurl's internal callback (multi_socket_callback()) but the fix wasn't working then you'd see runtime warnings (#712). I can't see anything wrong with do_curl_pause() either, it would raise an exception if anything was wrong with the arguments or if a call to libcurl returned an error.

I did some testing just in libcurl and it seems libcurl's curl_easy_pause() is itself spotty when pausing, socket callback isn't always invoked as expected. I'll report it upstream.

Regardless, pycurl makes no difference between platforms here, so I can't say why it consistently works on Linux yet consistently fails on macOS. Fortunately this particular callback invocation isn't that crucial since libcurl is still pausing the transfer internally. The impact on downstream applications is that socket polling gets no performance benefit from pausing a transfer because the application isn't told that the associated sockets should be removed from its polling list.

Can you comment out that assert line and see if the rest of the test passes? That would tell us if at least unpausing works on macOS. On Linux unpausing always invokes timer callback as expected.

[swt2c]

Can you comment out that assert line and see if the rest of the test passes? That would tell us if at least unpausing works on macOS. On Linux unpausing always invokes timer callback as expected.

Thanks for the analysis. Yes, the rest of the test passes if I comment out that one assert.

[swt2c]

FYI, one other random thing I wanted to mention: when I was running the tests locally on Windows, a bunch of the multi-callback tests were failing initially - I was running on a VM with a single core. When I added a second core to the VM, all of the tests passed. I don't think that's relevant here as my Mac and the GitHub Macs both have multiple cores, but just wanted to mention it.

[fsbs]

when I was running the tests locally on Windows, a bunch of the multi-callback tests were failing initially - I was running on a VM with a single core. When I added a second core to the VM, all of the tests passed.

Thanks for the info, could you post the output from those tests? If more that just the pause test was failing then it might be something else.

The mock event loops in these tests assume a very simple transfer, maybe the bottle server behaves differently on different platforms. Replacing those mock event loops with asyncio might introduce another can of worms. I'm now thinking how to make this simple enough for the purpose of the tests and flexible enough for the unexpected differences between platforms.

[swt2c]

These are the tests that fail on Windows 10 on a single-core VM:

================================== FAILURES ===================================
______________________ MultiCallbackTest.test_easy_close ______________________

self = <tests.multi_callback_test.MultiCallbackTest testMethod=test_easy_close>

    def test_easy_close(self):
        self.multi.add_handle(self.easy)
        while self.multi.socket_action(*self.socket_action)[1]:
            if self.socket_result:
                # libcurl informed us about new sockets
                break
            # Without real event loop we just use blocking select call instead
            self.multi.select(0.1)
        self.socket_result = None
        # libcurl will now inform us that we should remove those sockets
        self.easy.close()
>       assert self.socket_result is not None
E       AssertionError: assert None is not None
E        +  where None = <tests.multi_callback_test.MultiCallbackTest testMethod=test_easy_close>.socket_result

tests\multi_callback_test.py:106: AssertionError
__________________ MultiCallbackTest.test_easy_pause_unpause __________________

self = <tests.multi_callback_test.MultiCallbackTest testMethod=test_easy_pause_unpause>

    def test_easy_pause_unpause(self):
        self.multi.add_handle(self.easy)
        while self.multi.socket_action(*self.socket_action)[1]:
            if self.socket_result:
                # libcurl informed us about new sockets
                break
            # Without real event loop we just use blocking select call instead
            self.multi.select(0.1)
        self.socket_result = None
        # libcurl will now inform us that we should remove those sockets
>       self.easy.pause(pycurl.PAUSE_ALL)
E       pycurl.error: (43, 'pause/unpause failed')

tests\multi_callback_test.py:85: error
_________________ MultiCallbackTest.test_multi_remove_handle __________________

self = <tests.multi_callback_test.MultiCallbackTest testMethod=test_multi_remove_handle>

    def test_multi_remove_handle(self):
        self.multi.add_handle(self.easy)
        while self.multi.socket_action(*self.socket_action)[1]:
            if self.socket_result:
                # libcurl informed us about new sockets
                break
            # Without real event loop we just use blocking select call instead
            self.multi.select(0.1)
        self.socket_result = None
        # libcurl will now inform us that we should remove those sockets
        self.multi.remove_handle(self.easy)
>       assert self.socket_result is not None
E       AssertionError: assert None is not None
E        +  where None = <tests.multi_callback_test.MultiCallbackTest testMethod=test_multi_remove_handle>.socket_result

tests\multi_callback_test.py:71: AssertionError
_________________ MultiCallbackTest.test_multi_socket_action __________________

self = <tests.multi_callback_test.MultiCallbackTest testMethod=test_multi_socket_action>

    def test_multi_socket_action(self):
        self.multi.add_handle(self.easy)
        self.timer_result = None
        self.socket_result = None
        while self.multi.socket_action(*self.socket_action)[1]:
            # Without real event loop we just use blocking select call instead
            self.multi.select(0.1)
        # both callbacks should be invoked multiple times by socket_action
>       assert self.socket_result is not None
E       AssertionError: assert None is not None
E        +  where None = <tests.multi_callback_test.MultiCallbackTest testMethod=test_multi_socket_action>.socket_result

tests\multi_callback_test.py:50: AssertionError

[fsbs]

Seems like a problem with the mock event loop, otherwise test_multi_add_handle would be failing as well. What is the output with #731 on a single core Windows?

[swt2c]

Seems like a problem with the mock event loop, otherwise test_multi_add_handle would be failing as well. What is the output with #731 on a single core Windows?

============================= test session starts =============================
platform win32 -- Python 3.8.7, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- c:\users\scott talbert\pycurl\env\scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\Scott Talbert\pycurl, configfile: pytest.ini
plugins: flaky-3.7.0
collecting ... collected 5 items

tests/multi_callback_test.py::MultiCallbackTest::test_easy_close FAILED  [ 20%]
tests/multi_callback_test.py::MultiCallbackTest::test_easy_pause_unpause PASSED [ 40%]
tests/multi_callback_test.py::MultiCallbackTest::test_multi_add_handle PASSED [ 60%]
tests/multi_callback_test.py::MultiCallbackTest::test_multi_remove_handle PASSED [ 80%]
tests/multi_callback_test.py::MultiCallbackTest::test_multi_socket_action PASSED [100%]

================================== FAILURES ===================================
______________________ MultiCallbackTest.test_easy_close ______________________

self = <tests.multi_callback_test.MultiCallbackTest testMethod=test_easy_close>

    def test_easy_close(self):
        self.easy.setopt(pycurl.URL, 'http://%s:8380/success' % localhost)
        self.multi.add_handle(self.easy)
        self.multi.socket_action(pycurl.SOCKET_TIMEOUT, 0)
        self.socket_result = None
        self.easy.close()
>       assert self.socket_result is not None
E       AssertionError: assert None is not None
E        +  where None = <tests.multi_callback_test.MultiCallbackTest testMethod=test_easy_close>.socket_result

tests\multi_callback_test.py:96: AssertionError
=========================== short test summary info ===========================
FAILED tests/multi_callback_test.py::MultiCallbackTest::test_easy_close - Ass...
========================= 1 failed, 4 passed in 0.70s =========================

[fsbs]

Thanks again, looks like the event loop was indeed the main issue on Windows.

test_easy_close FAILED
test_multi_remove_handle PASSED

That's the strangest one yet. do_easy_close() actually calls curl_mutli_remove_handle() before closing the handle, so the callback invocation should depend on the same conditions. Anyway, I'll make this test use the same approach as test_easy_pause_unpause.

What's most important is that there are no runtime warnings in any of these failures. The problem so far is just with the tests themselves not forcing libcurl to invoke the callbacks across different platforms.

IMO your CI changes (#725 and #726) can be accepted without passing these callback tests. I'll also have easier time fine-tuning these tests to different platforms if Windows and macOS CI is already in place.

[swt2c]

Sounds good. Hopefully, @p will find some time soon to review and merge them.

[swt2c]

Hey @fsbs, I thought your latest PR #731 had resolved all these issues, but it seems there are now some failures, see CI results:

=================================== FAILURES ===================================
______________________ MultiCallbackTest.test_easy_close _______________________

self = <tests.multi_callback_test.MultiCallbackTest testMethod=test_easy_close>

    def test_easy_close(self):
        self.partial_transfer()
        self.socket_result = None
        self.easy.close()
>       assert self.socket_result is not None
E       assert None is not None
E        +  where None = <tests.multi_callback_test.MultiCallbackTest testMethod=test_easy_close>.socket_result

tests/multi_callback_test.py:97: AssertionError
__________________ MultiCallbackTest.test_multi_remove_handle __________________

self = <tests.multi_callback_test.MultiCallbackTest testMethod=test_multi_remove_handle>

    def test_multi_remove_handle(self):
        self.multi.add_handle(self.easy)
        self.multi.socket_action(pycurl.SOCKET_TIMEOUT, 0)
        self.socket_result = None
        self.multi.remove_handle(self.easy)
>       assert self.socket_result is not None
E       assert None is not None
E        +  where None = <tests.multi_callback_test.MultiCallbackTest testMethod=test_multi_remove_handle>.socket_result

tests/multi_callback_test.py:74: AssertionError
__________________ MultiCallbackTest.test_multi_socket_action __________________

self = <tests.multi_callback_test.MultiCallbackTest testMethod=test_multi_socket_action>

    def test_multi_socket_action(self):
        self.multi.add_handle(self.easy)
        self.timer_result = None
        self.socket_result = None
        self.multi.socket_action(pycurl.SOCKET_TIMEOUT, 0)
>       assert self.socket_result is not None
E       assert None is not None
E        +  where None = <tests.multi_callback_test.MultiCallbackTest testMethod=test_multi_socket_action>.socket_result

tests/multi_callback_test.py:60: AssertionError

[swt2c]

Note: I added more xfails to the multi-callback tests so that the mac tests will show a green board for now.

[mcepl]

I can't say why it consistently works on Linux yet consistently fails on macOS

Actually it doesn’t. These are build logs from the test suite failing on openSUSE/Tumbleweed while building the official package for python-pycurl.

_log.txt

[mtelka]

I see the following failure on OpenIndiana (sunos) with curl 8.4.0 and pycurl 7.45.2:

__________________ MultiCallbackTest.test_multi_socket_action __________________
            
self = <tests.multi_callback_test.MultiCallbackTest testMethod=test_multi_socket_action>
            
    @pytest.mark.xfail(sys.platform == 'darwin', reason='https://github.com/pycurl/pycurl/issues/729')
    def test_multi_socket_action(self):
        self.multi.add_handle(self.easy)
        self.timer_result = None
        self.socket_result = None
        self.multi.socket_action(pycurl.SOCKET_TIMEOUT, 0)
        assert self.socket_result is not None
>       assert self.timer_result is not None
E       assert None is not None
E        +  where None = <tests.multi_callback_test.MultiCallbackTest testMethod=test_multi_socket_action>.timer_result

tests/multi_callback_test.py:62: AssertionError

[mtelka]

I also see this likely related failure:

_______________________ PauseTest.test_pause_via_return ________________________

self = <tests.pause_test.PauseTest testMethod=test_pause_via_return>

    def test_pause_via_return(self):
>       self.check_pause(False)

tests/pause_test.py:28:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/util.py:208: in decorated
    return fn(*args, **kwargs)
tests/pause_test.py:95: in check_pause
    self.assertTrue(end-start > 1)
E   AssertionError: False is not true

[mtelka]

There is also one more (and the last one) pause related test failure here:

________________________ PauseTest.test_pause_via_call _________________________

self = <tests.pause_test.PauseTest testMethod=test_pause_via_call>

    def test_pause_via_call(self):
>       self.check_pause(True)

tests/pause_test.py:27:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/util.py:208: in decorated
    return fn(*args, **kwargs)
tests/pause_test.py:98: in check_pause
    self.assertTrue(end-start > 1)
E   AssertionError: False is not true