BUG: Failing tests on multiple architectures when building scipy 1.10.0 for Fedora

[psimovec]

Describe your issue.

I am updating Fedora package for scipy 1.8.3 -> 1.10.0
(We skipped version 1.9, as there were test failures on i686 architecture, as reported in #17213 , and disabling the tests led to another tests failing)
I did 65+ builds, as disabling some tests led to other tests failing directly or being flaky.

For first build, all of important failed tests and error messages are in i686-build.log and s390x-build.log.

For rest of the builds - rest of archived logs ( as logs in koji will expire in <14 days) : https://drive.google.com/file/d/1op-FgNp1bHHdJB8GAEWowiBrYo5GwDWG/view?usp=sharing (5.3MB compressed, 943MB uncompressed)
(check kojilogs/noarch-<taskID>/<architecture>-*/build.log in the archive)
You can also check scipy-notes.txt to see which tests were disabled at specific moment/in which tasks the tests failed.

Test failures in first build:

https://koji.fedoraproject.org/koji/taskinfo?taskID=96702545
=========================== short test summary info ============================
all architectures s390x-build.log or i686-build.log
FAILED scipy/integrate/tests/test_banded_ode_solvers.py::test_banded_ode_solvers
FAILED scipy/integrate/tests/test_integrate.py::TestOde::test_vode - Assertio...
FAILED scipy/integrate/tests/test_integrate.py::TestOde::test_zvode - Asserti...
FAILED scipy/integrate/tests/test_integrate.py::TestOde::test_concurrent_ok
FAILED scipy/integrate/tests/test_integrate.py::TestComplexOde::test_vode - A...
FAILED scipy/integrate/tests/test_integrate.py::TestVODECheckParameterUse::test_no_params
FAILED scipy/integrate/tests/test_integrate.py::TestVODECheckParameterUse::test_one_scalar_param
FAILED scipy/integrate/tests/test_integrate.py::TestVODECheckParameterUse::test_two_scalar_params
FAILED scipy/integrate/tests/test_integrate.py::TestVODECheckParameterUse::test_vector_param
FAILED scipy/integrate/tests/test_integrate.py::TestVODECheckParameterUse::test_warns_on_failure
FAILED scipy/integrate/tests/test_integrate.py::TestZVODECheckParameterUse::test_no_params
FAILED scipy/integrate/tests/test_integrate.py::TestZVODECheckParameterUse::test_one_scalar_param
FAILED scipy/integrate/tests/test_integrate.py::TestZVODECheckParameterUse::test_two_scalar_params
FAILED scipy/integrate/tests/test_integrate.py::TestZVODECheckParameterUse::test_vector_param
FAILED scipy/integrate/tests/test_integrate.py::TestZVODECheckParameterUse::test_warns_on_failure

i686 unique, same as #17213, i686-build.log
FAILED scipy/sparse/linalg/tests/test_propack.py::test_examples[False-float64]
FAILED scipy/sparse/linalg/tests/test_propack.py::test_examples[True-float32]
FAILED scipy/sparse/linalg/tests/test_propack.py::test_shifts[float32-10]
FAILED scipy/sparse/linalg/tests/test_propack.py::test_shifts[float64-10]

aarch64 + s390x #17859 most likely solved here
FAILED scipy/sparse/linalg/_isolve/tests/test_iterative.py::test_maxiter_worsening[lgmres]

s390x only s390x-build.log, Comment about possible suspect -> #17038
FAILED scipy/interpolate/tests/test_bsplines.py::TestInterop::test_splint - A...
FAILED scipy/interpolate/tests/test_fitpack.py::TestSplder::test_splantider_vs_splint
FAILED scipy/interpolate/tests/test_fitpack.py::test_dblint - AssertionError:
FAILED scipy/interpolate/tests/test_fitpack2.py::TestUnivariateSpline::test_derivative_and_antiderivative
FAILED scipy/interpolate/tests/test_fitpack2.py::TestLSQBivariateSpline::test_integral
FAILED scipy/interpolate/tests/test_interpolate.py::TestPPoly::test_integrate
FAILED scipy/stats/tests/test_distributions.py::TestLevyStable::test_cdf_nolan_samples[pct_range0-alpha_range0-beta_range0]
FAILED scipy/stats/tests/test_distributions.py::TestLevyStable::test_cdf_nolan_samples[pct_range1-alpha_range1-beta_range1]

Test failures in following builds:

These tests also failed at some point, in different builds, when specific tests were enabled/disabled (more info in the attachment)
If you think it is worth to look at them, you can find their error messages in log files I have mentioned. Also, you can check scipy-notes.txt to see in which tasks the tests failed and which tests were disabled/enabled at the moment.

x86_64, s390x, ppc64le
scipy/sparse/linalg/tests/test_propack.py::test_examples[True-complex64]

x86_64
scipy/stats/tests/test_multivariate.py::TestWishart::test_frozen - Ass...

aarch64
scipy/sparse/tests/test_construct.py::TestConstructUtils::test_concatenate_int32_overflow

i686: #17213 possibly the same behaviour as disabling i686 tests in this Issue, on 1.9.3
scipy/stats/tests/test_distributions.py::TestLevyStable::test_cdf_nolan_samples[pct_range0-alpha_range0-beta_range0]
scipy/stats/tests/test_distributions.py::TestNct::test_broadcasting
scipy/sparse/tests/test_base.py::Test64Bit::test_resiliency_all_32[TestBSR-test_mean]
scipy/sparse/tests/test_base.py::Test64Bit::test_resiliency_all_32[TestCSC-test_sum]
scipy/sparse/tests/test_base.py::Test64Bit::test_resiliency_all_32[TestDOK-test_sum]
scipy/sparse/tests/test_base.py::TestCOO::test_mu - AssertionError:
scipy/sparse/tests/test_base.py::TestCOO::test_sum - AssertionError:
scipy/sparse/tests/test_base.py::TestCSR::test_mu - AssertionError:
scipy/sparse/tests/test_base.py::TestCSR::test_sum - AssertionError:
scipy/stats/tests/test_continuous_basic.py::test_methods_with_lists[levy-args62-isf]
scipy/stats/tests/test_continuous_basic.py::test_methods_with_lists[nct-args76-pdf]
scipy/sparse/tests/test_base.py::TestCSR::test_fancy_assignment_dtypes

(following tests also failed at some point, but I don't have the logs right now)
test_resiliency_random[TestCSC-test_sum_dtype]
test_boost
test_entropy_broadcasting
test_svdp

Reproducing Code Example

runtests.py -n

Error message

If you need additional info, search _ test_name in build.log files to see Error messages. (for example search _ test_banded_ode_solvers)
i686-build.log
s390x-build.log
Or you can check rest of the logs in this archive:
scipylogs.tar.xz

Errors messages

___________________________ test_banded_ode_solvers ____________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
    def test_banded_ode_solvers():
        # Test the "lsoda", "vode" and "zvode" solvers of the `ode` class
        # with a system that has a banded Jacobian matrix.
    
        t_exact = np.linspace(0, 1.0, 5)
    
        # --- Real arrays for testing the "lsoda" and "vode" solvers ---
    
        # lband = 2, uband = 1:
        a_real = np.array([[-0.6, 0.1, 0.0, 0.0, 0.0],
                           [0.2, -0.5, 0.9, 0.0, 0.0],
                           [0.1, 0.1, -0.4, 0.1, 0.0],
                           [0.0, 0.3, -0.1, -0.9, -0.3],
                           [0.0, 0.0, 0.1, 0.1, -0.7]])
    
        # lband = 0, uband = 1:
        a_real_upper = np.triu(a_real)
    
        # lband = 2, uband = 0:
        a_real_lower = np.tril(a_real)
    
        # lband = 0, uband = 0:
        a_real_diag = np.triu(a_real_lower)
    
        real_matrices = [a_real, a_real_upper, a_real_lower, a_real_diag]
        real_solutions = []
    
        for a in real_matrices:
            y0 = np.arange(1, a.shape[0] + 1)
            y_exact = _analytical_solution(a, y0, t_exact)
            real_solutions.append((y0, t_exact, y_exact))
    
        def check_real(idx, solver, meth, use_jac, with_jac, banded):
            a = real_matrices[idx]
            y0, t_exact, y_exact = real_solutions[idx]
            t, y = _solve_linear_sys(a, y0,
                                     tend=t_exact[-1],
                                     dt=t_exact[1] - t_exact[0],
                                     solver=solver,
                                     method=meth,
                                     use_jac=use_jac,
                                     with_jacobian=with_jac,
                                     banded=banded)
            assert_allclose(t, t_exact)
            assert_allclose(y, y_exact)
    
        for idx in range(len(real_matrices)):
            p = [['vode', 'lsoda'],  # solver
                 ['bdf', 'adams'],   # method
                 [False, True],      # use_jac
                 [False, True],      # with_jacobian
                 [False, True]]      # banded
            for solver, meth, use_jac, with_jac, banded in itertools.product(*p):
>               check_real(idx, solver, meth, use_jac, with_jac, banded)
scipy/integrate/tests/test_banded_ode_solvers.py:180: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
idx = 0, solver = 'vode', meth = 'bdf', use_jac = False, with_jac = False
banded = False
    def check_real(idx, solver, meth, use_jac, with_jac, banded):
        a = real_matrices[idx]
        y0, t_exact, y_exact = real_solutions[idx]
        t, y = _solve_linear_sys(a, y0,
                                 tend=t_exact[-1],
                                 dt=t_exact[1] - t_exact[0],
                                 solver=solver,
                                 method=meth,
                                 use_jac=use_jac,
                                 with_jacobian=with_jac,
                                 banded=banded)
>       assert_allclose(t, t_exact)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0
E       
E       (shapes (3,), (5,) mismatch)
E        x: array([0.  , 0.25, 0.25])
E        y: array([0.  , 0.25, 0.5 , 0.75, 1.  ])
scipy/integrate/tests/test_banded_ode_solvers.py:170: AssertionError
______________________________ TestOde.test_vode _______________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestOde object at 0x3ff7a204890>
    def test_vode(self):
        # Check the vode solver
        for problem_cls in PROBLEMS:
            problem = problem_cls()
            if problem.cmplx:
                continue
            if not problem.stiff:
>               self._do_problem(problem, 'vode', 'adams')
scipy/integrate/tests/test_integrate.py:99: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestOde object at 0x3ff7a204890>
problem = <scipy.integrate.tests.test_integrate.SimpleOscillator object at 0x3ff6c948fd0>
integrator = 'vode', method = 'adams'
    def _do_problem(self, problem, integrator, method='adams'):
    
        # ode has callback arguments in different order than odeint
        f = lambda t, z: problem.f(z, t)
        jac = None
        if hasattr(problem, 'jac'):
            jac = lambda t, z: problem.jac(z, t)
    
        integrator_params = {}
        if problem.lband is not None or problem.uband is not None:
            integrator_params['uband'] = problem.uband
            integrator_params['lband'] = problem.lband
    
        ig = self.ode_class(f, jac)
        ig.set_integrator(integrator,
                          atol=problem.atol/10,
                          rtol=problem.rtol/10,
                          method=method,
                          **integrator_params)
    
        ig.set_initial_value(problem.z0, t=0.0)
        z = ig.integrate(problem.stop_t)
    
        assert_array_equal(z, ig.y)
        assert_(ig.successful(), (problem, method))
        assert_(ig.get_return_code() > 0, (problem, method))
>       assert_(problem.verify(array([z]), problem.stop_t), (problem, method))
E       AssertionError: (<scipy.integrate.tests.test_integrate.SimpleOscillator object at 0x3ff6c948fd0>, 'adams')
scipy/integrate/tests/test_integrate.py:85: AssertionError
______________________________ TestOde.test_zvode ______________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestOde object at 0x3ff7a204d90>
    def test_zvode(self):
        # Check the zvode solver
        for problem_cls in PROBLEMS:
            problem = problem_cls()
            if not problem.stiff:
>               self._do_problem(problem, 'zvode', 'adams')
scipy/integrate/tests/test_integrate.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestOde object at 0x3ff7a204d90>
problem = <scipy.integrate.tests.test_integrate.SimpleOscillator object at 0x3ff6c92b410>
integrator = 'zvode', method = 'adams'
    def _do_problem(self, problem, integrator, method='adams'):
    
        # ode has callback arguments in different order than odeint
        f = lambda t, z: problem.f(z, t)
        jac = None
        if hasattr(problem, 'jac'):
            jac = lambda t, z: problem.jac(z, t)
    
        integrator_params = {}
        if problem.lband is not None or problem.uband is not None:
            integrator_params['uband'] = problem.uband
            integrator_params['lband'] = problem.lband
    
        ig = self.ode_class(f, jac)
        ig.set_integrator(integrator,
                          atol=problem.atol/10,
                          rtol=problem.rtol/10,
                          method=method,
                          **integrator_params)
    
        ig.set_initial_value(problem.z0, t=0.0)
        z = ig.integrate(problem.stop_t)
    
        assert_array_equal(z, ig.y)
        assert_(ig.successful(), (problem, method))
        assert_(ig.get_return_code() > 0, (problem, method))
>       assert_(problem.verify(array([z]), problem.stop_t), (problem, method))
E       AssertionError: (<scipy.integrate.tests.test_integrate.SimpleOscillator object at 0x3ff6c92b410>, 'adams')
scipy/integrate/tests/test_integrate.py:85: AssertionError
__________________________ TestOde.test_concurrent_ok __________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestOde object at 0x3ff7a289450>
    def test_concurrent_ok(self):
        f = lambda t, y: 1.0
    
        for k in range(3):
            for sol in ('vode', 'zvode', 'lsoda', 'dopri5', 'dop853'):
                r = ode(f).set_integrator(sol)
                r.set_initial_value(0, 0)
    
                r2 = ode(f).set_integrator(sol)
                r2.set_initial_value(0, 0)
    
                r.integrate(r.t + 0.1)
                r2.integrate(r2.t + 0.1)
                r2.integrate(r2.t + 0.1)
    
>               assert_allclose(r.y, 0.1)
E               AssertionError: 
E               Not equal to tolerance rtol=1e-07, atol=0
E               
E               Mismatched elements: 1 / 1 (100%)
E               Max absolute difference: 0.1
E               Max relative difference: 1.
E                x: array([0.])
E                y: array(0.1)
scipy/integrate/tests/test_integrate.py:172: AssertionError
----------------------------- Captured stdout call -----------------------------
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.2500000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.5000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Trouble from DVINDY.  ITASK = I1, TOUT = R1.       
      In above message,  I1 =         1
      In above message,  R1 =  0.5000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1090000000000D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVODE--  Warning: internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1090000000000D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVODE--  Warning: internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVODE--  Warning: internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
___________________________ TestComplexOde.test_vode ___________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestComplexOde object at 0x3ff7a28ab50>
    def test_vode(self):
        # Check the vode solver
        for problem_cls in PROBLEMS:
            problem = problem_cls()
            if not problem.stiff:
>               self._do_problem(problem, 'vode', 'adams')
scipy/integrate/tests/test_integrate.py:201: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestComplexOde object at 0x3ff7a28ab50>
problem = <scipy.integrate.tests.test_integrate.SimpleOscillator object at 0x3ff6e27af10>
integrator = 'vode', method = 'adams'
    def _do_problem(self, problem, integrator, method='adams'):
    
        # ode has callback arguments in different order than odeint
        f = lambda t, z: problem.f(z, t)
        jac = None
        if hasattr(problem, 'jac'):
            jac = lambda t, z: problem.jac(z, t)
    
        integrator_params = {}
        if problem.lband is not None or problem.uband is not None:
            integrator_params['uband'] = problem.uband
            integrator_params['lband'] = problem.lband
    
        ig = self.ode_class(f, jac)
        ig.set_integrator(integrator,
                          atol=problem.atol/10,
                          rtol=problem.rtol/10,
                          method=method,
                          **integrator_params)
    
        ig.set_initial_value(problem.z0, t=0.0)
        z = ig.integrate(problem.stop_t)
    
        assert_array_equal(z, ig.y)
        assert_(ig.successful(), (problem, method))
        assert_(ig.get_return_code() > 0, (problem, method))
>       assert_(problem.verify(array([z]), problem.stop_t), (problem, method))
E       AssertionError: (<scipy.integrate.tests.test_integrate.SimpleOscillator object at 0x3ff6e27af10>, 'adams')
scipy/integrate/tests/test_integrate.py:85: AssertionError
___________________ TestVODECheckParameterUse.test_no_params ___________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a242fd0>
    def test_no_params(self):
        solver = self._get_solver(f, jac)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:602: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a242fd0>
solver = <scipy.integrate._ode.ode object at 0x3ff6e19de50>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1., 0.])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
_______________ TestVODECheckParameterUse.test_one_scalar_param ________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a243610>
    def test_one_scalar_param(self):
        solver = self._get_solver(f1, jac1)
        omega = 1.0
        solver.set_f_params(omega)
        if self.solver_uses_jac:
            solver.set_jac_params(omega)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:610: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a243610>
solver = <scipy.integrate._ode.ode object at 0x3ff6e1beb10>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1., 0.])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
_______________ TestVODECheckParameterUse.test_two_scalar_params _______________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a243190>
    def test_two_scalar_params(self):
        solver = self._get_solver(f2, jac2)
        omega1 = 1.0
        omega2 = 1.0
        solver.set_f_params(omega1, omega2)
        if self.solver_uses_jac:
            solver.set_jac_params(omega1, omega2)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:619: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a243190>
solver = <scipy.integrate._ode.ode object at 0x3ff6e243390>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1., 0.])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
_________________ TestVODECheckParameterUse.test_vector_param __________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a242750>
    def test_vector_param(self):
        solver = self._get_solver(fv, jacv)
        omega = [1.0, 1.0]
        solver.set_f_params(omega)
        if self.solver_uses_jac:
            solver.set_jac_params(omega)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:627: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a242750>
solver = <scipy.integrate._ode.ode object at 0x3ff6c97ee90>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1., 0.])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
_______________ TestVODECheckParameterUse.test_warns_on_failure ________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a243b90>
    def test_warns_on_failure(self):
        # Set nsteps small to ensure failure
        solver = self._get_solver(f, jac)
        solver.set_integrator(self.solver_name, nsteps=1)
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
>       assert_warns(UserWarning, solver.integrate, pi)
scipy/integrate/tests/test_integrate.py:635: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.11/site-packages/numpy/testing/_private/utils.py:1799: in assert_warns
    with _assert_warns_context(warning_class, name=func.__name__):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <contextlib._GeneratorContextManager object at 0x3ff6c966790>, typ = None
value = None, traceback = None
    def __exit__(self, typ, value, traceback):
        if typ is None:
            try:
>               next(self.gen)
E               AssertionError: No warning raised when calling integrate
/usr/lib64/python3.11/contextlib.py:144: AssertionError
__________________ TestZVODECheckParameterUse.test_no_params ___________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a27ac50>
    def test_no_params(self):
        solver = self._get_solver(f, jac)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:602: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a27ac50>
solver = <scipy.integrate._ode.ode object at 0x3ff6e1cc5d0>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1.+0.j, 0.+0.j])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
_______________ TestZVODECheckParameterUse.test_one_scalar_param _______________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a28a9d0>
    def test_one_scalar_param(self):
        solver = self._get_solver(f1, jac1)
        omega = 1.0
        solver.set_f_params(omega)
        if self.solver_uses_jac:
            solver.set_jac_params(omega)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:610: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a28a9d0>
solver = <scipy.integrate._ode.ode object at 0x3ff6c878890>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1.+0.j, 0.+0.j])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
----------------------------- Captured stdout call -----------------------------
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.2000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Trouble from DVINDY.  ITASK = I1, TOUT = R1.       
      In above message,  I1 =         1
      In above message,  R1 =  0.2000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1090000000000D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.3141592653590D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.3141592653590D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.3141592653590D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.3141592653590D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.3141592653590D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVODE--  Warning: internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.3141592653590D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVODE--  Warning: internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
______________ TestZVODECheckParameterUse.test_two_scalar_params _______________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a28ac10>
    def test_two_scalar_params(self):
        solver = self._get_solver(f2, jac2)
        omega1 = 1.0
        omega2 = 1.0
        solver.set_f_params(omega1, omega2)
        if self.solver_uses_jac:
            solver.set_jac_params(omega1, omega2)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:619: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a28ac10>
solver = <scipy.integrate._ode.ode object at 0x3ff6e1ceb10>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1.+0.j, 0.+0.j])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
_________________ TestZVODECheckParameterUse.test_vector_param _________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a205990>
    def test_vector_param(self):
        solver = self._get_solver(fv, jacv)
        omega = [1.0, 1.0]
        solver.set_f_params(omega)
        if self.solver_uses_jac:
            solver.set_jac_params(omega)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:627: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a205990>
solver = <scipy.integrate._ode.ode object at 0x3ff6c685050>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1.+0.j, 0.+0.j])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
_______________ TestZVODECheckParameterUse.test_warns_on_failure _______________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a242b10>
    def test_warns_on_failure(self):
        # Set nsteps small to ensure failure
        solver = self._get_solver(f, jac)
        solver.set_integrator(self.solver_name, nsteps=1)
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
>       assert_warns(UserWarning, solver.integrate, pi)
scipy/integrate/tests/test_integrate.py:635: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.11/site-packages/numpy/testing/_private/utils.py:1799: in assert_warns
    with _assert_warns_context(warning_class, name=func.__name__):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <contextlib._GeneratorContextManager object at 0x3ff6c87bf10>, typ = None
value = None, traceback = None
    def __exit__(self, typ, value, traceback):
        if typ is None:
            try:
>               next(self.gen)
E               AssertionError: No warning raised when calling integrate
/usr/lib64/python3.11/contextlib.py:144: AssertionError

s390x only failures:

___________________________ TestInterop.test_splint ____________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.interpolate.tests.test_bsplines.TestInterop object at 0x3ff77aae5d0>
    def test_splint(self):
        # test that splint accepts BSpline objects
        b, b2 = self.b, self.b2
        assert_allclose(splint(0, 1, b),
                        splint(0, 1, b.tck), atol=1e-14)
>       assert_allclose(splint(0, 1, b),
                        b.integrate(0, 1), atol=1e-14)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=1e-14
E       
E       Mismatched elements: 1 / 1 (100%)
E       Max absolute difference: 0.39296037
E       Max relative difference: 0.46697394
E        x: array(0.448543)
E        y: array(0.841504)
scipy/interpolate/tests/test_bsplines.py:873: AssertionError
_____________________ TestSplder.test_splantider_vs_splint _____________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.interpolate.tests.test_fitpack.TestSplder object at 0x3ff779ba110>
    def test_splantider_vs_splint(self):
        # Check antiderivative vs. FITPACK
        spl2 = splantider(self.spl)
    
        # no extrapolation, splint assumes function is zero outside
        # range
        xx = np.linspace(0, 1, 20)
    
        for x1 in xx:
            for x2 in xx:
                y1 = splint(x1, x2, self.spl)
                y2 = splev(x2, spl2) - splev(x1, spl2)
>               assert_allclose(y1, y2)
E               AssertionError: 
E               Not equal to tolerance rtol=1e-07, atol=0
E               
E               Mismatched elements: 1 / 1 (100%)
E               Max absolute difference: 0.01986645
E               Max relative difference: 0.78723695
E                x: array(0.005369)
E                y: array(0.025236)
scipy/interpolate/tests/test_fitpack.py:277: AssertionError
_________________________________ test_dblint __________________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
    def test_dblint():
        # Basic test to see it runs and gives the correct result on a trivial
        # problem. Note that `dblint` is not exposed in the interpolate namespace.
        x = np.linspace(0, 1)
        y = np.linspace(0, 1)
        xx, yy = np.meshgrid(x, y)
        rect = RectBivariateSpline(x, y, 4 * xx * yy)
        tck = list(rect.tck)
        tck.extend(rect.degrees)
    
>       assert_almost_equal(dblint(0, 1, 0, 1, tck), 1)
E       AssertionError: 
E       Arrays are not almost equal to 7 decimals
E        ACTUAL: 0.019024309965098775
E        DESIRED: 1
scipy/interpolate/tests/test_fitpack.py:391: AssertionError
___________ TestUnivariateSpline.test_derivative_and_antiderivative ____________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.interpolate.tests.test_fitpack2.TestUnivariateSpline object at 0x3ff7798acd0>
    def test_derivative_and_antiderivative(self):
        # Thin wrappers to splder/splantider, so light smoke test only.
        x = np.linspace(0, 1, 70)**3
        y = np.cos(x)
    
        spl = UnivariateSpline(x, y, s=0)
        spl2 = spl.antiderivative(2).derivative(2)
        assert_allclose(spl(0.3), spl2(0.3))
    
        spl2 = spl.antiderivative(1)
>       assert_allclose(spl2(0.6) - spl2(0.2),
                        spl.integral(0.2, 0.6))
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0
E       
E       Mismatched elements: 1 / 1 (100%)
E       Max absolute difference: 0.34922361
E       Max relative difference: 20.84975171
E        x: array(0.365973)
E        y: array(0.01675)
scipy/interpolate/tests/test_fitpack2.py:168: AssertionError
_____________________ TestLSQBivariateSpline.test_integral _____________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.interpolate.tests.test_fitpack2.TestLSQBivariateSpline object at 0x3ff777c6550>
    def test_integral(self):
        x = [1,1,1,2,2,2,8,8,8]
        y = [1,2,3,1,2,3,1,2,3]
        z = array([0,7,8,3,4,7,1,3,4])
    
        s = 0.1
        tx = [1+s,3-s]
        ty = [1+s,3-s]
        with suppress_warnings() as sup:
            r = sup.record(UserWarning, "\nThe coefficients of the spline")
            lut = LSQBivariateSpline(x, y, z, tx, ty, kx=1, ky=1)
            assert_equal(len(r), 1)
        tx, ty = lut.get_knots()
        tz = lut(tx, ty)
        trpz = .25*(diff(tx)[:,None]*diff(ty)[None,:]
                    * (tz[:-1,:-1]+tz[1:,:-1]+tz[:-1,1:]+tz[1:,1:])).sum()
    
>       assert_almost_equal(lut.integral(tx[0], tx[-1], ty[0], ty[-1]),
                            trpz)
E       AssertionError: 
E       Arrays are not almost equal to 7 decimals
E        ACTUAL: 75.93932045329475
E        DESIRED: 180.04605637737518
scipy/interpolate/tests/test_fitpack2.py:443: AssertionError
___________________________ TestPPoly.test_integrate ___________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.interpolate.tests.test_interpolate.TestPPoly object at 0x3ff7771e710>
    def test_integrate(self):
        np.random.seed(1234)
        x = np.sort(np.r_[0, np.random.rand(11), 1])
        y = np.random.rand(len(x))
    
        spl = splrep(x, y, s=0, k=5)
        pp = PPoly.from_spline(spl)
    
        a, b = 0.3, 0.9
        ig = pp.integrate(a, b)
    
        ipp = pp.antiderivative()
        assert_allclose(ig, ipp(b) - ipp(a))
>       assert_allclose(ig, splint(a, b, spl))
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0
E       
E       Mismatched elements: 1 / 1 (100%)
E       Max absolute difference: 10.65264483
E       Max relative difference: 1.00916222
E        x: array(-0.096716)
E        y: array(10.555929)
scipy/interpolate/tests/test_interpolate.py:1465: AssertionError
________________________ test_maxiter_worsening[lgmres] ________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
solver = <function lgmres at 0x3ff7e409940>
    @pytest.mark.parametrize("solver", [
        pytest.param(gmres, marks=pytest.mark.xfail(platform.machine() == 'aarch64'
                                                    and sys.version_info[1] == 9,
                                                    reason="gh-13019")),
        qmr,
        pytest.param(lgmres, marks=pytest.mark.xfail(platform.machine() == 'ppc64le',
                                                     reason="fails on ppc64le")),
        pytest.param(cgs, marks=pytest.mark.xfail),
        pytest.param(bicg, marks=pytest.mark.xfail),
        pytest.param(bicgstab, marks=pytest.mark.xfail),
        pytest.param(gcrotmk, marks=pytest.mark.xfail),
        pytest.param(tfqmr, marks=pytest.mark.xfail)])
    def test_maxiter_worsening(solver):
        # Check error does not grow (boundlessly) with increasing maxiter.
        # This can occur due to the solvers hitting close to breakdown,
        # which they should detect and halt as necessary.
        # cf. gh-9100
    
        # Singular matrix, rhs numerically not in range
        A = np.array([[-0.1112795288033378, 0, 0, 0.16127952880333685],
                      [0, -0.13627952880333782+6.283185307179586j, 0, 0],
                      [0, 0, -0.13627952880333782-6.283185307179586j, 0],
                      [0.1112795288033368, 0j, 0j, -0.16127952880333785]])
        v = np.ones(4)
        best_error = np.inf
        tol = 7 if platform.machine() == 'aarch64' else 5
    
        for maxiter in range(1, 20):
            x, info = solver(A, v, maxiter=maxiter, tol=1e-8, atol=0)
    
            if info == 0:
                assert_(np.linalg.norm(A.dot(x) - v) <= 1e-8*np.linalg.norm(v))
    
            error = np.linalg.norm(A.dot(x) - v)
            best_error = min(best_error, error)
    
            # Check with slack
>           assert_(error <= tol*best_error)
E           AssertionError
scipy/sparse/linalg/_isolve/tests/test_iterative.py:487: AssertionError
__ TestLevyStable.test_cdf_nolan_samples[pct_range0-alpha_range0-beta_range0] __
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.stats.tests.test_distributions.TestLevyStable object at 0x3ff6df71090>
nolan_cdf_sample_data = rec.array([(-5.54809272e+19, 0.01, 0.1, -1., 0.01),
           (-1.93540945e-07, 0.99, 0.1, -1., 0.99),
           (-5....95, 2. ,  1., 0.95)],
          dtype=[('x', '<f8'), ('p', '<f8'), ('alpha', '<f8'), ('beta', '<f8'), ('pct', '<f8')])
pct_range = [0.01, 0.5, 0.99], alpha_range = [0.1, 1, 2]
beta_range = [-1, 0, 0.8]
    @pytest.mark.parametrize(
        "pct_range,alpha_range,beta_range", [
            pytest.param(
                [.01, .5, .99],
                [.1, 1, 2],
                [-1, 0, .8],
            ),
            pytest.param(
                [.01, .05, .5, .95, .99],
                [.1, .5, 1, 1.5, 2],
                [-.9, -.5, 0, .3, .6, 1],
                marks=pytest.mark.slow
            ),
            pytest.param(
                [.01, .05, .1, .25, .35, .5, .65, .75, .9, .95, .99],
                np.linspace(0.1, 2, 20),
                np.linspace(-1, 1, 21),
                marks=pytest.mark.xslow,
            ),
        ]
    )
    def test_cdf_nolan_samples(
            self, nolan_cdf_sample_data, pct_range, alpha_range, beta_range
    ):
        """ Test cdf values against Nolan's stablec.exe output."""
        data = nolan_cdf_sample_data
        tests = [
            # piecewise generally good accuracy
            [
                'piecewise', 1e-12, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    ~(
                        (
                            (r['alpha'] == 1.) &
                            np.isin(r['beta'], [-0.3, -0.2, -0.1]) &
                            (r['pct'] == 0.01)
                        ) |
                        (
                            (r['alpha'] == 1.) &
                            np.isin(r['beta'], [0.1, 0.2, 0.3]) &
                            (r['pct'] == 0.99)
                        )
                    )
                )
            ],
            # for some points with alpha=1, Nolan's STABLE clearly
            # loses accuracy
            [
                'piecewise', 5e-2, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (
                        (r['alpha'] == 1.) &
                        np.isin(r['beta'], [-0.3, -0.2, -0.1]) &
                        (r['pct'] == 0.01)
                    ) |
                    (
                        (r['alpha'] == 1.) &
                        np.isin(r['beta'], [0.1, 0.2, 0.3]) &
                        (r['pct'] == 0.99)
                    )
                )
            ],
            # fft accuracy poor, very poor alpha < 1
            [
                'fft-simpson', 1e-5, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.7)
                )
            ],
            [
                'fft-simpson', 1e-4, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.5) &
                    (r['alpha'] <= 1.7)
                )
            ],
            [
                'fft-simpson', 1e-3, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.3) &
                    (r['alpha'] <= 1.5)
                )
            ],
            [
                'fft-simpson', 1e-2, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.0) &
                    (r['alpha'] <= 1.3)
                )
            ],
        ]
        for ix, (default_method, rtol,
                 filter_func) in enumerate(tests):
            stats.levy_stable.cdf_default_method = default_method
            subdata = data[filter_func(data)
                           ] if filter_func is not None else data
            with suppress_warnings() as sup:
                sup.record(
                    RuntimeWarning,
                    'Cumulative density calculations experimental for FFT'
                    + ' method. Use piecewise method instead.*'
                )
                p = stats.levy_stable.cdf(
                    subdata['x'],
                    subdata['alpha'],
                    subdata['beta'],
                    scale=1,
                    loc=0
                )
                with np.errstate(over="ignore"):
                    subdata2 = rec_append_fields(
                        subdata,
                        ['calc', 'abserr', 'relerr'],
                        [
                            p,
                            np.abs(p - subdata['p']),
                            np.abs(p - subdata['p']) / np.abs(subdata['p'])
                        ]
                    )
                failures = subdata2[
                  (subdata2['relerr'] >= rtol) |
                  np.isnan(p)
                ]
>               assert_allclose(
                    p,
                    subdata['p'],
                    rtol,
                    err_msg="cdf test %s failed with method '%s'\n%s\n%s" %
                    (ix, default_method, failures.dtype.names, failures),
                    verbose=False
                )
E               AssertionError: 
E               Not equal to tolerance rtol=1e-05, atol=0
E               cdf test 2 failed with method 'fft-simpson'
E               ('x', 'p', 'alpha', 'beta', 'pct', 'calc', 'abserr', 'relerr')
E               [(-3.28995271e+00, 0.01, 2., -1. , 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2., -1. , 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                (-3.28995271e+00, 0.01, 2.,  0. , 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2.,  0. , 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                (-3.28995271e+00, 0.01, 2.,  0.8, 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2.,  0.8, 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                ( 1.47149782e-15, 0.5 , 2., -1. , 0.5 , 0.08088676, 0.41911324, 0.83822649)
E                ( 6.55042902e-17, 0.5 , 2.,  0. , 0.5 , 0.08088676, 0.41911324, 0.83822649)
E                (-1.16864291e-15, 0.5 , 2.,  0.8, 0.5 , 0.13317373, 0.36682627, 0.73365254)]
E               Mismatched elements: 9 / 9 (100%)
E               Max absolute difference: 0.9825569
E               Max relative difference: 0.99248172
scipy/stats/tests/test_distributions.py:4141: AssertionError
__ TestLevyStable.test_cdf_nolan_samples[pct_range1-alpha_range1-beta_range1] __
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.stats.tests.test_distributions.TestLevyStable object at 0x3ff6df70b90>
nolan_cdf_sample_data = rec.array([(-5.54809272e+19, 0.01, 0.1, -1., 0.01),
           (-1.93540945e-07, 0.99, 0.1, -1., 0.99),
           (-5....95, 2. ,  1., 0.95)],
          dtype=[('x', '<f8'), ('p', '<f8'), ('alpha', '<f8'), ('beta', '<f8'), ('pct', '<f8')])
pct_range = [0.01, 0.05, 0.5, 0.95, 0.99], alpha_range = [0.1, 0.5, 1, 1.5, 2]
beta_range = [-0.9, -0.5, 0, 0.3, 0.6, 1]
    @pytest.mark.parametrize(
        "pct_range,alpha_range,beta_range", [
            pytest.param(
                [.01, .5, .99],
                [.1, 1, 2],
                [-1, 0, .8],
            ),
            pytest.param(
                [.01, .05, .5, .95, .99],
                [.1, .5, 1, 1.5, 2],
                [-.9, -.5, 0, .3, .6, 1],
                marks=pytest.mark.slow
            ),
            pytest.param(
                [.01, .05, .1, .25, .35, .5, .65, .75, .9, .95, .99],
                np.linspace(0.1, 2, 20),
                np.linspace(-1, 1, 21),
                marks=pytest.mark.xslow,
            ),
        ]
    )
    def test_cdf_nolan_samples(
            self, nolan_cdf_sample_data, pct_range, alpha_range, beta_range
    ):
        """ Test cdf values against Nolan's stablec.exe output."""
        data = nolan_cdf_sample_data
        tests = [
            # piecewise generally good accuracy
            [
                'piecewise', 1e-12, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    ~(
                        (
                            (r['alpha'] == 1.) &
                            np.isin(r['beta'], [-0.3, -0.2, -0.1]) &
                            (r['pct'] == 0.01)
                        ) |
                        (
                            (r['alpha'] == 1.) &
                            np.isin(r['beta'], [0.1, 0.2, 0.3]) &
                            (r['pct'] == 0.99)
                        )
                    )
                )
            ],
            # for some points with alpha=1, Nolan's STABLE clearly
            # loses accuracy
            [
                'piecewise', 5e-2, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (
                        (r['alpha'] == 1.) &
                        np.isin(r['beta'], [-0.3, -0.2, -0.1]) &
                        (r['pct'] == 0.01)
                    ) |
                    (
                        (r['alpha'] == 1.) &
                        np.isin(r['beta'], [0.1, 0.2, 0.3]) &
                        (r['pct'] == 0.99)
                    )
                )
            ],
            # fft accuracy poor, very poor alpha < 1
            [
                'fft-simpson', 1e-5, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.7)
                )
            ],
            [
                'fft-simpson', 1e-4, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.5) &
                    (r['alpha'] <= 1.7)
                )
            ],
            [
                'fft-simpson', 1e-3, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.3) &
                    (r['alpha'] <= 1.5)
                )
            ],
            [
                'fft-simpson', 1e-2, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.0) &
                    (r['alpha'] <= 1.3)
                )
            ],
        ]
        for ix, (default_method, rtol,
                 filter_func) in enumerate(tests):
            stats.levy_stable.cdf_default_method = default_method
            subdata = data[filter_func(data)
                           ] if filter_func is not None else data
            with suppress_warnings() as sup:
                sup.record(
                    RuntimeWarning,
                    'Cumulative density calculations experimental for FFT'
                    + ' method. Use piecewise method instead.*'
                )
                p = stats.levy_stable.cdf(
                    subdata['x'],
                    subdata['alpha'],
                    subdata['beta'],
                    scale=1,
                    loc=0
                )
                with np.errstate(over="ignore"):
                    subdata2 = rec_append_fields(
                        subdata,
                        ['calc', 'abserr', 'relerr'],
                        [
                            p,
                            np.abs(p - subdata['p']),
                            np.abs(p - subdata['p']) / np.abs(subdata['p'])
                        ]
                    )
                failures = subdata2[
                  (subdata2['relerr'] >= rtol) |
                  np.isnan(p)
                ]
>               assert_allclose(
                    p,
                    subdata['p'],
                    rtol,
                    err_msg="cdf test %s failed with method '%s'\n%s\n%s" %
                    (ix, default_method, failures.dtype.names, failures),
                    verbose=False
                )
E               AssertionError: 
E               Not equal to tolerance rtol=1e-05, atol=0
E               cdf test 2 failed with method 'fft-simpson'
E               ('x', 'p', 'alpha', 'beta', 'pct', 'calc', 'abserr', 'relerr')
E               [(-3.28995271e+00, 0.01, 2., -0.9, 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2., -0.9, 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                (-3.28995271e+00, 0.01, 2., -0.5, 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2., -0.5, 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                (-3.28995271e+00, 0.01, 2.,  0. , 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2.,  0. , 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                (-3.28995271e+00, 0.01, 2.,  0.3, 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2.,  0.3, 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                (-3.28995271e+00, 0.01, 2.,  0.6, 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2.,  0.6, 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                ( 1.25270582e-15, 0.5 , 2., -0.9, 0.5 , 0.08088676, 0.41911324, 0.83822649)
E                ( 7.31638105e-16, 0.5 , 2., -0.5, 0.5 , 0.08088676, 0.41911324, 0.83822649)
E                ( 6.55042902e-17, 0.5 , 2.,  0. , 0.5 , 0.08088676, 0.41911324, 0.83822649)
E                (-3.78584920e-16, 0.5 , 2.,  0.3, 0.5 , 0.13317373, 0.36682627, 0.73365254)
E                (-8.22674129e-16, 0.5 , 2.,  0.6, 0.5 , 0.13317373, 0.36682627, 0.73365254)
E                (-2.32617431e+00, 0.05, 2., -0.9, 0.05, 0.02578472, 0.02421528, 0.48430567)
E                ( 2.32617431e+00, 0.95, 2., -0.9, 0.95, 0.02648234, 0.92351766, 0.97212385)
E                (-2.32617431e+00, 0.05, 2., -0.5, 0.05, 0.02578472, 0.02421528, 0.48430567)
E                ( 2.32617431e+00, 0.95, 2., -0.5, 0.95, 0.02648234, 0.92351766, 0.97212385)
E                (-2.32617431e+00, 0.05, 2.,  0. , 0.05, 0.02578472, 0.02421528, 0.48430567)
E                ( 2.32617431e+00, 0.95, 2.,  0. , 0.95, 0.02648234, 0.92351766, 0.97212385)
E                (-2.32617431e+00, 0.05, 2.,  0.3, 0.05, 0.02578472, 0.02421528, 0.48430567)
E                ( 2.32617431e+00, 0.95, 2.,  0.3, 0.95, 0.02648234, 0.92351766, 0.97212385)
E                (-2.32617431e+00, 0.05, 2.,  0.6, 0.05, 0.02578472, 0.02421528, 0.48430567)
E                ( 2.32617431e+00, 0.95, 2.,  0.6, 0.95, 0.02648234, 0.92351766, 0.97212385)
E                (-3.28995271e+00, 0.01, 2.,  1. , 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2.,  1. , 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                (-1.37941195e-15, 0.5 , 2.,  1. , 0.5 , 0.13317373, 0.36682627, 0.73365254)
E                (-2.32617431e+00, 0.05, 2.,  1. , 0.05, 0.02578472, 0.02421528, 0.48430567)
E                ( 2.32617431e+00, 0.95, 2.,  1. , 0.95, 0.02648234, 0.92351766, 0.97212385)]
E               Mismatched elements: 30 / 30 (100%)
E               Max absolute difference: 0.9825569
E               Max relative difference: 0.99248172
scipy/stats/tests/test_distributions.py:4141: AssertionError

i686 only failures:

_________________________ test_examples[False-float64] _________________________
[gw5] linux -- Python 3.11.1 /usr/bin/python3
dtype = <class 'numpy.float64'>, irl = False
    @pytest.mark.parametrize('dtype', _dtypes)
    @pytest.mark.parametrize('irl', (False, True))
    @pytest.mark.timeout(120)  # True, complex64 > 60 s: prerel deps cov 64bit blas
    def test_examples(dtype, irl):
        # Note: atol for complex64 bumped from 1e-4 to 1e-3 due to test failures
        # with BLIS, Netlib, and MKL+AVX512 - see
        # https://github.com/conda-forge/scipy-feedstock/pull/198#issuecomment-999180432
        atol = {
            np.float32: 1.3e-4,
            np.float64: 1e-9,
            np.complex64: 1e-3,
            np.complex128: 1e-9,
        }[dtype]
    
        path_prefix = os.path.dirname(__file__)
        # Test matrices from `illc1850.coord` and `mhd1280b.cua` distributed with
        # PROPACK 2.1: http://sun.stanford.edu/~rmunk/PROPACK/
        relative_path = "propack_test_data.npz"
        filename = os.path.join(path_prefix, relative_path)
        data = np.load(filename, allow_pickle=True)
    
        if is_complex_type(dtype):
            A = data['A_complex'].item().astype(dtype)
        else:
            A = data['A_real'].item().astype(dtype)
    
        k = 200
        u, s, vh, _ = _svdp(A, k, irl_mode=irl, random_state=0)
    
        # complex example matrix has many repeated singular values, so check only
        # beginning non-repeated singular vectors to avoid permutations
        sv_check = 27 if is_complex_type(dtype) else k
        u = u[:, :sv_check]
        vh = vh[:sv_check, :]
        s = s[:sv_check]
    
        # Check orthogonality of singular vectors
        assert_allclose(np.eye(u.shape[1]), u.conj().T @ u, atol=atol)
        assert_allclose(np.eye(vh.shape[0]), vh @ vh.conj().T, atol=atol)
    
        # Ensure the norm of the difference between the np.linalg.svd and
        # PROPACK reconstructed matrices is small
        u3, s3, vh3 = np.linalg.svd(A.todense())
        u3 = u3[:, :sv_check]
        s3 = s3[:sv_check]
        vh3 = vh3[:sv_check, :]
        A3 = u3 @ np.diag(s3) @ vh3
        recon = u @ np.diag(s) @ vh
>       assert_allclose(np.linalg.norm(A3 - recon), 0, atol=atol)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=1e-09
E       
E       Mismatched elements: 1 / 1 (100%)
E       Max absolute difference: 1.84554848
E       Max relative difference: inf
E        x: array(1.845548)
E        y: array(0)
scipy/sparse/linalg/tests/test_propack.py:161: AssertionError
_________________________ test_examples[True-float32] __________________________
[gw5] linux -- Python 3.11.1 /usr/bin/python3
dtype = <class 'numpy.float32'>, irl = True
    @pytest.mark.parametrize('dtype', _dtypes)
    @pytest.mark.parametrize('irl', (False, True))
    @pytest.mark.timeout(120)  # True, complex64 > 60 s: prerel deps cov 64bit blas
    def test_examples(dtype, irl):
        # Note: atol for complex64 bumped from 1e-4 to 1e-3 due to test failures
        # with BLIS, Netlib, and MKL+AVX512 - see
        # https://github.com/conda-forge/scipy-feedstock/pull/198#issuecomment-999180432
        atol = {
            np.float32: 1.3e-4,
            np.float64: 1e-9,
            np.complex64: 1e-3,
            np.complex128: 1e-9,
        }[dtype]
    
        path_prefix = os.path.dirname(__file__)
        # Test matrices from `illc1850.coord` and `mhd1280b.cua` distributed with
        # PROPACK 2.1: http://sun.stanford.edu/~rmunk/PROPACK/
        relative_path = "propack_test_data.npz"
        filename = os.path.join(path_prefix, relative_path)
        data = np.load(filename, allow_pickle=True)
    
        if is_complex_type(dtype):
            A = data['A_complex'].item().astype(dtype)
        else:
            A = data['A_real'].item().astype(dtype)
    
        k = 200
        u, s, vh, _ = _svdp(A, k, irl_mode=irl, random_state=0)
    
        # complex example matrix has many repeated singular values, so check only
        # beginning non-repeated singular vectors to avoid permutations
        sv_check = 27 if is_complex_type(dtype) else k
        u = u[:, :sv_check]
        vh = vh[:sv_check, :]
        s = s[:sv_check]
    
        # Check orthogonality of singular vectors
        assert_allclose(np.eye(u.shape[1]), u.conj().T @ u, atol=atol)
        assert_allclose(np.eye(vh.shape[0]), vh @ vh.conj().T, atol=atol)
    
        # Ensure the norm of the difference between the np.linalg.svd and
        # PROPACK reconstructed matrices is small
        u3, s3, vh3 = np.linalg.svd(A.todense())
        u3 = u3[:, :sv_check]
        s3 = s3[:sv_check]
        vh3 = vh3[:sv_check, :]
        A3 = u3 @ np.diag(s3) @ vh3
        recon = u @ np.diag(s) @ vh
>       assert_allclose(np.linalg.norm(A3 - recon), 0, atol=atol)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0.00013
E       
E       Mismatched elements: 1 / 1 (100%)
E       Max absolute difference: 1.84416294
E       Max relative difference: inf
E        x: array(1.844163, dtype=float32)
E        y: array(0)
scipy/sparse/linalg/tests/test_propack.py:161: AssertionError
_____________________ sparse/linalg/tests/test_propack.py ______________________
[gw5] linux -- Python 3.11.1 /usr/bin/python3
worker 'gw5' crashed while running 'sparse/linalg/tests/test_propack.py::test_shifts[float32-10]'
_____________________ sparse/linalg/tests/test_propack.py ______________________
[gw6] linux -- Python 3.11.1 /usr/bin/python3
worker 'gw6' crashed while running 'sparse/linalg/tests/test_propack.py::test_shifts[float64-10]'

SciPy/NumPy/Python version information

scipy1.10.0/numpy1.24.1/python3.11.1

[ev-br]

For s390x failures in scipy.interpolate a possible suspect is #17038. It's be useful to try reverting it and see if failures persist.

[psimovec]

For s390x failures in scipy.interpolate a possible suspect is #17038. It's be useful to try reverting it and see if failures persist.

I can try that tomorrow, thank you for the suggestion.

[psimovec]

For s390x failures in scipy.interpolate a possible suspect is #17038. It's be useful to try reverting it and see if failures persist.

I have re-enabled tests for s390x:
not TestInterop and not TestSplder and not test_dblint and not TestUnivariateSpline and not TestLSQBivariateSpline and not TestPPoly

revert patch I have used on v1.10.0 tag:
scipy-1.10.0-deduplicate_splint.patch

Build log for s390x
https://kojipkgs.fedoraproject.org//work/tasks/1705/97041705/build.log
build.log (archived)

...
+ /usr/lib/rpm/rpmuncompress /builddir/build/SOURCES/scipy-1.10.0-deduplicate_splint.patch
... (output ommited)
FAILED scipy/interpolate/tests/test_bsplines.py::TestInterop::test_splint - A...
FAILED scipy/interpolate/tests/test_fitpack.py::TestSplder::test_splantider_vs_splint
FAILED scipy/interpolate/tests/test_fitpack.py::test_dblint - AssertionError: 
FAILED scipy/interpolate/tests/test_fitpack2.py::TestUnivariateSpline::test_derivative_and_antiderivative
FAILED scipy/interpolate/tests/test_fitpack2.py::TestLSQBivariateSpline::test_integral
FAILED scipy/interpolate/tests/test_interpolate.py::TestPPoly::test_integrate
...

The tests are still failing and I didn't notice any difference in error messages either. So it seems to me that #17038 did not cause these errors.

[ev-br]

Ok good to know, thanks!

I'd venture a guess that the next suspect would be the build system change between 1.8.x (distutils) and 1.10.x (meson).
Maybe some build flags differ. I'm not an expert on these matters though.

[tylerjereddy]

@psimovec I checked in an i386/debian container and the test_examples and test_shifts tests are passing there for both the old setup.py and new meson builds of SciPy, at both main and 1.10.0 release point.

It looks to me like you have PROPACK issues specific to your setup given the massive number of warnings like this in your build log:

Warning: call-back function cb_daprod_in___user__routines did not provide return value (index=0, type=float)

I think combining PROPACK with different linalg backends can be painful too--what's going on with the flexiblas you are using? What is the actual linear algebra backend in use there at runtime?

I know @h-vetinari saw a bunch of testing issues on the conda-forge side re: linear algebra backends combined with PROPACK, and some discussion is in gh-15108. For my spot checking here, I just used libopenblas-dev from apt out of convenience.

I suppose if it gets too time consuming in the short term, there is always export SCIPY_USE_PROPACK=0?

[h-vetinari]

I suppose if it gets too time consuming in the short term, there is always export SCIPY_USE_PROPACK=0?

That should be the default?

I know @h-vetinari saw a bunch of testing issues on the conda-forge side re: linear algebra backends combined with PROPACK, and some discussion is in gh-15108.

We keep a list of the failures in https://github.com/conda-forge/scipy-feedstock/blob/main/recipe/meta.yaml, though the only relevant ones (for our CI) are:

# skip a test that fails with MKL + AVX512 (non-AVX512 passes), scipy/scipy#15533
{% set tests_to_skip = tests_to_skip + " or test_x0_equals_Mb[bicgstab]" %}
# scipy/scipy#16927
{% set tests_to_skip = tests_to_skip + " or test_failure_to_run_iterations" %}  # [linux]

And as Tyler mentioned, PROPACK used to be an issues (leading to it being deactivate by default), but with 1.10, those are passing now as well.

We skipped version 1.9, as there were test failures on i686 architecture, as reported in #17213 , and disabling the tests led to another tests failing

(With my scipy hat on,) I'm surprised that one arch - especially one bordering on being obsolete - would block an entire release.

Reproducing Code Example

runtests.py -n

Not sure if there are meaningful differences, but in conda-forge we use

python -c "import scipy, sys; sys.exit(not scipy.test(label='full'))"

(minus some options I'm skipping over).

Error message

A small request: could you please use detail tags for long error/stack traces?

<details>
<summary>Optional</summary>

triple-back-ticked-code-here (newline before and after is necessary)

</details>

This leads to

Optional

triple-back-ticked-code-here (newline before and after is necessary)

[psimovec]

@psimovec I checked in an i386/debian container and the test_examples and test_shifts tests are passing there for both the old setup.py and new meson builds of SciPy, at both main and 1.10.0 release point.

It looks to me like you have PROPACK issues specific to your setup given the massive number of warnings like this in your build log:

Warning: call-back function cb_daprod_in___user__routines did not provide return value (index=0, type=float)

I think combining PROPACK with different linalg backends can be painful too--what's going on with the flexiblas you are using? What is the actual linear algebra backend in use there at runtime?

I know @h-vetinari saw a bunch of testing issues on the conda-forge side re: linear algebra backends combined with PROPACK, and some discussion is in gh-15108. For my spot checking here, I just used libopenblas-dev from apt out of convenience.

I suppose if it gets too time consuming in the short term, there is always export SCIPY_USE_PROPACK=0?

@tylerjereddy I have tried export SCIPY_USE_PROPACK=0:
aside of i686, the difference I have noticed is just ~100 less skipped tests, no effect on failures
x86_64 with enabled tests, for reference: x86_64, and with export SCIPY_USE_PROPACK=0 x86_64 PROPACK=0

on i686, ~30 PROPACK related tests fail additionally, I would expect them to be skipped as PROPACK is disabled.
build-scipy-x86-re-enabled_tests_propack_disabled.log

In our test run we use:
(from scipy.spec, https://src.fedoraproject.org/rpms/scipy/blob/rawhide/f/scipy.spec , PR with update to 1.10.0)

%check
# check against the reference BLAS/LAPACK
export FLEXIBLAS=netlib

I have tried removing export FLEXIBLAS=netlib, thus using the default openblas-openmp linalg backend.
It made the difference with i686, these warnings Warning: call-back function cb_daprod_in___user__routines did not provide return value (index=0, type=float) disappeared, and also these i686 specific failures test_examples and test_shifts. But a new test failure appeared:
test_x0_equals_Mb[bicgstab] x86_no_netlib
and after trying to disable the test another one:
test_cont_basic[500-200-johnsonsu-arg51] x86_no_netlib_disable_bicgstab.log

After disabling this test, there were no more i686 specific test failures. x86_no_netlib_disable_johnsonsu

[tylerjereddy]

@psimovec test_x0_equals_Mb is known to be an issue when swapping linalg backends per the comments about conda-forge above. Perhaps we could add an appropriate skip/xfail there if you need us to?

Can you perhaps summarize what you'd minimally need on the SciPy side to move forward with Fedora? Perhaps we could prioritize things that you'd consider as "definitely blockers," vs. things you may be able to live with for a while at least (more exotic archs + linalg backend combos?).

[tylerjereddy]

Basically, I'm just trying to prioritize a bit so we can look at another bug-fix patch release soon-ish.

[psimovec]

@psimovec test_x0_equals_Mb is known to be an issue when swapping linalg backends per the comments about conda-forge above. Perhaps we could add an appropriate skip/xfail there if you need us to?

Can you perhaps summarize what you'd minimally need on the SciPy side to move forward with Fedora? Perhaps we could prioritize things that you'd consider as "definitely blockers," vs. things you may be able to live with for a while at least (more exotic archs + linalg backend combos?).

@tylerjereddy Skip/xfail for test_x0_equals_Mb would be great as it is know issue, so it is better when tracked by upstream.
With other stuff it is possible to live with for some time, and makes mores sense to skip the remaining tests in our .spec file as it is Fedora specific now. (We can also live with switching testing i686 on default linalg backend instead of netlib - reference implementation, as most likely majority of users there use the default one anyways)

[tylerjereddy]

Does Fedora still officially support 32-bit Linux? Are there Docker containers I could pull from Dockerhub for this?

For Debian it is easy to find: https://hub.docker.com/r/i386/debian/

Maybe I'm just doing the search incorrectly for Fedora?

[psimovec]

I wouldn't say Fedora officially supports 32-bit Linux, but there is support for multilib packages (for example 32-bit for wine and steam + their dependencies). It is encouraged to remove 32-bit architecture from leaf packages that are not a dependence of multilib packages.

I have considered the option of removing i686 scipy, but scipy is recursively a build time dependency of all packages.
For example:
(everything) <- rpm <- rust-rpm-sequoia <- rust-packaging <- pytest <- python-hypothesis <- python-pandas <- scipy

And it doesn't seem realistic in practice to get rid of the 32-bit architecture for scipy at the moment, as it would force at least hundreds of other packages to drop 32-bit support too, even if we cut the dependency tree somewhere.

We use mock [1] to build packages for different architectures, and specifically i686 is possible to do even from x86_64 machine (or container) with Fedora.

[1] https://rpm-software-management.github.io/mock/

[h-vetinari]

pytest <- python-hypothesis <- python-pandas <- scipy

Not sure if this helps within the constraints of your infra, but scipy is an optional dependency of pandas, which is an optional dependency of hypothesis, which is an optional dependency of pytest

[tylerjereddy]

I tried messing around with mock in a fedora:38 podman container, and did end up dropping into a chroot set up for i686, but then I didn't have access to dnf or yum (etc.), so it seems like this is still a more challenging route for debugging interactively than providing i386/fedora for Docker/podman.

Or maybe I still don't fully understand mock, but it does seem a little bit more involved than just getting auto-dropped into a 32-bit world with the matching package repos available to pull in. It does seem to have access to those 32-bit repos when it sets up the chroot though, so maybe there is a path to making this easier...

So, I'm probably not going to go much deeper here for now--I'll start on a hopefully-final series of backports for the next point release. I'd suggest that we really need a super-easy way to debug/reproduce what is going on here if it is to be addressed by the SciPy team.

We may also need to know the version of OpenBLAS used by openblas-openmp package, if it isn't clear from the logs.