BUG: Failing tests on multiple architectures when building scipy 1.10.0 for Fedora

[psimovec]

Describe your issue.

I am updating Fedora package for scipy 1.8.3 -> 1.10.0
(We skipped version 1.9, as there were test failures on i686 architecture, as reported in #17213 , and disabling the tests led to another tests failing)
I did 65+ builds, as disabling some tests led to other tests failing directly or being flaky.

For first build, all of important failed tests and error messages are in i686-build.log and s390x-build.log.

For rest of the builds - rest of archived logs ( as logs in koji will expire in <14 days) : https://drive.google.com/file/d/1op-FgNp1bHHdJB8GAEWowiBrYo5GwDWG/view?usp=sharing (5.3MB compressed, 943MB uncompressed)
(check kojilogs/noarch-<taskID>/<architecture>-*/build.log in the archive)
You can also check scipy-notes.txt to see which tests were disabled at specific moment/in which tasks the tests failed.

Test failures in first build:

https://koji.fedoraproject.org/koji/taskinfo?taskID=96702545
=========================== short test summary info ============================
all architectures s390x-build.log or i686-build.log
FAILED scipy/integrate/tests/test_banded_ode_solvers.py::test_banded_ode_solvers
FAILED scipy/integrate/tests/test_integrate.py::TestOde::test_vode - Assertio...
FAILED scipy/integrate/tests/test_integrate.py::TestOde::test_zvode - Asserti...
FAILED scipy/integrate/tests/test_integrate.py::TestOde::test_concurrent_ok
FAILED scipy/integrate/tests/test_integrate.py::TestComplexOde::test_vode - A...
FAILED scipy/integrate/tests/test_integrate.py::TestVODECheckParameterUse::test_no_params
FAILED scipy/integrate/tests/test_integrate.py::TestVODECheckParameterUse::test_one_scalar_param
FAILED scipy/integrate/tests/test_integrate.py::TestVODECheckParameterUse::test_two_scalar_params
FAILED scipy/integrate/tests/test_integrate.py::TestVODECheckParameterUse::test_vector_param
FAILED scipy/integrate/tests/test_integrate.py::TestVODECheckParameterUse::test_warns_on_failure
FAILED scipy/integrate/tests/test_integrate.py::TestZVODECheckParameterUse::test_no_params
FAILED scipy/integrate/tests/test_integrate.py::TestZVODECheckParameterUse::test_one_scalar_param
FAILED scipy/integrate/tests/test_integrate.py::TestZVODECheckParameterUse::test_two_scalar_params
FAILED scipy/integrate/tests/test_integrate.py::TestZVODECheckParameterUse::test_vector_param
FAILED scipy/integrate/tests/test_integrate.py::TestZVODECheckParameterUse::test_warns_on_failure

i686 unique, same as #17213, i686-build.log
FAILED scipy/sparse/linalg/tests/test_propack.py::test_examples[False-float64]
FAILED scipy/sparse/linalg/tests/test_propack.py::test_examples[True-float32]
FAILED scipy/sparse/linalg/tests/test_propack.py::test_shifts[float32-10]
FAILED scipy/sparse/linalg/tests/test_propack.py::test_shifts[float64-10]

aarch64 + s390x #17859 most likely solved here
FAILED scipy/sparse/linalg/_isolve/tests/test_iterative.py::test_maxiter_worsening[lgmres]

s390x only s390x-build.log, Comment about possible suspect -> #17038
FAILED scipy/interpolate/tests/test_bsplines.py::TestInterop::test_splint - A...
FAILED scipy/interpolate/tests/test_fitpack.py::TestSplder::test_splantider_vs_splint
FAILED scipy/interpolate/tests/test_fitpack.py::test_dblint - AssertionError:
FAILED scipy/interpolate/tests/test_fitpack2.py::TestUnivariateSpline::test_derivative_and_antiderivative
FAILED scipy/interpolate/tests/test_fitpack2.py::TestLSQBivariateSpline::test_integral
FAILED scipy/interpolate/tests/test_interpolate.py::TestPPoly::test_integrate
FAILED scipy/stats/tests/test_distributions.py::TestLevyStable::test_cdf_nolan_samples[pct_range0-alpha_range0-beta_range0]
FAILED scipy/stats/tests/test_distributions.py::TestLevyStable::test_cdf_nolan_samples[pct_range1-alpha_range1-beta_range1]

Test failures in following builds:

These tests also failed at some point, in different builds, when specific tests were enabled/disabled (more info in the attachment)
If you think it is worth to look at them, you can find their error messages in log files I have mentioned. Also, you can check scipy-notes.txt to see in which tasks the tests failed and which tests were disabled/enabled at the moment.

x86_64, s390x, ppc64le
scipy/sparse/linalg/tests/test_propack.py::test_examples[True-complex64]

x86_64
scipy/stats/tests/test_multivariate.py::TestWishart::test_frozen - Ass...

aarch64
scipy/sparse/tests/test_construct.py::TestConstructUtils::test_concatenate_int32_overflow

i686: #17213 possibly the same behaviour as disabling i686 tests in this Issue, on 1.9.3
scipy/stats/tests/test_distributions.py::TestLevyStable::test_cdf_nolan_samples[pct_range0-alpha_range0-beta_range0]
scipy/stats/tests/test_distributions.py::TestNct::test_broadcasting
scipy/sparse/tests/test_base.py::Test64Bit::test_resiliency_all_32[TestBSR-test_mean]
scipy/sparse/tests/test_base.py::Test64Bit::test_resiliency_all_32[TestCSC-test_sum]
scipy/sparse/tests/test_base.py::Test64Bit::test_resiliency_all_32[TestDOK-test_sum]
scipy/sparse/tests/test_base.py::TestCOO::test_mu - AssertionError:
scipy/sparse/tests/test_base.py::TestCOO::test_sum - AssertionError:
scipy/sparse/tests/test_base.py::TestCSR::test_mu - AssertionError:
scipy/sparse/tests/test_base.py::TestCSR::test_sum - AssertionError:
scipy/stats/tests/test_continuous_basic.py::test_methods_with_lists[levy-args62-isf]
scipy/stats/tests/test_continuous_basic.py::test_methods_with_lists[nct-args76-pdf]
scipy/sparse/tests/test_base.py::TestCSR::test_fancy_assignment_dtypes

(following tests also failed at some point, but I don't have the logs right now)
test_resiliency_random[TestCSC-test_sum_dtype]
test_boost
test_entropy_broadcasting
test_svdp

Reproducing Code Example

runtests.py -n

Error message

If you need additional info, search _ test_name in build.log files to see Error messages. (for example search _ test_banded_ode_solvers)
i686-build.log
s390x-build.log
Or you can check rest of the logs in this archive:
scipylogs.tar.xz

Errors messages

___________________________ test_banded_ode_solvers ____________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
    def test_banded_ode_solvers():
        # Test the "lsoda", "vode" and "zvode" solvers of the `ode` class
        # with a system that has a banded Jacobian matrix.
    
        t_exact = np.linspace(0, 1.0, 5)
    
        # --- Real arrays for testing the "lsoda" and "vode" solvers ---
    
        # lband = 2, uband = 1:
        a_real = np.array([[-0.6, 0.1, 0.0, 0.0, 0.0],
                           [0.2, -0.5, 0.9, 0.0, 0.0],
                           [0.1, 0.1, -0.4, 0.1, 0.0],
                           [0.0, 0.3, -0.1, -0.9, -0.3],
                           [0.0, 0.0, 0.1, 0.1, -0.7]])
    
        # lband = 0, uband = 1:
        a_real_upper = np.triu(a_real)
    
        # lband = 2, uband = 0:
        a_real_lower = np.tril(a_real)
    
        # lband = 0, uband = 0:
        a_real_diag = np.triu(a_real_lower)
    
        real_matrices = [a_real, a_real_upper, a_real_lower, a_real_diag]
        real_solutions = []
    
        for a in real_matrices:
            y0 = np.arange(1, a.shape[0] + 1)
            y_exact = _analytical_solution(a, y0, t_exact)
            real_solutions.append((y0, t_exact, y_exact))
    
        def check_real(idx, solver, meth, use_jac, with_jac, banded):
            a = real_matrices[idx]
            y0, t_exact, y_exact = real_solutions[idx]
            t, y = _solve_linear_sys(a, y0,
                                     tend=t_exact[-1],
                                     dt=t_exact[1] - t_exact[0],
                                     solver=solver,
                                     method=meth,
                                     use_jac=use_jac,
                                     with_jacobian=with_jac,
                                     banded=banded)
            assert_allclose(t, t_exact)
            assert_allclose(y, y_exact)
    
        for idx in range(len(real_matrices)):
            p = [['vode', 'lsoda'],  # solver
                 ['bdf', 'adams'],   # method
                 [False, True],      # use_jac
                 [False, True],      # with_jacobian
                 [False, True]]      # banded
            for solver, meth, use_jac, with_jac, banded in itertools.product(*p):
>               check_real(idx, solver, meth, use_jac, with_jac, banded)
scipy/integrate/tests/test_banded_ode_solvers.py:180: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
idx = 0, solver = 'vode', meth = 'bdf', use_jac = False, with_jac = False
banded = False
    def check_real(idx, solver, meth, use_jac, with_jac, banded):
        a = real_matrices[idx]
        y0, t_exact, y_exact = real_solutions[idx]
        t, y = _solve_linear_sys(a, y0,
                                 tend=t_exact[-1],
                                 dt=t_exact[1] - t_exact[0],
                                 solver=solver,
                                 method=meth,
                                 use_jac=use_jac,
                                 with_jacobian=with_jac,
                                 banded=banded)
>       assert_allclose(t, t_exact)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0
E       
E       (shapes (3,), (5,) mismatch)
E        x: array([0.  , 0.25, 0.25])
E        y: array([0.  , 0.25, 0.5 , 0.75, 1.  ])
scipy/integrate/tests/test_banded_ode_solvers.py:170: AssertionError
______________________________ TestOde.test_vode _______________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestOde object at 0x3ff7a204890>
    def test_vode(self):
        # Check the vode solver
        for problem_cls in PROBLEMS:
            problem = problem_cls()
            if problem.cmplx:
                continue
            if not problem.stiff:
>               self._do_problem(problem, 'vode', 'adams')
scipy/integrate/tests/test_integrate.py:99: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestOde object at 0x3ff7a204890>
problem = <scipy.integrate.tests.test_integrate.SimpleOscillator object at 0x3ff6c948fd0>
integrator = 'vode', method = 'adams'
    def _do_problem(self, problem, integrator, method='adams'):
    
        # ode has callback arguments in different order than odeint
        f = lambda t, z: problem.f(z, t)
        jac = None
        if hasattr(problem, 'jac'):
            jac = lambda t, z: problem.jac(z, t)
    
        integrator_params = {}
        if problem.lband is not None or problem.uband is not None:
            integrator_params['uband'] = problem.uband
            integrator_params['lband'] = problem.lband
    
        ig = self.ode_class(f, jac)
        ig.set_integrator(integrator,
                          atol=problem.atol/10,
                          rtol=problem.rtol/10,
                          method=method,
                          **integrator_params)
    
        ig.set_initial_value(problem.z0, t=0.0)
        z = ig.integrate(problem.stop_t)
    
        assert_array_equal(z, ig.y)
        assert_(ig.successful(), (problem, method))
        assert_(ig.get_return_code() > 0, (problem, method))
>       assert_(problem.verify(array([z]), problem.stop_t), (problem, method))
E       AssertionError: (<scipy.integrate.tests.test_integrate.SimpleOscillator object at 0x3ff6c948fd0>, 'adams')
scipy/integrate/tests/test_integrate.py:85: AssertionError
______________________________ TestOde.test_zvode ______________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestOde object at 0x3ff7a204d90>
    def test_zvode(self):
        # Check the zvode solver
        for problem_cls in PROBLEMS:
            problem = problem_cls()
            if not problem.stiff:
>               self._do_problem(problem, 'zvode', 'adams')
scipy/integrate/tests/test_integrate.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestOde object at 0x3ff7a204d90>
problem = <scipy.integrate.tests.test_integrate.SimpleOscillator object at 0x3ff6c92b410>
integrator = 'zvode', method = 'adams'
    def _do_problem(self, problem, integrator, method='adams'):
    
        # ode has callback arguments in different order than odeint
        f = lambda t, z: problem.f(z, t)
        jac = None
        if hasattr(problem, 'jac'):
            jac = lambda t, z: problem.jac(z, t)
    
        integrator_params = {}
        if problem.lband is not None or problem.uband is not None:
            integrator_params['uband'] = problem.uband
            integrator_params['lband'] = problem.lband
    
        ig = self.ode_class(f, jac)
        ig.set_integrator(integrator,
                          atol=problem.atol/10,
                          rtol=problem.rtol/10,
                          method=method,
                          **integrator_params)
    
        ig.set_initial_value(problem.z0, t=0.0)
        z = ig.integrate(problem.stop_t)
    
        assert_array_equal(z, ig.y)
        assert_(ig.successful(), (problem, method))
        assert_(ig.get_return_code() > 0, (problem, method))
>       assert_(problem.verify(array([z]), problem.stop_t), (problem, method))
E       AssertionError: (<scipy.integrate.tests.test_integrate.SimpleOscillator object at 0x3ff6c92b410>, 'adams')
scipy/integrate/tests/test_integrate.py:85: AssertionError
__________________________ TestOde.test_concurrent_ok __________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestOde object at 0x3ff7a289450>
    def test_concurrent_ok(self):
        f = lambda t, y: 1.0
    
        for k in range(3):
            for sol in ('vode', 'zvode', 'lsoda', 'dopri5', 'dop853'):
                r = ode(f).set_integrator(sol)
                r.set_initial_value(0, 0)
    
                r2 = ode(f).set_integrator(sol)
                r2.set_initial_value(0, 0)
    
                r.integrate(r.t + 0.1)
                r2.integrate(r2.t + 0.1)
                r2.integrate(r2.t + 0.1)
    
>               assert_allclose(r.y, 0.1)
E               AssertionError: 
E               Not equal to tolerance rtol=1e-07, atol=0
E               
E               Mismatched elements: 1 / 1 (100%)
E               Max absolute difference: 0.1
E               Max relative difference: 1.
E                x: array([0.])
E                y: array(0.1)
scipy/integrate/tests/test_integrate.py:172: AssertionError
----------------------------- Captured stdout call -----------------------------
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.2500000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.5000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Trouble from DVINDY.  ITASK = I1, TOUT = R1.       
      In above message,  I1 =         1
      In above message,  R1 =  0.5000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1090000000000D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVODE--  Warning: internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1090000000000D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVODE--  Warning: internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVODE--  Warning: internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
___________________________ TestComplexOde.test_vode ___________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestComplexOde object at 0x3ff7a28ab50>
    def test_vode(self):
        # Check the vode solver
        for problem_cls in PROBLEMS:
            problem = problem_cls()
            if not problem.stiff:
>               self._do_problem(problem, 'vode', 'adams')
scipy/integrate/tests/test_integrate.py:201: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestComplexOde object at 0x3ff7a28ab50>
problem = <scipy.integrate.tests.test_integrate.SimpleOscillator object at 0x3ff6e27af10>
integrator = 'vode', method = 'adams'
    def _do_problem(self, problem, integrator, method='adams'):
    
        # ode has callback arguments in different order than odeint
        f = lambda t, z: problem.f(z, t)
        jac = None
        if hasattr(problem, 'jac'):
            jac = lambda t, z: problem.jac(z, t)
    
        integrator_params = {}
        if problem.lband is not None or problem.uband is not None:
            integrator_params['uband'] = problem.uband
            integrator_params['lband'] = problem.lband
    
        ig = self.ode_class(f, jac)
        ig.set_integrator(integrator,
                          atol=problem.atol/10,
                          rtol=problem.rtol/10,
                          method=method,
                          **integrator_params)
    
        ig.set_initial_value(problem.z0, t=0.0)
        z = ig.integrate(problem.stop_t)
    
        assert_array_equal(z, ig.y)
        assert_(ig.successful(), (problem, method))
        assert_(ig.get_return_code() > 0, (problem, method))
>       assert_(problem.verify(array([z]), problem.stop_t), (problem, method))
E       AssertionError: (<scipy.integrate.tests.test_integrate.SimpleOscillator object at 0x3ff6e27af10>, 'adams')
scipy/integrate/tests/test_integrate.py:85: AssertionError
___________________ TestVODECheckParameterUse.test_no_params ___________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a242fd0>
    def test_no_params(self):
        solver = self._get_solver(f, jac)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:602: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a242fd0>
solver = <scipy.integrate._ode.ode object at 0x3ff6e19de50>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1., 0.])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
_______________ TestVODECheckParameterUse.test_one_scalar_param ________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a243610>
    def test_one_scalar_param(self):
        solver = self._get_solver(f1, jac1)
        omega = 1.0
        solver.set_f_params(omega)
        if self.solver_uses_jac:
            solver.set_jac_params(omega)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:610: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a243610>
solver = <scipy.integrate._ode.ode object at 0x3ff6e1beb10>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1., 0.])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
_______________ TestVODECheckParameterUse.test_two_scalar_params _______________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a243190>
    def test_two_scalar_params(self):
        solver = self._get_solver(f2, jac2)
        omega1 = 1.0
        omega2 = 1.0
        solver.set_f_params(omega1, omega2)
        if self.solver_uses_jac:
            solver.set_jac_params(omega1, omega2)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:619: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a243190>
solver = <scipy.integrate._ode.ode object at 0x3ff6e243390>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1., 0.])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
_________________ TestVODECheckParameterUse.test_vector_param __________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a242750>
    def test_vector_param(self):
        solver = self._get_solver(fv, jacv)
        omega = [1.0, 1.0]
        solver.set_f_params(omega)
        if self.solver_uses_jac:
            solver.set_jac_params(omega)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:627: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a242750>
solver = <scipy.integrate._ode.ode object at 0x3ff6c97ee90>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1., 0.])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
_______________ TestVODECheckParameterUse.test_warns_on_failure ________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestVODECheckParameterUse object at 0x3ff7a243b90>
    def test_warns_on_failure(self):
        # Set nsteps small to ensure failure
        solver = self._get_solver(f, jac)
        solver.set_integrator(self.solver_name, nsteps=1)
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
>       assert_warns(UserWarning, solver.integrate, pi)
scipy/integrate/tests/test_integrate.py:635: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.11/site-packages/numpy/testing/_private/utils.py:1799: in assert_warns
    with _assert_warns_context(warning_class, name=func.__name__):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <contextlib._GeneratorContextManager object at 0x3ff6c966790>, typ = None
value = None, traceback = None
    def __exit__(self, typ, value, traceback):
        if typ is None:
            try:
>               next(self.gen)
E               AssertionError: No warning raised when calling integrate
/usr/lib64/python3.11/contextlib.py:144: AssertionError
__________________ TestZVODECheckParameterUse.test_no_params ___________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a27ac50>
    def test_no_params(self):
        solver = self._get_solver(f, jac)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:602: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a27ac50>
solver = <scipy.integrate._ode.ode object at 0x3ff6e1cc5d0>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1.+0.j, 0.+0.j])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
_______________ TestZVODECheckParameterUse.test_one_scalar_param _______________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a28a9d0>
    def test_one_scalar_param(self):
        solver = self._get_solver(f1, jac1)
        omega = 1.0
        solver.set_f_params(omega)
        if self.solver_uses_jac:
            solver.set_jac_params(omega)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:610: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a28a9d0>
solver = <scipy.integrate._ode.ode object at 0x3ff6c878890>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1.+0.j, 0.+0.j])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
----------------------------- Captured stdout call -----------------------------
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.2000000000000D+00
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Trouble from DVINDY.  ITASK = I1, TOUT = R1.       
      In above message,  I1 =         1
      In above message,  R1 =  0.2000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.1090000000000D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.3141592653590D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.3141592653590D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.3141592653590D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.3141592653590D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVODE--  Warning..internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 DVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.3141592653590D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVODE--  Warning: internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVINDY-- T (=R1) illegal      
      In above message,  R1 =  0.3141592653590D+01
       T not in interval TCUR - HU (= R1) to TCUR (=R2)      
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
 ZVODE--  Warning: internal T (=R1) and H (=R2) are
       such that in the machine, T + H = T on the next step  
       (H = step size). solver will continue anyway
      In above,  R1 =  0.0000000000000D+00   R2 =  0.0000000000000D+00
______________ TestZVODECheckParameterUse.test_two_scalar_params _______________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a28ac10>
    def test_two_scalar_params(self):
        solver = self._get_solver(f2, jac2)
        omega1 = 1.0
        omega2 = 1.0
        solver.set_f_params(omega1, omega2)
        if self.solver_uses_jac:
            solver.set_jac_params(omega1, omega2)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:619: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a28ac10>
solver = <scipy.integrate._ode.ode object at 0x3ff6e1ceb10>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1.+0.j, 0.+0.j])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
_________________ TestZVODECheckParameterUse.test_vector_param _________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a205990>
    def test_vector_param(self):
        solver = self._get_solver(fv, jacv)
        omega = [1.0, 1.0]
        solver.set_f_params(omega)
        if self.solver_uses_jac:
            solver.set_jac_params(omega)
>       self._check_solver(solver)
scipy/integrate/tests/test_integrate.py:627: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a205990>
solver = <scipy.integrate._ode.ode object at 0x3ff6c685050>
    def _check_solver(self, solver):
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
        solver.integrate(pi)
>       assert_array_almost_equal(solver.y, [-1.0, 0.0])
E       AssertionError: 
E       Arrays are not almost equal to 6 decimals
E       
E       Mismatched elements: 1 / 2 (50%)
E       Max absolute difference: 2.
E       Max relative difference: 2.
E        x: array([1.+0.j, 0.+0.j])
E        y: array([-1.,  0.])
scipy/integrate/tests/test_integrate.py:598: AssertionError
_______________ TestZVODECheckParameterUse.test_warns_on_failure _______________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.integrate.tests.test_integrate.TestZVODECheckParameterUse object at 0x3ff7a242b10>
    def test_warns_on_failure(self):
        # Set nsteps small to ensure failure
        solver = self._get_solver(f, jac)
        solver.set_integrator(self.solver_name, nsteps=1)
        ic = [1.0, 0.0]
        solver.set_initial_value(ic, 0.0)
>       assert_warns(UserWarning, solver.integrate, pi)
scipy/integrate/tests/test_integrate.py:635: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.11/site-packages/numpy/testing/_private/utils.py:1799: in assert_warns
    with _assert_warns_context(warning_class, name=func.__name__):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <contextlib._GeneratorContextManager object at 0x3ff6c87bf10>, typ = None
value = None, traceback = None
    def __exit__(self, typ, value, traceback):
        if typ is None:
            try:
>               next(self.gen)
E               AssertionError: No warning raised when calling integrate
/usr/lib64/python3.11/contextlib.py:144: AssertionError

s390x only failures:

___________________________ TestInterop.test_splint ____________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.interpolate.tests.test_bsplines.TestInterop object at 0x3ff77aae5d0>
    def test_splint(self):
        # test that splint accepts BSpline objects
        b, b2 = self.b, self.b2
        assert_allclose(splint(0, 1, b),
                        splint(0, 1, b.tck), atol=1e-14)
>       assert_allclose(splint(0, 1, b),
                        b.integrate(0, 1), atol=1e-14)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=1e-14
E       
E       Mismatched elements: 1 / 1 (100%)
E       Max absolute difference: 0.39296037
E       Max relative difference: 0.46697394
E        x: array(0.448543)
E        y: array(0.841504)
scipy/interpolate/tests/test_bsplines.py:873: AssertionError
_____________________ TestSplder.test_splantider_vs_splint _____________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.interpolate.tests.test_fitpack.TestSplder object at 0x3ff779ba110>
    def test_splantider_vs_splint(self):
        # Check antiderivative vs. FITPACK
        spl2 = splantider(self.spl)
    
        # no extrapolation, splint assumes function is zero outside
        # range
        xx = np.linspace(0, 1, 20)
    
        for x1 in xx:
            for x2 in xx:
                y1 = splint(x1, x2, self.spl)
                y2 = splev(x2, spl2) - splev(x1, spl2)
>               assert_allclose(y1, y2)
E               AssertionError: 
E               Not equal to tolerance rtol=1e-07, atol=0
E               
E               Mismatched elements: 1 / 1 (100%)
E               Max absolute difference: 0.01986645
E               Max relative difference: 0.78723695
E                x: array(0.005369)
E                y: array(0.025236)
scipy/interpolate/tests/test_fitpack.py:277: AssertionError
_________________________________ test_dblint __________________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
    def test_dblint():
        # Basic test to see it runs and gives the correct result on a trivial
        # problem. Note that `dblint` is not exposed in the interpolate namespace.
        x = np.linspace(0, 1)
        y = np.linspace(0, 1)
        xx, yy = np.meshgrid(x, y)
        rect = RectBivariateSpline(x, y, 4 * xx * yy)
        tck = list(rect.tck)
        tck.extend(rect.degrees)
    
>       assert_almost_equal(dblint(0, 1, 0, 1, tck), 1)
E       AssertionError: 
E       Arrays are not almost equal to 7 decimals
E        ACTUAL: 0.019024309965098775
E        DESIRED: 1
scipy/interpolate/tests/test_fitpack.py:391: AssertionError
___________ TestUnivariateSpline.test_derivative_and_antiderivative ____________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.interpolate.tests.test_fitpack2.TestUnivariateSpline object at 0x3ff7798acd0>
    def test_derivative_and_antiderivative(self):
        # Thin wrappers to splder/splantider, so light smoke test only.
        x = np.linspace(0, 1, 70)**3
        y = np.cos(x)
    
        spl = UnivariateSpline(x, y, s=0)
        spl2 = spl.antiderivative(2).derivative(2)
        assert_allclose(spl(0.3), spl2(0.3))
    
        spl2 = spl.antiderivative(1)
>       assert_allclose(spl2(0.6) - spl2(0.2),
                        spl.integral(0.2, 0.6))
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0
E       
E       Mismatched elements: 1 / 1 (100%)
E       Max absolute difference: 0.34922361
E       Max relative difference: 20.84975171
E        x: array(0.365973)
E        y: array(0.01675)
scipy/interpolate/tests/test_fitpack2.py:168: AssertionError
_____________________ TestLSQBivariateSpline.test_integral _____________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.interpolate.tests.test_fitpack2.TestLSQBivariateSpline object at 0x3ff777c6550>
    def test_integral(self):
        x = [1,1,1,2,2,2,8,8,8]
        y = [1,2,3,1,2,3,1,2,3]
        z = array([0,7,8,3,4,7,1,3,4])
    
        s = 0.1
        tx = [1+s,3-s]
        ty = [1+s,3-s]
        with suppress_warnings() as sup:
            r = sup.record(UserWarning, "\nThe coefficients of the spline")
            lut = LSQBivariateSpline(x, y, z, tx, ty, kx=1, ky=1)
            assert_equal(len(r), 1)
        tx, ty = lut.get_knots()
        tz = lut(tx, ty)
        trpz = .25*(diff(tx)[:,None]*diff(ty)[None,:]
                    * (tz[:-1,:-1]+tz[1:,:-1]+tz[:-1,1:]+tz[1:,1:])).sum()
    
>       assert_almost_equal(lut.integral(tx[0], tx[-1], ty[0], ty[-1]),
                            trpz)
E       AssertionError: 
E       Arrays are not almost equal to 7 decimals
E        ACTUAL: 75.93932045329475
E        DESIRED: 180.04605637737518
scipy/interpolate/tests/test_fitpack2.py:443: AssertionError
___________________________ TestPPoly.test_integrate ___________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.interpolate.tests.test_interpolate.TestPPoly object at 0x3ff7771e710>
    def test_integrate(self):
        np.random.seed(1234)
        x = np.sort(np.r_[0, np.random.rand(11), 1])
        y = np.random.rand(len(x))
    
        spl = splrep(x, y, s=0, k=5)
        pp = PPoly.from_spline(spl)
    
        a, b = 0.3, 0.9
        ig = pp.integrate(a, b)
    
        ipp = pp.antiderivative()
        assert_allclose(ig, ipp(b) - ipp(a))
>       assert_allclose(ig, splint(a, b, spl))
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0
E       
E       Mismatched elements: 1 / 1 (100%)
E       Max absolute difference: 10.65264483
E       Max relative difference: 1.00916222
E        x: array(-0.096716)
E        y: array(10.555929)
scipy/interpolate/tests/test_interpolate.py:1465: AssertionError
________________________ test_maxiter_worsening[lgmres] ________________________
[gw1] linux -- Python 3.11.1 /usr/bin/python3
solver = <function lgmres at 0x3ff7e409940>
    @pytest.mark.parametrize("solver", [
        pytest.param(gmres, marks=pytest.mark.xfail(platform.machine() == 'aarch64'
                                                    and sys.version_info[1] == 9,
                                                    reason="gh-13019")),
        qmr,
        pytest.param(lgmres, marks=pytest.mark.xfail(platform.machine() == 'ppc64le',
                                                     reason="fails on ppc64le")),
        pytest.param(cgs, marks=pytest.mark.xfail),
        pytest.param(bicg, marks=pytest.mark.xfail),
        pytest.param(bicgstab, marks=pytest.mark.xfail),
        pytest.param(gcrotmk, marks=pytest.mark.xfail),
        pytest.param(tfqmr, marks=pytest.mark.xfail)])
    def test_maxiter_worsening(solver):
        # Check error does not grow (boundlessly) with increasing maxiter.
        # This can occur due to the solvers hitting close to breakdown,
        # which they should detect and halt as necessary.
        # cf. gh-9100
    
        # Singular matrix, rhs numerically not in range
        A = np.array([[-0.1112795288033378, 0, 0, 0.16127952880333685],
                      [0, -0.13627952880333782+6.283185307179586j, 0, 0],
                      [0, 0, -0.13627952880333782-6.283185307179586j, 0],
                      [0.1112795288033368, 0j, 0j, -0.16127952880333785]])
        v = np.ones(4)
        best_error = np.inf
        tol = 7 if platform.machine() == 'aarch64' else 5
    
        for maxiter in range(1, 20):
            x, info = solver(A, v, maxiter=maxiter, tol=1e-8, atol=0)
    
            if info == 0:
                assert_(np.linalg.norm(A.dot(x) - v) <= 1e-8*np.linalg.norm(v))
    
            error = np.linalg.norm(A.dot(x) - v)
            best_error = min(best_error, error)
    
            # Check with slack
>           assert_(error <= tol*best_error)
E           AssertionError
scipy/sparse/linalg/_isolve/tests/test_iterative.py:487: AssertionError
__ TestLevyStable.test_cdf_nolan_samples[pct_range0-alpha_range0-beta_range0] __
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.stats.tests.test_distributions.TestLevyStable object at 0x3ff6df71090>
nolan_cdf_sample_data = rec.array([(-5.54809272e+19, 0.01, 0.1, -1., 0.01),
           (-1.93540945e-07, 0.99, 0.1, -1., 0.99),
           (-5....95, 2. ,  1., 0.95)],
          dtype=[('x', '<f8'), ('p', '<f8'), ('alpha', '<f8'), ('beta', '<f8'), ('pct', '<f8')])
pct_range = [0.01, 0.5, 0.99], alpha_range = [0.1, 1, 2]
beta_range = [-1, 0, 0.8]
    @pytest.mark.parametrize(
        "pct_range,alpha_range,beta_range", [
            pytest.param(
                [.01, .5, .99],
                [.1, 1, 2],
                [-1, 0, .8],
            ),
            pytest.param(
                [.01, .05, .5, .95, .99],
                [.1, .5, 1, 1.5, 2],
                [-.9, -.5, 0, .3, .6, 1],
                marks=pytest.mark.slow
            ),
            pytest.param(
                [.01, .05, .1, .25, .35, .5, .65, .75, .9, .95, .99],
                np.linspace(0.1, 2, 20),
                np.linspace(-1, 1, 21),
                marks=pytest.mark.xslow,
            ),
        ]
    )
    def test_cdf_nolan_samples(
            self, nolan_cdf_sample_data, pct_range, alpha_range, beta_range
    ):
        """ Test cdf values against Nolan's stablec.exe output."""
        data = nolan_cdf_sample_data
        tests = [
            # piecewise generally good accuracy
            [
                'piecewise', 1e-12, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    ~(
                        (
                            (r['alpha'] == 1.) &
                            np.isin(r['beta'], [-0.3, -0.2, -0.1]) &
                            (r['pct'] == 0.01)
                        ) |
                        (
                            (r['alpha'] == 1.) &
                            np.isin(r['beta'], [0.1, 0.2, 0.3]) &
                            (r['pct'] == 0.99)
                        )
                    )
                )
            ],
            # for some points with alpha=1, Nolan's STABLE clearly
            # loses accuracy
            [
                'piecewise', 5e-2, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (
                        (r['alpha'] == 1.) &
                        np.isin(r['beta'], [-0.3, -0.2, -0.1]) &
                        (r['pct'] == 0.01)
                    ) |
                    (
                        (r['alpha'] == 1.) &
                        np.isin(r['beta'], [0.1, 0.2, 0.3]) &
                        (r['pct'] == 0.99)
                    )
                )
            ],
            # fft accuracy poor, very poor alpha < 1
            [
                'fft-simpson', 1e-5, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.7)
                )
            ],
            [
                'fft-simpson', 1e-4, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.5) &
                    (r['alpha'] <= 1.7)
                )
            ],
            [
                'fft-simpson', 1e-3, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.3) &
                    (r['alpha'] <= 1.5)
                )
            ],
            [
                'fft-simpson', 1e-2, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.0) &
                    (r['alpha'] <= 1.3)
                )
            ],
        ]
        for ix, (default_method, rtol,
                 filter_func) in enumerate(tests):
            stats.levy_stable.cdf_default_method = default_method
            subdata = data[filter_func(data)
                           ] if filter_func is not None else data
            with suppress_warnings() as sup:
                sup.record(
                    RuntimeWarning,
                    'Cumulative density calculations experimental for FFT'
                    + ' method. Use piecewise method instead.*'
                )
                p = stats.levy_stable.cdf(
                    subdata['x'],
                    subdata['alpha'],
                    subdata['beta'],
                    scale=1,
                    loc=0
                )
                with np.errstate(over="ignore"):
                    subdata2 = rec_append_fields(
                        subdata,
                        ['calc', 'abserr', 'relerr'],
                        [
                            p,
                            np.abs(p - subdata['p']),
                            np.abs(p - subdata['p']) / np.abs(subdata['p'])
                        ]
                    )
                failures = subdata2[
                  (subdata2['relerr'] >= rtol) |
                  np.isnan(p)
                ]
>               assert_allclose(
                    p,
                    subdata['p'],
                    rtol,
                    err_msg="cdf test %s failed with method '%s'\n%s\n%s" %
                    (ix, default_method, failures.dtype.names, failures),
                    verbose=False
                )
E               AssertionError: 
E               Not equal to tolerance rtol=1e-05, atol=0
E               cdf test 2 failed with method 'fft-simpson'
E               ('x', 'p', 'alpha', 'beta', 'pct', 'calc', 'abserr', 'relerr')
E               [(-3.28995271e+00, 0.01, 2., -1. , 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2., -1. , 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                (-3.28995271e+00, 0.01, 2.,  0. , 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2.,  0. , 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                (-3.28995271e+00, 0.01, 2.,  0.8, 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2.,  0.8, 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                ( 1.47149782e-15, 0.5 , 2., -1. , 0.5 , 0.08088676, 0.41911324, 0.83822649)
E                ( 6.55042902e-17, 0.5 , 2.,  0. , 0.5 , 0.08088676, 0.41911324, 0.83822649)
E                (-1.16864291e-15, 0.5 , 2.,  0.8, 0.5 , 0.13317373, 0.36682627, 0.73365254)]
E               Mismatched elements: 9 / 9 (100%)
E               Max absolute difference: 0.9825569
E               Max relative difference: 0.99248172
scipy/stats/tests/test_distributions.py:4141: AssertionError
__ TestLevyStable.test_cdf_nolan_samples[pct_range1-alpha_range1-beta_range1] __
[gw1] linux -- Python 3.11.1 /usr/bin/python3
self = <scipy.stats.tests.test_distributions.TestLevyStable object at 0x3ff6df70b90>
nolan_cdf_sample_data = rec.array([(-5.54809272e+19, 0.01, 0.1, -1., 0.01),
           (-1.93540945e-07, 0.99, 0.1, -1., 0.99),
           (-5....95, 2. ,  1., 0.95)],
          dtype=[('x', '<f8'), ('p', '<f8'), ('alpha', '<f8'), ('beta', '<f8'), ('pct', '<f8')])
pct_range = [0.01, 0.05, 0.5, 0.95, 0.99], alpha_range = [0.1, 0.5, 1, 1.5, 2]
beta_range = [-0.9, -0.5, 0, 0.3, 0.6, 1]
    @pytest.mark.parametrize(
        "pct_range,alpha_range,beta_range", [
            pytest.param(
                [.01, .5, .99],
                [.1, 1, 2],
                [-1, 0, .8],
            ),
            pytest.param(
                [.01, .05, .5, .95, .99],
                [.1, .5, 1, 1.5, 2],
                [-.9, -.5, 0, .3, .6, 1],
                marks=pytest.mark.slow
            ),
            pytest.param(
                [.01, .05, .1, .25, .35, .5, .65, .75, .9, .95, .99],
                np.linspace(0.1, 2, 20),
                np.linspace(-1, 1, 21),
                marks=pytest.mark.xslow,
            ),
        ]
    )
    def test_cdf_nolan_samples(
            self, nolan_cdf_sample_data, pct_range, alpha_range, beta_range
    ):
        """ Test cdf values against Nolan's stablec.exe output."""
        data = nolan_cdf_sample_data
        tests = [
            # piecewise generally good accuracy
            [
                'piecewise', 1e-12, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    ~(
                        (
                            (r['alpha'] == 1.) &
                            np.isin(r['beta'], [-0.3, -0.2, -0.1]) &
                            (r['pct'] == 0.01)
                        ) |
                        (
                            (r['alpha'] == 1.) &
                            np.isin(r['beta'], [0.1, 0.2, 0.3]) &
                            (r['pct'] == 0.99)
                        )
                    )
                )
            ],
            # for some points with alpha=1, Nolan's STABLE clearly
            # loses accuracy
            [
                'piecewise', 5e-2, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (
                        (r['alpha'] == 1.) &
                        np.isin(r['beta'], [-0.3, -0.2, -0.1]) &
                        (r['pct'] == 0.01)
                    ) |
                    (
                        (r['alpha'] == 1.) &
                        np.isin(r['beta'], [0.1, 0.2, 0.3]) &
                        (r['pct'] == 0.99)
                    )
                )
            ],
            # fft accuracy poor, very poor alpha < 1
            [
                'fft-simpson', 1e-5, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.7)
                )
            ],
            [
                'fft-simpson', 1e-4, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.5) &
                    (r['alpha'] <= 1.7)
                )
            ],
            [
                'fft-simpson', 1e-3, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.3) &
                    (r['alpha'] <= 1.5)
                )
            ],
            [
                'fft-simpson', 1e-2, lambda r: (
                    np.isin(r['pct'], pct_range) &
                    np.isin(r['alpha'], alpha_range) &
                    np.isin(r['beta'], beta_range) &
                    (r['alpha'] > 1.0) &
                    (r['alpha'] <= 1.3)
                )
            ],
        ]
        for ix, (default_method, rtol,
                 filter_func) in enumerate(tests):
            stats.levy_stable.cdf_default_method = default_method
            subdata = data[filter_func(data)
                           ] if filter_func is not None else data
            with suppress_warnings() as sup:
                sup.record(
                    RuntimeWarning,
                    'Cumulative density calculations experimental for FFT'
                    + ' method. Use piecewise method instead.*'
                )
                p = stats.levy_stable.cdf(
                    subdata['x'],
                    subdata['alpha'],
                    subdata['beta'],
                    scale=1,
                    loc=0
                )
                with np.errstate(over="ignore"):
                    subdata2 = rec_append_fields(
                        subdata,
                        ['calc', 'abserr', 'relerr'],
                        [
                            p,
                            np.abs(p - subdata['p']),
                            np.abs(p - subdata['p']) / np.abs(subdata['p'])
                        ]
                    )
                failures = subdata2[
                  (subdata2['relerr'] >= rtol) |
                  np.isnan(p)
                ]
>               assert_allclose(
                    p,
                    subdata['p'],
                    rtol,
                    err_msg="cdf test %s failed with method '%s'\n%s\n%s" %
                    (ix, default_method, failures.dtype.names, failures),
                    verbose=False
                )
E               AssertionError: 
E               Not equal to tolerance rtol=1e-05, atol=0
E               cdf test 2 failed with method 'fft-simpson'
E               ('x', 'p', 'alpha', 'beta', 'pct', 'calc', 'abserr', 'relerr')
E               [(-3.28995271e+00, 0.01, 2., -0.9, 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2., -0.9, 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                (-3.28995271e+00, 0.01, 2., -0.5, 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2., -0.5, 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                (-3.28995271e+00, 0.01, 2.,  0. , 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2.,  0. , 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                (-3.28995271e+00, 0.01, 2.,  0.3, 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2.,  0.3, 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                (-3.28995271e+00, 0.01, 2.,  0.6, 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2.,  0.6, 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                ( 1.25270582e-15, 0.5 , 2., -0.9, 0.5 , 0.08088676, 0.41911324, 0.83822649)
E                ( 7.31638105e-16, 0.5 , 2., -0.5, 0.5 , 0.08088676, 0.41911324, 0.83822649)
E                ( 6.55042902e-17, 0.5 , 2.,  0. , 0.5 , 0.08088676, 0.41911324, 0.83822649)
E                (-3.78584920e-16, 0.5 , 2.,  0.3, 0.5 , 0.13317373, 0.36682627, 0.73365254)
E                (-8.22674129e-16, 0.5 , 2.,  0.6, 0.5 , 0.13317373, 0.36682627, 0.73365254)
E                (-2.32617431e+00, 0.05, 2., -0.9, 0.05, 0.02578472, 0.02421528, 0.48430567)
E                ( 2.32617431e+00, 0.95, 2., -0.9, 0.95, 0.02648234, 0.92351766, 0.97212385)
E                (-2.32617431e+00, 0.05, 2., -0.5, 0.05, 0.02578472, 0.02421528, 0.48430567)
E                ( 2.32617431e+00, 0.95, 2., -0.5, 0.95, 0.02648234, 0.92351766, 0.97212385)
E                (-2.32617431e+00, 0.05, 2.,  0. , 0.05, 0.02578472, 0.02421528, 0.48430567)
E                ( 2.32617431e+00, 0.95, 2.,  0. , 0.95, 0.02648234, 0.92351766, 0.97212385)
E                (-2.32617431e+00, 0.05, 2.,  0.3, 0.05, 0.02578472, 0.02421528, 0.48430567)
E                ( 2.32617431e+00, 0.95, 2.,  0.3, 0.95, 0.02648234, 0.92351766, 0.97212385)
E                (-2.32617431e+00, 0.05, 2.,  0.6, 0.05, 0.02578472, 0.02421528, 0.48430567)
E                ( 2.32617431e+00, 0.95, 2.,  0.6, 0.95, 0.02648234, 0.92351766, 0.97212385)
E                (-3.28995271e+00, 0.01, 2.,  1. , 0.01, 0.00602404, 0.00397596, 0.3975957 )
E                ( 3.28995271e+00, 0.99, 2.,  1. , 0.99, 0.0074431 , 0.9825569 , 0.99248172)
E                (-1.37941195e-15, 0.5 , 2.,  1. , 0.5 , 0.13317373, 0.36682627, 0.73365254)
E                (-2.32617431e+00, 0.05, 2.,  1. , 0.05, 0.02578472, 0.02421528, 0.48430567)
E                ( 2.32617431e+00, 0.95, 2.,  1. , 0.95, 0.02648234, 0.92351766, 0.97212385)]
E               Mismatched elements: 30 / 30 (100%)
E               Max absolute difference: 0.9825569
E               Max relative difference: 0.99248172
scipy/stats/tests/test_distributions.py:4141: AssertionError

i686 only failures:

_________________________ test_examples[False-float64] _________________________
[gw5] linux -- Python 3.11.1 /usr/bin/python3
dtype = <class 'numpy.float64'>, irl = False
    @pytest.mark.parametrize('dtype', _dtypes)
    @pytest.mark.parametrize('irl', (False, True))
    @pytest.mark.timeout(120)  # True, complex64 > 60 s: prerel deps cov 64bit blas
    def test_examples(dtype, irl):
        # Note: atol for complex64 bumped from 1e-4 to 1e-3 due to test failures
        # with BLIS, Netlib, and MKL+AVX512 - see
        # https://github.com/conda-forge/scipy-feedstock/pull/198#issuecomment-999180432
        atol = {
            np.float32: 1.3e-4,
            np.float64: 1e-9,
            np.complex64: 1e-3,
            np.complex128: 1e-9,
        }[dtype]
    
        path_prefix = os.path.dirname(__file__)
        # Test matrices from `illc1850.coord` and `mhd1280b.cua` distributed with
        # PROPACK 2.1: http://sun.stanford.edu/~rmunk/PROPACK/
        relative_path = "propack_test_data.npz"
        filename = os.path.join(path_prefix, relative_path)
        data = np.load(filename, allow_pickle=True)
    
        if is_complex_type(dtype):
            A = data['A_complex'].item().astype(dtype)
        else:
            A = data['A_real'].item().astype(dtype)
    
        k = 200
        u, s, vh, _ = _svdp(A, k, irl_mode=irl, random_state=0)
    
        # complex example matrix has many repeated singular values, so check only
        # beginning non-repeated singular vectors to avoid permutations
        sv_check = 27 if is_complex_type(dtype) else k
        u = u[:, :sv_check]
        vh = vh[:sv_check, :]
        s = s[:sv_check]
    
        # Check orthogonality of singular vectors
        assert_allclose(np.eye(u.shape[1]), u.conj().T @ u, atol=atol)
        assert_allclose(np.eye(vh.shape[0]), vh @ vh.conj().T, atol=atol)
    
        # Ensure the norm of the difference between the np.linalg.svd and
        # PROPACK reconstructed matrices is small
        u3, s3, vh3 = np.linalg.svd(A.todense())
        u3 = u3[:, :sv_check]
        s3 = s3[:sv_check]
        vh3 = vh3[:sv_check, :]
        A3 = u3 @ np.diag(s3) @ vh3
        recon = u @ np.diag(s) @ vh
>       assert_allclose(np.linalg.norm(A3 - recon), 0, atol=atol)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=1e-09
E       
E       Mismatched elements: 1 / 1 (100%)
E       Max absolute difference: 1.84554848
E       Max relative difference: inf
E        x: array(1.845548)
E        y: array(0)
scipy/sparse/linalg/tests/test_propack.py:161: AssertionError
_________________________ test_examples[True-float32] __________________________
[gw5] linux -- Python 3.11.1 /usr/bin/python3
dtype = <class 'numpy.float32'>, irl = True
    @pytest.mark.parametrize('dtype', _dtypes)
    @pytest.mark.parametrize('irl', (False, True))
    @pytest.mark.timeout(120)  # True, complex64 > 60 s: prerel deps cov 64bit blas
    def test_examples(dtype, irl):
        # Note: atol for complex64 bumped from 1e-4 to 1e-3 due to test failures
        # with BLIS, Netlib, and MKL+AVX512 - see
        # https://github.com/conda-forge/scipy-feedstock/pull/198#issuecomment-999180432
        atol = {
            np.float32: 1.3e-4,
            np.float64: 1e-9,
            np.complex64: 1e-3,
            np.complex128: 1e-9,
        }[dtype]
    
        path_prefix = os.path.dirname(__file__)
        # Test matrices from `illc1850.coord` and `mhd1280b.cua` distributed with
        # PROPACK 2.1: http://sun.stanford.edu/~rmunk/PROPACK/
        relative_path = "propack_test_data.npz"
        filename = os.path.join(path_prefix, relative_path)
        data = np.load(filename, allow_pickle=True)
    
        if is_complex_type(dtype):
            A = data['A_complex'].item().astype(dtype)
        else:
            A = data['A_real'].item().astype(dtype)
    
        k = 200
        u, s, vh, _ = _svdp(A, k, irl_mode=irl, random_state=0)
    
        # complex example matrix has many repeated singular values, so check only
        # beginning non-repeated singular vectors to avoid permutations
        sv_check = 27 if is_complex_type(dtype) else k
        u = u[:, :sv_check]
        vh = vh[:sv_check, :]
        s = s[:sv_check]
    
        # Check orthogonality of singular vectors
        assert_allclose(np.eye(u.shape[1]), u.conj().T @ u, atol=atol)
        assert_allclose(np.eye(vh.shape[0]), vh @ vh.conj().T, atol=atol)
    
        # Ensure the norm of the difference between the np.linalg.svd and
        # PROPACK reconstructed matrices is small
        u3, s3, vh3 = np.linalg.svd(A.todense())
        u3 = u3[:, :sv_check]
        s3 = s3[:sv_check]
        vh3 = vh3[:sv_check, :]
        A3 = u3 @ np.diag(s3) @ vh3
        recon = u @ np.diag(s) @ vh
>       assert_allclose(np.linalg.norm(A3 - recon), 0, atol=atol)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-07, atol=0.00013
E       
E       Mismatched elements: 1 / 1 (100%)
E       Max absolute difference: 1.84416294
E       Max relative difference: inf
E        x: array(1.844163, dtype=float32)
E        y: array(0)
scipy/sparse/linalg/tests/test_propack.py:161: AssertionError
_____________________ sparse/linalg/tests/test_propack.py ______________________
[gw5] linux -- Python 3.11.1 /usr/bin/python3
worker 'gw5' crashed while running 'sparse/linalg/tests/test_propack.py::test_shifts[float32-10]'
_____________________ sparse/linalg/tests/test_propack.py ______________________
[gw6] linux -- Python 3.11.1 /usr/bin/python3
worker 'gw6' crashed while running 'sparse/linalg/tests/test_propack.py::test_shifts[float64-10]'

SciPy/NumPy/Python version information

scipy1.10.0/numpy1.24.1/python3.11.1

[ev-br]

For s390x failures in scipy.interpolate a possible suspect is #17038. It's be useful to try reverting it and see if failures persist.

[psimovec]

For s390x failures in scipy.interpolate a possible suspect is #17038. It's be useful to try reverting it and see if failures persist.

I can try that tomorrow, thank you for the suggestion.

[psimovec]

For s390x failures in scipy.interpolate a possible suspect is #17038. It's be useful to try reverting it and see if failures persist.

I have re-enabled tests for s390x:
not TestInterop and not TestSplder and not test_dblint and not TestUnivariateSpline and not TestLSQBivariateSpline and not TestPPoly

revert patch I have used on v1.10.0 tag:
scipy-1.10.0-deduplicate_splint.patch

Build log for s390x
https://kojipkgs.fedoraproject.org//work/tasks/1705/97041705/build.log
build.log (archived)

...
+ /usr/lib/rpm/rpmuncompress /builddir/build/SOURCES/scipy-1.10.0-deduplicate_splint.patch
... (output ommited)
FAILED scipy/interpolate/tests/test_bsplines.py::TestInterop::test_splint - A...
FAILED scipy/interpolate/tests/test_fitpack.py::TestSplder::test_splantider_vs_splint
FAILED scipy/interpolate/tests/test_fitpack.py::test_dblint - AssertionError: 
FAILED scipy/interpolate/tests/test_fitpack2.py::TestUnivariateSpline::test_derivative_and_antiderivative
FAILED scipy/interpolate/tests/test_fitpack2.py::TestLSQBivariateSpline::test_integral
FAILED scipy/interpolate/tests/test_interpolate.py::TestPPoly::test_integrate
...

The tests are still failing and I didn't notice any difference in error messages either. So it seems to me that #17038 did not cause these errors.

[ev-br]

Ok good to know, thanks!

I'd venture a guess that the next suspect would be the build system change between 1.8.x (distutils) and 1.10.x (meson).
Maybe some build flags differ. I'm not an expert on these matters though.

[tylerjereddy]

@psimovec I checked in an i386/debian container and the test_examples and test_shifts tests are passing there for both the old setup.py and new meson builds of SciPy, at both main and 1.10.0 release point.

It looks to me like you have PROPACK issues specific to your setup given the massive number of warnings like this in your build log:

Warning: call-back function cb_daprod_in___user__routines did not provide return value (index=0, type=float)

I think combining PROPACK with different linalg backends can be painful too--what's going on with the flexiblas you are using? What is the actual linear algebra backend in use there at runtime?

I know @h-vetinari saw a bunch of testing issues on the conda-forge side re: linear algebra backends combined with PROPACK, and some discussion is in gh-15108. For my spot checking here, I just used libopenblas-dev from apt out of convenience.

I suppose if it gets too time consuming in the short term, there is always export SCIPY_USE_PROPACK=0?

[h-vetinari]

I suppose if it gets too time consuming in the short term, there is always export SCIPY_USE_PROPACK=0?

That should be the default?

I know @h-vetinari saw a bunch of testing issues on the conda-forge side re: linear algebra backends combined with PROPACK, and some discussion is in gh-15108.

We keep a list of the failures in https://github.com/conda-forge/scipy-feedstock/blob/main/recipe/meta.yaml, though the only relevant ones (for our CI) are:

# skip a test that fails with MKL + AVX512 (non-AVX512 passes), scipy/scipy#15533
{% set tests_to_skip = tests_to_skip + " or test_x0_equals_Mb[bicgstab]" %}
# scipy/scipy#16927
{% set tests_to_skip = tests_to_skip + " or test_failure_to_run_iterations" %}  # [linux]

And as Tyler mentioned, PROPACK used to be an issues (leading to it being deactivate by default), but with 1.10, those are passing now as well.

We skipped version 1.9, as there were test failures on i686 architecture, as reported in #17213 , and disabling the tests led to another tests failing

(With my scipy hat on,) I'm surprised that one arch - especially one bordering on being obsolete - would block an entire release.

Reproducing Code Example

runtests.py -n

Not sure if there are meaningful differences, but in conda-forge we use

python -c "import scipy, sys; sys.exit(not scipy.test(label='full'))"

(minus some options I'm skipping over).

Error message

A small request: could you please use detail tags for long error/stack traces?

<details>
<summary>Optional</summary>

triple-back-ticked-code-here (newline before and after is necessary)

</details>

This leads to

Optional

triple-back-ticked-code-here (newline before and after is necessary)

[psimovec]

@psimovec I checked in an i386/debian container and the test_examples and test_shifts tests are passing there for both the old setup.py and new meson builds of SciPy, at both main and 1.10.0 release point.

It looks to me like you have PROPACK issues specific to your setup given the massive number of warnings like this in your build log:

Warning: call-back function cb_daprod_in___user__routines did not provide return value (index=0, type=float)

I think combining PROPACK with different linalg backends can be painful too--what's going on with the flexiblas you are using? What is the actual linear algebra backend in use there at runtime?

I know @h-vetinari saw a bunch of testing issues on the conda-forge side re: linear algebra backends combined with PROPACK, and some discussion is in gh-15108. For my spot checking here, I just used libopenblas-dev from apt out of convenience.

I suppose if it gets too time consuming in the short term, there is always export SCIPY_USE_PROPACK=0?

@tylerjereddy I have tried export SCIPY_USE_PROPACK=0:
aside of i686, the difference I have noticed is just ~100 less skipped tests, no effect on failures
x86_64 with enabled tests, for reference: x86_64, and with export SCIPY_USE_PROPACK=0 x86_64 PROPACK=0

on i686, ~30 PROPACK related tests fail additionally, I would expect them to be skipped as PROPACK is disabled.
build-scipy-x86-re-enabled_tests_propack_disabled.log

In our test run we use:
(from scipy.spec, https://src.fedoraproject.org/rpms/scipy/blob/rawhide/f/scipy.spec , PR with update to 1.10.0)

%check
# check against the reference BLAS/LAPACK
export FLEXIBLAS=netlib

I have tried removing export FLEXIBLAS=netlib, thus using the default openblas-openmp linalg backend.
It made the difference with i686, these warnings Warning: call-back function cb_daprod_in___user__routines did not provide return value (index=0, type=float) disappeared, and also these i686 specific failures test_examples and test_shifts. But a new test failure appeared:
test_x0_equals_Mb[bicgstab] x86_no_netlib
and after trying to disable the test another one:
test_cont_basic[500-200-johnsonsu-arg51] x86_no_netlib_disable_bicgstab.log

After disabling this test, there were no more i686 specific test failures. x86_no_netlib_disable_johnsonsu

[tylerjereddy]

@psimovec test_x0_equals_Mb is known to be an issue when swapping linalg backends per the comments about conda-forge above. Perhaps we could add an appropriate skip/xfail there if you need us to?

Can you perhaps summarize what you'd minimally need on the SciPy side to move forward with Fedora? Perhaps we could prioritize things that you'd consider as "definitely blockers," vs. things you may be able to live with for a while at least (more exotic archs + linalg backend combos?).

[tylerjereddy]

Basically, I'm just trying to prioritize a bit so we can look at another bug-fix patch release soon-ish.

[psimovec]

@psimovec test_x0_equals_Mb is known to be an issue when swapping linalg backends per the comments about conda-forge above. Perhaps we could add an appropriate skip/xfail there if you need us to?

Can you perhaps summarize what you'd minimally need on the SciPy side to move forward with Fedora? Perhaps we could prioritize things that you'd consider as "definitely blockers," vs. things you may be able to live with for a while at least (more exotic archs + linalg backend combos?).

@tylerjereddy Skip/xfail for test_x0_equals_Mb would be great as it is know issue, so it is better when tracked by upstream.
With other stuff it is possible to live with for some time, and makes mores sense to skip the remaining tests in our .spec file as it is Fedora specific now. (We can also live with switching testing i686 on default linalg backend instead of netlib - reference implementation, as most likely majority of users there use the default one anyways)

[tylerjereddy]

Does Fedora still officially support 32-bit Linux? Are there Docker containers I could pull from Dockerhub for this?

For Debian it is easy to find: https://hub.docker.com/r/i386/debian/

Maybe I'm just doing the search incorrectly for Fedora?

[psimovec]

I wouldn't say Fedora officially supports 32-bit Linux, but there is support for multilib packages (for example 32-bit for wine and steam + their dependencies). It is encouraged to remove 32-bit architecture from leaf packages that are not a dependence of multilib packages.

I have considered the option of removing i686 scipy, but scipy is recursively a build time dependency of all packages.
For example:
(everything) <- rpm <- rust-rpm-sequoia <- rust-packaging <- pytest <- python-hypothesis <- python-pandas <- scipy

And it doesn't seem realistic in practice to get rid of the 32-bit architecture for scipy at the moment, as it would force at least hundreds of other packages to drop 32-bit support too, even if we cut the dependency tree somewhere.

We use mock [1] to build packages for different architectures, and specifically i686 is possible to do even from x86_64 machine (or container) with Fedora.

[1] https://rpm-software-management.github.io/mock/

[h-vetinari]

pytest <- python-hypothesis <- python-pandas <- scipy

Not sure if this helps within the constraints of your infra, but scipy is an optional dependency of pandas, which is an optional dependency of hypothesis, which is an optional dependency of pytest

[tylerjereddy]

I tried messing around with mock in a fedora:38 podman container, and did end up dropping into a chroot set up for i686, but then I didn't have access to dnf or yum (etc.), so it seems like this is still a more challenging route for debugging interactively than providing i386/fedora for Docker/podman.

Or maybe I still don't fully understand mock, but it does seem a little bit more involved than just getting auto-dropped into a 32-bit world with the matching package repos available to pull in. It does seem to have access to those 32-bit repos when it sets up the chroot though, so maybe there is a path to making this easier...

So, I'm probably not going to go much deeper here for now--I'll start on a hopefully-final series of backports for the next point release. I'd suggest that we really need a super-easy way to debug/reproduce what is going on here if it is to be addressed by the SciPy team.

We may also need to know the version of OpenBLAS used by openblas-openmp package, if it isn't clear from the logs.

[stratakis]

An additional possibly relevant test failure I've noticed with 1.10.0 on an i686 RHEL8 system, with Python 3.11.2 and Numpy 1.23.5 is the TestKSTwoSamples.test_gh12218 failing with:

>      cddiffs = cdf1 - cdf2
E       numpy.core._exceptions._ArrayMemoryError: Unable to allocate 32.0 MiB for an array with shape (4194304,) and data type float64
scipy/stats/_stats_py.py:8119: MemoryError

Full snippet:

________________________ TestKSTwoSamples.test_gh12218 _________________________
self = <scipy.stats.tests.test_stats.TestKSTwoSamples object at 0x8ac16fd0>
    @pytest.mark.slow
    def test_gh12218(self):
        """Ensure gh-12218 is fixed."""
        # gh-1228 triggered a TypeError calculating sqrt(n1*n2*(n1+n2)).
        # n1, n2 both large integers, the product exceeded 2^64
        np.random.seed(12345678)
        n1 = 2097152  # 2*^21
        rvs1 = stats.uniform.rvs(size=n1, loc=0., scale=1)
        rvs2 = rvs1 + 1  # Exact value of rvs2 doesn't matter.
>       stats.ks_2samp(rvs1, rvs2, alternative='greater', mode='asymp')
scipy/stats/tests/test_stats.py:4221: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
scipy/_lib/_util.py:700: in wrapper
    return fun(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
data1 = array([1.31659993e-06, 1.38255591e-06, 1.56707242e-06, ...,
       9.99997863e-01, 9.99998846e-01, 9.99999093e-01])
data2 = array([1.00000132, 1.00000138, 1.00000157, ..., 1.99999786, 1.99999885,
       1.99999909])
alternative = 'greater', method = 'asymp'
    @_rename_parameter("mode", "method")
    def ks_2samp(data1, data2, alternative='two-sided', method='auto'):
        """
        Performs the two-sample Kolmogorov-Smirnov test for goodness of fit.
    
        This test compares the underlying continuous distributions F(x) and G(x)
        of two independent samples.  See Notes for a description of the available
        null and alternative hypotheses.
    
        Parameters
        ----------
        data1, data2 : array_like, 1-Dimensional
            Two arrays of sample observations assumed to be drawn from a continuous
            distribution, sample sizes can be different.
        alternative : {'two-sided', 'less', 'greater'}, optional
            Defines the null and alternative hypotheses. Default is 'two-sided'.
            Please see explanations in the Notes below.
        method : {'auto', 'exact', 'asymp'}, optional
            Defines the method used for calculating the p-value.
            The following options are available (default is 'auto'):
    
              * 'auto' : use 'exact' for small size arrays, 'asymp' for large
              * 'exact' : use exact distribution of test statistic
              * 'asymp' : use asymptotic distribution of test statistic
    
        Returns
        -------
        res: KstestResult
            An object containing attributes:
    
            statistic : float
                KS test statistic.
            pvalue : float
                One-tailed or two-tailed p-value.
            statistic_location : float
                Value from `data1` or `data2` corresponding with the KS statistic;
                i.e., the distance between the empirical distribution functions is
                measured at this observation.
            statistic_sign : int
                +1 if the empirical distribution function of `data1` exceeds
                the empirical distribution function of `data2` at
                `statistic_location`, otherwise -1.
    
        See Also
        --------
        kstest, ks_1samp, epps_singleton_2samp, anderson_ksamp
    
        Notes
        -----
        There are three options for the null and corresponding alternative
        hypothesis that can be selected using the `alternative` parameter.
    
        - `less`: The null hypothesis is that F(x) >= G(x) for all x; the
          alternative is that F(x) < G(x) for at least one x. The statistic
          is the magnitude of the minimum (most negative) difference between the
          empirical distribution functions of the samples.
    
        - `greater`: The null hypothesis is that F(x) <= G(x) for all x; the
          alternative is that F(x) > G(x) for at least one x. The statistic
          is the maximum (most positive) difference between the empirical
          distribution functions of the samples.
    
        - `two-sided`: The null hypothesis is that the two distributions are
          identical, F(x)=G(x) for all x; the alternative is that they are not
          identical. The statistic is the maximum absolute difference between the
          empirical distribution functions of the samples.
    
        Note that the alternative hypotheses describe the *CDFs* of the
        underlying distributions, not the observed values of the data. For example,
        suppose x1 ~ F and x2 ~ G. If F(x) > G(x) for all x, the values in
        x1 tend to be less than those in x2.
    
        If the KS statistic is large, then the p-value will be small, and this may
        be taken as evidence against the null hypothesis in favor of the
        alternative.
    
        If ``method='exact'``, `ks_2samp` attempts to compute an exact p-value,
        that is, the probability under the null hypothesis of obtaining a test
        statistic value as extreme as the value computed from the data.
        If ``method='asymp'``, the asymptotic Kolmogorov-Smirnov distribution is
        used to compute an approximate p-value.
        If ``method='auto'``, an exact p-value computation is attempted if both
        sample sizes are less than 10000; otherwise, the asymptotic method is used.
        In any case, if an exact p-value calculation is attempted and fails, a
        warning will be emitted, and the asymptotic p-value will be returned.
    
        The 'two-sided' 'exact' computation computes the complementary probability
        and then subtracts from 1.  As such, the minimum probability it can return
        is about 1e-16.  While the algorithm itself is exact, numerical
        errors may accumulate for large sample sizes.   It is most suited to
        situations in which one of the sample sizes is only a few thousand.
    
        We generally follow Hodges' treatment of Drion/Gnedenko/Korolyuk [1]_.
    
        References
        ----------
        .. [1] Hodges, J.L. Jr.,  "The Significance Probability of the Smirnov
               Two-Sample Test," Arkiv fiur Matematik, 3, No. 43 (1958), 469-86.
    
        Examples
        --------
        Suppose we wish to test the null hypothesis that two samples were drawn
        from the same distribution.
        We choose a confidence level of 95%; that is, we will reject the null
        hypothesis in favor of the alternative if the p-value is less than 0.05.
    
        If the first sample were drawn from a uniform distribution and the second
        were drawn from the standard normal, we would expect the null hypothesis
        to be rejected.
    
        >>> import numpy as np
        >>> from scipy import stats
        >>> rng = np.random.default_rng()
        >>> sample1 = stats.uniform.rvs(size=100, random_state=rng)
        >>> sample2 = stats.norm.rvs(size=110, random_state=rng)
        >>> stats.ks_2samp(sample1, sample2)
        KstestResult(statistic=0.5454545454545454, pvalue=7.37417839555191e-15)
    
        Indeed, the p-value is lower than our threshold of 0.05, so we reject the
        null hypothesis in favor of the default "two-sided" alternative: the data
        were *not* drawn from the same distribution.
    
        When both samples are drawn from the same distribution, we expect the data
        to be consistent with the null hypothesis most of the time.
    
        >>> sample1 = stats.norm.rvs(size=105, random_state=rng)
        >>> sample2 = stats.norm.rvs(size=95, random_state=rng)
        >>> stats.ks_2samp(sample1, sample2)
        KstestResult(statistic=0.10927318295739348, pvalue=0.5438289009927495)
    
        As expected, the p-value of 0.54 is not below our threshold of 0.05, so
        we cannot reject the null hypothesis.
    
        Suppose, however, that the first sample were drawn from
        a normal distribution shifted toward greater values. In this case,
        the cumulative density function (CDF) of the underlying distribution tends
        to be *less* than the CDF underlying the second sample. Therefore, we would
        expect the null hypothesis to be rejected with ``alternative='less'``:
    
        >>> sample1 = stats.norm.rvs(size=105, loc=0.5, random_state=rng)
        >>> stats.ks_2samp(sample1, sample2, alternative='less')
        KstestResult(statistic=0.4055137844611529, pvalue=3.5474563068855554e-08)
    
        and indeed, with p-value smaller than our threshold, we reject the null
        hypothesis in favor of the alternative.
    
        """
        mode = method
    
        if mode not in ['auto', 'exact', 'asymp']:
            raise ValueError(f'Invalid value for mode: {mode}')
        alternative = {'t': 'two-sided', 'g': 'greater', 'l': 'less'}.get(
            alternative.lower()[0], alternative)
        if alternative not in ['two-sided', 'less', 'greater']:
            raise ValueError(f'Invalid value for alternative: {alternative}')
        MAX_AUTO_N = 10000  # 'auto' will attempt to be exact if n1,n2 <= MAX_AUTO_N
        if np.ma.is_masked(data1):
            data1 = data1.compressed()
        if np.ma.is_masked(data2):
            data2 = data2.compressed()
        data1 = np.sort(data1)
        data2 = np.sort(data2)
        n1 = data1.shape[0]
        n2 = data2.shape[0]
        if min(n1, n2) == 0:
            raise ValueError('Data passed to ks_2samp must not be empty')
    
        data_all = np.concatenate([data1, data2])
        # using searchsorted solves equal data problem
        cdf1 = np.searchsorted(data1, data_all, side='right') / n1
        cdf2 = np.searchsorted(data2, data_all, side='right') / n2
>       cddiffs = cdf1 - cdf2
E       numpy.core._exceptions._ArrayMemoryError: Unable to allocate 32.0 MiB for an array with shape (4194304,) and data type float64
scipy/stats/_stats_py.py:8119: MemoryError

[tylerjereddy]

Ok, I'll see if I can circle back here for 1.10.2 -- I'd like to have a low-friction way to reproduce for the Fedora/RHEL/CentOS 32-bit stuff. It may just mean using mock as above, but with a custom config file to add the package managers I need, not sure.

[stratakis]

I am using this mock config file for CentOS stream 8 to emulate the RHEL8 failures, placed in ~/.config/mock/c8s-i686.cfg of my Fedora machine:

Config

# Auto-generated by the Koji build system

config_opts['basedir'] = '/var/lib/mock'
config_opts['chroot_setup_cmd'] = 'groupinstall build'
config_opts['chroothome'] = '/builddir'
config_opts['dnf_warning'] = False
config_opts['package_manager'] = 'dnf'
config_opts['root'] = 'c8s-build-repo_221370'
config_opts['rpmbuild_networking'] = False
config_opts['rpmbuild_timeout'] = 86400
config_opts['target_arch'] = 'i686'
config_opts['chroot_additional_packages'] = 'autoconf bluez-libs-devel bzip2 bzip2-devel expat-devel gcc-c++ gdb gdbm-devel glibc-all-langpacks gmp-devel libX11-devel libappstream-glib libffi-devel libnsl2-devel libtirpc-devel libuuid-devel mesa-libGL-devel ncurses-devel net-tools readline-devel sqlite-devel systemtap-sdt-devel tcl-devel tix-devel tk-devel valgrind-devel chrpath gcc-gfortran lapack-devel openblas-devel python3-Cython fftw-devel suitesparse-devel qhull-devel'
config_opts['use_host_resolv'] = False
config_opts['yum.conf'] = '[main]\ncachedir=/var/cache/yum\ndebuglevel=1\nlogfile=/var/log/yum.log\nreposdir=/dev/null\nretries=20\nobsoletes=1\ngpgcheck=0\nassumeyes=1\nkeepcache=1\ninstall_weak_deps=0\nstrict=1\n\n# repos\n\n[build]\nname=build\nbaseurl=http://kojihub.stream.rdu2.redhat.com/kojifiles/repos/c8s-build/latest/i386\n'

config_opts['plugin_conf']['ccache_enable'] = False
config_opts['plugin_conf']['root_cache_enable'] = False
config_opts['plugin_conf']['yum_cache_enable'] = False

config_opts['macros']['%_host'] = 'i686-koji-linux-gnu'
config_opts['macros']['%_host_cpu'] = 'i686'
config_opts['macros']['%_rpmfilename'] = '%%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm'
config_opts['macros']['%_topdir'] = '/builddir/build'
config_opts['macros']['%dist'] = '%{!?distprefix0:%{?distprefix}}%{expand:%{lua:for i=0,9999 do print("%{?distprefix" .. i .."}") end}}.el8%{?with_bootstrap:%{__bootstrap}}'
config_opts['macros']['%distribution'] = 'Koji Testing'
config_opts['macros']['%packager'] = 'Koji'
config_opts['macros']['%vendor'] = 'Koji'

However Python3.11 and Numpy need to be compiled manually first inside the chroot.

Adding dnf or yum in the chroot should be done with something like:

mock -r c8s-i686 --install dnf

and then shelling with network would be:

mock -r c8s-i686 --enable-network --shell

Edit: Added some development packages to be installed at the chroot initialization in the config file.

[drew-parsons]

aarch64 + s390x #17859 most likely solved here
FAILED scipy/sparse/linalg/_isolve/tests/test_iterative.py::test_maxiter_worsening[lgmres]

This is still a problem in v1.10.1 on debian arm64. test_maxiter_worsening[lgmres] was not one of the tests patched by #17859 . Recent test log at https://ci.debian.net/data/autopkgtest/testing/arm64/s/scipy/41887950/log.gz