Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TEST: 1.9.x + blas variants #223

Closed
wants to merge 6 commits into from
Closed

TEST: 1.9.x + blas variants #223

wants to merge 6 commits into from

Conversation

h-vetinari
Copy link
Member

Following the same scheme as #199, #172, #153 & #130. Should not be merged for the same reasons as #153.

@conda-forge-webservices
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@h-vetinari h-vetinari mentioned this pull request Jan 27, 2023
Closed
@h-vetinari
Copy link
Member Author

h-vetinari commented Jan 27, 2023 •
edited
Loading

Scipy 1.9.3

I had gotten stuck in #199 and didn't get around to doing the analysis for the 1.9 series yet. Now picking this up again to have a reasonable chain record of the changes from here to 1.10.

TL;DR: Improvements all around

From 40 failures out of 110 runs for 1.8.1, we're now at 32 failures out of 96 CI runs (added CPython 3.11, no PyPy, distinguished avx512 absence/presence for openblas)

Notable

The good news:

  • failures related to TestCOO from 1.8.1 disappeared
  • lots of other test failures disappeared

The bad news

Restrictions

  • aarch64/ppc64le only run tests with label='fast', not the full test suite
  • the following test skips:
# these multithreading tests occasionally hang in emulation
{% set tests_to_skip = tests_to_skip + " or test_immediate_updating" %}         # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or test_mixed_threads_processes" %}    # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or test_multiprocess[fft]" %}          # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test__differential_evolution and test_parallel)" %}  # [aarch64 or ppc64le]
# on the slowest agents, these tests take more than 20min in emulation
{% set tests_to_skip = tests_to_skip + " or (test_lsq_linear and test_large_rank_deficient)" %}   # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_iterative and test_precond_inverse[case1])" %}  # [aarch64 or ppc64le]

Details

lib for 1.8.1 now updated
version		 updated
build
scipy 1.8.1 1.9.3 X
numpy 1.24.1 1.24.1
libblas 3.9.0-16 3.9.0-16
blis 0.9.0-0 0.9.0-0
openblas 0.3.21-pthreads-3 0.3.21-pthreads-3
mkl 2022.1.0-915 2022.1.0-915
netlib 3.9.0-5 3.9.0-5
pypy 7.3.9-7 not used
pythran 0.12.1-0 0.12.1-0
qemu-user-static 7.2.0-1 7.2.0-1
variant 1.8.1 1.9.3
all builds 7 test failures related to TestCOO gone 🥳
linux + x64 + mkl + avx512 2 test failures;
incl. scipy/scipy#15533		 only scipy/scipy#15533
linux + x64 + openblas + non-avx512 ✔️ 1 test failure
scipy/scipy#16927
linux + x64 + netlib 1-2 test failures depending on
absence/presence of avx512		 passes 🥳
linux + x64 + openblas 2-3 test failures depending on
absence/presence of avx512		 passes 🥳
linux + aarch64 + openblas 2 test failures 1 test failure
linux + ppc64le test suite disabled due to emulation problems cautious signs that this might
finally get solved 🥳
osx + accelerate 8 test failures 7 test failures
osx + blis ✔️ 1 test failure
scipy/scipy#16927
osx + x64 + mkl 2 failures passes 🥳
osx + x64 + openblas 3 failures passes 🥳
win + mkl + avx512 1 test failure;
scipy/scipy#15533		 remains
win + openblas 8-9 test failures depending on
absence/presence of avx512		 7 test failures
variant accelerate blis mkl
w/ avx512		 mkl
w/o avx512		 netlib openblas
w/ avx512		 openblas
w/o avx512		 sum**
linux / x64 ➖ ✔️ ❌ ✔️ ✔️ ✔️ ❌ 8F
linux / aarch64 ➖ ➖ ➖ ➖ ✔️ ➖ ❌ 4F
linux / ppc64le ➖ ➖ ➖ ➖ ❔ ➖ ❔ -
osx / arm ❔ ➖ ➖ ➖ ❔ ➖ ❔ -
osx / x64 ❌ ❌ ➖ ✔️ ✔️ ➖ ✔️ 8F
win / x64 ➖ ✔️ ❌ ✔️ ✔️ ❌ ❌ 12F
sum** 4F 4F 8F - - 4F 12F 32F

** sum of Segfaults (S), Failures (F), resp. Timeouts (T); out of a total of 96 CI combinations being tested (the openblas builds haven't actually been split into avx512 present/absent, but I inferred them from the others)

Build logs:
Azure

linux + mkl + avx512: 1 failure 
=========================== short test summary info ============================
FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab] - AssertionError: 
Items are not equal:
 ACTUAL: -11
 DESIRED: 0
= 1 failed, 49590 passed, 2881 skipped, 253 xfailed, 9 xpassed, 477 warnings in 956.56s (0:15:56) =
linux + openblas + non-avx512: 1 failure 
=========================== short test summary info ============================
FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
The list of emitted warnings is: [].
= 1 failed, 49589 passed, 2882 skipped, 247 xfailed, 15 xpassed, 481 warnings in 1249.46s (0:20:49) =
linux + aarch64 + openblas: 1 failure 
=========================== short test summary info ============================
FAILED linalg/tests/test_solvers.py::test_solve_discrete_are - AssertionError: 
Arrays are not almost equal to 2 decimals

Mismatched elements: 1 / 4 (25%)
Max absolute difference: 0.01635742
Max relative difference: inf
 x: array([[0.00e+00, 1.23e-20],
       [1.23e-20, 1.64e-02]])
 y: array([[0., 0.],
       [0., 0.]])
= 1 failed, 37670 passed, 2848 skipped, 135 xfailed, 12 xpassed, 3231 warnings in 5731.15s (1:35:31) =
windows + mkl + avx512: 1 failure 
=========================== short test summary info ===========================
FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab]
= 1 failed, 49097 passed, 3374 skipped, 249 xfailed, 13 xpassed, 488 warnings in 1137.86s (0:18:57) =
windows + openblas: 7 failures 
=========================== short test summary info ===========================
FAILED linalg/tests/test_blas.py::TestFBLAS2Simple::test_syr_her - AssertionE...
FAILED linalg/tests/test_decomp.py::TestEig::test_bad_geneig - AssertionError: 
FAILED linalg/tests/test_decomp_ldl.py::test_simple - AssertionError: 
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex64]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex128]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex64]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex128]
= 7 failed, 49092 passed, 3373 skipped, 249 xfailed, 13 xpassed, 478 warnings in 1161.57s (0:19:21) =
osx + accelerate: 7 failures 
=========================== short test summary info ============================
FAILED linalg/tests/test_cython_blas.py::TestWfuncPointers::test_float_args
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_v0 - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 11 / 100 (11%)
Max absolute difference: 2.77555756e-17
Max relative difference: 2.72996743e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...

FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 43 / 100 (43%)
Max absolute difference: 4.16333634e-17
Max relative difference: 4.02567023e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...

FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[0] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 63 / 100 (63%)
Max absolute difference: 4.16333634e-17
Max relative difference: 4.15574571e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...

FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[1] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 50 / 100 (50%)
Max absolute difference: 2.77555756e-17
Max relative difference: 3.40554184e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...

FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state2] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 77 / 100 (77%)
Max absolute difference: 2.22044605e-16
Max relative difference: 2.02120457e-15
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...

FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state3] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 68 / 100 (68%)
Max absolute difference: 2.22044605e-16
Max relative difference: 2.02120457e-15
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
= 7 failed, 49573 passed, 2891 skipped, 249 xfailed, 14 xpassed, 476 warnings in 1241.17s (0:20:41) =
osx + blis: 1 failure 
=========================== short test summary info ============================
FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
The list of emitted warnings is: [].
= 1 failed, 49579 passed, 2891 skipped, 248 xfailed, 15 xpassed, 475 warnings in 1202.21s (0:20:02) =

This was referenced Jan 27, 2023
Closed
Closed
@rgommers
Copy link
Contributor

Looks great, thanks! The picture gets slightly better with every release cycle it looks like.

We're unlikely to do a 1.9.4 at this point, so let's close this PR and focus on 1.10x?

@h-vetinari
Copy link
Member Author

We're unlikely to do a 1.9.4 at this point, so let's close this PR and focus on 1.10x?

I close them as new versions become available. Not planning to invest more time here, but I have a slight preference to leave this open while it's still somewhat relevant (e.g. the last two available versions; will close once we have 1.11).

@h-vetinari h-vetinari mentioned this pull request Jun 26, 2023
Closed
@h-vetinari
Copy link
Member Author

Not that 1.11 is here, let's close this one.

@h-vetinari h-vetinari closed this Jun 26, 2023
@h-vetinari h-vetinari deleted the 1.9_blas_vars branch June 26, 2023 07:28
@h-vetinari h-vetinari mentioned this pull request Jan 23, 2024
Closed
@h-vetinari h-vetinari mentioned this pull request Apr 12, 2024
Closed
@h-vetinari h-vetinari mentioned this pull request Jun 25, 2024
Draft
@h-vetinari h-vetinari mentioned this pull request Jan 8, 2025
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jakirkham jakirkham Awaiting requested review from jakirkham jakirkham will be requested when the pull request is marked ready for review jakirkham is a code owner

@msarahan msarahan Awaiting requested review from msarahan msarahan will be requested when the pull request is marked ready for review msarahan is a code owner

@ocefpaf ocefpaf Awaiting requested review from ocefpaf ocefpaf will be requested when the pull request is marked ready for review ocefpaf is a code owner

@rgommers rgommers Awaiting requested review from rgommers rgommers will be requested when the pull request is marked ready for review rgommers is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@h-vetinari @rgommers