Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tensorflow and pytorch random seed setting & Update requirements. #612

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

StarryZhang-whu
Copy link

Tests and README are included.

Add support for setting tensorflow and pytorch random seed. It can help detect flaky tests due to the randomness, and provide reproducibility.

Reference:
[1] Dutta, Saikat, et al. "Detecting flaky tests in probabilistic and machine learning applications." Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis. 2020.

@adamchainz
Copy link
Member

Why did you close this? I started reviewing and tidying last night. I think it is a great PR, thank you very much.

@StarryZhang-whu
Copy link
Author

Why did you close this? I started reviewing and tidying last night. I think it is a great PR, thank you very much.

I tried adding pytorch and tensorflow requirements to pass the test, but it seems I did something wrong. This PR failed checks.

@StarryZhang-whu
Copy link
Author

Should I resubmit this PR?

@adamchainz
Copy link
Member

You didn’t mess anything up. The requirements are compiled on the OS you run requirements/compile.py, in your case macOS. Normally there aren’t any OS-specific packages, indeed across all my 33 open source projects there aren’t. But it seems that tensorflow is split into OS-specific packages, so the requirements currently have tensorflow-macos compiled in. Naturally, this doesn't install on the Linux CI runner.

I will try come up with a solution here, perhaps adding tensorflow to the .txt files without pinning the OS-specific packages.

@StarryZhang-whu
Copy link
Author

Maybe you can use compile.py to regenerate these py3*.txt files as before, with I added tensorflow and pytorch in requirements.in.

Thanks for your explanation!

@StarryZhang-whu
Copy link
Author

How is this PR going?

@adamchainz
Copy link
Member

I will work on it when I can. Please do not ping. It is not helpful and you should have no expectation of free service. If you are going to post a message, provide information, such as "I tried using my fork in a project and it worked well" or similar.

StarryZhang-whu and others added 5 commits March 26, 2024 10:47
Tests and README are included.

Add support for setting tensorflow and pytorch random seed. It can help detect flaky tests due to the randomness, provide reproducibility.

Co-Authored-By: Saikat Dutta <[email protected]>
@adamchainz
Copy link
Member

Python 3.12 has a segmentation fault, both locally and on CI.

Traceback
Fatal Python error: Segmentation fault

Current thread 0x00007f6e828aeb80 (most recent call first):
  Garbage-collecting
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/ast.py", line 52 in parse
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/source.py", line 185 in getstatementrange_ast
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 261 in getsource
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 834 in _getentrysource
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 932 in repr_traceback_entry
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 994 in repr_traceback
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 1064 in repr_excinfo
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/_code/code.py", line 699 in getrepr
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/nodes.py", line 464 in _repr_failure_py
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/nodes.py", line 541 in repr_failure
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/runner.py", line 409 in pytest_make_collect_report
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/runner.py", line 565 in collect_one_node
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 839 in _collect_one_node
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 976 in genitems
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 981 in genitems
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 813 in perform_collect
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 349 in pytest_collection
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 338 in _main
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 174 in main
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 197 in console_main
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/pytest/__main__.py", line 7 in <module>
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/execfile.py", line 211 in run
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/cmdline.py", line 858 in do_run
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/cmdline.py", line 681 in command_line
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/cmdline.py", line 970 in main
  File "/home/runner/work/pytest-randomly/pytest-randomly/.tox/py312/lib/python3.12/site-packages/coverage/__main__.py", line 10 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special (total: 20)

This occurs reliably and disappears when removing the tensorflow import. But it seems to be part of numpy. I have looked on both projects’ issue trackers and found nothing, and haven’t got the time to chase creating a minimal example. Would you be interested in doing that?

@StarryZhang-whu
Copy link
Author

Yes, I'm trying to do so. But I'm not sure how to create a minimal example of this case.

I noticed that except Python 3.12, other versions of CI worked well. With the below dependencies installed:

py311: absl-py==2.1.0,asgiref==3.8.0,astunparse==1.6.3,certifi==2024.2.2,charset-normalizer==3.3.2,coverage==7.4.4,Django==5.0.3,execnet==2.0.2,factory-boy==3.3.0,Faker==24.3.0,filelock==3.13.1,flatbuffers==24.3.25,fsspec==2024.3.1,gast==0.5.4,google-pasta==0.2.0,grpcio==1.62.1,h5py==3.10.0,idna==3.6,iniconfig==2.0.0,Jinja2==3.1.3,keras==3.1.1,libclang==18.1.1,Markdown==3.6,markdown-it-py==3.0.0,MarkupSafe==2.1.5,mdurl==0.1.2,ml-dtypes==0.3.2,model-bakery==1.17.0,mpmath==1.3.0,namex==0.0.7,networkx==3.2.1,numpy==1.26.4,nvidia-cublas-cu12==12.1.3.1,nvidia-cuda-cupti-cu12==12.1.105,nvidia-cuda-nvrtc-cu12==12.1.105,nvidia-cuda-runtime-cu12==12.1.105,nvidia-cudnn-cu12==8.9.2.26,nvidia-cufft-cu12==11.0.2.54,nvidia-curand-cu12==10.3.2.106,nvidia-cusolver-cu12==11.4.5.107,nvidia-cusparse-cu12==12.1.0.106,nvidia-nccl-cu12==2.19.3,nvidia-nvjitlink-cu12==12.4.99,nvidia-nvtx-cu12==12.1.105,opt-einsum==3.3.0,optree==0.11.0,packaging==24.0,pip==24.0,pluggy==1.4.0,protobuf==4.25.3,Pygments==2.17.2,pytest==8.1.1,pytest-randomly @ file:///home/runner/work/pytest-randomly/pytest-randomly/.tox/.tmp/package/1/pytest_randomly-3.15.0-py3-none-any.whl#sha256=33ad92c347928f2b348b0575fe78fb9f5ee21fe8a97540399b386a28a2b4b17f,pytest-xdist==3.5.0,python-dateutil==2.9.0.post0,requests==2.31.0,rich==13.7.1,setuptools==69.1.0,six==1.16.0,sqlparse==0.4.4,sympy==1.12,tensorboard==2.16.2,tensorboard-data-server==0.7.2,tensorflow==2.16.1,tensorflow-io-gcs-filesystem==0.36.0,termcolor==2.4.0,torch==2.2.1,triton==2.2.0,typing_extensions==4.10.0,urllib3==2.2.1,Werkzeug==3.0.1,wheel==0.42.0,wrapt==1.16.0
py311: commands[0]> python -W error::ResourceWarning -W error::DeprecationWarning -W error::PendingDeprecationWarning -W 'ignore:pkg_resources is deprecated as an API:DeprecationWarning' -W 'ignore:datetime.datetime.utcfromtimestamp() is deprecated:DeprecationWarning' -m coverage run -m pytest -p no:randomly tests
============================= test session starts ==============================

While the Python 3.12 CI failed with the below dependencies installed:

py312: absl-py==2.1.0,asgiref==3.8.1,astunparse==1.6.3,certifi==2024.2.2,charset-normalizer==3.3.2,colorama==0.4.6,coverage==7.4.4,Django==5.0.3,execnet==2.0.2,factory-boy==3.3.0,Faker==24.4.0,filelock==3.13.3,flatbuffers==24.3.25,fsspec==2024.3.1,gast==0.5.4,google-pasta==0.2.0,grpcio==1.62.1,h5py==3.10.0,idna==3.6,iniconfig==2.0.0,Jinja2==3.1.3,keras==3.1.1,libclang==18.1.1,Markdown==3.6,markdown-it-py==3.0.0,MarkupSafe==2.1.5,mdurl==0.1.2,ml-dtypes==0.3.2,model-bakery==1.17.0,mpmath==1.3.0,namex==0.0.7,networkx==3.2.1,numpy==1.26.4,nvidia-cublas-cu12==12.1.3.1,nvidia-cuda-cupti-cu12==12.1.105,nvidia-cuda-nvrtc-cu12==12.1.105,nvidia-cuda-runtime-cu12==12.1.105,nvidia-cudnn-cu12==8.9.2.26,nvidia-cufft-cu12==11.0.2.54,nvidia-curand-cu12==10.3.2.106,nvidia-cusolver-cu12==11.4.5.107,nvidia-cusparse-cu12==12.1.0.106,nvidia-nccl-cu12==2.19.3,nvidia-nvjitlink-cu12==12.4.99,nvidia-nvtx-cu12==12.1.105,opt-einsum==3.3.0,optree==0.11.0,packaging==24.0,pip==24.0,pluggy==1.4.0,protobuf==4.25.3,Pygments==2.17.2,pytest==8.1.1,pytest-randomly @ file:///home/runner/work/pytest-randomly/pytest-randomly/.tox/.tmp/package/1/pytest_randomly-3.15.0-py3-none-any.whl#sha256=1480f46b08e60d5fc895996cda1e7d98ead282b7ea49cb95665a2f654e3e264d,pytest-xdist==3.5.0,python-dateutil==2.9.0.post0,requests==2.31.0,rich==13.7.1,setuptools==69.2.0,six==1.16.0,sqlparse==0.4.4,sympy==1.12,tensorboard==2.16.2,tensorboard-data-server==0.7.2,tensorflow==2.16.1,termcolor==2.4.0,torch==2.2.1,typing_extensions==4.10.0,tzdata==2024.1,urllib3==2.2.1,Werkzeug==3.0.1,wheel==0.43.0,wrapt==1.16.0
py312: commands[0]> python -W error::ResourceWarning -W error::DeprecationWarning -W error::PendingDeprecationWarning -W 'ignore:pkg_resources is deprecated as an API:DeprecationWarning' -W 'ignore:datetime.datetime.utcfromtimestamp() is deprecated:DeprecationWarning' -m coverage run -m pytest -p no:randomly tests
Fatal Python error: Segmentation fault

which tensorflow-io-gcs-filesystem package is missing. I guess there might be some compatibility issue between python and package. I will inquire about this issue in community and try to figure it out. Thanks for your help!

@adamchainz
Copy link
Member

Huh, interesting. It would be surprising though if the missing package caused the segmentation fault. A segfault is when code reaches for memory outside of its allotted area.

@adamchainz
Copy link
Member

Please don’t just close PRs when you give up. I may find the time to finish this, or another contributor may be interested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants