Ability to compile using MinGW build tools #2114

Kreijstal · 2023-11-04T11:49:57Z

In this pull request I have done some code changes to allow mingw to build FBGEMM, everything builds and links, tests and benchmarks too. fix #2113

This is the build procedure

#run on a msys2 enviroment
mkdir -p build
cd build
cmake .. -G"MSYS Makefiles"
make
make test

I had trouble setting some flags for the tests, so I hardcoded FBGEMM_STATIC under mingw builds, I only did this to prove the point that mingw supports this compilation, and runs tests, however I lack the skills to edit the cmake so that the compilation just works, linkage error only happens when building the tests.

These are the result of the tests

$ make test
Running tests...
Test project C:/path/pytorch-help/FBGEMM/build
      Start  1: Bfloat16ConvertTest
 1/25 Test  #1: Bfloat16ConvertTest ..............   Passed    0.09 sec
      Start  2: EmbeddingSpMDM8BitTest
 2/25 Test  #2: EmbeddingSpMDM8BitTest ...........   Passed    0.54 sec
      Start  3: EmbeddingSpMDMNBitTest
 3/25 Test  #3: EmbeddingSpMDMNBitTest ...........   Passed    0.95 sec
      Start  4: EmbeddingSpMDMTest
 4/25 Test  #4: EmbeddingSpMDMTest ...............***Failed    4.57 sec
      Start  5: FP16Test
 5/25 Test  #5: FP16Test .........................***Exception: SegFault  0.05 sec
      Start  6: Float16ConvertTest
 6/25 Test  #6: Float16ConvertTest ...............   Passed    0.06 sec
      Start  7: GConvTest
 7/25 Test  #7: GConvTest ........................Exit code 0xc0000374
***Exception:   4.72 sec
      Start  8: I64Test
 8/25 Test  #8: I64Test ..........................   Passed    1.20 sec
      Start  9: I8DepthwiseTest
 9/25 Test  #9: I8DepthwiseTest ..................   Passed   45.08 sec
      Start 10: I8DirectconvTest
10/25 Test #10: I8DirectconvTest .................   Passed    5.64 sec
      Start 11: I8SpmdmTest
11/25 Test #11: I8SpmdmTest ......................   Passed   14.93 sec
      Start 12: Im2ColFusedRequantizeTest
12/25 Test #12: Im2ColFusedRequantizeTest ........   Passed   18.40 sec
      Start 13: PackedRequantizeAcc16Test
13/25 Test #13: PackedRequantizeAcc16Test ........   Passed    1.61 sec
      Start 14: PackedRequantizeTest
14/25 Test #14: PackedRequantizeTest .............   Passed   31.11 sec
      Start 15: QuantUtilsTest
15/25 Test #15: QuantUtilsTest ...................   Passed    0.16 sec
      Start 16: RadixSortTest
16/25 Test #16: RadixSortTest ....................   Passed    0.05 sec
      Start 17: RequantizeOnlyTest
17/25 Test #17: RequantizeOnlyTest ...............   Passed    0.04 sec
      Start 18: RowWiseSparseAdagradFusedTest
18/25 Test #18: RowWiseSparseAdagradFusedTest ....   Passed   27.23 sec
      Start 19: SparseAdagradTest
19/25 Test #19: SparseAdagradTest ................   Passed    0.53 sec
      Start 20: SparseDenseMMFP32Test
20/25 Test #20: SparseDenseMMFP32Test ............   Passed    3.37 sec
      Start 21: SparseDenseMMInt8Test
21/25 Test #21: SparseDenseMMInt8Test ............   Passed    5.57 sec
      Start 22: SparsePackUnpackTest
22/25 Test #22: SparsePackUnpackTest .............   Passed    0.08 sec
      Start 23: TransposeTest
23/25 Test #23: TransposeTest ....................   Passed    0.07 sec
      Start 24: TransposedRequantizeTest
24/25 Test #24: TransposedRequantizeTest .........   Passed    0.06 sec
      Start 25: UniConvTest
25/25 Test #25: UniConvTest ......................   Passed   26.91 sec

88% tests passed, 3 tests failed out of 25

Total Test time (real) = 193.31 sec

The following tests FAILED:
          4 - EmbeddingSpMDMTest (Failed)
          5 - FP16Test (SEGFAULT)
          7 - GConvTest (Exit code 0xc0000374
)
Errors while running CTest
Output from these tests are in: C:/path/pytorch-help/FBGEMM/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
make: *** [Makefile:71: test] Error 8

I am open for feedback and suggestions.

In case you're curious in the previous commit, you can also build using the shared library of asmjit that you can get with your msys distribution, but this probably polutes the project, so I took it out.

netlify · 2023-11-04T11:50:03Z

✅ Deploy Preview for pytorch-fbgemm-docs canceled.

Name	Link
🔨 Latest commit	`77cb1f9`
🔍 Latest deploy log	https://app.netlify.com/sites/pytorch-fbgemm-docs/deploys/654e9b2722be1c0008d384ea

facebook-github-bot · 2023-11-04T11:50:03Z

Hi @Kreijstal!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

Biswa96 · 2023-11-04T13:00:44Z

bench/CMakeLists.txt

@@ -65,6 +65,10 @@ macro(add_benchmark BENCHNAME)
 if (${BLAS_FOUND})
 target_compile_options(${BENCHNAME} PRIVATE "-DUSE_BLAS")
 target_link_libraries(${BENCHNAME} "${BLAS_LIBRARIES}")
+ if (DEFINED ENV{MINGW_PREFIX})
+ # Include the directory using the value of the environment variable
+ include_directories("$ENV{MINGW_PREFIX}/include/openblas")


MINGW_PREFIX is a private environment variable in msys2. It should not be used in any other repository.

ahh, I think I added it because I couldn't get openblas to work, thank you, I'll update it and fix it

facebook-github-bot · 2023-11-04T14:05:20Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

to compile you will do it like this for mingw: cmake -G"MSYS Makefiles" -DFBGEMM_LIBRARY_TYPE=shared -DBLAS_INCLUDE_DIRS="$MINGW_PREFIX/include/openblas" .. I hope this is okay

Biswa96 · 2023-11-11T04:14:38Z

CMakeLists.txt

@@ -244,7 +252,7 @@ if(NOT TARGET asmjit)

 #build asmjit
 #For MSVC shared build, asmjit needs to be shared also.
- if (MSVC AND (FBGEMM_LIBRARY_TYPE STREQUAL "shared"))
+ if ((MSVC OR MINGW) AND (FBGEMM_LIBRARY_TYPE STREQUAL "shared"))


MSVC or MINGW is probably same as WIN32.

Biswa96 · 2023-11-11T04:15:54Z

test/CMakeLists.txt

@@ -62,6 +63,11 @@ macro(add_gtest TESTNAME)
 target_compile_definitions(${TESTNAME} PRIVATE FBGEMM_STATIC)
 endif()
 else(MSVC)
+ if(MINGW)
+ #if (FBGEMM_LIBRARY_TYPE STREQUAL "static")
+ target_compile_definitions(${TESTNAME} PRIVATE FBGEMM_STATIC)


I assume the macro should not be defined for all cases. That static library condition should be present, right?

q10 · 2023-11-15T02:04:29Z

Could you let me know what version of msys2 gcc you are using, and how to install them on a Windows machine?

Biswa96 · 2023-11-15T03:55:06Z

Could you let me know what version of msys2 gcc you are using, and how to install them on a Windows machine?

The installation procedure is documented here https://www.msys2.org/

q10 · 2023-11-16T01:45:20Z

Could you let me know what version of msys2 gcc you are using, and how to install them on a Windows machine?

The installation procedure is documented here https://www.msys2.org/

The reason I asked this question is because the instructions at msys2 don't seem to apply well to the Windows CI runners.
The default gcc appears to be already installed in the runners, but are stuck at 8.0, while the more recent-versioned gcc packaged with mingw or msys2 (both installed through choco) can't replace the default one in %PATH%.

I've been trying to integrate MinGW builds into GitHub CI over at #2134, but compilation is failing at the linking stage (see here). Could you see if you can integrate the changes from that PR into this PR?

Kreijstal · 2023-11-16T09:08:32Z

Could you let me know what version of msys2 gcc you are using, and how to install them on a Windows machine?

The installation procedure is documented here https://www.msys2.org/

The reason I asked this question is because the instructions at msys2 don't seem to apply well to the Windows CI runners. The default gcc appears to be already installed in the runners, but are stuck at 8.0, while the more recent-versioned gcc packaged with mingw or msys2 (both installed through choco) can't replace the default one in %PATH%.

I've been trying to integrate MinGW builds into GitHub CI over at #2134, but compilation is failing at the linking stage (see here). Could you see if you can integrate the changes from that PR into this PR?

Sure, I can work on that, do you mind if I use this helper? https://github.com/marketplace/actions/setup-msys2

q10 · 2023-11-16T09:12:50Z

Could you let me know what version of msys2 gcc you are using, and how to install them on a Windows machine?

The installation procedure is documented here https://www.msys2.org/

The reason I asked this question is because the instructions at msys2 don't seem to apply well to the Windows CI runners. The default gcc appears to be already installed in the runners, but are stuck at 8.0, while the more recent-versioned gcc packaged with mingw or msys2 (both installed through choco) can't replace the default one in %PATH%.
I've been trying to integrate MinGW builds into GitHub CI over at #2134, but compilation is failing at the linking stage (see here). Could you see if you can integrate the changes from that PR into this PR?

Sure, I can work on that, do you mind if I use this helper? https://github.com/marketplace/actions/setup-msys2

That should be fine, yes.

Kreijstal · 2023-11-23T19:00:52Z

Could you let me know what version of msys2 gcc you are using, and how to install them on a Windows machine?

The installation procedure is documented here https://www.msys2.org/

The reason I asked this question is because the instructions at msys2 don't seem to apply well to the Windows CI runners. The default gcc appears to be already installed in the runners, but are stuck at 8.0, while the more recent-versioned gcc packaged with mingw or msys2 (both installed through choco) can't replace the default one in %PATH%.
I've been trying to integrate MinGW builds into GitHub CI over at #2134, but compilation is failing at the linking stage (see here). Could you see if you can integrate the changes from that PR into this PR?

Sure, I can work on that, do you mind if I use this helper? https://github.com/marketplace/actions/setup-msys2

That should be fine, yes.

please forgive the delays, I have lots of uni stuff at the moment, I'll catch up when I can :)

CI added 2 commits November 4, 2023 12:26

Adding mingw headers, so that it compiles with msys2 buildtools

210b7a4

Undo adding asmjit shared on mingw

992299d

Kreijstal changed the title ~~Mingwcompile~~ Ability to compile using MinGW build tools Nov 4, 2023

Biswa96 reviewed Nov 4, 2023

View reviewed changes

facebook-github-bot added the cla signed label Nov 4, 2023

Remove Mingw dependent things

77cb1f9

to compile you will do it like this for mingw: cmake -G"MSYS Makefiles" -DFBGEMM_LIBRARY_TYPE=shared -DBLAS_INCLUDE_DIRS="$MINGW_PREFIX/include/openblas" .. I hope this is okay

Biswa96 reviewed Nov 11, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ability to compile using MinGW build tools #2114

Ability to compile using MinGW build tools #2114

Kreijstal commented Nov 4, 2023

netlify bot commented Nov 4, 2023 •

edited

facebook-github-bot commented Nov 4, 2023

Biswa96 Nov 4, 2023

Kreijstal Nov 4, 2023

facebook-github-bot commented Nov 4, 2023

Biswa96 Nov 11, 2023

Biswa96 Nov 11, 2023

q10 commented Nov 15, 2023

Biswa96 commented Nov 15, 2023

q10 commented Nov 16, 2023 •

edited

Kreijstal commented Nov 16, 2023

q10 commented Nov 16, 2023

Kreijstal commented Nov 23, 2023

Ability to compile using MinGW build tools #2114

Are you sure you want to change the base?

Ability to compile using MinGW build tools #2114

Conversation

Kreijstal commented Nov 4, 2023

netlify bot commented Nov 4, 2023 • edited

✅ Deploy Preview for pytorch-fbgemm-docs canceled.

facebook-github-bot commented Nov 4, 2023

Action Required

Process

Biswa96 Nov 4, 2023

Choose a reason for hiding this comment

Kreijstal Nov 4, 2023

Choose a reason for hiding this comment

facebook-github-bot commented Nov 4, 2023

Biswa96 Nov 11, 2023

Choose a reason for hiding this comment

Biswa96 Nov 11, 2023

Choose a reason for hiding this comment

q10 commented Nov 15, 2023

Biswa96 commented Nov 15, 2023

q10 commented Nov 16, 2023 • edited

Kreijstal commented Nov 16, 2023

q10 commented Nov 16, 2023

Kreijstal commented Nov 23, 2023

netlify bot commented Nov 4, 2023 •

edited

q10 commented Nov 16, 2023 •

edited