Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use code-coverage.cmake from cmake-scripts #124

Open
mscofield0 opened this issue Apr 29, 2021 · 7 comments
Open

Use code-coverage.cmake from cmake-scripts #124

mscofield0 opened this issue Apr 29, 2021 · 7 comments

Comments

@mscofield0
Copy link
Contributor

Currently, the template just applies -fprofile-arcs -ftest-coverage to the Greeter target from test. As far as I could tell, it doesn't really generate the code coverage data locally, and I think it'd be useful to be able to do that.

The code-coverage.cmake script exposes targets that we can run if we want to generate the data locally too. It also does checks that the current ModernCppStarter solution doesn't.

@mscofield0
Copy link
Contributor Author

mscofield0 commented Apr 29, 2021

I've been trying for an hour to get it to work and it seems target_code_coverage() just doesn't find Greeter...
Worth noting that it works with GreeterTests.

Maybe you will have more luck?

Okay I figured out why the Greeter target doesn't work. It's because target_code_coverage() only generates the ccov-* targets if the passed-in target is an executable one.

What is even weirder is that when generating the code coverage data, it includes doctest in the report, but not Greeter.

@mscofield0
Copy link
Contributor Author

Okay, I don't know how I didn't see this before. It's so simple. To instrument a library target and an executable one, just call target_code_coverage() on both. The library one won't get its custom ccov-* targets, but the executable will. The report that cmake --build build/test --target ccov now generates includes the greeter.cpp file.

Although it also imports fmt now along with doctest. I'll try to figure out which directory to exclude so that it doesn't show up anymore.

@TheLartians
Copy link
Owner

Currently, the template just applies -fprofile-arcs -ftest-coverage to the Greeter target from test. As far as I could tell, it doesn't really generate the code coverage data locally, and I think it'd be useful to be able to do that.

It should though, e.g. using Makefiles on Mac, the code coverage data is stored in <build_dir>/CMakeFiles/GreeterTests.dir/. This is generated by simply calling the executable, e.g. after

cmake -Htest -Bbuild/test -DENABLE_TEST_COVERAGE=1
cmake --build build/test
./build/test/GreeterTests

To summarise the data you could simply call gcovr.

> gcovr
------------------------------------------------------------------------------
                           GCC Code Coverage Report
Directory: .
------------------------------------------------------------------------------
File                                       Lines    Exec  Cover   Missing
------------------------------------------------------------------------------
source/greeter.cpp                             8       8   100%   
------------------------------------------------------------------------------
TOTAL                                          8       8   100%
------------------------------------------------------------------------------

TBH I don't do much local analysis myself and rely on platforms such as codecov.io for a nice visualisation. But all the data should be there and you should be able to easily generate the html reports locally.

I'm not really sure about the advantage of adding additional abstraction for the coverage, as the flags are straightforward to use and give most user control without additional required code.

To read more about gcov and how the flags work, see the wikipedia entry.

@mscofield0
Copy link
Contributor Author

As far as I see it, the current solution doesn't support llvm-cov? Also, the script makes it really simple and automate by exposing ccov, ccov-all and ccov-SpecificTarget targets. It already comes with the cmake-scripts dependency so it really isn't adding too much complexity. I made sure the previous option ENABLE_TEST_COVERAGE remains there.

@acgetchell
Copy link

This doesn't seem to work anymore:

cmake -Htest -Bbuild/test -DENABLE_TEST_COVERAGE=1
-- CPM: Adding package [email protected] (v2.4.9)
-- CPM: Adding package [email protected] (v1.7.3)
-- CPM: Adding package CDT@ (/Users/adam/projects/cdt/test/..)
-- CPM: CDT: Adding package [email protected] (v1.8.0)
-- CPM: CDT: Adding package [email protected] (9.1.0)
-- Module support is disabled.
-- Version: 9.1.0
-- Build type:
-- CXX_STANDARD: 11
-- Required features: cxx_variadic_templates
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/adam/projects/cdt/build/test
cmake --build build/test
[ 37%] Built target fmt
[ 62%] Built target CDT
[100%] Built target CDTTests
./build/test/CDTTests
[doctest] doctest version is "2.4.9"
[doctest] run with "--help" for options
===============================================================================
[doctest] test cases: 2 | 2 passed | 0 failed | 0 skipped
[doctest] assertions: 5 | 5 passed | 0 failed |
[doctest] Status: SUCCESS!

gcovr
zsh: correct 'gcovr' to 'gcov' [nyae]? n
------------------------------------------------------------------------------
                           GCC Code Coverage Report
Directory: .
------------------------------------------------------------------------------
File                                       Lines    Exec  Cover   Missing
------------------------------------------------------------------------------
build/test/_deps/doctest-src/doctest/doctest.h
                                            1995     679    34%   829,831,835,862,1024-1025,1029,1032,1036-1039,1045-1046,1078-1079,1123,1127,1152-1153,1256-1257,1300,1316,1318,1794,1801,1830,3238,3240,3283,3289-3291,3294-3296,3298-3302,3306-3307,3310-3311,3503,3506-3507,3510-3513,3516,3518-3521,3532,3563-3565,3593-3594,3602,3610-3615,3617,3619-3620,3622,3624,3626-3629,3631,3633-3634,3636-3638,3640-3642,3644,3646,3648,3650-3651,3653,3657,3669,3677-3678,3681-3684,3691,3695-3698,3701-3706,3709-3711,3714-3720,3723-3728,3741,3743-3744,3746-3748,3750,3754-3755,3758-3759,3762-3765,3779,3786-3789,3791,3793,3795,3797,3799,3801,3803-3808,3810-3811,3813,3819-3826,3832,3834-3840,3844,3849-3852,3855-3860,3863,3866-3868,3872-3875,3888,3890,3892,3894-3896,3898-3908,3915-3919,3926-3928,3936-3946,3948-3949,3955-3956,3963-3966,4034-4036,4038-4042,4044,4048-4050,4075-4077,4079-4087,4092,4108-4109,4113-4114,4116-4118,4121-4122,4125-4129,4131,4134-4137,4139,4143-4148,4150,4153-4157,4159,4161-4164,4166-4167,4169-4175,4177-4180,4189-4191,4193,4195-4198,4200,4204,4208-4209,4214,4217,4225,4273,4282,4284,4293-4299,4316,4319,4323-4327,4331-4335,4350,4352,4355-4359,4414-4416,4419,4421-4425,4429-4437,4466-4467,4473-4476,4478-4479,4482,4518-4521,4526-4527,4530-4532,4534-4535,4545,4547,4551-4553,4555,4746-4752,4755-4757,4823-4825,4829-4830,4832,4834-4836,4839,4841,4843,4864,4866,4873-4875,4880,4882,4884,4886,4888,4892,4899,4901,4905,4910,4913-4915,4917,4920,4922,4928-4930,4933-4937,4940-4942,4945,4947-4950,4953,4955,4958-4960,4963-4964,4967-4969,5012-5014,5040-5043,5062-5063,5078-5080,5082-5083,5085-5086,5088,5091-5093,5095-5096,5098-5099,5101,5104-5109,5114-5117,5119,5123-5127,5131-5132,5134-5135,5138-5139,5141-5142,5149-5151,5155-5157,5166-5169,5172,5174-5176,5181-5186,5191,5193-5195,5197,5199-5200,5204-5205,5207-5208,5213-5215,5218-5220,5236-5238,5241-5243,5246,5251-5253,5256-5263,5266-5269,5272-5277,5280,5282-5284,5287-5290,5293-5296,5299-5301,5304-5311,5313,5333-5336,5340-5341,5344-5347,5364,5366-5368,5370-5378,5383,5385-5390,5394,5397-5399,5402-5407,5409-5414,5421-5438,5440-5448,5450,5453-5454,5457,5463-5465,5468-5477,5480-5482,5484-5486,5488-5494,5496,5499-5501,5504,5506-5516,5518,5521-5522,5524-5526,5529-5534,5537,5539-5541,5543,5545-5549,5551,5553-5554,5556-5561,5563,5565,5568-5569,5571-5574,5576,5578,5580,5583-5587,5594-5595,5597-5598,5600-5611,5613-5614,5616,5618-5620,5623-5624,5626-5629,5632-5640,5642,5658,5660,5664-5665,5671,5675,5677-5678,5683-5684,5686-5687,5694-5695,5702-5703,5706-5709,5712-5716,5719-5721,5724-5726,5730-5731,5738,5740-5742,5744,5750-5751,5754-5755,5758,5760,5765-5772,5774-5775,5777-5782,5784,5786-5790,5793-5796,5799,5801-5802,5805-5807,5810-5813,5815-5816,5819-5822,5825-5827,5830-5831,5834,5836-5838,5840,5842-5844,5846-5848,5851,5853,5855-5858,5860-5864,5875-5876,5879-5882,5915,5917-5919,5922-5924,5927,5929-5930,5933-5936,5938-5942,5946,5950,5952-5954,5957-5959,5961-5969,5971-5973,5976-5980,5984,5986,6003-6005,6007-6016,6018-6020,6022-6036,6038-6112,6116-6117,6120-6127,6129-6131,6140,6142,6144,6162-6163,6165-6166,6168,6170-6175,6186,6226-6227,6232,6238,6241-6242,6245-6246,6249,6251,6253,6255-6256,6258-6259,6262,6267-6270,6272,6274-6275,6277-6278,6280-6286,6289,6292-6295,6298-6300,6307,6309,6311-6312,6314,6316,6319-6321,6323,6325,6327-6332,6335,6388-6389,6395-6399,6401,6435-6439,6441-6442,6444-6453,6455,6457-6462,6466-6467,6469-6470,6488,6491-6494,6496,6498-6499,6502-6505,6507-6509,6513,6526,6530-6533,6632-6633,6637-6638,6642-6643,6652-6653,6657-6658,6663,6666-6668,6672-6673,6677-6678,6682-6685,6689,6691,6693,6695,6707,6709-6711,6736,6739-6741,6753,6765,6783,6792,6794,6807-6811,6814-6816,6818,6820-6821,6823,6843,6846,6848,6850,6852,6854,6856,6864,6867-6869,6874,6884-6887,6889,6935-6937,6939,6945-6947,6951,6964,6983-6985,6988-6990,6998
build/test/_deps/fmt-src/include/fmt/core.h
                                              52      52   100%
include/cdt/greeter.h                          0       0    --%
source/greeter.cpp                             8       8   100%
test/source/greeter.cpp                       10      10   100%
test/source/main.cpp                           0       0    --%
------------------------------------------------------------------------------
TOTAL                                       2065     749    36%
------------------------------------------------------------------------------

@ClausKlein
Copy link
Contributor

@acgetchell try a gcovr config file too like this: https://github.com/ClausKlein/ModernCmakeStarter/blob/develop/gcovr.cfg

@acgetchell
Copy link

acgetchell commented Feb 24, 2023

@ClausKlein That fixes it, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants