Roadmap for testing

[Beakerboy]

I found that PHP+PCOV runs unit tests much faster than PHP+Xdebug, when all you need is coverage reporting. However, PCOV will only run on PHPunit >7.0. I thought I'd document some of the future testing possibilities.

• PHP 7.0 is past end of life.
  • PHPunit 6.5 (current tests work, adding void to setup() causes failure)
• PHP 7.1 is past end of life
  • PHPunit 7.5 (current tests work, adding void to setup works)
  • could use Pcov with 'clobber' to backport to phpunit 7.x
• PHP 7.2 just started security only
  • PHPunit 8.5 (Current tests fail, must add void to setup(). Exception when string type hint is given an int )
  • could use PCOV
• PHP 7.3 is active
  • PHPunit 8.5 same as above
• PHP 7.4 is active (newly active...should add to test matrix)

[markrogoyski]

I hadn't heard of PCOV. I'll check it out. Seems like it might be a good choice for locally generating code coverage very quickly for a large codebase. It looks like there might be some discrepancies in code branching accuracy compared to XDebug, so maybe XDebug generating the code coverage for build-system reports is still the way to go.

For what version of PHP to support, an application and a library usually take two different approaches.

If you are developing an application, you have specific needs and you define what versions of dependencies you are going to support. And these days with Docker, you can ship the environmental dependencies with your application. So it is easy to use the latest version of things.

For a library that others will include in their projects to build apps, supporting the lowest version you can makes the library more accessible to others. A large company's codebase might be stuck on an old version of PHP and it is not an easy thing to upgrade. If your library only works in the newest version of PHP, you limit who can benefit from using your library.

For MathPHP v1.0, it is going to work on PHP 7.0 and up. Should be released soon (I've been busy with other stuff).

For MathPHP v2.0, we can look at are there any interesting features in 7.1 that make it worthwhile to end support for 7.0 users. I know for sure that nullable types, void functions, [] style list destructuring, class constant visibility, etc. will all make the code base better and have immediate uses in current code, so it seems like a good tradeoff. Also, 2.0 is where I want to do a lot of the backwards-incompatible refactorings, so the timing seems right.

I've gone ahead and added PHP 7.4 as a travis build target. Thanks for the reminder.

[Beakerboy]

My suggestion would be to lock MathPHP 1.x to php >7.0 and provide bugfixes, and move the development branch to php >=7.2, or maybe even 7.3 PHP 7.2 will be no longer supported in one year. We could then update phpunit to 8.x, which would require some refactoring (but there are tools to automate that).

I'll try to add pcov with phpunit 7.5 on PHP7.1 and compare the coverage report. Since we run so many tests, I doubt one of the branching-difference edge cases will be an issue. it sounds like PSR-12 helps avoid some of the scenarios that give dubious coverage reports

[Beakerboy]

I changed the travis script to:

install:
  - pecl install pcov
  - composer install
  - vendor/bin/pcov clobber

and the test suite ran in 3 minutes versus 8 minutes. However, it ran out of memory on one of the ArbitraryInteger tests.

[markrogoyski]

I should look into if Travis can do individual jobs for a build. How you can do things in Jenkins is that unit tests (without any debugger) runs in its own job, and that runs super fast. Code coverage can be a separate job that runs with the debugger. You can do the same with linting and stuff, so you get immediate feedback from linting errors and unit test failures, and the code coverage will finish whenever and doesn't slow you down from getting feedback.

[markrogoyski]

Version 2.0 supports a minimum PHP version of 7.2 and PHPUnit ^8.5.