Benchmarks and performance improvements

[Ocramius]

Scope of work

This patch attempts to improve happy-path execution time for Psl\Type components, and specifically:

• Psl\Type\TypeInterface#coerce()
• Psl\Type\TypeInterface#assert()
• Psl\Type\TypeInterface#matches()

Results

Results as per 2021-09-08 (af6d378) are as follows:

root@6d63f978332e:/app# ./tools/phpbench/vendor/bin/phpbench run --config tools/phpbench/phpbench.json --ref=benchmark_reference --iterations=15 --revs=8
PHPBench (1.1.0) running benchmarks...
with configuration file: /app/tools/phpbench/phpbench.json
with PHP version 8.0.10, xdebug ✔, opcache ✔
comparing [actual vs. benchmark_reference]

\Psl\Tests\Benchmark\Type\StringTypeBench

    benchCoerce # string....................R1 I5 - [Mo1.000μs vs. Mo1.000μs] 0.00% (±0.00%)
    benchCoerce # int.......................R1 I1 - [Mo1.125μs vs. Mo1.125μs] 0.00% (±0.00%)
    benchCoerce # instanceof Stringable (ex.R1 I8 - [Mo1.375μs vs. Mo1.875μs] -26.67% (±0.00%)
    benchCoerce # instanceof Stringable (im.R1 I7 - [Mo1.375μs vs. Mo1.875μs] -26.67% (±0.00%)
    benchAssert.............................R1 I5 - [Mo1.125μs vs. Mo1.125μs] 0.00% (±0.00%)
    benchMatch..............................R2 I14 - [Mo1.000μs vs. Mo1.000μs] 0.00% (±0.00%)

\Psl\Tests\Benchmark\Type\ShapeTypeBench

    benchCoerce # empty shape, empty array .R1 I1 - [Mo3.125μs vs. Mo17.202μs] -81.83% (±1.00%)
    benchCoerce # empty shape, empty iterab.R2 I7 - [Mo5.497μs vs. Mo17.548μs] -68.68% (±1.40%)
    benchCoerce # empty shape, non-empty ar.R1 I4 - [Mo3.502μs vs. Mo23.732μs] -85.24% (±2.32%)
    benchCoerce # empty shape, non-empty it.R1 I14 - [Mo7.004μs vs. Mo24.129μs] -70.97% (±1.27%)
    benchCoerce # complex shape with option.R1 I14 - [Mo5.625μs vs. Mo62.697μs] -91.03% (±1.21%)
    benchCoerce # complex shape with option.R1 I10 - [Mo29.472μs vs. Mo64.065μs] -54.00% (±0.86%)
    benchCoerce # complex shape with option.R1 I12 - [Mo8.672μs vs. Mo93.012μs] -90.68% (±1.57%)
    benchCoerce # complex shape with option.R1 I14 - [Mo39.567μs vs. Mo94.873μs] -58.30% (±0.94%)
    benchAssert # empty shape, empty array .R1 I0 - [Mo1.250μs vs. Mo15.543μs] -91.96% (±0.00%)
    benchAssert # empty shape, non-empty ar.R1 I7 - [Mo2.000μs vs. Mo18.186μs] -89.00% (±2.89%)
    benchAssert # complex shape with option.R1 I2 - [Mo22.480μs vs. Mo49.896μs] -54.95% (±1.44%)
    benchAssert # complex shape with option.R1 I2 - [Mo29.613μs vs. Mo63.770μs] -53.56% (±1.22%)
    benchMatch # empty shape, empty array v.R1 I3 - [Mo1.500μs vs. Mo15.995μs] -90.62% (±0.00%)
    benchMatch # empty shape, non-empty arr.R2 I7 - [Mo2.500μs vs. Mo18.625μs] -86.58% (±1.24%)
    benchMatch # complex shape with optiona.R1 I7 - [Mo22.995μs vs. Mo50.246μs] -54.24% (±0.97%)
    benchMatch # complex shape with optiona.R1 I3 - [Mo30.085μs vs. Mo64.037μs] -53.02% (±0.44%)

\Psl\Tests\Benchmark\Type\VecTypeBench

    benchCoerce # mixed, empty array........R1 I14 - [Mo3.753μs vs. Mo5.125μs] -26.77% (±2.02%)
    benchCoerce # mixed, empty iterable.....R1 I14 - [Mo4.116μs vs. Mo5.125μs] -19.69% (±2.44%)
    benchCoerce # mixed, non-empty array....R1 I14 - [Mo5.602μs vs. Mo6.749μs] -17.00% (±2.27%)
    benchCoerce # mixed, non-empty iterable.R1 I14 - [Mo5.750μs vs. Mo7.000μs] -17.85% (±0.96%)
    benchCoerce # mixed, large array........R1 I14 - [Mo48.082μs vs. Mo49.339μs] -2.55% (±1.26%)
    benchCoerce # mixed, large iterable.....R1 I14 - [Mo51.736μs vs. Mo52.998μs] -2.38% (±2.04%)
    benchCoerce # int, empty array..........R1 I10 - [Mo3.770μs vs. Mo5.125μs] -26.45% (±2.76%)
    benchCoerce # int, empty iterable.......R2 I8 - [Mo3.998μs vs. Mo5.191μs] -22.99% (±2.05%)
    benchCoerce # int, non-empty array......R2 I4 - [Mo5.620μs vs. Mo6.699μs] -16.10% (±1.89%)
    benchCoerce # int, non-empty iterable...R2 I14 - [Mo5.779μs vs. Mo7.000μs] -17.45% (±2.14%)
    benchAssert # mixed, empty array........R1 I10 - [Mo3.625μs vs. Mo5.125μs] -29.26% (±2.52%)
    benchAssert # mixed, non-empty array....R1 I14 - [Mo5.750μs vs. Mo7.245μs] -20.64% (±0.87%)
    benchAssert # mixed, large array........R1 I14 - [Mo64.692μs vs. Mo66.096μs] -2.12% (±1.18%)
    benchAssert # int, empty array..........R1 I14 - [Mo3.741μs vs. Mo5.125μs] -27.00% (±2.38%)
    benchAssert # int, non-empty array......R13 I14 - [Mo5.752μs vs. Mo7.269μs] -20.86% (±1.41%)
    benchMatch # mixed, empty array.........R1 I13 - [Mo1.125μs vs. Mo1.250μs] -10.00% (±0.00%)
    benchMatch # mixed, non-empty array.....R1 I4 - [Mo2.875μs vs. Mo3.125μs] -8.03% (±2.21%)
    benchMatch # mixed, large array.........R2 I6 - [Mo54.026μs vs. Mo53.978μs] +0.09% (±0.69%)
    benchMatch # int, empty array...........R1 I0 - [Mo1.125μs vs. Mo1.250μs] -10.00% (±0.00%)
    benchMatch # int, non-empty array.......R1 I4 - [Mo2.875μs vs. Mo3.125μs] -8.01% (±2.49%)

\Psl\Tests\Benchmark\Type\DictTypeBench

    benchCoerce # generic array, empty arra.R3 I7 - [Mo6.128μs vs. Mo8.708μs] -29.63% (±1.45%)
    benchCoerce # generic array, empty iter.R1 I1 - [Mo6.376μs vs. Mo8.850μs] -27.95% (±1.33%)
    benchCoerce # generic array, non-empty .R1 I2 - [Mo7.374μs vs. Mo10.614μs] -30.53% (±1.70%)
    benchCoerce # generic array, non-empty .R1 I14 - [Mo7.600μs vs. Mo10.875μs] -30.12% (±1.80%)
    benchCoerce # generic array, large arra.R1 I5 - [Mo99.035μs vs. Mo3.702ms] -97.33% (±2.06%)
    benchCoerce # generic array, large iter.R1 I5 - [Mo103.920μs vs. Mo3.086ms] -96.63% (±1.47%)
    benchCoerce # int array, empty array....R1 I3 - [Mo6.103μs vs. Mo8.436μs] -27.65% (±1.67%)
    benchCoerce # int array, empty iterable.R1 I10 - [Mo6.153μs vs. Mo8.760μs] -29.77% (±1.85%)
    benchCoerce # int array, non-empty arra.R1 I13 - [Mo9.011μs vs. Mo11.639μs] -22.58% (±1.40%)
    benchCoerce # int array, non-empty iter.R1 I6 - [Mo9.428μs vs. Mo12.013μs] -21.52% (±1.26%)
    benchCoerce # int array, large array....R2 I9 - [Mo100.810μs vs. Mo102.101μs] -1.26% (±2.02%)
    benchCoerce # int array, large iterable.R1 I4 - [Mo104.550μs vs. Mo107.096μs] -2.38% (±1.10%)
    benchCoerce # map, empty array..........R2 I8 - [Mo6.082μs vs. Mo8.475μs] -28.24% (±1.93%)
    benchCoerce # map, empty iterable.......R2 I13 - [Mo6.246μs vs. Mo8.604μs] -27.40% (±1.44%)
    benchCoerce # map, non-empty array......R1 I9 - [Mo9.001μs vs. Mo11.503μs] -21.75% (±0.82%)
    benchCoerce # map, non-empty iterable...R2 I13 - [Mo9.575μs vs. Mo11.875μs] -19.38% (±1.56%)
    benchCoerce # map, large array..........R2 I14 - [Mo102.754μs vs. Mo104.467μs] -1.64% (±2.33%)
    benchCoerce # map, large iterable.......R1 I2 - [Mo106.440μs vs. Mo109.807μs] -3.07% (±1.45%)
    benchAssert # generic array, empty arra.R1 I14 - [Mo5.750μs vs. Mo8.248μs] -30.29% (±0.54%)
    benchAssert # generic array, non-empty .R1 I14 - [Mo6.986μs vs. Mo9.750μs] -28.35% (±1.45%)
    benchAssert # generic array, large arra.R4 I10 - [Mo98.287μs vs. Mo3.511ms] -97.20% (±1.85%)
    benchAssert # int array, empty array....R3 I12 - [Mo5.750μs vs. Mo8.250μs] -30.31% (±1.10%)
    benchAssert # int array, non-empty arra.R3 I13 - [Mo8.737μs vs. Mo11.286μs] -22.59% (±1.64%)
    benchAssert # int array, large array....R2 I8 - [Mo98.272μs vs. Mo101.161μs] -2.86% (±1.21%)
    benchAssert # map, empty array..........R2 I14 - [Mo5.745μs vs. Mo8.250μs] -30.36% (±1.49%)
    benchAssert # map, non-empty array......R6 I11 - [Mo8.702μs vs. Mo11.181μs] -22.17% (±1.47%)
    benchAssert # map, large array..........R1 I7 - [Mo102.052μs vs. Mo104.321μs] -2.17% (±1.40%)
    benchMatch # generic array, empty array.R1 I14 - [Mo6.148μs vs. Mo8.651μs] -28.93% (±2.49%)
    benchMatch # generic array, non-empty a.R7 I13 - [Mo7.375μs vs. Mo10.256μs] -28.09% (±1.77%)
    benchMatch # generic array, large array.R1 I6 - [Mo100.021μs vs. Mo4.486ms] -97.77% (±1.98%)
    benchMatch # int array, empty array.....R1 I14 - [Mo6.227μs vs. Mo8.699μs] -28.41% (±2.21%)
    benchMatch # int array, non-empty array.R1 I14 - [Mo9.000μs vs. Mo11.664μs] -22.83% (±0.69%)
    benchMatch # int array, large array.....R1 I1 - [Mo98.781μs vs. Mo101.539μs] -2.72% (±1.67%)
    benchMatch # map, empty array...........R6 I14 - [Mo6.248μs vs. Mo8.625μs] -27.57% (±2.31%)
    benchMatch # map, non-empty array.......R6 I14 - [Mo9.017μs vs. Mo11.552μs] -21.95% (±1.28%)
    benchMatch # map, large array...........R6 I14 - [Mo102.852μs vs. Mo105.609μs] -2.61% (±1.44%)

\Psl\Tests\Benchmark\Type\ArrayKeyTypeBench

    benchCoerce # string....................R1 I0 - [Mo1.000μs vs. Mo1.750μs] -42.86% (±0.00%)
    benchCoerce # int.......................R1 I12 - [Mo1.000μs vs. Mo19.582μs] -94.89% (±0.00%)
    benchCoerce # instanceof Stringable (ex.R2 I7 - [Mo61.066μs vs. Mo53.186μs] +14.82% (±1.27%)
    benchCoerce # instanceof Stringable (im.R1 I4 - [Mo61.126μs vs. Mo52.867μs] +15.62% (±1.32%)
    benchAssert # string....................R1 I0 - [Mo1.125μs vs. Mo1.500μs] -25.00% (±0.00%)
    benchAssert # int.......................R2 I4 - [Mo1.125μs vs. Mo17.236μs] -93.47% (±0.00%)
    benchMatch # string.....................R1 I11 - [Mo1.000μs vs. Mo1.375μs] -27.27% (±0.00%)
    benchMatch # int........................R1 I10 - [Mo1.000μs vs. Mo1.875μs] -46.67% (±0.00%)

\Psl\Tests\Benchmark\Type\NonEmptyStringTypeBench

    benchCoerce # string....................R2 I8 - [Mo1.000μs vs. Mo1.375μs] -27.27% (±0.00%)
    benchCoerce # int.......................R2 I13 - [Mo1.125μs vs. Mo1.500μs] -25.00% (±0.00%)
    benchCoerce # instanceof Stringable (ex.R1 I5 - [Mo1.625μs vs. Mo2.500μs] -35.01% (±3.89%)
    benchCoerce # instanceof Stringable (im.R1 I11 - [Mo1.625μs vs. Mo2.500μs] -34.99% (±0.00%)
    benchAssert.............................R1 I7 - [Mo1.125μs vs. Mo1.375μs] -18.18% (±0.00%)
    benchMatch..............................R1 I7 - [Mo1.000μs vs. Mo1.250μs] -20.00% (±0.00%)

\Psl\Tests\Benchmark\Type\IntTypeBench

    benchCoerce # int.......................R1 I2 - [Mo1.000μs vs. Mo1.000μs] 0.00% (±0.00%)
    benchCoerce # string....................R3 I13 - [Mo1.250μs vs. Mo1.875μs] -33.32% (±0.00%)
    benchCoerce # float.....................R2 I5 - [Mo1.125μs vs. Mo1.250μs] -10.00% (±0.00%)
    benchCoerce # instanceof Stringable (ex.R5 I11 - [Mo1.750μs vs. Mo2.625μs] -33.35% (±3.69%)
    benchCoerce # instanceof Stringable (im.R2 I12 - [Mo1.750μs vs. Mo2.500μs] -30.01% (±0.00%)
    benchAssert.............................R1 I2 - [Mo1.125μs vs. Mo1.125μs] 0.00% (±0.00%)
    benchMatch..............................R1 I7 - [Mo0.875μs vs. Mo1.000μs] -12.50% (±0.00%)

Subjects: 21, Assertions: 0, Failures: 0, Errors: 0

Dependencies

• Prevent registering #[ParamProviders(...)] multiple times when benchmark methods are inherited phpbench/phpbench#918

[Ocramius]

@azjezz one big part that is relevant to me is optimizing the IterableType and VecType, which are intensively used in JSON decode operations in most biz domains I'm touching.

If you like where this is going (and before I add any more terrible things to it), I'll gladly expand the patch.

[Ocramius]

Note:

opcache ❌

I expect even better results when running with OPCache enabled, but don't have a local php installation where to try it out at this moment.

[Ocramius]

One failing test reports:

bash-5.1# ./tools/phpunit/vendor/bin/phpunit -c tools/phpunit/phpunit.xml.dist tests/unit/Type/ShapeAllowUnknownFieldsTypeTest.php 
PHPUnit 9.5.9 by Sebastian Bergmann and contributors.

Warning:       No code coverage driver available

...................F

Time: 00:00.020, Memory: 8.00 MB

There was 1 failure:

1) Psl\Tests\Unit\Type\ShapeAllowUnknownFieldsTypeTest::testValidCoercion with data set #1 (array('saif', '[email protected]', Psl\Collection\Vector Object (...)), array('saif', array(array('Foo', 'Baz', 0)), '[email protected]'))
Failed asserting that false is true.

/app/tests/unit/Type/TypeTest.php:111

Comparing $expected and $actual, I saw this:

array(3) {
  ["name"]=>
  string(4) "saif"
  ["articles"]=>
  array(1) {
    [0]=>
    array(3) {
      ["title"]=>
      string(3) "Foo"
      ["content"]=>
      string(3) "Baz"
      ["likes"]=>
      int(0)
    }
  }
  ["email"]=>
  string(18) "[email protected]"
}
array(3) {
  ["name"]=>
  string(4) "saif"
  ["email"]=>
  string(18) "[email protected]"
  ["articles"]=>
  array(1) {
    [0]=>
    array(3) {
      ["title"]=>
      string(3) "Foo"
      ["content"]=>
      string(3) "Baz"
      ["likes"]=>
      int(0)
    }
  }
}

I wonder if this scenario is valid? Is coerce() supposed to change the key order?

[azjezz]

If you like where this is going (and before I add any more terrible things to it), I'll gladly expand the patch.

This looks good to me so far 👍

I wonder if this scenario is valid? Is coerce() supposed to change the key order?

No 🤔 not sure why this is happening

[coveralls]

Pull Request Test Coverage Report for Build 1213779701

• 141 of 141 (100.0%) changed or added relevant lines in 8 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.0005%) to 99.904%

---

Totals
Change from base Build 1125701052:	0.0005%
Covered Lines:	3118
Relevant Lines:	3121

---

💛 - Coveralls

[Ocramius]

Latest results can be seen in the commit message of 68da707 - overall quite happy with it, but wish I could make VecType and DictType faster (couldn't find a way, they are O(n), and there's no shortcut, IMO).

I've resorted to improving some of the base types, where possible.

The ridiculous amount of scenarios is due to a @phpbench bug that I haven't yet investigated: it creates permutations of every benchmark scenario, because parent class / child class data providers are both considered, for some reason (parent class is supposed to only provide the signature).

Note: I've only run unit tests and benchmarks.

I'll fight Psalm/PHPCS another time :)

[Ocramius]

@azjezz I think this is good to go.

Here's the action run on my repo: https://github.com/Ocramius/psl/runs/3544847629

Benchmarks are running forever - hopefully fixed after phpbench/phpbench#918 :-\

EDIT: perhaps I need to remove the retry threshold, so it does not retry benchmarks when there's a lot of standard deviation error...

[azjezz]

Amazing work 🎉 Thank you!

( small issue )

The base branch was changed.

[azjezz]

Thank you for you work @Ocramius

[Ocramius]

@azjezz I see this isn't released yet - would a patch with release automation additions help on that?

Trying to kinda fill the time distance between improvements and discovered regressions :D

[azjezz]

would a patch with release automation additions help on that?

Yes!

i just didn't have the time for it :D i tried to integrate laminas auto release before but with no success

[bendavies]

think this is buggy.
keys need to be sorted before comparison. same on line 65