Does Chrome-for-testing Canary (or any release) have experimentation or A/B testing things happening in the background?

[ab-plankton]

Hello!

In the past I wanted to try some performance tests on Canary (https://www.google.com/intl/en_ca/chrome/canary/), but if i recall correctly I abandoned that idea as there was experimentation / AB testing flags that I couldn't control and which were skewing results.

What I'd like to ask is, does the CfT version of Canary (or other release version) have these experiments happening?

Thank you

[ab-plankton]

@thiagowfx tagging you, hope it's ok :)

[92kns]

I am interested in knowing this as well and had a couple follow ups to the OP:
1 - if there is experimentation, is there a (convenient) way to disable it?
2 - is there any thing to be aware of with regards to running performance tests on CfT binaries? (i.e. anything unique with how these are built that might skew performance numbers when compared to a non-cft chrome binary?)

[mathiasbynens]

What I'd like to ask is, does the CfT version of Canary (or other release version) have these experiments happening?

IIUC, there are two types of “experiments”. Here’s how to check which are enabled:

• Those in chrome://flags (to disable all these, you could use the --no-experiments CLI flag)
• Finch-enabled experiments in chrome://version/?show-variations-cmd under “Active Variations”

I’ve asked the domain experts for more background on this. I’ll post here once I learn more.

is there any thing to be aware of with regards to running performance tests on CfT binaries? (i.e. anything unique with how these are built that might skew performance numbers when compared to a non-cft chrome binary?)

One difference is that CfT’s chrome binaries don’t have Profile-Guided Optimization (PGO) whereas regular Chrome binaries do.

[92kns]

@mathiasbynens thanks for the details! Do you have any reservations if I file an issue requesting PGO builds for CfT builds? That would be quite useful :)

[mathiasbynens]

@mathiasbynens thanks for the details! Do you have any reservations if I file an issue requesting PGO builds for CfT builds? That would be quite useful :)

That’d be perfectly alright :) Please file it here: https://goo.gle/cftbug

[mathiasbynens]

What I'd like to ask is, does the CfT version of Canary (or other release version) have these experiments happening?

IIUC, there are two types of “experiments”. Here’s how to check which are enabled:

• Those in chrome://flags (to disable all these, you could use the --no-experiments CLI flag)
• Finch-enabled experiments in chrome://version/?show-variations-cmd under “Active Variations”

I’ve asked the domain experts for more background on this. I’ll post here once I learn more.

The “Active Variations” mentioned above come from the field trial testing config at https://source.chromium.org/chromium/chromium/src/+/main:testing/variations/fieldtrial_testing_config.json.

[92kns]

@mathiasbynens thanks for the details! Do you have any reservations if I file an issue requesting PGO builds for CfT builds? That would be quite useful :)

That’d be perfectly alright :) Please file it here: https://goo.gle/cftbug

When I hit submit issue nothing seems to happen unfortunately. (I hit it about 3 times, so apologies if i've spammed it, but I see nothing)

I've instead filed an issue here in github #89 hope that's okay

[ab-plankton]

many thanks @mathiasbynens for the details !
is there a flag similar to --no-experiments that will also disable Finch-enabled experiments?

I also am interested in #89 about PGO as well, as you and @92kns have brought up!

[guest271314]

@mathiasbynens thanks for the details! Do you have any reservations if I file an issue requesting PGO builds for CfT builds? That would be quite useful :)

That’d be perfectly alright :) Please file it here: https://goo.gle/cftbug

I havn't been able to look in to my Google account for a couple weeks now. Not sure why. It would be helpful if there was a "Discussion" type tab here in this repository where users could report bugs.

E.g., CfT Version 121.0.6167.0 (Official Build) (64-bit) is crashing randomly on tabs with this

[13419:1:1205/125349.234842:FATAL:v8_initializer.cc(784)] V8 error: Cannot create a handle without a HandleScope (v8::HandleScope::CreateHandle()).
[13593:1:1205/125515.368016:FATAL:v8_initializer.cc(784)] V8 error: Cannot create a handle without a HandleScope (v8::HandleScope::CreateHandle()).