Improve robustness of reporting scripts

[fhoering]

As a follow up of the discussion in #1354 several limitations of reporting scripts came up.

They are summarized in the documentation:
https://github.com/WICG/turtledove/blob/main/PA_implementation_overview.md#phase-3---reporting-scripts

In particular reporting scripts compete for resources with bidding scripts and might be lost when the running auction frame is navigated away.

Reporting is one of the most important features as it is used for billing. So any robustness improvement made here would be useful. It should be noted that it might not be necessary to send out reports in strict real time.
Some delay of sending out those events could be acceptable if it improves robustness and improves resource availability for bidding and scoring.

[JensenPaul]

While it is true that in theory the reporting scripts compete with bidding scripts, in practice, given Chrome's queuing and dispatch and timeout logic, we don't anticipate this significantly affecting report reliability.

Similarly, while it is true that reports will not be sent if the parent frame is navigated away while reporting worklets are running, the window for this happening is very short as the default reporting timeout is 50ms and reporting worklets are started immediately after the auction completes, before the winning ad has started being navigated to. Even if the parent frame were closed before the reporting worklet finished, the ad rendering frame is unlikely to have made much progress towards displaying the ad in that brief window of time. The navigation to the ad may not have even begun, in which case the reports aren't supposed to be sent. It's unclear if buyers and sellers would consider such a marginal impression as an actual impression and want to receive reports about it.

[fhoering]

We conducted an AB test using Criteo DSP as a buyer and Criteo SSP as a component seller from 09/02/2025 to 13/02/2025 and playing with bidding script caching and reportingTimeout parameters on different CMA labels.

AB test setup

Population	CMA test label	seller reportingTimeout parameter	buyer bidding script caching strategy
A: Reference	Other labels	500 ms	private, max-age=900, stale-while-revalidate=3600
B: Small reportingTimeout	label_only_4	50 ms (default Chrome value)	private, max-age=900, stale-while-revalidate=3600
C: Small reportingTimeout + no cache	label_only_5	50 ms (default Chrome value)	private, no-cache

We performed a full outer join on the seller and buyer events, event level wise using an auction related join key, to detect:

• how many events overlap
• how many events were missing on seller side
• how many events were missing on buyer side

Results

• We observed a significant loss of 10% of buyers sendReportTo events compared to sellers sendReportTo events on population B & C (timeout 50ms). Bidding script caching in population B had no impact on the loss of events
• We observed only a 1% loss of buyer sendReportTo events compared to sellers sendReportTo events on the reference population (timeout 500ms)
• The 10% loss of buyer events applies to buyer's sendReportTo events and fenced frame beacons but not to forDebuggingOnly events. The buyers forDebuggingOnly event has only a small difference with the sellers sendReportTo event

To conclude, it seems to confirm that there might be issues with reacquiring the JavaScript execution context for buyer reporting. For each bid the buyer receives its forDebuggingOnly events, sellers scoring and reporting events are also sent out but we loose a significant part of buyer's reporting events.

As a short terms fix, in order to mitigate this effect, it looks reasonable to increase the default reportingTimeout value to 500ms or even 1 sec. However, it doesn’t look like a long term solution to just increase the default of reportingTimeout to a higher value as this might have other side effects on how bidding can be performed in other auctions on the same page. The reporting scripts generally contain few logic and should largely be finished within 50ms.

Also those observations are not marginal and somehow contradict the previous expectations on how it PAAPI reporting behaves:

While it is true that in theory the reporting scripts compete with bidding scripts, in practice, given Chrome's queuing and dispatch and timeout logic, we don't anticipate this significantly affecting report reliability.

So it looks important to fully understand what is happening here to be able to mitigate this effect.

Appendix

Example results of the share of seller only / buyer only sendReportTo events on each label

• event
  • “both”, we see the event on seller & buyer side
  • “buyer only”, we see the event on buyer side
  • “seller only”, we see the event on seller side
• the change of reducing reportingTimeout from 500ms to 50ms on on label_only_4 and label_only_5 was introduced on 2025-02-10
• example, on 2025-02-11 on label_only_4 we see
  • 90.5% both events
  • 0.54% buyer only events, no seller events
  • 8.9% seller only events, no buyer events

[JensenPaul]

This topic was discussed in the Feb 26, 2025 WICG call.
It was noted that the implementation overview may be useful to consult here.

[fhoering]

Thanks for the discussion in the WICG call.

The summary, with my understanding, is that the seller reporting script is executed straight away in the same JS context whereas there is a larger timeframe between buyer bid and buyer reporting. And therefore the buyer JS context (running in a process) gets released and instantiating the process from cache takes time. Bidding script caching didn't have an impact because this time is not taken into account in the reportingTimeout.

Overall the explanation seems clear. And the analysis gives some importance to the ticket that an improvement looks useful.

A note on applying the timeouts: It seems that bidding script network loading is included in other timeouts however like the perBuyerCumulative timeout as we discussed in #1355. Not saying it should be included here but it makes things difficult to understand as the loading from network is not included but the JS process instantiation actually is included.

[rdgordon-index]

The summary, with my understanding, is that the seller reporting script is executed straight away in the same JS context

Mine as well, based on https://github.com/WICG/turtledove/blob/main/PA_implementation_overview.md#phase-3---reporting-scripts, which reads:

On auction completion, Chrome immediately reuses the seller executor to call reportResult()