Lighthouse fails to produce accessibility score altogether for incomplete axe-core result

[rajsite]

FAQ

• Yes, my issue is not about variability or throttling.
• Yes, my issue is not about a specific accessibility audit (file with axe-core instead).

URL

https://axecore-lighthouse-failure.glitch.me/

What happened?

Running lighthouse on the above page fails to produce any accessibility score at all:

What did you expect?

It would be preferable for lighthouse to be robust to errors in axe-core and still produce an accessibility score.

We try to gate code submissions on accessibility scores and while lighthouse / axe-core reports plenty of other successful accessibility checks, it is completely masked by the incomplete check (in-terms of gating on accessibility checks and tracking history of the score).

I would expect:

• An accessibility score to still be calculated to prevent breaking PR gates completely based on score and fail to see general progress
• A link / more resources to report the incomplete result to axe-core

What have you tried?

Isolated and disabled lighthouse testing for pages using the component that causes the lighthouse failure.
Reported the incomplete result and reproducing example to axe-core: dequelabs/axe-core#4335
But would still like lighthouse to be more robust to axe-core errors and generate a useful score (might be a bit lower because of the error or maybe the error does not contribute to the score).

How were you running Lighthouse?

PageSpeed Insights, Chrome DevTools

Lighthouse Version

11.3.0

Chrome Version

121.0.6167.185

Node Version

No response

OS

No response

Relevant log output

axe-core Error: Cannot read properties of undefined (reading 'props')

as mentioned above, the axe-core error was reported at: dequelabs/axe-core#4335
but this issue is for improving lighthouses score calculation robustness to axe-core errors

[adamraine]

Thanks for filing a bug with axe-core, we will close this once it's fixed and the dependency is updated.

An accessibility score to still be calculated to prevent breaking PR gates completely based on score and fail to see general progress

If an axe check throws an error, we don't know if the audit passes and so we can't know what the accessibility score would be.

Maybe we can treat the audit as failing which could be less punishing than having no score at all, but I'm not completely sold on the idea.

A link / more resources to report the incomplete result to axe-core

I think this could be useful. 90% of the time issues related to accessibility we wind up upstreaming to axe-core.

[rajsite]

Maybe we can treat the audit as failing which could be less punishing than having no score at all, but I'm not completely sold on the idea.

In our case we had to disable the CI gate for the accessibility check for several weeks (until we had time to come up with a reproducing example and make the reports) in which time other accessibility issues that would have been caught otherwise were allowed through.

[adamraine]

We are not going to change how the accessibility category is scored since there is value in displaying errors loudly. Reporting this error to us when it happens is exactly what we want :)

If you weren't doing this already, you can use skipAudits to disable a single erroneous audit without disabling the entire accessibility score check.

---

I will leave this issue open because there is still an upstream issue and we should update the accessibility category description and docs to mention Axe and direct users to file bugs there.

[paulirish]

@rajsite It was fun to walk back the history in ni/nimble#1650 and see everything. In this case I think you managed to find a really nice bug in axe. It's wildly infrequent we see those sorts of DOM level exceptions from axe. So.. hats off to you for really stressing the system. :)