net::ERR_HTTP2_PROTOCOL_ERROR via PlayWright

[jeremysherriff]

I have a watch that used to work, that checks my local supermarket search results to see if my favourite coffee is on sale.
Recently it stopped working and always generates a "net::ERR_HTTP2_PROTOCOL_ERROR" error. All my research points to an issue on the server side, but the site works fine via any browser I try interactively.

Error is: net::ERR_HTTP2_PROTOCOL_ERROR at https://www.countdown.co.nz/shop/searchproducts?search=farmers%20origins%20india%20espresso

Watch URL: https://www.countdown.co.nz/shop/searchproducts?search=farmers%20origins%20india%20espresso
Request fetch method: Playwright Chromium/Javascript via 'ws://playwright-chrome:3000/?stealth=1&--disable-web-security=true'
xpath filter: xpath1:/html/body/wnz-content/div/wnz-search/div[1]/main/product-grid/cdx-card/product-stamp-grid/a

ChangeDetection docker version: v0.45.8.1
Playwright version: v1.29.2 (current)

I've tried routing Playwright via a local proxy - same result.
I've tried to force Playwright to use http1.1 (due to the HTTP2 part of the error) but that made no difference.

No idea what else to try. Hoping someone can help!

TLDR It's a possible anti-robot trick, see the recommended answer

[kevindelaney]

Also experiencing this same issue. Wondering if you ever found a fix?

[jeremysherriff]

No, still having the same situation today.
I suspect the site is detecting that the browser session is not interactive. I haven't got enough technical knowledge to see if I can use cookies to get further.

[dgtlmoon]

Could anyone reading this try and perform curl -v https://thesite.com... and paste the output?

[kevindelaney]

I ended up resolving this error last night by:

1. Ensuring the following environment variables were present in my docker-compose for playwright:

  - SCREEN_WIDTH=1920
  - SCREEN_HEIGHT=1024
  - SCREEN_DEPTH=16
  - ENABLE_DEBUGGER=false
  - PREBOOT_CHROME=true
  - CONNECTION_TIMEOUT=300000
  - MAX_CONCURRENT_SESSIONS=10
  - CHROME_REFRESH_TIME=600000
  - DEFAULT_BLOCK_ADS=true
  - DEFAULT_STEALTH=true
  - DEFAULT_IGNORE_HTTPS_ERRORS=true

2. Specifying a User Agent in the request headers (under Request > Show Advanced Options)

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

[dgtlmoon]

Could anyone reading this try and perform curl -v https://thesite.com... and paste the output?

(and which URL)

[jeremysherriff]

@dgtlmoon here you go:

jeremy@toolbox:~$ curl -v https://www.countdown.co.nz
*   Trying 104.95.105.50:443...
* Connected to www.countdown.co.nz (104.95.105.50) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=AU; ST=New South Wales; L=Bella Vista; O=Woolworths Group Limited;                                                        CN=woolworths.com.au
*  start date: Aug 14 00:00:00 2023 GMT
*  expire date: Aug 14 23:59:59 2024 GMT
*  subjectAltName: host "www.countdown.co.nz" matched cert's "*.countdown.co.nz"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55e009d16980)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: www.countdown.co.nz
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)
* stopped the pause stream!
* Connection #0 to host www.countdown.co.nz left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

[jeremysherriff]

This seems really relevant: https://lwthiker.com/networks/2022/06/17/http2-fingerprinting.html
I have been unable to find any way to disable http2 in browserless-chrome

[jeremysherriff]

The response from @kevindelaney solved it for me, specifically the User-Agent header:

2. Specifying a User Agent in the request headers (under Request > Show Advanced Options)

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

Unless @dgtlmoon comes up with something extra/different based on the curl -v output, I reckon this is the answer.
Probably would be useful to add a few more headers too, specifically some Accept values. My full header for watchers that use PlayWright is:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Language: en-GB,en-NZ;q=0.9,en-AU;q=0.8,en;q=0.7,en-US;q=0.6
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36

I got these values from my browser dev mode, so they are exactly my current interactive settings.

[dgtlmoon]

@jeremysherriff super interesting! i'll give it a try, thanks

[MikeChristianson]

2. Specifying a User Agent in the request headers (under Request > Show Advanced Options)

It's also possible to set User-Agent, and other headers, globally using a /datastore/headers.txt file.

[dgtlmoon]

@MikeChristianson has setting a user-agent helped?

[plangin]

Encountered the same issue with change detection 0.45.14 and the sockpuppetbrowser on tesla.com

[yyakee]

Just bought the subscription to make my work less time consuming, now I get this error all the time
Any fix before I start to regret buying the subscription?

[dgtlmoon]

Since you havent supplied any information (URL you are following, or even email'ed the customer support by using the contact form on the bottom of the webpage) its impossible to help you

its not possible to help people who dont provide any information

[yyakee]

You are right, I am pretty new to this and have low to none knowledge, I overreacted

the url I am trying to watch is called https://zelenka.guru/. When I try to step by step login, the typicall ERR_HTTP2_PROTOCOL_ERROR appears

[dgtlmoon]

@yyakee i can confirm that i get that error in BrowserSteps but not just scanning for differences :( looking into it

[dgtlmoon]

"net::ERR_HTTP2_PROTOCOL_ERROR" happens when in headless mode but not 'non headless' - i'm wondering if its some anti robot detection,their SSL for zelenka.guru looks good in cURL

[yyakee]

Thank you! - Am I able to run it in 'non headless'?

[tkg61]

TLDR - It is because of their anti-robot protection - I am also having this issue with disney's website:

URL: https://disneycruise.disney.go.com
Playwright Chromium/Javascript via 'ws://playwright-chrome:3000?stealth=1&--disable-web-security=true'

Advanced Request Headers:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Language: en-GB,en-NZ;q=0.9,en-AU;q=0.8,en;q=0.7,en-US;q=0.6
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36

docker-compose:

        hostname: playwright-chrome
        image: dgtlmoon/sockpuppetbrowser:latest
        restart: unless-stopped
        environment:
            - SCREEN_WIDTH=1920
            - SCREEN_HEIGHT=1024
            - SCREEN_DEPTH=16
            - MAX_CONCURRENT_CHROME_PROCESSES=10
            - ENABLE_DEBUGGER=false
            - PREBOOT_CHROME=true
            - CONNECTION_TIMEOUT=300000
            - MAX_CONCURRENT_SESSIONS=10
            - CHROME_REFRESH_TIME=600000
            - DEFAULT_BLOCK_ADS=true
            - DEFAULT_STEALTH=true
            - DEFAULT_IGNORE_HTTPS_ERRORS=true

The visual filter selector will work but not the browser steps. Need to be able to login...

[dgtlmoon]

yes, I set those headers in my headers.txt and it solved it.. I have no idea why, I suspect its some kind of anti-bot trick

However I can report that it didnt solve any of the 403 access denied problems

[tkg61]

Ok I will add the header to the text file and try again later, it didn’t work in the advanced settings header section but maybe there is a disconnect somewhere with that?

[edwardyuanslalom]

Added an user agent based on @kevindelaney recommendation under Request - Request headers e.g.

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

Worked for me against Ticktek website, which I am certain there is bot protection enabled.