[Bug] Stealth being detected by Chrome DevTools Protocol (CDP)

[brauliobo]

Puppeeteer stealth is now being easily detected, checkout https://deviceandbrowserinfo.com/learning_zone/articles/detecting-headless-chrome-puppeteer-2024

[vladtreny]

It is not possible to fix it properly without modifying the Chromium source.

If anyone could start a project for a custom Chromium, would be great

[brauliobo]

It is possible to replace puppeteer with a websocket connection through a browser extension (loaded with --load-extension) and then control the browser with chrome.scripting.executeScript calls. I've tested and verified in a PoC.

I wonder though if it is enough to use Webdriver instead of CDP to communicate with Puppeteer

[vladtreny]

Interesting idea. Before puppeteer, I used extensions. Puppeteer is cool, but extensions provide additional useful apis.

You can also call cdp commands on the background of extension. It is not detectable.

[nwebson]

I made a patch for this issue, it disables Runtime.enable that causes this leak.
You can check it out here: https://github.com/rebrowser/rebrowser-patches

[vladtreny]

^ This way we lose a lot of funcs of puppeteer. We need a chromium patch... if anyone could start a project

[nwebson]

^ This way we lose a lot of funcs of puppeteer. We need a chromium patch... if anyone could start a project

@vladtreny I see no loss in functions of puppeteer after the patch. If you could find any, please let me know via issues section, I will be happy to address it.

[vladtreny]

console.log does not work
click does not work
other cdp functions do not work

[nwebson]

• console.log works just fine, but you don't get Runtime.consoleAPICalled events
• click and elements query work fine, too
• what other cdp functions?

[vladtreny]

Do you get even here?
page.on('console', async message => {

Also, show how do you click?

[nwebson]

page.on('console') relies on Runtime.consoleAPICalled, it won't work, that's true.
page.click(selector, clickOptions) - this one works fine with my patch

[vladtreny]

how does it detect element to click? can i select it correctly inside random 10 iframes?

does it click inside closed shadow root?

[nwebson]

@vladtreny it feels like I have to defend myself for something... I'm not trying to sell you anything.

You can try my solution, find any non-working stuff in patched version, and open a new issue for this. I will be glad to assist. Thanks.

[andrewmcwatters]

It seems less than optimal that one has to patch a browser to prevent this non-standard stack field from leaking side-effects.

Has anyone else here also tried just not using Chrome? A preliminary test from our end showed that the only leaked behavior from Firefox was window.navigator.webdriver, and I'm not sure what the state of the art is to patch that, but IIRC, it's not as simple as setting it to false.

[andrewmcwatters]

I should add that most detection strategies explicitly target Google Chrome, but there's far fewer explicit detection strategies for Firefox it seems. It's also Tor Browser's browser of choice for a reason.

[nwebson]

@andrewmcwatters since this post yesterday - https://hacks.mozilla.org/2024/08/puppeteer-support-for-firefox/ - I guess detection strategies will evolve quite quickly to target Firefox, too.

[andrewmcwatters]

Yeah, I'm wanting to diversify away from just automating with Google Chrome and stealth measures, since it's a bit of a risk factor at this point.

I'm not sure what WebDriver BiDi's equivalent of Page.addScriptToEvaluateOnNewDocument is, though, or if it has one. It might be necessary to build an equivalent event from existing standard ones.

[andrewmcwatters]

andrewmcwatters@Andrews-iMac redacted % node --test
▶ tests
  ✔ https://arh.antoinevastel.com/bots/ (5887.661417ms)
  ✔ https://arh.antoinevastel.com/bots/areyouheadless (2896.569986ms)
  ✔ BotD (1423.917009ms)
  ✖ Fingerprint Pro Bot Detection (3997.748462ms)
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    + actual - expected
    
    + 'You are a bot'
    - 'You are not a bot'
               ^
        at TestContext.<anonymous> (redacted)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async Test.run (node:internal/test_runner/test:857:9)
        at async Suite.processPendingSubtests (node:internal/test_runner/test:565:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: 'You are a bot',
      expected: 'You are not a bot',
      operator: 'strictEqual'
    }

  ✖ BrowserScan (1719.802858ms)
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    + actual - expected
    
    + 'Robot'
    - 'Normal'
        at TestContext.<anonymous> (redacted)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async Test.run (node:internal/test_runner/test:857:9)
        at async Suite.processPendingSubtests (node:internal/test_runner/test:565:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: 'Robot',
      expected: 'Normal',
      operator: 'strictEqual'
    }

▶ tests (15927.932604ms)
ℹ tests 5
ℹ suites 1
ℹ pass 3
ℹ fail 2
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 16215.926107

[vladtreny]

Need to patch chromium. Nothing to do.
All these hacks are detectable.

Also, the new protection in cloudflare via shadow root, is hard to bypass. Possible, but not ideal.

[ottodriver]

Seems to me that this or some other leak is affecting recaptcha bypass since yesterday.
Getting challenge 90%+ of the time on v2 and low score on v3.
Can anyone confirm?

[andrewmcwatters]

Need to patch chromium. Nothing to do. All these hacks are detectable.

Also, the new protection in cloudflare via shadow root, is hard to bypass. Possible, but not ideal.

Bummer.

[Vinyzu]

Need to patch chromium. Nothing to do.

All these hacks are detectable.

Also, the new protection in cloudflare via shadow root, is hard to bypass. Possible, but not ideal.

Its possible. See selenium-driverless. Im also currently working on an open-source (playwright based) solution.

[vladtreny]

Its possible. See selenium-driverless. Im also currently working on an open-source (playwright based) solution.

I know it... just rechecked if they added something new.
What kind of value does it bring? Turns off runtime? This way we lose a lot of features.
Including ability to bypass this new cloudflare shadow root.

Cloudflare reads these threads, but anyway :)

[nwebson]

Its possible. See selenium-driverless. Im also currently working on an open-source (playwright based) solution.

I know it... just rechecked if they added something new. What kind of value does it bring? Turns off runtime? This way we lose a lot of features. Including ability to bypass this new cloudflare shadow root.

Cloudflare reads these threads, but anyway :)

You keep saying about losing some features, but you never provide any specific code that stop working when Runtime is off. If you could, that would be really useful for the community.

[vladtreny]

For example, universally find an element in runtime.
Click inside shadowroot closed.

[andrewmcwatters]

Its possible. See selenium-driverless. Im also currently working on an open-source (playwright based) solution.

I know it... just rechecked if they added something new. What kind of value does it bring? Turns off runtime? This way we lose a lot of features. Including ability to bypass this new cloudflare shadow root.
Cloudflare reads these threads, but anyway :)

You keep saying about losing some features, but you never provide any specific code that stop working when Runtime is off. If you could, that would be really useful for the community.

It's also not desirable to use Selenium, but use a non-Selenium API.

[nwebson]

@vladtreny do you have any example of code that breaks after disabling Runtime.enable command?
@andrewmcwatters could you please clarify?

[Vinyzu]

Many features break after not enabling Runtime.
But as i said im currently working on a playwright solution, which fixes !every! issue.

[andrewmcwatters]

@vladtreny do you have any example of code that breaks after disabling Runtime.enable command? @andrewmcwatters could you please clarify?

I'm talking specifically about Selenium-Driverless, not so much about your patches disabling the Runtime.enable command. The README.md of https://github.com/kaliiiiiiiiii/Selenium-Driverless reads:

Note: This project is moving away from the selenium syntax

I'm not going to try and speak for everyone, but I think a lot of us are looking for drop-in solutions like yours.

My business uses Selenium, though, we don't use the other automation frameworks in part because they're explicitly not designed for anything other than testing. You can use them for other purposes, but it's leads to hacking around their APIs.

[NCLnclNCL]

Runtime.enable

you can wrap cdp to use, dont use Runtime.enable and Runtime.consoleAPICalled, it can detected
Pp, playwrite, selenium use Runtime.enable

[nwebson]

📣 I just made a post how to access main world objects from isolated context, please read it here: How to Access Main Context Objects from Isolated Context in Puppeteer & Playwright

ℹ️ TLDR: use page.evaluateOnNewDocument to listen for window messages and eval code, use page.evaluate to send window message with your payload and get response from the main world.
It worked just fine for several clients of mine, please let me know (discord or here) if it doesn't work for you.

[vladtreny]

@nwebson

can we add you to our Telegram group about anti-detect?
there are anti-detect browsers and other advanced bot writes

[nwebson]

@vladtreny sure, I'll be glad to join, my tg username is Nickwebson

[incapdns]

@Vinyzu What exactly does not enabling Runtime.enable break?
@vladtreny

For example, universally find an element in runtime.
Click inside shadowroot closed.

Can you give me a code example?
I don't understand what you mean.

[andrew-aladjev]

I've read carefully the blog post and found this sentence:

To restrict the serialization only to situations where CDP is being used,
we leverage the fact that Chrome buffers the console messages when the DevTools (CDP) are not open.

This is the key. You don't need to block Runtime.enable, you don't need to stop Runtime.consoleApiCalled and so on. Don't try to make the same behaviour for chrome and firefox stack variable, it is too hard. Just find a place where chrome are caching console messages and drop this if. Let chrome cache console messages without respect to CDP open or closed. And that's it.

[andrew-aladjev]

Where to find the code that is caching console messages? You can find it in v8/src/inspector/v8-console-agent-impl.cc:

Response V8ConsoleAgentImpl::enable() {
  if (m_enabled) return Response::Success();
  m_state->setBoolean(ConsoleAgentState::consoleEnabled, true);
  m_enabled = true;
  reportAllMessages();
   return Response::Success();
 }

void V8ConsoleAgentImpl::reportAllMessages() {
  V8ConsoleMessageStorage* storage =
      m_session->inspector()->ensureConsoleMessageStorage(
          m_session->contextGroupId());
  for (const auto& message : storage->messages()) {
    if (message->origin() == V8MessageOrigin::kConsole) {
      if (!reportMessage(message.get(), false)) return;
    }
  }
}

This code is reporting all messages that appeared in storage once devtools become enabled.

Another thing is in v8/src/inspector/v8-runtime-agent-impl.cc:

 Response V8RuntimeAgentImpl::enable() {
   if (m_enabled) return Response::Success();
 TRACE_EVENT_WITH_FLOW0(TRACE_DISABLED_BY_DEFAULT("v8.inspector"),
                        "V8RuntimeAgentImpl::enable", this,
                        TRACE_EVENT_FLAG_FLOW_OUT);
 m_inspector->client()->beginEnsureAllContextsInGroup(
     m_session->contextGroupId());
 m_enabled = true;
 m_state->setBoolean(V8RuntimeAgentImplState::runtimeEnabled, true);
 m_inspector->debugger()->setMaxCallStackSizeToCapture(
     this, V8StackTraceImpl::kDefaultMaxCallStackSizeToCapture);
 m_session->reportAllContexts(this);
 V8ConsoleMessageStorage* storage =
     m_inspector->ensureConsoleMessageStorage(m_session->contextGroupId());
 for (const auto& message : storage->messages()) {
   if (!reportMessage(message.get(), false)) break;
 }

You are free to do whatever you like with console messages. Everything you want to do with console message is inside v8/src folder. Unfortunately, I won't public my patches.

I've described the reason here.

In short: if I will public my patch, than Antoine Vastel and his command will try to find a way to work around me. I can public another patch and start endless chain. The only one way to break this chain and remain the winner is not to public your patch.

[incapdns]

Has anyone noticed that the mouseleave and mouseenter events are not emitted in the iframe when you are using the devtools inspect in headless mode?

Example:
http://127.0.0.1:9222/devtools/inspector.html?ws=127.0.0.1:9222/devtools/page/F58D8137DCC015804F5DF59C0AC9D383

You hover over the iframe and then leave, but the mouseleave is not emitted

[andrew-aladjev]

Sorry to interrupt, I want just to mention the source of this issue v8/src/inspector/value-mirror.cc:

// Build a description from an exception using the following pattern:
//   * The first line is "<name || constructor name>: <message property>". We
//     use the constructor name if the "name" property is "Error". Most custom
//     Error subclasses don't overwrite the "name" property.
//   * The rest is the content of the "stack" property but only with the actual
//     stack trace part.
String16 descriptionForError(v8::Local<v8::Context> context,
                             v8::Local<v8::Object> object) {
  {
    v8::Local<v8::Value> nameValue;
    if (object->Get(context, toV8String(isolate, "name")).ToLocal(&nameValue) && nameValue->IsString()) {
      ...
    }
  }

  {
    v8::Local<v8::Value> stackValue;
    if (object->Get(context, toV8String(isolate, "stack")).ToLocal(&stackValue) && stackValue->IsString()) {
      ...
    }
  }

  {
    v8::Local<v8::Value> messageValue;
    if (object->Get(context, toV8String(isolate, "message")).ToLocal(&messageValue) && messageValue->IsString()) {
      ...
    }
  }

So Antoine Vastel was wrong: stack is not a special variable and Error is not a special object. You may reproduce this issue using name, stack and message variables, for example:

var result = false;
var error = new Error();
Object.defineProperty(error, 'message', {
  get() {
    console.error('ololo message');
    result = true;
  }
});
console.warn(error);
console.log(result);

You may reproduce this issue with any object and any kind of property if you will be able to find related ->Get in v8 source code.

var result = false;
var array = [];
Object.defineProperty(array, 'length', {
  get() {
    console.error('ololo length');
    result = true;
  }
});
console.warn(array);
console.log(result);

You won't be able to reproduce this thing in regular chrome, because length can't be redefined. But I've dropped this exception and reproduced this vulnerability. I can't predict what objects and properties will be vulnerable today and what objects and properties will be vulnerable tomorrow. Someone will just add new ->Get, commit it and almost nobody will mention that new object and property became vulnerable.

I am sure you want just to drop m_enabled properties I've mentioned previously. The problem is that some parts of v8 code doesn't care about m_enabled. For example you've dropped m_enabled:

var result = false;
var error = new Error();
Object.defineProperty(error, 'name', {
  get() {
    console.error('ololo name');
    result = true;
  }
});
console.warn(error);

You will test it and found that vulnerability has gone. But when you will try to launch result from inspector console it will immediately call getter and result will still be true (despite devtools are formally disabled). So there are much more vulnerabilities in v8 inspector than you can imagine.

So I've found a reliable solution: block all InjectedScript::wrapObject calls that comes from console context. I am pretty sure it will be easy for you to implement this patch. But please don't publish your patches as I said. Thank you.

[incapdns]

@andrew-aladjev Do you are in https://discord.com/invite/vz7PeKk group ?