single POST requests #11
-
|
(Hi @kaliiiiiiiiii , sure, a new channel is more appropriate. I tried to do it by myself without success...) The question : how to make a post request using your scraper ? (taking profit of its antibot features + session) I tried without success the following process (my aim being to post a form online) :
Would be curious/interested about the interception method mentionned |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 12 replies
-
|
Assuming you used the right proxies, headers and data, and it still doesn't work: Looks like your wesite makes some other checks, and doesn't "trust" on valid cookies and constanly checks the browser. Personally, I'd prefer manual posts then (over .click(), .sendkeys, ..) , even if it takes longer, just to make shure the bot doesn't get detected (normal webiste ineraction). But as you asked for post methods, thats what I came up with:
import time
from selenium_profiles import driver as mydriver
from selenium_profiles.profiles import profiles
from selenium_interceptor.interceptor import cdp_listener # automatically gets installed with selenium-profiles
mydriver = mydriver()
profile = profiles.Windows()
#profile["options"]["browser"]["load_images"] = False # performance
profile["options"]["browser"]["gpu"] = True # gpu-fingerprinting
driver = mydriver.start(profile, uc_driver=False)
async def at_request(event, connection):
# event will return a https://chromedevtools.github.io/devtools-protocol/tot/Fetch/#event-requestPaused object
cdp_listener.print_event(event)
session, devtools = connection.session, connection.devtools
''' selenium.webdriver.common.devtools.v108.fetch.continue_request(): https://chromedevtools.github.io/devtools-protocol/tot/Fetch/#method-continueRequest -object:
:param request_id: An id the client received in requestPaused event.
:param url: *(Optional)* If set, the request url will be modified in a way that's not observable by page.
:param method: *(Optional)* If set, the request method is overridden.
:param post_data: *(Optional)* If set, overrides the post data in the request. (Base64)
:param headers: *(Optional)* If set, overrides the request headers.
:param intercept_response: **(EXPERIMENTAL)** *(Optional)* If set, overrides response interception behavior for this request.
'''
return devtools.fetch.continue_request(request_id=event.request_id)
cdp_listener = cdp_listener(driver=driver)
thread = cdp_listener.start_threaded(listener= {"listener":cdp_listener.requests,"at_event":at_request})
time.sleep(4) # wait for cdp_listener to start (experimental)
# specify interception patterns, for keys, have a look at https://chromedevtools.github.io/devtools-protocol/tot/Network/#type-RequestPattern
cdp_listener.specify_patterns([{"urlPattern":"*"}]) # Wildcards: ('*' -> zero or more, '?' -> exactly one) for "urlPattern"
# get url
driver.get('https://nowsecure.nl/#relax')
input("Press ENTER to quit..")
cdp_listener.terminate_all() # experimental, throws exception because trio async function gets lostto encrypt or decrypt post-data or request-body (base64) you can use the That won't help you initiating a post-request, but meybe to mess with it//modify the post content after sent. Note: cdp_listener and selenium-interceptor syntax might change in the future because it is in developement, means if you want a reliable package, you'll need to use one specific version |
Beta Was this translation helpful? Give feedback.
-
Let's say we want : |
Beta Was this translation helpful? Give feedback.
-
....
headers = headers = event.request.headers.to_json() # current headers
my_data = {'key1': 'value1', 'key2':'value2'} # encode to base64 string
my_headers = {'referer': 'https://www.example.fr', 'origin':'https://www.example.fr'}
my_cookies = cookies: {'cookie1':'cvalue1', 'cookie2':'cvalue2'} # I think the cookies gotta be parsed somehow into the headers, but don't know how?
my_data = cdp_listener.encode_body(decoded=my_data)
headers.update(my_headers) # append my headers
# format headers object
my_headers = []
for item in headers.items():
my_headers.append(devtools.fetch.HeaderEntry.from_json({"name": item[0], "value": item[1]}))
return devtools.fetch.continue_request(request_id=event.request_id, headers=my_headers, post_data=my_data)
... |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, I believe I was wrong (since it was not working, I found this explanation online without checking it...) My issue seems to be the following, how to both :
My POST request returns a 403 response with this page... |
Beta Was this translation helpful? Give feedback.
-
|
Well for that I con't help you more. Since it sems to be some cloudfare-browser check, it should automatiocally pass with Selenium-Profiles ( not in headless tho).. There's no (realistic) way passing this without a browser, unless the website thinks you're another browser instance (cookies..) Oh and I hope (and don't think) your website isn't www.xxx.xom .. |
Beta Was this translation helpful? Give feedback.
-
|
Haha, no it's not ! |
Beta Was this translation helpful? Give feedback.
-
|
@Pingo-v0 |
Beta Was this translation helpful? Give feedback.
-
|
Well yes, I agree, a simple requests.post with the right cookies, headers and proxy should do the trick |
Beta Was this translation helpful? Give feedback.
-
|
Well then the website detects, that you're not the same session as browser..
Try finding your desired post request using the devloper-console in chrome (network-tab) and have a look what values are sent.. |
Beta Was this translation helpful? Give feedback.
-
|
@Pingo-v0 maybe there's some authorisation-header you missed? |
Beta Was this translation helpful? Give feedback.
-
|
The website just recently added its new protection so I know that the post-data is valid (and that cookies + headers + ip rotations used to be valid) I'm extracting user_agent and cookies from your |
Beta Was this translation helpful? Give feedback.
-
|
Hmm well if you could provide me the url or sample request-and-post data (for example as XHR), I could have a look at it. (Could also be over E-Mail in case you don't wan to post it public, oh and you can censor the cookies and host-url in the request data of course) But if you can't tell me that, I absolutely understand that. |
Beta Was this translation helpful? Give feedback.
-
Post-requestsExample Script: post-data = {"message":"Hello world!"}
headers = {
"accept": "application/json",
"accept-encoding": "gzip, deflate, br",
"accept-language": profile["cdp"]["useragent"]["acceptLanguage"],
"content-type": "application/json",
"sec-ch-ua": "'Google Chrome';v='107', 'Chromium';v='107', 'Not=A?Brand';v='24'",
"sec-ch-ua-mobile": "?0", # "?1" for mobile
"sec-ch-ua-platform": "'" + profile['cdp']['useragent']['userAgentMetadata']['platform'] + "'",
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
"user-agent": profile['cdp']['useragent']['userAgent']
}
answer = driver.profiles.fetch("https://www.example.com/","body": ),
method="POST",
headers:dict=headers,
body=json.dumps(post_data)
)syntaxmethods = ["GET","HEAD" "POST", "PUT", "DELETE","OPTIONS"] # TRACE, CONNECT exluded here
supported_credentials = ["omit", "same-origin", "include"]
modes = ["cors", "no-cors", "same-origin"]
cache_values = ["default", "no-store", "reload", "no-cache", "force-cache", "only-if-cached"]
redirect_values = ["follow", "error"] # "manual" excluded here
referrer_policies = ["no-referrer", "no-referrer-when-downgrade", "same-origin", "origin", "strict-origin", "origin-when-cross-origin", "strict-origin-when-cross-origin", "unsafe-url"]
priorities = ["high", "low", "auto"]
EDIT:
|
Beta Was this translation helpful? Give feedback.
-
|
Closing as it seems to be resolved |
Beta Was this translation helpful? Give feedback.
Post-requests
Example Script: