-
-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
aws lambda support? #434
Comments
I definitely want to support this, but this is going to take some effort to troubleshoot. I think we need to figure out what isn't working on Lambda. I'm guessing Lambda is balking either at the size of Chrome, or the architecture on the machine is somehow incompatible with the Chrome builds we have. I think the starting place is to look at how Playwright and Puppeteer are getting running on AWS. You can change out the "Chrome" that SecretAgent is using as a temporary measure by using the environment variable |
Hi! I tried with latest chomium 88 binary from https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Linux/ and as before it works just fine on local but on lambda I got an error, a different one this time: On the lambda call where the exception error is logged: On the cloundfront log This two flags are being used to launch chrome: '--homedir=/tmp', I have no idea where that .validated dir comes from, is it something custom from the browser emulator? Edit So I noticed that .validated "dir" is just an empty file on a default secret-agent install I touched it and now it loads but the launch args are being ignored, which seems quite strange. Edit2 I see the .validated file flags if the current browser dependencies are met. I tried to load a different chromium 88 binary, this time the build number was closest to the default one in secret agent, the build was so "old" that it used gtk2 and thus different dependencies were needed I had to resort to symlinking the bzip2 .so because the build was expecting libbzip2.1.0 instead onf libbzip2.1 or libbzip2.1.06 which was what amazon linux bzip2 package provides. This time the browser loaded but failed with the error: Gtk-WARNING **: 14:39:47.950: cannot open display: ' this was in the local enviroment I also used the --disable-gpu and --headless launch args for chrome Edit 3 Tried with the binary dev-headless-chromium-88.0.4298.4-amazonlinux-2017-03.zip from: https://github.com/adieuadieu/serverless-chrome/releases And got this error on aws lambda, works just fine in the docker container locally:
There seems to be something wrong with the communication with the core, could it be some mismatch on chrome dev tools/debug protocols or whatever secret agent is using to communicate with chrome? Mind you I haven't got yet full understanding on how secret agent operates in this regard. |
Thanks for running these experiments! It's been awhile since I've used Lambda. You used to have to pre-compile native modules and package them. Is that still true? You're correct - the .validated folder is whether we've checked linux dependencies. On debian based Operating systems, we have a .deb file that will install any missing deps so that Chrome has everything it needs. I'm not sure if the binaries you're finding are coming with everything or not. This is an example of one of the projects I'd seen in the past that precompiled some binaries: https://github.com/alixaxel/chrome-aws-lambda/tree/master/bin Regarding Core->Client, SecretAgent is forking a process and then (if you are using the default full-client project), it communicates over a socket to that process. I'm not sure what restrictions Lambda puts on launching processes. I'm guessing there are probably some other logs that are happening that show something else breaking prior to this? |
Well for deploying to aws I'm using docker images which you can build taking base on other images, there's one with node14 provided by amazon, so it's just a matter of installing any node modules with npm install and the required dependencies of those modules (in this case not only the modules but the chrome binary too) on the base system ex: gtk, alsa, etc.... Using the binary from chrome-aws-lambda I think I have it working but with a lot of kludges, still need to do more testing code/cleanup. |
Hi, although I got it to run under aws-lambda I couldn't get clodflare checks to pass, and cloudflare support is the main feature I need to get working for my specific use case. Here's a detalied description on how I got it to run: I'm using aws-lambda docker images built with this dockerfile:
I had to resort to hardcoded launch args replacing the whole configureBrowserLaunchArgs.js file as I couldn't get the launchargs plugin to work, the plugin worked just fine on my local machine inside the docker container and out of it, it didn't worked in the real lambda enviroment, it was giving some errors related to the connection to Core. The flags I add are:
Package.json:
app.js:
I guess the differences between the binary provided by chrome-aws-lambda and the default binary in secret agent are causing it to fail in passing the cloudflare checks. I tried with other versions of the chrome-aws-lambda package (one with chrome version 88.xx) and found no differences in the outcome. Will a different build of the secret agent binary be needed to support aws-lambda and pass cloudflare checks? Here are some logs: With default binary, the lambda fails with the aforementioned PipeTransport error:
With chrome-aws-lambda binary, the lambda runs fine but cloudflare checks are not passed and ends up in a timeout error, on local (inside container or outside container) it usually passes the test on the first 5 retries:
|
I guess it's unsupported right now but... would it be possible to run secret-agent on aws lambda with or without any limitations?
I'm trying to get it running but I keep getting PipeTransport errors on the real aws enviroment while on a local invocation works just fine.
This is an excerpt of the log:
`2022-01-28T11:42:40.802Z ERROR [/var/task/node_modules/@secret-agent/puppet/lib/PipeTransport] PipeTransport.WriteError { context: {}, sessionId: �[1mnull�[22m, sessionName: �[90mundefined�[39m } Error: read ECONNRESET
�[90m at Pipe.onStreamRead (internal/stream_base_commons.js:209:20)�[39m {
errno: �[33m-104�[39m,
code: �[32m'ECONNRESET'�[39m,
syscall: �[32m'read'�[39m
}
2022-01-28T11:42:40.803Z ERROR [/var/task/node_modules/@secret-agent/puppet/lib/PipeTransport] PipeTransport.WriteError { context: {}, sessionId: �[1mnull�[22m, sessionName: �[90mundefined�[39m } Error: read ECONNRESET
�[90m at Pipe.onStreamRead (internal/stream_base_commons.js:209:20)�[39m {
errno: �[33m-104�[39m,
code: �[32m'ECONNRESET'�[39m,
syscall: �[32m'read'�[39m
}
2022-01-28T11:42:40.804Z INFO [/var/task/node_modules/@secret-agent/puppet/lib/PipeTransport] PipeTransport.Closed { context: {} }
2022-01-28T11:42:40.820Z STATS [/var/task/node_modules/@secret-agent/puppet/lib/BrowserProcess] chrome.ProcessExited { context: {} }
2022-01-28T11:42:40.821Z INFO [/var/task/node_modules/@secret-agent/core/lib/Session] Session.Closing { context: {} }`
Looking at the stacktrace it seems that there's some problem while launching chrome and thus the connection is closed.
The text was updated successfully, but these errors were encountered: