Add Support for Safaridriver Debug Logs

[lread]

Problem/Opportunity
Safaridriver is different from other WebDriver implementations.
It has fewer features, and its docs induce head-scratching, at least for me.

When diagnosing mysterious behavior, it can be very helpful to enable WebDriver logging.
We support this easily for all WebDriver implementations, except for safaridriver, via :log-stdout and :log-stderr (and :driver-log-level).

When I need safaridriver logging, I will hack some temporary code into Etaoin to get it to spit out safaridriver debug logs. After the mystery is solved, the hack is abandoned, and how I achieved it fades from my memory.

Technical Details
Safaridriver debug logging is enabled via its --diagnose command line option.
There don't seem to be any logging levels to set.

Other WebDrivers send output to stdout/stderr, but safaridriver always writes its logs to ~/Library/Logs/com.apple.WebDriver/safaridriver.{safaridriver pid}.{random chars}.txt.

Proposed Solution
It would be nice to have a way to ask Etaoin to spit out safaridriver logs.

Interpret :driver-log-level of debug to turn safaridriver logging on.
Stream to log to stdout; this will allow the reuse of the existing :log-stdout feature.

Alternative Solutions
Don't stream log; spit it to stdout when the safaridriver process is closed.
This would be simpler to implement but less useful.

Additional context
The safaridriver PID is part of its log filename. Determining the safaridriver PID should be straightforward in jdk9+, but we currently still support jdk8+. If I decide to stick with jdk8+ support, whatever technique I come up with only has to work on macOS, the only platform where safaridriver is currently supported.

Action
I'll likely tackle this one. If he remembers, future-me might thank me.

[lread]

Although safaridriver --help suggests reading its man page for details:

Usage: safaridriver [options]
	-h, --help                Prints out this usage information.
	--version                 Prints out version information and exits.
	-p, --port                Port number the driver should use. If the server
	                          is already running, the port cannot be changed.
	                          If port 0 is specified, a default port will be used.
	--enable                  Applies configuration changes so that subsequent WebDriver
	                          sessions will run without further authentication.
	--diagnose                Causes safaridriver to log diagnostic information for
	                          all sessions hosted by this instance. See the safaridriver(1)
	                          man page for more details about diagnostic logging.

... man safaridriver does not find a man page, at lest on macOS 14.3.1.

The only recent-ish manpage I found on the web is here: https://gist.github.com/foolip/a0d7ac345a67a09a709ef7b28b3924ce

Contents pasted just in case gist goes poof:

SAFARIDRIVER(1)           BSD General Commands Manual          SAFARIDRIVER(1)

NAME
     safaridriver -- Safari WebDriver REST API service

SYNOPSIS
     safaridriver -p port [-h | --help] [--version] [--enable] [--diagnose]

DESCRIPTION
     The safaridriver utility is used to launch an HTTP server that implements the Selenium WebDriver
     REST API. When launched, safaridriver allows for automated testing of web content using the ver-
     sion of Safari that is installed with macOS.

     safaridriver supports several capabilities that can customize an automation session's behavior for
     a particular testing purpose. Capabilities are provided as arguments when requesting a new ses-
     sion.  The capabilities supported by safaridriver are listed in the OPTIONS section below. Unless
     noted below, the values of requested capability keys are not read and are assumed to be true if
     present, and false otherwise.

     To use capabilities in your tests, please refer to the relevant 3rd-party documentation to learn
     how to request extra capabilities with the WebDriver client library that you are using.

OPTIONS
   Command line options
     -p, --port <port>
              Specifies the port on which the HTTP server should listen for incoming connections. If
              the port is already in use or otherwise unavailable, safaridriver will exit immediately
              with a non-zero return code.

     -h, --help
              Prints a usage message and exits.

     --version
              Prints version information and exits.

     --enable
              Applies configuration changes so that subsequent WebDriver sessions will run without fur-
              ther authentication. This includes checking "Enable Remote Automation" in Safari's
              Develop menu. The user must authenticate via password for the changes to be applied.

              When this option is specified, safaridriver exits immediately without starting up the
              REST API service. If the changes were successful or already applied, safaridriver exits
              0; otherwise, safaridriver exits >0 and prints an error message to stderr.

     --diagnose
              Enables diagnostic logging for all sessions hosted by this safaridriver instance. See
              DIAGNOSTICS for more information.

   Session Creation Capabilities
     browserName
              safaridriver can only create WebDriver sessions for Safari. If the value of browserName
              is not equal to `Safari', session creation will fail.

              Values of browserName are compared case-insensitively.

     browserVersion
              safaridriver will only create a session using hosts whose Safari version matches the
              value of browserVersion.

              Browser version numbers are prefix-matched. For example, if the value of browserVersion
              is `12', this will allow hosts with a Safari version of `12.0.1' or `12.1`.

     platformName
              If the value of platformName is `mac' or `macOS', safaridriver will only create a session
              using the macOS host on which safaridriver is running.

              If the value of platformName is `iOS', safaridriver will only create a session on a
              paired iOS device or simulator.

              Values of platformName are compared case-insensitively.

     safari:platformVersion
              safaridriver will only create a session using hosts whose OS version matches the value of
              safari:platformVersion.

              OS version numbers are prefix-matched. For example, if the value of
              safari:platformVersion is `12', this will allow hosts with an OS version of `12.0' or
              `12.1' but not `10.12'.

     safari:platformBuildVersion
              safaridriver will only create a session using hosts whose OS build version matches the
              value of safari:platformBuildVersion. example of a macOS build version is `18E193'.

              On macOS, the OS build version can be determined by running the sw_vers(1) utility.

     safari:useSimulator
              If the value of safari:useSimulator is true, safaridriver will only use iOS Simulator
              hosts.

              If the value of safari:useSimulator is false, safaridriver will not use iOS Simulator
              hosts.

              NOTE: An Xcode installation is required in order to run WebDriver tests on iOS Simulator
              hosts.

     safari:deviceType
              If the value of safari:deviceType is `iPhone', safaridriver will only create a session
              using an iPhone device or iPhone simulator.

              If the value of safari:deviceType is `iPad', safaridriver will only create a session
              using an iPad device or iPad simulator.

              Values of safari:deviceType are compared case-insensitively.

     safari:deviceName
              safaridriver will only create a session using hosts whose device name matches the value
              of safari:deviceName.  Device names are compared case-insensitively.

              NOTE: Device names for connected devices are shown in iTunes.  If Xcode is installed,
              device names for connected devices are available via the output of instruments(1) and in
              the Devices and Simulators window (accessed in Xcode via "Window > Devices and Simula-
              tors").

     safari:deviceUDID
              safaridriver will only create a session using hosts whose device UDID matches the value
              of safari:deviceUDID.  Device UDIDs are compared case-insensitively.

              NOTE: If Xcode is installed, UDIDs for connected devices are available via the output of
              instruments(1) and in the Devices and Simulators window (accessed in Xcode via "Window >
              Devices and Simulators").

   Other Capabilities
     safari:automaticInspection
              This capability instructs Safari to preload the Web Inspector and JavaScript debugger in
              the background prior to returning a newly-created window. To pause the test's execution
              in JavaScript and bring up Web Inspector's Debugger tab, you can simply evaluate a
              debugger; statement in the test page.

     safari:automaticProfiling
              This capability instructs Safari to preload the Web Inspector and start a Timeline
              recording in the background prior to returning a newly-created window. To view the
              recording, open the Web Inspector through Safari's Develop menu.

     safari:diagnose
              This capability requests that diagnostics be enabled for the session.  See DIAGNOSTICS
              for more information.

     webkit:WebRTC
              This capability allows a test to temporarily change Safari's policies for WebRTC and
              Media Capture.  The value of the webkit:WebRTC capability is a dictionary with the fol-
              lowing sub-keys, all of which are optional:

                    DisableInsecureMediaCapture
                    Normally, Safari refuses to allow media capture over insecure connections. This
                    capability suppresses that restriction for testing purposes. For example, it would
                    allow a test to exercise media capture code paths using a local test web server
                    that is not configured to use HTTPS.

                    DisableICECandidateFiltering
                    To protect a user's privacy, Safari normally filters out WebRTC ICE candidates that
                    correspond to internal network addresses when capture devices are not in use. This
                    capability suppresses ICE candidate filtering so that both internal and external
                    network addresses are always sent as ICE candidates.

EXIT STATUS
     The safaridriver utility exits 0 on success, and >0 if an error occurs.

ERRORS
     When a REST API command fails, safaridriver includes a detailed error message in the response. If
     you use a 3rd-party library on top of the REST API service, consult the library's documentation
     for how to access these error messages.

SESSION CREATION
     safaridriver can create WebDriver sessions using Safari on a macOS machine, a paired iOS device,
     or an iOS simulator.  Capabilities listed in the Session Creation Capabilities subsection provide
     criteria that affect which hosts safaridriver will attempt to use when handling a New Session com-
     mand. A host must match all criteria to be usable.  If no hosts match all of the criteria, then
     the New Session command will fail.  If multiple hosts match all of the criteria, the order in
     which safaridriver will use them is unspecified, except that booted simulators are used before
     unbooted simulators.

     If a simulator host matches the criteria but is not booted, safaridriver will attempt to boot the
     simulator instance and wait for it to become usable.  If Safari is not running on a host that oth-
     erwise matches the criteria, safaridriver will automatically launch Safari and wait for it to
     become usable.  If a host's Safari instance is associated with an inactive WebDriver session,
     safaridriver will replace the old session unless the session was manually interrupted by the user
     or is being inspected by Web Inspector.

NOTES
     For security reasons, the HTTP server accepts connections from localhost only. The HTTP server can
     accept connections from multiple test clients. Safari on macOS and iOS can only host one WebDriver
     session at a time, so it is not recommended to run multiple safaridriver instances at the same
     time.

     safaridriver is typically executed manually at the command line or automatically by a WebDriver
     client library. The Selenium project provides client libraries for most popular programming lan-
     guages. More information is available on the Selenium project website:

           https://www.seleniumhq.org/

SEE ALSO
     A summary of which REST API endpoints safaridriver supports, as well as the Safari version in
     which each endpoint became available, can be found on the Apple Developer website:

           https://developer.apple.com/

STANDARDS
     safaridriver implements the W3C WebDriver specification:

           https://www.w3.org/TR/webdriver/

DIAGNOSTICS
     When filing a bug report against safaridriver, it is highly recommended that you capture and
     include diagnostics generated by safaridriver. This can be accomplished in several ways:

     To diagnose a single session, pass the safari:diagnose capability when requesting a new session.

     To diagnose all sessions from one safaridriver instance, use the --diagnose command line option.

     To diagnose all sessions in all instances of safaridriver, set the DiagnosticsEnabled default in
     the com.apple.WebDriver domain to YES using defaults(1).

     Diagnostic files are saved to ~/Library/Logs/com.apple.WebDriver/ and are uniquely named using the
     pid of safaridriver and a timestamp.  When using the safari:diagnose capability to turn on diag-
     nostics for a particular session, diagnostic files additionally include the session identifier in
     file names.

Darwin                         November 2, 2019                         Darwin

[lread]

I'll stick with my plan to use the --diagnose option.
In testing, I have discovered that for ~/Library/Logs/com.apple.WebDriver/safaridriver.{safaridriver pid}.{random chars}.txt, the {safaridriver pid} does not match the safaridriver process pid:

Clear out any existing logs...

rm ~/Library/Logs/com.apple.WebDriver/*

Start safaridriver:

$ safaridriver --diagnose --port 9999

From a separate terminal...

Force some logging:

$ curl http://localhost:9999/status
{"value":{"message":"","ready":true}}

Have a look at the generated log filename:

$ ls -l ~/Library/Logs/com.apple.WebDriver
total 8
-rw-r--r--@ 1 lee  staff  120 19 Mar 12:11 safaridriver.50629.qzqwjh.txt

Look at pid for safaridriver process.

$ ps -ef
  UID   PID  PPID   C STIME   TTY           TIME CMD
...
  502 50628 13512   0 12:10pm ttys000    0:00.02 safaridriver --diagnose --port 9999
...

Zut alors, 50628 does not match the filename PID of 50629.

So what process has a PID of 50629?

$ ps -ef
  UID   PID  PPID   C STIME   TTY           TIME CMD
...
  502 50629     1   0 12:10pm ??         0:00.14 /System/Library/PrivateFrameworks/WebDriver.framework/Versions/A/XPCServices/com.apple.WebDriver.HTTPService.xpc/Contents/MacOS/com.apple.WebDriver.HTTPService
...

And I don't see a process connection between the com.apple.WebDriver.HTTPService process and the safaridriver process. (Ie. safaridriver is not the parent process of com.apple.WebDriver.HTTPService).

Huh. That all jives with safaridriver being user-cruel.

[lread]

Option 1 - assume only 1 safaridriver running

I could make the assumption that the doc recommendations of only having one safaridriver running are followed. Could check for new log files created after launching safaridriver. There should only be one. Assume that the new log file belongs to the safaridriver process.

Pro:

• simple

Con:

• breaks if someone decides to run multiple safaridriver processes at the same time

Option 2 - dummy correlation id

Call WebDriver status with a dummy parameter set to a unique id. This unique id would be written to the log file and be used to correlate the file to the process.

Let's see if that might work:

# clear out existing logs again..
rm ~/Library/Logs/com.apple.WebDriver/*
# start up safaridriver
safaridriver --diagnose --port 9999

From another terminal:

$ curl "http://localhost:9999/status?etaoin-dummy=3b990e7a-64a0-4d3f-ad40-e0fbb22c5e5e"
{"value":{"message":"","ready":true}}

And let's take a peek to see if that dummy value is written:

$ ls ~/Library/Logs/com.apple.WebDriver
safaridriver.51172.rqvera.txt
$ cat 
2024/03/19 17:37:47:615  HTTPServer: GET http://localhost:9999/status?etaoin-dummy=3b990e7a-64a0-4d3f-ad40-e0fbb22c5e5e
2024/03/19 17:37:47:615  HTTPServer: --> HTTP 200

Pro:

• absolutely identifies log file to safaridriver

Con:

• breaks if safaridriver stops logging request, probably unlikely?
• breaks if safaridriver fails status request because of unrecognized parameter

Option 3 - Use lsof

Safaridriver is running on port 9999 (See setup in option 2)

$ lsof -ti :9999
51172

And this is the correct pid for the log file.

Pro:

• easy peasy

Con:

• breaks if safaridriver fixes what seems to be a bug, the pid should be for safaridriver process, not the com.apple.WebDriver.HTTPService process right?

[lread]

I am looking at optionally pretty printing log output as part of this work. My initial idea was to show prefixes like:

safari 1234 out| some stdout log line
safari 1234 err| some stderr log line
safari 1234 log| some safaridriver logfile log line
...

Where safari is the web driver type, 1234 is the web driver port, and out, err, and log distinguish the log line source.

A few rabbit holes later, I have not found a way to preserve the log line order of stdout and stderr and also distinguish them. So when pretty printing, my next plan is to redirect stderr to stdout and lose the distinction.

safari 1234| some stdout log line
safari 1234| some stderr log line
safari 1234 log| some safaridriver logfile log line
...

I think, for this use case, distinguishing log lines will work out ok, but we'll see.