sapling getdeps cli build and test support for python 3.12 (#964)

Summary:
Ubuntu 24.04 and Fedora 40 are both python 3.12 based.  Update sapling's OSS getdeps build to allow build/run/test on python 3.12 while keeping 3.10 support.

* distutils is deprecated in python 3.12 stdlib, but fortunately setuptools is pretty similar and includes a vendored distutils, so:
  * added a manifest so we get python3-setuptools installed.
  * setup.py: removed deprecated usage like `find_executable` in favor of `shutil.which()`, also removed some super old xcode 4/5.1 detection I found
* util.py makedate():  datetime.utcfromtimestamp is deprecated in 3.12, so updated to use non-deprecated methods. Tested with test-command-template.t which shows tz offset.
* Makefile: disabled test-eager-exchange.t on 3.12 where updating the expectation to run on 3.12 and 3.10 is tricky (debug output differs)
* tests modified to run on 3.12 and 3.10:
  * test-import-eol.t:  updated for invalid escape sequence warnings
  * test-install.t: updated to filter out message that appears in different order on 3.12 vs earlier versions
  * test-merge-driver2.t: match different debug output in 3.12, remove check for mercurial package
  * test-sign-commit.t: updated to add --yes for newer gpg that python 3.12 using distros have
* found some tests needed bunzip2, so update manifest for sapling and bz2 to install it

Pull Request resolved: #964
X-link: facebook/openzl#1005

Test Plan:
sync the code somewhere visible to toolbox:

enter ubuntu 24.04 toolbox:
`toolbox enter ubuntu-toolbox-24.04 `

make sure system packages are installed
`./build/fbcode_builder/getdeps.py install-system-deps --recursive  sapling`

build sapling getdeps
`./build/fbcode_builder/getdeps.py build --allow-system-packages --no-facebook-internal --src-dir=. sapling  --project-install-prefix sapling:/`

run the tests locally to validate python 3.12 operation:
```
./build/fbcode_builder/getdeps.py --allow-system-packages test --src-dir=. sapling --num-jobs=64 --project-install-prefix sapling:/
---------------------------------------------------------------------
Skipped 2 tests (missing feature: Mononoke server available):
  test-fb-ext-remotefilelog-log.t
  test-fb-ext-remotefilelog-worker.t

Skipped 3 tests (missing feature: running tests with fsmonitor):
  test-fsmonitor-filemerge.t
  test-fsmonitor-nonutf8-path.t
  test-rust-hgevents.t

Skipped 48 tests (skipped):
  test-atexit-epipe.t
  test-casefolding.t
  test-check-clang-format.t
  test-check-code.t
  test-check-fix-code.t
  test-check-help.t
  test-checkserverbookmark.t
  test-chg.t
  test-commitcloud-sync-omission.t
  test-committer-date.t
  test-debugbacktrace.t
  test-debugshell-namespace.t
  test-eden-clone.t
  test-eden-commands.t
  test-eden-du.t
  test-eden-glob.t
  test-eden-sparse.t
  test-eden-watchman-edenapi-glob.t
  test-eden-watchman-noedenapi-glob.t
  test-fb-ext-fbconduit.t
  test-fb-ext-scm-prompt-compat.t
  test-fb-ext-scm-prompt-git.t
  test-fb-ext-tweakdefaults-grep.t
  test-fsmonitor-casefolding.t
  test-fsmonitor-fallback.t
  test-fsmonitor-metadata.t
  test-fsmonitor-refreshclock.t
  test-fsmonitor-status.t
  test-fsmonitor-warn-fresh.t
  test-histedit-reorder.t
  test-matcher-expand-globs.t
  test-mutation-infinitepush.t
  test-non-working-symlink.t
  test-progress-rust-renderer.t
  test-progressfile.t
  test-purge-fsmonitor.t
  test-remotenames-journal.t
  test-runlog.t
  test-rust-status-in-transaction.t
  test-sparse-track-ignored.t
  test-status-case-insensitive.t
  test-status-fresh-instance.t
  test-status-root-ignored-py.t
  test-status-watchman.t
  test-symlink-migration.t
  test-tracing-under-chg.t
  test-treestate-fresh-instance.t
  test-treestate-trackignore.t

# Ran 727 tests, 53 skipped, 0 failed.
passed on try 0
```

generate github actions CI to pick up python3-setuptools
```
./build/fbcode_builder/getdeps.py --allow-system-packages generate-github-actions --free-up-disk --os-type=linux --src-dir=. --output-dir=.github/workflows --job-name "Sapling CLI Getdeps " --job-file-prefix=sapling-cli-getdeps_ sapling --num-jobs=16  --project-install-prefix sapling:/
```

github CI

Reviewed By: quark-zju, singhsrb

Differential Revision: D63958742

Pulled By: ahornby

fbshipit-source-id: 460c42eb1315f2e1631c74d356e4976469104c1b

Author: ahornby

.github/workflows/sapling-cli-getdeps_linux.yml

@@ -70,6 +70,8 @@ jobs:
       run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests snappy
     - name: Fetch zlib
       run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests zlib
+    - name: Fetch python-setuptools
+      run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests python-setuptools
     - name: Fetch openssl
       run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests openssl
     - name: Fetch liboqs
@@ -142,6 +144,8 @@ jobs:
       run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --free-up-disk --no-tests snappy
     - name: Build zlib
       run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --free-up-disk --no-tests zlib
+    - name: Build python-setuptools
+      run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --free-up-disk --no-tests python-setuptools
     - name: Build openssl
       run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --free-up-disk --no-tests openssl
     - name: Build liboqs

build/fbcode_builder/manifests/bz2

@@ -3,12 +3,14 @@ name = bz2
 
 [debs]
 libbz2-dev
+bzip2
 
 [homebrew]
 bzip2
 
 [rpms]
 bzip2-devel
+bzip2
 
 [download]
 url = https://sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz

build/fbcode_builder/manifests/python-setuptools

@@ -7,3 +7,12 @@ sha256 = 02fa291a0471b3a18b2b2481ed902af520c69e8ae0919c13da936542754b4c56
 
 [build]
 builder = python-wheel
+
+[rpms]
+python3-setuptools
+
+[homebrew]
+python-setuptools
+
+[debs]
+python3-setuptools

build/fbcode_builder/manifests/sapling

@@ -65,6 +65,7 @@ hexdump
 
 [dependencies.not(os=windows)]
 python
+python-setuptools
 
 # We use the system openssl on linux
 [dependencies.not(os=linux)]

eden/integration/lib/find_executables.py

@@ -11,9 +11,9 @@
 
 # pyre-strict
 
-import distutils.spawn
 import logging
 import os
+import shutil
 import sys
 from pathlib import Path
 from typing import Any, Callable, Dict, Generic, List, Optional, Tuple, Type, TypeVar
@@ -192,9 +192,7 @@ def DROP_PRIVS(self) -> str:
 
     @cached_property
     def GIT(self) -> str:
-        git = distutils.spawn.find_executable(
-            "git.real"
-        ) or distutils.spawn.find_executable("git")
+        git = shutil.which("git.real") or shutil.which("git")
         if git is None:
             raise Exception("unable to find git binary")
         return git
@@ -224,7 +222,7 @@ def _find_hg(self) -> str:
         if hg_bin:
             return hg_bin
 
-        hg_real_bin = distutils.spawn.find_executable("hg")
+        hg_real_bin = shutil.which("hg")
         if hg_real_bin:
             return hg_real_bin
 
@@ -238,11 +236,11 @@ def _find_hg_real(self) -> str:
         if hg_real_bin:
             return hg_real_bin
 
-        hg_real_bin = distutils.spawn.find_executable("hg.real")
+        hg_real_bin = shutil.which("hg.real")
         if hg_real_bin:
             return hg_real_bin
 
-        hg_real_bin = distutils.spawn.find_executable("hg")
+        hg_real_bin = shutil.which("hg")
         if hg_real_bin:
             return hg_real_bin
 

eden/scm/Makefile

@@ -21,6 +21,8 @@ PYTHON := python
 endif
 PYTHON3 := python3
 
+PYTHON_SYS_EXECUTABLE=$(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3))
+
 BUCK_BUILD_MODE = @fbcode//mode/opt
 ifeq ($(OS),Windows_NT)
   BUCK_BUILD_MODE := @fbcode//mode/opt-win
@@ -31,7 +33,7 @@ else
   endif
 endif
 
-PYTHON_MINOR_VERSION=$(shell $(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) -c "import sys; print(sys.version_info.minor)")
+PYTHON_MINOR_VERSION=$(shell $(PYTHON_SYS_EXECUTABLE) -c "import sys; print(sys.version_info.minor)")
 
 ifeq ($(RUST_DEBUG),1)
 RUST_FLAG = --debug
@@ -105,7 +107,7 @@ install-oss: oss
 
 local:
 	SAPLING_OSS_BUILD=$(OSS) HGNAME=$(HGNAME) \
-	  $(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) setup.py $(PURE) \
+	  $(PYTHON_SYS_EXECUTABLE) setup.py $(PURE) \
 	  build_interactive_smartlog \
 	  build_clib $(COMPILERFLAG) \
 	  build_rust_ext -i -l $(RUST_FLAG) \
@@ -116,7 +118,7 @@ endif
 	cp build/scripts-3.$(PYTHON_MINOR_VERSION)/$(HG_BIN_NAME) $(HG_BIN_NAME)
 
 hg:
-	SAPLING_SKIP_OTHER_RUST_BINARIES=true $(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) setup.py $(PURE) \
+	SAPLING_SKIP_OTHER_RUST_BINARIES=true $(PYTHON_SYS_EXECUTABLE) setup.py $(PURE) \
 	  build_clib $(COMPILERFLAG) \
 	  build_rust_ext -i -l $(RUST_FLAG) \
 	  build_mo
@@ -145,9 +147,9 @@ getdepsbuild:
 	mkdir -p $(GETDEPS_BUILD_DIR)/sapling
 	ln -sfn $(GETDEPS_BUILD_DIR)/sapling build
 	HGNAME=$(HGNAME) GETDEPS_BUILD=1 \
-		PYTHON_SYS_EXECUTABLE=$(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) \
+		PYTHON_SYS_EXECUTABLE=$(PYTHON_SYS_EXECUTABLE) \
 		THRIFT="$(GETDEPS_INSTALL_DIR)/fbthrift/bin/thrift1" \
-		$(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) setup.py \
+		$(PYTHON_SYS_EXECUTABLE) setup.py \
 		$(PURE) build $(COMPILERFLAG)
 
 cleanbutpackages:
@@ -176,8 +178,8 @@ install-home: build
 	$(PYTHON) setup.py $(PURE) install --home="$(HOME)" --prefix="" --force
 
 install-getdeps:
-	PYTHON_SYS_EXECUTABLE=$(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) \
-	    GETDEPS_BUILD=1 $(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) \
+	PYTHON_SYS_EXECUTABLE=$(PYTHON_SYS_EXECUTABLE) \
+	    GETDEPS_BUILD=1 $(PYTHON_SYS_EXECUTABLE) \
 		setup.py $(PURE) install --skip-build --prefix="$(PREFIX)" --install-scripts="$(PREFIX)/bin"  --install-lib="$(PREFIX)/bin" --force && \
 		cd "$(PREFIX)/bin" && ln -f "$(SL_BIN_NAME)" "$(HG_BIN_NAME)"
 
@@ -218,7 +220,13 @@ GETDEPS_TEST_EXCLUSION_LIST := test-cats.t \
 	test-rust-checkout.t \
 	test-smartlog-interactive.t \
 	test-smartlog-interactive-highlighting.t
-	
+
+ifeq ($(PYTHON_MINOR_VERSION),12)
+# output differs for these on 3.12 in ways that are hard to match for 3.10 and 3.12 simultaneously
+GETDEPS_TEST_EXCLUSION_LIST := $(GETDEPS_TEST_EXCLUSION_LIST) \
+	test-eager-exchange.t
+endif
+
 # convert to a sed expression
 GETDEPS_TEST_EXCLUSIONS := $(subst $() $(),|,$(GETDEPS_TEST_EXCLUSION_LIST))
 
@@ -230,14 +238,14 @@ test-getdeps:
 	# ensure that fbpython is present, as some tests depend on it being on PATH
 	if ! which fbpython >/dev/null 2>&1; then \
 		FBPYTHON="$(GETDEPS_INSTALL_DIR)/sapling/bin/fbpython"; \
-		PYTHON_SYS_EXECUTABLE=$(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)); \
+		PYTHON_SYS_EXECUTABLE=$(PYTHON_SYS_EXECUTABLE); \
 		printf "#!/bin/sh\nexec \"$$PYTHON_SYS_EXECUTABLE\" \"\$$@\"\n"  > $$FBPYTHON; \
 		chmod +x "$$FBPYTHON"; \
 	fi;
 	# Run tests and retry any failures
 	export GETDEPS_BUILD=1; \
 	export HGTEST_HG=$(GETDEPS_INSTALL_DIR)/sapling/bin/$(HG_BIN_NAME); \
-	cd tests && export PYTHON_SYS_EXECUTABLE=$(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)); \
+	cd tests && export PYTHON_SYS_EXECUTABLE=$(PYTHON_SYS_EXECUTABLE); \
 	for try in $$(seq 0 $(GETDEPS_TEST_RETRY)); do \
 		RERUN_ARG=""; \
 		GETDEPS_TEST_FILTER=$(GETDEPS_TEST_FILTER); \
@@ -260,8 +268,8 @@ check: tests
 
 .PHONY: tests
 tests:
-	cd tests && PYTHON_SYS_EXECUTABLE=$(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) \
-	    $(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) run-tests.py
+	cd tests && PYTHON_SYS_EXECUTABLE=$(PYTHON_SYS_EXECUTABLE) \
+	    $(PYTHON_SYS_EXECUTABLE) run-tests.py
 
 update-pot: i18n/hg.pot
 

eden/scm/contrib/check-code.py

@@ -415,7 +415,7 @@ def rephere(m):
 
 txtpats = [
     [
-        ("\s$", "trailing whitespace"),
+        ("\\s$", "trailing whitespace"),
         (".. note::[ \n][^\n]", "add two newlines after note::"),
     ],
     [],