sapling getdeps cli build and test support for python 3.12 (#964)

Summary:
Ubuntu 24.04 and Fedora 40 are both python 3.12 based.  Update sapling's OSS getdeps build to allow build/run/test on python 3.12 while keeping 3.10 support.

* distutils is deprecated in python 3.12 stdlib, but fortunately setuptools is pretty similar and includes a vendored distutils, so:
  * added a manifest so we get python3-setuptools installed.
  * setup.py: removed deprecated usage like `find_executable` in favor of `shutil.which()`, also removed some super old xcode 4/5.1 detection I found
* util.py makedate():  datetime.utcfromtimestamp is deprecated in 3.12, so updated to use non-deprecated methods. Tested with test-command-template.t which shows tz offset.
* Makefile: disabled test-eager-exchange.t on 3.12 where updating the expectation to run on 3.12 and 3.10 is tricky (debug output differs)
* tests modified to run on 3.12 and 3.10:
  * test-import-eol.t:  updated for invalid escape sequence warnings
  * test-install.t: updated to filter out message that appears in different order on 3.12 vs earlier versions
  * test-merge-driver2.t: match different debug output in 3.12, remove check for mercurial package
  * test-sign-commit.t: updated to add --yes for newer gpg that python 3.12 using distros have
* found some tests needed bunzip2, so update manifest for sapling and bz2 to install it

Pull Request resolved: #964
X-link: facebookincubator/zstrong#1005

Test Plan:
sync the code somewhere visible to toolbox:

enter ubuntu 24.04 toolbox:
`toolbox enter ubuntu-toolbox-24.04 `

make sure system packages are installed
`./build/fbcode_builder/getdeps.py install-system-deps --recursive  sapling`

build sapling getdeps
`./build/fbcode_builder/getdeps.py build --allow-system-packages --no-facebook-internal --src-dir=. sapling  --project-install-prefix sapling:/`

run the tests locally to validate python 3.12 operation:
```
./build/fbcode_builder/getdeps.py --allow-system-packages test --src-dir=. sapling --num-jobs=64 --project-install-prefix sapling:/
---------------------------------------------------------------------
Skipped 2 tests (missing feature: Mononoke server available):
  test-fb-ext-remotefilelog-log.t
  test-fb-ext-remotefilelog-worker.t

Skipped 3 tests (missing feature: running tests with fsmonitor):
  test-fsmonitor-filemerge.t
  test-fsmonitor-nonutf8-path.t
  test-rust-hgevents.t

Skipped 48 tests (skipped):
  test-atexit-epipe.t
  test-casefolding.t
  test-check-clang-format.t
  test-check-code.t
  test-check-fix-code.t
  test-check-help.t
  test-checkserverbookmark.t
  test-chg.t
  test-commitcloud-sync-omission.t
  test-committer-date.t
  test-debugbacktrace.t
  test-debugshell-namespace.t
  test-eden-clone.t
  test-eden-commands.t
  test-eden-du.t
  test-eden-glob.t
  test-eden-sparse.t
  test-eden-watchman-edenapi-glob.t
  test-eden-watchman-noedenapi-glob.t
  test-fb-ext-fbconduit.t
  test-fb-ext-scm-prompt-compat.t
  test-fb-ext-scm-prompt-git.t
  test-fb-ext-tweakdefaults-grep.t
  test-fsmonitor-casefolding.t
  test-fsmonitor-fallback.t
  test-fsmonitor-metadata.t
  test-fsmonitor-refreshclock.t
  test-fsmonitor-status.t
  test-fsmonitor-warn-fresh.t
  test-histedit-reorder.t
  test-matcher-expand-globs.t
  test-mutation-infinitepush.t
  test-non-working-symlink.t
  test-progress-rust-renderer.t
  test-progressfile.t
  test-purge-fsmonitor.t
  test-remotenames-journal.t
  test-runlog.t
  test-rust-status-in-transaction.t
  test-sparse-track-ignored.t
  test-status-case-insensitive.t
  test-status-fresh-instance.t
  test-status-root-ignored-py.t
  test-status-watchman.t
  test-symlink-migration.t
  test-tracing-under-chg.t
  test-treestate-fresh-instance.t
  test-treestate-trackignore.t

# Ran 727 tests, 53 skipped, 0 failed.
passed on try 0
```

generate github actions CI to pick up python3-setuptools
```
./build/fbcode_builder/getdeps.py --allow-system-packages generate-github-actions --free-up-disk --os-type=linux --src-dir=. --output-dir=.github/workflows --job-name "Sapling CLI Getdeps " --job-file-prefix=sapling-cli-getdeps_ sapling --num-jobs=16  --project-install-prefix sapling:/
```

github CI

Reviewed By: quark-zju, singhsrb

Differential Revision: D63958742

Pulled By: ahornby

fbshipit-source-id: 460c42eb1315f2e1631c74d356e4976469104c1b

.github/workflows/sapling-cli-getdeps_linux.yml

@@ -70,6 +70,8 @@ jobs:
  run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests snappy
  - name: Fetch zlib
  run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests zlib
+ - name: Fetch python-setuptools
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests python-setuptools
  - name: Fetch openssl
  run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests openssl
  - name: Fetch liboqs
@@ -142,6 +144,8 @@ jobs:
  run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --free-up-disk --no-tests snappy
  - name: Build zlib
  run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --free-up-disk --no-tests zlib
+ - name: Build python-setuptools
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --free-up-disk --no-tests python-setuptools
  - name: Build openssl
  run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --free-up-disk --no-tests openssl
  - name: Build liboqs

build/fbcode_builder/manifests/bz2

@@ -3,12 +3,14 @@ name = bz2
 
 [debs]
 libbz2-dev
+bzip2
 
 [homebrew]
 bzip2
 
 [rpms]
 bzip2-devel
+bzip2
 
 [download]
 url = https://sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz

build/fbcode_builder/manifests/python-setuptools

@@ -7,3 +7,12 @@ sha256 = 02fa291a0471b3a18b2b2481ed902af520c69e8ae0919c13da936542754b4c56
 
 [build]
 builder = python-wheel
+
+[rpms]
+python3-setuptools
+
+[homebrew]
+python-setuptools
+
+[debs]
+python3-setuptools

build/fbcode_builder/manifests/sapling

@@ -65,6 +65,7 @@ hexdump
 
 [dependencies.not(os=windows)]
 python
+python-setuptools
 
 # We use the system openssl on linux
 [dependencies.not(os=linux)]

eden/integration/lib/find_executables.py

@@ -11,9 +11,9 @@
 
 # pyre-strict
 
-import distutils.spawn
 import logging
 import os
+import shutil
 import sys
 from pathlib import Path
 from typing import Any, Callable, Dict, Generic, List, Optional, Tuple, Type, TypeVar
@@ -192,9 +192,7 @@ def DROP_PRIVS(self) -> str:
 
  @cached_property
  def GIT(self) -> str:
- git = distutils.spawn.find_executable(
- "git.real"
- ) or distutils.spawn.find_executable("git")
+ git = shutil.which("git.real") or shutil.which("git")
  if git is None:
  raise Exception("unable to find git binary")
  return git
@@ -224,7 +222,7 @@ def _find_hg(self) -> str:
  if hg_bin:
  return hg_bin
 
- hg_real_bin = distutils.spawn.find_executable("hg")
+ hg_real_bin = shutil.which("hg")
  if hg_real_bin:
  return hg_real_bin
 
@@ -238,11 +236,11 @@ def _find_hg_real(self) -> str:
  if hg_real_bin:
  return hg_real_bin
 
- hg_real_bin = distutils.spawn.find_executable("hg.real")
+ hg_real_bin = shutil.which("hg.real")
  if hg_real_bin:
  return hg_real_bin
 
- hg_real_bin = distutils.spawn.find_executable("hg")
+ hg_real_bin = shutil.which("hg")
  if hg_real_bin:
  return hg_real_bin
 

eden/scm/Makefile

@@ -21,6 +21,8 @@ PYTHON := python
 endif
 PYTHON3 := python3
 
+PYTHON_SYS_EXECUTABLE=$(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3))
+
 BUCK_BUILD_MODE = @fbcode//mode/opt
 ifeq ($(OS),Windows_NT)
  BUCK_BUILD_MODE := @fbcode//mode/opt-win
@@ -31,7 +33,7 @@ else
  endif
 endif
 
-PYTHON_MINOR_VERSION=$(shell $(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) -c "import sys; print(sys.version_info.minor)")
+PYTHON_MINOR_VERSION=$(shell $(PYTHON_SYS_EXECUTABLE) -c "import sys; print(sys.version_info.minor)")
 
 ifeq ($(RUST_DEBUG),1)
 RUST_FLAG = --debug
@@ -105,7 +107,7 @@ install-oss: oss
 
 local:
  SAPLING_OSS_BUILD=$(OSS) HGNAME=$(HGNAME) \
- $(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) setup.py $(PURE) \
+ $(PYTHON_SYS_EXECUTABLE) setup.py $(PURE) \
  build_interactive_smartlog \
  build_clib $(COMPILERFLAG) \
  build_rust_ext -i -l $(RUST_FLAG) \
@@ -116,7 +118,7 @@ endif
  cp build/scripts-3.$(PYTHON_MINOR_VERSION)/$(HG_BIN_NAME) $(HG_BIN_NAME)
 
 hg:
- SAPLING_SKIP_OTHER_RUST_BINARIES=true $(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) setup.py $(PURE) \
+ SAPLING_SKIP_OTHER_RUST_BINARIES=true $(PYTHON_SYS_EXECUTABLE) setup.py $(PURE) \
  build_clib $(COMPILERFLAG) \
  build_rust_ext -i -l $(RUST_FLAG) \
  build_mo
@@ -145,9 +147,9 @@ getdepsbuild:
  mkdir -p $(GETDEPS_BUILD_DIR)/sapling
  ln -sfn $(GETDEPS_BUILD_DIR)/sapling build
  HGNAME=$(HGNAME) GETDEPS_BUILD=1 \
- PYTHON_SYS_EXECUTABLE=$(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) \
+ PYTHON_SYS_EXECUTABLE=$(PYTHON_SYS_EXECUTABLE) \
  THRIFT="$(GETDEPS_INSTALL_DIR)/fbthrift/bin/thrift1" \
- $(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) setup.py \
+ $(PYTHON_SYS_EXECUTABLE) setup.py \
  $(PURE) build $(COMPILERFLAG)
 
 cleanbutpackages:
@@ -176,8 +178,8 @@ install-home: build
  $(PYTHON) setup.py $(PURE) install --home="$(HOME)" --prefix="" --force
 
 install-getdeps:
- PYTHON_SYS_EXECUTABLE=$(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) \
- GETDEPS_BUILD=1 $(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) \
+ PYTHON_SYS_EXECUTABLE=$(PYTHON_SYS_EXECUTABLE) \
+ GETDEPS_BUILD=1 $(PYTHON_SYS_EXECUTABLE) \
  setup.py $(PURE) install --skip-build --prefix="$(PREFIX)" --install-scripts="$(PREFIX)/bin" --install-lib="$(PREFIX)/bin" --force && \
  cd "$(PREFIX)/bin" && ln -f "$(SL_BIN_NAME)" "$(HG_BIN_NAME)"
 
@@ -218,7 +220,13 @@ GETDEPS_TEST_EXCLUSION_LIST := test-cats.t \
  test-rust-checkout.t \
  test-smartlog-interactive.t \
  test-smartlog-interactive-highlighting.t
-
+
+ifeq ($(PYTHON_MINOR_VERSION),12)
+# output differs for these on 3.12 in ways that are hard to match for 3.10 and 3.12 simultaneously
+GETDEPS_TEST_EXCLUSION_LIST := $(GETDEPS_TEST_EXCLUSION_LIST) \
+ test-eager-exchange.t
+endif
+
 # convert to a sed expression
 GETDEPS_TEST_EXCLUSIONS := $(subst $() $(),|,$(GETDEPS_TEST_EXCLUSION_LIST))
 
@@ -230,14 +238,14 @@ test-getdeps:
  # ensure that fbpython is present, as some tests depend on it being on PATH
  if ! which fbpython >/dev/null 2>&1; then \
  FBPYTHON="$(GETDEPS_INSTALL_DIR)/sapling/bin/fbpython"; \
- PYTHON_SYS_EXECUTABLE=$(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)); \
+ PYTHON_SYS_EXECUTABLE=$(PYTHON_SYS_EXECUTABLE); \
  printf "#!/bin/sh\nexec \"$$PYTHON_SYS_EXECUTABLE\" \"\$$@\"\n" > $$FBPYTHON; \
  chmod +x "$$FBPYTHON"; \
  fi;
  # Run tests and retry any failures
  export GETDEPS_BUILD=1; \
  export HGTEST_HG=$(GETDEPS_INSTALL_DIR)/sapling/bin/$(HG_BIN_NAME); \
- cd tests && export PYTHON_SYS_EXECUTABLE=$(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)); \
+ cd tests && export PYTHON_SYS_EXECUTABLE=$(PYTHON_SYS_EXECUTABLE); \
  for try in $$(seq 0 $(GETDEPS_TEST_RETRY)); do \
  RERUN_ARG=""; \
  GETDEPS_TEST_FILTER=$(GETDEPS_TEST_FILTER); \
@@ -260,8 +268,8 @@ check: tests
 
 .PHONY: tests
 tests:
- cd tests && PYTHON_SYS_EXECUTABLE=$(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) \
- $(shell $(PYTHON3) contrib/pick_python.py $(PYTHON3)) run-tests.py
+ cd tests && PYTHON_SYS_EXECUTABLE=$(PYTHON_SYS_EXECUTABLE) \
+ $(PYTHON_SYS_EXECUTABLE) run-tests.py
 
 update-pot: i18n/hg.pot
 

eden/scm/contrib/check-code.py

@@ -415,7 +415,7 @@ def rephere(m):
 
 txtpats = [
  [
- ("\s$", "trailing whitespace"),
+ ("\\s$", "trailing whitespace"),
  (".. note::[ \n][^\n]", "add two newlines after note::"),
  ],
  [],