Windows Support: Testing Suite integration

Broaden support for execution of the test suite
on Windows.
General bug and review fixups

.github/workflows/windows_python.yml

@@ -3,13 +3,12 @@ name: windows tests
 on:
   push:
     branches:
-      - features/windows-support
-      - windows-ci*
+      - develop
+      - releases/**
   pull_request:
     branches:
-      - features/windows-support
-      - windows-ci*
       - develop
+      - releases/**
 defaults:
   run:
     shell:
@@ -70,7 +69,7 @@ jobs:
     - name: Unit Test
       run: |
         echo F|xcopy .\spack\share\spack\qa\configuration\windows_config.yaml $env:USERPROFILE\.spack\windows\config.yaml
-        spack unit-test -x --verbose --ignore=lib/spack/spack/test/cmd
+        spack unit-test --verbose --ignore=lib/spack/spack/test/cmd
   unittest-cmd:
     runs-on: windows-latest
     steps:
@@ -89,7 +88,7 @@ jobs:
     - name: Command Unit Test
       run: |
         echo F|xcopy .\spack\share\spack\qa\configuration\windows_config.yaml $env:USERPROFILE\.spack\windows\config.yaml
-        spack unit-test lib/spack/spack/test/cmd -x --verbose
+        spack unit-test lib/spack/spack/test/cmd --verbose
   buildtest:
     runs-on: windows-latest
     steps:

.gitignore

@@ -269,11 +269,6 @@ local.properties
 # CDT-specific (C/C++ Development Tooling)
 .cproject
 
-# VSCode files
-.vs
-.vscode
-.devcontainer
-
 # CDT- autotools
 .autotools
 

bin/spack_cmd.bat

@@ -68,4 +68,5 @@ set
 GOTO:EOF
 
 :continue
+set PROMPT=[spack] %PROMPT%
 %comspec% /k

lib/spack/docs/getting_started.rst

@@ -1580,8 +1580,8 @@ Python 3 can be downloaded and installed from the Windows Store, and will be aut
 to your ``PATH`` in this case.
 
 .. note::
-
    Spack currently supports Python versions later than 3.2 inclusive.
+
 """
 Git
 """
@@ -1594,12 +1594,10 @@ When given the option of adjusting your ``PATH``, choose the ``Git from the
 command line and also from 3rd-party software`` option. This will automatically
 update your ``PATH`` variable to include the ``git`` command.
 
-.. note::
-
-   Spack support on Windows is currently dependent on installing the Git for Windows project
-   as the project providing Git support on Windows. This is additionally the recommended method
-   for installing Git on Windows, a link to which can be found above. Spack requires the
-   utilities vendored by this project.
+Spack support on Windows is currently dependent on installing the Git for Windows project
+as the project providing Git support on Windows. This is additionally the recommended method
+for installing Git on Windows, a link to which can be found above. Spack requires the
+utilities vendored by this project.
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Step 2: Install and setup Spack
@@ -1609,32 +1607,20 @@ We are now ready to get the Spack environment set up on our machine. We
 begin by using Git to clone the Spack repo, hosted at https://github.com/spack/spack.git
 into a desired directory, for our purposes today, called ``spack_install``.
 
-Presently, Windows operations are supported by Spack soley through the
-features/windows-support branch on the upstream Spack repository,
-located at the above url.
-
 In order to install Spack with Windows support, run the following one liner
 in a Windows CMD prompt.
 
 .. code-block:: console
 
-   git clone https://github.com/spack/spack.git -b features/windows-support win_spack
-
-or if working from a previous clone of Spack, simply checkout the Windows support feature branch
-with
-
-.. code-block:: console
-
-  git checkout -b features/windows-support --track <spack_upstream>/features/windows-support
+   git clone https://github.com/spack/spack.git
 
 .. note::
+   If you chose to install Spack into a directory on Windows that is set up to require Administrative
+   Privleges, Spack will require elevated privleges to run.
+   Administrative Privleges can be denoted either by default such as
+   ``C:\Program Files``, or aministrator applied administrative restrictions
+   on a directory that spack installs files to such as ``C:\Users``
 
-  If you chose to install Spack into a directory on Windows that is set up to require Administrative
-  Privleges*, Spack will require elevated privleges to run.
-
-  *Administrative Privleges can be denoted either by default such as
-  `C:\Program Files`, or aministrator applied administrative restrictions
-  on a directory that spack installs files to such as `C:\Users\`
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Step 3: Run and configure Spack
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -1646,6 +1632,9 @@ and its prerequisites. If you receive a warning message that Python is not in yo
 of the Python executable to your ``PATH`` now. You can permanently add Python to your ``PATH`` variable
 by using the ``Edit the system environment variables`` utility in Windows Control Panel.
 
+.. note::
+   Alternatively, Powershell can be used in place of CMD
+
 To configure Spack, first run the following command inside the Spack console:
 
 .. code-block:: console
@@ -1712,6 +1701,7 @@ and not tabs, so ensure that this is the case when editing one directly.
    The use of Cygwin is not officially supported by Spack and is not tested.
    However Spack will not throw an error, so use if choosing to use Spack
    with Cygwin, know that no functionality is garunteed.
+
 ^^^^^^^^^^^^^^^^^
 Step 4: Use Spack
 ^^^^^^^^^^^^^^^^^

lib/spack/external/ctest_log_parser.py

@@ -71,8 +71,6 @@
 import re
 import math
 import multiprocessing
-import sys
-import threading
 import time
 from contextlib import contextmanager
 
@@ -411,12 +409,7 @@ def parse(self, stream, context=6, jobs=None):
             pool = multiprocessing.Pool(jobs)
             try:
                 # this is a workaround for a Python bug in Pool with ctrl-C
-                if sys.version_info >= (3, 2):
-                    max_timeout = threading.TIMEOUT_MAX
-                else:
-                    max_timeout = 9999999
-                results = pool.map_async(_parse_unpack, args, 1).get(max_timeout)
-
+                results = pool.map_async(_parse_unpack, args, 1).get(9999999)
                 errors, warnings, timings = zip(*results)
             finally:
                 pool.terminate()

lib/spack/external/macholib/util.py

@@ -6,8 +6,6 @@
 
 from macholib import mach_o
 
-from llnl.util.symlink import symlink
-
 MAGIC = [
     struct.pack("!L", getattr(mach_o, "MH_" + _))
     for _ in ["MAGIC", "CIGAM", "MAGIC_64", "CIGAM_64"]
@@ -142,7 +140,7 @@ def mergetree(src, dst, condition=None, copyfn=mergecopy, srcbase=None):
         try:
             if os.path.islink(srcname):
                 realsrc = os.readlink(srcname)
-                symlink(realsrc, dstname)
+                os.symlink(realsrc, dstname)
             elif os.path.isdir(srcname):
                 mergetree(
                     srcname,

lib/spack/external/pytest-fallback/py/_path/local.py

@@ -12,8 +12,6 @@
 
 from os.path import abspath, normpath, isabs, exists, isdir, isfile, islink, dirname
 
-from llnl.util.symlink import symlink
-
 if sys.version_info > (3,0):
     def map_as_list(func, iter):
         return list(map(func, iter))
@@ -81,15 +79,15 @@ def mklinkto(self, oldname):
     def mksymlinkto(self, value, absolute=1):
         """ create a symbolic link with the given value (pointing to another name). """
         if absolute:
-            py.error.checked_call(symlink, str(value), self.strpath)
+            py.error.checked_call(os.symlink, str(value), self.strpath)
         else:
             base = self.common(value)
             # with posix local paths '/' is always a common base
             relsource = self.__class__(value).relto(base)
             reldest = self.relto(base)
             n = reldest.count(self.sep)
             target = self.sep.join(('..', )*n + (relsource, ))
-            py.error.checked_call(symlink, target, self.strpath)
+            py.error.checked_call(os.symlink, target, self.strpath)
 
 def getuserid(user):
     import pwd
@@ -894,7 +892,7 @@ def try_remove_lockfile():
         except OSError:
             pass
         try:
-            symlink(src, dest)
+            os.symlink(src, dest)
         except (OSError, AttributeError, NotImplementedError):
             pass
 

lib/spack/llnl/util/filesystem.py

@@ -1094,12 +1094,23 @@ def remove_linked_tree(path):
     Parameters:
         path (str): Directory to be removed
     """
+    # On windows, cleaning a Git stage can be an issue
+    # as git leaves readonly files that Python handles
+    # poorly on Windows. Remove readonly status and try again
+    def onerror(func, path, exe_info):
+        os.chmod(path, stat.S_IWUSR)
+        try:
+            func(path)
+        except Exception as e:
+            tty.warn(e)
+            pass
+
     if os.path.exists(path):
         if os.path.islink(path):
-            shutil.rmtree(os.path.realpath(path), True)
+            shutil.rmtree(os.path.realpath(path), onerror=onerror)
             os.unlink(path)
         else:
-            shutil.rmtree(path, True)
+            shutil.rmtree(path, onerror=onerror)
 
 
 @contextmanager
@@ -1237,9 +1248,6 @@ def find(root, files, recursive=True):
         return _find_non_recursive(root, files)
 
 
-# here and in _find_non_recursive below we only take the first
-# index to check for system path safety as glob handles this
-# w.r.t. search_files
 @system_path_filter
 def _find_recursive(root, search_files):
 

lib/spack/llnl/util/lock.py

@@ -191,6 +191,7 @@ def is_valid(op):
         return op == LockType.READ \
             or op == LockType.WRITE
 
+
 class Lock(object):
     """This is an implementation of a filesystem lock using Python's lockf.
 
@@ -617,6 +618,12 @@ def release_write(self, release_fn=None):
             else:
                 return False
 
+    def cleanup(self):
+        if self._reads == 0 and self._writes == 0:
+            os.unlink(self.path)
+        else:
+            raise LockError("Attempting to cleanup active lock.")
+
     def _get_counts_desc(self):
         return '(reads {0}, writes {1})'.format(self._reads, self._writes) \
             if tty.is_verbose() else ''

lib/spack/llnl/util/tty/__init__.py

@@ -395,7 +395,7 @@ def ioctl_gwinsz(fd):
         return int(rc[0]), int(rc[1])
     else:
         if sys.version_info[0] < 3:
-            raise RuntimeError("""Terminal size not obtainable on Windows with a
-                                  Python version older than 3""")
+            raise RuntimeError("Terminal size not obtainable on Windows with a\
+Python version older than 3")
         rc = (os.environ.get('LINES', 25), os.environ.get('COLUMNS', 80))
         return int(rc[0]), int(rc[1])

lib/spack/llnl/util/tty/log.py

@@ -418,6 +418,7 @@ def log_output(*args, **kwargs):
         with log_output('logfile.txt', echo=True):
             # do things ... output will be logged and printed out
 
+    The following is available on Unix only. No-op on Windows.
     And, if you just want to echo *some* stuff from the parent, use
     ``force_echo``::
 
@@ -427,20 +428,11 @@ def log_output(*args, **kwargs):
             with logger.force_echo():
                 # things here will be echoed *and* logged
 
-    Under the hood, we spawn a daemon and set up a pipe between this
-    process and the daemon.  The daemon writes our output to both the
-    file and to stdout (if echoing).  The parent process can communicate
-    with the daemon to tell it when and when not to echo; this is what
-    force_echo does.  You can also enable/disable echoing by typing 'v'.
-
-    We try to use OS-level file descriptors to do the redirection, but if
-    stdout or stderr has been set to some Python-level file object, we
-    use Python-level redirection instead.  This allows the redirection to
-    work within test frameworks like nose and pytest.
+    See individual log classes for more information.
 
 
     This method is actually a factory serving a per platform
-    (nix vs windows) log_output class
+    (unix vs windows) log_output class
     """
     if sys.platform == 'win32':
         return winlog(*args, **kwargs)
@@ -449,29 +441,7 @@ def log_output(*args, **kwargs):
 
 
 class nixlog(object):
-    """Context manager that logs its output to a file.
-
-    In the simplest case, the usage looks like this::
-
-        with log_output('logfile.txt'):
-            # do things ... output will be logged
-
-    Any output from the with block will be redirected to ``logfile.txt``.
-    If you also want the output to be echoed to ``stdout``, use the
-    ``echo`` parameter::
-
-        with log_output('logfile.txt', echo=True):
-            # do things ... output will be logged and printed out
-
-    And, if you just want to echo *some* stuff from the parent, use
-    ``force_echo``::
-
-        with log_output('logfile.txt', echo=False) as logger:
-            # do things ... output will be logged
-
-            with logger.force_echo():
-                # things here will be echoed *and* logged
-
+    """
     Under the hood, we spawn a daemon and set up a pipe between this
     process and the daemon.  The daemon writes our output to both the
     file and to stdout (if echoing).  The parent process can communicate
@@ -748,7 +718,6 @@ def __init__(self, sys_attr):
             self.libc = libc
             self.c_stream = c_stdout
         else:
-            # The original fd stdout points to. Usually 1 on POSIX systems for stdout.
             self.libc = ctypes.CDLL(None)
             self.c_stream = ctypes.c_void_p.in_dll(self.libc, self.sys_attr)
         self.sys_stream = getattr(sys, self.sys_attr)
@@ -793,6 +762,12 @@ def close(self):
 
 
 class winlog(object):
+    """
+    Similar to nixlog, with underlying
+    functionality ported to support Windows.
+
+    Does not support the use of 'v' toggling as nixlog does.
+    """
     def __init__(self, file_like=None, echo=False, debug=0, buffer=False,
                  env=None, filter_fn=None):
         self.env = env

lib/spack/spack/build_systems/cmake.py

@@ -19,6 +19,7 @@
 import spack.build_environment
 from spack.directives import conflicts, depends_on, variant
 from spack.package import InstallError, PackageBase, run_after
+from spack.util.path import convert_to_posix_path
 
 # Regex to extract the primary generator from the CMake generator
 # string.
@@ -173,7 +174,7 @@ def _std_args(pkg):
         define = CMakePackage.define
         args = [
             '-G', generator,
-            define('CMAKE_INSTALL_PREFIX', pkg.prefix.replace('\\', '/')),
+            define('CMAKE_INSTALL_PREFIX', convert_to_posix_path(pkg.prefix)),
             define('CMAKE_BUILD_TYPE', build_type),
         ]
 

lib/spack/spack/cmd/create.py

@@ -758,7 +758,15 @@ def get_versions(args, name):
     # Default guesser
     guesser = BuildSystemGuesser()
 
-    if args.url is not None and args.template != 'bundle':
+    valid_url = True
+    try:
+        spack.util.url.require_url_format(args.url)
+        if args.url.startswith('file://'):
+            valid_url = False  # No point in spidering these
+    except AssertionError:
+        valid_url = False
+
+    if args.url is not None and args.template != 'bundle' and valid_url:
         # Find available versions
         try:
             url_dict = spack.util.web.find_versions_of_archive(args.url)