[FR] When sorting sources, move Message Compiler (`.mc`) files first

[Avasam]

What's the problem this feature will solve?

This is re-opening python/cpython#86175 for setuptools.

If I'm not mistaken, python/cpython#12341 is responsible for sorting sources in distutils to enable reproducible builds.

But since then, pywin32 has had to manually re-sort the sources by doing this:

from distutils import ccompiler
from distutils._msvccompiler import MSVCCompiler

def my_new_compiler(**kw):
    if "compiler" in kw and kw["compiler"] in (None, "msvc"):
        return my_compiler()
    return orig_new_compiler(**kw)

# No way to cleanly wedge our compiler sub-class in.
orig_new_compiler = ccompiler.new_compiler
ccompiler.new_compiler = my_new_compiler

# This import has to be delayed to AFTER the compiler hack
from setuptools.command.build_ext import build_ext  # noqa: E402

class my_compiler(MSVCCompiler):
    # Work around bpo-36302/bpo-42009 - it sorts sources but this breaks
    # support for building .mc files etc :(
    def compile(self, sources, **kwargs) -> list[str]:
        # re-sort the list of source files but ensure all .mc files come first.
        def key_reverse_mc(a):
            b, e = os.path.splitext(a)
            e = "" if e == ".mc" else e
            return (e, b)

        sources = sorted(sources, key=key_reverse_mc)
        return MSVCCompiler.compile(self, sources, **kwargs)

    # [...]

If pywin32 didn't have to do this "compiler patching", I'm pretty sure I could remove this hack and 2 deprecated distutils imports

Describe the solution you'd like

I'd like setuptools (distutils) to sort sources in such a way that Message Compiler (.mc) files come first.

If there's an already existing better way to handle .mc files. Please let me know!

Alternative Solutions

(see the current workaround posted above)

Additional context

• https://github.com/mhammond/pywin32/blob/main/setup.py
• [BUG] v76.1.0 broke pywin32 builds #4968
• The docs make a brief mention of .mc files: https://setuptools.pypa.io/en/latest/deprecated/distutils/setupscript.html#extension-source-files
• https://learn.microsoft.com/en-us/windows/win32/wes/message-compiler--mc-exe-

Code of Conduct

• I agree to follow the PSF Code of Conduct

[Avasam]

I know this would have to be done in https://github.com/pypa/distutils

Do you think build_ext.build_extension should update its sorting?

setuptools/setuptools/_distutils/command/build_ext.py

Lines 529 to 530 in f37845b

	# sort to make the resulting .so file build reproducible
	sources = sorted(sources)

Or should that happen in compilers.C.msvc.Compiler.compile if compiling .mc files is a concern specific to MSVC?

setuptools/setuptools/_distutils/compilers/C/msvc.py

Lines 385 to 387 in f37845b

	compile_info = self._setup_compile(
	output_dir, macros, include_dirs, sources, depends, extra_postargs
	)

Or even compilers.C.base.Compiler._setup_compile to return object in a compilable order.

setuptools/setuptools/_distutils/compilers/C/base.py

Lines 387 to 388 in f37845b

	# Get the list of expected output (object) files
	objects = self.object_filenames(sources, strip_dir=False, output_dir=outdir)

(It doesn't feel like compilers.C.base.Compiler.object_filenames would be the right place)

[abravalheri]

Thanks @Avasam, I am not very familiar with Message Compiler so I have a question:

(a) Is it always the case that .mc files need to come first in Windows, or
(b) is it a concern particular to a project and the way that project is designed/organised?

If (a), then I wouldn't see a problem in possibly modifying build_ext.build_extension or compilers/C/msvc.py.
If (b), then the best approach would be allowing projects to explicitly opt out of automatic sorting¹.

That said, it is up to the pypa/distutils maintainers to decide and those are just personal opinions.

Footnotes

1. Vaguely related we have an example of setuptools allowing opt-out of version normalisation: https://github.com/pypa/setuptools/blob/v80.3.1/setuptools/dist.py#L367 ↩

[Avasam]

I think either solution would work for pywin32. But my preference of course lies in the build backend being "smarter" and working out of the box in more scenarios where its concerned.

My current understanding and experience (which is limited to "pywin32 uses it") is that Message Compiler (.mc) files produces Header (.h) and Resource Compiler Script (.rc) files. So if those aren't compiled first, then the headers are missing, and compiling other files referencing those headers fails. (that was the symptoms of #4968).

You'll find that for every .mc file in pywin32, there's no .h file checked-in, because they're generated on build.

Windows' official doc on the Message Compiler: https://learn.microsoft.com/en-us/windows/win32/wes/message-compiler--mc-exe-

[raedrizqie]

Thanks @Avasam for this PR. Could you please made the sorting in build_ext.build_extension, so that MinGW could also benefit from it?

[Avasam]

I tried to only apply the change to compilers that would be relevant to building MC files. I'm not too familiar with MinGW, but I don't mind adding the sorting to both. Or directly in the build_ext command if the distutils maintainers think it would belong there.
Just waiting on some answer on the PR itself by maintainers.

@abravalheri thoughts on @raedrizqie 's request ?