How to see which tools are used?

[abitrolly]

To strip down portable SCons installation (scons-local) which is now 17Mb because of shipped DocBook, is there a way to see all tools that are used in the hierarchy of SConstruct files in my repo?

[mwichmann]

First off, you can probably remove the vendored docbook-xsl if you don't use docbook. We've debated splitting that out at the package (wheel) level, although scons-local hadn't come into the discussion. If I were to do this, I'd change the last line of SCons/Tool/docbook/__init__.py - the exists function - to return False instead (just to make sure that the sequence "if tool exists, call its generate function" doesn't trigger) - but since docbook isn't in any of the default lists, it should only be looked at if you ask for it explicitly in a tools= keyword arg or Tool() call, you should be safe if you don't. Then remove the docbook-xsl* directory. I think it was originally included inline because nobody could figure out how to deliver it separately in a reliable way, although I'm just guessing about that part; included at all because if getting it from other sources, there was no way to know where it would end up. We haven't put in the time to figure it out now that Python packaging has evolved a lot.

It's pretty hard to figure out which tools are used, since the tool modules themselves just do setup, and many of them are called as part of a default list unless you say otherwise - that is, a lot of tools may get initialized that you never call. You could perhaps instrument the underlying builder class (BuilderBase) to record when the various builders get called, but this would be a little fiddly, and not all builders end up as instances of the builder class, as there are "pseudo builders" too. The fact that builders are all set up dynamically at runtime contributes to making it a bit hard to trace. There's not a direct correlation from the outside between tool module names and builder names. Lots of pieces can contribute to having a Program builder, while at the other extreme, the docbook tool adds maybe ten builders - and they're pseudo-builders at that, so won't appear in the BUILDERS variable.

[abitrolly]

What would be the best place to instrument to catch when a new tool is being registered? And to trace what it added.

[mwichmann]

Sadly (note: that word is a personal editorial comment, not SCons policy!!!), builders don't have a registration method, they're added manually, like:

    env['BUILDERS']['Program'] = program

So you can't trigger on that. The two places I would look at for instrumenting (in the form of "log to a file") would be the __call__ methods of the Tool class in SCons/Tool/__init__.py and of the BuilderBase class in SCons/Builder.py. In BuilderBase you should be able to derive the name via self.get_name (see the comments at the top of the file for that). Untested, lots of caveats, etc.

[abitrolly]

Maybe it might work using __getitem__ on env when somebody requests an item, and then checking if the requested value changed. Neither way seems easy.

[abitrolly]

Will just removing SCons/Tool/docbook dir and other unused tool dirs work? Or is it somehow hardwired in SCons internals that needs patching?

[mwichmann]

docbook should be okay as noted above (should be okay to remove it entirely) - "not finding" a tool is normal (at least for non-default ones). Don't see any point in removing anything else, docbook is 90% of the space consumed by tools.

[abitrolly]

One point could be to speed up initialization if every Python tool tries to run some binary to discover its version and other parameters.

[mwichmann]

Yes, if you know accurately what you're using you can specify just that list, and avoid the defaults. The test suite does this, since it has to kick off scons over 1,000 times, so you'll often see something like:

DefaultEnvironment(tools=[])
env = Environment(tools=['gcc','g++','gnulink'])

as a recipe to speed up startup. On a larger project, the time to compute dependencies, etc. usually causes startup time to be lost in the noise, and you're not launching SCons numerous times in a short space, but one can use something in the above style if one wants.

[bdbaddog]

Just remove docbook. That's the biggest piece.
The rest is not really worth the effort to narrow it down IMHO.

[bdbaddog]

The only tool initialization which takes significant time is MSVC. The rest are pretty trivial, and in most cases don't launch a subprocess.

So to summarize:

1. Remove docbook xsl from your scons-local, change docbook's exist to return False
2. Explicitly set which tools are initialized for your build

That should minimize your footprint (without a lot of extra work to determine which files are safe to remove/aren't used), and minimize your tool initialization time.

gcc/g++ do launch gcc to determine version.
On non-windows. This would be trivial, but you could override/modify gcc's detect_version() to just return some true value or the version string from the gcc on your system.

Likely not a real contributor to SCons start up time.
Most likley you're incremental builds spend the bulk of their time

1. Reading .sconsign
2. evaluating the DAG
3. writing the .sconsign

If there's some other goal you have that won't be addressed by the above, please explain?

[abitrolly]

If there's some other goal you have that won't be addressed by the above, please explain?

The goal is to add scons to https://github.com/godotengine/godot/ repo with minimal overhead and with no extra files in root, except ./scons binary to avoid extra installation steps to build godot.

[bdbaddog]

I'd just use the scons-local tree and remove the docbook. more than that yields little improvment in size and lots of hassle when you need to upgrade to the next version

[bdbaddog]

Note: MongoDB does something similar. They "vendor" scons-local

[mwichmann]

Ah... curious if #4127 would be of interest in aid of that goal (that would be one file added, rather than an unpacked tree). Again, the omission of the docbook blob is not an issue that was looked at. I don't think there's been a decision on whether to go this way or not, however.

[abitrolly]

Reminds me of https://pex.readthedocs.io/en/latest/ but without complete virtualenv inside. Could be useful.