CanonicalizePath should resolve symbolic links

[xtexChooser]

On AOSC OS (a usr-merged distribution), when using Ninja + CMake + Clang, here is a issue:

• cmake calls clang to scan for implicit dependencies
• clang -M generate things like this:
  /bin/../lib64/gcc/x86_64-aosc-linux-gnu/13.2.0/../../../../include/c++/13.2.0/iosfwd
• it is legal, as /bin is symbolically linked to /usr/bin on usr-merged systems, so it is actually /usr/include
• when Ninja tries to load it, it canonicalize it to /include/xxx, without realizing /bin is a symbolic link
• Ninja saved the bad path to .ninja_deps, and sees /include/xxx as missing and dirty, thus rebuilding all sources, everytime

ninja/src/graph.cc

Lines 712 to 719 in a3fda2b

	for (std::vector<StringPiece>::iterator i = depfile_ins->begin();
	i != depfile_ins->end(); ++i, ++implicit_dep) {
	uint64_t slash_bits;
	CanonicalizePath(const_cast<char*>(i->str_), &i->len_, &slash_bits);
	Node* node = state_->GetNode(*i, slash_bits);
	*implicit_dep = node;
	node->AddOutEdge(edge);
	}

[digit-google]

The problem seems to be that your compiler is producing paths that appear to be ambiguous, i.e. whose interpretation depends on the file system state (which in general is a mutable thing out of control from Ninja).

CanonicalizePath() is a simple "pure" function (output only depends on its input). This makes it easy to understand, unit-test and maintain, and its performance is critical for Ninja. Making its result depend on the file system state is a terrible idea (it would make things much slower due to all the extra syscalls, and now error cases need to be handled in some way, plus the file system may change at runtime unexpectedly).

An alternative would be to change how depfiles are processed by Ninja, but again, this introduces a ton of complexity, and will slow down builds in general. I suggest you instead use a compiler wrapper to rewrite the depfile paths instead before Ninja parses them. Another option is to use an option like -MM to omit system header files.

[xtexChooser]

Thanks! It seems to be a clang problem (while GCC does expands symlinked paths).

[xtexChooser]

I did a simple benchmark. When compiling clang, ~3.6k object files, on Ninja's startup, CanonicalizePath was called ~120000 times. It tooks ~0.7secs to finish these canonicalization (clang++, -O2, Linux, btrfs + LUKS + LVM, realpath from glibc 2.40).

I think this is does not introduce a huge performance impact, so I still think that the canonicalization process should involve symbolic link resolving.

The problem seems to be that your compiler is producing paths that appear to be ambiguous, i.e. whose interpretation depends on the file system state (which in general is a mutable thing out of control from Ninja).

I am not sure how the path is ambiguous, it should always only point to the underlying file. There seem not to be any filesystems who allows users to change the path canonicalization process' behavior.

[digit-google]

Performance of CanonicalizePath is not important when building lots of things, but it is for Ninja startup time during incremental builds (in particular to perform a Ninja no-op check, as we do in CI in my project), or running tools, as it is heavily used when loading the manifest before building anything. For example, in PR #2358, I benchmarked that a small optimization in CanonicalizePath itself reduced that time by 200ms.

I would be very wary about anything that makes startup slower. Do you have an actual PR to benchmark against real use cases?