PGO optimization for non-Rust dependencies in Rust projects

[zamazan4ik]

During the PGO optimization, I found an interesting example, when a Rust program has a C/C++ dependency in a critical performance path. As an example - TiKV. TiKV depends on https://github.com/tikv/rust-rocksdb/ that under-the-hood compiles RocksDB (https://github.com/tikv/rust-rocksdb/tree/master/librocksdb_sys).

The problem is that PGO instrumentation flags are not propagated to the non-Rust dependencies, so they are not PGO-optimized during the cargo-pgo optimization.

Honestly, not sure what we can/should do here. Because I understand that implementing PGO flags propagation to the non-Rust dependencies would be difficult like hell: different C compilers with different options, need to think somehow modify build.rs scripts, etc.

Right now I see the only option to remediate it - optimize with BOLT since it works on the resulting binary. Probably we need to document this limitation somewhere in cargo-pgo.

[Kobzol]

Hi, yeah, I agree that this would be quite difficult to support in the general case (although for specific crates it shouldn't be that hard). I'm not sure how we should document this though. "If you have native code, cargo-pgo won't PGO optimize it?". I'm not sure if that adds any relevant value, as cargo-pgo doesn't promise anything about C/C++ PGO optimizations.

[Kobzol]

In any case, if you have something specific in mind, I'd be happy to accept a PR :)

[zamazan4ik]

I agree that this would be quite difficult to support in the general case (although for specific crates it shouldn't be that hard)

I think the default recommendation could be something like "If you need to optimize your native non-Rust dependency with PGO - please create an issue in the upstream package to support PGO build mode". And in the Rust crates something_sys the maintainers could implement some build.rs PGO-related magic (if the real need exists).

I'm not sure how we should document this though. "If you have native code, cargo-pgo won't PGO optimize it?". I'm not sure if that adds any relevant value, as cargo-pgo doesn't promise anything about C/C++ PGO optimizations.

Hah, I also do not know how would be better to highlight (and should we at do it or not). But maybe for someone it would surprising to understand, that only Rust dependencies are optimized in the Rust projects, even if you (re)compile whole Rust project with cargo-pgo.

[Kobzol]

I think that the mere existence of this issue is enough, if someone will look for PGO + native/C/C++ dependencies, they will find the discussion about it here :)