want-ref or want in .prepare_fetch to avoid transfers of large ref lists?

[vlad-ivanov-name]

Is there a way to ask the server to limit the list of refs when fetching, or to avoid listing refs altogether when fetching? For example given the following code:

#![allow(unused_variables)]

use anyhow::Result;

#[tokio::main]
async fn main() -> Result<()> {
    let dir = tempfile::Builder::new().prefix("gix").tempdir()?;
    let repo = gix::init_bare(dir.path())?;

    let remote = repo
        .remote_at("https://github.com/libgit2/libgit2.git")?
        .with_fetch_tags(gix::remote::fetch::Tags::None)
        .with_refspecs(
            Some("+refs/heads/main:refs/heads/main"),
            gix::remote::Direction::Fetch,
        )?;

    let conn = remote.connect(gix::remote::Direction::Fetch)?;
    let fetch_options = gix::remote::ref_map::Options {
        prefix_from_spec_as_filter_on_remote: true,
        handshake_parameters: vec![],
        extra_refspecs: vec![],
    };

    let prepare_fetch = conn.prepare_fetch(gix::progress::Discard, fetch_options)?;
    let ref_map = prepare_fetch.ref_map();

    eprintln!("{} ref mappings", ref_map.remote_refs.len());

    Ok(())
}

Cargo.toml

[package]
name = "gix-fetch-test"
version = "0.1.0"
edition = "2021"

[dependencies]
anyhow = "1.0.75"
async-trait = "0.1.73"
tempfile = "3.8.0"
tokio = { version = "1.32.0", features = ["full"] }

[dependencies.gix]
version = "0.54.1"
features = [
    "gix-protocol",
    "blocking-http-transport-reqwest-rust-tls"
]

I get 139 ref mappings (and of course this number would be bigger for bigger repos).

GitHub doesn't seem to support want-ref:

23:46:45.523323 pkt-line.c:80           packet:          git< # service=git-upload-pack
23:46:45.523375 pkt-line.c:80           packet:          git< 0000
23:46:45.523384 pkt-line.c:80           packet:          git< version 2
23:46:45.523393 pkt-line.c:80           packet:          git< agent=git/github-b60c2b516187
23:46:45.523400 pkt-line.c:80           packet:          git< ls-refs=unborn
23:46:45.523413 pkt-line.c:80           packet:          git< fetch=shallow wait-for-done filter
23:46:45.523420 pkt-line.c:80           packet:          git< server-option
23:46:45.523426 pkt-line.c:80           packet:          git< object-format=sha1
23:46:45.523432 pkt-line.c:80           packet:          git< 0000

But if I wanted to fetch only one sha1 via want without even listing refs -- am I starting in the wrong place with prepare_fetch? The end goal is to fetch a history tree starting with a particular sha1 into a bare repo.

Thank you!

[vlad-ivanov-name]

Edited to set prefix_from_spec_as_filter_on_remote = true -- this adds ref-prefix argument but still seemingly only limits to refs/heads, not to refs/heads/main

[Byron]

Thanks a lot for bringing this up! I never really thought about the possible applications of want-ref (and thus have no concept on how this would be represented in an API, yet), or have I thought about what ref-filter could do for refspecs without globs. This is probably the reason that it currently creates quite rough prefixes.

Having a prefix that isn't a directory is something I didn't know was working, did you try it? One way to do that would be to send a request through a proxy, and than re-send the request with an edited payload.

Lastly, ref-in-want seems like a very desirable feature to surface in the API (or maybe even make use of it automatically if refspecs permit) as one could indeed skip the ls-refs step entirely in protocol V2.

But as we know, this isn't supported by all forges.

But if I wanted to fetch only one sha1 via want without even listing refs -- am I starting in the wrong place with prepare_fetch?

RefSpecs can also be object names, so if the object is known in advance it's no problem to pass it. It's just that right now gix will always list refs, even though with some refspecs that can probably be avoided entirely as in V2 it's optional.

As it stands, there are two possible improvements:

• don't list refs unless needed by refspecs
• figure out if refs-in-want could be used automatically depending on refspecs and server capabilities.

I welcome you to share your opinion and ideas, maybe it's possible to sketch something out. Thank you.

[vlad-ivanov-name]

Thanks a lot for the response!

Having a prefix that isn't a directory is something I didn't know was working, did you try it? One way to do that would be to send a request through a proxy, and than re-send the request with an edited payload.

Do you mean ref-prefix during ls-refs here? From what I understand it really only works as a prefix, with no exact match:

https://github.com/git/git/blob/493f4622739e9b64f24b465b21aa85870dd9dc09/git-compat-util.h#L702-L712

So even if the client requests prefix refs/heads/main, other branches like refs/heads/main-foo and refs/heads/main-bar would also be matched. Although to be fair that will still improve performance in a lot of cases.

possible applications

When it comes to fast fetching by sha1 I think there's at least one application where this is desirable -- it's apps that accept GitHub webhooks (and probably webhooks from other platforms like gitlab, gerrit etc). Most of the time in case of push event, the payload already contains the sha1 of the commit that has been pushed, and even though the branch ref is probably also there, sha1 is actually better for using in any downstream data processing than a ref because it stays consistent while a ref may change.

So this means for example that if someone runs CI builds from GitHub on a service like Jenkins, the clone is not as fast as it could be because it involves ls-refs when fetching.

Another use case is Josh git proxy -- both want-ref and want with sha1 (and perhaps ability to pass ref-prefix values directly) would be desirable for fetching because large monorepos often have a lot of refs.

[Byron]

So even if the client requests prefix refs/heads/main, other branches like refs/heads/main-foo and refs/heads/main-bar would also be matched. Although to be fair that will still improve performance in a lot of cases.

That's a valid point, but I also think that perfect shouldn't be in the way of 'good-enough' :). Git will also optimize the traversal to only list refs that are in a particular common prefix, which will help accelerate the server response as well.
So I think it would make sense to adjust the ref-prefix that gix sends to already be the common prefix, which would be the name of the ref if there is only a single one to fetch.

And if there is no ref to fetch, because a SHA1 is mentioned, then there is no use for ls-refs at all - but it's an optimization it would still have to learn and quite a bit of code relies on having remote-ref listings.

By the looks of it, all these improvements can be done under the hood.

For want-ref one would need an API change and I have no idea how that could look, but also feel like it has lower priority than what was just mentioned as support for it seems to be somewhat lacking (until GitHub changes its mind about it).

For the ref-prefix optimizations, contributions would definitely be welcome, and I think it won't be too hard to implement either.