Allow overriding web bundle at _build time_

[bryango]

Note: I would like to postpone the work on the missing documentations, until the monumental bundle rework has settled down, and then I will be rebasing this as soon as possible. Thank you all very much for maintaining tectonic!

This PR enables us to override the default web bundle at build time, through the environment variables (or in build.rs):

• TECTONIC_WEB_BUNDLE_PREFIX
• TECTONIC_WEB_BUNDLE_LOCKED

The PREFIX variable makes it easy for people to track a mirrored bundle, while the LOCKED variable ensures reproducible builds across all CLIs. This is related to #1002 and its resolution in #1132.

Currently, the default web bundle is hardcoded, as observed in #1002. A related issue is the version mismatch between biber and the tectonic bundle #893, which we would like to addressed in #1132 and nixpkgs NixOS/nixpkgs#273740.

P.S. I am very new to rust, so if the code / interface is no good, please help me improve! And thank you for this wonderful package!

[codecov]

Codecov Report

Attention: Patch coverage is 71.42857% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 46.41%. Comparing base (78fd977) to head (0cb32d3).

Files	Patch %	Lines
crates/bundles/src/lib.rs	71.42%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1131   +/-   ##
=======================================
  Coverage   46.41%   46.41%           
=======================================
  Files         176      176           
  Lines       65118    65123    +5     
=======================================
+ Hits        30222    30227    +5     
  Misses      34896    34896           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rm-dr]

This is somewhat related to #1121, which completely overhauls Tectonic's bundles (and thus conflicts with the changes here)

Notably, #1121 adds a new bundle format, and setting a variable for just the url prefix won't work. It may be a better idea (even without extra bundles) to have the variable contain the whole URL.

Also, I don't think I understand the difference between the LOCKED and PREFIX variables. Is there any reason to have two? I think one variable that overrides the default bundle url should be enough.

A few more notes about the issue in #1002:

• Should this be a cli option or a env variable?
• Why does tectonic new touch the network in the first place? I'd argue that the command should create a skeleton project completely offline (with the default url in tectonic.toml), which can be changed later by the user. This is clearer and doesn't need any extra config options.

[bryango]

Hi @rm-dr! Thank you for the comment!

This is somewhat related to #1121, which completely overhauls Tectonic's bundles (and thus conflicts with the changes here)

Wow amazing! I didn't know this before! This is a huge change though. Are you aware of #1122? I think being a member would probably make it easier to land these changes.

Notably, #1121 adds a new bundle format, and setting a variable for just the url prefix won't work. It may be a better idea (even without extra bundles) to have the variable contain the whole URL.

Also, I don't think I understand the difference between the LOCKED and PREFIX variables. Is there any reason to have two? I think one variable that overrides the default bundle url should be enough.

Currently, the bundle url depends on the format_version:

tectonic/crates/bundles/src/lib.rs

Lines 113 to 123 in c64e524

	/// webservice.
	pub fn get_fallback_bundle_url(format_version: u32) -> String {
	// Format version 32 (TeXLive 2021) was when we introduced versioning to the
	// URL.
	if format_version < 32 {
	"https://relay.fullyjustified.net/default_bundle.tar".to_owned()
	} else {
	format!("https://relay.fullyjustified.net/default_bundle_v{format_version}.tar")
	}
	}

So the PREFIX variant only modifies the prefix, in this case https://relay.fullyjustified.net, and keeps the version suffix default_bundle_v{format_version}.tar. If this is going away in the future, then only the LOCKED variant is needed. What do you think? Should I simply remove the PREFIX variant?

Should this be a cli option or a env variable?

They are different! Let me clarify:

• the env variable provides a build time override for distro packagers, such that the tectonic bundle they distributed will match the biber version they distributed. This is related to Biblatex fails due to version mismatch #893.
• the --web-bundle option is for runtime override by the end user. This relates to Provide options for V2 cli to override the default bundle_url #1002.

I think for the runtime override, we should only provide the CLI option, because env variables are more implicit and thus reduce reproducibility. For build time, however, I think env variables are the only way to pass around the customized build constants, so I have to use it.

Why does tectonic new touch the network in the first place? I'd argue that the command should create a skeleton project completely offline (with the default url in tectonic.toml), which can be changed later by the user. This is clearer and doesn't need any extra config options.

AFAIK, tectonic -X new and init:

• takes a generic web bundle link https://relay.fullyjustified.net/default_bundle_v33.tar,
• redirects it to a version-locked one, such like https://data1.fullyjustified.net/tlextras-2022.0r0.tar,
• and finally pins it down in the Tectonic.toml file.

I think this behavior is nice as it maximizes reproducibility. There are some suggestions in issues to put the locked url ...tlextras-2022.0r0.tar in a dedicated Tectonic.lock file. When and if that happens, tectonic -X new and init can be completely offline, as it only needs to write the unlocked link ...default_bundle_v33.tar.

$ tectonic -X init
note: "version 2" Tectonic command-line interface activated
note: creating new document in this directory ($HOME/apps/tectest)
note: connecting to https://relay.fullyjustified.net/default_bundle_v33.tar
note: resolved to https://data1.fullyjustified.net/tlextras-2022.0r0.tar

$ cat Tectonic.toml
[doc]
name = "tectest"
bundle = "https://data1.fullyjustified.net/tlextras-2022.0r0.tar"
...

Note how the url gets redirected and locked down.

[doronbehar]

I really like the idea of having an environment variable for this, but I also don't understand why we need 2 environment variables, and not only 1.

Besides that, I also support having this environment variable read in runtime and not build time, as this would also help users to test different bundles in a different way, I think there is no advantage in terms of reproducibility between reading this environment variable in runtime vs build time; since Tectonic from different distributions may have different environments anywhere.

I also think it'd be fair for Tectonic to not guarantee reproducibility in case it reads a dedicated environment variable, and there's a conflict with the URL in the Tectonic.toml file. In such a case I think it'd be best to simply print a big warning, saying that we used the URL from the environment variable and not the the URL from Tectonic.toml file.

[bryango]

Hi @doronbehar! Thank you very much for the comment!

1. I am absolutely okay with a single env variable such as TECTONIC_WEB_BUNDLE_URL. As explained above, the PREFIX variant is only introduced for backward compatibility and ease of mirroring. If we decide that it is confusing, I can easily remove it!

2. When it comes to runtime, I would still prefer explicit flags such as --web-bundle, which can be overridden by a second --web-bundle. This is not yet the current behavior, and I can look into fixing it it has been fixed with the latest commit in this PR.

I think there is no advantage in terms of reproducibility between reading this environment variable in runtime vs build time; since Tectonic from different distributions may have different environments anywhere.

Yes, this is true in reality, but ideally, I would like Tectonic to not read any environment variables at all (at runtime), by design. It's like the new flake-y nix no longer reads the NIXPKGS_* env variables; sure, it makes things a bit more inconvenient, but I like the purity in the UI.

I know @pkgw is probably busy, but since this is now more than a bugfix and related to the UI design, I would very much like to hear your opinion, and I can fix the code accordingly!

[doronbehar]

2. When it comes to runtime, I would still prefer explicit flags such as --web-bundle, which can be overridden by a second --web-bundle. This is not yet the current behavior, and I can look into fixing it.

I did noticed that the 2nd commit of this PR as of currently is referring to this? But not completely? Maybe it'd be easier for upstream to digest this change only in a PR of its own.

[rm-dr]

They are different! Let me clarify:

Ah, I see. We really do need a compile-time option for this.
The two-link config is a bit messy, but I see why it's necessary...

and I don't think I know a better way to do this, if we want easy bundle updates through a "default" redirect.

[bryango]

Following the suggestion of @doronbehar I have moved the CLI changes to a separate PR #1132. These two PRs are both related to the web bundle interface, but they are technically independent, so it would be easier to review this way.

[pkgw]

I've merged #1132, so that this PR becomes a lot smaller.

It does seem reasonable to me to have the default bundle URL be build-time configurable, for specialized applications.

My only comment upon first glance is one of the evergreen ones: these changes should come with corresponding documentation somewhere!

[bryango]

My only comment upon first glance is one of the evergreen ones: these changes should come with corresponding documentation somewhere!

Yes indeed! But since #1132 (run time web bundle override) has been merged, this one is no longer a blocking issue for me 😆 But I will try to document it sometime!