Indicate launch module updates in app layer versioning

[ncoghlan]

The automated application layer versioning in 0.2.0 is misleading, as the given version number only reflects the lock version, it doesn't reflect the launch module version. This makes it hard for the deploying application to work out if the latest version of an application layer is already installed, or if it needs to download and install a new one.

To fix this, the meaning of versioned=True on application layers should change to include a app_launch_module_version field (based on the app_launch_module_hash changing) in addition to the existing lock_version field.

For app layers, the deployed layer name format would change from {layer_name}@{lock_version} to {layer_name}@{app_launch_module_version}.{lock_version}

[ncoghlan]

I initially thought this could use the same approach as the existing archive_build field, and just put the launch module version number in the layer output metadata. However, that approach won't work for this use case, as the "previous launch module hash" and "previous launch module version" information needs to be consistent across both archive builds and local exports, and it needs to be reliably available.

Instead, we need a dedicated (automatically generated) versioning file checked into git for each application layer, just as we already have for the environment lock versioning.

Proposed location for that file:

• emit a app-{layer_name}.json file in the requirements/app-{layer_name} folder for each application layer (this folder is already required to be under version control, as it contains the per-platform lock files for the layer)
• Potential variation: launch-module-app-{layer_name}.json (to be more consistent with the way the per-platform lock files are named)

Proposed fields for that file:

• launch_name: the name to pass to python -m to launch the module (version is bumped if this changes)
• source_hash: the file-or-directory hash for the launch module source (version is bumped if this changes)
• app_version: counterpart to the lock_version field in the layer lock version files

Output metadata (option 1 - flat):

• app_launch_module: rename to app_launch_name
• app_launch_module_hash: rename to app_source_hash
• app_version: new field

Output metadata (option 2 - nested):

• app: new field with the same structure as the committed app layer versioning file

My current preference is to keep a flat output metadata structure (for consistency with the way the corresponding lock versioning info is already handled), but could be potentially talked into the nested option if that might be easier to consume.

Either way, the old output metadata field names would be temporarily retained for 0.3.0, but removed in 0.4.0.

As part of this, we should explicitly document the expectation that venvstacks projects should consistently use LF line-endings, even on Windows, to avoid cross-platform hash inconsistencies (referencing https://github.com/lmstudio-ai/venvstacks/blob/main/.gitattributes as an example)

[ncoghlan]

After chatting to @neilmehta24 about it, I'm going to try an approach that treats the launch module hash as an additional input to the environment lock file for application layers.

However, I still have to decide how the visible layer versioning numbering should work:

• {layer_name}@{lock_version}.{app_version} (suggested by Neil)
• {layer_name}@{app_version}.{lock_version} (my first idea above)
• {layer_name}@{layer_version} (incremented when lock or app version is updated)
• {layer_name}@{lock_version+app_version} (incremented twice if lock version and app version both change)

[ncoghlan]

I'm currently leaning towards Neil's suggested app layer versioning format, where the part immediately after the @ symbol is the lock version for all layers, and app layers add the launch module version after that.

The general idea being that complex launch modules should ideally be separated out as regular versioned Python packages, with the launch modules being a relatively thin layer that adapts the application interface to the CLI interface expected by the embedding application.

[ncoghlan]

Further discussion between Neil and myself narrowed the four options above down to two candidates:

• {layer_name}@{lock_version}.{app_version}
• {layer_name}@{lock_version+app_version}

These are the two that feel the most "similar in spirit" to the way the runtime and framework layer versioning works (with just a lock version defined, since venvstacks only allows injecting unpackaged code into the application layers).

The (arguable) problem with the first option (exposing the two numbers separately), is that it promotes an implementation detail (whether an application is written as a complex launch module only distributed with venvstacks, or as a published Python module that the application layer definition depends on) into a visible part of the layer name. It also uses the dotted version notation for something that is quite different from the way semantic versioning or calendar versioning uses it.

Distinguished versioning with a complex launch module and the occasional dependency update:

• [email protected] (initial layer publication)
• [email protected] (launch module update)
• [email protected] (launch module update)
• [email protected] (dependency update)
• [email protected] (dependency and launch module update)

Distinguished versioning with a simple launch module and all functional changes happening via dependency updates:

• [email protected] (initial layer publication)
• [email protected] (app module update)
• [email protected] (app module update)
• [email protected] (other dependency update)
• [email protected] (app module and other dependency update)

If we make the launch module versioning zero-based instead of one based, then summing the two version numbers would give the following progression:

Combined versioning with a complex launch module and the occasional dependency update:

• app-layer@1 (initial layer publication: 1+0)
• app-layer@2 (launch module update: 1+1)
• app-layer@3 (launch module update: 1+2)
• app-layer@4 (dependency update: 2+2)
• app-layer@6 (dependency and launch module update: 3+3)

Combined versioning with a simple launch module and all functional changes happening via dependency updates:

• app-layer@1 (initial layer publication: 1+0)
• app-layer@2 (launch module update: 2+0)
• app-layer@3 (launch module update: 3+0)
• app-layer@4 (dependency update: 4+0)
• app-layer@5 (dependency and launch module update: 5+0)

[ncoghlan]

I'm still not loving either of the app layer versioning approaches explored in my previous comment. The fact I felt compelled to explain the sums involved when writing out the "combined version number" examples isn't a good sign, and neither is the fact that the combined approach needs zero-based launch modules versions to avoid making the initial layer version number weird.

It did inspire a fifth idea, though, which is to use + as the separator instead of ., so it's clearer that these are two different version numbers more than they are two parts of one version number:

Distinguished versioning with a complex launch module and the occasional dependency update:

• app-layer@1+1 (initial layer publication)
• app-layer@1+2 (launch module update)
• app-layer@1+3 (launch module update)
• app-layer@2+3 (dependency update)
• app-layer@3+4 (dependency and launch module update)

Distinguished versioning with a simple launch module and all functional changes happening via dependency updates:

• app-layer@1+1 (initial layer publication)
• app-layer@2+1 (app module update)
• app-layer@3+1 (app module update)
• app-layer@4+1 (other dependency update)
• app-layer@5+1 (app module and other dependency update)

[ncoghlan]

{layer_name}@{lock_version}+{app_version} looks good to @neilmehta24 as well, so that's the approach we're going to go with.

[ncoghlan]

After several months of working around this limitation via explicit layer versioning, @neilmehta24 is now less sure it's worthwhile exposing the lock version separately from the overall layer version in the deployed layer name when using automatic layer versioning.

This favours a variation on combined versioning, where the layer version is updated when the lock version or any other relevant application layer input changes (such as the launch module contents, the post-install module contents, the underlying runtime name, or the relative deployed framework layer locations that will be added to sys.path on startup).

[ncoghlan]

The resolution of the remaining parts of #149 in #154 offer a potential path forward without needing to track a separate app version directly: the launch module hash could be treated as an additional "other input" when determining the layer lock validity.

This would be particularly appropriate if that change was combined with an enhancement to accept app layer dependency declarations via inline script metadata.

[ncoghlan]

#166 implements the approach of treating the app module version as "another input" for the layer versioning.

Adding inline script metadata support is left as a potential future enhancement for now.