Straight seems to miss updates or installation of dependencies #1083
Comments
|
Are we sure about compat failing to install in any of these cases? Installing Consult(straight-bug-report
:user-dir "straight.consult"
:post-bootstrap
(straight-use-package 'consult))
OutputBootstrapping straight.el...
Bootstrapping straight.el...done
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...done
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...done
Looking for emacsmirror-mirror recipe → Cloning el-get...
Looking for emacsmirror-mirror recipe → Cloning el-get...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done
Test run with version: prerelease (HEAD -> develop, origin/master, origin/develop, origin/HEAD) 039e5c9 2023-03-12
Cloning consult...
Cloning consult...done
Building consult...
Building consult → Cloning compat...
Building consult → Cloning compat...done
Building consult → Building compat...
Building consult → Building compat...done
Building consult...
Building consult...done
Packages:
"straight" n/a develop 039e5c9 2023-03-12
"org-elpa" n/a n/a
"melpa" n/a master 86ca8d06 2023-05-02
"gnu-elpa-mirror" n/a master eda4c96 2023-05-01
"nongnu-elpa" n/a main 0120f3d 2023-03-14
"el-get" melpa master 22c83206 2023-03-19
"emacsmirror-mirror" n/a master 606c3dc 2023-05-02
"consult" melpa main 625edc0 2023-05-02
"compat" gnu-elpa-mirror master cd28402 2023-04-24
Installing Embark(straight-bug-report
:user-dir "straight.embark"
:post-bootstrap
(straight-use-package 'embark))
OutputBootstrapping straight.el...
Bootstrapping straight.el...done
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...done
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...done
Looking for emacsmirror-mirror recipe → Cloning el-get...
Looking for emacsmirror-mirror recipe → Cloning el-get...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done
Test run with version: prerelease (HEAD -> develop, origin/master, origin/develop, origin/HEAD) 039e5c9 2023-03-12
Cloning embark...
Cloning embark...done
Building embark...
Building embark → Cloning compat...
Building embark → Cloning compat...done
Building embark → Building compat...
Building embark → Building compat...done
Building embark...
Building embark...done
Packages:
"straight" n/a develop 039e5c9 2023-03-12
"org-elpa" n/a n/a
"melpa" n/a master 86ca8d06 2023-05-02
"gnu-elpa-mirror" n/a master eda4c96 2023-05-01
"nongnu-elpa" n/a main 0120f3d 2023-03-14
"el-get" melpa master 22c83206 2023-03-19
"emacsmirror-mirror" n/a master 606c3dc 2023-05-02
"embark" melpa master 1586905 2023-05-01
"compat" gnu-elpa-mirror master cd28402 2023-04-24
Installing Marginalia(straight-bug-report
:user-dir "straight.marginalia"
:post-bootstrap
(straight-use-package 'marginalia))
OutputBootstrapping straight.el...
Bootstrapping straight.el...done
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...done
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...done
Looking for emacsmirror-mirror recipe → Cloning el-get...
Looking for emacsmirror-mirror recipe → Cloning el-get...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done
Test run with version: prerelease (HEAD -> develop, origin/master, origin/develop, origin/HEAD) 039e5c9 2023-03-12
Cloning marginalia...
Cloning marginalia...done
Building marginalia...
Building marginalia → Cloning compat...
Building marginalia → Cloning compat...done
Building marginalia → Building compat...
Building marginalia → Building compat...done
Building marginalia...
Building marginalia...done
Packages:
"straight" n/a develop 039e5c9 2023-03-12
"org-elpa" n/a n/a
"melpa" n/a master 86ca8d06 2023-05-02
"gnu-elpa-mirror" n/a master eda4c96 2023-05-01
"nongnu-elpa" n/a main 0120f3d 2023-03-14
"el-get" melpa master 22c83206 2023-03-19
"emacsmirror-mirror" n/a master 606c3dc 2023-05-02
"marginalia" melpa main 3ddd2b7 2023-04-23
"compat" gnu-elpa-mirror master cd28402 2023-04-24
Installing Magit(straight-bug-report
:user-dir "straight.magit"
:post-bootstrap
(straight-use-package 'magit))
OutputBootstrapping straight.el...
Bootstrapping straight.el...done
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...done
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...done
Looking for emacsmirror-mirror recipe → Cloning el-get...
Looking for emacsmirror-mirror recipe → Cloning el-get...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done
Test run with version: prerelease (HEAD -> develop, origin/master, origin/develop, origin/HEAD) 039e5c9 2023-03-12
Cloning magit...
Cloning magit...done
Building magit...
Building magit → Cloning compat...
Building magit → Cloning compat...done
Building magit → Building compat...
Building magit → Building compat...done
Building magit → Cloning dash.el...
Building magit → Cloning dash.el...done
Building magit → Building dash...
Building magit → Building dash...done
Building magit → Building git-commit...
Building magit → Building git-commit → Cloning transient...
Building magit → Building git-commit → Cloning transient...done
Building magit → Building git-commit → Building transient...
Building magit → Building git-commit → Building transient...done
Building magit → Building git-commit → Cloning with-editor...
Building magit → Building git-commit → Cloning with-editor...done
Building magit → Building git-commit → Building with-editor...
Building magit → Building git-commit → Building with-editor...done
Building magit → Building git-commit...
Building magit → Building git-commit...done
Building magit → Building magit-section...
Building magit → Building magit-section...done
Building magit...
Building magit...done
Packages:
"straight" n/a develop 039e5c9 2023-03-12
"org-elpa" n/a n/a
"melpa" n/a master 86ca8d06 2023-05-02
"gnu-elpa-mirror" n/a master eda4c96 2023-05-01
"nongnu-elpa" n/a main 0120f3d 2023-03-14
"el-get" melpa master 22c83206 2023-03-19
"emacsmirror-mirror" n/a master 606c3dc 2023-05-02
"magit" melpa main 6067f92c 2023-05-02
"compat" gnu-elpa-mirror master cd28402 2023-04-24
"dash" melpa master b6eef1a 2023-04-16
"git-commit" melpa main 6067f92c 2023-05-02
"transient" melpa main af7fe42 2023-05-01
"with-editor" melpa main df74385 2023-04-12
"magit-section" melpa main 6067f92c 2023-05-02
Installing Vertico(straight-bug-report
:user-dir "straight.vertico"
:post-bootstrap
(straight-use-package 'vertico))
OutputBootstrapping straight.el...
Bootstrapping straight.el...done
Looking for gnu-elpa-mirror recipe → Cloning melpa...
Looking for gnu-elpa-mirror recipe → Cloning melpa...done
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...
Looking for nongnu-elpa recipe → Cloning gnu-elpa-mirror...done
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...done
Looking for emacsmirror-mirror recipe → Cloning el-get...
Looking for emacsmirror-mirror recipe → Cloning el-get...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done
Test run with version: prerelease (HEAD -> develop, origin/master, origin/develop, origin/HEAD) 039e5c9 2023-03-12
Cloning vertico...
Cloning vertico...done
Building vertico...
Building vertico → Cloning compat...
Building vertico → Cloning compat...done
Building vertico → Building compat...
Building vertico → Building compat...done
Building vertico...
Building vertico...done
Packages:
"straight" n/a develop 039e5c9 2023-03-12
"org-elpa" n/a n/a
"melpa" n/a master 86ca8d06 2023-05-02
"gnu-elpa-mirror" n/a master eda4c96 2023-05-01
"nongnu-elpa" n/a main 0120f3d 2023-03-14
"el-get" melpa master 22c83206 2023-03-19
"emacsmirror-mirror" n/a master 606c3dc 2023-05-02
"vertico" gnu-elpa-mirror master dd8eb3a 2023-04-26
"compat" gnu-elpa-mirror master cd28402 2023-04-24The issue, as I understand it, is that updating a package which depends on compat does not automatically update compat to meet the dependent's declared compat version. It's a matter of deciding what to do in these cases. Automatically updating a dependencies doesn't seem like the best solution to me. The update may fix one package and break others. A warning could be signaled to let users know that a dependent declares version X.Y.Z, but a lower version is installed. This would still require intervention on the part of the user (decide whether to upgrade the dependency, or downgrade the dependent), but it seems like the best way to handle this to me. Also related: #822 @raxod502: Thoughts? |
Is this specific to compat or does straight just ignore versions of dependencies in general, only ensuring they are installed but not caring what version is installed? I think the right thing is to update dependencies, like package.el does. I always thought the version numbers in dependency requirements where minimum versions, so if updating a dependency to a newer version broke one of my packages I'd consider it a bug in my package (albeit a bug that isn't my fault, for once!) and fix it. |
It seems so. The particular problem with Compat is that it has seen multiple updates recently. Since Compat is used as a library, packages depending on newly added APIs bumped the version of the Compat dependency.
Yes, it is. Straight could at least print a prominent warning if dependency version requirements are not met. |
It's not compat specifically.
Let's say, for example, a dependency makes some breaking changes.
This seems like the simplest solution, and should prevent most of the bug reports in the issue description. A more complex solution would be: Automatically bump a dependency when a dependent requests a higher version and:
or
Otherwise, warn the user about the conflict. e.g. |
I believe this is the only reasonable strategy. As far as I know, there is no way in Package-Requires to specify a maximum version of a library, only a minimum version. So, whether we like it or not (probably not), package maintainers are implicitly promising that their package works with any version of the dependencies with version greater than or equal to the listed one. Is this a problematic system Emacs has adopted? Yes! But it is what we have.
This message would be arguably wrong, according to the documentation, Package-Requires doesn't mean "I require this specific version" it means "I require this version or any later version". |
|
Yes, it would be worded more carefully, the example was just that. |
|
OK, how about this wording?
😛 |
|
I'm only half-joking. I'd personally like such a tongue-in-cheek error message. |
This is a gap in the documentation or design, if that's so. |
This is also very sensible. |
For me the problem is critical and it cropped up quite often already. The crucial point here is that Compat is a library which is hard to debug. Users have difficulties locating the problem since Compat functions are not prefixed with a package name. For this reason, Compat is developed in a defensive style and tested extensively, to make absolutely sure that there are no regressions due to it. Also the versions can be relied on. Straight users generally seem well-versed with Emacs and Elisp and yet they have a hard time with Straight+Compat. For other libraries, users may usually figure out themselves that they have to update another package, e.g., if the new API
It seems you are discussing version bounds. Note that Compat uses a special versioning scheme of the form |
|
Why does compat need to invent its own versioning scheme instead of using semver? |
Obviously because Compat is tied to the Emacs versions, e.g., Also note that semver is not a commonly agreed standard for Emacs packages. It would be nice if there were some formal rules, but in reality, there are none. Therefore it would be better to not rely on such assumptions for such a Straight version check. |
That's not obvious to me (or perhaps anyone else who isn't developing compat) why the Emacs version would need to be expressed in compat's versioning.
That sounds like an Emacs/package.el bug. |
|
If anything, I would've expected the emacs major/minor version to be encoded after compats major version. e.g. compat 1.28.1 |
Maybe I should elaborate a bit more on the versioning problem for Compat. Users of Compat specify an "API level", which is the Emacs version. This makes sense since Emacs APIs are generally introduced with new Emacs versions. This means the Compat version carries meaning and is updated in lock-step with Emacs releases itself. As soon as an Emacs version is frozen, e.g., 29.1, we could release a version of Compat 29.1.1, where only the minor version could be bumped. However it can then happen that we backport additional APIs. Compat is not frozen like the Emacs version itself. Some APIs are hard to back port and can be made available only a while later when we figured out a reasonable approach. I also suggest that you take a look at the Compat manual.
A warning is fine with me, if it is sufficiently visible and clear. The warning should suggest that a package dependency must be updated too. |
|
Also note how Compat ;; Package-Requires: ((emacs "27.1") (compat "29.1.4.1"))We require Emacs 27.1, API level 29.1, and more precisely Compat version 29.1.4.1. If you wouldn't care about the Compat details you could also specify: ;; Package-Requires: ((emacs "27.1") (compat "29.1"))But then the problem is that you may miss additional Compat updates, which were added after the initial release of 29.1.0.0, the very issue we are discussing here. |
|
Does package.el have special casing baked in to account for this versioning scheme? How does it handle the case where package "A" requires compat "28.x.y" and package "B" requires "29.x.y"? Indiscriminately update? This would work for compat, but cause breakage for any package adhering to semantic versioning. |
|
As I mentioned above, there is no way at all to say you require version exactly 28.x.y, only a way to say you require at least 28.x.y, therefore package.el just updates the dependency. |
|
Yes, it would indiscriminately update and nothing will break since Compat will always stay backward compatible. Note that package.el doesn't know upper bounds for package versions. Newer versions are always considered as "better". |
I see. Thanks for the explanations. So it sounds to me like a warning is the best option for straight.el. |
I find it hard to tell. A version bound system like in Haskell or Ruby seems like a good idea. Otoh upper version bounds are also a huge pita since they can cause updates being stuck for a long time. A better approach may be to avoid upper version bounds as is currently the case and develop libraries in a strictly backward compatible way, or at least with long deprecation periods. For Compat this works well since Emacs APIs are usually backward compatible. For other libraries the story may be different. Imo it also makes a big difference if we talk about libraries vs user packages. User packages can be developed much less rigoursly since they usually won't be depended on. |
Further development in elisp name spacing (or, allegedly, read-symbol-shorthands, though I haven't looked into how) would help here because it would be possible to load multiple versions of a library without having them interfere. Of course, that would need a lot of work in many areas of Emacs.
Authors still need a mechanism to communicate "the deprecation period is over".
The more dependents a package has, the higher the chance its breaking changes will affect some of those dependents. |
Indeed. But I am not sure if this is even desirable. I'd say time is better spent if we update packages which depend on old libraries.
Yes, but introducing upper version bounds is a steep price to pay and it is not without downsides.
Agree. In my packages I make a distinction between private and public APIs, which makes it clear that depending packages move into dangerous territory. I've often seen that packages make too much functionality public right away such that the stable API boundary is unclear. |
|
The decision to ignore version bounds in I think signaling a warning when a version bound is not satisfied, but not automatically incorporating the information into the dependency resolution algorithm, is a great compromise to resolve the biggest issues here. |
|
According to: purcell/flycheck-package#11 we can't even rely on the "Version" header metadata to reliably get a package's declared version. To summarize, package-lint declares a "version" of 0 as a place holder in the main source file. It's assumed that an ELPA will dynamically generate a package-lint-pkg.el file with the real version info. Good luck to whoever takes on this feature. |
|
@progfolio That doesn't seem very relevant for this issue. We are talking about packages, in particular libraries, which care about versioning and which specify proper versions in their version header. Packages depending on those packages can specify these versions and can reasonably except that the version constraints are satisfied. Packages which don't specify dependencies or versions could be excluded from the check. |
|
As a data point, libraries like dash, s, f, ht and compat specify version headers. I'd say it is a win if these were checked. Btw I wonder how well package-lint works if straight doesn't update version headers during installation? In my installation I get warnings when depending on unavailable package versions. |
It's not that they don't declare a version, its that the proper declaration is not available outside of an ELPA-dependent tarball build. Those versions don't exist from Straight's perspective, and we have no way of easily getting them. So, even in order to do the simplest warning based solution, we'll have to decide on rules for which package Versions (as declared in the package's source repository) to ignore completely Personally I'm in favor of ignoring a Version: "0". |
|
I've also noticed that there are issues where users have pinned Compat to a specific commit, which is incompatible with other unpinned packages. It would be great if Straight could also check these. It should generally warn about every version constraint, which is not met, for pinned and unpinned packages. This will help users to unpin packages or to update the pinned commit where needed. |
|
Pinned? Did we add a feature to support pinning a specific package to a specific commit? I wasn't aware this was possible with |
No. Perhaps they meant a lockfile is in use? |
|
My case is the same as magit/magit#4913 (comment) which @minad linked above. However this was not an issue of Compat not being updated along with other libraries - it was updated to the latest along with Magit. Instead Deleting Besides deleting And why didn't |
While the symptom may have been the same, the cause may have been different.
My hunch is that deleting everything wasn't necessary, but had the side effect of updating straight.el itself, which bumped you to a version where a couple byte-compilation bugs have been fixed. What version of straight.el were you running before?
|
|
I've made some progress on this with Elpaca.
This handles most cases, but still required the addition of a I also made the decision to fail package orders which do not have all their dependency versions satisfied. If anyone is interested in implementing something similar for straight.el, here is the relevant Elpaca code: |
|
@progfolio Thank you, this sounds really good. I hope that Straight also gets adapted accordingly. |
There have been numerous issues reported to packages using the Compat library as dependency. Straight fails to update or install Compat properly.
All these issues required manual intervention, basically manual installation of Compat and reinstallation of the affected package. Is this related to #964? Can the dependency handling of Straight be made more robust please?
The text was updated successfully, but these errors were encountered: