From 085a0ca17d5a5d892299db638b31e0c8fdc65d16 Mon Sep 17 00:00:00 2001 From: "Justin M. Keyes" Date: Mon, 17 Jul 2023 01:46:05 +0200 Subject: [PATCH] pkg.json "pivot" part of #41 --- README.md | 183 +++++++++----------------------------------- docs/README.md | 14 ++-- docs/client-spec.md | 19 +++++ docs/spec.md | 136 ++++++++++++++++++++++++++++++++ 4 files changed, 202 insertions(+), 150 deletions(-) create mode 100644 docs/client-spec.md create mode 100644 docs/spec.md diff --git a/README.md b/README.md index f2cd392..6314688 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,33 @@ # pkg.json +`pkg.json` is a wild-west "package" format for defining packages without a package system. +It's a (very) limited subset of NPM's `package.json` that allows any project to declare dependencies on arbitrary URLs. + +The initial use-case is for Vim and Emacs plugins (which can be downloaded from anywhere), but the format is designed to be generic. + +## Example + +``` +{ + "name" : "lspconfig", // OPTIONAL cosmetic name, not used for resolution nor filesystem locations. + "description" : "Quickstart configurations for the Nvim-lsp client", // OPTIONAL + "engines": { + "nvim": "^0.10.0", + "vim": "^9.1.0" + }, + "repository": { // REQUIRED + "type": "git", // reserved for future use + "url": "https://github.com/neovim/nvim-lspconfig" + }, + "dependencies" : { // OPTIONAL + "https://github.com/neovim/neovim" : "0.6.1", + "https://github.com/lewis6991/gitsigns.nvim" : "0.3" + }, +} +``` + +## Overview + - `pkg.json` is just a way to declare dependencies on URLs and versions - Decentralized ("federated", omg) - Subset of `package.json` @@ -26,153 +54,18 @@ ## What about LuaRocks? -LuaRocks is a natural as the Nvim plugin manager, but defining a ~~"plugin spec"~~ "federated package spec" also makes sense because: +LuaRocks is a natural choice as the Nvim plugin manager, but defining a "federated package spec" also makes sense because: -- There is no "federated" plugin spec (corrections welcome!). LuaRocks is a "centralized" approach. -- LuaRocks + Nvim is starting to see real progress in the form of https://github.com/nvim-neorocks , but thus far has not gained momentum. A decentralized, lowest-common-denominator, "infectious" approach is high-leverage, while work continues on the centralized LuaRocks approach at its own pace. -- There's no central _asset_ registry, just a bunch of URLs. - - Could have a central _list_ of plugins, but not assets. -- We can do both, at low cost. `pkg.json` is a fairly "cheap" approach. LuaRocks +- We can do both, at low cost. `pkg.json` is a fairly "cheap" approach. +- LuaRocks is a "centralized" approach that requires active participation from many plugins. + In contrast, `pkg.json` is a decentralized, "infectious" approach that is useful at the "leaf nodes": + it only requires the consumer to provide a `pkg.json`, the upstream dependencies don't need to be "compliant" or participate in any way. +- LuaRocks + Nvim is starting to see [progress](https://github.com/nvim-neorocks), but momentum will take time. + A decentralized, lowest-common-denominator, "infectious" approach can be tried without losing much time or effort. +- There's no central _asset registry_, just a bunch of URLs. (Though "aggregators" are possible and welcome.) +- LuaRocks has 10x more scope than `pkg.json` and 100x more [unresolved edge cases](https://github.com/luarocks/luarocks/issues/905). + `pkg.json` side-steps all of that by punting the ecosystem-dependent questions to the ecosystem-dependent package manager client. ## Release TBD - ---- - -# OLD - -The neovim package specification consists of three components: -1. Guidelines which provide guidance to package authors, -2. the `packspec` file format, and -3. guidelines for `packspec`-compatible plugin manager implementers - -# Guidelines for plugin authors - -### Semantic versioning - -All Neovim packages should be [semantically versioned](https://semver.org/). While other versioning schema have uses, semver allows for plugin managers to provide smart dependency resolution. - -# The `packspec` file -The Neovim package specification supports a single, top-level package metadata file. This file can be *either* 'packspec.lua' or 'packspec.json'. The format is loosely based on the [Rockspec Format](https://github.com/luarocks/luarocks/wiki/Rockspec-format) and can contain the following fields: - -* `package` (String) the name of the package - -* `version` (String) the version of the package. Should obey semantic versioning conventions, for example `0.1.0`. Plugins should have a git commit with a `tag` matching this version. For all version identifiers, implementation should check for a `version` prefixed with `v` in the git repository, as this is a common convention. - -* `packspec` (String) the current specification version. (0.1.0) at this time. - -* `source` (String) The URL of the package source archive. Examples: "http://github.com/downloads/keplerproject/wsapi/wsapi-1.3.4.tar.gz", "git://github.com/keplerproject/wsapi.git". Different protocols are supported: - - * `file://` - for URLs in the local filesystem (note that for Unix paths, the root slash is the third slash, resulting in paths like "file:///full/path/filename" - * `git://` - for the Git source control manager - * `git+https://` - for the Git source control manager when using repositories that need https:// URLs - * `git+ssh://` - for the Git source control manager when using repositories that need SSH login, such as git@example.com/myrepo - * `http://` - for HTTP URLs - * `https://` - for HTTPS URLs - * `luarocks://` - for Luarocks packages - -* `description` (Table) the description is a table that includes the following nested fields: - * `summary` (String) a short description of the package, typically less than 100 character long. - * `detailed` (String) a long-form description of the package, this should convey the package's principal functionality to the user without being as detailed as the package readme. - * `homepage` (String) This is the homepage of the package, which in most cases will be the GitHub URL. - * `license` (String) This is [SPDX](https://spdx.org/licenses/) license identifier. Dual licensing is indicated via joining the relevant licenses via `/`. - -* `dependencies` (List[Table]) A list of tables describing the package dependencies. Each entry in the table has the following, only `source` is mandatory: - * `version` (String) The version constraints on the package. - * Accepted operators are the relational operators of Lua: == \~= < > <= >= , as well as a special operator, \~>, inspired by the "pessimistic operator" of RubyGems ("\~> 2" means ">= 2, < 3"; "~> 2.4" means ">= 2.4, < 2.5"). No operator means an implicit == (i.e., "lfs 1.0" is the same as "lfs == 1.0"). "lua" is an special dependency name; it matches not a rock, but the version of Lua in use. Multiple version constraints can be joined with a `comma`, e.g. `"neovim >= 5.0, < 7.0"`. - * If no version is specified, then HEAD is assumed valid. - * If no upper bound is specified, then any commit after the tag corresponding to the lower bound is assumed valid. The commit chosen is up to the plugin manager's discretion, but implementers are strongly encouraged to always use the latest valid commit. - * If an upper bound is specified, then the the tag corresponding to that upper bound is the latest commit that is valid - - * `source` (String) The source of the dependency. See previous `source` description. - * `releases_only` (Boolean) Whether the package manager should only resolve version constraints to include tagged releases. - - -* `external_dependencies` (Table) Like dependencies, this specifies packages which are required for the package but should *not* be managed by the Neovim package manager, such as `gcc` or `cmake`. Package managers are encouraged to provide a notification to the user if the dependency is not available. - * `version` (String) same as `dependencies` - -# Example - -```lua -package = "lspconfig" -version = "0.1.2" -specification_version = "0.1.0" -source = "git://github.com/neovim/nvim-lspconfig.git", -description = { - summary = "Quickstart configurations for the Nvim-lsp client", - detailed = [[ - lspconfig is a set of configurations for language servers for use with Neovim's built-in language server client. Lspconfig handles configuring, launching, and attaching language servers. - ]], - homepage = "git://github.com/neovim/nvim-lspconfig/", - license = "Apache-2.0" -} -dependencies = { - neovim = { - version = ">= 0.6.1", - source = "git://github.com/neovim/neovim.git" - }, - gitsigns = { - version = "> 0.3", - source = "git://github.com/lewis6991/gitsigns.nvim.git" - } -} -external_dependencies = { - git = { - version = ">= 1.6.0", - }, -} -``` - -And in json format -```json -{ - "package" : "lspconfig", - "version" : "0.1.2", - "specification_version" : "0.1.0", - "source" : "git://github.com/neovim/nvim-lspconfig.git", - "description" : { - "summary" : "Quickstart configurations for the Nvim-lsp client", - "detailed" : "lspconfig is a set of configurations for language servers for use with Neovim's built-in language server client. Lspconfig handles configuring, launching, and attaching language servers", - "homepage" : "https://github.com/neovim/nvim-lspconfig/", - "license" : "Apache-2.0" - }, - "dependencies" : { - "neovim" : { - "version" : ">= 0.6.1", - "source" : "git://github.com/neovim/neovim.git" - }, - "gitsigns" : { - "version" : "> 0.3", - "source" : "git://github.com/lewis6991/gitsigns.nvim.git" - } - }, - "external_dependencies" : { - "git" : { - "version" : ">= 1.6.0", - }, - } -} -``` - -# Guidelines for `packspec` implementers - -The main features that must be implemented to be considered `packspec`-compatible are: - -## Managing package versions -- Must be able to fetch and parse `packspec.json` files -- Must be able to use `git` to retrieve and manipulate package sources -- Must be able to fetch tagged commits for specified package versions -- Must be able to check for updated `packspec.json` files - -## Managing Neovim dependencies - -- Must be able to check the current version of `Neovim` and warn on incompatibility -- Must be able to retrieve and manage the specified versions of dependencies transitively, starting from user-specified packages -- Must either be able to solve for compatible versions of dependency packages across all dependency relationships, or warn users if using a potentially inconsistent version resolution strategy (e.g. picking the first specified version of a dependency). -- [optional] remove dependencies when they are no longer required (transitively) by any user-specified packages - -## Managing external dependencies - -- Must be able to check for the existence of a corresponding executable on the user's system -- [optional] check the version constraints and warn the user if the external dependency are incompatible diff --git a/docs/README.md b/docs/README.md index ce0310c..804e216 100644 --- a/docs/README.md +++ b/docs/README.md @@ -22,16 +22,19 @@ a formalized way to list dependencies by URL. - [lazy.nvim](https://github.com/folke/lazy.nvim/) - TBD -# Client requirements +# Package requirements -- `git` (packages can live at any git URL) -- JSON parser - -# Package server requirements +The [package specification](./spec.md) specifies the structure of a package and the `pkg.json` format. - Dependency URLs are expected to be git repos. - TODO: support other kinds of artifacts, like zip archives or binaries. +# Client requirements + +- `git` (packages can live at any git URL) +- JSON parser +- [client guidelines](./client-spec.md) + # Design 1. Support _all "assets" or "artifacts" of any kind_. @@ -47,5 +50,6 @@ a formalized way to list dependencies by URL. # References - https://json-schema.org/ +- https://github.com/luarocks/luarocks/wiki/Rockspec-format - lazy.nvim [pkg.json impl](https://github.com/folke/lazy.nvim/pull/910/files#diff-eeb8f2e48ace6e2f4c40bf159b7f59e5eb1208e056a3f9f1b9cc6822ecb45371) - [A way for artifacts to depend on other artifacts.](https://sink.io/jmk/artifact-system) diff --git a/docs/client-spec.md b/docs/client-spec.md new file mode 100644 index 0000000..3fa6593 --- /dev/null +++ b/docs/client-spec.md @@ -0,0 +1,19 @@ +# pkg.json client specification + +_Work-in-progress: These guideliens are subject to change._ + +pkg.json clients... + +## Fetching dependencies + +- MUST be able to fetch and parse `pkg.json` files +- MUST be able to use `git` to clone dependencies and query git repo info +- MUST be able to fetch tagged commits for specified package versions +- MUST be able to check for updated `pkg.json` files + +## Resolving dependencies + +- MUST be able to check the current version of `Neovim` and warn on incompatibility +- MUST be able to fetch and manage the specified versions of dependencies transitively: if dependencies have `pkg.json`, read it and process it, recursively. +- MUST either be able to solve for compatible versions of dependency packages across all dependency relationships, or warn users if using a potentially inconsistent version resolution strategy (e.g. picking the first specified version of a dependency). +- MAY remove dependencies when they are no longer required (transitively) by any user-specified packages diff --git a/docs/spec.md b/docs/spec.md new file mode 100644 index 0000000..f2751fd --- /dev/null +++ b/docs/spec.md @@ -0,0 +1,136 @@ +# pkg.json format specification + +## Example + +``` +{ + "name" : "lspconfig", // OPTIONAL cosmetic name, not used for resolution nor filesystem locations. + "description" : "Quickstart configurations for the Nvim-lsp client", // OPTIONAL + "engines": { + "nvim": "^0.10.0", + "vim": "^9.1.0" + }, + "repository": { // REQUIRED + "type": "git", // reserved for future use + "url": "https://github.com/neovim/nvim-lspconfig" + }, + "dependencies" : { // OPTIONAL + "https://github.com/neovim/neovim" : "0.6.1", + "https://github.com/lewis6991/gitsigns.nvim" : "0.3" + }, +} +``` + +- Dependencies aren't required to have a `pkg.json` file. Only required for the "leaf nodes". + - `pkg.json` can declare a dependency on any random artifact fetchable by URL. The upstream dependency doesn't need a `pkg.json`. +- Version specifiers in `dependencies` follow the [NPM version range spec](https://devhints.io/semver) ~~[cargo spec](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html)~~ + - Supported by Nvim `vim.version.range()`. + - Extensions to npm version spec: + - `"HEAD"` means git HEAD. (npm version spec defines `""` and `"*"` as latest stable version.) + - ~~Do NOT support "Combined ranges".~~ + - Treat any string of length >=7 and lacking "." as a commit-id. + - Only support commit-id, tags, and HEAD. + - Tags must contain a non-alphanumeric char. +- Out of scope: + - "pack" (creating a package) + - "publish" is out of scope, because `pkg.json` is decentralized. Publishing a package means pushing it to a git repo with a top-level `pkg.json`. + - "uninstall" https://docs.npmjs.com/cli/v9/using-npm/scripts#a-note-on-a-lack-of-npm-uninstall-scripts + +## Semantic versioning + +Packages SHOULD be [semantically versioned](https://semver.org/). While other versioning schema have uses, semver allows for plugin managers to provide smart dependency resolution. + +## File location + +Packages MUST have a single, top-level package metadata file named `pkg.json`. +This may be relaxed in the future. + +## Fields + +* metadata: `pkg.json` allows arbitrary user-defined fields at any nesting level, as long as they don't conflict with specification-owned fields. + * Like NPM's `package.json`, this makes the format easy to extend. But application fields should be chosen to avoid potential conflict with fields added to future versions of the spec. For example, `devDependencies` may be added to the spec so applications SHOULD NOT extend the format with that field. +* `package` (String) the name of the package +* `version` (String) the version of the package. SHOULD obey semantic versioning conventions. Plugins SHOULD have a git _tag_ matching this version. For all version identifiers, implementation should check for a `version` prefixed with `v` in the git repository, as this is a common convention. +* `source` (String) The URL of the package source archive. Examples: "http://github.com/downloads/keplerproject/wsapi/wsapi-1.3.4.tar.gz", "git://github.com/keplerproject/wsapi.git". Different protocols are supported: + * `file://` - for URLs in the local filesystem (note that for Unix paths, the root slash is the third slash, resulting in paths like "file:///full/path/filename" +* `dependencies` (`List[Table]`) Object whose keys are URLs and values are version specifiers. + * Compare NPM, where URL is the value rather than the key: [URLs as Dependencies](https://docs.npmjs.com/cli/v9/configuring-npm/package-json#urls-as-dependencies) + +## Changes + +- renamed `packspec.json` to `pkg.json` ~~`deps.json`~~ (to hint that it's basically a subset of NPM's `package.json`) +- removed `"version" : "0.1.2",` because package version is provided by the `.git` repo info +- removed `external_dependencies` +- removed `specification_version`. The lack of a "spec version" field means the spec version is `1.0.0`. If breaking changes are ever needed then we could introduce a "spec version" field. +- renamed `"source" : "git:…",` to `repository.url` +- renamed `package` to `name` (to align with NPM) +- changed the shape of `description` from object to string (to align with NPM) +- changed `dependencies` shape to align with NPM. Except the keys are URLs. + - Leaves the door open for non-URL keys in the future. + +## Closed questions + +- Top-level application-defined "metadata" field (`client`, `user`, `metadata`, ...?) for use by clients (package managers)? + - `pkg.json` allows arbitrary application-defined fields, as `package.json` does. +- "Ecosystem-agnostic" means that https://luarocks.org packages can't be consumed? + - If Nvim plugins can successfully use luarocks then `pkg.json` is redundant. `pkg.json` is only useful for ecosystems that don't have centralized package management. +- Are git submodules/subtrees a viable solution for git-only dependency trees? + - https://stackoverflow.com/a/61961021/152142 + - pro: avoids another package/deps format + - con: + - not easy for package authors to implement (run `git` commands instead of editing a json file) + - no `engines` field: how will aggregators build a package list? + - no support for non-git blobs +- Does the lack of a `version` field mean that a manifest file always tracks HEAD of the git repo? + - The dependents declare what version they need, which must be available as a git tag in the dependency. Thus no need for `pkg.json` to repeat that information. The reason that `package.json` and other package formats need a `version` field is because they _don't_ require a `.git` repo to be present. +- Should consumers of dependencies need to control how a dependency is resolved? + - `repository.type` is available for future use if we want to deal with that. +- It'd be nice if the spec enforces globally unique names... Then `dependencies` could look like `{ "dependencies": { "plenary.nvim": "1.0.0" } }` + - Requiring URIs achieves that, without a central registry. +- Should `name` be removed? Because `repository.url` already defines the "name" (which can be prettified in UIs). + - Defined `name` as OPTIONAL and strictly cosmetic (not used for programmatic decisions or filesystem paths). +- `package.json` has an [`engines` field](https://docs.npmjs.com/cli/v9/configuring-npm/package-json#engines) that declares what software can _run_ the package. Example: + ``` + "engines": { + "vscode": "^1.71.0" + }, +- How to deal with dependencies moving to a new host? Should `pkg.json` support "fallback" URLs? + - The downstream must update its URLs. + +## Open questions + +- should URL be the version (value) rather than the key? see NPM [URLs as Dependencies](https://docs.npmjs.com/cli/v9/configuring-npm/package-json#urls-as-dependencies) +- via @folke: most important to ideally be in the spec: + - ✅ dependencies + - ✅ metadata probably makes sense, NPM itself allows arbitrary fields in `package.json` + - ❓ build + - ❓whether the plugin needs/supports setup() + - ❓main module for the plugin (lazy guesses that automatically, but would be better to have this part of the spec) +- Can `pkg.json` be a strict subset of NPM `package.json` ? The ability to validate it with https://www.npmjs.com/package/read-package-json is attractive... +- Non-git dependencies ("blobs"): require version specifier to be object (instead of string): + - `"https://www.leonerd.org.uk/code/libvterm/libvterm-0.3.2.tar.gz": { "type": "tar+gzip", "version": "…" }` + - How can a package manager know the blob has been updated if there's no git info? (Answer: undefined.) +- `scripts` and "build-time" tasks ([lifecycle](https://docs.npmjs.com/cli/v9/using-npm/scripts#life-cycle-operation-order)) + - Scripts must be array of strings (unlike npm package.json). + - Scripts are run from the root of the package folder, regardless of what the current working directory is. + - Predefined script names and lifecycle order: + - These all run after fetching and writing the package contents to the engine-defined package path, in order. + - `preinstall` + - `install` + - `postinstall` +- Naming conflict: what happens if `https://github.com/.../foo` and `https://sr.ht/.../foo` are in the dependency tree? + ``` + .local/share/nvim/site/pack/github.com/start/ + .local/share/nvim/site/pack/sr.ht/start/ + ``` + + +# Strategy + +- [x] specify packspec (above) +- [ ] specify ecosystem-agnostic client behavior (report conflicts, fetch things into `pack/` dir, update existing dir, ...) +- [ ] specify what is undefined (i.e. owned by the per-ecosystem "engine", for example vim/nvim packages are fetched into 'packpath') + +# TODO (future) + +- nested packages in workspace (`foo/pkg.json`, `foo/bar/pkg.json`)