A template engine for skeletons that enables multiple styles.

[johanneskoester]

This pull request rewrites the skeleton framework to use jinja2-based templates. By that, we can easily modify the style without touching the code. Moreover the skeleton codebase is simplified, since formatting can happen entirely the templates. Based on that, PR adds a command line argument --style {jinja|plain} which allows to select whether jinja-variables shall be used for the version number or not.
With jinja, I tried to stick as much as reasonable to the current style. In contrast, plain tries to avoid jinja variables when possible (e.g., using {{ PKG_VERSION }} in pypi urls).

Important: The plain style does not increase redundancy in the recipe. When doing an update there is still needs to be only a single entity of version, checksum, and build number changed. See PKG_VERSION above or the cpan example below. URLs can remain unchanged on updates, because they reuse the version.

Notes:

• I could not test luarocks, because there seems to be a syntax error in the parser script that is delivered with the skeleton.
• I did not migrate rpm, because I need some input about why it uses outputs instead of the top-level sections for run_depends and about.

Any comments welcome.

xref: bioconda/bioconda-utils#339

cc: @bgruening, @daler, @jakirkham, @ocefpaf, @msarahan, @mingwandroid, @kalefranz

[johanneskoester]

Here comes an example for the perl-aard recipe:

conda skeleton --style jinja cpan Aard:

{% set name = "perl-aard" %}
{% set version = "0.001" %}
{% set sha256 = "3867fd7ea67bf192b9c64d56313eadcaf49efe595292ca81df408c056a5153ff" %}

package:
  name: {{ name }}
  version: "{{ version }}"

source:
  url: https://cpan.metacpan.org/authors/id/M/MG/MGV/Aard-{{ version }}.tar.gz
  sha256: {{ sha256 }}

build:
  # If this is a new build for the same version, increment the build number.
  number: 0

requirements:
  host:
    - perl
    - perl-extutils-makemaker
    - perl-io-uncompress-bunzip2
    - perl-io-uncompress-inflate
    - perl-json-maybexs
    - perl-list-util
    - perl-uuid-tiny
  run:
    - perl
    - perl-io-uncompress-bunzip2
    - perl-io-uncompress-inflate
    - perl-json-maybexs
    - perl-list-util
    - perl-uuid-tiny

test:
  imports:
    - Aard

about:
  home: http://metacpan.org/pod/Aard
  summary: 'Read aarddict dictionaries'
  license: perl_5
  license_family: ""
  license_file: ""
  doc_url: ""
  dev_url: ""

extra:
  recipe-maintainers:
    - your-github-id-here


# See
# http://docs.continuum.io/conda/build.html for
# more information about meta.yaml

conda skeleton --style plain cpan Aard:

package:
  name: perl-aard
  version: "0.001"

source:
  url: https://cpan.metacpan.org/authors/id/M/MG/MGV/Aard-{{ PKG_VERSION }}.tar.gz
  sha256: 3867fd7ea67bf192b9c64d56313eadcaf49efe595292ca81df408c056a5153ff

build:
  # If this is a new build for the same version, increment the build number.
  number: 0

requirements:
  host:
    - perl
    - perl-extutils-makemaker
    - perl-io-uncompress-bunzip2
    - perl-io-uncompress-inflate
    - perl-json-maybexs
    - perl-list-util
    - perl-uuid-tiny
  run:
    - perl
    - perl-io-uncompress-bunzip2
    - perl-io-uncompress-inflate
    - perl-json-maybexs
    - perl-list-util
    - perl-uuid-tiny

test:
  imports:
    - Aard

about:
  home: http://metacpan.org/pod/Aard
  summary: 'Read aarddict dictionaries'
  license: perl_5
  license_family: ""
  license_file: ""
  doc_url: ""
  dev_url: ""

extra:
  recipe-maintainers:
    - your-github-id-here


# See
# http://docs.continuum.io/conda/build.html for
# more information about meta.yaml

[msarahan]

I'm OK with this, but the communities need to decide which standard they will adhere to. For conda-forge, a large part of that decision will be how easy the bot can adapt to one style or the other. If we already know that the "simple" style will not fly for conda-forge, it is counterintuitive to me to add this functionality to conda-build.

CC @mariusvniekerk @CJ-Wright @scopatz

[johanneskoester]

@msarahan that is exactly the point of this PR. It allows multiple styles. Jinja (the conda-forge style) is the default. But plain is also possible if people like. An advantage of plain is that recipes become more readable. The different communities can simply refer to whatever style they prefer or even easily add their own.

[msarahan]

The possibility of that choice is promotion of a schism. Anything that moves away from the goal of having fewer sources of recipes is not effort in the right direction. Either we choose the new, simpler way and keep the old way around for a while as a transition, or we stick with the old way.

[johanneskoester]

By getting rid of jinja variables when they are not needed, the readability of recipes is increased. This is what we strive for at Bioconda. The reason I made this PR is that I thought it would be utopic to convince all communities to quickly switch their style just because it is possible now (with the availability of PKG_VERSION in the recipe). Via providing styles, the transition can be smooth, without enforcing something unwanted to any of the communities.

I don't think that you can enforce certain recipe styles via skeletons. Conda-forge used the jinja-style before skeletons had been adapted. For Bioconda, it was vice-versa. With this PR it is simpler for both communities , because nobody needs to manually adapt what the skeleton commands print out.

Of course, it would be nice to have a common style (although I think the differences are too marginal to call them a schism). But maybe it is better to let this evolve by providing the choice than to make some kind of decision that probably not everybody can agree with.

[mingwandroid]

I believe tooling and workflow is more important than readability and also contest that jinja2 can if used sensibly increase readability and certainly redundancy.

I'm specifically talking about easy diff and merge here, and also the differences are a lot clearer to see when using jinja2.

Either way it needs a vote IMHO.

[msarahan]

I'm not sure if you've followed all the past discussion about unifying Anaconda's recipes with conda-forge. We ditched all of our old recipes, deprecated the conda-recipes repo, and we fork conda-forge recipes to replace ours. The goal is to be in-sync, and if not in-sync, to at least have git be a useful tool for helping us manage differences. I certainly can't force that practice on bioconda, but I hope you'll agree that having a single source of truth for a given recipe is a good thing to shoot for. If you can't get agreement from conda-forge on heading towards the "simple" style, you are hindering the effort to have that single source of truth.

FWIW, I like the simple style. I respect @mingwandroid's opinion on the diffs. Which do people spend more time looking at? The diffs, or the recipe itself? I think you should put up a proposal/vote at conda-forge and try to get them to agree to move towards your simple style. @mingwandroid should participate in that and make sure his points are considered.

I do not want conda-build to become a tool that has forked behavior for different communities. Especially so because we don't generally follow bioconda's practices at Anaconda or at conda-forge, so any branched logic would be unfamiliar and with a smaller potential maintainer pool. If we can get your utopia, I'm all for it.

[mingwandroid]

90% of packages do not need any modification (these days), about 10% need to be worked on now and then. For those, if version appears twice as literal text, then that's redundant and risks bugs when editing them.

If I update a load of packages at once (as I do, 380 odd now), then the diffs are pure blocks of jinja2 for any 'trivial' update and it's very easy to eyeball (and indeed for automatic tools to detect) that there's no other significant differences.

[mingwandroid]

I'd like to bring awareness of this PR and the discussion whether to use jinja2 or not to @mcg1969, @dsludwig, @quasiben.

[johanneskoester]

@msarahan @mingwandroid I can understand you. In the end, it is your decision how conda-build should behave. So, I am fine with removing the style option from this PR for now, and keeping the "jinja" variant until a decision whether to switch to "plain" has been made.

Just to clarify: the plain style does not introduce redundancy, please see my cpan example above and the added note in the initial description in this PR.

[scopatz]

conda-forge has been slowly evolving to a style that uses Jinja only where approriate, ie when the variable is actually used more than once. I am pretty sure that the auto-tick bot can handle both the jinja and plain styles already, and in any event, the bot will have to adapt to any style that becomes sufficiently popular.

[github-actions]

Hi there, thank you for your contribution!

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this pull request to remain open please:

1. Rebase and verify the changes still work
2. Leave a comment with the current status

NOTE: If this pull request was closed prematurely, please leave a comment.

Thanks!

[github-actions]

Hi there, thank you for your contribution!

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this pull request to remain open please:

1. Rebase and verify the changes still work
2. Leave a comment with the current status

NOTE: If this pull request was closed prematurely, please leave a comment.

Thanks!