[WIP]: Version 2 #380
Draft
[WIP]: Version 2 #380
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is the beginnings of a major rewrite of the `showyourwork` back end that I've been working on for the last little while. My plan is to get this to the point where the user experience won't be very different. The configuration won't be 100% backwards compatible, but it should be pretty close.
The main user-facing difference will be that the figure dependencies will no longer be specified using the `\script` command in the TeX manuscript. Instead, the data dependency structure will be completely inferred from the `snakemake` DAG. To get `showyourwork` to automatically generate rules for your figures or other dynamically generated files, you'll need to instead specify these dependencies using the following syntax in the config file:
```yaml
dynamic:
- script: path/to/script.py
output: path/to/figure.pdf
```
where `output` can also be a list of output files. This change in syntax is nice for a few reasons, but I big one is that it means that the dependencies labelled in the document will always be _exactly the dependencies required to generate the file_ (because `snakemake` knows that already without us needing to specify it!), even without the `dependencies:` entry in the config file (which no longer exists).
From the back end perspective this rewrite is motivated by the following:
1. I wanted to be able to support more flexible workflows. For example, I didn't want to require the specific `showyourwork` directory structure. Rewriting makes this easier because expectations about the file system were strongly baked into the previous implementation. Other workflows include future developments like support for mystjs-based build systems.
2. As part of supporting these flexible workflows, I wanted to make `showyourwork` more flexible and extensible. This requires well-documented (to-do) hooks to allow customization. This was hard with the existing implementation in part because of the fact that `snakemake` was executed twice for every build. The change above ☝️ means that we only need to fire `snakemake` up once, and makes it somewhat easier to provide hooks. This should also make everything faster!
3. I wanted to improve the testing infrastructure, and in particular add unit tests for the core functionality that didn't require the expensive tests that generate new GitHub repositories that currently provide the majority of test coverage.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* logging and verbosity * updating rule names * fixing rendering test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.