|
| 1 | +--- |
| 2 | +title: "Useful code snippets" |
| 3 | +subtitle: "Key commands and code for research software development" |
| 4 | +--- |
| 5 | + |
| 6 | +## Essential linux/bash commands |
| 7 | + |
| 8 | +These commands will be useful if you are using the Linux Codespace machine, or if you are working locally on your machine either via GitBash for Windows, or using Terminal on Mac. |
| 9 | + |
| 10 | +```bash |
| 11 | +cd # change directory to home |
| 12 | +cd /workspaces # return to the /workspaces directory |
| 13 | +cd .. # go up a level in the directory structure |
| 14 | +ls # list the contents of the current directory |
| 15 | +pwd # get the path to the current working directory |
| 16 | +``` |
| 17 | + |
| 18 | +## Essential git commands |
| 19 | + |
| 20 | +We will use git and a GitHub remote to track our changes. You can use git in the same way you would from your local machine. |
| 21 | + |
| 22 | +If you want to use git on your machine, [install git here](https://git-scm.com/). When you create a new repository on GitHub, instead of launching a Codespace, instead follow the directions on your new repository page to set up a git repository on your machine. |
| 23 | + |
| 24 | +```bash |
| 25 | +git status # check on status of current git repo |
| 26 | +git branch NAME # create a branch called NAME |
| 27 | +git checkout NAME # swap over to the branch called NAME |
| 28 | +git add NAME # stage FILE for commit |
| 29 | +git commit # commit the staged files (this will open your text editor to create a commit message) |
| 30 | +git push origin NAME # push local commits to the remote branch tracking the branch NAME |
| 31 | + |
| 32 | +# Added something unintentional? |
| 33 | +git reset --soft HEAD^ # undo a git commit |
| 34 | +git reset # undo git add |
| 35 | +git restore --staged FILE # undo git add to specific file |
| 36 | +git restore FILE # undo all changes to an unstaged file since last commit |
| 37 | + |
| 38 | +# after merging a pull request |
| 39 | +git fetch -p # delete branches that no longer exist in the remote |
| 40 | + |
| 41 | +# go back to an old version and put it on a branch |
| 42 | +git checkout -b NEW-BRANCH-NAME-FOR-OLD-VERSION git-hash-here |
| 43 | +``` |
| 44 | + |
| 45 | +## Essential conda commands |
| 46 | + |
| 47 | +We recommend you use [Miniforge](https://github.com/conda-forge/miniforge) to set up an open-source version of Conda on your machine. |
| 48 | + |
| 49 | +To install on Linux (on your virtual Codespaces machine), follow these commands: |
| 50 | + |
| 51 | +```bash |
| 52 | +wget -O Miniforge3.sh "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" |
| 53 | + |
| 54 | +bash Miniforge3.sh -b -p "${HOME}/conda" |
| 55 | + |
| 56 | +source "${HOME}/conda/etc/profile.d/conda.sh" |
| 57 | +``` |
| 58 | + |
| 59 | +See the Miniforge link above for installation instructions for other operating systems. |
| 60 | + |
| 61 | +```bash |
| 62 | +# from terminal/outside a conda env |
| 63 | +conda env list # list built environments |
| 64 | +conda env create --file PATH/TO/A/FILE # build a conda env from a file |
| 65 | +conda env create --file environment.yml # build a conda env from a file in the current directory |
| 66 | +conda env update --file environment.yml --prune # update a conda env from file and remove unused libraries |
| 67 | +conda activate ENV-NAME # activate the environment ENV-NAME |
| 68 | + |
| 69 | +# from inside a conda env (after activating the env) |
| 70 | +conda list # lists installed packages in the env |
| 71 | +conda env export --no-builds > exported-env.yml # exports all packages in the env |
| 72 | +conda env export --from-history > exported-env.yml # exports the packages that were explicitly installed |
| 73 | +``` |
| 74 | + |
| 75 | +Export your conda and pip dependencies from the active environment without version numbers (so that the resulting `environment.yml` file can be used to build a new environment): |
| 76 | + |
| 77 | +```bash |
| 78 | +### Extract installed pip packages |
| 79 | +pip_packages=$(conda env export | grep -A9999 ".*- pip:" | grep -v "^prefix: " | cut -f1 -d"=") |
| 80 | + |
| 81 | +### Export conda environment without builds, and append pip packages |
| 82 | +conda env export --from-history | grep -v "^prefix: " > new-environment.yml |
| 83 | +echo "$pip_packages" >> new-environment.yml |
| 84 | +``` |
| 85 | + |
| 86 | +Conda environment file template: |
| 87 | + |
| 88 | +```yml |
| 89 | +name: my-env-name |
| 90 | + |
| 91 | +channels: |
| 92 | + - conda-forge |
| 93 | + - nodefaults |
| 94 | + |
| 95 | +dependencies: |
| 96 | + - python=3.13 |
| 97 | + - pytest |
| 98 | + - blackd |
| 99 | + - isort |
| 100 | + |
| 101 | + # remove/modify these as needed |
| 102 | + - numpy |
| 103 | + - pandas |
| 104 | + |
| 105 | + # keep this to install your Python package locally |
| 106 | + - pip |
| 107 | + - pip: |
| 108 | + - --editable . |
| 109 | +``` |
| 110 | +
|
| 111 | +## Pyproject.toml |
| 112 | +
|
| 113 | +In order to install your package in your environment, you need to have a TOML file in your project directory. |
| 114 | +
|
| 115 | +```toml |
| 116 | +[build-system] |
| 117 | +requires = ["setuptools", "wheel"] |
| 118 | +build-backend = "setuptools.build_meta" |
| 119 | + |
| 120 | +[project] |
| 121 | +name = "amazing_project" |
| 122 | +version = "0.1.0" |
| 123 | +description = "Brief description" |
| 124 | +authors = [{name = "Your Name"}] |
| 125 | +requires-python = ">=3.13" |
| 126 | + |
| 127 | +[tool.setuptools.packages.find] |
| 128 | +where = ["src"] |
| 129 | +``` |
| 130 | + |
| 131 | +## Essential pytest hints |
| 132 | + |
| 133 | +Add the following to the `__init__.py` file in your `tests/` directory: |
| 134 | + |
| 135 | +```python |
| 136 | +import sys |
| 137 | + |
| 138 | +sys.path.append("src") |
| 139 | +``` |
| 140 | + |
| 141 | +You can then run `pytest` from the main repo directory. |
| 142 | + |
| 143 | +Basic test format: |
| 144 | + |
| 145 | +```python |
| 146 | +def test_example(): |
| 147 | + '''Test for the example function''' |
| 148 | + |
| 149 | + # Arrange |
| 150 | + test_variable_1 = 0 |
| 151 | + test_variable_2 = 1 |
| 152 | + expected_output = 7 |
| 153 | + |
| 154 | + # Act |
| 155 | + output = your_function(test_variable_1, test_variable_2) |
| 156 | + |
| 157 | + # Assert |
| 158 | + assert output == expected_output |
| 159 | + |
| 160 | + # No cleanup needed |
| 161 | +``` |
| 162 | + |
| 163 | +## Using Python notebooks |
| 164 | + |
| 165 | +### Working with editable installs |
| 166 | + |
| 167 | +The benefit of editable installs is you can modify your source code, and don't need to reinstall your environment. However, if you are working in a Jupyter notebook or similar, you will have to force reload your module. |
| 168 | + |
| 169 | +Where your locally installed package is called `amazing_project`: |
| 170 | + |
| 171 | +```python |
| 172 | +import amazing_project as ap |
| 173 | +import importlib |
| 174 | +importlib.reload(ap) |
| 175 | +``` |
| 176 | + |
| 177 | +Just re-running the line `import amazing_project as ap` will not update the module. |
| 178 | + |
| 179 | +### Version control with notebooks |
| 180 | + |
| 181 | +Because Jupyter notebooks are not plain Python,. and include information about the rendering of the page you see in your browser, they can be messy when using version control. Additionally, as you can run cells out of order, they can occasionally lead to non-reproducible workflows. |
| 182 | + |
| 183 | +If you only use notebooks for prototyping or showcasing results, and move all your important functional code to plain `.py` files as soon as you can, this might not be too annoying an issue for you. However, if notebooks are an integral part of your process, you might want to look at ways to make them work better for reproducibility and version control. |
| 184 | + |
| 185 | +#### Working with Jupytext |
| 186 | + |
| 187 | +Here's the [Jupytext documentation](https://jupytext.readthedocs.io/en/latest/index.html). |
| 188 | + |
| 189 | +Create a conda env with Jupyter and Jupytext installed, and add the Jupyter extension to VSCode. |
| 190 | + |
| 191 | +To "pair" a notebook (e.g. create a `.py` plaintext version that can be version controlled), use the Jupytext CLI: |
| 192 | + |
| 193 | +``` |
| 194 | +jupytext --set-formats ipynb,py:percent notebook.ipynb |
| 195 | +``` |
| 196 | + |
| 197 | +Now you can add `.ipynb` files to your `.gitignore` and only track the `.py` version. |
| 198 | + |
| 199 | +You can sync these notebooks: |
| 200 | +``` |
| 201 | +jupytext --sync notebook.ipynb |
| 202 | +``` |
| 203 | + |
| 204 | +If you download the code to another machine (`git clone` the repository), you should only have the `.py` files. |
| 205 | + |
| 206 | +You can get `jupytext` to generate the notebook output: |
| 207 | + |
| 208 | +``` |
| 209 | +jupytext --sync notebook.py |
| 210 | +``` |
| 211 | + |
| 212 | +You can also just work from the `jupytext` Python file (the Juptyer VSCode extension will allow sections of it to run as a notebook) and not use the `.ipynb` format file at all. |
| 213 | + |
| 214 | +#### Working with Marimo |
| 215 | + |
| 216 | +Here's the [Marimo documentation](https://docs.marimo.io/). |
| 217 | + |
| 218 | +Create a conda env with Marimo installed. You can launch the Marimo tutorial with `marimo tutorial intro` from inside the active environment. |
| 219 | + |
| 220 | +While technically Marimo notebooks ate `.py` files, you can easily strip them of extra Marimo-related formatting (used to create cells): |
| 221 | + |
| 222 | +```bash |
| 223 | +marimo export script notebook.py -o notebook.script.py |
| 224 | +``` |
| 225 | + |
| 226 | +This is useful when migrating your prototyping work from a notebook to your source code or Python package. |
0 commit comments