add inital files

rashley-iqt · rashley-iqt · commit 32d9e2045fbc · 2024-03-18T11:31:59.000-04:00
diff --git a/.gitignore b/.gitignore
@@ -1,160 +1,10 @@
-# Byte-compiled / optimized / DLL files
-__pycache__/
-*.py[cod]
-*$py.class
-
-# C extensions
-*.so
-
-# Distribution / packaging
-.Python
-build/
-develop-eggs/
-dist/
-downloads/
-eggs/
-.eggs/
-lib/
-lib64/
-parts/
-sdist/
-var/
-wheels/
-share/python-wheels/
-*.egg-info/
-.installed.cfg
-*.egg
-MANIFEST
-
-# PyInstaller
-#  Usually these files are written by a python script from a template
-#  before PyInstaller builds the exe, so as to inject date/other infos into it.
-*.manifest
-*.spec
-
-# Installer logs
-pip-log.txt
-pip-delete-this-directory.txt
-
-# Unit test / coverage reports
-htmlcov/
-.tox/
-.nox/
-.coverage
-.coverage.*
-.cache
-nosetests.xml
-coverage.xml
-*.cover
-*.py,cover
-.hypothesis/
-.pytest_cache/
-cover/
-
-# Translations
-*.mo
-*.pot
-
-# Django stuff:
-*.log
-local_settings.py
-db.sqlite3
-db.sqlite3-journal
-
-# Flask stuff:
-instance/
-.webassets-cache
-
-# Scrapy stuff:
-.scrapy
-
-# Sphinx documentation
-docs/_build/
-
-# PyBuilder
-.pybuilder/
-target/
-
-# Jupyter Notebook
-.ipynb_checkpoints
-
-# IPython
-profile_default/
-ipython_config.py
-
-# pyenv
-#   For a library or package, you might want to ignore these files since the code is
-#   intended to run in multiple environments; otherwise, check them in:
-# .python-version
-
-# pipenv
-#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
-#   However, in case of collaboration, if having platform-specific dependencies or dependencies
-#   having no cross-platform support, pipenv may install dependencies that don't work, or not
-#   install all needed dependencies.
-#Pipfile.lock
-
-# poetry
-#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
-#   This is especially recommended for binary packages to ensure reproducibility, and is more
-#   commonly ignored for libraries.
-#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
-#poetry.lock
-
-# pdm
-#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
-#pdm.lock
-#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
-#   in version control.
-#   https://pdm.fming.dev/#use-with-ide
-.pdm.toml
-
-# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
-__pypackages__/
-
-# Celery stuff
-celerybeat-schedule
-celerybeat.pid
-
-# SageMath parsed files
-*.sage.py
-
-# Environments
-.env
-.venv
-env/
-venv/
-ENV/
-env.bak/
-venv.bak/
-
-# Spyder project settings
-.spyderproject
-.spyproject
-
-# Rope project settings
-.ropeproject
-
-# mkdocs documentation
-/site
-
-# mypy
-.mypy_cache/
-.dmypy.json
-dmypy.json
-
-# Pyre type checker
-.pyre/
-
-# pytype static type analyzer
-.pytype/
-
-# Cython debug symbols
-cython_debug/
-
-# PyCharm
-#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
-#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
-#  and can be added to the global gitignore or merged into this file.  For a more nuclear
-#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
-#.idea/
+.trash
+.DS_Store
+
+.obsidian/*
+!.obsidian/app.json
+!.obsidian/config
+!.obsidian/community-plugins.json
+!.obsidian/core-plugins.json
+!.obsidian/graph.json
+!.obsidian/templates.json
diff --git a/.obsidian/app.json b/.obsidian/app.json
@@ -0,0 +1 @@
+{}
diff --git a/.obsidian/core-plugins.json b/.obsidian/core-plugins.json
@@ -0,0 +1,20 @@
+[
+  "file-explorer",
+  "global-search",
+  "switcher",
+  "graph",
+  "backlink",
+  "canvas",
+  "outgoing-link",
+  "tag-pane",
+  "page-preview",
+  "daily-notes",
+  "templates",
+  "note-composer",
+  "command-palette",
+  "editor-status",
+  "bookmarks",
+  "outline",
+  "word-count",
+  "file-recovery"
+]
diff --git a/Development Practices/Best Practices.md b/Development Practices/Best Practices.md
@@ -0,0 +1,27 @@
+# Source Code level
+
+1. Dependency pinning - Specific versions of all dependencies for proper reproducibility - Relevant tools: Poetry, NPM, Nix
+2. Lockfiles - goes with dependency pinning to ensure that the entire web of transitive dependencies are fixed and mutuall compatible - Relevant tools: Poetry, Pip-compile, NPM
+3. Unit Tests - ensure that an atomic unit of code (function or class) functions correctly. Ideally each addition or change to the code should have a corresponding unit test to show that it is working correctly. - Relevant tools: Pytest, mocha, Jest, rhino, mock
+4. Integration Tests - Show that units of code work together properly. Not every change will require modifications to integration tests, but it is important to maintain appropriate coverage levels and to monitor breaking changes in interfaces
+5. Code Style - This is fancy talk for readability. Use tools to monitor this to ensure consistency within your codebase. - relevant tools: Flake8, black, prettier
+6. Linting - linting is the process of checking the source code of a file looking for anti-patterns or artifacts that may indicate the presence of bugs - related tools: lint, eslint, pylint
+7. Doc Comments - comments that are placed at the top of classes and methods to allow generation of documentation - related tools: Sphinx,PyDoc, Doxygen
+8. Containerization - Where applicable it is nice to have container(s) for your code. Containers can also make a great platform for certain types of testing. Remember that you can have multiple different `Dockerfile`s and that you can specify the `Dockerfile` to use with the `-f` flag. Containers can also be used ephemerally to perform tasks and generate output. Avoid having containers that need to be managed by containers as this requires using the `--privileged` flag/exposing the docker socket which opens the way to conatiner escape techniques. Base images are your friend.
+
+# Repo Level
+
+I'm trying to build out a repo template that will help with creating huge chunks of this and just allow you to fill in the blanks relevant to your project, but i'm not 100% there yet.
+
+1. Community Standards - All repos should include and have regularly updated:
+   - README
+   - License
+   - Security Policy
+   - Code of Conduct
+   - Contributor guide
+  If possible it is aslo desirable to have issue and PR templates to make it easier for outside contributors.
+2. Maintainers - each repo should have at least 2 top line maintainers who are familar with the codebase as a whole and who are willing to take on the work of reviewing/merging PRs
+3. Ignore files - `.gitignore` and `.dockerignore` are the 2 main ones, but depending on purpose and tooling you may need others. May require a `.semgrepignore` for certain scenarios
+4. Pre-commit hooks - setup tools that you want run locally by contributors before code is commited. You need to install pre-commit on your machine for this to work but it can be a real time saver to not have to invoke an entire CI loop to find out that unit tests fail or Flake8 has flagged a bunch of code. This is also a good place to run things like Black to clean up code.
+5. Workflows - There will probably be some overlap with pre-commit on these since pre commit can be unreliable. The goal of these is to simplify the lives of PR reviewers so add workflows that give good heuristics about whether or not a PR is valid. 2 specific workflows will be provided before a repo is made public. One to Scan for any kinds of secrets within the repo and another to look for anti-patterns in the code. You don't need to worry about building these out. They will eventually be part of the standard template. 
+6. Codeowners - having this in place will allow GitHub to automatically tag appropriate reviewers. do this by adding a file called `CODEOWNERS` underneath of the `.github` directory. it uses a similar syntax to the `.gitignore` file. The first entry should be `*` followed by the usernames of the repo's maintainers. Subsequent entries should call out reviewers with specialized expertise on particular types of files or specific areas of the code. Please go in increasing order of specificity as the most specific rule to match is what will be used. 
diff --git a/Development Practices/Tools/Containers/Dockerfiles.md b/Development Practices/Tools/Containers/Dockerfiles.md
@@ -0,0 +1,6 @@
+# Best Practices
+ - have a `USER` directive whenever possible. Running as root is unwise unless it is absolutely necessary.
+ - To minimize the number of layers created at build time try to consolidate multiple commands either by using shell scripts or by consolidating multiple commands into one `RUN` directive by using `&&` and `\`
+ - **DO NOT** embed secrets inside of your dockerfile at build time. Providing them as environment variables is also not encouraged. If possible use the built in secrets mechanisms to provide and keys or credentials.
+ - It is preferable to provide environment variables via a `.env` file rather than passing them on the command line.
+ 
diff --git a/Development Practices/Tools/Containers/README.md b/Development Practices/Tools/Containers/README.md
@@ -0,0 +1,3 @@
+Container specific tools and knowledge:
+- [Dockerfiles](Dockerfiles.md)
+- Compose Plugin - Planned but not yet available
diff --git a/Development Practices/Tools/Pre-commit.md b/Development Practices/Tools/Pre-commit.md
@@ -0,0 +1,76 @@
+# Why use it?
+---
+### Pre-Commit Hooks
+Git supports a variety of [hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks). Client-side pre-commit hooks run on your development machine before a commit actually happens to your local repo. These hooks live in your repo's `.git/hooks` directory. It is pre-populated with a variety of sample scripts. In practice these scripts can be any valid bash script. However, by necessity, these scripts live outside of Git's version control workflow which means that they aren't especially convenient or portable.
+### pre-commit the tool
+[pre-commit](https://pre-commit.com) is a tool designed to make it easy for developers to version control, run, and share pre-commit scripts. It runs a set of scripts defined by a `.yaml` file on changed files prior to a commit happening. If the scripts fail, it will prevent the commit from occurring.
+
+# Installation
+---
+Information is based on [the official documentation](https://pre-commit.com/#installation) but uses [pipx](https://github.com/pypa/pipx) instead of `pip` because `pipx` installs the application and all of it's dependencies inside of a sandbox, so you do not have to worry about an installed application breaking your python environment.
+### Macos/Linux/Windows
+```sh
+pipx install pre-commit
+```
+# Usage
+---
+Upon cloning a repo containing an existing `.pre-commit-config.yaml` file, or after creating one install hook scripts with:
+```sh
+pre-commit install
+```
+to see what the results of running against currently uncommited files:
+```sh
+pre-commit run
+```
+
+to see the results of a run against all files in the current repo:
+```sh
+pre-commit run --all-files
+```
+
+# Configuration
+---
+Configuration is handled by the file `.pre-commit-config.yaml` . a basic starting configuration can be found below.
+## Starting config
+**Note:** this config assumes a python project. Entries may change for projects in other languages but the basic structure remains the same
+```yml
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.0.1
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-case-conflict
+      - id: check-json
+      - id: pretty-format-json
+        args: ['--autofix']
+      - id: double-quote-string-fixer
+      - id: check-yaml
+  - repo: https://github.com/asottile/reorder_python_imports
+    rev: v2.6.0
+    hooks:
+      - id: reorder-python-imports
+  - repo: https://github.com/python-poetry/poetry
+    rev: 1.5.1
+    hooks:
+    -   id: poetry-check
+    -   id: poetry-lock
+        args: [--no-update]
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.0.291
+    hooks:
+      - id: ruff
+        args: [--respect-gitignore]
+  - repo: https://github.com/psf/black
+    rev: 23.9.1
+    hooks:
+      - id: black
+        args: [-q, --check]
+  - repo: local
+    hooks: 
+      - id: trufflehog 
+        name: TruffleHog 
+        description: Detect secrets in your data. 
+        entry: bash -c 'docker run --rm -i -v "$PWD:/pwd" trufflesecurity/trufflehog:latest git file:///pwd --since-commit HEAD --fail --only-verified --json'
+        language: system 
+        stages: ["commit", "push"]
diff --git a/Development Practices/Tools/Python/Poetry.md b/Development Practices/Tools/Python/Poetry.md
@@ -0,0 +1,79 @@
+# Why use it?
+### Dependency Pinning
+Python Poetry simplifies multiple aspects of dependency management in Python. The most important job it will perform is dependency pinning. This means that instead of specifying your dependency as `httpx` (which is tacitly latest) it is specified as `httpx==0.23.3` Traditionally this has been painful in Python because you need to look up the version on PyPi (or whatever package repo you are using) and make an appropriate entry in the `requirements.txt` file. Instead with Poetry the command is a simple `poetry add httpx` which handles the lookup for you and makes an appropriate entry in the `pyproject.toml` file.
+### Setup
+Poetry also handles setup, replacing the deprecated and dangerous `setup.py` convention. So instead of running `python3 -m pip install .` which is tacitly running `setup.py` which contains an arbitrary set of python commands you would use `poetry install` which handles the installation declaratively and by default sandboxes each application into its own virtual environment.
+
+# Installation
+Information below comes from [the official documentation](https://python-poetry.org/docs/)
+
+### Macos/Linux
+```sh
+curl -sSL https://install.python-poetry.org | python3 -
+```
+
+### Windows
+```
+(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -
+```
+If using a shell that supports them, tab completions can be enabled using the instructions in the documentation.
+
+# Configuration
+
+It is strongly recommended to configure poetry to use virtualenvs inside of a project directory instead of in their default location. to set this configuration use the command:
+```sh
+poetry config virtualenvs.in-project true
+```
+
+# Usage
+The following are some common usage patterns for poetry:
+
+### Adding a dependency
+**Latest:**
+```sh
+poetry add <dependency>
+```
+**Specific version:**
+```sh
+poetry add <dependency>==<version>
+```
+OR
+```sh
+poetry add <dependency>@<version>
+```
+Please note that version constraints can also be specified using [multiple notations](https://python-poetry.org/docs/dependency-specification/#version-constraints)
+
+**As an optional Extra dpendency:**
+```sh
+poetry add -E <extra> <dependency>
+```
+A common use for this pattern is to define development or test 
+### Removing a dependency
+```sh
+poetry remove <dependency>
+```
+
+### Installation
+```sh
+poetry install
+```
+
+### Running a script 
+```sh
+poetry run <script> <args>
+```
+
+### Update dependencies
+```sh
+poetry update
+```
+This will update the lockfile in accordance with the constraints specified in the `pyproject.toml`
+
+### Regenerate the lockfile
+```sh
+poetry lock --no-update
+```
+As an example you might run unit tests using:
+```sh
+poetry run pytest ./
+```
diff --git a/Development Practices/Tools/Python/README.md b/Development Practices/Tools/Python/README.md
diff --git a/Development Practices/Tools/README.md b/Development Practices/Tools/README.md
diff --git a/README.md b/README.md

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+Container specific tools and knowledge:`
	`2`	`+- [Dockerfiles](Dockerfiles.md)`
	`3`	`+- Compose Plugin - Planned but not yet available`