Skip to content

Commit

Permalink
Merge pull request #60 from HicResearch/github-precommit
Browse files Browse the repository at this point in the history
Automatically run pre-commit in GitHub workflow for pushes and pull requests
  • Loading branch information
manics authored Jun 14, 2023
2 parents ba4e45d + e794304 commit b2fbe1f
Show file tree
Hide file tree
Showing 33 changed files with 978 additions and 865 deletions.
2 changes: 1 addition & 1 deletion .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Description

----
---

Declaration : _By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license_
31 changes: 31 additions & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
name: Lint

on:
pull_request:
push:
workflow_dispatch:

jobs:
pre-commit:
runs-on: ubuntu-22.04
timeout-minutes: 5

steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: "3.10"

- name: Install git-secrets
run: |
git clone https://github.com/awslabs/git-secrets
cd git-secrets
git checkout 99d01d58ebcc06e237c0e3f3ff5ae628aeef6aa6
sudo make install
# ref: https://github.com/pre-commit/action
# Run "pre-commit run --all-files"
- uses: pre-commit/[email protected]

- name: pre-commit workspace_backup
run: pre-commit run -a --config src/components/workspace_backup/.pre-commit-config.yaml
107 changes: 75 additions & 32 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,43 +1,86 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.1.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-docstring-first
- id: check-json
- id: check-added-large-files
- id: debug-statements
- id: name-tests-test
- id: requirements-txt-fixer
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-docstring-first
- id: check-json
- id: check-added-large-files
- id: debug-statements
- id: name-tests-test
- id: requirements-txt-fixer
- repo: https://github.com/pycqa/isort
rev: 5.11.5
hooks:
- id: isort
args: ["--profile", "black"]
- repo: https://github.com/psf/black
rev: 23.1.0
hooks:
- id: black
language_version: python3 # Should be a command that runs python3.7+
# - repo: https://github.com/pycqa/flake8
# rev: 6.0.0
# hooks:
# - id: flake8
# args:
# [
# "--max-line-length=200",
# "--ignore=E203,W503",
# "--max-cognitive-complexity=17",
# "--max-expression-complexity=9",
# ]
# additional_dependencies: [
# flake8-bugbear, # Detect potential bugs
# flake8-builtins, # Check for built-ins being used as variables
# flake8-cognitive-complexity, # Check max function complexity
# flake8-comprehensions, # Suggestions for better list/set/dict comprehensions
# flake8-eradicate, # Find dead/commented out code
# flake8-expression-complexity, # Check max expression complexity
# flake8-fixme, # Check for FIXME, TODO, and XXX left in comments
# flake8-logging-format, # Validate (lack of) logging format strings
# flake8-mutable, # Check for mutable default arguments
# flake8-pie, # Misc. linting rules
# flake8-pytest-style, # Check against pytest style guide
# flake8-return, # Check return statements
# flake8-simplify, # Suggestions to simplify code
# flake8-use-fstring, # Encourages use of f-strings vs old style
# pep8-naming, # Check PEP8 class naming
# ]
- repo: https://github.com/Lucas-C/pre-commit-hooks-safety
rev: v1.3.1
hooks:
- id: python-safety-dependencies-check
- repo: https://github.com/awslabs/git-secrets
rev: 80230afa8c8bdeac766a0fece36f95ffaa0be778
rev: 99d01d58ebcc06e237c0e3f3ff5ae628aeef6aa6
hooks:
- id: git-secrets
verbose: true
entry: 'git-secrets --register-aws'
entry: "git-secrets --register-aws"
language: script
name: git-secrets-register-aws-provider
- id: git-secrets
verbose: true
entry: 'git-secrets --scan'
entry: "git-secrets --scan"
language: script
name: git-secrets-scan
- repo: https://github.com/markdownlint/markdownlint
rev: v0.11.0
hooks:
- id: markdownlint
name: Markdownlint
description: Run markdownlint on your Markdown files
entry: mdl .
language: ruby
files: \.(md|mdown|markdown)$
verbose: true
args:
- "-s"
- "mdl_style.rb"
- repo: https://github.com/awslabs/cfn-python-lint
rev: v0.59.1
hooks:
- id: cfn-python-lint
files: src/.*(Cfn)\.(yml|yaml)$
- repo: https://github.com/pre-commit/mirrors-prettier
rev: v2.7.1
hooks:
- id: prettier
- repo: https://github.com/awslabs/cfn-python-lint
rev: v0.59.1
hooks:
- id: cfn-python-lint
files: src/.*(Cfn)\.(yml|yaml)$
- repo: https://github.com/PyCQA/bandit
rev: "1.7.4"
hooks:
- id: bandit
entry: bandit
exclude: ^tests/

# workspace_backup/ has it's own pre-commit-config.yaml
# egress_app_* are being moved to their own repos
exclude: "^src/components/(egress_app_backend|egress_app_frontend|workspace_backup)/"
17 changes: 8 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ Open Original Science Exploration

## What is TREEHOOSE

Trusted Research Environments (TREs) are secure computing environments providing
secure access to sensitive data for research purposes. HDR UK has a set of
[resources and guides](https://www.hdruk.ac.uk/access-to-health-data/trusted-research-environments/)
explaining TREs in more detail in terms of health data research, although TREs
Trusted Research Environments (TREs) are secure computing environments providing
secure access to sensitive data for research purposes. HDR UK has a set of
[resources and guides](https://www.hdruk.ac.uk/access-to-health-data/trusted-research-environments/)
explaining TREs in more detail in terms of health data research, although TREs
are not exclusive to health data.

TREEHOOSE is an open-source platform for deploying TREs on Amazon Web Services
Expand Down Expand Up @@ -53,9 +53,9 @@ and discuss future enhancements.

## Use cases

TREEHOOSE was originally developed for use with confidential healthcare data
such as patient electronic health records, but is designed to be used and
customised for all research and analysis disciplines which require access to
TREEHOOSE was originally developed for use with confidential healthcare data
such as patient electronic health records, but is designed to be used and
customised for all research and analysis disciplines which require access to
sensitive data.

---
Expand Down Expand Up @@ -101,7 +101,6 @@ This project is licensed under the [Apache-2.0 License](./LICENSE).

## Funding

This work was funded by UK Research & Innovation Grant Number MC_PC_21032 as
This work was funded by UK Research & Innovation Grant Number MC_PC_21032 as
part of Phase 1 of the DARE UK (Data and Analytics Research Environments UK)
programme, delivered in partnership with HDR UK and ADRUK.

26 changes: 13 additions & 13 deletions doc/architecture/Architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,19 +57,19 @@ of the numbered steps in the diagram.
with a comprehensive review process with multiple approvers
before the data is available for download.
1. Egress requests that are approved can be downloaded by Data Egress Managers
and shared with the Researcher who requested the data egress.
There is a configurable limit to the number of downloads which can be made.
and shared with the Researcher who requested the data egress.
There is a configurable limit to the number of downloads which can be made.
1. Audit & Compliance teams get full visibility into all
user activities resulting in AWS API calls through centralised
CloudTrail logs. Additionally, they get breakglass
access to all TRE projects/accounts in the TRE through
a Lambda function role in the Audit account.
user activities resulting in AWS API calls through centralised
CloudTrail logs. Additionally, they get breakglass
access to all TRE projects/accounts in the TRE through
a Lambda function role in the Audit account.

## Component Overview

---

### *AWS Control Tower*
### _AWS Control Tower_

---

Expand All @@ -87,7 +87,7 @@ that will be setup by using the TREEHOOSE solution.

![Multi-account structure](../../res/images/multi-account-setup.png)

### *Service Workbench on AWS Solution*
### _Service Workbench on AWS Solution_

---

Expand All @@ -107,7 +107,7 @@ Key Components :
(more services as desired; this is customisable by providing Service Catalog templates).
- For the secure access environment: AWS AppStream 2.0

### *Data Lake*
### _Data Lake_

---

Expand All @@ -122,7 +122,7 @@ Key Components :

- AWS Lake Formation, Amazon S3, AWS KMS, AWS Glue, Amazon Athena

### *Data Egress Application*
### _Data Egress Application_

---

Expand All @@ -149,7 +149,7 @@ Key Components :
- For the backend: AWS Step Functions, Amazon EFS,
AWS Lambda, Amazon DynamoDB, Amazon SES, Amazon S3, AWS KMS, Amazon SNS, Amazon Cognito, AWS AppSync

### *Workspace backup*
### _Workspace backup_

---

Expand Down Expand Up @@ -189,7 +189,7 @@ Key Components:
- For the backend: AWS Step Functions,
AWS Lambda, Amazon CloudWatch Events, AWS CloudFormation, AWS Backup, Amazon S3

### *Budget controls*
### _Budget controls_

---

Expand All @@ -204,7 +204,7 @@ each TRE project and allows to
- **Respond** : automate actions to avoid over-spending

The component uses [AWS Budgets](https://aws.amazon.com/aws-cost-management/aws-budgets/)
to plan and set expectations around TRE project costs.
to plan and set expectations around TRE project costs.

Key Components:

Expand Down
30 changes: 15 additions & 15 deletions doc/architecture/Cost.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ in the EU West (London) AWS Region is approximately **$30** for TRE account with
Prices are subject to change.
For full details, see the pricing page for each AWS service used in this solution.

> **_NOTE:_** Many AWS Services include a Free Tier – a baseline amount of the service that customers can use at no charge.
> **_NOTE:_** Many AWS Services include a Free Tier – a baseline amount of the service that customers can use at no charge.
> Actual costs may be more or less than the pricing examples provided.
The baseline cost is just for spinning up the infrastructure.
Expand Down Expand Up @@ -42,20 +42,20 @@ solution with the default settings in EU West (Ireland) AWS Region.

An installation of TRE without any workspaces and users.

|AWS Service|Monthly cost|
|----|----|
|Networking services|$11|
|KMS|$6|
|Config|$4|
|CloudTrail|$3.5|
|EC2-other|$1.5|
|DynamoDB|$6|
|Service Catalog|$1|
|Step Functions|$0.09|
|Lambdas|$0.003|
|CloudFront|$0.0002|
|CloudWatch|$0.0003|
|Total|$33.0935|
| AWS Service | Monthly cost |
| ------------------- | ------------ |
| Networking services | $11 |
| KMS | $6 |
| Config | $4 |
| CloudTrail | $3.5 |
| EC2-other | $1.5 |
| DynamoDB | $6 |
| Service Catalog | $1 |
| Step Functions | $0.09 |
| Lambdas | $0.003 |
| CloudFront | $0.0002 |
| CloudWatch | $0.0003 |
| Total | $33.0935 |

### EC2 Usage

Expand Down
1 change: 1 addition & 0 deletions doc/architecture/Design-Considerations.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ TREEHOOSE should make based on their functional and
non-functional requirements.

- Centralise and enable AWS Security services like:

- [AWS Security Hub](https://aws.amazon.com/security-hub/)
- [Amazon GuardDuty](https://aws.amazon.com/guardduty/)
- [Amazon Macie](https://aws.amazon.com/macie/)
Expand Down
Loading

0 comments on commit b2fbe1f

Please sign in to comment.