diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index cf61f919..c30a478b 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,5 +1,5 @@ # Description ----- +--- Declaration : _By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license_ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 16f1b073..2d3ce709 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -83,4 +83,4 @@ repos: # workspace_backup/ has it's own pre-commit-config.yaml # egress_app_* are being moved to their own repos -exclude: '^src/components/(egress_app_backend|egress_app_frontend|workspace_backup)/' +exclude: "^src/components/(egress_app_backend|egress_app_frontend|workspace_backup)/" diff --git a/README.md b/README.md index d775e1fb..995bade5 100644 --- a/README.md +++ b/README.md @@ -13,10 +13,10 @@ Open Original Science Exploration ## What is TREEHOOSE -Trusted Research Environments (TREs) are secure computing environments providing -secure access to sensitive data for research purposes. HDR UK has a set of -[resources and guides](https://www.hdruk.ac.uk/access-to-health-data/trusted-research-environments/) -explaining TREs in more detail in terms of health data research, although TREs +Trusted Research Environments (TREs) are secure computing environments providing +secure access to sensitive data for research purposes. HDR UK has a set of +[resources and guides](https://www.hdruk.ac.uk/access-to-health-data/trusted-research-environments/) +explaining TREs in more detail in terms of health data research, although TREs are not exclusive to health data. TREEHOOSE is an open-source platform for deploying TREs on Amazon Web Services @@ -53,9 +53,9 @@ and discuss future enhancements. ## Use cases -TREEHOOSE was originally developed for use with confidential healthcare data -such as patient electronic health records, but is designed to be used and -customised for all research and analysis disciplines which require access to +TREEHOOSE was originally developed for use with confidential healthcare data +such as patient electronic health records, but is designed to be used and +customised for all research and analysis disciplines which require access to sensitive data. --- @@ -101,7 +101,6 @@ This project is licensed under the [Apache-2.0 License](./LICENSE). ## Funding -This work was funded by UK Research & Innovation Grant Number MC_PC_21032 as +This work was funded by UK Research & Innovation Grant Number MC_PC_21032 as part of Phase 1 of the DARE UK (Data and Analytics Research Environments UK) programme, delivered in partnership with HDR UK and ADRUK. - diff --git a/doc/architecture/Architecture.md b/doc/architecture/Architecture.md index cd228c39..82decfde 100644 --- a/doc/architecture/Architecture.md +++ b/doc/architecture/Architecture.md @@ -57,19 +57,19 @@ of the numbered steps in the diagram. with a comprehensive review process with multiple approvers before the data is available for download. 1. Egress requests that are approved can be downloaded by Data Egress Managers - and shared with the Researcher who requested the data egress. - There is a configurable limit to the number of downloads which can be made. + and shared with the Researcher who requested the data egress. + There is a configurable limit to the number of downloads which can be made. 1. Audit & Compliance teams get full visibility into all - user activities resulting in AWS API calls through centralised - CloudTrail logs. Additionally, they get breakglass - access to all TRE projects/accounts in the TRE through - a Lambda function role in the Audit account. + user activities resulting in AWS API calls through centralised + CloudTrail logs. Additionally, they get breakglass + access to all TRE projects/accounts in the TRE through + a Lambda function role in the Audit account. ## Component Overview --- -### *AWS Control Tower* +### _AWS Control Tower_ --- @@ -87,7 +87,7 @@ that will be setup by using the TREEHOOSE solution. ![Multi-account structure](../../res/images/multi-account-setup.png) -### *Service Workbench on AWS Solution* +### _Service Workbench on AWS Solution_ --- @@ -107,7 +107,7 @@ Key Components : (more services as desired; this is customisable by providing Service Catalog templates). - For the secure access environment: AWS AppStream 2.0 -### *Data Lake* +### _Data Lake_ --- @@ -122,7 +122,7 @@ Key Components : - AWS Lake Formation, Amazon S3, AWS KMS, AWS Glue, Amazon Athena -### *Data Egress Application* +### _Data Egress Application_ --- @@ -149,7 +149,7 @@ Key Components : - For the backend: AWS Step Functions, Amazon EFS, AWS Lambda, Amazon DynamoDB, Amazon SES, Amazon S3, AWS KMS, Amazon SNS, Amazon Cognito, AWS AppSync -### *Workspace backup* +### _Workspace backup_ --- @@ -189,7 +189,7 @@ Key Components: - For the backend: AWS Step Functions, AWS Lambda, Amazon CloudWatch Events, AWS CloudFormation, AWS Backup, Amazon S3 -### *Budget controls* +### _Budget controls_ --- @@ -204,7 +204,7 @@ each TRE project and allows to - **Respond** : automate actions to avoid over-spending The component uses [AWS Budgets](https://aws.amazon.com/aws-cost-management/aws-budgets/) - to plan and set expectations around TRE project costs. +to plan and set expectations around TRE project costs. Key Components: diff --git a/doc/architecture/Cost.md b/doc/architecture/Cost.md index 23c4c016..183c1c95 100644 --- a/doc/architecture/Cost.md +++ b/doc/architecture/Cost.md @@ -8,7 +8,7 @@ in the EU West (London) AWS Region is approximately **$30** for TRE account with Prices are subject to change. For full details, see the pricing page for each AWS service used in this solution. -> **_NOTE:_** Many AWS Services include a Free Tier – a baseline amount of the service that customers can use at no charge. +> **_NOTE:_** Many AWS Services include a Free Tier – a baseline amount of the service that customers can use at no charge. > Actual costs may be more or less than the pricing examples provided. The baseline cost is just for spinning up the infrastructure. @@ -42,20 +42,20 @@ solution with the default settings in EU West (Ireland) AWS Region. An installation of TRE without any workspaces and users. -|AWS Service|Monthly cost| -|----|----| -|Networking services|$11| -|KMS|$6| -|Config|$4| -|CloudTrail|$3.5| -|EC2-other|$1.5| -|DynamoDB|$6| -|Service Catalog|$1| -|Step Functions|$0.09| -|Lambdas|$0.003| -|CloudFront|$0.0002| -|CloudWatch|$0.0003| -|Total|$33.0935| +| AWS Service | Monthly cost | +| ------------------- | ------------ | +| Networking services | $11 | +| KMS | $6 | +| Config | $4 | +| CloudTrail | $3.5 | +| EC2-other | $1.5 | +| DynamoDB | $6 | +| Service Catalog | $1 | +| Step Functions | $0.09 | +| Lambdas | $0.003 | +| CloudFront | $0.0002 | +| CloudWatch | $0.0003 | +| Total | $33.0935 | ### EC2 Usage diff --git a/doc/architecture/Design-Considerations.md b/doc/architecture/Design-Considerations.md index 47e1143f..8ecf170d 100644 --- a/doc/architecture/Design-Considerations.md +++ b/doc/architecture/Design-Considerations.md @@ -42,6 +42,7 @@ TREEHOOSE should make based on their functional and non-functional requirements. - Centralise and enable AWS Security services like: + - [AWS Security Hub](https://aws.amazon.com/security-hub/) - [Amazon GuardDuty](https://aws.amazon.com/guardduty/) - [Amazon Macie](https://aws.amazon.com/macie/) diff --git a/doc/deployment/Prerequisites.md b/doc/deployment/Prerequisites.md index 56d72007..6a597581 100644 --- a/doc/deployment/Prerequisites.md +++ b/doc/deployment/Prerequisites.md @@ -5,20 +5,20 @@ **Total time to configure**: Approximately 105 minutes Ensure all steps below are executed in AWS region: - [London (eu-west-2)](https://eu-west-2.console.aws.amazon.com/). +[London (eu-west-2)](https://eu-west-2.console.aws.amazon.com/). ## Step 1. Setup Accounts **Time to configure**: Approximately 85 minutes Log in to the [AWS Management Console](https://console.aws.amazon.com/) - using your organization's **Management** account and Admin privileges. +using your organization's **Management** account and Admin privileges. The solution must be deployed in a multi-account environment created and - managed using AWS Control Tower. The structure is shown in the image below. +managed using AWS Control Tower. The structure is shown in the image below. ->Optional: If you want to create more environments (e.g. Dev), then ->replicate the structure of the **TRE Environment (Prod)** group in the image and repeat steps 1E and 1F. +> Optional: If you want to create more environments (e.g. Dev), then +> replicate the structure of the **TRE Environment (Prod)** group in the image and repeat steps 1E and 1F. ![Deployment Account Structure](../../res/images/Diagram-DeploymentAccountStructure.png) @@ -27,14 +27,14 @@ The solution must be deployed in a multi-account environment created and **Time to configure**: Approximately 5 minutes - [ ] Go to Service: [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) -- [ ] Select the [*Stacks*](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks) - menu option on the left side -- [ ] Press button: [*Create Stack* with new resources](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks/create/template) -- [ ] Select option *Upload a template file* to upload CloudFormation - template file: [landing zone encryption](../../src/pre_requisites/LandingZoneEncryption-Cfn.yaml) - and press on button *Next* -- [ ] Provide *Stack name*: "ControlTowerSetup-EncryptionKey-ManagementAccount", - press on button *Next* twice and then press on button *Create stack* +- [ ] Select the [_Stacks_](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks) + menu option on the left side +- [ ] Press button: [_Create Stack_ with new resources](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks/create/template) +- [ ] Select option _Upload a template file_ to upload CloudFormation + template file: [landing zone encryption](../../src/pre_requisites/LandingZoneEncryption-Cfn.yaml) + and press on button _Next_ +- [ ] Provide _Stack name_: "ControlTowerSetup-EncryptionKey-ManagementAccount", + press on button _Next_ twice and then press on button _Create stack_ - [ ] Confirm the stack status is "CREATE_COMPLETE" ### Step 1B. Create Landing Zone in AWS Control Tower @@ -42,20 +42,20 @@ The solution must be deployed in a multi-account environment created and **Time to configure**: Approximately 25 minutes - [ ] Go to Service: [AWS Control Tower](https://eu-west-2.console.aws.amazon.com/controltower) -- [ ] Press button: [*Set up landing zone*](https://eu-west-2.console.aws.amazon.com/controltower/home/setup?region=eu-west-2) +- [ ] Press button: [_Set up landing zone_](https://eu-west-2.console.aws.amazon.com/controltower/home/setup?region=eu-west-2) ->Troubleshooting Note: If you see the message "Your AWS Environment is not ready for AWS Control Tower to be set up.", ->please refer to the [troubleshooting guide](../troubleshooting/TroubleshootingRunbook.md#compute-service-limits). +> Troubleshooting Note: If you see the message "Your AWS Environment is not ready for AWS Control Tower to be set up.", +> please refer to the [troubleshooting guide](../troubleshooting/TroubleshootingRunbook.md#compute-service-limits). Leave every option set to default in the Control Tower Landing Zone Setup, except: -- [ ] Step 1 Page - Ensure Home Region is *London*. Enable *Region deny setting*. Add *Additional AWS Regions for governance*: - US East (N. Virginia) - us-east-1 -- [ ] Step 2 Page - For *Additional OU*, add **TRE Solution Prod** +- [ ] Step 1 Page - Ensure Home Region is _London_. Enable _Region deny setting_. Add _Additional AWS Regions for governance_: + US East (N. Virginia) - us-east-1 +- [ ] Step 2 Page - For _Additional OU_, add **TRE Solution Prod** - [ ] Step 3 Page - Provide email addresses for the **Log Archive** and **Audit** accounts. - Enable KMS Encryption and select the **ControlTowerSetup-Landing-Zone** key created in Step 1A + Enable KMS Encryption and select the **ControlTowerSetup-Landing-Zone** key created in Step 1A - [ ] Step 4 Page - Ensure the list matches the diagram above for the **Default Setup** for the 3 initial accounts - plus **TRE Solution Prod**, then press on button *Set up landing zone* + plus **TRE Solution Prod**, then press on button _Set up landing zone_ Wait until the Control Tower Landing Zone Setup completes successfully. @@ -68,16 +68,16 @@ Wait until the Control Tower Landing Zone Setup completes successfully. Ensure the encryption key setup in Step 1A is also automatically applied to all enrolled Control Tower accounts. - [ ] Go to Service: [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) -- [ ] Select the [*StackSets*](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacksets) - menu option on the left side +- [ ] Select the [_StackSets_](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacksets) + menu option on the left side - [ ] If you get message "Enable trusted access with AWS Organizations to use service-managed permissions.", press on - button *Enable trusted access* -- [ ] Press button: [*Create StackSet*](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacksets/create) -- [ ] Select option *Service-managed permissions* -- [ ] Select option *Upload a template file* to upload CloudFormation template file: [landing zone encryption](../../src/pre_requisites/LandingZoneEncryption-Cfn.yaml) and press on button *Next* -- [ ] Provide *StackSet name*: "ControlTowerSetup-EncryptionKey" and press on button *Next* twice -- [ ] For *Deployment targets*, ensure *Automatic deployment* is set to Enabled and select region *eu-west-2 (London)*, - then press on button *Next* and *Submit* + button _Enable trusted access_ +- [ ] Press button: [_Create StackSet_](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacksets/create) +- [ ] Select option _Service-managed permissions_ +- [ ] Select option _Upload a template file_ to upload CloudFormation template file: [landing zone encryption](../../src/pre_requisites/LandingZoneEncryption-Cfn.yaml) and press on button _Next_ +- [ ] Provide _StackSet name_: "ControlTowerSetup-EncryptionKey" and press on button _Next_ twice +- [ ] For _Deployment targets_, ensure _Automatic deployment_ is set to Enabled and select region _eu-west-2 (London)_, + then press on button _Next_ and _Submit_ - [ ] Click on the stack set created and confirm the status is "ACTIVE" ### Step 1D. Change default settings in AWS Control Tower @@ -85,11 +85,11 @@ Ensure the encryption key setup in Step 1A is also automatically applied to all **Time to configure**: Approximately 5 minutes - [ ] Go to Service: - [AWS Control Tower](https://eu-west-2.console.aws.amazon.com/controltower) + [AWS Control Tower](https://eu-west-2.console.aws.amazon.com/controltower) - [ ] Select - [Account factory](https://eu-west-2.console.aws.amazon.com/controltower/home/accountfactory?region=eu-west-2) + [Account factory](https://eu-west-2.console.aws.amazon.com/controltower/home/accountfactory?region=eu-west-2) - [ ] Follow these [instructions](https://docs.aws.amazon.com/controltower/latest/userguide/configure-without-vpc.html#create-without-vpc) - to modify the default network configuration. The intent is to ensure AWS Control Tower doesn't create a VPC for every account created + to modify the default network configuration. The intent is to ensure AWS Control Tower doesn't create a VPC for every account created ### Step 1E. Create Organizational Units @@ -99,13 +99,13 @@ Manually create the remaining OUs in the diagram's **TRE Environment (Prod)** gr - [ ] Go to Service: [AWS Control Tower](https://eu-west-2.console.aws.amazon.com/controltower) - [ ] Select [Organizational units](https://eu-west-2.console.aws.amazon.com/controltower/home/organizationunits?region=eu-west-2) -- [ ] Press button *Add an OU*. Use OU name **TRE Projects Prod** and place it under Parent OU **TRE Solution Prod**. - Wait until registration completes successfully -- [ ] Press button *Add an OU*. Use OU name **TRE Data Prod** and place it under Parent OU **TRE Solution Prod**. Wait - until registration completes successfully +- [ ] Press button _Add an OU_. Use OU name **TRE Projects Prod** and place it under Parent OU **TRE Solution Prod**. + Wait until registration completes successfully +- [ ] Press button _Add an OU_. Use OU name **TRE Data Prod** and place it under Parent OU **TRE Solution Prod**. Wait + until registration completes successfully The current [organizational structure](https://eu-west-2.console.aws.amazon.com/controltower/home/organizationunits?region=eu-west-2) - should match the image below. +should match the image below. ![Control Tower Organizational Structure](../../res/images/Status-OrganizationalStructure.png) @@ -116,24 +116,24 @@ The current [organizational structure](https://eu-west-2.console.aws.amazon.com/ Manually create the remaining Accounts in the diagram's **TRE Environment (Prod)** group. - [ ] Go to Service: - [AWS Control Tower](https://eu-west-2.console.aws.amazon.com/controltower) + [AWS Control Tower](https://eu-west-2.console.aws.amazon.com/controltower) - [ ] Select - [Account factory](https://eu-west-2.console.aws.amazon.com/controltower/home/accountfactory?region=eu-west-2) -- [ ] Press button *Enroll account*. - Set *Display name* to **TRE-Project-1-Prod**. - Place it under *Parent OU* **TRE Projects Prod**. - Provide the required email address and press on button *Enroll account*. - Wait until account creation completes successfully - (check state under [Accounts](https://eu-west-2.console.aws.amazon.com/controltower/home/accounts?region=eu-west-2)) -- [ ] Press button *Enroll account*. - Set *Display name* to **TRE-Datalake-1-Prod**. - Place it under *Parent OU* **TRE Data Prod**. - Provide the required email address and press on button *Enroll account*. - Wait until account creation completes successfully - (check state under [Accounts](https://eu-west-2.console.aws.amazon.com/controltower/home/accounts?region=eu-west-2)) + [Account factory](https://eu-west-2.console.aws.amazon.com/controltower/home/accountfactory?region=eu-west-2) +- [ ] Press button _Enroll account_. + Set _Display name_ to **TRE-Project-1-Prod**. + Place it under _Parent OU_ **TRE Projects Prod**. + Provide the required email address and press on button _Enroll account_. + Wait until account creation completes successfully + (check state under [Accounts](https://eu-west-2.console.aws.amazon.com/controltower/home/accounts?region=eu-west-2)) +- [ ] Press button _Enroll account_. + Set _Display name_ to **TRE-Datalake-1-Prod**. + Place it under _Parent OU_ **TRE Data Prod**. + Provide the required email address and press on button _Enroll account_. + Wait until account creation completes successfully + (check state under [Accounts](https://eu-west-2.console.aws.amazon.com/controltower/home/accounts?region=eu-west-2)) The resulting [organizational structure](https://eu-west-2.console.aws.amazon.com/controltower/home/organizationunits?region=eu-west-2) - should match the image below. +should match the image below. ![Control Tower Organizational Structure with Accounts](../../res/images/Status-OrganizationalStructure-with-Accounts.png) @@ -142,7 +142,7 @@ The resulting [organizational structure](https://eu-west-2.console.aws.amazon.co **Time to configure**: Approximately 5 minutes Log in to the [AWS Management Console](https://console.aws.amazon.com/) using your organization's **Management** account - and Admin privileges. +and Admin privileges. - [ ] Go to Service: [AWS Cost Explorer](https://us-east-1.console.aws.amazon.com/cost-management/home?region=eu-west-2) @@ -157,12 +157,12 @@ If the service is not already initialised, a message will appear like in the ima Required for all accounts created under the **TRE Data Prod** OU. Log in to the [AWS Management Console](https://console.aws.amazon.com/) using your **TRE Datalake 1 Prod** account and - Admin privileges. +Admin privileges. - [ ] Go to Service: [AWS Lake Formation](https://eu-west-2.console.aws.amazon.com/lakeformation/home?region=eu-west-2) A prompt will appear to add the current account as an administrator for Lake Formation. Select the option like in the - image below and press on button *Get started*. +image below and press on button _Get started_. ![Lake Formation](../../res/images/Init-LakeFormation.png) @@ -173,22 +173,22 @@ A prompt will appear to add the current account as an administrator for Lake For Required for all accounts created under the **TRE Projects Prod** OU. Log in to the [AWS Management Console](https://console.aws.amazon.com/) using your **TRE Project 1 Prod** account and - Admin privileges. +Admin privileges. - [ ] Go to Service: [Amazon AppStream 2.0](https://eu-west-2.console.aws.amazon.com/appstream2/home?region=eu-west-2#/) -- [ ] Press on button *Get Started*, then *Skip*. This will create a role in the background which will be used later - in the deployment process. +- [ ] Press on button _Get Started_, then _Skip_. This will create a role in the background which will be used later + in the deployment process. ## Appendix These steps are optional, but recommended for implementing best practices on AWS. - [ ] To secure the **Management** account, please follow these - [instructions](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_best-practices_mgmt-acct.html) + [instructions](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_best-practices_mgmt-acct.html) - [ ] To manage SSO access to your AWS accounts, please follow these - [instructions](https://docs.aws.amazon.com/singlesignon/latest/userguide/manage-your-accounts.html) + [instructions](https://docs.aws.amazon.com/singlesignon/latest/userguide/manage-your-accounts.html) - [ ] To allow IAM users to view Billing on AWS, please follow these - [instructions](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/control-access-billing.html#ControllingAccessWebsite-Activate) + [instructions](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/control-access-billing.html#ControllingAccessWebsite-Activate) - [ ] To learn how to manage AWS Organizations effectively, please follow the - [best practices notes](https://aws.amazon.com/blogs/mt/best-practices-for-organizational-units-with-aws-organizations/) and the - [recommended OU setup](https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/recommended-ous.html) + [best practices notes](https://aws.amazon.com/blogs/mt/best-practices-for-organizational-units-with-aws-organizations/) and the + [recommended OU setup](https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/recommended-ous.html) diff --git a/doc/deployment/README.md b/doc/deployment/README.md index 441faa3a..e4f043f7 100644 --- a/doc/deployment/README.md +++ b/doc/deployment/README.md @@ -16,19 +16,19 @@ Please ensure you switch back to the correct region after the redirect. The TREEHOOSE TRE uses the **ServiceWorkbench** open-source software as the core component and deploys additional add-ons to enable other features. -1) The prerequisites will cover the setup for an AWS Control Tower environment with a multi-account structure. +1. The prerequisites will cover the setup for an AWS Control Tower environment with a multi-account structure. -2) The solution deployment will be done from a pre-configured EC2 instance. +2. The solution deployment will be done from a pre-configured EC2 instance. -3) The following components are part of the TRE solution: +3. The following components are part of the TRE solution: -| Component | Type | Name | Purpose | -|:---------:|:--------|:-------------------------|:-----------------------------------| -| 1 | Mandatory | [ServiceWorkbench](https://aws.amazon.com/government-education/research-and-technical-computing/service-workbench/) | Core engine for TRE. It provides a simple GUI interface for Researchers to provision secure cloud compute resources with data analytics tools. | -| 2 | Mandatory | [Data Lake](https://aws.amazon.com/lake-formation) | A pre-configured data lake to store and manage sensitive datasets. | -| 3 | Optional | Data Egress Application | Provides a GUI-based data egress approval workflow for researchers to take out data from the TRE with the permission of an Information Governance Lead and Research IT Admin | -| 4 | Optional | Project Budget Controls | Allows TRE administrators to set policies to stop new ServiceWorkbench workspace creation when the provided budget limit is reached | -| 5 | Optional | Workspace Backup | Allows TRE administrators to backup and restore ServiceWorkbench workspaces | +| Component | Type | Name | Purpose | +| :-------: | :-------- | :------------------------------------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | Mandatory | [ServiceWorkbench](https://aws.amazon.com/government-education/research-and-technical-computing/service-workbench/) | Core engine for TRE. It provides a simple GUI interface for Researchers to provision secure cloud compute resources with data analytics tools. | +| 2 | Mandatory | [Data Lake](https://aws.amazon.com/lake-formation) | A pre-configured data lake to store and manage sensitive datasets. | +| 3 | Optional | Data Egress Application | Provides a GUI-based data egress approval workflow for researchers to take out data from the TRE with the permission of an Information Governance Lead and Research IT Admin | +| 4 | Optional | Project Budget Controls | Allows TRE administrators to set policies to stop new ServiceWorkbench workspace creation when the provided budget limit is reached | +| 5 | Optional | Workspace Backup | Allows TRE administrators to backup and restore ServiceWorkbench workspaces | ## Tools and Dependencies @@ -57,13 +57,13 @@ Any package management requirements such as availability and security will need --- -| Component |Name | Source code location | -|:---------:|:------------------------|:-----------------------------------| -| 1 | ServiceWorkbench | [Official Open-Source Repository](https://github.com/awslabs/service-workbench-on-aws/releases/tag/v5.2.3) | -| 2 | Data Lake | [CloudFormation Template](../../src/data_lake/DataLake-Cfn.yaml) | -| 3 | Data Egress Application | [CDKv1 Application](./) | -| 4 | Project Budget Controls | [CloudFormation Template](../../src/components/budget_controls/ProjectBudgetControl-Cfn.yaml) | -| 5 | Workspace Backup | [CDKv2 Application](./) | +| Component | Name | Source code location | +| :-------: | :---------------------- | :--------------------------------------------------------------------------------------------------------- | +| 1 | ServiceWorkbench | [Official Open-Source Repository](https://github.com/awslabs/service-workbench-on-aws/releases/tag/v5.2.3) | +| 2 | Data Lake | [CloudFormation Template](../../src/data_lake/DataLake-Cfn.yaml) | +| 3 | Data Egress Application | [CDKv1 Application](./) | +| 4 | Project Budget Controls | [CloudFormation Template](../../src/components/budget_controls/ProjectBudgetControl-Cfn.yaml) | +| 5 | Workspace Backup | [CDKv2 Application](./) | ## [Installation Guide](./INSTALLATION.md) diff --git a/doc/deployment/Step1-SetupDeploymentInstance.md b/doc/deployment/Step1-SetupDeploymentInstance.md index 100f5462..8ecf06f1 100644 --- a/doc/deployment/Step1-SetupDeploymentInstance.md +++ b/doc/deployment/Step1-SetupDeploymentInstance.md @@ -1,7 +1,7 @@ # Deployment Instance Ensure all steps below are executed in AWS region: - [London (eu-west-2)](https://eu-west-2.console.aws.amazon.com/). +[London (eu-west-2)](https://eu-west-2.console.aws.amazon.com/). ## Step 1. Create Deployment Instance @@ -10,22 +10,22 @@ Ensure all steps below are executed in AWS region: ### Step 1A. Create the EC2 instance Log in to the [AWS Management Console](https://console.aws.amazon.com/) using your **TRE Project 1 Prod** - account and Admin privileges. +account and Admin privileges. - [ ] Go to Service: [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) -- [ ] Select the [*Stacks*](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks) - menu option on the left side -- [ ] Press button: [*Create Stack* with new resources](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks/create/template) -- [ ] Select option *Upload a template file* to upload CloudFormation template file: - [deployment instance](../../src/deployment/DeploymentInstance-Cfn.yaml) and press on button *Next* -- [ ] Provide *Stack name*: "TREDeploymentInstance". Adjust default parameter values if needed. - Press on button *Next* twice and then on button *Create stack* +- [ ] Select the [_Stacks_](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks) + menu option on the left side +- [ ] Press button: [_Create Stack_ with new resources](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks/create/template) +- [ ] Select option _Upload a template file_ to upload CloudFormation template file: + [deployment instance](../../src/deployment/DeploymentInstance-Cfn.yaml) and press on button _Next_ +- [ ] Provide _Stack name_: "TREDeploymentInstance". Adjust default parameter values if needed. + Press on button _Next_ twice and then on button _Create stack_ - [ ] Confirm the stack status is "CREATE_COMPLETE" ### Step 1B. Log in to the EC2 instance - [ ] Follow these [instructions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/session-manager.html) - to learn how to connect via SSM to the EC2 instance created in Step 1A. + to learn how to connect via SSM to the EC2 instance created in Step 1A. - [ ] Run the following command to initialise your environment: ```shell @@ -48,10 +48,10 @@ Change `VERSION` if you want to install a different release of TREEHOOSE. If you are familiar with [`git`](https://git-scm.com/) you can clone the repository instead. > Note: The installation process requires 3rd party -open-source libraries to be installed on the -EC2 instance that does not come bundled by default -with the Operating System. \ -The list is as below \ +> open-source libraries to be installed on the +> EC2 instance that does not come bundled by default +> with the Operating System. \ +> The list is as below \ > > - yum-utils, git, jq, and packer from yum repository. > - AWS CLI V2, AWS CDK V1.0 and V2.0 diff --git a/doc/deployment/Step2-DeployServiceWorkbench.md b/doc/deployment/Step2-DeployServiceWorkbench.md index 746d0a68..eb51f7f5 100644 --- a/doc/deployment/Step2-DeployServiceWorkbench.md +++ b/doc/deployment/Step2-DeployServiceWorkbench.md @@ -108,10 +108,10 @@ npm run start-image-builder -- default eu-west-2 default default Log in to the [AWS Management Console](https://console.aws.amazon.com/) using your **TRE Project 1 Prod** account and Admin privileges. - [ ] Go to Service: [AWS AppStream](https://eu-west-2.console.aws.amazon.com/appstream2/home?region=eu-west-2) -- [ ] Select menu option *Images* -> [*Image builder*](https://eu-west-2.console.aws.amazon.com/appstream2/home?region=eu-west-2#/image-builder) -- [ ] Select the image builder created in Part 1 above, then press on button *Connect* -- [ ] On the new tab page that opened in your browser, log in as *Administrator* -- [ ] On the Windows Desktop provided by AppStream, press on the Start button and right click on Windows Powershell to select *Run as administrator* +- [ ] Select menu option _Images_ -> [_Image builder_](https://eu-west-2.console.aws.amazon.com/appstream2/home?region=eu-west-2#/image-builder) +- [ ] Select the image builder created in Part 1 above, then press on button _Connect_ +- [ ] On the new tab page that opened in your browser, log in as _Administrator_ +- [ ] On the Windows Desktop provided by AppStream, press on the Start button and right click on Windows Powershell to select _Run as administrator_ - [ ] Run the following commands in Windows Powershell to create an AppStream image: ```bash @@ -121,7 +121,7 @@ Invoke-WebRequest -Uri https://raw.githubusercontent.com/awslabs/service-workben ``` If the script successfully completes you will be automatically disconnected. -You can view the image created in AppStream's menu option: [*Image registry*](https://eu-west-2.console.aws.amazon.com/appstream2/home?region=eu-west-2#/images). +You can view the image created in AppStream's menu option: [_Image registry_](https://eu-west-2.console.aws.amazon.com/appstream2/home?region=eu-west-2#/images). Wait for the Appstream image status to become `Available` (this will take several minutes). ### Step 2G. Onboard Account @@ -135,11 +135,11 @@ cd /home/ec2-user/tmp/service-workbench-on-aws-5.2.3 ./scripts/get-info.sh treprod ``` -- [ ] Copy the value for *Website URL* and open the browser to access the page -- [ ] Log in using the *rootUserEmail* user set in the configuration file in step 2C. For the first time login, use the temporary password received in an email. - Alternatively, use the password shown in the get-info.sh output under *Temporary Native Pool Password* -- [ ] In the SWB website, select *Accounts* on the left side menu -- [ ] Select *AWS Accounts* and press on button *Add AWS Account*. Some of the settings are adjustable (e.g. time limits, instance type). Below you can find suggested values: +- [ ] Copy the value for _Website URL_ and open the browser to access the page +- [ ] Log in using the _rootUserEmail_ user set in the configuration file in step 2C. For the first time login, use the temporary password received in an email. + Alternatively, use the password shown in the get-info.sh output under _Temporary Native Pool Password_ +- [ ] In the SWB website, select _Accounts_ on the left side menu +- [ ] Select _AWS Accounts_ and press on button _Add AWS Account_. Some of the settings are adjustable (e.g. time limits, instance type). Below you can find suggested values: ```console Account Name: TRE-Project-1-Prod @@ -159,8 +159,8 @@ Note there are 3 options you can select for the AppStream Fleet Type: ON_DEMAND, ALWAYS_ON will reduce the waiting time to establish an AppStream connection, but it will cost more. To learn more about AppStream fleet types, follow this [guide](https://docs.aws.amazon.com/appstream2/latest/developerguide/fleet-type.html). -- [ ] Press on button *Onboard AWS Account* and follow all remaining instructions on the web page. - Do not edit any of the fields in CloudFormation. +- [ ] Press on button _Onboard AWS Account_ and follow all remaining instructions on the web page. + Do not edit any of the fields in CloudFormation. - [ ] Confirm the account status matches the image below ![Onboard AWS Account on SWB](../../res/images/Status-SWBOnboardAccount-Success.png) diff --git a/doc/deployment/Step3-CreateDataLake.md b/doc/deployment/Step3-CreateDataLake.md index c0b89119..70104e30 100644 --- a/doc/deployment/Step3-CreateDataLake.md +++ b/doc/deployment/Step3-CreateDataLake.md @@ -11,22 +11,22 @@ Due to design considerations, each account in the **TRE Data Prod** OU must cont Do not create a Datalake in the main (**TRE Project 1 Prod**) account, you will not be able to use it in the TRE. Log in to the [AWS Management Console](https://console.aws.amazon.com/) using your **TRE Datalake 1 Prod** - account and Admin privileges. +account and Admin privileges. - [ ] Go to Service: [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) -- [ ] Select the [*Stacks*](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks) - menu option on the left side +- [ ] Select the [_Stacks_](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks) + menu option on the left side - [ ] Press button: - [*Create Stack* with new resources](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks/create/template) -- [ ] Select option *Upload a template file* to upload CloudFormation template file: [data lake](../../src/data_lake/DataLake-Cfn.yaml) - and press on button *Next* -- [ ] Provide *Stack name*: "TREDataLake1". Add the parameters required. Press on button *Next* twice - and then press on button *Create stack* - -|Parameter Name|Description|Default value| -|-----------------|-----------|-------------| -|EgressAppAccount|Account number which is hosting the Egress add-on application (Add **TRE Project 1 Prod** account number)|*No default - must be specified*| -|LFDatabaseName|Lake Formation database name that will be created|*No default - must be specified*| + [_Create Stack_ with new resources](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks/create/template) +- [ ] Select option _Upload a template file_ to upload CloudFormation template file: [data lake](../../src/data_lake/DataLake-Cfn.yaml) + and press on button _Next_ +- [ ] Provide _Stack name_: "TREDataLake1". Add the parameters required. Press on button _Next_ twice + and then press on button _Create stack_ + +| Parameter Name | Description | Default value | +| ---------------- | --------------------------------------------------------------------------------------------------------- | -------------------------------- | +| EgressAppAccount | Account number which is hosting the Egress add-on application (Add **TRE Project 1 Prod** account number) | _No default - must be specified_ | +| LFDatabaseName | Lake Formation database name that will be created | _No default - must be specified_ | - [ ] Confirm the stack status is "CREATE_COMPLETE" diff --git a/doc/deployment/Step4-DeployDataEgressApp.md b/doc/deployment/Step4-DeployDataEgressApp.md index a9189307..4eed6285 100644 --- a/doc/deployment/Step4-DeployDataEgressApp.md +++ b/doc/deployment/Step4-DeployDataEgressApp.md @@ -3,7 +3,7 @@ Ensure all steps below are executed in AWS region: [London (eu-west-2)](https://eu-west-2.console.aws.amazon.com/). If this add-on application is added, a researcher can use a GUI-based data egress approval workflow - to take out data from the TRE with the permission of multiple parties (Information Governance Lead, Research IT). +to take out data from the TRE with the permission of multiple parties (Information Governance Lead, Research IT). **Total time to deploy**: Approximately 35 minutes (without prerequisites) @@ -12,19 +12,19 @@ If this add-on application is added, a researcher can use a GUI-based data egres Apply these prerequisites only to accounts that are part of the **TRE Projects Prod** OU. Log in to the [AWS Management Console](https://console.aws.amazon.com/) using your **TRE Project 1 Prod** - account and Admin privileges. +account and Admin privileges. ### Remove email restrictions By default, a new AWS account will be placed in the [Amazon SES](https://aws.amazon.com/ses/) sandbox - which enforces a set of [restrictions](https://docs.aws.amazon.com/ses/latest/dg/request-production-access.html). +which enforces a set of [restrictions](https://docs.aws.amazon.com/ses/latest/dg/request-production-access.html). To enable the egress app to send email notifications to signed-up users (information governance leads, - IT admins and researchers), an admin must manually add each user's email as a verified entity in SES. Following - that, the user must confirm the subscription using a link received in an email. +IT admins and researchers), an admin must manually add each user's email as a verified entity in SES. Following +that, the user must confirm the subscription using a link received in an email. To skip the need to manually add and verify each email address in Amazon SES, you should request production - access to SES by following these [instructions](https://docs.aws.amazon.com/ses/latest/dg/request-production-access.html). +access to SES by following these [instructions](https://docs.aws.amazon.com/ses/latest/dg/request-production-access.html). ## Step 4. Deploy Data Egress App @@ -46,22 +46,22 @@ sudo -iu ec2-user Apply these steps only to accounts that are part of the **TRE Projects Prod** OU. Log in to the [AWS Management Console](https://console.aws.amazon.com/) using your **TRE Project 1 Prod** - account and Admin privileges. +account and Admin privileges. - [ ] Edit file [`cdk.json`](../../src/components/egress_app_backend/cdk.json) in the `src/components/egress_app_backend/` directory (Step 1C). Change the following required - parameters for the CDK backend stack: - -|Parameter Name|Description|Location|AWS Account| -|:-----------------|:-----------|:-------------|:------------| -|swb_egress_store_arn|Provide resource created in Step 2 - S3 Bucket: Egress Store Bucket Arn (`EgressStoreBucket`)|Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Resources* tab for *Stack* "treprod-ldn-pj1-backend" or go to [Amazon S3 Buckets](https://s3.console.aws.amazon.com/s3/buckets?region=eu-west-2)| **TRE Project 1 Prod** account | -|swb_egress_notification_topic|Provide resource created in Step 2 - SNS Topic: Egress Notification Topic Arn (`EgressNotificationTopic`)|Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Resources* tab for *Stack* "treprod-ldn-pj1-backend" or go to [Amazon SNS Topics](https://eu-west-2.console.aws.amazon.com/sns/v3/home?region=eu-west-2#/topics)| **TRE Project 1 Prod** account | -|swb_egress_notification_bucket_arn|Provide resource created in Step 2 - S3 Bucket: Egress Notification Bucket Arn (`EgressNotificationBucket`)|Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Resources* tab for *Stack* "treprod-ldn-pj1-backend" or go to [Amazon S3 Buckets](https://s3.console.aws.amazon.com/s3/buckets?region=eu-west-2)| **TRE Project 1 Prod** account | -|swb_egress_notification_bucket_kms_arn|Provide resource created in Step 2 - KMS Key: Egress Store Encryption Key Arn (`EgressStoreEncryptionKey`)|Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Resources* tab for *Stack* "treprod-ldn-pj1-backend" or go to [AWS KMS Keys](https://eu-west-2.console.aws.amazon.com/kms/home?region=eu-west-2#/kms/keys)| **TRE Project 1 Prod** account | -|swb_egress_store_db_table|Provide resource created in Step 2 - DynamoDB Table: Egress Store Table Arn (`EgressStoreDb`)|Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Resources* tab for *Stack* "treprod-ldn-pj1-backend" or go to [Amazon DynamoDB Tables](https://eu-west-2.console.aws.amazon.com/dynamodbv2/home?region=eu-west-2#tables)| **TRE Project 1 Prod** account | -|datalake_target_bucket_arn|Provide resource created in Step 3 - S3 Bucket: TRE Target Bucket (output `TRETargetBucketArn`)|Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Resources* tab for *Stack* "TREDataLake1" or go to [Amazon S3 Buckets](https://s3.console.aws.amazon.com/s3/buckets?region=eu-west-2)| **TRE Datalake 1 Prod** account | -|datalake_target_bucket_kms_arn|Provide resource created in Step 3 - KMS Key: TRE Target Bucket KMS Key (output `TRETargetBucketKMSKeyArn`)|Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Resources* tab for *Stack* "TREDataLake1" or go to [AWS KMS Keys](https://eu-west-2.console.aws.amazon.com/kms/home?region=eu-west-2#/kms/keys)| **TRE Datalake 1 Prod** account | -|cognito_userpool_domain|Provide name for a new Amazon Cognito domain to be created, e.g. `treprod-pj1-egress-userpool`|To view resources created after deployment of this CDK stack, go to service [Amazon Cognito](https://eu-west-2.console.aws.amazon.com/cognito/home?region=eu-west-2)| **TRE Project 1 Prod** account | -|tre_admin_email_address|Provide a TRE admin email address that will need to be verified after deployment|To view verified identities after deployment of this CDK stack, go to service [Amazon SES](https://eu-west-2.console.aws.amazon.com/ses/home?region=eu-west-2#/verified-identities)| **TRE Project 1 Prod** account | + parameters for the CDK backend stack: + +| Parameter Name | Description | Location | AWS Account | +| :------------------------------------- | :---------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :------------------------------ | +| swb_egress_store_arn | Provide resource created in Step 2 - S3 Bucket: Egress Store Bucket Arn (`EgressStoreBucket`) | Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Resources_ tab for _Stack_ "treprod-ldn-pj1-backend" or go to [Amazon S3 Buckets](https://s3.console.aws.amazon.com/s3/buckets?region=eu-west-2) | **TRE Project 1 Prod** account | +| swb_egress_notification_topic | Provide resource created in Step 2 - SNS Topic: Egress Notification Topic Arn (`EgressNotificationTopic`) | Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Resources_ tab for _Stack_ "treprod-ldn-pj1-backend" or go to [Amazon SNS Topics](https://eu-west-2.console.aws.amazon.com/sns/v3/home?region=eu-west-2#/topics) | **TRE Project 1 Prod** account | +| swb_egress_notification_bucket_arn | Provide resource created in Step 2 - S3 Bucket: Egress Notification Bucket Arn (`EgressNotificationBucket`) | Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Resources_ tab for _Stack_ "treprod-ldn-pj1-backend" or go to [Amazon S3 Buckets](https://s3.console.aws.amazon.com/s3/buckets?region=eu-west-2) | **TRE Project 1 Prod** account | +| swb_egress_notification_bucket_kms_arn | Provide resource created in Step 2 - KMS Key: Egress Store Encryption Key Arn (`EgressStoreEncryptionKey`) | Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Resources_ tab for _Stack_ "treprod-ldn-pj1-backend" or go to [AWS KMS Keys](https://eu-west-2.console.aws.amazon.com/kms/home?region=eu-west-2#/kms/keys) | **TRE Project 1 Prod** account | +| swb_egress_store_db_table | Provide resource created in Step 2 - DynamoDB Table: Egress Store Table Arn (`EgressStoreDb`) | Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Resources_ tab for _Stack_ "treprod-ldn-pj1-backend" or go to [Amazon DynamoDB Tables](https://eu-west-2.console.aws.amazon.com/dynamodbv2/home?region=eu-west-2#tables) | **TRE Project 1 Prod** account | +| datalake_target_bucket_arn | Provide resource created in Step 3 - S3 Bucket: TRE Target Bucket (output `TRETargetBucketArn`) | Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Resources_ tab for _Stack_ "TREDataLake1" or go to [Amazon S3 Buckets](https://s3.console.aws.amazon.com/s3/buckets?region=eu-west-2) | **TRE Datalake 1 Prod** account | +| datalake_target_bucket_kms_arn | Provide resource created in Step 3 - KMS Key: TRE Target Bucket KMS Key (output `TRETargetBucketKMSKeyArn`) | Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Resources_ tab for _Stack_ "TREDataLake1" or go to [AWS KMS Keys](https://eu-west-2.console.aws.amazon.com/kms/home?region=eu-west-2#/kms/keys) | **TRE Datalake 1 Prod** account | +| cognito_userpool_domain | Provide name for a new Amazon Cognito domain to be created, e.g. `treprod-pj1-egress-userpool` | To view resources created after deployment of this CDK stack, go to service [Amazon Cognito](https://eu-west-2.console.aws.amazon.com/cognito/home?region=eu-west-2) | **TRE Project 1 Prod** account | +| tre_admin_email_address | Provide a TRE admin email address that will need to be verified after deployment | To view verified identities after deployment of this CDK stack, go to service [Amazon SES](https://eu-west-2.console.aws.amazon.com/ses/home?region=eu-west-2#/verified-identities) | **TRE Project 1 Prod** account | - [ ] Run the following commands to create an isolated Python environment and deploy the CDK backend stack, replacing `DEPLOYMENT_ACCOUNT` with TRE Project 1 Prod account ID: @@ -82,21 +82,21 @@ cdkv1 deploy Apply these steps only to accounts that are part of the **TRE Projects Prod** OU. Log in to the [AWS Management Console](https://console.aws.amazon.com/) using your **TRE Project 1 Prod** - account and Admin privileges. +account and Admin privileges. - [ ] Edit file [`.env.local`](../../src/components/egress_app_frontend/.env.local) in the `src/components/egress_app_frontend` directory (Step 1C). Change the following required - parameters for the web application: - -|Parameter Name|Description|Location|AWS Account| -|:-----------------|:-----------|:-------------|:------------| -|REACT_APP_APPSYNC_API|Provide resource created in Step 4B - Egress API URL (e.g. ") |Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Outputs* tab for *Stack* "EgressAppBackend" and locate *AppSyncGraphQLURL* or go to [AWS AppSync APIs](https://eu-west-2.console.aws.amazon.com/appsync/home?region=eu-west-2#/apis) -> Select the API created -> *Settings* -> *API URL* | **TRE Project 1 Prod** account | -|REACT_APP_USER_POOL_CLIENT_ID|Provide resource created in Step 4B - App Client Id |Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Outputs* tab for *Stack* "EgressAppBackend" and locate *CognitoAppClientId* or go to [Amazon Cognito Pools](https://eu-west-2.console.aws.amazon.com/cognito/users/?region=eu-west-2#) -> Select the User Pool created -> Under *General settings* -> *App clients* -> *App client id* | **TRE Project 1 Prod** account | -|REACT_APP_USER_POOL_ID|Provide resource created in Step 4B - User Pool Id |Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Outputs* tab for *Stack* "EgressAppBackend" and locate *CognitoUserPoolId* or go to [Amazon Cognito Pools](https://eu-west-2.console.aws.amazon.com/cognito/users/?region=eu-west-2#) -> *General settings* -> *Pool Id* | **TRE Project 1 Prod** account | -|REACT_APP_USER_POOL_DOMAIN|Provide resource created in Step 4B - User Pool Domain Name (e.g. {cognito_userpool_domain}.auth.eu-west-2.amazoncognito.com) |Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Outputs* tab for *Stack* "EgressAppBackend" and locate *CognitoUserPoolDomain* or go to [Amazon Cognito Pools](https://eu-west-2.console.aws.amazon.com/cognito/users/?region=eu-west-2#) -> Under *App integration* -> *Domain name* | **TRE Project 1 Prod** account | -|REACT_APP_REGION|Provide current AWS Region (i.e. eu-west-2)|NA| **TRE Project 1 Prod** account | -|REACT_APP_EGRESS_IG_ROLE|Provide same value as in cdk.json file edited in Step 4B for parameter *egress_reviewer_roles* - value 1|Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Outputs* tab for *Stack* "EgressAppBackend" and locate *EgressReviewerRole1*| **TRE Project 1 Prod** account | -|REACT_APP_EGRESS_RIT_ROLE|Provide same value as in cdk.json file edited in Step 4B for parameter *egress_reviewer_roles* - value 2|Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Outputs* tab for *Stack* "EgressAppBackend" and locate *EgressReviewerRole2*| **TRE Project 1 Prod** account | -|REACT_APP_MAX_DOWNLOADS_ALLOWED|Provide same value as in cdk.json file edited in Step 4B for parameter *max_downloads_allowed* |Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Outputs* tab for *Stack* "EgressAppBackend" and locate *MaxDownloadsAllowed*| **TRE Project 1 Prod** account | + parameters for the web application: + +| Parameter Name | Description | Location | AWS Account | +| :------------------------------ | :---------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------- | +| REACT_APP_APPSYNC_API | Provide resource created in Step 4B - Egress API URL (e.g. ") | Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Outputs_ tab for _Stack_ "EgressAppBackend" and locate _AppSyncGraphQLURL_ or go to [AWS AppSync APIs](https://eu-west-2.console.aws.amazon.com/appsync/home?region=eu-west-2#/apis) -> Select the API created -> _Settings_ -> _API URL_ | **TRE Project 1 Prod** account | +| REACT_APP_USER_POOL_CLIENT_ID | Provide resource created in Step 4B - App Client Id | Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Outputs_ tab for _Stack_ "EgressAppBackend" and locate _CognitoAppClientId_ or go to [Amazon Cognito Pools](https://eu-west-2.console.aws.amazon.com/cognito/users/?region=eu-west-2#) -> Select the User Pool created -> Under _General settings_ -> _App clients_ -> _App client id_ | **TRE Project 1 Prod** account | +| REACT_APP_USER_POOL_ID | Provide resource created in Step 4B - User Pool Id | Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Outputs_ tab for _Stack_ "EgressAppBackend" and locate _CognitoUserPoolId_ or go to [Amazon Cognito Pools](https://eu-west-2.console.aws.amazon.com/cognito/users/?region=eu-west-2#) -> _General settings_ -> _Pool Id_ | **TRE Project 1 Prod** account | +| REACT_APP_USER_POOL_DOMAIN | Provide resource created in Step 4B - User Pool Domain Name (e.g. {cognito_userpool_domain}.auth.eu-west-2.amazoncognito.com) | Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Outputs_ tab for _Stack_ "EgressAppBackend" and locate _CognitoUserPoolDomain_ or go to [Amazon Cognito Pools](https://eu-west-2.console.aws.amazon.com/cognito/users/?region=eu-west-2#) -> Under _App integration_ -> _Domain name_ | **TRE Project 1 Prod** account | +| REACT_APP_REGION | Provide current AWS Region (i.e. eu-west-2) | NA | **TRE Project 1 Prod** account | +| REACT_APP_EGRESS_IG_ROLE | Provide same value as in cdk.json file edited in Step 4B for parameter _egress_reviewer_roles_ - value 1 | Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Outputs_ tab for _Stack_ "EgressAppBackend" and locate _EgressReviewerRole1_ | **TRE Project 1 Prod** account | +| REACT_APP_EGRESS_RIT_ROLE | Provide same value as in cdk.json file edited in Step 4B for parameter _egress_reviewer_roles_ - value 2 | Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Outputs_ tab for _Stack_ "EgressAppBackend" and locate _EgressReviewerRole2_ | **TRE Project 1 Prod** account | +| REACT_APP_MAX_DOWNLOADS_ALLOWED | Provide same value as in cdk.json file edited in Step 4B for parameter _max_downloads_allowed_ | Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Outputs_ tab for _Stack_ "EgressAppBackend" and locate _MaxDownloadsAllowed_ | **TRE Project 1 Prod** account | - [ ] Run the following commands to build the React frontend code: @@ -116,7 +116,7 @@ zip -r ../build.zip ./ ``` - [ ] Run the following commands to copy the React app to S3 and trigger an automatic - deployment to Amplify: + deployment to Amplify: ```bash cd .. @@ -125,14 +125,14 @@ aws s3 cp build.zip s3://{egress web app bucket created in Step 4B} To find the egress web app bucket name, you can check - using the **TRE Project 1 Prod** account - the [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) -*Outputs* tab for *Stack* "EgressAppBackend" and locate *EgressWebAppS3BucketName* +_Outputs_ tab for _Stack_ "EgressAppBackend" and locate _EgressWebAppS3BucketName_ Verify the Amplify app has been updated automatically and the website is reachable: - [ ] Go to Service: [AWS Amplify](https://eu-west-2.console.aws.amazon.com/amplify/home?region=eu-west-2#/) - [ ] Select the app and branch created in Step 4B -- [ ] Confirm the status in the app branch is: *Deployment successfully completed.* -- [ ] Open the URL from *Domain* and confirm a login prompt appears like in the image below +- [ ] Confirm the status in the app branch is: _Deployment successfully completed._ +- [ ] Open the URL from _Domain_ and confirm a login prompt appears like in the image below ![Egress App Website](../../res/images/Status-EgressAppDeployed.png) diff --git a/doc/deployment/Step5-AddProjectBudgetControls.md b/doc/deployment/Step5-AddProjectBudgetControls.md index ac28e5bd..a957ad08 100644 --- a/doc/deployment/Step5-AddProjectBudgetControls.md +++ b/doc/deployment/Step5-AddProjectBudgetControls.md @@ -15,9 +15,9 @@ Log in to the [AWS Management Console](https://console.aws.amazon.com/) using yo ### Step 5A. Locate existing IAM role - [ ] Go to Service: [AWS Identity and Access Management](https://us-east-1.console.aws.amazon.com/iamv2/home#/home) -- [ ] Select the [*Roles*](https://us-east-1.console.aws.amazon.com/iamv2/home#/roles) menu option on the left side -- [ ] Search for *initial-stack* -- [ ] Extract number ID from role: *initial-stack-{number_ID}-xacc-env-mgmt*. This number is required in Step 5C +- [ ] Select the [_Roles_](https://us-east-1.console.aws.amazon.com/iamv2/home#/roles) menu option on the left side +- [ ] Search for _initial-stack_ +- [ ] Extract number ID from role: _initial-stack-{number_ID}-xacc-env-mgmt_. This number is required in Step 5C ### Step 5B - Optional. Locate default SWB Workspace Types @@ -27,9 +27,9 @@ The default behaviour is to restrict all SWB workspace types when the budget con To restrict workspace creation for just the 4 default workspace types SWB creates, follow the instructions below. - [ ] Go to Service: [AWS Service Catalog](https://eu-west-2.console.aws.amazon.com/servicecatalog/home?region=eu-west-2#/home) -- [ ] Select the [*Portfolios*](https://eu-west-2.console.aws.amazon.com/servicecatalog/home?region=eu-west-2#portfolios?activeTab=localAdminPortfolios) - menu option on the left side and click on the local portfolio created during SWB deployment (e.g. treprod-ldn-pj1) -- [ ] Extract just the ID from *prod-ID* from the list which should contain 4 default SWB products. These ID strings are required in Step 5C +- [ ] Select the [_Portfolios_](https://eu-west-2.console.aws.amazon.com/servicecatalog/home?region=eu-west-2#portfolios?activeTab=localAdminPortfolios) + menu option on the left side and click on the local portfolio created during SWB deployment (e.g. treprod-ldn-pj1) +- [ ] Extract just the ID from _prod-ID_ from the list which should contain 4 default SWB products. These ID strings are required in Step 5C For guidance identifying the default workspace types (products) created by SWB, please refer to the image below. @@ -38,27 +38,27 @@ For guidance identifying the default workspace types (products) created by SWB, ### Step 5C. Add Project Budget Control - [ ] Go to Service: [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) -- [ ] Select the [*Stacks*](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks) menu option on the left side -- [ ] Press button: [*Create Stack* with new resources](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks/create/template) -- [ ] Select option *Upload a template file* to upload CloudFormation template file: [project budget controls](../../src/components/budget_controls/ProjectBudgetControl-Cfn.yaml) and press on button *Next* -- [ ] Provide *Stack name*: "TREProject1ProdBudgetControl". Add the parameters required. Press on button *Next* twice and then press on button *Create stack* - -|Parameter Name|Description|Default value| -|:-----------------|:-----------|:-------------| -|BudgetLimit|Estimated annual account spend in USD|*No default - must be specified*| -|BudgetTimeWindow|Choose between an annually or monthly recurring budget|*ANNUALLY*| -|NotificationThresholdActualCost1|Budget threshold percentage for receiving first notification based on actual costs|*80*| -|NotificationThresholdActualCost2|Budget threshold percentage for receiving second notification based on actual costs|*99*| -|NotificationThresholdForecastedCost|Budget threshold percentage for notification based on forecasted costs|*90*| -|ActionThreshold|Budget threshold percentage for stopping new SWB workspace creation based on forecasted costs|*99*| -|BudgetNotifySNSTopicName|The name of the SNS topic whose subscribers (includes TREAdminEmailAddress) receive alerts regarding project budget|*TREProject1Prod-BudgetNotifications*| -|TREAdminEmailAddress|The email address for the TRE admin who will receive alerts regarding project budget|*No default - must be specified*| -|SWBStackID|Specify the ID of existing IAM role initial-stack-ID-xacc-env-mgmt (the number from `CrossAccountEnvMgmtRoleArn`)|*No default - must be specified*| -|ServiceCatalogProductsList|Leave blank if you want to restrict all SWB workspace types. Otherwise, specify the 4 default products ID (prod-{ID}) from the Service Catalog portfolio created by SWB|*""*| +- [ ] Select the [_Stacks_](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks) menu option on the left side +- [ ] Press button: [_Create Stack_ with new resources](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks/create/template) +- [ ] Select option _Upload a template file_ to upload CloudFormation template file: [project budget controls](../../src/components/budget_controls/ProjectBudgetControl-Cfn.yaml) and press on button _Next_ +- [ ] Provide _Stack name_: "TREProject1ProdBudgetControl". Add the parameters required. Press on button _Next_ twice and then press on button _Create stack_ + +| Parameter Name | Description | Default value | +| :---------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------ | +| BudgetLimit | Estimated annual account spend in USD | _No default - must be specified_ | +| BudgetTimeWindow | Choose between an annually or monthly recurring budget | _ANNUALLY_ | +| NotificationThresholdActualCost1 | Budget threshold percentage for receiving first notification based on actual costs | _80_ | +| NotificationThresholdActualCost2 | Budget threshold percentage for receiving second notification based on actual costs | _99_ | +| NotificationThresholdForecastedCost | Budget threshold percentage for notification based on forecasted costs | _90_ | +| ActionThreshold | Budget threshold percentage for stopping new SWB workspace creation based on forecasted costs | _99_ | +| BudgetNotifySNSTopicName | The name of the SNS topic whose subscribers (includes TREAdminEmailAddress) receive alerts regarding project budget | _TREProject1Prod-BudgetNotifications_ | +| TREAdminEmailAddress | The email address for the TRE admin who will receive alerts regarding project budget | _No default - must be specified_ | +| SWBStackID | Specify the ID of existing IAM role initial-stack-ID-xacc-env-mgmt (the number from `CrossAccountEnvMgmtRoleArn`) | _No default - must be specified_ | +| ServiceCatalogProductsList | Leave blank if you want to restrict all SWB workspace types. Otherwise, specify the 4 default products ID (prod-{ID}) from the Service Catalog portfolio created by SWB | _""_ | - [ ] Confirm the stack status is "CREATE_COMPLETE" -When the budget action gets triggered (depends on *AnnualBudgetLimit* and *ActionThreshold*), any allowed user trying to create a new workspace in SWB will see this error message: +When the budget action gets triggered (depends on _AnnualBudgetLimit_ and _ActionThreshold_), any allowed user trying to create a new workspace in SWB will see this error message: ![SWB Workspace Creation Expected Failure](../../res/images/Status-DenySWBWorkspaceCreation.png) @@ -67,18 +67,18 @@ When the budget action gets triggered (depends on *AnnualBudgetLimit* and *Actio Follow these instructions if you need to update the project budget settings. - [ ] Go to Service: [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) -- [ ] Select the [*Stacks*](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks) menu option on the left side -- [ ] Select the stack created in Step 5C and press on button *Update* to adjust the parameters. Please note the changes will take up to 24 hours to reflect in AWS Budgets in terms of alerts and actions. +- [ ] Select the [_Stacks_](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks) menu option on the left side +- [ ] Select the stack created in Step 5C and press on button _Update_ to adjust the parameters. Please note the changes will take up to 24 hours to reflect in AWS Budgets in terms of alerts and actions. ### Step 5E - Optional. Remove Project Budget Control Follow these instructions if you need to remove the project budget control policy after e.g. increasing your budget. - [ ] Go to Service: [AWS Identity and Access Management](https://us-east-1.console.aws.amazon.com/iamv2/home) -- [ ] Select the [*Roles*](https://us-east-1.console.aws.amazon.com/iamv2/home#/roles) menu option on the left side -- [ ] Search for *initial-stack* to locate the role with name *initial-stack-ID-xacc-env-mgmt* (ID is a number) -- [ ] Click on the role identified above and select the policy that has *-DenyCreateWorkSpacePolicy-* in its name -- [ ] Use the *Remove* button on the right side to detach the policy from the role +- [ ] Select the [_Roles_](https://us-east-1.console.aws.amazon.com/iamv2/home#/roles) menu option on the left side +- [ ] Search for _initial-stack_ to locate the role with name _initial-stack-ID-xacc-env-mgmt_ (ID is a number) +- [ ] Click on the role identified above and select the policy that has _-DenyCreateWorkSpacePolicy-_ in its name +- [ ] Use the _Remove_ button on the right side to detach the policy from the role When the budget control policy has been removed, any allowed user trying to create a new workspace in SWB will see the Available status message: @@ -89,5 +89,5 @@ When the budget control policy has been removed, any allowed user trying to crea For further customisations, additional notes can be found below. - [ ] AWS Budgets limits action types to applying IAM or SCP policies or stopping EC2 and RDS instances. - For more flexibility, [Amazon SNS can be integrated with AWS Lambda](https://docs.aws.amazon.com/sns/latest/dg/sns-lambda-as-subscriber.html) - functions to support custom actions when a budget notification is sent. + For more flexibility, [Amazon SNS can be integrated with AWS Lambda](https://docs.aws.amazon.com/sns/latest/dg/sns-lambda-as-subscriber.html) + functions to support custom actions when a budget notification is sent. diff --git a/doc/deployment/Step6-DeployBackupComponent.md b/doc/deployment/Step6-DeployBackupComponent.md index b8d77dde..51ce5756 100644 --- a/doc/deployment/Step6-DeployBackupComponent.md +++ b/doc/deployment/Step6-DeployBackupComponent.md @@ -24,7 +24,7 @@ steps for installing the TRE setup and the source code already downloaded. ### Step 6A. Log in to the EC2 instance - [ ] Follow these [instructions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/session-manager.html) - to learn how to connect via SSM to the EC2 instance created in Step 1. + to learn how to connect via SSM to the EC2 instance created in Step 1. - [ ] Run the following commands to initialise your environment: ```console @@ -37,12 +37,12 @@ alias cdk2="npx aws-cdk@2.x" - [ ] Update the [`cdk.json`](../../src/components/workspace_backup/cdk.json) file to meet your requirements. Below are the options available -|Parameter|Usage|Default Value|Notes| -|---------|-----|-------------|-----| -|frequency|controls the frequency of taking backups. possible values are daily and hourly|daily|For SageMaker notebooks, the instance needs to be running for the backup to happen. This might not always be the case so a backup is also taken when instance is started| -|retention_period|defines the number of days backup is retained before deletion|180|| -|move_to_cold_storage_after|defines the number of days after which the backup is archived to cold storage|90|AWS Backup for EBS currently does not support this. Uses S3 lifecycle policy for SageMaker backups| -|sagemaker_enable_selfservice_restore|controls if SageMaker notebook user is able to restore backed-up files|true|| +| Parameter | Usage | Default Value | Notes | +| ------------------------------------ | ------------------------------------------------------------------------------ | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| frequency | controls the frequency of taking backups. possible values are daily and hourly | daily | For SageMaker notebooks, the instance needs to be running for the backup to happen. This might not always be the case so a backup is also taken when instance is started | +| retention_period | defines the number of days backup is retained before deletion | 180 | | +| move_to_cold_storage_after | defines the number of days after which the backup is archived to cold storage | 90 | AWS Backup for EBS currently does not support this. Uses S3 lifecycle policy for SageMaker backups | +| sagemaker_enable_selfservice_restore | controls if SageMaker notebook user is able to restore backed-up files | true | | - [ ] Change directory to root folder of the backup component code: `src/components/workspace_backup/`. @@ -75,7 +75,7 @@ this functionality. ### Validating EBS volume backups 1. Login into the SWB (Service Workbench) Web App as an Admin. - Navigate to `Workspaces`. In the list of + Navigate to `Workspaces`. In the list of all Available and Stopped workspaces check for a workspace based on EC2 and has been created using a configuration that has backup enabled. @@ -83,9 +83,9 @@ this functionality. 1. Once such workspace is identified, click on `View Detail` , navigate to `CloudFormation Output` and note `Ec2WorkspaceInstanceId` - *Note : You need to wait for the backup window to pass before + _Note : You need to wait for the backup window to pass before performing below steps. For Daily backup frequency wait for a day - and for hourly back frequency wait for atleast a few hours to pass.* + and for hourly back frequency wait for atleast a few hours to pass._ 1. Log in to the [AWS Management Console](https://console.aws.amazon.com/) of the TRE account using Admin privileges. @@ -104,7 +104,7 @@ this functionality. ### Validating SageMaker Notebook backups 1. Login into the SWB Web App as an Admin. - Navigate to `Workspaces`. In the list of + Navigate to `Workspaces`. In the list of all Available and Stopped workspaces check for a workspace based on SageMaker Notebook and has been created using a configuration that has backup enabled. @@ -112,9 +112,9 @@ this functionality. 1. Once such workspace is identified, click on `View Detail` , navigate to `CloudFormation Output` and note `NotebookInstanceName` - *Note : You need to wait for the backup window to pass before + _Note : You need to wait for the backup window to pass before performing below steps Or the workspace to be restarted once - after files have been created in it.* + after files have been created in it._ 1. Log in to the [AWS Management Console](https://console.aws.amazon.com/) of the TRE account using Admin privileges. Navigate to Amazon S3 console. diff --git a/doc/deployment/Testing.md b/doc/deployment/Testing.md index c0a9674c..29199034 100644 --- a/doc/deployment/Testing.md +++ b/doc/deployment/Testing.md @@ -18,7 +18,6 @@ Perform the following tests for Linux and Windows workspaces - A researcher can copy files to the egress folder, and can request egress - ### Egress (administrators) - A notification is received when a researcher requests egress @@ -27,4 +26,3 @@ Perform the following tests for Linux and Windows workspaces - If an IG approves a request a notification is sent to the IT administrator - The IT admin can approve or deny an egress request - If the IT admin approves the request the IG can download the egressed files in the egress app - diff --git a/doc/operations/BackupComponent.md b/doc/operations/BackupComponent.md index 53449f2a..3913a6ac 100644 --- a/doc/operations/BackupComponent.md +++ b/doc/operations/BackupComponent.md @@ -75,7 +75,7 @@ EBS by replacing it on the Workspace. 1. Click on the `Volume ID` which should take you to `volumes` page, select the volume, click on `Action`->`Detach volume`, confirm `Detach` - *Note : At this point if the detached volume is no longer required it should be deleted.* + _Note : At this point if the detached volume is no longer required it should be deleted._ 1. Navigate to [AWS Backup](https://eu-west-2.console.aws.amazon.com/backup/home?region=eu-west-2#/backupvaults) console, navigate to `Backup vaults`, click on the vault name. @@ -94,14 +94,14 @@ EBS by replacing it on the Workspace. that will have the snapshot ID, and other configurations. Fill the details as per below table and click on `Restore backup` button. - |Parameter|Value| - |----|----| - |Resource Type| Specify EBS volume.| - |Volume type| Select either the original volume type as noted earlier or a more appropriate type based on cost and performance requirements| - |Size| Select equivalent size of the backed up EBS volume as noted earlier.| - |IOS| 300/3000 - Baseline of 3 iops per GiB with a minimum of 100 IOPS, burstable to 3000 IOPS.| - |Availability Zone| Select the Availability Zone for the EC2 instance as noted in previous step| - |Restore role| Select Default role| + | Parameter | Value | + | ----------------- | ----------------------------------------------------------------------------------------------------------------------------- | + | Resource Type | Specify EBS volume. | + | Volume type | Select either the original volume type as noted earlier or a more appropriate type based on cost and performance requirements | + | Size | Select equivalent size of the backed up EBS volume as noted earlier. | + | IOS | 300/3000 - Baseline of 3 iops per GiB with a minimum of 100 IOPS, burstable to 3000 IOPS. | + | Availability Zone | Select the Availability Zone for the EC2 instance as noted in previous step | + | Restore role | Select Default role | 1. This will take you to restored jobs screen. The restored backup job will appear under Restore jobs in the the @@ -196,33 +196,26 @@ files the restoration work needs to be undertaken by the TRE Admin. Replace values for `BACKUP_BUCKET` and `NOTEBOOK_NAME` variables. ```json - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": ["s3:ListBucket"], - "Resource": [ - "arn:aws:s3:::BACKUP_BUCKET/NOTEBOOK_NAME" - ], - "Condition": { - "StringEquals": { - "s3:prefix": "NOTEBOOK_NAME" - } - } - }, - { - "Effect": "Allow", - "Action": [ - "s3:GetObject", - "s3:GetObjectVersion" - ], - "Resource": [ - "arn:aws:s3:::BACKUP_BUCKET/NOTEBOOK_NAME/*" - ], - }, - ], - } + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": ["s3:ListBucket"], + "Resource": ["arn:aws:s3:::BACKUP_BUCKET/NOTEBOOK_NAME"], + "Condition": { + "StringEquals": { + "s3:prefix": "NOTEBOOK_NAME" + } + } + }, + { + "Effect": "Allow", + "Action": ["s3:GetObject", "s3:GetObjectVersion"], + "Resource": ["arn:aws:s3:::BACKUP_BUCKET/NOTEBOOK_NAME/*"] + } + ] + } ``` 1. Click on Review Policy. Provide policy name as `sagemaker-restore-policy-for-NOTEBOOK_NAME` diff --git a/doc/operations/Costs.md b/doc/operations/Costs.md index d9e187a2..ba27535c 100644 --- a/doc/operations/Costs.md +++ b/doc/operations/Costs.md @@ -1,36 +1,36 @@ # Guidance - View Costs One TRE project is defined using one AWS account (e.g. **TRE Project 1 Prod**) as described in the - [deployment Prerequisites section](../deployment/Prerequisites.md). The costs associated with - the TRE project are the costs incurred when using this dedicated AWS account. +[deployment Prerequisites section](../deployment/Prerequisites.md). The costs associated with +the TRE project are the costs incurred when using this dedicated AWS account. The AWS account (e.g. **TRE Project 1 Prod**) will host only one SWB instance with one SWB project. - We can view the total cost associated with a TRE project by checking the costs incurred when using - the AWS account (e.g. **TRE Project 1 Prod**). +We can view the total cost associated with a TRE project by checking the costs incurred when using +the AWS account (e.g. **TRE Project 1 Prod**). There will also be costs for the data lake account (e.g. **TRE Datalake 1 Prod**) linked to a - TRE project, depending on how much data is stored and processed in it. +TRE project, depending on how much data is stored and processed in it. ## AWS Cost Explorer [AWS Cost Explorer](https://aws.amazon.com/aws-cost-management/aws-cost-explorer/) provides an - easy-to-use interface to visualise costs. An example of daily costs associated with - the all the accounts used for a TRE deployment is provided below. +easy-to-use interface to visualise costs. An example of daily costs associated with +the all the accounts used for a TRE deployment is provided below. > Note: The Management account should already have AWS Cost Explorer enabled as per the instructions > in [Step 2 in the deployment prerequisites](../deployment/Prerequisites.md) section. Log in to the [AWS Management Console](https://console.aws.amazon.com/) - using your organization's **Management** account and Admin privileges. +using your organization's **Management** account and Admin privileges. - [ ] Go to Service: - [AWS Cost Management](https://us-east-1.console.aws.amazon.com/cost-management/home?region=eu-west-2) -- [ ] Select the *Cost Explorer* menu option on the left side + [AWS Cost Management](https://us-east-1.console.aws.amazon.com/cost-management/home?region=eu-west-2) +- [ ] Select the _Cost Explorer_ menu option on the left side ![Guidance CostExplorer TRE](../../res/images/Guidance-CostExplorer-TRE.png) AWS Cost Explorer provides several filters to generate a different view of the costs data. An example - of using the *Service* filter is shown below. +of using the _Service_ filter is shown below. ![Guidance CostExplorer TRE - Service filter](../../res/images/Guidance-CostExplorer-TRE-ServicesView.png) @@ -41,54 +41,54 @@ You can also use AWS Cost Explorer in any of the member AWS accounts as shown be > to enable it. This process can take around 24 hours to complete. Log in to the [AWS Management Console](https://console.aws.amazon.com/) - using your **TRE Project 1 Prod** account and Admin privileges. +using your **TRE Project 1 Prod** account and Admin privileges. - [ ] Go to Service: - [AWS Cost Management](https://us-east-1.console.aws.amazon.com/cost-management/home?region=eu-west-2) -- [ ] Select the *Cost Explorer* menu option on the left side + [AWS Cost Management](https://us-east-1.console.aws.amazon.com/cost-management/home?region=eu-west-2) +- [ ] Select the _Cost Explorer_ menu option on the left side ![Guidance CostExplorer TRE Project](../../res/images/Guidance-CostExplorer-TRE-Project.png) ## Cost Control To set a budget limit for a TRE project and prevent further resource creation when the limit - is reached, deploy the optional - [Project Budget Controls component](../deployment/Step5-AddProjectBudgetControls.md). +is reached, deploy the optional +[Project Budget Controls component](../deployment/Step5-AddProjectBudgetControls.md). ## Service Workbench This section applies only to the payer account, e.g. your organization's **Management** account. An IT admin can add tags to workspace configurations in SWB. These custom tags can be - seen in [AWS Billing](https://aws.amazon.com/aws-cost-management/aws-billing/). You can activate the - associated [cost allocation tag](https://us-east-1.console.aws.amazon.com/billing/home?region=eu-west-2#/tags) - to track costs for the group of resources (workspaces) created with that tag. An example of using the tag - that shows if a workspace has backup enabled or not can be seen below. +seen in [AWS Billing](https://aws.amazon.com/aws-cost-management/aws-billing/). You can activate the +associated [cost allocation tag](https://us-east-1.console.aws.amazon.com/billing/home?region=eu-west-2#/tags) +to track costs for the group of resources (workspaces) created with that tag. An example of using the tag +that shows if a workspace has backup enabled or not can be seen below. ![Guidance Cost Explorer TRE Cost Allocation Tag](../../res/images/Guidance-CostExplorer-TRE-CostAllocationTag.png) To learn more about cost allocation tags, view the - [AWS Billing user guide](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html). +[AWS Billing user guide](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html). ## Management Account The **Management** account is defined in the [deployment prerequisites](../deployment/Prerequisites.md) section. From a cost perspective, it has the responsibilities of a payer account and as such, it is responsible for - also paying all the charges that are accrued by the member accounts (e.g. **TRE Project 1 Prod**). +also paying all the charges that are accrued by the member accounts (e.g. **TRE Project 1 Prod**). This account can use - [consolidated billing](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/consolidated-billing.html) - to view AWS charges incurred by all other accounts in the AWS Organizations structure. +[consolidated billing](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/consolidated-billing.html) +to view AWS charges incurred by all other accounts in the AWS Organizations structure. To view costs for a particular category (e.g. combined charges for a TRE project covered by **TRE Project 1 Prod** - and **TRE Datalake 1 Prod**, data storage costs in **TRE Datalake 1 Prod**, etc.), you can create cost - categories using these - [instructions](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/manage-cost-categories.html). +and **TRE Datalake 1 Prod**, data storage costs in **TRE Datalake 1 Prod**, etc.), you can create cost +categories using these +[instructions](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/manage-cost-categories.html). If you need to get a comprehensive set of cost and usage data, you can generate reports using the - [AWS Cost and Usage Reports](https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur.html) service. +[AWS Cost and Usage Reports](https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur.html) service. ## Learn More To learn about all the services that help with cost management, view the - [AWS Cost Management user guide](https://docs.aws.amazon.com/cost-management/latest/userguide/what-is-costmanagement.html). +[AWS Cost Management user guide](https://docs.aws.amazon.com/cost-management/latest/userguide/what-is-costmanagement.html). diff --git a/doc/operations/DataLake.md b/doc/operations/DataLake.md index f5ca0ba8..fb4e5326 100644 --- a/doc/operations/DataLake.md +++ b/doc/operations/DataLake.md @@ -1,68 +1,68 @@ # Data Lake The TREEHOOSE TRE solution contains a data lake that needs to be deployed - with every TRE project. The data lake is used to store and manage datasets in an - environment which can be governed securely. +with every TRE project. The data lake is used to store and manage datasets in an +environment which can be governed securely. ## Data Ingestion [Amazon S3](https://aws.amazon.com/s3/) provides a cost-effective way to store - large amounts of data in an easy-to-use secure way. The data lake deployed - uses S3 buckets to store datasets and results for research activities. +large amounts of data in an easy-to-use secure way. The data lake deployed +uses S3 buckets to store datasets and results for research activities. Data Lake Managers are responsible for uploading datasets into the data lake - and ensuring the data has been processed and it's ready for research activities. +and ensuring the data has been processed and it's ready for research activities. The **TRE Source Bucket** is created after - [deploying the data lake](../deployment/Step3-CreateDataLake.md) and is registered - with [AWS Lake Formation](https://aws.amazon.com/lake-formation/). [Data Lake Managers](../architecture/User-Personas.md#Data-Lake-Manager) - can use this encrypted bucket to store data suitable as input for research activities. +[deploying the data lake](../deployment/Step3-CreateDataLake.md) and is registered +with [AWS Lake Formation](https://aws.amazon.com/lake-formation/). [Data Lake Managers](../architecture/User-Personas.md#Data-Lake-Manager) +can use this encrypted bucket to store data suitable as input for research activities. Data Lake Managers can upload data to the **TRE Source Bucket** using - [multiple methods](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html) - that are available for S3. When uploading large files to S3, they can leverage available - [optimisation methods](https://aws.amazon.com/premiumsupport/knowledge-center/s3-upload-large-files/). +[multiple methods](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html) +that are available for S3. When uploading large files to S3, they can leverage available +[optimisation methods](https://aws.amazon.com/premiumsupport/knowledge-center/s3-upload-large-files/). Optionally, Data Lake Managers: -* can apply ETL jobs on S3 buckets using [AWS Glue](https://aws.amazon.com/glue/) to prepare - or transform the data before they store it in the **TRE Source Bucket** to share it with researchers. -* can query S3 data buckets like **TRE Source Bucket** using [Amazon Athena](https://aws.amazon.com/athena) - and store the results in another S3 bucket e.g. **TRE Analyst Bucket** +- can apply ETL jobs on S3 buckets using [AWS Glue](https://aws.amazon.com/glue/) to prepare + or transform the data before they store it in the **TRE Source Bucket** to share it with researchers. +- can query S3 data buckets like **TRE Source Bucket** using [Amazon Athena](https://aws.amazon.com/athena) + and store the results in another S3 bucket e.g. **TRE Analyst Bucket** To learn more about Amazon S3, check the - [official user guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html). +[official user guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html). To learn more about AWS Glue, check the - [official documentation](https://docs.aws.amazon.com/glue/index.html). +[official documentation](https://docs.aws.amazon.com/glue/index.html). To learn more about Amazon Athena, check the - [official user guide](https://docs.aws.amazon.com/athena/latest/ug/getting-started.html). +[official user guide](https://docs.aws.amazon.com/athena/latest/ug/getting-started.html). ## Data Usage Before proceeding with the instructions below, review the - [TRE user personas](../architecture/User-Personas.md). +[TRE user personas](../architecture/User-Personas.md). ### Step 1. Register Data Study in Service Workbench User personas involved: IT Administrators and Data Lake Managers As an IT Administrator, log in to the SWB Web App deployed in - [Step 2 in the deployment guide](../deployment/Step2-DeployServiceWorkbench.md) using - a SWB user account of type *admin*. Select menu option *Data Sources* on the left side - and press on button *Register Studies*. +[Step 2 in the deployment guide](../deployment/Step2-DeployServiceWorkbench.md) using +a SWB user account of type _admin_. Select menu option _Data Sources_ on the left side +and press on button _Register Studies_. First provide the AWS account where the data resides. This will be the data lake account, - e.g. **TRE Datalake 1 Prod**. +e.g. **TRE Datalake 1 Prod**. ![Data Usage - 1](../../res/images/data_lake/DataUsage-1.png) Provide the **TRE Source Bucket** bucket name and the KMS key used to encrypt it. In addition, - specify the path to the dataset (in section *Studies*) that you wish to make available to researchers. - If you need help finding these details, use the **TRE Datalake 1 Prod** account and check the - [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) - *Resources* tab for *Stack* "TREDataLake1". +specify the path to the dataset (in section _Studies_) that you wish to make available to researchers. +If you need help finding these details, use the **TRE Datalake 1 Prod** account and check the +[AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) +_Resources_ tab for _Stack_ "TREDataLake1". ![Data Usage - 2](../../res/images/data_lake/DataUsage-2.png) @@ -75,16 +75,16 @@ SWB will generate a CloudFormation template as shown below. ![Data Usage - 4](../../res/images/data_lake/DataUsage-4.png) As an IT administrator, you will first need approval from a Data Lake Manager - to run the CloudFormation template that will make the data study available in SWB. - If approval has been given, log in to the - [AWS Management Console](https://console.aws.amazon.com/) - using your **TRE Datalake 1 Prod** account and Admin privileges. Then use the - *Create Stack* button from SWB to load the page below. +to run the CloudFormation template that will make the data study available in SWB. +If approval has been given, log in to the +[AWS Management Console](https://console.aws.amazon.com/) +using your **TRE Datalake 1 Prod** account and Admin privileges. Then use the +_Create Stack_ button from SWB to load the page below. ![Data Usage - 5](../../res/images/data_lake/DataUsage-5.png) Run the stack and ensure it completes successfully. Then return to SWB and test the - data study connection and ensure the status matches the image below. +data study connection and ensure the status matches the image below. ![Data Usage - 6](../../res/images/data_lake/DataUsage-6.png) @@ -93,14 +93,14 @@ Run the stack and ensure it completes successfully. Then return to SWB and test User personas involved: IT Administrators and Data Lake Managers As an IT Administrator, log in to the SWB Web App deployed in - [Step 2 in the deployment guide](../deployment/Step2-DeployServiceWorkbench.md) using - a SWB user account of type *admin*. Select menu option *Studies* on the left side - and select tab *Organization* to locate the previously created data study, e.g. *Dataset1*. +[Step 2 in the deployment guide](../deployment/Step2-DeployServiceWorkbench.md) using +a SWB user account of type _admin_. Select menu option _Studies_ on the left side +and select tab _Organization_ to locate the previously created data study, e.g. _Dataset1_. The data study admin can edit the permissions for the data study and in the example - below the admin provided a researcher with read-only access to *Dataset1*. Note the - admin should change permissions only if they got approval from the Data Lake Managers - in charge of that data study. +below the admin provided a researcher with read-only access to _Dataset1_. Note the +admin should change permissions only if they got approval from the Data Lake Managers +in charge of that data study. ![Data Usage - 7](../../res/images/data_lake/DataUsage-7.png) @@ -108,18 +108,18 @@ The data study admin can edit the permissions for the data study and in the exam User personas involved: Researchers -The researcher who was provided with read-only access to the data study, e.g. *Dataset1*, - can now create a workspace with this study attached. As a Researcher, select the study - like in the image below. +The researcher who was provided with read-only access to the data study, e.g. _Dataset1_, +can now create a workspace with this study attached. As a Researcher, select the study +like in the image below. ![Data Usage - 8](../../res/images/data_lake/DataUsage-8.png) Then create a SWB workspace with a suitable configuration that is available. An example - is provided below. +is provided below. ![Data Usage - 9](../../res/images/data_lake/DataUsage-9.png) Open a connection to the workspace to check the study data is mounted and the shared - dataset can be accessed. +dataset can be accessed. ![Data Usage - 10](../../res/images/data_lake/DataUsage-10.png) diff --git a/doc/operations/ServiceWorkbench.md b/doc/operations/ServiceWorkbench.md index 5ca2b763..2bd7bcdc 100644 --- a/doc/operations/ServiceWorkbench.md +++ b/doc/operations/ServiceWorkbench.md @@ -1,30 +1,30 @@ # Service Workbench [Service Workbench on AWS](https://aws.amazon.com/government-education/research-and-technical-computing/service-workbench/) - is a cloud solution that enables IT teams to provide secure, repeatable, and federated control of access to data, tooling, - and compute power that researchers need. +is a cloud solution that enables IT teams to provide secure, repeatable, and federated control of access to data, tooling, +and compute power that researchers need. ## Design Constraints As described in the [TREEHOOSE TRE architecture](../architecture/Architecture.md), one TRE Project AWS account - (e.g. **TRE Project 1 Prod**) will host only one SWB instance. +(e.g. **TRE Project 1 Prod**) will host only one SWB instance. For a SWB instance (web application), a TRE admin should **create only one SWB Project** which will represent the TRE project - whose boundaries are defined by the AWS account (e.g. **TRE Project 1 Prod**) where all project related resources are deployed. +whose boundaries are defined by the AWS account (e.g. **TRE Project 1 Prod**) where all project related resources are deployed. ## Create the SWB Project To create the SWB project required to perform any tasks in SWB (create workspaces, register data studies, etc.), follow the - [instructions](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf) - from the official SWB user guide, pages 17-18. +[instructions](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf) +from the official SWB user guide, pages 17-18. ## Create SWB Users ### Roles To learn about the predefined user roles available in SWB, follow the - [guidance](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf) - from the official SWB user guide, page 16. +[guidance](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf) +from the official SWB user guide, page 16. ### Users @@ -33,61 +33,61 @@ Follow the instructions below to create Cognito users who can authenticate to th Apply these steps only to accounts that are part of the **TRE Projects Prod** OU. Log in to the [AWS Management Console](https://console.aws.amazon.com/) using your **TRE Project 1 Prod** - account and Admin privileges. +account and Admin privileges. In Cognito (default IdP): - [ ] Go to Service: [AWS Cognito](https://eu-west-2.console.aws.amazon.com/cognito/home?region=eu-west-2) -- [ ] Select [*Manage User Pools*](https://eu-west-2.console.aws.amazon.com/cognito/users/?region=eu-west-2) -- [ ] Select the *User Pool* for SWB called e.g. *treprod-pj1-userPool* (based on the SWB config file provided during - [deployment Step 2C](../../deployment/Step2-DeployServiceWorkbench.md)) -- [ ] Use button *Create user* to create a SWB user +- [ ] Select [_Manage User Pools_](https://eu-west-2.console.aws.amazon.com/cognito/users/?region=eu-west-2) +- [ ] Select the _User Pool_ for SWB called e.g. _treprod-pj1-userPool_ (based on the SWB config file provided during + [deployment Step 2C](../../deployment/Step2-DeployServiceWorkbench.md)) +- [ ] Use button _Create user_ to create a SWB user In SWB: - [ ] Log in to SWB using the root account (based on the SWB config file provided during - [deployment Step 2C](../../deployment/Step2-DeployServiceWorkbench.md)) -- [ ] Go to menu option *Users* -- [ ] For each user previously created in Cognito use buttons *Detail* -> *Activate User* to activate them to allow login -- [ ] For each user previously created in Cognito use buttons *Detail* -> *Edit* to select a suitable *User Role* for them + [deployment Step 2C](../../deployment/Step2-DeployServiceWorkbench.md)) +- [ ] Go to menu option _Users_ +- [ ] For each user previously created in Cognito use buttons _Detail_ -> _Activate User_ to activate them to allow login +- [ ] For each user previously created in Cognito use buttons _Detail_ -> _Edit_ to select a suitable _User Role_ for them While SWB does support other identity providers, only Cognito is in scope for the TREEHOOSE TRE solution at this time. To learn more about SWB IdP support, check the - [official SWB configuration guide](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_Configuration_Guide.pdf). +[official SWB configuration guide](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_Configuration_Guide.pdf). ### Add Users to Project To add users to the SWB project, follow the - [instructions](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf) - from the official SWB user guide, page 18 - section *Adding a User to a Project*. +[instructions](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf) +from the official SWB user guide, page 18 - section _Adding a User to a Project_. ## Register Data Studies An admin in SWB can register data studies and assign permissions to those studies. A researcher can then - attach the read-only data studies to a compute workspace to perform their research activities. +attach the read-only data studies to a compute workspace to perform their research activities. To learn how to register external data studies, follow the - [instructions](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf) - from the official SWB user guide, pages 26-28. +[instructions](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf) +from the official SWB user guide, pages 26-28. To learn how to set permissions for data studies, follow the - [instructions](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf) - from the official SWB user guide, page 25. +[instructions](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf) +from the official SWB user guide, page 25. For known issues with registered data studies in SWB, please refer to the - [troubleshooting guidance](../troubleshooting/TroubleshootingRunbook.md), section *External Data Studies*. +[troubleshooting guidance](../troubleshooting/TroubleshootingRunbook.md), section _External Data Studies_. ## Create Workspaces An admin in SWB can define workspace types and configurations. A researcher can then use those configurations to - create compute workspaces to perform research activities. +create compute workspaces to perform research activities. To learn about workspaces, follow the - [instructions](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf) - from the official SWB user guide, pages 11-14. +[instructions](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf) +from the official SWB user guide, pages 11-14. ## Learn More For more SWB guidance, please consult the - [official SWB user guide](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf) - or ask questions in the [project's Issues page on GitHub](https://github.com/awslabs/service-workbench-on-aws/issues). +[official SWB user guide](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf) +or ask questions in the [project's Issues page on GitHub](https://github.com/awslabs/service-workbench-on-aws/issues). diff --git a/doc/operations/egress_app/ConfigurationGuide.md b/doc/operations/egress_app/ConfigurationGuide.md index cec1653d..6f442f01 100644 --- a/doc/operations/egress_app/ConfigurationGuide.md +++ b/doc/operations/egress_app/ConfigurationGuide.md @@ -5,16 +5,16 @@ Ensure all steps below are executed in AWS region: [London (eu-west-2)](https:// **Total time to configure**: Approximately 30 minutes To use the Egress Application add-on after deployment, a TRE admin must additionally configure user accounts - and enable the information governance leads to review the data in the egress requests triggered by researchers. +and enable the information governance leads to review the data in the egress requests triggered by researchers. There are 2 types of users involved in the Egress App workflow: -1) Information Governance Leads +1. Information Governance Leads - uses a SWB user account of type researcher to create a new workspace to view the contents of the data egress request - uses an Egress App user account to provide the 1st approval for a data egress request -1) Research IT Admins +1. Research IT Admins - uses a SWB user account of type admin to validate that an information governance lead is authorised to grant their approval - uses an Egress App user account to provide the 2nd approval for data egress requests, only after the 1st approval has been granted @@ -26,7 +26,7 @@ There are 2 types of users involved in the Egress App workflow: Apply these steps only to accounts that are part of the **TRE Projects Prod** OU. Log in to the [AWS Management Console](https://console.aws.amazon.com/) using your **TRE Project 1 Prod** - account and Admin privileges. +account and Admin privileges. ### Part 1. SWB @@ -35,101 +35,100 @@ Create Cognito users who can authenticate to the SWB website In Cognito: - [ ] Go to Service: [AWS Cognito](https://eu-west-2.console.aws.amazon.com/cognito/home?region=eu-west-2) -- [ ] Select [*Manage User Pools*](https://eu-west-2.console.aws.amazon.com/cognito/users/?region=eu-west-2) -- [ ] Select the *User Pool* for SWB called e.g. *treprod-pj1-userPool* (based on the SWB config file provided - during [deployment Step 2C](../../deployment/Step2-DeployServiceWorkbench.md)) -- [ ] Under *General settings*, select menu option *Users and groups* -- [ ] Use button *Create user* to create at least one SWB user to represent an Information Governance Lead -- [ ] Use button *Create user* to create at least one SWB user to represent a Research IT Admin +- [ ] Select [_Manage User Pools_](https://eu-west-2.console.aws.amazon.com/cognito/users/?region=eu-west-2) +- [ ] Select the _User Pool_ for SWB called e.g. _treprod-pj1-userPool_ (based on the SWB config file provided + during [deployment Step 2C](../../deployment/Step2-DeployServiceWorkbench.md)) +- [ ] Under _General settings_, select menu option _Users and groups_ +- [ ] Use button _Create user_ to create at least one SWB user to represent an Information Governance Lead +- [ ] Use button _Create user_ to create at least one SWB user to represent a Research IT Admin In SWB: - [ ] Log in to SWB using the root account (based on the SWB config file provided during - [deployment Step 2C](../../deployment/Step2-DeployServiceWorkbench.md)) -- [ ] Go to menu option *Users* -- [ ] For each user previously created in Cognito use buttons *Detail* -> *Activate User* to activate them to allow login -- [ ] For each user of type Information Governance Lead previously created in Cognito use buttons *Detail* -> *Edit* - to select *User Role*: researcher -- [ ] For each user of type Research IT Admin previously created in Cognito use buttons *Detail* -> *Edit* to select *User Role*: admin + [deployment Step 2C](../../deployment/Step2-DeployServiceWorkbench.md)) +- [ ] Go to menu option _Users_ +- [ ] For each user previously created in Cognito use buttons _Detail_ -> _Activate User_ to activate them to allow login +- [ ] For each user of type Information Governance Lead previously created in Cognito use buttons _Detail_ -> _Edit_ + to select _User Role_: researcher +- [ ] For each user of type Research IT Admin previously created in Cognito use buttons _Detail_ -> _Edit_ to select _User Role_: admin ### Part 2. Egress App Create Cognito users who can authenticate to the Egress App website - [ ] Go to Service: [AWS Cognito](https://eu-west-2.console.aws.amazon.com/cognito/home?region=eu-west-2) -- [ ] Select [*Manage User Pools*](https://eu-west-2.console.aws.amazon.com/cognito/users/?region=eu-west-2) -- [ ] Select the *User Pool* for the Egress App called e.g. *EgressUserPool** -- [ ] Under *General settings*, select menu option *Users and groups*. View tab *Groups*, you should see the 2 types - of reviewers: InformationGovernance and TREAdmin -- [ ] Select tab *Users* -- [ ] Use button *Create user* to create at least one user of type Information Governance Lead. After creating the user, - select it and use button *Add to group* to add it to the InformationGovernance group -- [ ] Use button *Create user* to create at least one user of type Research IT Admin. After creating the user, select - it and use button *Add to group* to add it to the TREAdmin group +- [ ] Select [_Manage User Pools_](https://eu-west-2.console.aws.amazon.com/cognito/users/?region=eu-west-2) +- [ ] Select the _User Pool_ for the Egress App called e.g. \*EgressUserPool\*\* +- [ ] Under _General settings_, select menu option _Users and groups_. View tab _Groups_, you should see the 2 types + of reviewers: InformationGovernance and TREAdmin +- [ ] Select tab _Users_ +- [ ] Use button _Create user_ to create at least one user of type Information Governance Lead. After creating the user, + select it and use button _Add to group_ to add it to the InformationGovernance group +- [ ] Use button _Create user_ to create at least one user of type Research IT Admin. After creating the user, select + it and use button _Add to group_ to add it to the TREAdmin group ### Part 3. Notifications Ensure Egress App users receive notifications when egress requests are triggered. - [ ] Go to Service: [Amazon SNS](https://eu-west-2.console.aws.amazon.com/sns/v3/home?region=eu-west-2#/homepage) -- [ ] Select [*Topics*](https://eu-west-2.console.aws.amazon.com/sns/v3/home?region=eu-west-2#/topics) +- [ ] Select [_Topics_](https://eu-west-2.console.aws.amazon.com/sns/v3/home?region=eu-west-2#/topics) -Select topic *Information-Governance-Notifications*: +Select topic _Information-Governance-Notifications_: -- [ ] For each user of type Information Governance Lead created in Part 2, use button *Create subscription* and - select *Protocol*: Email. In *Endpoint* provide the user's email address. Submit and the user will receive an email to confirm the subscription +- [ ] For each user of type Information Governance Lead created in Part 2, use button _Create subscription_ and + select _Protocol_: Email. In _Endpoint_ provide the user's email address. Submit and the user will receive an email to confirm the subscription -Select topic *ResearchIT-Notifications*: +Select topic _ResearchIT-Notifications_: -- [ ] For each user of type Research IT Admin created in Part 2, use button *Create subscription* and select - *Protocol*: Email. In *Endpoint* provide the user's email address. Submit and the user will receive an email to confirm the subscription +- [ ] For each user of type Research IT Admin created in Part 2, use button _Create subscription_ and select + _Protocol_: Email. In _Endpoint_ provide the user's email address. Submit and the user will receive an email to confirm the subscription If you are able to create your own email groups (e.g. Outlook groups) that can receive external emails you can create a subscription for the group email address instead, and manage subscriptions outside of AWS. - ## Step 2. Setup Access for Information Governance Leads **Time to configure**: Approximately 15 minutes Before proceeding with the steps below, please be aware of the following limitations in SWB: -- The *Admin* permissions for a registered data study should always have at least one SWB user - (of type Information Governance Lead) listed -- Do not add the same SWB user under both *Admin* and *Read Only* permissions for a registered data study as - it leads to permission errors when viewing the study +- The _Admin_ permissions for a registered data study should always have at least one SWB user + (of type Information Governance Lead) listed +- Do not add the same SWB user under both _Admin_ and _Read Only_ permissions for a registered data study as + it leads to permission errors when viewing the study - Once an external S3 bucket is registered as data study, it cannot be updated nor deleted. Please view the - [troubleshooting guide](../../troubleshooting/TroubleshootingRunbook.md#external-data-studies) for instructions - on how to handle this situation + [troubleshooting guide](../../troubleshooting/TroubleshootingRunbook.md#external-data-studies) for instructions + on how to handle this situation Follow the instructions below to provide SWB users of type Information Governance Lead with access to view - the egress requests made by researchers in SWB. +the egress requests made by researchers in SWB. In SWB: - [ ] Log in to SWB using an admin account -- [ ] Select menu option *Data Sources* and use button *Register Studies*, press *Next* +- [ ] Select menu option _Data Sources_ and use button _Register Studies_, press _Next_ - [ ] Provide the following details: -|Parameter Name|Description| -|:-----------------|:-----------| -|AWS Account ID|Provide the ID for the **TRE Project 1 Prod** account| -|Account Name|Provide e.g. **TRE Project 1 Prod**| -|Region|Provide the AWS Region where the TRE project was deployed, e.g. *eu-west-2* for London | -|Bucket Name|Provide the name of the S3 bucket created by the Egress App as staging area for egress requests, e.g. EgressStagingArea... ; Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Resources* tab for *Stack* "EgressAppBackend" to locate the S3 bucket| -|Bucket Region|Provide the AWS Region where the TRE project was deployed, e.g. *eu-west-2* for London | -|Bucket Default Encryption|Select *SSE-KMS* and provide the KMS Arn used for the Egress App staging area S3 Bucket's encryption key, e.g. EgressS3Key...; Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Resources* tab for *Stack* "EgressAppBackend" to locate the KMS key| +| Parameter Name | Description | +| :------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| AWS Account ID | Provide the ID for the **TRE Project 1 Prod** account | +| Account Name | Provide e.g. **TRE Project 1 Prod** | +| Region | Provide the AWS Region where the TRE project was deployed, e.g. _eu-west-2_ for London | +| Bucket Name | Provide the name of the S3 bucket created by the Egress App as staging area for egress requests, e.g. EgressStagingArea... ; Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Resources_ tab for _Stack_ "EgressAppBackend" to locate the S3 bucket | +| Bucket Region | Provide the AWS Region where the TRE project was deployed, e.g. _eu-west-2_ for London | +| Bucket Default Encryption | Select _SSE-KMS_ and provide the KMS Arn used for the Egress App staging area S3 Bucket's encryption key, e.g. EgressS3Key...; Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Resources_ tab for _Stack_ "EgressAppBackend" to locate the KMS key | - [ ] Press on button "Add Study": -|Parameter Name|Description| -|:-----------------|:-----------| -|Study Id & Study Name|Provide a name for the directory that will be mounted to a workspace, e.g. EgressRequests| -|Study Folder|Provide just the forward slash character */* . This will allow the Information Governance Lead to view all of the TRE project's egress requests for all workspaces| -|Project|Provide the associated SWB project e.g. *TREProject1Prod*| -|Type|Select *Organization Study*| -|Access|Select *Read Only*| -|Study KMS Arn|Provide the KMS Arn used for the Egress App staging area S3 Bucket's encryption key, e.g. EgressS3Key...; Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) *Resources* tab for *Stack* "EgressAppBackend" to locate the KMS key| -|Admin|Select an existing SWB user account for the Information Governance Lead, as created in Step 1 - Part 1| +| Parameter Name | Description | +| :-------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Study Id & Study Name | Provide a name for the directory that will be mounted to a workspace, e.g. EgressRequests | +| Study Folder | Provide just the forward slash character _/_ . This will allow the Information Governance Lead to view all of the TRE project's egress requests for all workspaces | +| Project | Provide the associated SWB project e.g. _TREProject1Prod_ | +| Type | Select _Organization Study_ | +| Access | Select _Read Only_ | +| Study KMS Arn | Provide the KMS Arn used for the Egress App staging area S3 Bucket's encryption key, e.g. EgressS3Key...; Check [AWS CloudFormation](https://eu-west-2.console.aws.amazon.com/cloudformation/home?region=eu-west-2#/) _Resources_ tab for _Stack_ "EgressAppBackend" to locate the KMS key | +| Admin | Select an existing SWB user account for the Information Governance Lead, as created in Step 1 - Part 1 | A message like this should appear after registering the study: @@ -140,4 +139,4 @@ A message like this should appear after successfully attaching the study: ![Successfully attached the staging area for egress requests](../../../res/images/Status-SetupDataStudy-EgressRequests.png) To view data egress requests, a SWB user of type Information Governance Lead needs to launch a SWB Workspace - with this Study attached to it. +with this Study attached to it. diff --git a/doc/operations/egress_app/UserGuide.md b/doc/operations/egress_app/UserGuide.md index f6147b24..4c23ef44 100644 --- a/doc/operations/egress_app/UserGuide.md +++ b/doc/operations/egress_app/UserGuide.md @@ -3,11 +3,11 @@ There are 2 types of reviewer personas involved in the Egress App workflow: 1. Information Governance Leads - - uses a SWB user account of type researcher to create a new workspace to view the contents of the data egress request - - uses an Egress App user account to provide the 1st approval for a data egress request + - uses a SWB user account of type researcher to create a new workspace to view the contents of the data egress request + - uses an Egress App user account to provide the 1st approval for a data egress request 1. Research IT Admins - - uses a SWB user account of type admin to validate that an information governance lead is authorised to grant their approval - - uses an Egress App user account to provide the 2nd approval for data egress requests, only after the 1st approval has been granted + - uses a SWB user account of type admin to validate that an information governance lead is authorised to grant their approval + - uses an Egress App user account to provide the 2nd approval for data egress requests, only after the 1st approval has been granted ## Workflow @@ -15,63 +15,63 @@ There are 2 types of reviewer personas involved in the Egress App workflow: 1. In the SWB Web App, the Researcher creates and uses a workspace to carry out research activities. - ![Egress App Workflow - 1](../../../res/images/egress_app/UserGuide-Workflow-1.png) + ![Egress App Workflow - 1](../../../res/images/egress_app/UserGuide-Workflow-1.png) 1. In the SWB Web App, the Researcher adds data to the egress store and submits an egress request from the SWB website. - ![Egress App Workflow - 2](../../../res/images/egress_app/UserGuide-Workflow-2.png) + ![Egress App Workflow - 2](../../../res/images/egress_app/UserGuide-Workflow-2.png) - ![Egress App Workflow - 3](../../../res/images/egress_app/UserGuide-Workflow-3.png) + ![Egress App Workflow - 3](../../../res/images/egress_app/UserGuide-Workflow-3.png) 1. The Information Governance Lead receives an email notification about the egress request triggered by the Researcher. - ![Egress App Workflow - 4](../../../res/images/egress_app/UserGuide-Workflow-4.png) + ![Egress App Workflow - 4](../../../res/images/egress_app/UserGuide-Workflow-4.png) 1. In the Egress Web App, the Information Governance Lead views the egress request searching for the ID from the email notification. Search box is present at the bottom left of the data table. - ![Egress App Workflow - 5](../../../res/images/egress_app/UserGuide-Workflow-5.png) + ![Egress App Workflow - 5](../../../res/images/egress_app/UserGuide-Workflow-5.png) 1. In the SWB Web App, the Information Governance Lead creates/uses a workspace with the study created in the [Configuration Guide](./ConfigurationGuide.md) - ![Egress App Workflow - 6](../../../res/images/egress_app/UserGuide-Workflow-6.png) + ![Egress App Workflow - 6](../../../res/images/egress_app/UserGuide-Workflow-6.png) - ![Egress App Workflow - 7](../../../res/images/egress_app/UserGuide-Workflow-7.png) + ![Egress App Workflow - 7](../../../res/images/egress_app/UserGuide-Workflow-7.png) 1. In the SWB Web App, the Information Governance Lead connects to the workspace to view the egress request contents searching for the ID from the email notification. - ![Egress App Workflow - 8](../../../res/images/egress_app/UserGuide-Workflow-8.png) + ![Egress App Workflow - 8](../../../res/images/egress_app/UserGuide-Workflow-8.png) 1. In the Egress Web App, the Information Governance Lead approves or rejects the egress request with the ID from the email notification. - ![Egress App Workflow - 9](../../../res/images/egress_app/UserGuide-Workflow-9.png) + ![Egress App Workflow - 9](../../../res/images/egress_app/UserGuide-Workflow-9.png) 1. If the egress request was rejected by the Information Governance Lead, the Researcher and Information Governance Lead will be notified by email. - **The steps below happen only if the egress request was approved by the Information Governance Lead.** - ![Egress App Workflow - 10](../../../res/images/egress_app/UserGuide-Workflow-10.png) + **The steps below happen only if the egress request was approved by the Information Governance Lead.** + ![Egress App Workflow - 10](../../../res/images/egress_app/UserGuide-Workflow-10.png) 1. If the egress request was approved by the Information Governance Lead, a Research IT Admin will be notified by email. - ![Egress App Workflow - 11](../../../res/images/egress_app/UserGuide-Workflow-11.png) + ![Egress App Workflow - 11](../../../res/images/egress_app/UserGuide-Workflow-11.png) 1. The Research IT Admin needs to check that the Information Governance Lead approval was granted by a person with the required clearance. 1. In the Egress Web App, the Research IT Admin approves or rejects the egress request with the ID from the email notification. - ![Egress App Workflow - 12](../../../res/images/egress_app/UserGuide-Workflow-12.png) + ![Egress App Workflow - 12](../../../res/images/egress_app/UserGuide-Workflow-12.png) 1. If the egress request was rejected by the Research IT Admin, the Researcher and Information Governance Lead will be notified by email. - **The steps below happen only if the egress request was approved by the Research IT Admin.** - ![Egress App Workflow - 13](../../../res/images/egress_app/UserGuide-Workflow-13.png) + **The steps below happen only if the egress request was approved by the Research IT Admin.** + ![Egress App Workflow - 13](../../../res/images/egress_app/UserGuide-Workflow-13.png) 1. If the egress request was approved by the Research IT Admin, the Researcher and Information Governance Lead will be notified by email. - ![Egress App Workflow - 14](../../../res/images/egress_app/UserGuide-Workflow-14.png) + ![Egress App Workflow - 14](../../../res/images/egress_app/UserGuide-Workflow-14.png) 1. In the Egress Web App, the Information Governance Lead can download the egress request file contents. The number of times a download is allowed is constrained by a parameter set during the Egress App deployment stage. - ![Egress App Workflow - 15](../../../res/images/egress_app/UserGuide-Workflow-15.png) + ![Egress App Workflow - 15](../../../res/images/egress_app/UserGuide-Workflow-15.png) 1. The Information Governance Lead is responsible for sending the egress request file contents to the Researcher via an approved manual method such as email. diff --git a/doc/security/SecurityControls.md b/doc/security/SecurityControls.md index ec02a396..4e33a7a8 100644 --- a/doc/security/SecurityControls.md +++ b/doc/security/SecurityControls.md @@ -1,87 +1,87 @@ # Security Controls To learn about AWS general security best practices, please review the - [Security Pillar](https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/security.html) - of the AWS Well-Architected Framework. +[Security Pillar](https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/security.html) +of the AWS Well-Architected Framework. ## Shared Responsibility Model The TREEHOOSE TRE solution runs on the AWS cloud where security and - compliance is a shared responsibility between AWS and the customer. +compliance is a shared responsibility between AWS and the customer. To learn more about the shared responsibility model, please review - the [official guidance](https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/shared-responsibility.html). +the [official guidance](https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/shared-responsibility.html). ## Data Classification Data classification is determined by a number of participants, including, - but not limited to, 3rd party dataset providers and principal investigators - who lead research projects. Classification is not a property of a dataset, - because a dataset’s sensitivity also depends on the data it can be combined - with and its use. +but not limited to, 3rd party dataset providers and principal investigators +who lead research projects. Classification is not a property of a dataset, +because a dataset’s sensitivity also depends on the data it can be combined +with and its use. - Under-classification can lead to legal and financial sanction - and the loss of the social license to operate in the research community. + and the loss of the social license to operate in the research community. - Over-classification can result in lost researcher productivity, - a loss of scientific engagement and increases data risk by encouraging workaround breach. + a loss of scientific engagement and increases data risk by encouraging workaround breach. The TREEHOOSE TRE solution is designed to be flexible and does not enforce - any tagging mechanism or workflows specific to data classification tiers. - It is the responsibility of the TRE administrator and the Data managers to - configure the TRE environment and datasets to meet their data classification requirements. +any tagging mechanism or workflows specific to data classification tiers. +It is the responsibility of the TRE administrator and the Data managers to +configure the TRE environment and datasets to meet their data classification requirements. ## Data Ingress The datasets used in research activities are stored in a data lake provided as part - of the TREEHOOSE TRE solution with encryption applied by default at rest and in transit. - The TRE administrator can configure custom access and security policies to - manage the data lake in a secure manner. +of the TREEHOOSE TRE solution with encryption applied by default at rest and in transit. +The TRE administrator can configure custom access and security policies to +manage the data lake in a secure manner. Researchers can only be provided with read-only access to a dataset. - If a dataset contains sensitive information or needs changes, a Data Manager - will need to process the data in the data lake before allowing access to it for research activities. +If a dataset contains sensitive information or needs changes, a Data Manager +will need to process the data in the data lake before allowing access to it for research activities. The data is protected at rest by [AWS KMS](https://aws.amazon.com/kms/) encryption keys - and in transit by TLS. +and in transit by TLS. ## Data Egress Researchers can extract data from the TRE environment only by undergoing a 2-stage approval process - involving Information Governance Leads and Research IT Admins. An add-on application enables a - secure data egress workflow that will move the encrypted data securely out of a workspace and - into the data lake provided as part of the TREEHOOSE TRE solution. +involving Information Governance Leads and Research IT Admins. An add-on application enables a +secure data egress workflow that will move the encrypted data securely out of a workspace and +into the data lake provided as part of the TREEHOOSE TRE solution. After all the approvals are given, Information Governance Leads will be able to download - the egressed data for a maximum number of times as set by the TRE administrator. +the egressed data for a maximum number of times as set by the TRE administrator. The data is protected at rest by [AWS KMS](https://aws.amazon.com/kms/) encryption keys - and in transit by TLS. +and in transit by TLS. ## Software Ingress The compute workspaces used by researchers are network-isolated and do not have Internet access. - The TRE administrator needs to configure a suitable workspace type or attach a storage location - with the required packages before any workspaces are created. +The TRE administrator needs to configure a suitable workspace type or attach a storage location +with the required packages before any workspaces are created. ## User Access Management The TRE administrators will use AWS SSO to access the AWS console. The 2 web applications part of the TREEHOOSE TRE solution use [Amazon Cognito](https://aws.amazon.com/cognito/) - as an identity provider and have a set of predefined roles that are assigned to users by a TRE administrator. - Some permissions can also be assigned to control user access to compute workspaces and datasets. +as an identity provider and have a set of predefined roles that are assigned to users by a TRE administrator. +Some permissions can also be assigned to control user access to compute workspaces and datasets. The SWB web application allows a new user to sign-up, but they will need to contact a TRE administrator - who will activate their account in SWB and assign an appropriate role. The Egress Addon web application - does not support user sign-up and requires a TRE administrator to manage user accounts in Amazon Cognito. +who will activate their account in SWB and assign an appropriate role. The Egress Addon web application +does not support user sign-up and requires a TRE administrator to manage user accounts in Amazon Cognito. ## User Device Management All resources part of the TREEHOOSE TRE solution are stored on the AWS cloud. Users will need an Internet - or dedicated network connection to access the TRE-related web applications and resources, the web-based - virtual remote desktop application or the AWS web console for administration purposes. +or dedicated network connection to access the TRE-related web applications and resources, the web-based +virtual remote desktop application or the AWS web console for administration purposes. ## Infrastructure Security The TREEHOOSE TRE solution is deployed in a multi-account structure managed using - [AWS Control Tower](https://aws.amazon.com/controltower/) with security guardrails and region protection. +[AWS Control Tower](https://aws.amazon.com/controltower/) with security guardrails and region protection. diff --git a/doc/troubleshooting/TroubleshootingRunbook.md b/doc/troubleshooting/TroubleshootingRunbook.md index 66ef808d..6fc736bd 100644 --- a/doc/troubleshooting/TroubleshootingRunbook.md +++ b/doc/troubleshooting/TroubleshootingRunbook.md @@ -7,26 +7,26 @@ The guidance below should help troubleshoot common issues identified in the TRE ### Compute service limits Some services such as AWS Control Tower or Amazon AppStream 2.0 have default service limits - which apply to new AWS accounts. To raise default service limits, you can try [launching and terminating at - least one Amazon EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/LaunchingAndUsingInstances.html) - (size of instance and run duration prior to termination do not matter). +which apply to new AWS accounts. To raise default service limits, you can try [launching and terminating at +least one Amazon EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/LaunchingAndUsingInstances.html) +(size of instance and run duration prior to termination do not matter). ## ServiceWorkbench (SWB) For general guidance on operating the ServiceWorkbench web application, please visit - the official [documentation page](https://github.com/awslabs/service-workbench-on-aws/tree/v5.2.3/docs). +the official [documentation page](https://github.com/awslabs/service-workbench-on-aws/tree/v5.2.3/docs). For known issues affecting the ServiceWorkbench codebase, please visit the official - [GitHub Issues page](https://github.com/awslabs/service-workbench-on-aws/issues). +[GitHub Issues page](https://github.com/awslabs/service-workbench-on-aws/issues). ### AppStream Due to limitations from the EC2 service, there is a 60 seconds time window to connect to a SWB workspace - via an AppStream connection. The process to connect usually takes longer than that if the AppStream instance - needs to warm up, so after the AppStream connection becomes available for use, you can click again on the - *Use this SSH Key* button in the SWB website which will trigger another 60 seconds countdown. Please note - you cannot connect to the workspace from the AppStream instance if the countdown has expired, so make sure - to return to the SWB website and repeat the *Use this SSH Key* procedure as many times as needed. +via an AppStream connection. The process to connect usually takes longer than that if the AppStream instance +needs to warm up, so after the AppStream connection becomes available for use, you can click again on the +_Use this SSH Key_ button in the SWB website which will trigger another 60 seconds countdown. Please note +you cannot connect to the workspace from the AppStream instance if the countdown has expired, so make sure +to return to the SWB website and repeat the _Use this SSH Key_ procedure as many times as needed. ![Troubleshooting SWB - 1A](../../res/images/troubleshooting_runbook/Troubleshooting-SWB-1A.png) @@ -38,38 +38,38 @@ you will receive this error when opening a SSH connection using Putty: ### Project Budget Controls When the optional component for project budget controls is deployed, a budget threshold will be set. When that - limit is reached, an automatic policy will be applied which will prevent new SWB workspace creation. The error - message a researcher or admin will see in SWB is shown below. +limit is reached, an automatic policy will be applied which will prevent new SWB workspace creation. The error +message a researcher or admin will see in SWB is shown below. ![Troubleshooting SWB - 2](../../res/images/troubleshooting_runbook/Troubleshooting-SWB-2.png) To allow more workspaces to be created, a TRE administrator needs to: 1. Adjust the budget amount or threshold (Follow [Step 5D](../deployment/Step5-AddProjectBudgetControls.md) - in the deployment guide for instructions) + in the deployment guide for instructions) 1. Remove the IAM policy attached to the SWB IAM role controlling the workspace creation workflow - (Follow [Step 5E](../deployment/Step5-AddProjectBudgetControls.md) in the deployment guide for instructions) + (Follow [Step 5E](../deployment/Step5-AddProjectBudgetControls.md) in the deployment guide for instructions) ### External Data Studies Limitations in SWB prevent changes to registered data studies. In some cases, selecting certain permissions could render the data study unusable. To be able to register the study again using different settings, a TRE administrator - will need to delete some SWB resources in the AWS Management Console. **There is a risk of breaking the SWB environment - if the wrong resources are deleted, therefore caution is advised.** +will need to delete some SWB resources in the AWS Management Console. **There is a risk of breaking the SWB environment +if the wrong resources are deleted, therefore caution is advised.** To remove the data study with issues from the SWB website, follow the instructions below. Log in to the [AWS Management Console](https://console.aws.amazon.com/) using your **TRE Project 1 Prod** account and Admin privileges. - [ ] Go to Service: [Amazon DynamoDB](https://eu-west-2.console.aws.amazon.com/dynamodbv2/home?region=eu-west-2#service) -- [ ] Select the [*Tables*](https://eu-west-2.console.aws.amazon.com/dynamodbv2/home?region=eu-west-2#tables) menu option on the left side -- [ ] Locate the data study (with the name provided when it was registered in SWB) in DynamoDB table *treprod-ldn-pj1-Studies* (please note - the table name differs based on the SWB config file details provided in [deployment Step 2C](../deployment/Step2-DeployServiceWorkbench.md)) - and delete the item/row -- [ ] If required, also locate the bucket name (the data study's source reference) in DynamoDB table *treprod-ldn-pj1-DsAccounts* (please note - the table name differs based on the SWB config file details provided in [deployment Step 2C](../deployment/Step2-DeployServiceWorkbench.md)) - and delete the item/row +- [ ] Select the [_Tables_](https://eu-west-2.console.aws.amazon.com/dynamodbv2/home?region=eu-west-2#tables) menu option on the left side +- [ ] Locate the data study (with the name provided when it was registered in SWB) in DynamoDB table _treprod-ldn-pj1-Studies_ (please note + the table name differs based on the SWB config file details provided in [deployment Step 2C](../deployment/Step2-DeployServiceWorkbench.md)) + and delete the item/row +- [ ] If required, also locate the bucket name (the data study's source reference) in DynamoDB table _treprod-ldn-pj1-DsAccounts_ (please note + the table name differs based on the SWB config file details provided in [deployment Step 2C](../deployment/Step2-DeployServiceWorkbench.md)) + and delete the item/row You can now try registering the data study again in SWB. @@ -80,54 +80,54 @@ If a researcher notices that the SWB workspace creation takes a long time, they ### S3 File Mounts SWB workspaces use S3 file mounts which come with some limitations described in the - [official SWB guide](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf), - pages 13-14. +[official SWB guide](https://github.com/awslabs/service-workbench-on-aws/blob/v5.2.3/docs/Service_Workbench_User_Guide.pdf), +pages 13-14. ## Egress Add-On Application ### Egress Workflow This section describes troubleshooting steps and tips for the egress workflow (StepFunction tasks). - The [AWS StepFunctions](https://eu-west-2.console.aws.amazon.com/states/home?region=eu-west-2#/statemachines) - service should be the first point of contact for any monitoring, debugging or troubleshooting actions. +The [AWS StepFunctions](https://eu-west-2.console.aws.amazon.com/states/home?region=eu-west-2#/statemachines) +service should be the first point of contact for any monitoring, debugging or troubleshooting actions. Below are some debugging steps which can be followed to identify an issue in the egress workflow. - This includes any errors arising between the time the egress request submission button is clicked in SWB - to the time the request has been approved or rejected. +This includes any errors arising between the time the egress request submission button is clicked in SWB +to the time the request has been approved or rejected. 1. Identify the egress request ID which caused the issue. This should be done by visiting the - [AWS StepFunctions](https://eu-west-2.console.aws.amazon.com/states/home?region=eu-west-2#/statemachines) service. - By clicking on the **DataEgressWorkflow** state machine, you will be able to see all of the current and past - workflow executions. The most recent execution will be listed at the top. Failed executions will be noticeable - through the execution status, as seen below. + [AWS StepFunctions](https://eu-west-2.console.aws.amazon.com/states/home?region=eu-west-2#/statemachines) service. + By clicking on the **DataEgressWorkflow** state machine, you will be able to see all of the current and past + workflow executions. The most recent execution will be listed at the top. Failed executions will be noticeable + through the execution status, as seen below. - The Execution ID is the same as the Egress Request ID, a unique 36 character string that can be used to - identify an egress request. The ID can also be found in the Egress App. + The Execution ID is the same as the Egress Request ID, a unique 36 character string that can be used to + identify an egress request. The ID can also be found in the Egress App. - ![Troubleshooting Egress App - 1](../../res/images/troubleshooting_runbook/Troubleshooting-EgressApp-1.png) + ![Troubleshooting Egress App - 1](../../res/images/troubleshooting_runbook/Troubleshooting-EgressApp-1.png) 1. Once the egress request has been identified, you can click on the execution to view the graph inspector, - which displays each stage of the worfklow execution. The stages show the workflow order and are either - displayed in green or red to indicate a success or a failure. + which displays each stage of the worfklow execution. The stages show the workflow order and are either + displayed in green or red to indicate a success or a failure. - ![Troubleshooting Egress App - 2](../../res/images/troubleshooting_runbook/Troubleshooting-EgressApp-2.png) + ![Troubleshooting Egress App - 2](../../res/images/troubleshooting_runbook/Troubleshooting-EgressApp-2.png) 1. The red stage on the graph indicates that the issue happened during its execution. By clicking on the stage, - you can analyse details associated with the stage such as the input data supplied to the stage, the data output - from the stage, and the exception (error logs) produced. A few examples are shown below. + you can analyse details associated with the stage such as the input data supplied to the stage, the data output + from the stage, and the exception (error logs) produced. A few examples are shown below. - ![Troubleshooting Egress App - 3](../../res/images/troubleshooting_runbook/Troubleshooting-EgressApp-3.png) + ![Troubleshooting Egress App - 3](../../res/images/troubleshooting_runbook/Troubleshooting-EgressApp-3.png) - ![Troubleshooting Egress App - 4](../../res/images/troubleshooting_runbook/Troubleshooting-EgressApp-4.png) + ![Troubleshooting Egress App - 4](../../res/images/troubleshooting_runbook/Troubleshooting-EgressApp-4.png) 1. Below the graph inspector, the execution event history can be used to analyse a timestamped series of events - from the workflow execution. If the failed stage involves a Lambda function, the Lambda configurations (code, - permissions) and logs can be accessed by clicking on the associated links. + from the workflow execution. If the failed stage involves a Lambda function, the Lambda configurations (code, + permissions) and logs can be accessed by clicking on the associated links. - ![Troubleshooting Egress App - 5](../../res/images/troubleshooting_runbook/Troubleshooting-EgressApp-5.png) + ![Troubleshooting Egress App - 5](../../res/images/troubleshooting_runbook/Troubleshooting-EgressApp-5.png) 1. If the root cause has still not been identified with a Lambda function, further analysis through CloudWatch logs - can be performed. Use the link in the execution event history to access the Lambda log group. + can be performed. Use the link in the execution event history to access the Lambda log group. - Click on the '**Search all**' button and use the '**filter events**' input - Pass the **Egress Request ID** in double quotes and search for @@ -137,25 +137,25 @@ Below are some debugging steps which can be followed to identify an issue in the - e.g. "f7f57dbb-2b1e-43c8-817d-b56a193d785a" - Click on any of the log stream names in the returned event messages - This will display all timestamped log messages output by the Lambda function during execution. All messages will - be associated with the specified egress request ID. + This will display all timestamped log messages output by the Lambda function during execution. All messages will + be associated with the specified egress request ID. - Depending on the log level configured, different logs will be visible in CloudWatch. Setting log level to - DEBUG will provide additional useful information such as the name of the S3 bucket used in the function. Log level - can be set by passing in the environment variable LOG_LEVEL to the Lambda function. + Depending on the log level configured, different logs will be visible in CloudWatch. Setting log level to + DEBUG will provide additional useful information such as the name of the S3 bucket used in the function. Log level + can be set by passing in the environment variable LOG_LEVEL to the Lambda function. - ![Troubleshooting Egress App - 6](../../res/images/troubleshooting_runbook/Troubleshooting-EgressApp-6.png) + ![Troubleshooting Egress App - 6](../../res/images/troubleshooting_runbook/Troubleshooting-EgressApp-6.png) - For additional troubleshooting guidance check the documentation for - [Lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/lambda-troubleshooting.html) - and [Appsync APIs](https://docs.aws.amazon.com/appsync/latest/devguide/troubleshooting-and-common-mistakes.html). + For additional troubleshooting guidance check the documentation for + [Lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/lambda-troubleshooting.html) + and [Appsync APIs](https://docs.aws.amazon.com/appsync/latest/devguide/troubleshooting-and-common-mistakes.html). ### Failure Scenarios The use of a task token has a side-effect that an egress request can only be updated once for a given token. - If something were to go wrong in the workflow after the user has made their decision in the front-end, it would - not be possible for them to save their decision again. If they tried to do so, they would get a `Null...coerced...` error. - One way of getting around this without needing the researcher to resubmit their egress request again is to do the following: +If something were to go wrong in the workflow after the user has made their decision in the front-end, it would +not be possible for them to save their decision again. If they tried to do so, they would get a `Null...coerced...` error. +One way of getting around this without needing the researcher to resubmit their egress request again is to do the following: - In the [AWS Step Functions](https://eu-west-2.console.aws.amazon.com/states/home?region=eu-west-2#/statemachines) service, identify and select the execution that has the issue diff --git a/src/components/budget_controls/ProjectBudgetControl-Cfn.yaml b/src/components/budget_controls/ProjectBudgetControl-Cfn.yaml index 9eef5fbc..915a2909 100644 --- a/src/components/budget_controls/ProjectBudgetControl-Cfn.yaml +++ b/src/components/budget_controls/ProjectBudgetControl-Cfn.yaml @@ -13,7 +13,7 @@ Parameters: Description: Specify the ID of existing IAM role initial-stack--xacc-env-mgmt Type: String MinLength: 1 - AllowedPattern : "[0-9]+" + AllowedPattern: "[0-9]+" ConstraintDescription: Ensure only the ID part of the IAM role name is provided ServiceCatalogProductsList: @@ -75,7 +75,7 @@ Parameters: Conditions: ApplyToAllSCProducts: !Equals - - !Join [ "", !Ref ServiceCatalogProductsList ] + - !Join ["", !Ref ServiceCatalogProductsList] - "" Resources: @@ -93,16 +93,16 @@ Resources: Properties: PolicyDocument: Id: BudgetNotifySNSPolicy - Version: '2012-10-17' + Version: "2012-10-17" Statement: - - Sid: BudgetNotifySNSPolicyId1 - Effect: Allow - Principal: - Service: budgets.amazonaws.com - Action: SNS:Publish - Resource: !Ref BudgetNotifySNS + - Sid: BudgetNotifySNSPolicyId1 + Effect: Allow + Principal: + Service: budgets.amazonaws.com + Action: SNS:Publish + Resource: !Ref BudgetNotifySNS Topics: - - !Ref BudgetNotifySNS + - !Ref BudgetNotifySNS AWSBudgetsExecutionRole: Type: AWS::IAM::Role @@ -115,7 +115,7 @@ Resources: Service: - budgets.amazonaws.com Action: - - 'sts:AssumeRole' + - "sts:AssumeRole" ManagedPolicyArns: - arn:aws:iam::aws:policy/AWSBudgetsActionsWithAWSResourceControlAccess Policies: @@ -135,7 +135,7 @@ Resources: Type: AWS::IAM::ManagedPolicy Properties: PolicyDocument: - Version: '2012-10-17' + Version: "2012-10-17" Statement: - Effect: Deny Action: @@ -143,10 +143,50 @@ Resources: Resource: !If - ApplyToAllSCProducts - "*" - - - !Join [ "", [ "arn:aws:catalog:", !Ref AWS::Region, ":", !Ref AWS::AccountId, ":product/prod-", !Select [ 0, !Ref ServiceCatalogProductsList ] ] ] - - !Join [ "", [ "arn:aws:catalog:", !Ref AWS::Region, ":", !Ref AWS::AccountId, ":product/prod-", !Select [ 1, !Ref ServiceCatalogProductsList ] ] ] - - !Join [ "", [ "arn:aws:catalog:", !Ref AWS::Region, ":", !Ref AWS::AccountId, ":product/prod-", !Select [ 2, !Ref ServiceCatalogProductsList ] ] ] - - !Join [ "", [ "arn:aws:catalog:", !Ref AWS::Region, ":", !Ref AWS::AccountId, ":product/prod-", !Select [ 3, !Ref ServiceCatalogProductsList ] ] ] + - - !Join [ + "", + [ + "arn:aws:catalog:", + !Ref AWS::Region, + ":", + !Ref AWS::AccountId, + ":product/prod-", + !Select [0, !Ref ServiceCatalogProductsList], + ], + ] + - !Join [ + "", + [ + "arn:aws:catalog:", + !Ref AWS::Region, + ":", + !Ref AWS::AccountId, + ":product/prod-", + !Select [1, !Ref ServiceCatalogProductsList], + ], + ] + - !Join [ + "", + [ + "arn:aws:catalog:", + !Ref AWS::Region, + ":", + !Ref AWS::AccountId, + ":product/prod-", + !Select [2, !Ref ServiceCatalogProductsList], + ], + ] + - !Join [ + "", + [ + "arn:aws:catalog:", + !Ref AWS::Region, + ":", + !Ref AWS::AccountId, + ":product/prod-", + !Select [3, !Ref ServiceCatalogProductsList], + ], + ] TREProjectBudget: Type: AWS::Budgets::Budget @@ -191,7 +231,7 @@ Resources: BudgetName: !Ref TREProjectBudget Definition: IamActionDefinition: - PolicyArn: !Ref 'DenyCreateWorkSpacePolicy' + PolicyArn: !Ref "DenyCreateWorkSpacePolicy" Roles: - !Sub "initial-stack-${SWBStackID}-xacc-env-mgmt" ExecutionRoleArn: !GetAtt AWSBudgetsExecutionRole.Arn diff --git a/src/data_lake/DataLake-Cfn.yaml b/src/data_lake/DataLake-Cfn.yaml index 3a6605f2..e192e41b 100644 --- a/src/data_lake/DataLake-Cfn.yaml +++ b/src/data_lake/DataLake-Cfn.yaml @@ -44,22 +44,22 @@ Resources: - Sid: Enable IAM Policies Effect: Allow Principal: - AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root' + AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root" Action: kms:* - Resource: '*' + Resource: "*" - Sid: Enable Access From Egress App Account Effect: Allow Principal: AWS: - - !Sub 'arn:aws:iam::${EgressAppAccount}:root' + - !Sub "arn:aws:iam::${EgressAppAccount}:root" Action: - kms:Encrypt - kms:Decrypt - kms:ReEncrypt - kms:GenerateDataKey - kms:DescribeKey - Resource: '*' + Resource: "*" TRESourceBucketKMSKey: Type: AWS::KMS::Key @@ -72,9 +72,9 @@ Resources: - Sid: Enable IAM Policies Effect: Allow Principal: - AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root' + AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root" Action: kms:* - Resource: '*' + Resource: "*" TRESourceBucketKMSAlias: Type: AWS::KMS::Alias @@ -93,9 +93,9 @@ Resources: - Sid: Enable IAM Policies Effect: Allow Principal: - AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root' + AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root" Action: kms:* - Resource: '*' + Resource: "*" TREAnalystBucketKMSAlias: Type: AWS::KMS::Alias @@ -114,9 +114,9 @@ Resources: - Sid: Enable IAM Policies Effect: Allow Principal: - AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root' + AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root" Action: kms:* - Resource: '*' + Resource: "*" TREAccessLogsBucketKMSKey: Type: AWS::KMS::Key @@ -129,9 +129,9 @@ Resources: - Sid: Enable IAM Policies Effect: Allow Principal: - AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root' + AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root" Action: kms:* - Resource: '*' + Resource: "*" TREAccessLogsBucket: Type: AWS::S3::Bucket @@ -162,60 +162,60 @@ Resources: Statement: - Sid: S3ServerAccessLogsPolicy-TRETargetBucket Action: - - 's3:PutObject' + - "s3:PutObject" Effect: Allow Resource: !Sub "arn:aws:s3:::${TREAccessLogsBucket}/logs-tre-target-bucket*" Principal: Service: logging.s3.amazonaws.com Condition: ArnLike: - 'aws:SourceArn': + "aws:SourceArn": - !GetAtt TRETargetBucket.Arn StringEquals: - 'aws:SourceAccount': - - !Sub "${AWS::AccountId}" + "aws:SourceAccount": + - !Sub "${AWS::AccountId}" - Sid: S3ServerAccessLogsPolicy-TRESourceBucket Action: - - 's3:PutObject' + - "s3:PutObject" Effect: Allow Resource: !Sub "arn:aws:s3:::${TREAccessLogsBucket}/logs-tre-source-bucket*" Principal: Service: logging.s3.amazonaws.com Condition: ArnLike: - 'aws:SourceArn': + "aws:SourceArn": - !GetAtt TRESourceBucket.Arn StringEquals: - 'aws:SourceAccount': - - !Sub "${AWS::AccountId}" + "aws:SourceAccount": + - !Sub "${AWS::AccountId}" - Sid: S3ServerAccessLogsPolicy-TREAnalystResultsBucket Action: - - 's3:PutObject' + - "s3:PutObject" Effect: Allow Resource: !Sub "arn:aws:s3:::${TREAccessLogsBucket}/logs-tre-analyst-results-bucket*" Principal: Service: logging.s3.amazonaws.com Condition: ArnLike: - 'aws:SourceArn': + "aws:SourceArn": - !GetAtt TREAnalystResultsBucket.Arn StringEquals: - 'aws:SourceAccount': - - !Sub "${AWS::AccountId}" + "aws:SourceAccount": + - !Sub "${AWS::AccountId}" - Sid: S3ServerAccessLogsPolicy-TRELakeAdminResultsBucket Action: - - 's3:PutObject' + - "s3:PutObject" Effect: Allow Resource: !Sub "arn:aws:s3:::${TREAccessLogsBucket}/logs-tre-lake-admin-results-bucket*" Principal: Service: logging.s3.amazonaws.com Condition: ArnLike: - 'aws:SourceArn': + "aws:SourceArn": - !GetAtt TRELakeAdminResultsBucket.Arn StringEquals: - 'aws:SourceAccount': - - !Sub "${AWS::AccountId}" + "aws:SourceAccount": + - !Sub "${AWS::AccountId}" TRETargetBucket: Type: AWS::S3::Bucket @@ -308,8 +308,8 @@ Resources: Action: - s3:PutObject Effect: Deny - Resource: !Sub '${TRETargetBucket.Arn}/*' - Principal: '*' + Resource: !Sub "${TRETargetBucket.Arn}/*" + Principal: "*" Condition: StringNotEquals: "s3:x-amz-server-side-encryption": "aws:kms" @@ -317,8 +317,8 @@ Resources: Action: - s3:PutObject Effect: Deny - Resource: !Sub '${TRETargetBucket.Arn}/*' - Principal: '*' + Resource: !Sub "${TRETargetBucket.Arn}/*" + Principal: "*" Condition: "Null": "s3:x-amz-server-side-encryption": true @@ -329,21 +329,21 @@ Resources: - s3:List* Effect: Allow Resource: - - !Sub '${TRETargetBucket.Arn}/*' - - !Sub '${TRETargetBucket.Arn}' + - !Sub "${TRETargetBucket.Arn}/*" + - !Sub "${TRETargetBucket.Arn}" Principal: AWS: - - !Sub 'arn:aws:iam::${EgressAppAccount}:root' + - !Sub "arn:aws:iam::${EgressAppAccount}:root" - Sid: AllowWriteFromEgressAppAccount Action: - s3:PutObject - s3:Abort* Effect: Allow Resource: - - !Sub '${TRETargetBucket.Arn}/*' + - !Sub "${TRETargetBucket.Arn}/*" Principal: AWS: - - !Sub 'arn:aws:iam::${EgressAppAccount}:root' + - !Sub "arn:aws:iam::${EgressAppAccount}:root" TRESourceBucketPolicy: Type: AWS::S3::BucketPolicy @@ -356,8 +356,8 @@ Resources: Action: - s3:PutObject Effect: Deny - Resource: !Sub '${TRESourceBucket.Arn}/*' - Principal: '*' + Resource: !Sub "${TRESourceBucket.Arn}/*" + Principal: "*" Condition: StringNotEquals: "s3:x-amz-server-side-encryption": "aws:kms" @@ -365,8 +365,8 @@ Resources: Action: - s3:PutObject Effect: Deny - Resource: !Sub '${TRESourceBucket.Arn}/*' - Principal: '*' + Resource: !Sub "${TRESourceBucket.Arn}/*" + Principal: "*" Condition: "Null": "s3:x-amz-server-side-encryption": true @@ -382,8 +382,8 @@ Resources: Action: - s3:PutObject Effect: Deny - Resource: !Sub '${TREAnalystResultsBucket.Arn}/*' - Principal: '*' + Resource: !Sub "${TREAnalystResultsBucket.Arn}/*" + Principal: "*" Condition: StringNotEquals: "s3:x-amz-server-side-encryption": "aws:kms" @@ -391,8 +391,8 @@ Resources: Action: - s3:PutObject Effect: Deny - Resource: !Sub '${TREAnalystResultsBucket.Arn}/*' - Principal: '*' + Resource: !Sub "${TREAnalystResultsBucket.Arn}/*" + Principal: "*" Condition: "Null": "s3:x-amz-server-side-encryption": true @@ -408,8 +408,8 @@ Resources: Action: - s3:PutObject Effect: Deny - Resource: !Sub '${TRELakeAdminResultsBucket.Arn}/*' - Principal: '*' + Resource: !Sub "${TRELakeAdminResultsBucket.Arn}/*" + Principal: "*" Condition: StringNotEquals: "s3:x-amz-server-side-encryption": "aws:kms" @@ -417,8 +417,8 @@ Resources: Action: - s3:PutObject Effect: Deny - Resource: !Sub '${TRELakeAdminResultsBucket.Arn}/*' - Principal: '*' + Resource: !Sub "${TRELakeAdminResultsBucket.Arn}/*" + Principal: "*" Condition: "Null": "s3:x-amz-server-side-encryption": true @@ -427,7 +427,7 @@ Resources: Type: AWS::IAM::ManagedPolicy Properties: PolicyDocument: - Version: '2012-10-17' + Version: "2012-10-17" Statement: - Sid: AllowKeyAdministration Effect: Allow @@ -454,7 +454,7 @@ Resources: Type: AWS::IAM::ManagedPolicy Properties: PolicyDocument: - Version: '2012-10-17' + Version: "2012-10-17" Statement: - Sid: AllowKeyUsage Effect: Allow @@ -472,22 +472,22 @@ Resources: Type: AWS::IAM::ManagedPolicy Properties: PolicyDocument: - Version: '2012-10-17' + Version: "2012-10-17" Statement: - Sid: AllowGlueUsage Effect: Allow Action: - iam:GetRole - iam:PassRole - Resource: !GetAtt 'TREGlueRole.Arn' + Resource: !GetAtt "TREGlueRole.Arn" TREDataAnalystRole: Type: AWS::IAM::Role Properties: ManagedPolicyArns: - arn:aws:iam::aws:policy/AmazonAthenaFullAccess - - !Ref 'TRELFQueryPolicy' - - !Ref 'TRES3DataAnalystPolicy' + - !Ref "TRELFQueryPolicy" + - !Ref "TRES3DataAnalystPolicy" AssumeRolePolicyDocument: Version: 2012-10-17 Statement: @@ -496,7 +496,7 @@ Resources: AWS: - !Sub ${AWS::AccountId} Action: - - 'sts:AssumeRole' + - "sts:AssumeRole" TREDataLakeAdminRole: Type: AWS::IAM::Role @@ -507,10 +507,10 @@ Resources: - arn:aws:iam::aws:policy/IAMReadOnlyAccess - arn:aws:iam::aws:policy/AmazonS3FullAccess - arn:aws:iam::aws:policy/AWSGlueConsoleFullAccess - - !Ref 'TRELFQueryPolicy' - - !Ref 'TRES3DataAnalystPolicy' - - !Ref 'TREDataLakeKMSUsagePolicy' - - !Ref 'TREGlueRolePolicy' + - !Ref "TRELFQueryPolicy" + - !Ref "TRES3DataAnalystPolicy" + - !Ref "TREDataLakeKMSUsagePolicy" + - !Ref "TREGlueRolePolicy" AssumeRolePolicyDocument: Version: 2012-10-17 Statement: @@ -519,7 +519,7 @@ Resources: AWS: - !Sub ${AWS::AccountId} Action: - - 'sts:AssumeRole' + - "sts:AssumeRole" TREAdminRole: Type: AWS::IAM::Role @@ -534,20 +534,20 @@ Resources: - !Ref TREDataLakeKMSManagementPolicy - !Ref TREGlueRolePolicy AssumeRolePolicyDocument: - Version: '2012-10-17' + Version: "2012-10-17" Statement: - Effect: Allow Principal: AWS: - !Ref AWS::AccountId Action: - - 'sts:AssumeRole' + - "sts:AssumeRole" TRELFRegisterLocationServiceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: - Version: '2012-10-17' + Version: "2012-10-17" Statement: - Effect: Allow Principal: @@ -565,7 +565,7 @@ Resources: reason: Lake Formation does not support resource ARNs Properties: AssumeRolePolicyDocument: - Version: '2012-10-17' + Version: "2012-10-17" Statement: - Effect: Allow Principal: @@ -578,7 +578,7 @@ Resources: Policies: - PolicyName: glue-tre-policy PolicyDocument: - Version: '2012-10-17' + Version: "2012-10-17" Statement: - Effect: Allow Action: @@ -587,10 +587,10 @@ Resources: - s3:DeleteObject - s3:ListBucket Resource: - - !GetAtt 'TRESourceBucket.Arn' - - !GetAtt 'TRETargetBucket.Arn' - - !Sub '${TRESourceBucket.Arn}/*' - - !Sub '${TRETargetBucket.Arn}/*' + - !GetAtt "TRESourceBucket.Arn" + - !GetAtt "TRETargetBucket.Arn" + - !Sub "${TRESourceBucket.Arn}/*" + - !Sub "${TRETargetBucket.Arn}/*" - Effect: Allow Action: - kms:Encrypt @@ -603,28 +603,28 @@ Resources: Action: - lakeformation:GetDataAccess - lakeformation:GrantPermissions - Resource: '*' + Resource: "*" Path: / TRES3DataLakePolicy: Type: AWS::IAM::ManagedPolicy Properties: PolicyDocument: - Version: '2012-10-17' + Version: "2012-10-17" Statement: - Effect: Allow Action: - s3:GetObject - s3:PutObject Resource: - - !Sub 'arn:aws:s3:::${TRETargetBucket}/*' - - !Sub 'arn:aws:s3:::${TRESourceBucket}/*' + - !Sub "arn:aws:s3:::${TRETargetBucket}/*" + - !Sub "arn:aws:s3:::${TRESourceBucket}/*" - Effect: Allow Action: - s3:ListBucket Resource: - - !GetAtt 'TRETargetBucket.Arn' - - !GetAtt 'TRESourceBucket.Arn' + - !GetAtt "TRETargetBucket.Arn" + - !GetAtt "TRESourceBucket.Arn" - Effect: Allow Action: - kms:Decrypt @@ -634,7 +634,7 @@ Resources: - !GetAtt TRESourceBucketKMSKey.Arn - !GetAtt TRETargetBucketKMSKey.Arn Roles: - - !Ref 'TRELFRegisterLocationServiceRole' + - !Ref "TRELFRegisterLocationServiceRole" TRES3DataAnalystPolicy: Type: AWS::IAM::ManagedPolicy @@ -645,25 +645,25 @@ Resources: reason: KMS ListAliases does not support resource ARNs Properties: PolicyDocument: - Version: '2012-10-17' + Version: "2012-10-17" Statement: - Effect: Allow Action: - s3:ListBucket Resource: - - !Sub 'arn:aws:s3:::${TREAnalystResultsBucket}' - - !Sub 'arn:aws:s3:::${TRESourceBucket}' + - !Sub "arn:aws:s3:::${TREAnalystResultsBucket}" + - !Sub "arn:aws:s3:::${TRESourceBucket}" - Effect: Allow Action: - s3:GetObject Resource: - - !Sub 'arn:aws:s3:::${TREAnalystResultsBucket}/*' - - !Sub 'arn:aws:s3:::${TRESourceBucket}/*' + - !Sub "arn:aws:s3:::${TREAnalystResultsBucket}/*" + - !Sub "arn:aws:s3:::${TRESourceBucket}/*" - Effect: Allow Action: - s3:PutObject Resource: - - !Sub 'arn:aws:s3:::${TREAnalystResultsBucket}/*' + - !Sub "arn:aws:s3:::${TREAnalystResultsBucket}/*" - Effect: Allow Action: kms:Decrypt Resource: @@ -677,7 +677,7 @@ Resources: - Effect: Allow Action: - kms:ListAliases - Resource: '*' + Resource: "*" TRELFQueryPolicy: Type: AWS::IAM::ManagedPolicy @@ -688,7 +688,7 @@ Resources: reason: Lake Formation does not support resource ARNs Properties: PolicyDocument: - Version: '2012-10-17' + Version: "2012-10-17" Statement: - Effect: Allow Action: @@ -701,7 +701,7 @@ Resources: - lakeformation:GetQueryState - lakeformation:GetWorkUnits - lakeformation:Execute - Resource: '*' + Resource: "*" TRELFLocationPolicy: Type: AWS::IAM::Policy @@ -711,9 +711,9 @@ Resources: - id: W12 reason: Lake Formation does not support resource ARNs Properties: - PolicyName: !Sub 'TRELFLocationPolicy-${AWS::StackName}' + PolicyName: !Sub "TRELFLocationPolicy-${AWS::StackName}" PolicyDocument: - Version: '2012-10-17' + Version: "2012-10-17" Statement: - Effect: Allow Action: @@ -722,36 +722,36 @@ Resources: - lakeformation:AbortTransaction - lakeformation:GetTableObjects - lakeformation:UpdateTableObjects - Resource: '*' + Resource: "*" Roles: - - !Ref 'TRELFRegisterLocationServiceRole' + - !Ref "TRELFRegisterLocationServiceRole" TRELFDataLakeSettings: Type: AWS::LakeFormation::DataLakeSettings Properties: Admins: - - DataLakePrincipalIdentifier: !GetAtt 'TREDataLakeAdminRole.Arn' - - DataLakePrincipalIdentifier: !GetAtt 'TREAdminRole.Arn' + - DataLakePrincipalIdentifier: !GetAtt "TREDataLakeAdminRole.Arn" + - DataLakePrincipalIdentifier: !GetAtt "TREAdminRole.Arn" TRELFDataLakeLocation1: Type: AWS::LakeFormation::Resource Properties: - ResourceArn: !Sub 'arn:aws:s3:::${TRETargetBucket}' - RoleArn: !GetAtt 'TRELFRegisterLocationServiceRole.Arn' + ResourceArn: !Sub "arn:aws:s3:::${TRETargetBucket}" + RoleArn: !GetAtt "TRELFRegisterLocationServiceRole.Arn" UseServiceLinkedRole: false TRELFDataLakeLocation2: Type: AWS::LakeFormation::Resource Properties: - ResourceArn: !Sub 'arn:aws:s3:::${TRESourceBucket}' - RoleArn: !GetAtt 'TRELFRegisterLocationServiceRole.Arn' + ResourceArn: !Sub "arn:aws:s3:::${TRESourceBucket}" + RoleArn: !GetAtt "TRELFRegisterLocationServiceRole.Arn" UseServiceLinkedRole: false TRELFDatabasePermissionForLFRegisterLocationServiceRole: Type: AWS::LakeFormation::Permissions Properties: DataLakePrincipal: - DataLakePrincipalIdentifier: !GetAtt 'TRELFRegisterLocationServiceRole.Arn' + DataLakePrincipalIdentifier: !GetAtt "TRELFRegisterLocationServiceRole.Arn" Permissions: - ALTER - CREATE_TABLE @@ -759,64 +759,64 @@ Resources: - DROP Resource: DatabaseResource: - Name: !Ref 'TRELakeFormationDatabase' + Name: !Ref "TRELakeFormationDatabase" TRELFDatabasePermissionForAdminRole: Type: AWS::LakeFormation::Permissions Properties: DataLakePrincipal: - DataLakePrincipalIdentifier: !GetAtt 'TREAdminRole.Arn' + DataLakePrincipalIdentifier: !GetAtt "TREAdminRole.Arn" Permissions: - ALL PermissionsWithGrantOption: - ALL Resource: DatabaseResource: - Name: !Ref 'TRELakeFormationDatabase' + Name: !Ref "TRELakeFormationDatabase" TRELFDatabasePermissionForDataLakeAdminRole: Type: AWS::LakeFormation::Permissions Properties: DataLakePrincipal: - DataLakePrincipalIdentifier: !GetAtt 'TREDataLakeAdminRole.Arn' + DataLakePrincipalIdentifier: !GetAtt "TREDataLakeAdminRole.Arn" Permissions: - ALL PermissionsWithGrantOption: - ALL Resource: DatabaseResource: - Name: !Ref 'TRELakeFormationDatabase' + Name: !Ref "TRELakeFormationDatabase" TRELFDatabasePermissionForGlueRole: Type: AWS::LakeFormation::Permissions Properties: DataLakePrincipal: - DataLakePrincipalIdentifier: !GetAtt 'TREGlueRole.Arn' + DataLakePrincipalIdentifier: !GetAtt "TREGlueRole.Arn" Permissions: - ALL PermissionsWithGrantOption: - ALL Resource: DatabaseResource: - Name: !Ref 'TRELakeFormationDatabase' + Name: !Ref "TRELakeFormationDatabase" TRELFDataPermissionForGlueRole: Type: AWS::LakeFormation::Permissions Properties: DataLakePrincipal: - DataLakePrincipalIdentifier: !GetAtt 'TREGlueRole.Arn' + DataLakePrincipalIdentifier: !GetAtt "TREGlueRole.Arn" Permissions: - CREATE_TABLE_READ_WRITE Resource: DataLocationResource: - S3Resource: !Sub '${TRESourceBucket.Arn}/' + S3Resource: !Sub "${TRESourceBucket.Arn}/" TRELakeFormationDatabase: Type: AWS::Glue::Database Properties: - CatalogId: !Ref 'AWS::AccountId' + CatalogId: !Ref "AWS::AccountId" DatabaseInput: - Name: !Ref 'LFDatabaseName' + Name: !Ref "LFDatabaseName" TREGlueSecurityConfiguration: Type: AWS::Glue::SecurityConfiguration diff --git a/src/data_lake/README.md b/src/data_lake/README.md index 0b564ed3..39efd8eb 100644 --- a/src/data_lake/README.md +++ b/src/data_lake/README.md @@ -1,7 +1,7 @@ # TRE Data Lake The TREEHOOSE TRE solution contains a data lake that needs to be deployed with every TRE project. - It is used to store and manage datasets in an environment which can be governed securely. +It is used to store and manage datasets in an environment which can be governed securely. The pre-configured data lake is created with the CloudFormation template provided as source code. @@ -10,72 +10,76 @@ The pre-configured data lake is created with the CloudFormation template provide ### Amazon Simple Storage Service (S3) [Amazon S3](https://aws.amazon.com/s3/) provides the storage for the data in the data lake - in a cost-effective easy-to-use way. +in a cost-effective easy-to-use way. There are 5 S3 buckets created: 1. TRE Source Bucket - * Can store input datasets for research activities - * Can be attached to SWB as a registered data study - * Can be used in data transformation pipelines to manage e.g. sensitive datasets + + - Can store input datasets for research activities + - Can be attached to SWB as a registered data study + - Can be used in data transformation pipelines to manage e.g. sensitive datasets 1. TRE Target Bucket - * Stores the approved egressed data from research activities - * Used by the Egress Addon web application + + - Stores the approved egressed data from research activities + - Used by the Egress Addon web application 1. TRE Analyst Bucket - * A data analyst can run queries using [Amazon Athena](https://aws.amazon.com/athena) - on the data and store query results here + + - A data analyst can run queries using [Amazon Athena](https://aws.amazon.com/athena) + on the data and store query results here 1. TRE Lake Admin Bucket - * A data lake administrator can run queries using [Amazon Athena](https://aws.amazon.com/athena) - on the data and store query results here + + - A data lake administrator can run queries using [Amazon Athena](https://aws.amazon.com/athena) + on the data and store query results here 1. TRE Access Logs Bucket - * S3 access logging destination for the other buckets + - S3 access logging destination for the other buckets Each of the data S3 buckets: -* Blocks public access +- Blocks public access -* Is encrypted at-rest +- Is encrypted at-rest -* Has versioning enabled +- Has versioning enabled -* Has access logging configured +- Has access logging configured ### AWS Identity and Access Management (IAM) [AWS IAM](https://aws.amazon.com/iam/) provides fine-grained access control to other AWS services. Some resources in this template have IAM policies attached to control access or enforce security - constraints. For example, the S3 buckets will deny attempts to upload data in unencrypted form - and expect the use of KMS keys for server-side encryption. +constraints. For example, the S3 buckets will deny attempts to upload data in unencrypted form +and expect the use of KMS keys for server-side encryption. There are 5 IAM roles created: 1. TREAdminRole -* Has PowerUser access -* Can manage KMS keys but cannot use the keys +- Has PowerUser access +- Can manage KMS keys but cannot use the keys 1. TREDataAnalystRole -* Can read data in the TRE Source Bucket and execute Amazon Athena queries -* Can put query results in the TRE Analyst Bucket +- Can read data in the TRE Source Bucket and execute Amazon Athena queries +- Can put query results in the TRE Analyst Bucket 1. TREDataLakeAdminRole -* The designated Lake Formation administrator with full access to the Data Lake -* Can use KMS keys but cannot manage or perform administrative tasks on KMS keys +- The designated Lake Formation administrator with full access to the Data Lake +- Can use KMS keys but cannot manage or perform administrative tasks on KMS keys 1 TRELFRegisterLocationServiceRole -* Needed to register TRE Source Bucket and TRE Target Bucket with AWS Lake Formation +- Needed to register TRE Source Bucket and TRE Target Bucket with AWS Lake Formation 1. TREGlueRole -* Role used by AWS Glue for Glue jobs and crawlers +- Role used by AWS Glue for Glue jobs and crawlers ### AWS Key Management Service (KMS) @@ -88,15 +92,15 @@ Each of the S3 buckets is encrypted server-side with its dedicated KMS encryptio [AWS Glue](https://aws.amazon.com/glue/) makes data discovery and preparation easy to simplify data integration. An AWS Glue database is created which will be used to catalog metadata about data sources. This is the initial phase - of the long-term vision of exposing a catalog of datasets available within the data lake so that researchers can - request access to datasets that are of interest to them. +of the long-term vision of exposing a catalog of datasets available within the data lake so that researchers can +request access to datasets that are of interest to them. ### AWS Lake Formation [AWS Lake Formation](https://aws.amazon.com/lake-formation/) creates a secure data lake from the data sources - provided and applies access and security policies. It builds on the capabilities provided by Amazon S3 and - AWS Glue. +provided and applies access and security policies. It builds on the capabilities provided by Amazon S3 and +AWS Glue. - The TRE Source and TRE Target S3 buckets are registered as data lake locations. Once S3 buckets - (or any other supported data sources) are registered with AWS Lake Formation, the service can be used to - provide fine-grained access to both the data catalogs and the underlying data being described by the catalog. +The TRE Source and TRE Target S3 buckets are registered as data lake locations. Once S3 buckets +(or any other supported data sources) are registered with AWS Lake Formation, the service can be used to +provide fine-grained access to both the data catalogs and the underlying data being described by the catalog. diff --git a/src/deployment/DeploymentInstance-Cfn.yaml b/src/deployment/DeploymentInstance-Cfn.yaml index 59e4524a..17a9f216 100644 --- a/src/deployment/DeploymentInstance-Cfn.yaml +++ b/src/deployment/DeploymentInstance-Cfn.yaml @@ -144,46 +144,46 @@ Resources: commands: dependencies: command: !Sub "su ec2-user -c ' \ - echo \"Initiating yum packages installation\"; \ - sudo yum update -y; \ - sudo yum install -y yum-utils git jq; \ - echo \"Completed yum packages installation\"; \ - echo \"---\"; \ - echo \"Initiating nodejs and serverless installs\"; \ - curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash; \ - export PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin; \ - . ~/.nvm/nvm.sh; \ - source ~/.bashrc; \ - nvm install v14.17.0; \ - npm install -g pnpm@5.18.9 serverless@3.8.0; \ - echo \"Completed nodejs and serverless installs\"; \ - echo \"---\"; \ - echo \"Downloading additional packages\"; \ - mkdir ~/tmp && cd ~/tmp; \ - curl \"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip\" -o \"awscliv2.zip\"; \ - wget https://go.dev/dl/go1.18.linux-amd64.tar.gz; \ - echo \"Finished downloading additional packages\"; \ - echo \"---\"; \ - echo \"Extracting additional packages\"; \ - sudo unzip awscliv2.zip; \ - sudo rm -rf /usr/local/go; \ - sudo tar -C /usr/local -xzf go1.18.linux-amd64.tar.gz; \ - echo \"Finished extracting additional packages\"; \ - echo \"---\"; \ - echo \"Installing additional packages\"; \ - ./aws/install; \ - echo \"export PATH=/usr/local/go/bin:$PATH\" >>~/.bash_profile; \ - echo \"Finished installing additional packages\"; \ - echo \"---\"; \ - echo \"Create default AWS profile\"; \ - mkdir ~/.aws; \ - echo \"[default]\" > ~/.aws/config; \ - echo \"region = ${AWS::Region}\" >> ~/.aws/config; \ - echo \"[default]\" > ~/.aws/credentials; \ - echo \"region = ${AWS::Region}\" >> ~/.aws/credentials; \ - echo \"Create default VPC\"; \ - existing_vpc=`aws ec2 describe-vpcs --filters Name=isDefault,Values=true --query \"Vpcs[].VpcId\" --profile default | jq .[0]`; \ - if [ \"$existing_vpc_id\" == \"null\" ]; then aws ec2 create-default-vpc; fi;'" + echo \"Initiating yum packages installation\"; \ + sudo yum update -y; \ + sudo yum install -y yum-utils git jq; \ + echo \"Completed yum packages installation\"; \ + echo \"---\"; \ + echo \"Initiating nodejs and serverless installs\"; \ + curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash; \ + export PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin; \ + . ~/.nvm/nvm.sh; \ + source ~/.bashrc; \ + nvm install v14.17.0; \ + npm install -g pnpm@5.18.9 serverless@3.8.0; \ + echo \"Completed nodejs and serverless installs\"; \ + echo \"---\"; \ + echo \"Downloading additional packages\"; \ + mkdir ~/tmp && cd ~/tmp; \ + curl \"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip\" -o \"awscliv2.zip\"; \ + wget https://go.dev/dl/go1.18.linux-amd64.tar.gz; \ + echo \"Finished downloading additional packages\"; \ + echo \"---\"; \ + echo \"Extracting additional packages\"; \ + sudo unzip awscliv2.zip; \ + sudo rm -rf /usr/local/go; \ + sudo tar -C /usr/local -xzf go1.18.linux-amd64.tar.gz; \ + echo \"Finished extracting additional packages\"; \ + echo \"---\"; \ + echo \"Installing additional packages\"; \ + ./aws/install; \ + echo \"export PATH=/usr/local/go/bin:$PATH\" >>~/.bash_profile; \ + echo \"Finished installing additional packages\"; \ + echo \"---\"; \ + echo \"Create default AWS profile\"; \ + mkdir ~/.aws; \ + echo \"[default]\" > ~/.aws/config; \ + echo \"region = ${AWS::Region}\" >> ~/.aws/config; \ + echo \"[default]\" > ~/.aws/credentials; \ + echo \"region = ${AWS::Region}\" >> ~/.aws/credentials; \ + echo \"Create default VPC\"; \ + existing_vpc=`aws ec2 describe-vpcs --filters Name=isDefault,Values=true --query \"Vpcs[].VpcId\" --profile default | jq .[0]`; \ + if [ \"$existing_vpc_id\" == \"null\" ]; then aws ec2 create-default-vpc; fi;'" Properties: Tags: - Key: "Name" @@ -201,8 +201,8 @@ Resources: DeleteOnTermination: true Encrypted: true UserData: !Base64 - 'Fn::Join': - - '' + "Fn::Join": + - "" - - | #!/bin/bash -x - | @@ -210,21 +210,21 @@ Resources: - |+ - | - - '/opt/aws/bin/cfn-init -v ' - - ' --stack ' - - !Ref 'AWS::StackName' - - ' --resource Instance ' - - ' --region ' - - !Ref 'AWS::Region' + - "/opt/aws/bin/cfn-init -v " + - " --stack " + - !Ref "AWS::StackName" + - " --resource Instance " + - " --region " + - !Ref "AWS::Region" - |+ - | - - '/opt/aws/bin/cfn-signal -e $? ' - - ' --stack ' - - !Ref 'AWS::StackName' - - ' --resource Instance ' - - ' --region ' - - !Ref 'AWS::Region' + - "/opt/aws/bin/cfn-signal -e $? " + - " --stack " + - !Ref "AWS::StackName" + - " --resource Instance " + - " --region " + - !Ref "AWS::Region" - |+ CreationPolicy: ResourceSignal: @@ -258,7 +258,7 @@ Resources: Service: - ec2.amazonaws.com Action: - - 'sts:AssumeRole' + - "sts:AssumeRole" ManagedPolicyArns: - arn:aws:iam::aws:policy/AmazonAppStreamFullAccess - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore