Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update mount points #2092

Merged
merged 22 commits into from
Sep 20, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 0 additions & 12 deletions data_safe_haven/resources/workspace/ansible/desired_state.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -67,14 +67,12 @@
fi
fi

# This will create directories if src or dest ends in '/'
JimMadge marked this conversation as resolved.
Show resolved Hide resolved
- name: Copy bashrc skeleton
ansible.builtin.copy:
src: etc/skel/bashrc
dest: /etc/skel/.bashrc
mode: '0755'

# This will create directories if src or dest ends in '/'
- name: Copy xsession skeleton
ansible.builtin.copy:
src: etc/skel/xsession
Expand Down Expand Up @@ -125,14 +123,12 @@
validate: sshd -T -f %s
notify: Restart sshd

# This will create directories if src or dest ends in '/'
- name: Copy xrdp settings
ansible.builtin.copy:
src: etc/xrdp/
dest: /etc/xrdp/
mode: '0644'

# This will create directories if src or dest ends in '/'
- name: Copy xrdp logo
ansible.builtin.copy:
src: usr/local/share/xrdp/
Expand Down Expand Up @@ -163,7 +159,6 @@
regexp: '^TerminalEmulator='
line: 'TerminalEmulator=xfce4-terminal'

# This will create directories if src or dest ends in '/'
- name: Copy default terminal colourscheme
ansible.builtin.copy:
src: etc/xdg/xfce4/terminal/
Expand Down Expand Up @@ -203,14 +198,12 @@
- xrdp
- xrdp-sesman

# This will create directories if src or dest ends in '/'
- name: Copy desktop icons directory
ansible.builtin.copy:
src: "usr/local/share/icons/"
dest: "/usr/local/share/icons/"
mode: '0444'

# This will create directories if src or dest ends in '/'
- name: Copy desktop files directory
ansible.builtin.copy:
src: "etc/skel/Desktop/"
Expand All @@ -226,7 +219,6 @@
- gitea
- hedgedoc

# This will create directories if src or dest ends in '/'
- name: Add polkit rule to allow colord
ansible.builtin.copy:
src: etc/polkit-1/localauthority/50-local.d/50-colord.pkla
Expand Down Expand Up @@ -255,7 +247,6 @@
mode: '0640'
notify: Restart auditd

# This will create directories if src or dest ends in '/'
- name: Copy auditd privileged executable rules script
tags: auditd
ansible.builtin.copy:
Expand All @@ -270,7 +261,6 @@
creates: /etc/audit/rules.d/50-privileged.rules
notify: Restart auditd

# This will create directories if src or dest ends in '/'
- name: Copy ClamAV daemon configuration
ansible.builtin.copy:
src: etc/clamav/clamd.conf
Expand Down Expand Up @@ -305,7 +295,6 @@
systemctl stop clamav-freshclam && freshclam && systemctl start clamav-freshclam
creates: '/var/lib/clamav/main.{c[vl]d,inc}'

# This will create directories if src or dest ends in '/'
- name: Copy ClamAV services directory
ansible.builtin.copy:
src: "etc/systemd/system/"
Expand All @@ -331,7 +320,6 @@
enabled: true
state: started

# This will create directories if src or dest ends in '/'
- name: Copy smoke test files directory
ansible.builtin.copy:
src: "usr/local/smoke_tests/"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,8 +89,8 @@ write_files:
permissions: "0700"
content: |
#!/usr/bin/env bash
pushd /desired_state
ansible-playbook /desired_state/desired_state.yaml
pushd /var/local/ansible
ansible-playbook desired_state.yaml
popd

- path: "/root/gitea.url"
Expand All @@ -105,12 +105,12 @@ write_files:

mounts:
# Desired state configuration is in a blob container mounted as NFSv3
- ["{{storage_account_data_desired_state_name}}.blob.core.windows.net:/{{storage_account_data_desired_state_name}}/desiredstate", /desired_state, nfs, "ro,_netdev,sec=sys,vers=3,nolock,proto=tcp"]
- ["{{storage_account_data_desired_state_name}}.blob.core.windows.net:/{{storage_account_data_desired_state_name}}/desiredstate", /var/local/ansible, nfs, "ro,_netdev,sec=sys,vers=3,nolock,proto=tcp"]
# Secure data is in a blob container mounted as NFSv3
- ["{{storage_account_data_private_sensitive_name}}.blob.core.windows.net:/{{storage_account_data_private_sensitive_name}}/ingress", /data, nfs, "ro,_netdev,sec=sys,vers=3,nolock,proto=tcp"]
- ["{{storage_account_data_private_sensitive_name}}.blob.core.windows.net:/{{storage_account_data_private_sensitive_name}}/egress", /output, nfs, "rw,_netdev,sec=sys,vers=3,nolock,proto=tcp"]
- ["{{storage_account_data_private_sensitive_name}}.blob.core.windows.net:/{{storage_account_data_private_sensitive_name}}/ingress", /mnt/input, nfs, "ro,_netdev,sec=sys,vers=3,nolock,proto=tcp"]
- ["{{storage_account_data_private_sensitive_name}}.blob.core.windows.net:/{{storage_account_data_private_sensitive_name}}/egress", /mnt/output, nfs, "rw,_netdev,sec=sys,vers=3,nolock,proto=tcp"]
# User data is in a file share mounted as NFSv4
- ["{{storage_account_data_private_user_name}}.file.core.windows.net:/{{storage_account_data_private_user_name}}/shared", /shared, nfs, "_netdev,sec=sys,nconnect=4"]
- ["{{storage_account_data_private_user_name}}.file.core.windows.net:/{{storage_account_data_private_user_name}}/shared", /mnt/shared, nfs, "_netdev,sec=sys,nconnect=4"]
- ["{{storage_account_data_private_user_name}}.file.core.windows.net:/{{storage_account_data_private_user_name}}/home", /home, nfs, "_netdev,sec=sys,nconnect=4"]

# Add additional apt repositories
Expand Down Expand Up @@ -144,11 +144,11 @@ runcmd:
- echo ">=== Mounting all external volumes... ===<"
- grep -v -e '^[[:space:]]*$' /etc/fstab | sed 's|^| /etc/fstab |'
- mount -fav
- while (! mountpoint -q /data); do sleep 5; mount /data; done
- while (! mountpoint -q /desired_state); do sleep 5; mount /desired_state; done
- while (! mountpoint -q /mnt/input); do sleep 5; mount /mnt/input; done
- while (! mountpoint -q /var/local/ansible); do sleep 5; mount /var/local/ansible; done
- while (! mountpoint -q /home); do sleep 5; mount /home; done
- while (! mountpoint -q /output); do sleep 5; mount /output; done
- while (! mountpoint -q /shared); do sleep 5; mount /shared; done
- while (! mountpoint -q /mnt/output); do sleep 5; mount /mnt/output; done
- while (! mountpoint -q /mnt/shared); do sleep 5; mount /mnt/shared; done
- findmnt

# Enable and start desired state timer
Expand Down
10 changes: 5 additions & 5 deletions docs/source/deployment/security_checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -454,12 +454,12 @@ To minimise the risk of unauthorised access to the dataset while the ingress vol

### Turing configuration setting:

- Research users can write to the `/output` volume.
- A {ref}`role_system_manager` can view and download data in the `/output` volume via `Azure Storage Explorer`.
- Research users can write to the `/mnt/output` volume.
- A {ref}`role_system_manager` can view and download data in the `/mnt/output` volume via `Azure Storage Explorer`.

### Implication:

- SREs contain an `/output` volume, in which SRE users can store data designated for egress.
- SREs contain an `/mnt/output` volume, in which SRE users can store data designated for egress.

### Verify by:

Expand All @@ -469,7 +469,7 @@ To minimise the risk of unauthorised access to the dataset while the ingress vol
- Open up a file explorer and search for the various storage volumes

```{attention}
{{white_check_mark}} Verify that: the `/output` volume exists and can be read and written to.
{{white_check_mark}} Verify that: the `/mnt/output` volume exists and can be read and written to.
```

```{attention}
Expand All @@ -481,7 +481,7 @@ To minimise the risk of unauthorised access to the dataset while the ingress vol
- As the {ref}`role_system_manager`, follow the instructions in the [project manager documentation](../roles/project_manager/data_egress.md#data-egress-process) on how to access files set for egress with `Azure Storage Explorer`.

```{attention}
{{white_check_mark}} Verify that: you can see the files written to the `/output` storage volume.
{{white_check_mark}} Verify that: you can see the files written to the `/mnt/output` storage volume.
```

```{attention}
Expand Down
6 changes: 3 additions & 3 deletions docs/source/design/security/reference_configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,12 +102,12 @@ To minimise the risk of unauthorised access to the dataset while the ingress vol

### Turing configuration setting::

- Users can write to the `/output` volume.
- A {ref}`role_system_manager` can view and download data in the `/output` volume via **Azure Storage Explorer**.
- Users can write to the `/mnt/output` volume.
- A {ref}`role_system_manager` can view and download data in the `/mnt/output` volume via **Azure Storage Explorer**.

### Implication:

- SRE users can mark data as ready for egress approval by placing it in the `/output` volume.
- SRE users can mark data as ready for egress approval by placing it in the `/mnt/output` volume.

## 9. Software ingress

Expand Down
2 changes: 1 addition & 1 deletion docs/source/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ We have developed:
- A proposed default set of technical security measures for each tier.
- A set of infrastructure-as-code tools which will allow anyone to deploy their own isolated research environment.

If this sounds interesting to you, take a look at our GitHub releases: [![Data Safe Haven releases](https://img.shields.io/static/v1?label=Data%20Safe%20Haven&message=Releases&style=flat&logo=github)](https://github.com/alan-turing-institute/data-safe-haven/releases).
If this sounds interesting to you, take a look at our GitHub releases: [![Data Safe Haven releases](https://img.shields.io/static/v1?label=Data%20Safe%20Haven&message=Releases&style=flat&logo=github)](https://github.com/alan-turing-institute/mnt/input-safe-haven/releases).

## Documentation structure

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,10 +61,10 @@ They are common algorithms built into many operating systems, and included in th
You can generate a checksum file, which can be used to verify the integrity of files.
If you upload this file then researchers will be able to independently verify data integrity within the environment.

Here are instructions to generate a checksum file using the `md5sum` algorithm for a data set stored in a directory called `data`.
Here are instructions to generate a checksum file using the `md5sum` algorithm for a data set stored in the `input` directory.

```console
find ./data/ -type fl -exec md5sum {} + > hashes.txt
find ./mnt/input/ -type fl -exec md5sum {} + > hashes.txt
```

`find` searches the `data` directory for files and symbolic links (`-type fl`).
Expand Down
2 changes: 1 addition & 1 deletion docs/source/roles/researcher/accessing_the_sre.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,5 +175,5 @@ When you are connected to a workspace, you may switch to another by bringing up
:::

:::{tip}
Any files in the **/output/**, **/home/** or **/shared** folders on other workspaces will be available in this workspace too.
Any files in the **/mnt/output/**, **/home/** or **/mnt/shared** folders on other workspaces will be available in this workspace too.
:::
34 changes: 17 additions & 17 deletions docs/source/roles/researcher/using_the_sre.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,8 @@ For instance, describing in detail what a dataset contains and how it will be us
SREs are designed to be ephemeral and only deployed for as long as necessary.
It is likely that the infrastructure, and data, will be permanently deleted when work has concluded.

The `/output/` directory is designed for storing output to be kept after a project concludes.
You should move such data to the `/output/` directory and contact your designated contact about data egress.
The `/mnt/output/` directory is designed for storing output to be kept after a project concludes.
You should move such data to the `/mnt/output/` directory and contact your designated contact about data egress.

:::{important}
You are responsible for deciding what is worth archiving.
Expand All @@ -63,8 +63,8 @@ You are responsible for deciding what is worth archiving.
While working on the project:

- store all your code in a **Gitea** repository.
- store all resources that might be useful to the rest of the project in the **/shared/** folder.
- store anything that might form an output from the project (_e.g._ images, documents or output datasets) in the **/output/** folder.
- store all resources that might be useful to the rest of the project in the **/mnt/shared/** folder.
- store anything that might form an output from the project (_e.g._ images, documents or output datasets) in the **/mnt/output/** folder.

See {ref}`the section on sharing files <role_researcher_shared_storage>` to find out more about where to store your files.

Expand Down Expand Up @@ -244,45 +244,45 @@ Type `yes` to install the packages.

There are several shared folder on each workspace that all collaborators within a research project team can see and access:

- [input data](#input-data): in the **/data/** folder
- [shared space](#shared-space): in the **/shared/** folder
- [output resources](#output-resources): in the **/output/** folder
- [input data](#input-data): in the **/mnt/input/** folder
- [shared space](#shared-space): in the **/mnt/shared/** folder
- [output resources](#output-resources): in the **/mnt/output/** folder

### Input data

Data that has been approved and brought into the secure research environment can be found in the **/data/** folder.
Data that has been approved and brought into the secure research environment can be found in the **/mnt/input/** folder.

- The contents of **/data/** will be identical on all workspaces in your SRE.
- The contents of **/mnt/input/** will be identical on all workspaces in your SRE.
- Everyone working on your project will be able to access it.
- Everyone has **read-only access** to the files stored here.

If you are using the Data Safe Haven as part of an organised event, you might find additional resources in the **/data/** folder, such as example slides or document templates.
If you are using the Data Safe Haven as part of an organised event, you might find additional resources in the **/mnt/input/** folder, such as example slides or document templates.

:::{important}
You will not be able to change any of the files in **/data/**.
If you want to make derived datasets, for example cleaned and reformatted data, please add those to the **/shared/** or **/output/** folders.
You will not be able to change any of the files in **/mnt/input/**.
If you want to make derived datasets, for example cleaned and reformatted data, please add those to the **/mnt/shared/** or **/mnt/output/** folders.
:::

### Shared space

The **/shared/** folder should be used for any work that you want to share with your group.
The **/mnt/shared/** folder should be used for any work that you want to share with your group.

- The contents of **/shared/** will be identical on all workspaces in your SRE.
- The contents of **/mnt/shared/** will be identical on all workspaces in your SRE.
- Everyone working on your project will be able to access it
- Everyone has **read-and-write access** to the files stored here.

### Output resources

Any outputs that you want to extract from the secure environment should be placed in the **/output/** folder on the workspace.
Any outputs that you want to extract from the secure environment should be placed in the **/mnt/output/** folder on the workspace.

- The contents of **/output/** will be identical on all workspaces in your SRE.
- The contents of **/mnt/output/** will be identical on all workspaces in your SRE.
- Everyone working on your project will be able to access it
- Everyone has **read-and-write access** to the files stored here.

Anything placed in here will be considered for data egress - removal from the secure research environment - by the project's principal investigator together with the data provider.

:::{tip}
You may want to consider having subfolders of **/output/** to make the review of this directory easier.
You may want to consider having subfolders of **/mnt/output/** to make the review of this directory easier.
:::

## {{pill}} Version control using Gitea
Expand Down
2 changes: 1 addition & 1 deletion tests/infrastructure/programs/sre/test_workspaces.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,6 @@ def test_template_cloudinit(self):
)

assert (
'- ["storageaccount.blob.core.windows.net:/storageaccount/desiredstate", /desired_state, nfs, "ro,'
'- ["storageaccount.blob.core.windows.net:/storageaccount/desiredstate", /var/local/ansible, nfs, "ro,'
in cloudinit
)
Loading