From db934dff4e03cb68090550f9e89bf9664cad6977 Mon Sep 17 00:00:00 2001 From: Davsarper <118986872+Davsarper@users.noreply.github.com> Date: Thu, 4 Apr 2024 11:53:54 +0100 Subject: [PATCH 01/12] Create ProgressReport-Feb24-Aug24.md --- Reports/ProgressReport-Feb24-Aug24.md | 301 ++++++++++++++++++++++++++ 1 file changed, 301 insertions(+) create mode 100644 Reports/ProgressReport-Feb24-Aug24.md diff --git a/Reports/ProgressReport-Feb24-Aug24.md b/Reports/ProgressReport-Feb24-Aug24.md new file mode 100644 index 0000000..6be758f --- /dev/null +++ b/Reports/ProgressReport-Feb24-Aug24.md @@ -0,0 +1,301 @@ +# Data Safe Haven project report + +**Period: [February 2023-January 2024]** + +This document contains a summary of progress across all stories in the [project roadmap](https://github.com/orgs/alan-turing-institute/projects/111/views/1). +It maps stories according to the (main) pillar and priority they contribute to. + +Included stories are those that were scheduled and/or prioritised over this period (February 2023-January 2024), some not scheduled stories are included when there are updates for them regardless (indirect contributions from other stories, relevant work recently started, or something to report in general). + +## Codebase development +Running projects working with sensitive data safely +Running cutting edge data science projects effectively + + +### Manage codebase releases and testing: [#50](https://github.com/alan-turing-institute/data-safe-haven-team/issues/50) + + +Contributes to: +- Running projects working with sensitive data safely +- Running cutting edge data science projects effectively + +#### Goal +Support for deployments of the Data Safe Haven at Turing and beyond + +#### Progress +- Pen testing done: little found +- Penetration tested arranged and will be done in September +- Preparation for release v4.1.0: Deployment of different SRE variants, Security checklist +- Reviewing v4.1.0: No significant problems is deployment logs, Problems found in security checklist relating to MSRDS +- Working on [Release 4.1.0](https://github.com/alan-turing-institute/data-safe-haven/issues/1544): fixes bugs and introduces necessary updates + + +### Codebase maintenance: [#47](https://github.com/alan-turing-institute/data-safe-haven-team/issues/47) + +Contributes to: +- Running projects working with sensitive data safely +- Running cutting edge data science projects effectively + +#### Goal +Ensure that codebase is kept up-to-date with bug fixes, security updates, external API changes etc. +- Ensure that DSH code is always deployable +- Ensure that known security issues are remediated/minimised as soon as possible +- Ensure that documentation is up-to-date with code base + +#### Progress + +This year saw the [4.1.0 release](https://github.com/alan-turing-institute/data-safe-haven/releases/tag/v4.1.0) with notable fixes, security and documentaiton updates. + +We are currently working towards a final PowerShell version ([Release 4.2.0 Milestone](https://github.com/alan-turing-institute/data-safe-haven/milestone/21)). +The aim is to make the release in time for next DSG and then focus exclusively on Pulumi. + +A more complete summary of work done is available in the story, but some key updates through the year are: +- Database permission issue debugging as PostgreSQL user privileges were not correctly applied. + - Now fixed in PR [#1708](https://github.com/alan-turing-institute/data-safe-haven/pull/1708). +- ClamAV On-access was not running. + - On-access virus scanning is a DSPT requirement, and this process was not running correctly. + - Fixed by PR [#1725](https://github.com/alan-turing-institute/data-safe-haven/pull/1725). +- Improve handling of file paths PR [#1705](https://github.com/alan-turing-institute/data-safe-haven/pull/1705). +- Investigating issues with Julia on AMD processors: + - During the building of VM images for deployment in SREs, Julia created and stored compiled versions of packages that were suitable only for Intel systems, causing crashes when users wanted to use AMD systems. +- Investigating issues with DBeaver on Tier 2+ SREs: + - DBeaver drivers were not installing correctly during VM building, so it tries to download them from the internet. + No problem on T1, but fails on T2. +- Factoring storage creation and account deployments out of main deployment script now allows for a more resilient process (not having to re-run everything when one fails) +- MS changed the name of Azure Directory to Microsoft Entra ID which made necessary to spend time updating documentation and code +- Factor SHM storage creation out of main deployment script: PR [#1673](https://github.com/alan-turing-institute/data-safe-haven/pull/1673) +- Add all contributors table to project README and docs: PR[#1649](https://github.com/alan-turing-institute/data-safe-haven/pull/1649). +- Removal of MSRDS (PR: [#1535](https://github.com/alan-turing-institute/data-safe-haven/pull/1535)) which reduces support burden and codebase complexity, instead Guacamole implementation is more robust and secure. +- Removal of CoCalc (PR: [#1554](https://github.com/alan-turing-institute/data-safe-haven/pull/1554) Reduces support burden for future releases by removing a largely unused feature. +- Drop Microsoft Remote Desktop: (PR: [#1159](https://github.com/alan-turing-institute/data-safe-haven/issues/1159) primarily for increased security as it shows more issues than Guacamole, in doing this several other open issues are resolved. +- Documentation improvements and updates + +### Identify and implement core IAC changes: [#28](https://github.com/alan-turing-institute/data-safe-haven-team/issues/28) + +#### Goal +Make DSH deployment more robust and development easier through using IAC and configuration management. +- Take advantage of IAC and configuration management in the DSH codebase which will + - Make deployments faster + - Make deployments more reliable + - Make development easier +- Move away from non-idempotent, bespoke scripts (Powershell, bash, cloud-init) + +##### Definition of done +On the release of a new major version which removes legacy, script-based deployment. + +#### Progress +Arrived at a IAC MVP version of the code, available as a penetration tested [pre-release](https://github.com/alan-turing-institute/data-safe-haven/releases/tag/v5.0.0-rc.1). + +This new code is better and easier for users to deploy, however some incompatibilities with the old code would require extensive work. +As PowerShell heads to its final release it was decided not to work on fixing these. + +Since finishing migration a lot of the work is focusing on structuring the code, small improvements aimed at the user experience and robustness. + +These are main references and milestones, a more complete list is available on the [story issue](https://github.com/alan-turing-institute/data-safe-haven-team/issues/28) +- Codebase pre-release https://github.com/alan-turing-institute/data-safe-haven/releases/tag/v5.0.0-rc.1 +- Next version milestone https://github.com/alan-turing-institute/data-safe-haven/milestone/20 + +## Information governance & standards +Infrastructure adhering to the latest agreed upon standard +Identifying, co-creating and supporting a TRE standard used across TRE infrastructures + +### Co-create a TRE standard (SATRE): [#23](https://github.com/alan-turing-institute/data-safe-haven-team/issues/23) + +Contributes to: +- Identifying, co-creating and supporting a TRE standard used across TRE infrastructures + +#### Goal +Develop the SATRE specification that UK TREs can evaluate themselves against + +##### Definition of done +SATRE specification published + +#### Progress +[SATRE specification](https://satre-specification.readthedocs.io/en/stable/) published, and available for contribution and reproducibility in its [open repository](https://github.com/sa-tre/satre-specification). + +Currently Turing and HIC have self-evaluated against it and evaluations are available openly, several conversations ongoing about other institutions doing the same and making them available. + +Organisations that have declared they are going through the self evaluation (at the time of reporting): +- KCL +- Sheffield +- North West NHS SNSDE +- NHS SDE technology group reviewing SATRE + +- Outputs: + - SATRE specification V1 and [associated technical paper](https://zenodo.org/records/10053383) + - Internal DARE report, link will be added and published openly + - [User report](https://zenodo.org/records/10066800): characterises users, with a wider notion than we started of what users are. Lays the fundation for futher usability work, going beyong technicla features and into training and documentation + - UK TRE Community - SATRE WG: the ongoing work and evolution of the specification is now a working group within the UK TRE Community + +### Documentation management [#32](https://github.com/alan-turing-institute/data-safe-haven-team/issues/32) + +Contributes to: +- Identifying, co-creating and supporting a TRE standard used across TRE infrastructures +- Creating resources for all stakeholders (inc. Citizens) to engage in the TRE conversation +- Identifying and documenting everything that can be openly documented + +#### Goal +Comprehensive and clear documentation for the DSH, SATRE & TRESA will ensure open, reproducible outputs from this project. +- Ensure all relevant information is captured in documentation +- Test accessibility and discoverability of docs with relevant groups +- Iterate documentation in line with wider project work (e.g. TRESA processes, DSH updates) +- Determine what can/can't be documented (e.g. from an IG perspective). + +#### Definition of Done +When funding ends for the project and we have openly documented everything that we feel we can + +#### Progress +Documentation management have not been a story actively worked in, yet some processes have been updated and documented and needs identified. +While the story goes beyond Production processes it is worth noting that those have been handed over to TRESA, who are already suggesting and appliying changes. + +### SATRE: stakeholder engagement and community buy out [#66](https://github.com/alan-turing-institute/data-safe-haven-team/issues/66) + +#### Goal +Ensure that institutions evaluate themselves against SATRE and that the momentum is maintained between funded phases + +At the end of the funded phase of SATRE there was a growing community interest, with institutions and stakeholders affirming they would evaluate themselves against it and contribute feedback. + +Without ongoing resources the necessary support to ensure that happens cannot be provided and SATRE may end up not being adopted. + +#### Definition of Done +There are a number of self evlauaitons completed, there is feedback on the spec repository and, ideally, there is an active WG within the TRE Community continuing to work on it. + +#### Progress +Work is folded into the [UK TRE Community](https://github.com/alan-turing-institute/data-safe-haven-team/issues/52), already SATRE is formally becoming a UK TRE Community WG. + +## Community building +Creating resources for all stakeholders (inc. members of the public) to engage in the TRE conversation +Creating and maintaining open and active communication spaces & workspaces (Slack, GH) +Identifying and documenting everything that can be openly documented + +### Stakeholder landscape review [#30](https://github.com/alan-turing-institute/data-safe-haven-team/issues/30) + +Contributes to: +- Creating resources for all stakeholders (inc. Citizens) to engage in the TRE conversation +- Creating and maintaining open and active communication spaces & workspaces (Slack, GH) + +#### Goal +Across a lot of our work (DSH project, SATRE, UK TRE community) there has been a lot of discussion around who the impacted parties are, how they are categorised, what their interests/needs are etc. + +An effective stakeholder map showing all parties we think we should engage with will help us prioritise who to collaborate with, and strengthen our work in community building within the TRE space (which is kind of where this project is heading, above and beyond getting others to use the DSH). + +- Brainstorm and identify potential stakeholder groups +- Engage different groups through interviews/workshops to better understand them +- Create engagement pipeline & priority for different groups + +#### Definition of Done +- When we have an intended end-output from engagement with our established groups (e.g. by the end of the project, we want X group to be part of the UK TRE community, we want Y group to have contributed to the DSH repo...) + +#### Progress +The work done in [SATRE](https://github.com/alan-turing-institute/data-safe-haven-team/issues/23) and the [UK TRE Community](https://github.com/alan-turing-institute/data-safe-haven-team/issues/52) have directly contributed to estbalishing a relationship with key stakeholders, identify and characterise them. + +Direct and explicit work on this story have not been carried out (not scheduled in this period). + +We are now establishing the engagement pipleline by creating an internal CRM (sharepoint based currently). + +### UK TRE Community leadership [#52](https://github.com/alan-turing-institute/data-safe-haven-team/issues/52) + +Contributes to: +- Creating resources for all stakeholders (inc. Citizens) to engage in the TRE conversation +- Creating and maintaining open and active communication spaces & workspaces (Slack, GH) + +#### Goal +Provide a space for those involved in building, using and responsible for governance of TREs to discuss and recommend best practices. +- Host online working spaces, events and workshops to support the UK TRE Community +- Share best practices i.e. for making radiology data available for researchers +- Empower the community to help influence policy decisions + +#### Progress +A lot of of effort has been put this year into the UK TRE Community, this year has seen the community mature and evolve from the original RSE TRE Community. +Currently we are delivering a funded project to ensure its sustainability which focuses one establishing the necessary spaces and governance processes. + +All work and progress during the funded phase can be consulted on [the community project board](https://github.com/orgs/uk-tre/projects/1) + +- The UK TRE Community event was held in Swansea on 4 September as a RSEcon23 satellite event. The event was very well attended with around 90 people in person and 50 online (figure to be revised), attendees were active on the day and had very positive feedback. + - [Report and notes](https://www.uktre.org/en/latest/events/wg_workshops/2023-09-04-september-meeting/index.html) +- We were awarded the DARE UK Community call. It was prepared and submitted to DARE UK community call, with full agreement and participation of the community itself +- Celebrated December community event (virtual). We presented the plans and work within this funded phase, which included a vision and mission for the community + - [Report and notes](https://www.uktre.org/en/latest/events/wg_workshops/2023-12-05-december-meeting/index.html) +- The first version of the Community Website is ready, it has been done using Hugo to balance quality and sustainability (being easily maintained and updated by the community after funding ends) +- Governance processes are being established, striving for simplicity in this phase. The conversation and work is open and welcomes all input and feedback. This is the [most active issue](https://github.com/uk-tre/community-management/pull/54) and a good starting point + +### Communication and outreach [#35](https://github.com/alan-turing-institute/data-safe-haven-team/issues/35) + +Contributes to: +- Creating and maintaining open and active communication spaces & workspaces (Slack, GH) +- Creating resources for all stakeholders (inc. Citizens) to engage in the TRE conversation + +#### Goal +- Supporting the user community of the DSH codebase +- Publicising our work via blogposts, reports or papers +- Communicating our work through conference/workshop talks or posters + +#### Progress +This story has changed in scope along the year, work done here has been that of presenting DSH externally via events and talks. + +Yet we have identified work to be done within this story to define and establish a DSH community and user base, what this means and entails needs to be discussed and agreed yet. + +- We have established a shared slack channel with UCL to discuss common approaches to information governance processes +- [AI UK demonstration proposal](https://thealanturininstitute.sharepoint.com/:x:/s/SafeHaven/EfKD3w8Gi9NFv6JBshOkugsBOnn4v3ZdU-FTeIcy5obQcg?e=Rhq9w4) + - [proposal collaborative note](https://hackmd.io/AmcYdsyETU2dVgtIdfVL-g) + - We are following a similar format to last year but want to bring forward the community work and the satre specification. We want to have an interactive activity that blends role playing the different stakeholder groups and collectively deciding on specification features. The demo challenge last year did not work so the technical side this year will be demonstrated by a video and project members "touring" the repositories, docs and environment +- Met with Nottingham to support them as users of the DSH codebase +- The team worked together on the content for DSH activities on RSEcon as well as the UK TRE community satellite event #46 . +- The team visited the Bennett Institute for a show and tell about DSH and OpenSafely https://github.com/alan-turing-institute/tps-project-management/issues/157 + - No immediate collaboration but agreed to be involved in the specification and TRE community +- For discussion: this story needs to be redefined to include external engagement or a new one created + +## TRESA +Over the year TRESA have increased its autonomy from the DSH research project, in terms of work ownership and management. +Therefore TRESA stories have not been independently updated and it is more comprehensive to update on the service area as a whole. + +This warrants updating and reviewing the stories we keep under the DSH roadmap, focusing on communicating with the service area rather than planning or prioritising for them. + +### Updates +A new team have been onboarded. +They have focused on learning the ropes while simultaneously supporting the December DSG. +With some support from the DSH team they were able to do so. + +We had a detailed [handover meeting](https://hackmd.io/kh6siuZcTdCxcfYryAvypw) to go over each process, the idea is that moving forward TRESA directly owns processes and their improvement. + +We have also identified areas of work for TRESA to fully establish itself as a service area, these need to be elaborated into a full proposal with sprints and stories but main areas are: +- Data protection +- Cost and recovery model +- Acredditation: ISO 27001 and DSPT renewal +- Client projects management + + +## Project management and strategy +Work and stories that do not belong directly in any pillars but are necessary for all + +### Project strategy and ways of working [#43](https://github.com/alan-turing-institute/data-safe-haven-team/issues/43) + +#### Goal +The aim is to develop a project strategy and revise best ways of work to achieve it + +Through several strategy sessions we will: +- Define our north star (vision & mision) +- Establish the project pilars or areas, defining what success looks like for each +- Prioritise the measure of success, which are essential to consider the project succesful +- Identify work required to achieve success +- Allocate work by team +- Produce an initial roadmap +- Evaluate required effort for the work against team capacity +- Develop and agree new ways of work, including meeting structure and use of project's repositories and projects + +#### Progress + +##### Strategy + +Through several team wide sessions we jointly produced a [project strategy](https://thealanturininstitute.sharepoint.com/:p:/s/SafeHaven/Ebrp4Iyc9M1NpPTgpgHdj5kB7HPvH-2gM0oNd97jJu6oxw?e=eN0ZFw)https://thealanturininstitute.sharepoint.com/:p:/s/SafeHaven/Ebrp4Iyc9M1NpPTgpgHdj5kB7HPvH-2gM0oNd97jJu6oxw?e=eN0ZFw a long, medium and short term levels. + +This resulting in a clear Vision & Mision that have allowed internal alignment and improved external communications + +>To remove barriers to working safely and effectively with sensitive data, +by promoting and demonstrating a culture of open, community-led development +of interoperable foundational infrastructure and governance. + +We also agreed the pillars of the project and established a [roadmap](https://github.com/orgs/alan-turing-institute/projects/111/views/1) of the necessary work for success. + +##### Ways of working +Throughout this year we have also iterated our ways of working which are openly available on GitHub in [WaysofWork.md](https://github.com/alan-turing-institute/data-safe-haven-team/blob/main/WaysofWork.md), they are focused in increased communication and work prioritisation. From 4c4f353867d323bc890bf458556584c3d0954009 Mon Sep 17 00:00:00 2001 From: Davsarper <118986872+Davsarper@users.noreply.github.com> Date: Mon, 8 Apr 2024 08:55:37 +0100 Subject: [PATCH 02/12] Update ProgressReport-Feb24-Aug24.md initial updates --- Reports/ProgressReport-Feb24-Aug24.md | 207 +++++++------------------- 1 file changed, 52 insertions(+), 155 deletions(-) diff --git a/Reports/ProgressReport-Feb24-Aug24.md b/Reports/ProgressReport-Feb24-Aug24.md index 6be758f..bd3cdda 100644 --- a/Reports/ProgressReport-Feb24-Aug24.md +++ b/Reports/ProgressReport-Feb24-Aug24.md @@ -1,11 +1,13 @@ # Data Safe Haven project report -**Period: [February 2023-January 2024]** +**Period: [February 2024-August 2024]** + +This update: 8 April 2024 This document contains a summary of progress across all stories in the [project roadmap](https://github.com/orgs/alan-turing-institute/projects/111/views/1). It maps stories according to the (main) pillar and priority they contribute to. -Included stories are those that were scheduled and/or prioritised over this period (February 2023-January 2024), some not scheduled stories are included when there are updates for them regardless (indirect contributions from other stories, relevant work recently started, or something to report in general). +Included stories are those that were scheduled and/or prioritised over this period, some not scheduled stories are included when there are updates for them regardless (indirect contributions from other stories, relevant work recently started, or something to report in general). ## Codebase development Running projects working with sensitive data safely @@ -23,11 +25,13 @@ Contributes to: Support for deployments of the Data Safe Haven at Turing and beyond #### Progress -- Pen testing done: little found -- Penetration tested arranged and will be done in September -- Preparation for release v4.1.0: Deployment of different SRE variants, Security checklist -- Reviewing v4.1.0: No significant problems is deployment logs, Problems found in security checklist relating to MSRDS -- Working on [Release 4.1.0](https://github.com/alan-turing-institute/data-safe-haven/issues/1544): fixes bugs and introduces necessary updates + +Having completed development of v4.2.0 we prepared the release which included preparing a release branch and deployment in an environment for pen testing. + +Extensive time was allocated to deploying and the errors/bugs that arose, as well as preparing for pen testing (this included deployment but also requesting specific tests like the removal of certain hardcoded IPs) + +Pen testing was arranged and carried out, managing to spend within the 2023-2024 FY. iStorm kept better communications than last time and did not find concerning issues. +- @craddm would you add here a bit on the results? ### Codebase maintenance: [#47](https://github.com/alan-turing-institute/data-safe-haven-team/issues/47) @@ -44,31 +48,19 @@ Ensure that codebase is kept up-to-date with bug fixes, security updates, extern #### Progress -This year saw the [4.1.0 release](https://github.com/alan-turing-institute/data-safe-haven/releases/tag/v4.1.0) with notable fixes, security and documentaiton updates. - -We are currently working towards a final PowerShell version ([Release 4.2.0 Milestone](https://github.com/alan-turing-institute/data-safe-haven/milestone/21)). -The aim is to make the release in time for next DSG and then focus exclusively on Pulumi. - -A more complete summary of work done is available in the story, but some key updates through the year are: -- Database permission issue debugging as PostgreSQL user privileges were not correctly applied. - - Now fixed in PR [#1708](https://github.com/alan-turing-institute/data-safe-haven/pull/1708). -- ClamAV On-access was not running. - - On-access virus scanning is a DSPT requirement, and this process was not running correctly. - - Fixed by PR [#1725](https://github.com/alan-turing-institute/data-safe-haven/pull/1725). -- Improve handling of file paths PR [#1705](https://github.com/alan-turing-institute/data-safe-haven/pull/1705). -- Investigating issues with Julia on AMD processors: - - During the building of VM images for deployment in SREs, Julia created and stored compiled versions of packages that were suitable only for Intel systems, causing crashes when users wanted to use AMD systems. -- Investigating issues with DBeaver on Tier 2+ SREs: - - DBeaver drivers were not installing correctly during VM building, so it tries to download them from the internet. - No problem on T1, but fails on T2. -- Factoring storage creation and account deployments out of main deployment script now allows for a more resilient process (not having to re-run everything when one fails) -- MS changed the name of Azure Directory to Microsoft Entra ID which made necessary to spend time updating documentation and code -- Factor SHM storage creation out of main deployment script: PR [#1673](https://github.com/alan-turing-institute/data-safe-haven/pull/1673) -- Add all contributors table to project README and docs: PR[#1649](https://github.com/alan-turing-institute/data-safe-haven/pull/1649). -- Removal of MSRDS (PR: [#1535](https://github.com/alan-turing-institute/data-safe-haven/pull/1535)) which reduces support burden and codebase complexity, instead Guacamole implementation is more robust and secure. -- Removal of CoCalc (PR: [#1554](https://github.com/alan-turing-institute/data-safe-haven/pull/1554) Reduces support burden for future releases by removing a largely unused feature. -- Drop Microsoft Remote Desktop: (PR: [#1159](https://github.com/alan-turing-institute/data-safe-haven/issues/1159) primarily for increased security as it shows more issues than Guacamole, in doing this several other open issues are resolved. -- Documentation improvements and updates +Have worked on updating software used within SREs to ensure the security and functionality of the environment: +- Guacamole server updated [PR](https://github.com/alan-turing-institute/data-safe-haven/pull/1741) +- Nexus server updated [PR](https://github.com/alan-turing-institute/data-safe-haven/pull/1744) +- CodiMD server updated [PR](https://github.com/alan-turing-institute/data-safe-haven/pull/1743) + +Added and tested a script to handle SAS access tokens renewal, currently expiring yearly. These are required manage access to data storage (and therefore ingress and egress). The relevant PR is here https://github.com/alan-turing-institute/data-safe-haven/pull/1739. In the process we realised SAS tokens are bound to Store Access Policies which could be modified to have no end date, we are currently considering the covenience of this approach versus potential security issues in https://github.com/alan-turing-institute/data-safe-haven/issues/1751 . + +Improved use of hardcoded domain names and IPs. The hardcoded lists are difficult to maintain and are prone to going out of date, despite not fully fixing the use of these improvements have been made for the 4.2.0 release by relaxing rules where security allows. For this the team checked individuals cases and applied where possible, no security issues where found and we added this as a specific thing to pent test. Related PR is https://github.com/alan-turing-institute/data-safe-haven/pull/1745 and explanatory issue is https://github.com/alan-turing-institute/data-safe-haven/issues/1549 . + +An issue with Jupyter notebooks not being able to use Python when launched from the menu was found, despite extensive work a fix was not found and decided to let it be by documenting the right workaround: launching Jupyter Notebooks from the terminal. The issue is https://github.com/alan-turing-institute/data-safe-haven/issues/1584 . + +Worked on updating documentation to reflect Azure Active Directory name change to Microsoft Entra. + ### Identify and implement core IAC changes: [#28](https://github.com/alan-turing-institute/data-safe-haven-team/issues/28) @@ -84,69 +76,15 @@ Make DSH deployment more robust and development easier through using IAC and con On the release of a new major version which removes legacy, script-based deployment. #### Progress -Arrived at a IAC MVP version of the code, available as a penetration tested [pre-release](https://github.com/alan-turing-institute/data-safe-haven/releases/tag/v5.0.0-rc.1). -This new code is better and easier for users to deploy, however some incompatibilities with the old code would require extensive work. -As PowerShell heads to its final release it was decided not to work on fixing these. +Over this period the story has not been prioritised/resourced as much, focusing on identifying and scoping future work which will be a priority after v4.2 release. The DSH code repo contains milestones that reflect related and planned issues (V5.x milestones) https://github.com/alan-turing-institute/data-safe-haven/milestones. -Since finishing migration a lot of the work is focusing on structuring the code, small improvements aimed at the user experience and robustness. - -These are main references and milestones, a more complete list is available on the [story issue](https://github.com/alan-turing-institute/data-safe-haven-team/issues/28) -- Codebase pre-release https://github.com/alan-turing-institute/data-safe-haven/releases/tag/v5.0.0-rc.1 -- Next version milestone https://github.com/alan-turing-institute/data-safe-haven/milestone/20 +Additionally a first version of the codebase roadmap has been put in place, linking to milestone but also recording potential desired features that are not currently planned to implement. A roadmap is a desired/required document to have in place for software funding opportunities https://github.com/alan-turing-institute/data-safe-haven/blob/develop/ROADMAP.md. ## Information governance & standards Infrastructure adhering to the latest agreed upon standard Identifying, co-creating and supporting a TRE standard used across TRE infrastructures -### Co-create a TRE standard (SATRE): [#23](https://github.com/alan-turing-institute/data-safe-haven-team/issues/23) - -Contributes to: -- Identifying, co-creating and supporting a TRE standard used across TRE infrastructures - -#### Goal -Develop the SATRE specification that UK TREs can evaluate themselves against - -##### Definition of done -SATRE specification published - -#### Progress -[SATRE specification](https://satre-specification.readthedocs.io/en/stable/) published, and available for contribution and reproducibility in its [open repository](https://github.com/sa-tre/satre-specification). - -Currently Turing and HIC have self-evaluated against it and evaluations are available openly, several conversations ongoing about other institutions doing the same and making them available. - -Organisations that have declared they are going through the self evaluation (at the time of reporting): -- KCL -- Sheffield -- North West NHS SNSDE -- NHS SDE technology group reviewing SATRE - -- Outputs: - - SATRE specification V1 and [associated technical paper](https://zenodo.org/records/10053383) - - Internal DARE report, link will be added and published openly - - [User report](https://zenodo.org/records/10066800): characterises users, with a wider notion than we started of what users are. Lays the fundation for futher usability work, going beyong technicla features and into training and documentation - - UK TRE Community - SATRE WG: the ongoing work and evolution of the specification is now a working group within the UK TRE Community - -### Documentation management [#32](https://github.com/alan-turing-institute/data-safe-haven-team/issues/32) - -Contributes to: -- Identifying, co-creating and supporting a TRE standard used across TRE infrastructures -- Creating resources for all stakeholders (inc. Citizens) to engage in the TRE conversation -- Identifying and documenting everything that can be openly documented - -#### Goal -Comprehensive and clear documentation for the DSH, SATRE & TRESA will ensure open, reproducible outputs from this project. -- Ensure all relevant information is captured in documentation -- Test accessibility and discoverability of docs with relevant groups -- Iterate documentation in line with wider project work (e.g. TRESA processes, DSH updates) -- Determine what can/can't be documented (e.g. from an IG perspective). - -#### Definition of Done -When funding ends for the project and we have openly documented everything that we feel we can - -#### Progress -Documentation management have not been a story actively worked in, yet some processes have been updated and documented and needs identified. -While the story goes beyond Production processes it is worth noting that those have been handed over to TRESA, who are already suggesting and appliying changes. ### SATRE: stakeholder engagement and community buy out [#66](https://github.com/alan-turing-institute/data-safe-haven-team/issues/66) @@ -161,37 +99,18 @@ Without ongoing resources the necessary support to ensure that happens cannot be There are a number of self evlauaitons completed, there is feedback on the spec repository and, ideally, there is an active WG within the TRE Community continuing to work on it. #### Progress -Work is folded into the [UK TRE Community](https://github.com/alan-turing-institute/data-safe-haven-team/issues/52), already SATRE is formally becoming a UK TRE Community WG. + +NHS-E R&D Programme Director stated that SATRE has become the reference framework for TREs within the Subnational SDE programme, placing SATRE as a key reference for what TREs are. + +We have also seen how institutions that already have other accreditations opt to self evaluate with SATRE because it is the only one specific to TREs. + +SATRE has continued to atract the interest of the community, with many attendees to its meeting wanting to be involved. ## Community building Creating resources for all stakeholders (inc. members of the public) to engage in the TRE conversation Creating and maintaining open and active communication spaces & workspaces (Slack, GH) Identifying and documenting everything that can be openly documented -### Stakeholder landscape review [#30](https://github.com/alan-turing-institute/data-safe-haven-team/issues/30) - -Contributes to: -- Creating resources for all stakeholders (inc. Citizens) to engage in the TRE conversation -- Creating and maintaining open and active communication spaces & workspaces (Slack, GH) - -#### Goal -Across a lot of our work (DSH project, SATRE, UK TRE community) there has been a lot of discussion around who the impacted parties are, how they are categorised, what their interests/needs are etc. - -An effective stakeholder map showing all parties we think we should engage with will help us prioritise who to collaborate with, and strengthen our work in community building within the TRE space (which is kind of where this project is heading, above and beyond getting others to use the DSH). - -- Brainstorm and identify potential stakeholder groups -- Engage different groups through interviews/workshops to better understand them -- Create engagement pipeline & priority for different groups - -#### Definition of Done -- When we have an intended end-output from engagement with our established groups (e.g. by the end of the project, we want X group to be part of the UK TRE community, we want Y group to have contributed to the DSH repo...) - -#### Progress -The work done in [SATRE](https://github.com/alan-turing-institute/data-safe-haven-team/issues/23) and the [UK TRE Community](https://github.com/alan-turing-institute/data-safe-haven-team/issues/52) have directly contributed to estbalishing a relationship with key stakeholders, identify and characterise them. - -Direct and explicit work on this story have not been carried out (not scheduled in this period). - -We are now establishing the engagement pipleline by creating an internal CRM (sharepoint based currently). ### UK TRE Community leadership [#52](https://github.com/alan-turing-institute/data-safe-haven-team/issues/52) @@ -206,18 +125,20 @@ Provide a space for those involved in building, using and responsible for govern - Empower the community to help influence policy decisions #### Progress -A lot of of effort has been put this year into the UK TRE Community, this year has seen the community mature and evolve from the original RSE TRE Community. -Currently we are delivering a funded project to ensure its sustainability which focuses one establishing the necessary spaces and governance processes. -All work and progress during the funded phase can be consulted on [the community project board](https://github.com/orgs/uk-tre/projects/1) +Funded phase came to an end on 31 March, along reporting it is necessary to organise and put together what we have produced but we have: +- Created a first version of all governance documents. Some will include pending conversations that may turn in the next version of those documents, for example the endorsement of outputs (v1 will only have community approval). This is the issue referencing to all documents https://github.com/orgs/uk-tre/projects/1/views/1?pane=issue&itemId=53738648 +- Created a new website, not launched yet. +- Created a public community calendar, for "official" events and all those happening in the space and that people may want to include and promote. + +We celebrated the March quarterly event with very positive feedback from participants. Keynote was delivered by NHS England R&D programme Director on the subnaitonal SDEs programme, this followed a request by community members to know more about it during the December event. Full notes can be found here while we produce a report https://hackmd.io/6t-s4x-wQN6riL6gPJLuWg + +With funding ending we received a community request/push to ensure we put the workshop budget to good use. In response we elaborated two events proposals that align with the goal of this grant: finalising governance docs with the community participation and set up WGs for success. +- Documentation sprints (https://hackmd.io/ty_bEhtBTrKjmYsJEEP8Qw) +- Working groups day (https://hackmd.io/nTAAo7fOT_quE18y0NYjnA). Which will be held on 29 April (This also implied figuring out a way to spend beyond March given it is not feasible to run the WGs day effectively before late April). + +Lastly we have been workign with Scriberia to use the remaining funding to produce an illustration that can represent the community in a number of ways, the most immediate one being the production of physical stand up banners for events (due to timeline DSH will cover the actual banner while DARE funds have covered the illustration itself). -- The UK TRE Community event was held in Swansea on 4 September as a RSEcon23 satellite event. The event was very well attended with around 90 people in person and 50 online (figure to be revised), attendees were active on the day and had very positive feedback. - - [Report and notes](https://www.uktre.org/en/latest/events/wg_workshops/2023-09-04-september-meeting/index.html) -- We were awarded the DARE UK Community call. It was prepared and submitted to DARE UK community call, with full agreement and participation of the community itself -- Celebrated December community event (virtual). We presented the plans and work within this funded phase, which included a vision and mission for the community - - [Report and notes](https://www.uktre.org/en/latest/events/wg_workshops/2023-12-05-december-meeting/index.html) -- The first version of the Community Website is ready, it has been done using Hugo to balance quality and sustainability (being easily maintained and updated by the community after funding ends) -- Governance processes are being established, striving for simplicity in this phase. The conversation and work is open and welcomes all input and feedback. This is the [most active issue](https://github.com/uk-tre/community-management/pull/54) and a good starting point ### Communication and outreach [#35](https://github.com/alan-turing-institute/data-safe-haven-team/issues/35) @@ -231,19 +152,10 @@ Contributes to: - Communicating our work through conference/workshop talks or posters #### Progress -This story has changed in scope along the year, work done here has been that of presenting DSH externally via events and talks. -Yet we have identified work to be done within this story to define and establish a DSH community and user base, what this means and entails needs to be discussed and agreed yet. +AI UK has been a priority for this story and the project. Held on 19-20 March DSH had its own stand, through a likert scale exercise on TREs (and a bowl of sweets) we engaged attendees to introduce them to TREs and the work of the project which we stressed to be not only an open codebase but also the governance & standards and the community. -- We have established a shared slack channel with UCL to discuss common approaches to information governance processes -- [AI UK demonstration proposal](https://thealanturininstitute.sharepoint.com/:x:/s/SafeHaven/EfKD3w8Gi9NFv6JBshOkugsBOnn4v3ZdU-FTeIcy5obQcg?e=Rhq9w4) - - [proposal collaborative note](https://hackmd.io/AmcYdsyETU2dVgtIdfVL-g) - - We are following a similar format to last year but want to bring forward the community work and the satre specification. We want to have an interactive activity that blends role playing the different stakeholder groups and collectively deciding on specification features. The demo challenge last year did not work so the technical side this year will be demonstrated by a video and project members "touring" the repositories, docs and environment -- Met with Nottingham to support them as users of the DSH codebase -- The team worked together on the content for DSH activities on RSEcon as well as the UK TRE community satellite event #46 . -- The team visited the Bennett Institute for a show and tell about DSH and OpenSafely https://github.com/alan-turing-institute/tps-project-management/issues/157 - - No immediate collaboration but agreed to be involved in the specification and TRE community -- For discussion: this story needs to be redefined to include external engagement or a new one created +The stand was busy on the first day and quieter on the second, in part due to its location out of sight from the rest, but that left time to properly engage in conversation with those who did (NHS England, Head of cyber-physica & digital twins at Innovate UK, Chief Exec of RSS). Overall the impression of those there has been that there was important potential leads, we have added to our private sharepoint for follow up. ## TRESA Over the year TRESA have increased its autonomy from the DSH research project, in terms of work ownership and management. @@ -252,21 +164,12 @@ Therefore TRESA stories have not been independently updated and it is more compr This warrants updating and reviewing the stories we keep under the DSH roadmap, focusing on communicating with the service area rather than planning or prioritising for them. ### Updates -A new team have been onboarded. -They have focused on learning the ropes while simultaneously supporting the December DSG. -With some support from the DSH team they were able to do so. -We had a detailed [handover meeting](https://hackmd.io/kh6siuZcTdCxcfYryAvypw) to go over each process, the idea is that moving forward TRESA directly owns processes and their improvement. - -We have also identified areas of work for TRESA to fully establish itself as a service area, these need to be elaborated into a full proposal with sprints and stories but main areas are: -- Data protection -- Cost and recovery model -- Acredditation: ISO 27001 and DSPT renewal -- Client projects management ## Project management and strategy -Work and stories that do not belong directly in any pillars but are necessary for all + +Work and stories that do not belong directly in any pillars but are necessary for all. ### Project strategy and ways of working [#43](https://github.com/alan-turing-institute/data-safe-haven-team/issues/43) @@ -285,17 +188,11 @@ Through several strategy sessions we will: #### Progress -##### Strategy - -Through several team wide sessions we jointly produced a [project strategy](https://thealanturininstitute.sharepoint.com/:p:/s/SafeHaven/Ebrp4Iyc9M1NpPTgpgHdj5kB7HPvH-2gM0oNd97jJu6oxw?e=eN0ZFw)https://thealanturininstitute.sharepoint.com/:p:/s/SafeHaven/Ebrp4Iyc9M1NpPTgpgHdj5kB7HPvH-2gM0oNd97jJu6oxw?e=eN0ZFw a long, medium and short term levels. +On 13 February the DSH had its second strategy session, based on last year's work it tried to define and prioritise specific work and activities for the project by creating milestones by project workstream. While we discussed aspects of the Community workstream in depth there was no time for other workstreams. -This resulting in a clear Vision & Mision that have allowed internal alignment and improved external communications +Therefore the definition of milestones and their prioritisation needs to be continued and more effectively integrated into monthly discussions. A current priority once the FY has ended is to revisit ways of work and specifically roadmap management. ->To remove barriers to working safely and effectively with sensitive data, -by promoting and demonstrating a culture of open, community-led development -of interoperable foundational infrastructure and governance. +A first effort in that direction has been the creation of a first project report to be used to better communicate and discuss some elements of the monthly meeting, in this way it resembles more weeklies where the asynchronous elements help make better use of the meeting time. -We also agreed the pillars of the project and established a [roadmap](https://github.com/orgs/alan-turing-institute/projects/111/views/1) of the necessary work for success. +As further improvement to ways of work a proposal on how to better engage with PIs was developed, but not yet discussed. -##### Ways of working -Throughout this year we have also iterated our ways of working which are openly available on GitHub in [WaysofWork.md](https://github.com/alan-turing-institute/data-safe-haven-team/blob/main/WaysofWork.md), they are focused in increased communication and work prioritisation. From b86b1b6c5c9585e1dac50fbb22101c395433866f Mon Sep 17 00:00:00 2001 From: Davsarper <118986872+Davsarper@users.noreply.github.com> Date: Mon, 8 Apr 2024 09:08:10 +0100 Subject: [PATCH 03/12] Update ProgressReport-Feb24-Aug24.md added missing stories: TRESA --- Reports/ProgressReport-Feb24-Aug24.md | 41 ++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/Reports/ProgressReport-Feb24-Aug24.md b/Reports/ProgressReport-Feb24-Aug24.md index bd3cdda..21bc84e 100644 --- a/Reports/ProgressReport-Feb24-Aug24.md +++ b/Reports/ProgressReport-Feb24-Aug24.md @@ -163,8 +163,47 @@ Therefore TRESA stories have not been independently updated and it is more compr This warrants updating and reviewing the stories we keep under the DSH roadmap, focusing on communicating with the service area rather than planning or prioritising for them. -### Updates +### TRESA team updates +### Other TRESA related stories + +(Stories and work directly impacting TRESA but still lead from the research project) + +#### TRESA Costs and cost recovery [#36](https://github.com/alan-turing-institute/data-safe-haven-team/issues/36) + +##### Goal + +An agreed and formal process to recharge ATI projects being served by TRESA. + +##### Progress + +TRESA has now its own code, people time has been changed in forecast to this code and Azure subscriptions associated to it (although currently covered by core). + +Next step is to formalise the recharge process, projects engaging with TRESA have already been advised there will be a staff related cost in addittion to their specific subscription. + +#### Review of requirements for security accreditation [37](https://github.com/alan-turing-institute/data-safe-haven-team/issues/37) + +##### Goal + +A clear list of requirements and necessary steps that DSH would need to take to be ISO027001 compliant. + +This work will include pulling a list of requirements for the specification that includes a clear idea of the steps to take and the effort involved. + +It also includes the work relating to DSPT certification: resubmitting and adapting answers if necessary. + +##### Definition of Done + +DSH remains DSPT compliant + +There is a documented plan for DSH to be ISO027001 compliant. + +##### Progress + +Revised DSPT v6 requirement, there being no effective changes for category 3 organisations (us). +Reviewed and copied last year answers for all mandatory requirements and made progress updating links and references (ongoing). +Held team meeting to review non mandatory requirements identifying a full list of them that could be positively answered. + +Issue is [here](https://github.com/alan-turing-institute/trusted-research/issues/158#issuecomment-1965134444) and document with in progress submission can be found on TRESA sharepoint (private as it contains internal information). ## Project management and strategy From 1d5ec31ecfbd66e4915d0aef5cb026f29241f63d Mon Sep 17 00:00:00 2001 From: Davsarper <118986872+Davsarper@users.noreply.github.com> Date: Mon, 8 Apr 2024 09:11:56 +0100 Subject: [PATCH 04/12] Update ProgressReport-Feb24-Aug24.md Added missing sotries: PM --- Reports/ProgressReport-Feb24-Aug24.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/Reports/ProgressReport-Feb24-Aug24.md b/Reports/ProgressReport-Feb24-Aug24.md index 21bc84e..031dfaa 100644 --- a/Reports/ProgressReport-Feb24-Aug24.md +++ b/Reports/ProgressReport-Feb24-Aug24.md @@ -235,3 +235,15 @@ A first effort in that direction has been the creation of a first project report As further improvement to ways of work a proposal on how to better engage with PIs was developed, but not yet discussed. +### Contracts, legal and budget work [#53](https://github.com/alan-turing-institute/data-safe-haven-team/issues/53) + +All work related to agreements, policies, expenses, contracts, budget. + +#### Progress + +Work has focused on managing UK TRE Community grant, which included ensuring actual allocation of costs to project and workign with DARE to agree on a reprofile that allowed us to use almost the totally of funds while delivering a final event past the grant end date. + +Substantial work has also gone into aligning project actuals with Finance records for an appropriate management of internal and external Institute funds. + + + From 3f5e6aa6675794ee9de6529b393b39e5489093c6 Mon Sep 17 00:00:00 2001 From: Davsarper <118986872+Davsarper@users.noreply.github.com> Date: Mon, 8 Apr 2024 09:44:32 +0100 Subject: [PATCH 05/12] Update ProgressReport-Feb24-Aug24.md added priorities and plans sections --- Reports/ProgressReport-Feb24-Aug24.md | 138 ++++++++++++++++++-------- 1 file changed, 96 insertions(+), 42 deletions(-) diff --git a/Reports/ProgressReport-Feb24-Aug24.md b/Reports/ProgressReport-Feb24-Aug24.md index 031dfaa..1d33f75 100644 --- a/Reports/ProgressReport-Feb24-Aug24.md +++ b/Reports/ProgressReport-Feb24-Aug24.md @@ -4,27 +4,27 @@ This update: 8 April 2024 -This document contains a summary of progress across all stories in the [project roadmap](https://github.com/orgs/alan-turing-institute/projects/111/views/1). -It maps stories according to the (main) pillar and priority they contribute to. +## Progress -Included stories are those that were scheduled and/or prioritised over this period, some not scheduled stories are included when there are updates for them regardless (indirect contributions from other stories, relevant work recently started, or something to report in general). +This section contains a summary of progress across all stories in the [project roadmap](https://github.com/orgs/alan-turing-institute/projects/111/views/1). +It maps stories according to the (main) pillar and priority they contribute to. -## Codebase development +### Codebase development Running projects working with sensitive data safely Running cutting edge data science projects effectively -### Manage codebase releases and testing: [#50](https://github.com/alan-turing-institute/data-safe-haven-team/issues/50) +#### Manage codebase releases and testing: [#50](https://github.com/alan-turing-institute/data-safe-haven-team/issues/50) Contributes to: - Running projects working with sensitive data safely - Running cutting edge data science projects effectively -#### Goal +##### Goal Support for deployments of the Data Safe Haven at Turing and beyond -#### Progress +##### Progress Having completed development of v4.2.0 we prepared the release which included preparing a release branch and deployment in an environment for pen testing. @@ -34,19 +34,19 @@ Pen testing was arranged and carried out, managing to spend within the 2023-2024 - @craddm would you add here a bit on the results? -### Codebase maintenance: [#47](https://github.com/alan-turing-institute/data-safe-haven-team/issues/47) +#### Codebase maintenance: [#47](https://github.com/alan-turing-institute/data-safe-haven-team/issues/47) Contributes to: - Running projects working with sensitive data safely - Running cutting edge data science projects effectively -#### Goal +##### Goal Ensure that codebase is kept up-to-date with bug fixes, security updates, external API changes etc. - Ensure that DSH code is always deployable - Ensure that known security issues are remediated/minimised as soon as possible - Ensure that documentation is up-to-date with code base -#### Progress +##### Progress Have worked on updating software used within SREs to ensure the security and functionality of the environment: - Guacamole server updated [PR](https://github.com/alan-turing-institute/data-safe-haven/pull/1741) @@ -62,9 +62,9 @@ An issue with Jupyter notebooks not being able to use Python when launched from Worked on updating documentation to reflect Azure Active Directory name change to Microsoft Entra. -### Identify and implement core IAC changes: [#28](https://github.com/alan-turing-institute/data-safe-haven-team/issues/28) +#### Identify and implement core IAC changes: [#28](https://github.com/alan-turing-institute/data-safe-haven-team/issues/28) -#### Goal +##### Goal Make DSH deployment more robust and development easier through using IAC and configuration management. - Take advantage of IAC and configuration management in the DSH codebase which will - Make deployments faster @@ -72,33 +72,33 @@ Make DSH deployment more robust and development easier through using IAC and con - Make development easier - Move away from non-idempotent, bespoke scripts (Powershell, bash, cloud-init) -##### Definition of done +###### Definition of done On the release of a new major version which removes legacy, script-based deployment. -#### Progress +##### Progress Over this period the story has not been prioritised/resourced as much, focusing on identifying and scoping future work which will be a priority after v4.2 release. The DSH code repo contains milestones that reflect related and planned issues (V5.x milestones) https://github.com/alan-turing-institute/data-safe-haven/milestones. Additionally a first version of the codebase roadmap has been put in place, linking to milestone but also recording potential desired features that are not currently planned to implement. A roadmap is a desired/required document to have in place for software funding opportunities https://github.com/alan-turing-institute/data-safe-haven/blob/develop/ROADMAP.md. -## Information governance & standards +### Information governance & standards Infrastructure adhering to the latest agreed upon standard Identifying, co-creating and supporting a TRE standard used across TRE infrastructures -### SATRE: stakeholder engagement and community buy out [#66](https://github.com/alan-turing-institute/data-safe-haven-team/issues/66) +#### SATRE: stakeholder engagement and community buy out [#66](https://github.com/alan-turing-institute/data-safe-haven-team/issues/66) -#### Goal +##### Goal Ensure that institutions evaluate themselves against SATRE and that the momentum is maintained between funded phases At the end of the funded phase of SATRE there was a growing community interest, with institutions and stakeholders affirming they would evaluate themselves against it and contribute feedback. Without ongoing resources the necessary support to ensure that happens cannot be provided and SATRE may end up not being adopted. -#### Definition of Done +##### Definition of Done There are a number of self evlauaitons completed, there is feedback on the spec repository and, ideally, there is an active WG within the TRE Community continuing to work on it. -#### Progress +##### Progress NHS-E R&D Programme Director stated that SATRE has become the reference framework for TREs within the Subnational SDE programme, placing SATRE as a key reference for what TREs are. @@ -106,25 +106,25 @@ We have also seen how institutions that already have other accreditations opt to SATRE has continued to atract the interest of the community, with many attendees to its meeting wanting to be involved. -## Community building +### Community building Creating resources for all stakeholders (inc. members of the public) to engage in the TRE conversation Creating and maintaining open and active communication spaces & workspaces (Slack, GH) Identifying and documenting everything that can be openly documented -### UK TRE Community leadership [#52](https://github.com/alan-turing-institute/data-safe-haven-team/issues/52) +#### UK TRE Community leadership [#52](https://github.com/alan-turing-institute/data-safe-haven-team/issues/52) Contributes to: - Creating resources for all stakeholders (inc. Citizens) to engage in the TRE conversation - Creating and maintaining open and active communication spaces & workspaces (Slack, GH) -#### Goal +##### Goal Provide a space for those involved in building, using and responsible for governance of TREs to discuss and recommend best practices. - Host online working spaces, events and workshops to support the UK TRE Community - Share best practices i.e. for making radiology data available for researchers - Empower the community to help influence policy decisions -#### Progress +##### Progress Funded phase came to an end on 31 March, along reporting it is necessary to organise and put together what we have produced but we have: - Created a first version of all governance documents. Some will include pending conversations that may turn in the next version of those documents, for example the endorsement of outputs (v1 will only have community approval). This is the issue referencing to all documents https://github.com/orgs/uk-tre/projects/1/views/1?pane=issue&itemId=53738648 @@ -140,50 +140,50 @@ With funding ending we received a community request/push to ensure we put the wo Lastly we have been workign with Scriberia to use the remaining funding to produce an illustration that can represent the community in a number of ways, the most immediate one being the production of physical stand up banners for events (due to timeline DSH will cover the actual banner while DARE funds have covered the illustration itself). -### Communication and outreach [#35](https://github.com/alan-turing-institute/data-safe-haven-team/issues/35) +#### Communication and outreach [#35](https://github.com/alan-turing-institute/data-safe-haven-team/issues/35) Contributes to: - Creating and maintaining open and active communication spaces & workspaces (Slack, GH) - Creating resources for all stakeholders (inc. Citizens) to engage in the TRE conversation -#### Goal +##### Goal - Supporting the user community of the DSH codebase - Publicising our work via blogposts, reports or papers - Communicating our work through conference/workshop talks or posters -#### Progress +##### Progress AI UK has been a priority for this story and the project. Held on 19-20 March DSH had its own stand, through a likert scale exercise on TREs (and a bowl of sweets) we engaged attendees to introduce them to TREs and the work of the project which we stressed to be not only an open codebase but also the governance & standards and the community. The stand was busy on the first day and quieter on the second, in part due to its location out of sight from the rest, but that left time to properly engage in conversation with those who did (NHS England, Head of cyber-physica & digital twins at Innovate UK, Chief Exec of RSS). Overall the impression of those there has been that there was important potential leads, we have added to our private sharepoint for follow up. -## TRESA +### TRESA Over the year TRESA have increased its autonomy from the DSH research project, in terms of work ownership and management. Therefore TRESA stories have not been independently updated and it is more comprehensive to update on the service area as a whole. This warrants updating and reviewing the stories we keep under the DSH roadmap, focusing on communicating with the service area rather than planning or prioritising for them. -### TRESA team updates +#### TRESA team updates -### Other TRESA related stories +#### Other TRESA related stories (Stories and work directly impacting TRESA but still lead from the research project) -#### TRESA Costs and cost recovery [#36](https://github.com/alan-turing-institute/data-safe-haven-team/issues/36) +##### TRESA Costs and cost recovery [#36](https://github.com/alan-turing-institute/data-safe-haven-team/issues/36) -##### Goal +###### Goal An agreed and formal process to recharge ATI projects being served by TRESA. -##### Progress +###### Progress TRESA has now its own code, people time has been changed in forecast to this code and Azure subscriptions associated to it (although currently covered by core). Next step is to formalise the recharge process, projects engaging with TRESA have already been advised there will be a staff related cost in addittion to their specific subscription. -#### Review of requirements for security accreditation [37](https://github.com/alan-turing-institute/data-safe-haven-team/issues/37) +##### Review of requirements for security accreditation [37](https://github.com/alan-turing-institute/data-safe-haven-team/issues/37) -##### Goal +###### Goal A clear list of requirements and necessary steps that DSH would need to take to be ISO027001 compliant. @@ -191,13 +191,13 @@ This work will include pulling a list of requirements for the specification that It also includes the work relating to DSPT certification: resubmitting and adapting answers if necessary. -##### Definition of Done +###### Definition of Done DSH remains DSPT compliant There is a documented plan for DSH to be ISO027001 compliant. -##### Progress +###### Progress Revised DSPT v6 requirement, there being no effective changes for category 3 organisations (us). Reviewed and copied last year answers for all mandatory requirements and made progress updating links and references (ongoing). @@ -206,13 +206,13 @@ Held team meeting to review non mandatory requirements identifying a full list o Issue is [here](https://github.com/alan-turing-institute/trusted-research/issues/158#issuecomment-1965134444) and document with in progress submission can be found on TRESA sharepoint (private as it contains internal information). -## Project management and strategy +### Project management and strategy Work and stories that do not belong directly in any pillars but are necessary for all. -### Project strategy and ways of working [#43](https://github.com/alan-turing-institute/data-safe-haven-team/issues/43) +#### Project strategy and ways of working [#43](https://github.com/alan-turing-institute/data-safe-haven-team/issues/43) -#### Goal +##### Goal The aim is to develop a project strategy and revise best ways of work to achieve it Through several strategy sessions we will: @@ -225,7 +225,7 @@ Through several strategy sessions we will: - Evaluate required effort for the work against team capacity - Develop and agree new ways of work, including meeting structure and use of project's repositories and projects -#### Progress +##### Progress On 13 February the DSH had its second strategy session, based on last year's work it tried to define and prioritise specific work and activities for the project by creating milestones by project workstream. While we discussed aspects of the Community workstream in depth there was no time for other workstreams. @@ -235,15 +235,69 @@ A first effort in that direction has been the creation of a first project report As further improvement to ways of work a proposal on how to better engage with PIs was developed, but not yet discussed. -### Contracts, legal and budget work [#53](https://github.com/alan-turing-institute/data-safe-haven-team/issues/53) +#### Contracts, legal and budget work [#53](https://github.com/alan-turing-institute/data-safe-haven-team/issues/53) All work related to agreements, policies, expenses, contracts, budget. -#### Progress +##### Progress Work has focused on managing UK TRE Community grant, which included ensuring actual allocation of costs to project and workign with DARE to agree on a reprofile that allowed us to use almost the totally of funds while delivering a final event past the grant end date. Substantial work has also gone into aligning project actuals with Finance records for an appropriate management of internal and external Institute funds. +## Plans and priorities + +This section contains project plans and priorities , currently focusing on work to be done over the next month it should eventually encompass longer term plans. + +### Monthly priorities: April + +The list of priority stories can eb found here https://github.com/orgs/alan-turing-institute/projects/111/views/11 + +#### Communication and outreach [35](https://github.com/alan-turing-institute/data-safe-haven-team/issues/35) + +Follow up on conversations from AI UK to keep momentum going, as resources are limited this month the work here should be to establish connections buyt any meetings or further work left for furhter ahead. + +#### Project Strategy and Ways of work [43](https://github.com/alan-turing-institute/data-safe-haven-team/issues/43) + +@davsarper keen to prioritise as possible to: + +- Create due reports +- Revisit reports and monthlies format +- Propose and discuss PIs WoW + +#### Manage codebase releases adn testing [50](https://github.com/alan-turing-institute/data-safe-haven-team/issues/50) + +This story was a March priority and the release completed, with less time dedicated there will be follow up actions. Including any fixes and also required communications about the release. + +#### Identify and implement core IAC changes [28](https://github.com/alan-turing-institute/data-safe-haven-team/issues/28) + +April work will focus on identifying and planning work within this story so it can be a top priority in May. + +#### UK TRE Community [52](https://github.com/alan-turing-institute/data-safe-haven-team/issues/52) + +Having finalised the funding phase there is three important things to do: +- Wrap up outputs, and celebrate WGs event +- Transition and knowledge transfer: ensure the community can continue delivering work + +#### Manage grant opportunities and pipeline [51](https://github.com/alan-turing-institute/data-safe-haven-team/issues/51) + +It is a priority to at least think and plan, this story can be directly linked to PI work and how we better coordinate. So not a high resources story but still very important one + +#### DSPT submission [37](https://github.com/alan-turing-institute/data-safe-haven-team/issues/37) + +Finalise submission, it should not require intensive effort but it will include revision by project PIs and possibly Data Protection (as Manager changed between submissions). + +New non-mandatory requirements that can be easily completed will be (already identified). + +#### TRESA costs and cost recovery [36](https://github.com/alan-turing-institute/data-safe-haven-team/issues/36) + +While capacity is stretched with the new year we have to clarify recharges process and prepare for the announcement in May's catch up. + +#### Not doing or prioritising + +This month we are not leaving much out (please add anything you believe we should keep in mind), with the new FY we are doing a bit everywhere in a month characterised by planning ahead. + +### Next up: upcoming priorities and work the will be necessary +*under construction: please add anything relevant, with dates where possible* From 92a46f33ae196cb39547167573954ea9f4f8338e Mon Sep 17 00:00:00 2001 From: Davsarper <118986872+Davsarper@users.noreply.github.com> Date: Mon, 8 Apr 2024 09:46:27 +0100 Subject: [PATCH 06/12] Update ProgressReport-Feb24-Aug24.md formatted some spacing (spaces after titles) --- Reports/ProgressReport-Feb24-Aug24.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/Reports/ProgressReport-Feb24-Aug24.md b/Reports/ProgressReport-Feb24-Aug24.md index 1d33f75..cfff008 100644 --- a/Reports/ProgressReport-Feb24-Aug24.md +++ b/Reports/ProgressReport-Feb24-Aug24.md @@ -10,20 +10,22 @@ This section contains a summary of progress across all stories in the [project r It maps stories according to the (main) pillar and priority they contribute to. ### Codebase development + Running projects working with sensitive data safely Running cutting edge data science projects effectively #### Manage codebase releases and testing: [#50](https://github.com/alan-turing-institute/data-safe-haven-team/issues/50) - Contributes to: - Running projects working with sensitive data safely - Running cutting edge data science projects effectively + ##### Goal Support for deployments of the Data Safe Haven at Turing and beyond + ##### Progress Having completed development of v4.2.0 we prepared the release which included preparing a release branch and deployment in an environment for pen testing. @@ -40,12 +42,14 @@ Contributes to: - Running projects working with sensitive data safely - Running cutting edge data science projects effectively + ##### Goal Ensure that codebase is kept up-to-date with bug fixes, security updates, external API changes etc. - Ensure that DSH code is always deployable - Ensure that known security issues are remediated/minimised as soon as possible - Ensure that documentation is up-to-date with code base + ##### Progress Have worked on updating software used within SREs to ensure the security and functionality of the environment: @@ -65,6 +69,7 @@ Worked on updating documentation to reflect Azure Active Directory name change t #### Identify and implement core IAC changes: [#28](https://github.com/alan-turing-institute/data-safe-haven-team/issues/28) ##### Goal + Make DSH deployment more robust and development easier through using IAC and configuration management. - Take advantage of IAC and configuration management in the DSH codebase which will - Make deployments faster @@ -73,6 +78,7 @@ Make DSH deployment more robust and development easier through using IAC and con - Move away from non-idempotent, bespoke scripts (Powershell, bash, cloud-init) ###### Definition of done + On the release of a new major version which removes legacy, script-based deployment. ##### Progress @@ -89,6 +95,7 @@ Identifying, co-creating and supporting a TRE standard used across TRE infrastru #### SATRE: stakeholder engagement and community buy out [#66](https://github.com/alan-turing-institute/data-safe-haven-team/issues/66) ##### Goal + Ensure that institutions evaluate themselves against SATRE and that the momentum is maintained between funded phases At the end of the funded phase of SATRE there was a growing community interest, with institutions and stakeholders affirming they would evaluate themselves against it and contribute feedback. @@ -96,6 +103,7 @@ At the end of the funded phase of SATRE there was a growing community interest, Without ongoing resources the necessary support to ensure that happens cannot be provided and SATRE may end up not being adopted. ##### Definition of Done + There are a number of self evlauaitons completed, there is feedback on the spec repository and, ideally, there is an active WG within the TRE Community continuing to work on it. ##### Progress @@ -106,7 +114,9 @@ We have also seen how institutions that already have other accreditations opt to SATRE has continued to atract the interest of the community, with many attendees to its meeting wanting to be involved. + ### Community building + Creating resources for all stakeholders (inc. members of the public) to engage in the TRE conversation Creating and maintaining open and active communication spaces & workspaces (Slack, GH) Identifying and documenting everything that can be openly documented @@ -119,6 +129,7 @@ Contributes to: - Creating and maintaining open and active communication spaces & workspaces (Slack, GH) ##### Goal + Provide a space for those involved in building, using and responsible for governance of TREs to discuss and recommend best practices. - Host online working spaces, events and workshops to support the UK TRE Community - Share best practices i.e. for making radiology data available for researchers @@ -147,6 +158,7 @@ Contributes to: - Creating resources for all stakeholders (inc. Citizens) to engage in the TRE conversation ##### Goal + - Supporting the user community of the DSH codebase - Publicising our work via blogposts, reports or papers - Communicating our work through conference/workshop talks or posters @@ -157,6 +169,7 @@ AI UK has been a priority for this story and the project. Held on 19-20 March DS The stand was busy on the first day and quieter on the second, in part due to its location out of sight from the rest, but that left time to properly engage in conversation with those who did (NHS England, Head of cyber-physica & digital twins at Innovate UK, Chief Exec of RSS). Overall the impression of those there has been that there was important potential leads, we have added to our private sharepoint for follow up. + ### TRESA Over the year TRESA have increased its autonomy from the DSH research project, in terms of work ownership and management. Therefore TRESA stories have not been independently updated and it is more comprehensive to update on the service area as a whole. @@ -181,6 +194,7 @@ TRESA has now its own code, people time has been changed in forecast to this cod Next step is to formalise the recharge process, projects engaging with TRESA have already been advised there will be a staff related cost in addittion to their specific subscription. + ##### Review of requirements for security accreditation [37](https://github.com/alan-turing-institute/data-safe-haven-team/issues/37) ###### Goal @@ -213,6 +227,7 @@ Work and stories that do not belong directly in any pillars but are necessary fo #### Project strategy and ways of working [#43](https://github.com/alan-turing-institute/data-safe-haven-team/issues/43) ##### Goal + The aim is to develop a project strategy and revise best ways of work to achieve it Through several strategy sessions we will: From 3c800f5a10c3a120d17fe9776808ca6b6e358a1c Mon Sep 17 00:00:00 2001 From: Davsarper <118986872+Davsarper@users.noreply.github.com> Date: Tue, 9 Apr 2024 11:50:56 +0100 Subject: [PATCH 07/12] Apply suggestions from code review Apply suggestions from review Co-authored-by: Jim Madge --- Reports/ProgressReport-Feb24-Aug24.md | 36 +++++++++++++++++++-------- 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/Reports/ProgressReport-Feb24-Aug24.md b/Reports/ProgressReport-Feb24-Aug24.md index cfff008..379a19f 100644 --- a/Reports/ProgressReport-Feb24-Aug24.md +++ b/Reports/ProgressReport-Feb24-Aug24.md @@ -30,10 +30,14 @@ Support for deployments of the Data Safe Haven at Turing and beyond Having completed development of v4.2.0 we prepared the release which included preparing a release branch and deployment in an environment for pen testing. -Extensive time was allocated to deploying and the errors/bugs that arose, as well as preparing for pen testing (this included deployment but also requesting specific tests like the removal of certain hardcoded IPs) +Extensive time was allocated to deploying and the errors/bugs that arose, +as well as preparing for pen testing. +This included deployment but also requesting specific tests like changes to the network and firewall configuration. -Pen testing was arranged and carried out, managing to spend within the 2023-2024 FY. iStorm kept better communications than last time and did not find concerning issues. -- @craddm would you add here a bit on the results? +Pen testing was arranged and carried out, managing to spend within the 2023-2024 FY. +iSTORM kept better communications than last time and did not find concerning issues. +All reported vulnerabilities were known to use before the test and we are confident all are mitigated by technical or process controls. +We don't believe any of the vulnerabilities identified present a strong risk of enabling unauthorised data ingress or egress. #### Codebase maintenance: [#47](https://github.com/alan-turing-institute/data-safe-haven-team/issues/47) @@ -83,9 +87,15 @@ On the release of a new major version which removes legacy, script-based deploym ##### Progress -Over this period the story has not been prioritised/resourced as much, focusing on identifying and scoping future work which will be a priority after v4.2 release. The DSH code repo contains milestones that reflect related and planned issues (V5.x milestones) https://github.com/alan-turing-institute/data-safe-haven/milestones. +Over this period the story has not been prioritised instead focusing on identifying and scoping future work which will be a priority after v4.2.0 release. +The DSH code repo contains milestones that reflect related and planned issues (v5.x.y milestones) https://github.com/alan-turing-institute/data-safe-haven/milestones. -Additionally a first version of the codebase roadmap has been put in place, linking to milestone but also recording potential desired features that are not currently planned to implement. A roadmap is a desired/required document to have in place for software funding opportunities https://github.com/alan-turing-institute/data-safe-haven/blob/develop/ROADMAP.md. +Additionally a first version of a project [roadmap](https://github.com/alan-turing-institute/data-safe-haven/blob/develop/ROADMAP.md) for DSH has been put in place. +This links to milestones but also records potential desired features that are not currently planned. +A roadmap provides value, + +- For users, to understand and have confidence in upcoming features and how to influence development. +- For funders, to see the products longer term vision/strategy and as evidence of maturity and continuity. ### Information governance & standards Infrastructure adhering to the latest agreed upon standard @@ -104,7 +114,7 @@ Without ongoing resources the necessary support to ensure that happens cannot be ##### Definition of Done -There are a number of self evlauaitons completed, there is feedback on the spec repository and, ideally, there is an active WG within the TRE Community continuing to work on it. +There are a number of self evaluaitons completed, there is feedback on the spec repository and, ideally, there is an active WG within the TRE Community continuing to work on it. ##### Progress @@ -117,7 +127,7 @@ SATRE has continued to atract the interest of the community, with many attendees ### Community building -Creating resources for all stakeholders (inc. members of the public) to engage in the TRE conversation +Creating resources for all stakeholders (including members of the public) to engage in the TRE conversation Creating and maintaining open and active communication spaces & workspaces (Slack, GH) Identifying and documenting everything that can be openly documented @@ -125,19 +135,21 @@ Identifying and documenting everything that can be openly documented #### UK TRE Community leadership [#52](https://github.com/alan-turing-institute/data-safe-haven-team/issues/52) Contributes to: -- Creating resources for all stakeholders (inc. Citizens) to engage in the TRE conversation +- Creating resources for all stakeholders (including citizens) to engage in the TRE conversation - Creating and maintaining open and active communication spaces & workspaces (Slack, GH) ##### Goal Provide a space for those involved in building, using and responsible for governance of TREs to discuss and recommend best practices. + - Host online working spaces, events and workshops to support the UK TRE Community -- Share best practices i.e. for making radiology data available for researchers +- Share best practices _e.g._ for making radiology data available for researchers - Empower the community to help influence policy decisions ##### Progress Funded phase came to an end on 31 March, along reporting it is necessary to organise and put together what we have produced but we have: + - Created a first version of all governance documents. Some will include pending conversations that may turn in the next version of those documents, for example the endorsement of outputs (v1 will only have community approval). This is the issue referencing to all documents https://github.com/orgs/uk-tre/projects/1/views/1?pane=issue&itemId=53738648 - Created a new website, not launched yet. - Created a public community calendar, for "official" events and all those happening in the space and that people may want to include and promote. @@ -148,7 +160,9 @@ With funding ending we received a community request/push to ensure we put the wo - Documentation sprints (https://hackmd.io/ty_bEhtBTrKjmYsJEEP8Qw) - Working groups day (https://hackmd.io/nTAAo7fOT_quE18y0NYjnA). Which will be held on 29 April (This also implied figuring out a way to spend beyond March given it is not feasible to run the WGs day effectively before late April). -Lastly we have been workign with Scriberia to use the remaining funding to produce an illustration that can represent the community in a number of ways, the most immediate one being the production of physical stand up banners for events (due to timeline DSH will cover the actual banner while DARE funds have covered the illustration itself). +Lastly we have been working with Scriberia to use the remaining funding to produce an illustration that can represent the community in a number of ways. +The most immediate use will be for the production of physical stand up banners for events. +Due to timeline DSH will cover the actual banner while DARE funds have covered the illustration itself. #### Communication and outreach [#35](https://github.com/alan-turing-institute/data-safe-haven-team/issues/35) @@ -172,7 +186,7 @@ The stand was busy on the first day and quieter on the second, in part due to it ### TRESA Over the year TRESA have increased its autonomy from the DSH research project, in terms of work ownership and management. -Therefore TRESA stories have not been independently updated and it is more comprehensive to update on the service area as a whole. +Therefore, TRESA stories have not been independently updated and this section is a comprehensive to update on the service area as a whole. This warrants updating and reviewing the stories we keep under the DSH roadmap, focusing on communicating with the service area rather than planning or prioritising for them. From 99f4a904e42a141030b3088f0a4c38c4f42ec258 Mon Sep 17 00:00:00 2001 From: Davsarper <118986872+Davsarper@users.noreply.github.com> Date: Thu, 11 Apr 2024 11:56:50 +0100 Subject: [PATCH 08/12] Apply suggestions from code review commit suggestions from review Co-authored-by: Arielle-Bennett <74651964+Arielle-Bennett@users.noreply.github.com> --- Reports/ProgressReport-Feb24-Aug24.md | 111 +++++++++++++++++++------- 1 file changed, 81 insertions(+), 30 deletions(-) diff --git a/Reports/ProgressReport-Feb24-Aug24.md b/Reports/ProgressReport-Feb24-Aug24.md index 379a19f..14f9fb7 100644 --- a/Reports/ProgressReport-Feb24-Aug24.md +++ b/Reports/ProgressReport-Feb24-Aug24.md @@ -4,11 +4,13 @@ This update: 8 April 2024 + ## Progress This section contains a summary of progress across all stories in the [project roadmap](https://github.com/orgs/alan-turing-institute/projects/111/views/1). It maps stories according to the (main) pillar and priority they contribute to. + ### Codebase development Running projects working with sensitive data safely @@ -18,14 +20,13 @@ Running cutting edge data science projects effectively #### Manage codebase releases and testing: [#50](https://github.com/alan-turing-institute/data-safe-haven-team/issues/50) Contributes to: + - Running projects working with sensitive data safely - Running cutting edge data science projects effectively - ##### Goal Support for deployments of the Data Safe Haven at Turing and beyond - ##### Progress Having completed development of v4.2.0 we prepared the release which included preparing a release branch and deployment in an environment for pen testing. @@ -36,36 +37,48 @@ This included deployment but also requesting specific tests like changes to the Pen testing was arranged and carried out, managing to spend within the 2023-2024 FY. iSTORM kept better communications than last time and did not find concerning issues. -All reported vulnerabilities were known to use before the test and we are confident all are mitigated by technical or process controls. +All reported vulnerabilities were known to us before the test and we are confident all are mitigated by technical or process controls. We don't believe any of the vulnerabilities identified present a strong risk of enabling unauthorised data ingress or egress. #### Codebase maintenance: [#47](https://github.com/alan-turing-institute/data-safe-haven-team/issues/47) Contributes to: + - Running projects working with sensitive data safely - Running cutting edge data science projects effectively - ##### Goal Ensure that codebase is kept up-to-date with bug fixes, security updates, external API changes etc. + - Ensure that DSH code is always deployable - Ensure that known security issues are remediated/minimised as soon as possible - Ensure that documentation is up-to-date with code base - ##### Progress Have worked on updating software used within SREs to ensure the security and functionality of the environment: + - Guacamole server updated [PR](https://github.com/alan-turing-institute/data-safe-haven/pull/1741) - Nexus server updated [PR](https://github.com/alan-turing-institute/data-safe-haven/pull/1744) - CodiMD server updated [PR](https://github.com/alan-turing-institute/data-safe-haven/pull/1743) -Added and tested a script to handle SAS access tokens renewal, currently expiring yearly. These are required manage access to data storage (and therefore ingress and egress). The relevant PR is here https://github.com/alan-turing-institute/data-safe-haven/pull/1739. In the process we realised SAS tokens are bound to Store Access Policies which could be modified to have no end date, we are currently considering the covenience of this approach versus potential security issues in https://github.com/alan-turing-institute/data-safe-haven/issues/1751 . +Added and tested a script to handle SAS access tokens renewal, currently expiring yearly. +These are required manage access to data storage (and therefore ingress and egress). +The relevant PR is here https://github.com/alan-turing-institute/data-safe-haven/pull/1739. +In the process we realised SAS tokens are bound to Store Access Policies which could be modified to have no end date, +we are currently considering the covenience of this approach versus potential security issues in https://github.com/alan-turing-institute/data-safe-haven/issues/1751 . -Improved use of hardcoded domain names and IPs. The hardcoded lists are difficult to maintain and are prone to going out of date, despite not fully fixing the use of these improvements have been made for the 4.2.0 release by relaxing rules where security allows. For this the team checked individuals cases and applied where possible, no security issues where found and we added this as a specific thing to pent test. Related PR is https://github.com/alan-turing-institute/data-safe-haven/pull/1745 and explanatory issue is https://github.com/alan-turing-institute/data-safe-haven/issues/1549 . +Improved use of hardcoded domain names and IPs. +The hardcoded lists are difficult to maintain and are prone to going out of date, +despite not fully fixing the use of these improvements have been made for the 4.2.0 release by relaxing rules where security allows. +For this the team checked individuals cases and applied where possible, + no security issues where found and we added this as a specific thing to pent test. + Related PR is https://github.com/alan-turing-institute/data-safe-haven/pull/1745 and explanatory issue is https://github.com/alan-turing-institute/data-safe-haven/issues/1549 . -An issue with Jupyter notebooks not being able to use Python when launched from the menu was found, despite extensive work a fix was not found and decided to let it be by documenting the right workaround: launching Jupyter Notebooks from the terminal. The issue is https://github.com/alan-turing-institute/data-safe-haven/issues/1584 . +An issue with Jupyter notebooks not being able to use Python when launched from the menu was found, +despite extensive work a fix was not found and decided to let it be by documenting the right workaround: launching Jupyter Notebooks from the terminal. +The issue is https://github.com/alan-turing-institute/data-safe-haven/issues/1584. Worked on updating documentation to reflect Azure Active Directory name change to Microsoft Entra. @@ -75,6 +88,7 @@ Worked on updating documentation to reflect Azure Active Directory name change t ##### Goal Make DSH deployment more robust and development easier through using IAC and configuration management. + - Take advantage of IAC and configuration management in the DSH codebase which will - Make deployments faster - Make deployments more reliable @@ -97,32 +111,39 @@ A roadmap provides value, - For users, to understand and have confidence in upcoming features and how to influence development. - For funders, to see the products longer term vision/strategy and as evidence of maturity and continuity. + ### Information governance & standards + Infrastructure adhering to the latest agreed upon standard Identifying, co-creating and supporting a TRE standard used across TRE infrastructures -#### SATRE: stakeholder engagement and community buy out [#66](https://github.com/alan-turing-institute/data-safe-haven-team/issues/66) +#### SATRE: stakeholder engagement and community buy in [#66](https://github.com/alan-turing-institute/data-safe-haven-team/issues/66) ##### Goal Ensure that institutions evaluate themselves against SATRE and that the momentum is maintained between funded phases -At the end of the funded phase of SATRE there was a growing community interest, with institutions and stakeholders affirming they would evaluate themselves against it and contribute feedback. +At the end of the funded phase of SATRE there was a growing community interest, +with institutions and stakeholders affirming they would evaluate themselves against it and contribute feedback. Without ongoing resources the necessary support to ensure that happens cannot be provided and SATRE may end up not being adopted. ##### Definition of Done -There are a number of self evaluaitons completed, there is feedback on the spec repository and, ideally, there is an active WG within the TRE Community continuing to work on it. +There are a number of self evaluations completed, + there is feedback on the spec repository and, ideally, there is an active WG within the TRE Community continuing to work on it. ##### Progress -NHS-E R&D Programme Director stated that SATRE has become the reference framework for TREs within the Subnational SDE programme, placing SATRE as a key reference for what TREs are. +NHS-E R&D Programme Director stated during the UK TRE Community March event that SATRE has become the reference framework for TREs within the Subnational SDE programme, +placing SATRE as a key reference for what TREs are. +A recording of the event is available here https://youtu.be/KJVcy_ZKyVE?si=mbf64cZOLMHAxjwk and the notes of the day will be added to a report publicily available on the community website We have also seen how institutions that already have other accreditations opt to self evaluate with SATRE because it is the only one specific to TREs. -SATRE has continued to atract the interest of the community, with many attendees to its meeting wanting to be involved. +SATRE has continued to attract the interest of the community, +with many attendees to its meeting wanting to be involved. ### Community building @@ -135,6 +156,7 @@ Identifying and documenting everything that can be openly documented #### UK TRE Community leadership [#52](https://github.com/alan-turing-institute/data-safe-haven-team/issues/52) Contributes to: + - Creating resources for all stakeholders (including citizens) to engage in the TRE conversation - Creating and maintaining open and active communication spaces & workspaces (Slack, GH) @@ -148,16 +170,22 @@ Provide a space for those involved in building, using and responsible for govern ##### Progress -Funded phase came to an end on 31 March, along reporting it is necessary to organise and put together what we have produced but we have: +Funded phase came to an end on 31 March, +along reporting, + it is necessary to organise and put together what we have produced but we have: - Created a first version of all governance documents. Some will include pending conversations that may turn in the next version of those documents, for example the endorsement of outputs (v1 will only have community approval). This is the issue referencing to all documents https://github.com/orgs/uk-tre/projects/1/views/1?pane=issue&itemId=53738648 - Created a new website, not launched yet. - Created a public community calendar, for "official" events and all those happening in the space and that people may want to include and promote. -We celebrated the March quarterly event with very positive feedback from participants. Keynote was delivered by NHS England R&D programme Director on the subnaitonal SDEs programme, this followed a request by community members to know more about it during the December event. Full notes can be found here while we produce a report https://hackmd.io/6t-s4x-wQN6riL6gPJLuWg +We celebrated the March quarterly event with very positive feedback from participants. +Keynote was delivered by NHS England R&D programme Director on the subnational SDEs programme, +this followed a request by community members to know more about it during the December event. +Full notes can be found here while we produce a report https://hackmd.io/6t-s4x-wQN6riL6gPJLuWg + +With funding ending we received a community request/push to ensure we put the workshop budget to good use. In response we elaborated two event proposals that align with the goal of this grant: finalising governance docs with the community participation and setting up WGs for success. -With funding ending we received a community request/push to ensure we put the workshop budget to good use. In response we elaborated two events proposals that align with the goal of this grant: finalising governance docs with the community participation and set up WGs for success. -- Documentation sprints (https://hackmd.io/ty_bEhtBTrKjmYsJEEP8Qw) +- Documentation sprints (https://hackmd.io/ty_bEhtBTrKjmYsJEEP8Qw). Held across several days on 29 February, 4 March and 5 March - Working groups day (https://hackmd.io/nTAAo7fOT_quE18y0NYjnA). Which will be held on 29 April (This also implied figuring out a way to spend beyond March given it is not feasible to run the WGs day effectively before late April). Lastly we have been working with Scriberia to use the remaining funding to produce an illustration that can represent the community in a number of ways. @@ -168,6 +196,7 @@ Due to timeline DSH will cover the actual banner while DARE funds have covered t #### Communication and outreach [#35](https://github.com/alan-turing-institute/data-safe-haven-team/issues/35) Contributes to: + - Creating and maintaining open and active communication spaces & workspaces (Slack, GH) - Creating resources for all stakeholders (inc. Citizens) to engage in the TRE conversation @@ -179,23 +208,31 @@ Contributes to: ##### Progress -AI UK has been a priority for this story and the project. Held on 19-20 March DSH had its own stand, through a likert scale exercise on TREs (and a bowl of sweets) we engaged attendees to introduce them to TREs and the work of the project which we stressed to be not only an open codebase but also the governance & standards and the community. +AI UK has been a priority for this story and the project. +Held on 19-20 March DSH had its own stand, through a likert scale exercise on TREs (and a bowl of sweets) we engaged attendees to introduce them to TREs and the work of the project which we stressed to be not only an open codebase but also the governance & standards and the community. -The stand was busy on the first day and quieter on the second, in part due to its location out of sight from the rest, but that left time to properly engage in conversation with those who did (NHS England, Head of cyber-physica & digital twins at Innovate UK, Chief Exec of RSS). Overall the impression of those there has been that there was important potential leads, we have added to our private sharepoint for follow up. +The stand was busy on the first day and quieter on the second, in part due to its location out of sight from the rest, +but that left time to properly engage in conversation with those who did (NHS England, Head of cyber-physical & digital twins at Innovate UK, Chief Exec of RSS). +Overall the impression of those there has been that there was important potential leads even if less conversations than last year, + we have added to our private sharepoint for follow up. ### TRESA -Over the year TRESA have increased its autonomy from the DSH research project, in terms of work ownership and management. -Therefore, TRESA stories have not been independently updated and this section is a comprehensive to update on the service area as a whole. +Over the year TRESA has increased its autonomy from the DSH research project, +in terms of work ownership and management. +Therefore, TRESA stories have not been independently updated and this section is an update on the service area as a whole. This warrants updating and reviewing the stories we keep under the DSH roadmap, focusing on communicating with the service area rather than planning or prioritising for them. + #### TRESA team updates + #### Other TRESA related stories (Stories and work directly impacting TRESA but still lead from the research project) + ##### TRESA Costs and cost recovery [#36](https://github.com/alan-turing-institute/data-safe-haven-team/issues/36) ###### Goal @@ -245,6 +282,7 @@ Work and stories that do not belong directly in any pillars but are necessary fo The aim is to develop a project strategy and revise best ways of work to achieve it Through several strategy sessions we will: + - Define our north star (vision & mision) - Establish the project pilars or areas, defining what success looks like for each - Prioritise the measure of success, which are essential to consider the project succesful @@ -256,7 +294,8 @@ Through several strategy sessions we will: ##### Progress -On 13 February the DSH had its second strategy session, based on last year's work it tried to define and prioritise specific work and activities for the project by creating milestones by project workstream. While we discussed aspects of the Community workstream in depth there was no time for other workstreams. +On 13 February the DSH had its second strategy session, based on last year's work it tried to define and prioritise specific work and activities for the project by creating milestones by project workstream. +While we discussed aspects of the Community workstream in depth there was no time for other workstreams. Therefore the definition of milestones and their prioritisation needs to be continued and more effectively integrated into monthly discussions. A current priority once the FY has ended is to revisit ways of work and specifically roadmap management. @@ -264,6 +303,7 @@ A first effort in that direction has been the creation of a first project report As further improvement to ways of work a proposal on how to better engage with PIs was developed, but not yet discussed. + #### Contracts, legal and budget work [#53](https://github.com/alan-turing-institute/data-safe-haven-team/issues/53) All work related to agreements, policies, expenses, contracts, budget. @@ -275,17 +315,21 @@ Work has focused on managing UK TRE Community grant, which included ensuring act Substantial work has also gone into aligning project actuals with Finance records for an appropriate management of internal and external Institute funds. + ## Plans and priorities -This section contains project plans and priorities , currently focusing on work to be done over the next month it should eventually encompass longer term plans. +This section contains project plans and priorities, +currently focusing on work to be done over the next month it should eventually encompass longer term plans. + ### Monthly priorities: April -The list of priority stories can eb found here https://github.com/orgs/alan-turing-institute/projects/111/views/11 +The list of priority stories can be found here https://github.com/orgs/alan-turing-institute/projects/111/views/11 #### Communication and outreach [35](https://github.com/alan-turing-institute/data-safe-haven-team/issues/35) -Follow up on conversations from AI UK to keep momentum going, as resources are limited this month the work here should be to establish connections buyt any meetings or further work left for furhter ahead. +Follow up on conversations from AI UK to keep momentum going, +as resources are limited this month the work here should be to establish connections but any meetings or further work left for furhter ahead. #### Project Strategy and Ways of work [43](https://github.com/alan-turing-institute/data-safe-haven-team/issues/43) @@ -295,9 +339,11 @@ Follow up on conversations from AI UK to keep momentum going, as resources are l - Revisit reports and monthlies format - Propose and discuss PIs WoW -#### Manage codebase releases adn testing [50](https://github.com/alan-turing-institute/data-safe-haven-team/issues/50) +#### Manage codebase releases and testing [50](https://github.com/alan-turing-institute/data-safe-haven-team/issues/50) -This story was a March priority and the release completed, with less time dedicated there will be follow up actions. Including any fixes and also required communications about the release. +This story was a March priority and the release completed, +with less time dedicated there will be follow up actions. +Including any fixes and also required communications about the release. #### Identify and implement core IAC changes [28](https://github.com/alan-turing-institute/data-safe-haven-team/issues/28) @@ -306,16 +352,18 @@ April work will focus on identifying and planning work within this story so it c #### UK TRE Community [52](https://github.com/alan-turing-institute/data-safe-haven-team/issues/52) Having finalised the funding phase there is three important things to do: + - Wrap up outputs, and celebrate WGs event - Transition and knowledge transfer: ensure the community can continue delivering work -#### Manage grant opportunities and pipeline [51](https://github.com/alan-turing-institute/data-safe-haven-team/issues/51) +#### Manage grant opportunities and pipeline [51](https://github.com/alan-turing-institute/data-safe-haven-team/issues/51) It is a priority to at least think and plan, this story can be directly linked to PI work and how we better coordinate. So not a high resources story but still very important one #### DSPT submission [37](https://github.com/alan-turing-institute/data-safe-haven-team/issues/37) -Finalise submission, it should not require intensive effort but it will include revision by project PIs and possibly Data Protection (as Manager changed between submissions). +Finalise submission, +it should not require intensive effort but it will include revision by project PIs and possibly Data Protection (as Manager changed between submissions). New non-mandatory requirements that can be easily completed will be (already identified). @@ -323,9 +371,12 @@ New non-mandatory requirements that can be easily completed will be (already ide While capacity is stretched with the new year we have to clarify recharges process and prepare for the announcement in May's catch up. + #### Not doing or prioritising -This month we are not leaving much out (please add anything you believe we should keep in mind), with the new FY we are doing a bit everywhere in a month characterised by planning ahead. +This month we are not leaving much out (please add anything you believe we should keep in mind), +with the new FY we are doing a bit everywhere in a month characterised by planning ahead. + ### Next up: upcoming priorities and work the will be necessary From 664318f001e842c16180a96f8be8fc7ac7dd7df0 Mon Sep 17 00:00:00 2001 From: Davsarper <118986872+Davsarper@users.noreply.github.com> Date: Fri, 3 May 2024 15:32:30 +0100 Subject: [PATCH 09/12] Update Reports/ProgressReport-Feb24-Aug24.md --- Reports/ProgressReport-Feb24-Aug24.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Reports/ProgressReport-Feb24-Aug24.md b/Reports/ProgressReport-Feb24-Aug24.md index 14f9fb7..fa26c64 100644 --- a/Reports/ProgressReport-Feb24-Aug24.md +++ b/Reports/ProgressReport-Feb24-Aug24.md @@ -358,7 +358,9 @@ Having finalised the funding phase there is three important things to do: #### Manage grant opportunities and pipeline [51](https://github.com/alan-turing-institute/data-safe-haven-team/issues/51) -It is a priority to at least think and plan, this story can be directly linked to PI work and how we better coordinate. So not a high resources story but still very important one +It is a priority to at least think and plan, +this story can be directly linked to PI work and how we better coordinate. +So not a high resources story but still very important one. #### DSPT submission [37](https://github.com/alan-turing-institute/data-safe-haven-team/issues/37) From 3a2c579cb0e5b21d0f2260704a8dd80bd9aef286 Mon Sep 17 00:00:00 2001 From: Davsarper <118986872+Davsarper@users.noreply.github.com> Date: Fri, 3 May 2024 16:20:46 +0100 Subject: [PATCH 10/12] Update Reports/ProgressReport-Feb24-Aug24.md Co-authored-by: Matt Craddock --- Reports/ProgressReport-Feb24-Aug24.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Reports/ProgressReport-Feb24-Aug24.md b/Reports/ProgressReport-Feb24-Aug24.md index fa26c64..c91eb57 100644 --- a/Reports/ProgressReport-Feb24-Aug24.md +++ b/Reports/ProgressReport-Feb24-Aug24.md @@ -70,7 +70,7 @@ In the process we realised SAS tokens are bound to Store Access Policies which c we are currently considering the covenience of this approach versus potential security issues in https://github.com/alan-turing-institute/data-safe-haven/issues/1751 . Improved use of hardcoded domain names and IPs. -The hardcoded lists are difficult to maintain and are prone to going out of date, +The hardcoded lists are difficult to maintain and are prone to going out of date. despite not fully fixing the use of these improvements have been made for the 4.2.0 release by relaxing rules where security allows. For this the team checked individuals cases and applied where possible, no security issues where found and we added this as a specific thing to pent test. From b40c54f4a01914ea373b8d6482da06adc183ba1c Mon Sep 17 00:00:00 2001 From: Davsarper <118986872+Davsarper@users.noreply.github.com> Date: Fri, 3 May 2024 16:20:57 +0100 Subject: [PATCH 11/12] Update Reports/ProgressReport-Feb24-Aug24.md Co-authored-by: Matt Craddock --- Reports/ProgressReport-Feb24-Aug24.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Reports/ProgressReport-Feb24-Aug24.md b/Reports/ProgressReport-Feb24-Aug24.md index c91eb57..a59e4ed 100644 --- a/Reports/ProgressReport-Feb24-Aug24.md +++ b/Reports/ProgressReport-Feb24-Aug24.md @@ -71,7 +71,7 @@ we are currently considering the covenience of this approach versus potential se Improved use of hardcoded domain names and IPs. The hardcoded lists are difficult to maintain and are prone to going out of date. -despite not fully fixing the use of these improvements have been made for the 4.2.0 release by relaxing rules where security allows. +Despite not fully stopping use of hardcoded domains and IPs, improvements have been made for the 4.2.0 release by relaxing rules where security allows. For this the team checked individuals cases and applied where possible, no security issues where found and we added this as a specific thing to pent test. Related PR is https://github.com/alan-turing-institute/data-safe-haven/pull/1745 and explanatory issue is https://github.com/alan-turing-institute/data-safe-haven/issues/1549 . From 30944eb5953c4bb0a0374f0e33f4bcddcf6ab808 Mon Sep 17 00:00:00 2001 From: Davsarper <118986872+Davsarper@users.noreply.github.com> Date: Fri, 3 May 2024 16:21:23 +0100 Subject: [PATCH 12/12] Update Reports/ProgressReport-Feb24-Aug24.md Co-authored-by: Matt Craddock --- Reports/ProgressReport-Feb24-Aug24.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Reports/ProgressReport-Feb24-Aug24.md b/Reports/ProgressReport-Feb24-Aug24.md index a59e4ed..2eea5d4 100644 --- a/Reports/ProgressReport-Feb24-Aug24.md +++ b/Reports/ProgressReport-Feb24-Aug24.md @@ -80,7 +80,7 @@ An issue with Jupyter notebooks not being able to use Python when launched from despite extensive work a fix was not found and decided to let it be by documenting the right workaround: launching Jupyter Notebooks from the terminal. The issue is https://github.com/alan-turing-institute/data-safe-haven/issues/1584. -Worked on updating documentation to reflect Azure Active Directory name change to Microsoft Entra. +Updated documentation to reflect Azure Active Directory name change to Microsoft Entra. #### Identify and implement core IAC changes: [#28](https://github.com/alan-turing-institute/data-safe-haven-team/issues/28)