Release: 8.4.0 (#2006)

* add changelog * Bugfix/rdmp 253 filter ordering (#2007) * add start * partial * tidy up code * filter updates * update db migrations * update correct db * update changelog * add base creation sql * fix create * tidy up * update insert * tidy up * interim * style updates * add some info * start of where clause * improved * improved dates * improve bookends * add changelog * update changelog * improve ui with frequency * tidy up for tests * add summary * rename chart * interim people * add people label * better where * move items around * fix up tests * tidy up code * tidy up * made db independant * add changelog * add missing file * Bugfix/rdmp 259 delta load off by one issue (#2024) * fix off by one issue * update tests * update changelog * check for multi-server query * update changelog * update changelog * Add override for RAW table date column in delta loads (#2052) * fix table attacher special column issue * add override * add extra override * fix tests * tidy up * add docs * update deps * add safe ref * interim * working check * tidy up * add changelog * fix typos * update typo * fix typos * fix typos * Changes made to popup buttons (#2076) * Changes made to popup buttons * Bump the aws-sdk group with 4 updates (#2074) Bumps the aws-sdk group with 4 updates: [AWSSDK.S3](https://github.com/aws/aws-sdk-net), [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net), [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) and [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net). Updates `AWSSDK.S3` from 3.7.407 to 3.7.407.1 - [Release notes](https://github.com/aws/aws-sdk-net/releases) - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SecurityToken` from 3.7.401.5 to 3.7.401.6 - [Release notes](https://github.com/aws/aws-sdk-net/releases) - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SSO` from 3.7.400.56 to 3.7.400.57 - [Release notes](https://github.com/aws/aws-sdk-net/releases) - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) Updates `AWSSDK.SSOOIDC` from 3.7.400.56 to 3.7.400.57 - [Release notes](https://github.com/aws/aws-sdk-net/releases) - [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD) - [Commits](https://github.com/aws/aws-sdk-net/commits) --- updated-dependencies: - dependency-name: AWSSDK.S3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SecurityToken dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SSO dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk - dependency-name: AWSSDK.SSOOIDC dependency-type: direct:production update-type: version-update:semver-patch dependency-group: aws-sdk ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update CHANGELOG.md * Tidied up code --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Task/rdmp 32 regex redaction (#2009) * interim * interim * redact and add some tests * add missing interface * add regex redaction keys * add helper * add concept * update interfaces * add icons and fix restore * add missing files * add docs * add regions * add ignore * add speed test * improve speed * improve speed * update padding * stress test * add test case * add fk names * faster data load * faster update * more timers * super fast redaction * tidy up code * woking keys * improved * add limiting * shared helper * shared * interim * working * add summary * restore tests working * add tests * start og ui update * start of ui * basic configuraion * working add * user interface * update for tests * update test * tests * tidy up * updates * fix merge * ad dicons * add missing icons * tidyup * fix tests * add missing file * update migration number * add changelog * add timeout * fix build * interim * tidy up * tidy up code * add publish * add todo * add redact all * add column * update table * add docs * update markdown * fix typo * fix typos * tidy up tests * tidy up tests * Include regex redaction doc in VS Solution file * Update ExecuteCommandPerformRegexRedactionOnCatalogueTests.cs Syntax cleanup * Update ThrowImmediatelyActivator.cs Remove obsolete comment * Update ExecuteCommandPerformRegexRedactionOnCatalogueTests.cs Start being green and recycling * Update ExecuteCommandPerformRegexRedactionOnCatalogueTests.cs Finish recycling test objects --------- Co-authored-by: James A Sutherland <[email protected]> * codeql updates * update client xml * code ql updates --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: rdteviotdale <[email protected]> Co-authored-by: James A Sutherland <[email protected]> Co-authored-by: Brian <[email protected]> Co-authored-by: JBaird00183 <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
HicServices · Nov 29, 2024 · b1457d0 · b1457d0
1 parent da60289
commit b1457d0
Show file tree

Hide file tree

Showing 117 changed files with 7,647 additions and 2,013 deletions.
diff --git a/Application/ResearchDataManagementPlatform/WindowManagement/TopBar/RDMPTaskBarUI.Designer.cs b/Application/ResearchDataManagementPlatform/WindowManagement/TopBar/RDMPTaskBarUI.Designer.cs
diff --git a/Application/ResearchDataManagementPlatform/WindowManagement/TopBar/RDMPTaskBarUI.cs b/Application/ResearchDataManagementPlatform/WindowManagement/TopBar/RDMPTaskBarUI.cs
@@ -51,11 +51,11 @@ public RDMPTaskBarUI()
         btnTables.Image = CatalogueIcons.TableInfo.ImageToBitmap();
         btnTables.BackgroundImage = BackColorProvider.GetBackgroundImage(btnTables.Size, RDMPCollection.Tables);
 
-        btnDataSets.Image = CatalogueIcons.Dataset.ImageToBitmap();
-        btnDataSets.BackgroundImage = BackColorProvider.GetBackgroundImage(btnDataSets.Size, RDMPCollection.Datasets);
+        btnConfiguration.Image = FamFamFamIcons.pencil_go.ImageToBitmap();
+        btnConfiguration.BackgroundImage = BackColorProvider.GetBackgroundImage(btnConfiguration.Size, RDMPCollection.Datasets);
 
         btnLoads.Image = CatalogueIcons.LoadMetadata.ImageToBitmap();
-        btnLoads.BackgroundImage = BackColorProvider.GetBackgroundImage(btnDataSets.Size, RDMPCollection.DataLoad);
+        btnLoads.BackgroundImage = BackColorProvider.GetBackgroundImage(btnConfiguration.Size, RDMPCollection.DataLoad);
 
         btnFavourites.Image = CatalogueIcons.Favourite.ImageToBitmap();
         btnDeleteLayout.Image = FamFamFamIcons.delete.ImageToBitmap();
@@ -111,7 +111,7 @@ private void SetupToolTipText()
             btnTables.ToolTipText = "Advanced features e.g. logging, credentials, dashboards etc";
             btnLoads.ToolTipText = "Load configurations for reading data into your databases";
             btnFavourites.ToolTipText = "Collection of all objects that you have favourited";
-            btnDataSets.ToolTipText = "All external datasets that have been configured for use in RDMP";
+            btnConfiguration.ToolTipText = "All external datasets that have been configured for use in RDMP";
         }
         catch (Exception e)
         {
@@ -190,8 +190,8 @@ private RDMPCollection ButtonToEnum(object button)
             collectionToToggle = RDMPCollection.SavedCohorts;
         else if (button == btnFavourites)
             collectionToToggle = RDMPCollection.Favourites;
-        else if (button == btnDataSets)
-            collectionToToggle = RDMPCollection.Datasets;
+        else if (button == btnConfiguration)
+            collectionToToggle = RDMPCollection.Configurations;
         else
             throw new ArgumentOutOfRangeException(nameof(button));
 

diff --git a/Application/ResearchDataManagementPlatform/WindowManagement/TopBar/RDMPTaskBarUI.resx b/Application/ResearchDataManagementPlatform/WindowManagement/TopBar/RDMPTaskBarUI.resx
@@ -178,7 +178,7 @@
         TgDQASA1MVpwzwAAAABJRU5ErkJggg==
 </value>
   </data>
-  <data name="btnDataSets.Image" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
+  <data name="btnConfiguration.Image" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
     <value>
         iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8
         YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAIDSURBVDhPpZLrS5NhGMb3j4SWh0oRQVExD4gonkDpg4hG

diff --git a/Application/ResearchDataManagementPlatform/WindowManagement/WindowManager.cs b/Application/ResearchDataManagementPlatform/WindowManagement/WindowManager.cs
@@ -148,10 +148,10 @@ public PersistableToolboxDockContent Create(RDMPCollection collectionToCreate,
                     Image.Load<Rgba32>(CatalogueIcons.Favourite));
                 break;
 
-            case RDMPCollection.Datasets:
-                collection = new DatasetsCollectionUI();
-                toReturn = Show(RDMPCollection.Datasets, collection, "Datasets",
-                    Image.Load<Rgba32>(CatalogueIcons.Dataset));
+            case RDMPCollection.Configurations:
+                collection = new ConfigurationsCollectionUI();
+                toReturn = Show(RDMPCollection.Configurations, collection, "Configurations",
+                     Image.Load<Rgba32>(FamFamFamIcons.pencil_go));
                 break;
 
             default: throw new ArgumentOutOfRangeException(nameof(collectionToCreate));

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,18 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [8.4.0] - Unreleased
+
+- Add Ordering to Filters
+- [MSSQL ONLY] Add ability to perform Regex redactions on data loads and existing catalogues
+- Add overview page for Catalogues
+- Add RAW Table Date Column Override for Delta Loads
+- Fix Delta Load off by one issue
+- Update Migration strategy to account for all Primary Keys when moving from staging -> live
+- Fix UI issue with viewing cross-database SQL results
+- Add UI Steps to deprecate old cohorts when importing a cohort
+- Add instance setting to enable and disable Yes/No to all
+
 ## [8.3.1] - 2024-10-22
 
 - Improve Performance of regenerating problems with child providers

diff --git a/Documentation/DataLoadEngine/RegexRedactions.md b/Documentation/DataLoadEngine/RegexRedactions.md
@@ -0,0 +1,44 @@
+# Regex Redactions
+
+RDMP has the ability to redact incoming and existing catalogue data.
+It uses user-defined regex to do this.
+This functionality may be useful if you know about problematic data that you wish to clean before it reaches RDMP
+
+## Setup & Configuration
+To begin, a redaction configuration is required.
+This can be added in the "Configurations" section of RDMP.
+Each configuration has:
+* A Name
+* A Description
+* A Regex string to match
+* A Replacement value
+
+N.B. The replacement value cannot be longer than the redacted string i.e. "string" => "myRedactedString" will not work.
+
+Once the configuration has been saved, it can be used to redact data.
+
+## Usage in Data Loads
+Within the Data Loads mutilators, there is a mutilator named "Regex Redaction Mutilator" that can be used in the RAW or STAGING areas of the load.
+This mutilator will redact based on your selected configuration.
+The configuration for this mutilator requires:
+* A predefined regex redaction configuration (see above)
+* Either a regex to select which columns to redact, or a selection of known columns
+
+N.B. Primary Key columns will not be redacted and this functionality is intended for string fields.
+
+Once the configuration has been set, it will redact the desired columns during a data load.
+
+During a run, the mutilator will replace any regex matches with the desired string and store the now known redaction in RDMP.
+To view the redactions made, see the section below.
+
+
+## Usage within RDMP
+You can view and restore existing redactions, along with adding new redactions within RDMP.
+For a Catalogue, right-click and open the "Catalogue Items" submenu. 
+From here, select the "Regex Redactions" option.
+This will open an interface displaying existing redactions.
+From here you can restore existing redactions either individually or in bulk.
+You can also apply regex redactions to existing data by selecting a redaction configuration and which columns to redact.
+
+
+[Catalogue](../CodeTutorials/Glossary.md#Catalogue)
diff --git a/Documentation/DataLoadEngine/RemoteAttachers.md b/Documentation/DataLoadEngine/RemoteAttachers.md
@@ -50,6 +50,7 @@ The full configuration options are
 | Culture                                      | Optionally specify a custom date format                                                                                                                                                                                                                                                                                                                     |
 | Explicit Date Time Format                    | Optionally specify a specific datetime format    
 | Selected Columns                             | Optionally select which columns you wish to pull from the remote server (defaults to "*" )
+| RawTableDateColumn                           | Optionally give the date column in RAW a different name than in the select statement for delta loads. This may be due to awkward joins in a custom select query 
 
 ## Configuring the Remote Database Attacher
 The Remote Database Attacher has a number of configuration options the required fields are:

diff --git a/HIC.DataManagementPlatform.sln b/HIC.DataManagementPlatform.sln
@@ -10,8 +10,8 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution
 		CHANGELOG.md = CHANGELOG.md
 		deadlinksconfig.json = deadlinksconfig.json
 		Directory.Build.props = Directory.Build.props
-		.github\workflows\links.yml = .github\workflows\links.yml
 		Directory.Packages.props = Directory.Packages.props
+		.github\workflows\links.yml = .github\workflows\links.yml
 		NoteForNewDevelopers.md = NoteForNewDevelopers.md
 		Documentation\CodeTutorials\Packages.md = Documentation\CodeTutorials\Packages.md
 		rdmp-client.xml = rdmp-client.xml
@@ -55,6 +55,7 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Documentation", "Documentat
 		Documentation\CodeTutorials\Packages.md = Documentation\CodeTutorials\Packages.md
 		Documentation\CodeTutorials\PluginWriting.md = Documentation\CodeTutorials\PluginWriting.md
 		Documentation\CodeTutorials\RdmpCommandLine.md = Documentation\CodeTutorials\RdmpCommandLine.md
+		Documentation\DataLoadEngine\RegexRedactions.md = Documentation\DataLoadEngine\RegexRedactions.md
 		Documentation\CodeTutorials\Tests.md = Documentation\CodeTutorials\Tests.md
 		Documentation\CodeTutorials\UserInterfaceOverview.md = Documentation\CodeTutorials\UserInterfaceOverview.md
 		Documentation\CodeTutorials\UserManual.md = Documentation\CodeTutorials\UserManual.md
@@ -315,4 +316,4 @@ Global
 	GlobalSection(Performance) = preSolution
 		HasPerformanceSessions = true
 	EndGlobalSection
-EndGlobal
+EndGlobal