Release 12.0.3 (#3107)

* merge CUMULUS-2557

* merge CUMULUS-2971

* merge CUMULUS-3021

* merge CUMULUS-3024

* 12.0.3 version bump

* remove duplicate api user

* Sort results in unit test before comparison

* add ci stack

* CUMULUS-3027 -- Tightly constrain typescript version due to knex typing (#3095)

* Tightly constrain typescript verseion due to knex typing

Unpinned dependency resulted in typescript 4.8 being installed,
however knex (all versions) has an outstanding bug:
knex/knex#5279

* Add CHANGELOG 🔔

* Update generate-ts-build-cache to always 'npm install'

* Update CHANGELOG

* Update CNM lambdas to utilize newer releases (#3094)

Co-authored-by: Charles Huang <[email protected]>
Co-authored-by: jennyhliu <[email protected]>
Co-authored-by: Jonathan Kovarik <[email protected]>
Co-authored-by: Charles Huang <[email protected]>

Author: 4 people

CHANGELOG.md

@@ -6,6 +6,31 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ## Unreleased
 
+## [v12.0.3] 2022-10-03 [BACKPORT]
+
+### Fixed
+
+- **CUMULUS-3024**
+  - Update PUT /granules endpoint to operate consistently across datastores
+    (PostgreSQL, ElasticSearch, DynamoDB). Previously it was possible, given a
+    partial Granule payload to have different data in Dynamo/ElasticSearch and PostgreSQL
+  - Given a partial Granule object, the /granules update endpoint now operates
+    with behavior more consistent with a PATCH operation where fields not provided
+    in the payload will not be updated in the datastores.
+  - Granule translation (db/src/granules.ts) now supports removing null/undefined fields when converting from API to Postgres
+    granule formats.
+  - Update granule write logic: if a `null` files key is provided in an update payload (e.g. `files: null`),
+    an error will be thrown. `null` files were not previously supported and would throw potentially unclear errors. This makes the error clearer and more explicit.
+  - Update granule write logic: If an empty array is provided for the `files` key, all files will be removed in all datastores
+- **CUMULUS-2971**
+  - Updated `@cumulus/aws-client/S3ObjectStore` class to take string query parameters and
+    its methods `signGetObject` and `signHeadObject` to take parameter presignOptions
+- **CUMULUS-2557**
+  - Updated `@cumulus/aws-client/S3/moveObject` to handle zero byte files (0 byte files).
+- **CUMULUS-3021**
+  - Updated `@cumulus/api-client/collections` and `@cumulus/integration-tests/api` to encode
+    collection version in the URI path
+
 ## [v12.0.2] 2022-08-10 [BACKPORT]
 
 ### Notable Changes
@@ -15,6 +40,12 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
       `cumulus_id`. Previous logic removed files by matching `file.cumulus_id`
       to `granule.cumulus_id`.
 
+### Changed
+
+- **CUMULUS-3027**
+  - Pinned typescript to ~4.7.x to address typing incompatibility issues
+    discussed in https://github.com/knex/knex/pull/5279
+  - Update generate-ts-build-cache script to always install root project dependencies
 ## [v12.0.1] 2022-07-18
 
 - **CUMULUS-2995**
@@ -5932,7 +5963,8 @@ Note: There was an issue publishing 1.12.0. Upgrade to 1.12.1.
 
 ## [v1.0.0] - 2018-02-23
 
-[unreleased]: https://github.com/nasa/cumulus/compare/v12.0.2...HEAD
+[unreleased]: https://github.com/nasa/cumulus/compare/v12.0.3...HEAD
+[v12.0.3]: https://github.com/nasa/cumulus/compare/v12.0.2...v12.0.3
 [v12.0.2]: https://github.com/nasa/cumulus/compare/v12.0.1...v12.0.2
 [v12.0.1]: https://github.com/nasa/cumulus/compare/v12.0.0...v12.0.1
 [v12.0.0]: https://github.com/nasa/cumulus/compare/v11.1.0...v12.0.0

bamboo/generate-ts-build-cache.sh

@@ -27,9 +27,10 @@ if [[ $USE_CACHED_BOOTSTRAP == true ]]; then
   git checkout "$GIT_SHA"
 else
   CURRENT_WORKING_DIR=$(pwd)
-  npm install
 fi
 
+npm install
+
 # Bootstrap to install/link packages
 npm run ci:bootstrap-no-scripts
 

bamboo/select-stack.js

@@ -15,6 +15,7 @@ function determineIntegrationTestStackName(cb) {
   if (branch === 'master') return cb('cumulus-source');
 
   const stacks = {
+    'Charles Huang': 'ch-ci-tf',
     'Jenny Liu': 'jl-rds',
     jennyhliu: 'jl-rds',
     kkelly51: 'kk2',

docs/data-cookbooks/cnm-workflow.md

@@ -71,7 +71,7 @@ The following are steps that are required to set up your Cumulus instance to run
 
 In this example, we're going to trigger a workflow by creating a Kinesis rule and sending a record to a Kinesis stream.
 
-The following [workflow definition](workflows/README.md) should be added to a new `.tf` workflow resource (e.g. `cnm_workflow.tf`) in your deployment directory.  For the complete CNM workflow example, see [examples/cumulus-tf/kinesis_trigger_test_workflow.tf](https://github.com/nasa/cumulus/blob/master/example/cumulus-tf/kinesis_trigger_test_workflow.tf).
+The following [workflow definition](workflows/README.md) should be added to a new `.tf` workflow resource (e.g. `cnm_workflow.tf`) in your deployment directory.  For the complete CNM workflow example, see [examples/cumulus-tf/cnm_workflow.tf](https://github.com/nasa/cumulus/blob/master/example/cumulus-tf/cnm_workflow.tf).
 
 Add the following to the new terraform file in your deployment directory, updating the following:
 

example/cumulus-tf/cnm_response_task.tf

@@ -1,5 +1,5 @@
 locals {
-  cnm_response_version  = "2.0.2"
+  cnm_response_version  = "2.0.3"
   cnm_response_filename = "cnmResponse-${local.cnm_response_version}.zip"
 }
 

example/cumulus-tf/cnm_to_cma_task.tf

@@ -1,5 +1,5 @@
 locals {
-  cnm_to_cma_version  = "1.5.3"
+  cnm_to_cma_version  = "1.5.4"
   cnm_to_cma_filename = "cnmToGranule-${local.cnm_to_cma_version}.zip"
 }
 

example/cumulus-tf/kinesis_trigger_test_workflow.asl.json renamed to example/cumulus-tf/cnm_workflow.asl.json

@@ -1,5 +1,5 @@
 {
-  "Comment": "Tests Workflow from Kinesis Stream",
+  "Comment": "Tests CNM workflow",
   "StartAt": "TranslateMessage",
   "States": {
     "TranslateMessage": {
@@ -380,7 +380,7 @@
             "distribution_endpoint": "{$.meta.distribution_endpoint}",
             "response-endpoint": "{$.meta.cnmResponseStream}",
             "region": "us-east-1",
-            "type": "kinesis",
+            "type": "{$.meta.cnmResponseMethod}",
             "WorkflowException": "{$.exception}",
             "cumulus_message": {
               "outputs": [
@@ -434,7 +434,7 @@
             "OriginalCNM": "{$.meta.cnm}",
             "response-endpoint": "{$.meta.cnmResponseStream}",
             "region": "us-east-1",
-            "type": "kinesis",
+            "type": "{$.meta.cnmResponseMethod}",
             "WorkflowException": "{$.exception}",
             "cumulus_message": {
               "outputs": [

example/cumulus-tf/kinesis_trigger_test_workflow.tf renamed to example/cumulus-tf/cnm_workflow.tf

@@ -1,4 +1,4 @@
-module "kinesis_trigger_test_workflow" {
+module "cnm_workflow" {
   depends_on = [
     aws_lambda_function.cnm_response_task,
     aws_lambda_function.cnm_to_cma_task
@@ -7,13 +7,13 @@ module "kinesis_trigger_test_workflow" {
   source = "../../tf-modules/workflow"
 
   prefix          = var.prefix
-  name            = "KinesisTriggerTest"
+  name            = "CNMExampleWorkflow"
   workflow_config = module.cumulus.workflow_config
   system_bucket   = var.system_bucket
   tags            = local.tags
 
   state_machine_definition = templatefile(
-    "${path.module}/kinesis_trigger_test_workflow.asl.json",
+    "${path.module}/cnm_workflow.asl.json",
     {
       cnm_to_cma_task_arn: aws_lambda_function.cnm_to_cma_task.arn,
       cnm_response_task_arn: aws_lambda_function.cnm_response_task.arn,

example/cumulus-tf/main.tf

@@ -162,20 +162,17 @@ module "cumulus" {
   # Archive API settings
   token_secret = var.token_secret
   archive_api_users = [
-    "aortega527",
+    "chuang14",
+    "ds_jennifertran",
     "jasmine",
     "jennyhliu",
     "jmccoy_uat",
-    "kbaynes",
     "kkelly",
     "kovarik",
-    "lfrederick",
-    "matthewsavoie",
-    "mboyd",
     "menno.vandiermen",
     "mobrien84",
-    "npauzenga",
-    "ds_jennifertran"
+    "nnageswa",
+    "npauzenga"
   ]
   archive_api_url             = var.archive_api_url
   archive_api_port            = var.archive_api_port

example/data/collections/ASCATB-L2-Coastal/ASCATB-L2-Coastal.json

@@ -0,0 +1,32 @@
+{
+  "duplicateHandling": "replace",
+  "files": [
+    {
+      "bucket": "protected",
+      "regex": "^ascat_[0-9]{8}_[0-9]{6}_metopb_[0-9]{5}_eps_o_coa_[0-9]{4}_ovw\\.l2\\.nc$",
+      "sampleFileName": "ascat_20121029_010301_metopb_00588_eps_o_coa_2101_ovw.l2.nc",
+      "type": "data"
+    },
+    {
+      "bucket": "private",
+      "regex": "^ascat_[0-9]{8}_[0-9]{6}_metopb_[0-9]{5}_eps_o_coa_[0-9]{4}_ovw\\.l2.*\\.cmr\\.json$",
+      "sampleFileName": "ascat_20121029_010301_metopb_00588_eps_o_coa_2101_ovw.l2.cmr.json",
+      "type": "metadata"
+    },
+    {
+      "bucket": "protected",
+      "regex": "^ascat_[0-9]{8}_[0-9]{6}_metopb_[0-9]{5}_eps_o_coa_[0-9]{4}_ovw\\.l2\\.nc\\.dmrpp$",
+      "sampleFileName": "ascat_20121029_010301_metopb_00588_eps_o_coa_2101_ovw.l2.nc.dmrpp",
+      "type": "metadata"
+    }
+  ],
+  "granuleId": "^ascat_[0-9]{8}_[0-9]{6}_metopb_[0-9]{5}_eps_o_coa_[0-9]{4}_ovw\\.l2$",
+  "granuleIdExtraction": "^(.*)(\\.nc|\\.cmr)",
+  "meta": {
+    "granuleMetadataFileExtension": ".cmr.json"
+  },
+  "name": "ASCATB-L2-Coastal",
+  "sampleFileName": "ascat_20121029_010301_metopb_00588_eps_o_coa_2101_ovw.l2.nc",
+  "url_path": "{cmrMetadata.CollectionReference.ShortName}",
+  "version": "Operational/Near-Real-Time"
+}