Releases: nasa/cumulus
v1.18.0
This is a fairly minor breaking release. See migration steps below for required changes.
Migration Steps
-
If not already set, set
ecs_cluster_instance_image_id
in yourterraform.tfvars
for yourcumulus
module deployment -
Change variable
saml_launchpad_metadata_path
tosaml_launchpad_metadata_url
in thecumulus
Terraform module. -
Remove
process
fromPostToCmr
task_config in workflow definitions. -
Please note function removals and deprecations listed below in the
Removed
andDeprecated
sections.
Breaking Changes
-
CUMULUS-1686
ecs_cluster_instance_image_id
is now a required variable of thecumulus
module, instead of optional.
-
CUMULUS-1698
- Change variable
saml_launchpad_metadata_path
tosaml_launchpad_metadata_url
in thetf-modules/cumulus
Terraform module.
- Change variable
-
CUMULUS-1703
- Remove the unused
forceDownload
option from thesync-granule
tasks's config - Remove the
@cumulus/ingest/granule.Discover
class - Remove the
@cumulus/ingest/granule.Granule
class - Remove the
@cumulus/ingest/pdr.Discover
class - Remove the
@cumulus/ingest/pdr.Granule
class - Remove the
@cumulus/ingest/parse-pdr.parsePdr
function
- Remove the unused
Other Notable Changes
- Upgrade Thin Egress App (TEA) to version 48
- Fixed granule discovery recursion algorithm used in S/FTP protocols.
All Changes
Added
-
CUMULUS-1040
- Added
@cumulus/aws-client
package to provide utilities for working with AWS services and the Node.js AWS SDK - Added
@cumulus/errors
package which exports error classes for use in Cumulus workflow code - Added
@cumulus/integration-tests/sfnStep
to provide utilities for parsing step function execution histories
- Added
-
CUMULUS-1102
- Adds functionality to the @cumulus/api package for better local testing.
- Adds data seeding for @cumulus/api's localAPI.
- seed functions allow adding collections, executions, granules, pdrs, providers, and rules to a Localstack Elasticsearch and DynamoDB via
addCollections
,addExecutions
,addGranules
,addPdrs
,addProviders
, andaddRules
.
- seed functions allow adding collections, executions, granules, pdrs, providers, and rules to a Localstack Elasticsearch and DynamoDB via
- Adds
eraseDataStack
function to local API server code allowing resetting of local datastack for testing (ES and DynamoDB). - Adds optional parameters to the @cumulus/api bin serve to allow for launching the api without destroying the current data.
- Adds data seeding for @cumulus/api's localAPI.
- Adds functionality to the @cumulus/api package for better local testing.
-
CUMULUS-1697
- Added the
@cumulus/tf-inventory
package that provides command line utilities for managing Terraform resources in your AWS account
- Added the
-
CUMULUS-1703
- Add
@cumulus/aws-client/S3.createBucket
function - Add
@cumulus/aws-client/S3.putFile
function - Add
@cumulus/common/string.isNonEmptyString
function - Add
@cumulus/ingest/FtpProviderClient
class - Add
@cumulus/ingest/HttpProviderClient
class - Add
@cumulus/ingest/S3ProviderClient
class - Add
@cumulus/ingest/SftpProviderClient
class - Add
@cumulus/ingest/providerClientUtils.buildProviderClient
function - Add
@cumulus/ingest/providerClientUtils.fetchTextFile
function
- Add
-
CUMULUS-1731
- Add new optional input variables to the Cumulus Terraform module to support TEA upgrade:
thin_egress_cookie_domain
- Valid domain for Thin Egress App cookiethin_egress_domain_cert_arn
- Certificate Manager SSL Cert ARN for Thin
Egress App if deployed outside NGAP/CloudFrontthin_egress_download_role_in_region_arn
- ARN for reading of Thin Egress
App data buckets for in-region requeststhin_egress_jwt_algo
- Algorithm with which to encode the Thin Egress
App JWT cookiethin_egress_jwt_secret_name
- Name of AWS secret where keys for the Thin
Egress App JWT encode/decode are storedthin_egress_lambda_code_dependency_archive_key
- Thin Egress App - S3
Key of packaged python modules for lambda dependency layer
- Add new optional input variables to the Cumulus Terraform module to support TEA upgrade:
-
CUMULUS-1733
- Add
discovery-filtering
operator doc to document previously undocumented functionality.
- Add
Changed
-
CUMULUS-1102
- Updates
@cumulus/api/auth/testAuth
to use JWT instead of random tokens. - Updates the default AMI for the ecs_cluster_instance_image_id.
- Updates
-
CUMULUS-1622
- Mutex class has been deprecated in
@cumulus/common/concurrency
and will be removed in a future release.
- Mutex class has been deprecated in
-
CUMULUS-1686
- Changed
ecs_cluster_instance_image_id
to be a required variable of thecumulus
module and removed the default value.
The default was not available across accounts and regions, nor outside of NGAP and therefore not particularly useful.
- Changed
-
CUMULUS-1688
- Updated
@cumulus/aws.receiveSQSMessages
not to replacemessage.Body
with a parsed object. This behavior was undocumented and confusing as received messages appeared to contradict AWS docs that statemessage.Body
is always a string. - Replaced
sf_watcher
CloudWatch rule fromcloudwatch-events.tf
with an EventSourceMapping onsqs2sf
mapped to thestart_sf
SQS queue (inevent-sources.tf
). - Updated
sqs2sf
with an EventSourceMapping handler and unit test.
- Updated
-
CUMULUS-1698
- Change variable
saml_launchpad_metadata_path
tosaml_launchpad_metadata_url
in thetf-modules/cumulus
Terraform module. - Updated
@cumulus/api/launchpadSaml
to download launchpad IDP metadata from configured location when the metadata in s3 is not valid, and to work with updated IDP metadata and SAML response.
- Change variable
-
CUMULUS-1731
- Upgrade the version of the Thin Egress App deployed by Cumulus to v48
- Note: New variables available, see the 'Added' section of this changelog.
- Upgrade the version of the Thin Egress App deployed by Cumulus to v48
Fixed
-
CUMULUS-1664
- Updated
dbIndexer
Lambda to remove hardcoded references to DynamoDB table names.
- Updated
-
CUMULUS-1733
- Fixed granule discovery recursion algorithm used in S/FTP protocols.
Removed
-
CUMULUS-1481
- removed
process
config and output from PostToCmr as it was not required by the task nor downstream steps, and should still be in the output message'smeta
regardless.
- removed
-
CUMULUS-1703
- Remove the unused
forceDownload
option from thesync-granule
tasks's config - Remove the
@cumulus/ingest/granule.Discover
class - Remove the
@cumulus/ingest/granule.Granule
class - Remove the
@cumulus/ingest/pdr.Discover
class - Remove the
@cumulus/ingest/pdr.Granule
class - Remove the
@cumulus/ingest/parse-pdr.parsePdr
function
- Remove the unused
Deprecated
- CUMULUS-1040
- Deprecated the following code. For cases where the code was moved into another package, the new code location is noted:
@cumulus/common/CloudFormationGateway
->@cumulus/aws-client/CloudFormationGateway
@cumulus/common/DynamoDb
->@cumulus/aws-client/DynamoDb
@cumulus/common/errors
->@cumulus/errors
@cumulus/common/StepFunctions
->@cumulus/aws-client/StepFunctions
- All of the exported functions in
@cumulus/commmon/aws
(moved into@cumulus/aws-client
), except:@cumulus/common/aws/isThrottlingException
->@cumulus/errors/isThrottlingException
@cumulus/common/aws/improveStackTrace
(not deprecated)@cumulus/common/aws/retryOnThrottlingException
(not deprecated)
@cumulus/common/sfnStep/SfnStep.parseStepMessage
->@cumulus/integration-tests/sfnStep/SfnStep.parseStepMessage
@cumulus/common/sfnStep/ActivityStep
->@cumulus/integration-tests/sfnStep/ActivityStep
@cumulus/common/sfnStep/LambdaStep
->@cumulus/integration-tests/sfnStep/LambdaStep
@cumulus/common/string/unicodeEscape
->@cumulus/aws-client/StepFunctions.unicodeEscape
@cumulus/common/util/setErrorStack
->@cumulus/aws-client/util/setErrorStack
@cumulus/ingest/aws/invoke
->@cumulus/aws-client/Lambda/invoke
@cumulus/ingest/aws/CloudWatch.bucketSize
@cumulus/ingest/aws/CloudWatch.cw
@cumulus/ingest/aws/ECS.ecs
@cumulus/ingest/aws/ECS
@cumulus/ingest/aws/Events.putEvent
->@cumulus/aws-client/CloudwatchEvents.putEvent
@cumulus/ingest/aws/Events.deleteEvent
->@cumulus/aws-client/CloudwatchEvents.deleteEvent
@cumulus/ingest/aws/Events.deleteTarget
->@cumulus/aws-client/CloudwatchEvents.deleteTarget
@cumulus/ingest/aws/Events.putTarget
->@cumulus/aws-client/CloudwatchEvents.putTarget
@cumulus/ingest/aws/SQS.attributes
->@cumulus/aws-client/SQS.getQueueAttributes
@cumulus/ingest/aws/SQS.deleteMessage
->@cumulus/aws-client/SQS.deleteSQSMessage
@cumulus/ingest/aws/SQS.deleteQueue
->@cumulus/aws-client/SQS.deleteQueue
@cumulus/ingest/aws/SQS.getUrl
->@cumulus/aws-client/SQS.getQueueUrlByName
@cumulus/ingest/aws/SQS.receiveMessage
->@cumulus/aws-client/SQS.receiveSQSMessages
@cumulus/ingest/aws/SQS.sendMessage
->@cumulus/aws-client/SQS.sendSQSMessage
@cumulus/ingest/aws/StepFunction.getExecutionStatus
->@cumulus/aws-client/StepFunction.getExecutionStatus
@cumulus/ingest/aws/StepFunction.getExecutionUrl
->@cumulus/aws-client/StepFunction.getExecutionUrl
- Deprecated the following code. For cases where the code was moved into another package, the new code location is noted:
v1.17.0
This release updates all Cumulus internal lambdas/provided workflow lambdas to utilize a node 10 runtime in anticipation of node 8's end-of-life (https://docs.aws.amazon.com/lambda/latest/dg/runtime-support-policy.html).
All users on releases > 1.15.0 should update to this release prior to node 8's deprecation dates on AWS.
This release also includes an update (CUMULUS-1498) that updates Cumulus's CMR functions such that they utilize AWS Secrets Manager, this is considered a breaking change due to the changed dependency.
Migration Instructions
- If your permissions boundaries or IAMs do not support AWS Secrets Manager, they will need to be updated. For NASA NGAP users, no change should be required.
- This update requires use of the Cumulus Message Adapter >= 1.1.2 to allow Cumulus's internal lambdas to update to node 10, and should also be used for any custom workflow lambdas that need to use node 10 or later:
- Follow the CMA deployment instructions to update your CMA layer, and ensure your terraform configuration has been updated to use the updated CMA layer
- Ensure that custom node workflow lambdas are updated to utilize CMA >= 1.1.2 and cumulus-message-adapter-js >= 1.1.0
Changes
Added
-
CUMULUS-630
- Added support for replaying Kinesis records on a stream into the Cumulus Kinesis workflow triggering mechanism: either all the records, or some time slice delimited by start and end timestamps.
- Added
/replays
endpoint to the operator API for triggering replays. - Added
Replay Kinesis Messages
documentation to Operator Docs. - Added
manualConsumer
lambda function to consume a Kinesis stream. Used by the replay AsyncOperation.
-
CUMULUS-1687
- Added new API endpoint for listing async operations at
/asyncOperations
- All asyncOperations now include the fields
description
andoperationType
.operationType
can be one of the following. [Bulk Delete
,Bulk Granules
,ES Index
,Kinesis Replay
]
- Added new API endpoint for listing async operations at
Changed
-
CUMULUS-1626
- Updates Cumulus to use node10/CMA 1.1.2 for all of its internal lambdas in prep for AWS node 8 EOL
-
CUMULUS-1498
- The
@cumulus/cmrjs.publish2CMR
function expects that the value of its
creds.password
parameter is a plaintext password. - Rather than using an encrypted password from the
cmr_password
environment
variable, the@cumulus/cmrjs.updateCMRMetadata
function now looks for an
environment variable calledcmr_password_secret_name
and fetches the CMR
password from that secret in AWS Secrets Manager. - The
@cumulus/post-to-cmr
task now expects a
config.cmr.passwordSecretName
value, rather thanconfig.cmr.password
.
The CMR password will be fetched from that secret in AWS Secrets Manager.
- The
v1.14.5
This release updates all Cumulus internal lambdas/provided workflow lambdas to utilize a node 10 runtime in anticipation of node 8's end-of-life (https://docs.aws.amazon.com/lambda/latest/dg/runtime-support-policy.html).
All users on releases < 1.14.5 who are not > 1.16 should update to this prior to node 8's deprecation dates on AWS.
Migration Instructions
- This update requires use of the Cumulus Message Adapter >= 1.1.2 to allow Cumulus's internal lambdas to update to node 10, and should also be used for any custom workflow lambdas that need to use node 10 or later:
- Follow the CMA deployment instructions to update your CMA configuration to utilize version >= 1.1.2
- Ensure that custom node workflow lambdas are updated to utilize CMA >= 1.1.2 and cumulus-message-adapter-js >= 1.1.0
Updated
- CUMULUS-1626
- Updates Cumulus to use node10/CMA 1.1.2 for all of its internal lambdas in prep for AWS node 8 EOL
v1.16.1
Please Note
-
The
region
argument to thecumulus
Terraform module has been removed. You may see a warning or error if you have that variable populated. -
Your workflow tasks should use the following versions of the CMA libraries to utilize new granule, parentArn, asyncOperationId, and stackName fields on the logs:
cumulus-message-adapter-js
version 1.0.10+cumulus-message-adapter-python
version 1.1.1+cumulus-message-adapter-java
version 1.2.11+
-
The
data-persistence
module no longer manages the creation of an Elasticsearch service-linked role for deploying Elasticsearch to a VPC. Follow the deployment instructions on preparing your VPC for guidance on how to create the Elasticsearch service-linked role manually. -
There is now a
distribution_api_gateway_stage
variable for thetf-modules/cumulus
Terraform module that will be used as the API gateway stage name used for the distribution API (Thin Egress App) -
Default value for the
urs_url
variable is nowhttps://uat.urs.earthdata.nasa.gov/
in thetf-modules/cumulus
andtf-modules/archive
Terraform modules. So deploying thecumulus
module without aurs_url
variable set will integrate your Cumulus deployment with the UAT URS environment.
Added
-
CUMULUS-1563
- Added
custom_domain_name
variable totf-modules/data-persistence
module
- Added
-
CUMULUS-1654
- Added new helpers to
@cumulus/common/execution-history
:getStepExitedEvent()
returns theTaskStateExited
event in a workflow execution history after the given step completion/failure eventgetTaskExitedEventOutput()
returns the output message for aTaskStateExited
event in a workflow execution history
- Added new helpers to
Changed
-
CUMULUS-1578
- Updates SAML launchpad configuration to authorize via configured userGroup.
See the NASA specific documentation (protected)
- Updates SAML launchpad configuration to authorize via configured userGroup.
-
CUMULUS-1579
- Elasticsearch list queries use
match
instead ofterm
.term
had been analyzing the terms and not supporting-
in the field values.
- Elasticsearch list queries use
-
CUMULUS-1619
- Adds 4 new keys to
@cumulus/logger
to display granules, parentArn, asyncOperationId, and stackName. - Depends on
cumulus-message-adapter-js
version 1.0.10+. Cumulus tasks updated to use this version.
- Adds 4 new keys to
-
CUMULUS-1654
- Changed
@cumulus/common/SfnStep.parseStepMessage()
to a static class method
- Changed
-
CUMULUS-1641
- Added
meta.retries
andmeta.visibilityTimeout
properties to sqs-type rule. To create sqs-type rule, you're required to configure a dead-letter queue on your queue. - Added
sqsMessageRemover
lambda which removes the message from SQS queue upon successful workflow execution. - Updated
sqsMessageConsumer
lambda to not delete message from SQS queue, and to retry the SQS message for configured number of times.
- Added
Removed
-
Removed
create_service_linked_role
variable fromtf-modules/data-persistence
module. -
CUMULUS-1321
- The
region
argument to thecumulus
Terraform module has been removed
- The
Fixed
-
CUMULUS-1668 - Fixed a race condition where executions may not have been
added to the database correctly -
CUMULUS-1654 - Fixed issue with
publishReports
Lambda not including workflow execution error information for failed workflows with a single step -
Fixed
tf-modules/cumulus
module so that theurs_url
variable is passed on to its invocation of thetf-modules/archive
module
v1.16.0
Migration Instructions
As of this release, kes deployments are considered deprecated. Deployments will now be done using Terraform and all new feature deployments will be available in Terraform only.
It is recommended you replace your stack with a Terraform deployment. If you need to migrate an existing kes stack, please reach out to the Cumulus team.
The documentation, especially the deployment documentation has been revamped, so please check out how to deploy and the updated template deploy repo.
Note the artifacts attached to this release. They will be needed for your Terraform deployment.
Changelog Notes
Added
- CUMULUS-1580
- Added
/granules/bulk
endpoint to@cumulus/api
to perform bulk actions on granules given either a list of granule ids or an Elasticsearch query and the workflow to perform.
- Added
Changed
-
CUMULUS-1561
- Fix the way that we are handling Terraform provider version requirements
- Pass provider configs into child modules using the method that the
Terraform documentation
suggests - Remove the
region
input variable from thes3_access_test
Terraform module - Remove the
aws_profile
andaws_region
input variables from the
s3-replicator
Terraform module
-
CUMULUS-1639
- Because of
S3's Data Consistency Model,
there may be situations where a GET operation for an object can temporarily
return aNoSuchKey
response even if that object has been created. The
@cumulus/common/aws.getS3Object()
function has been updated to support
retries if aNoSuchKey
response is returned by S3. This behavior can be
enabled by passing aretryOptions
object to that function. Supported
values for that object can be found here:
https://github.com/tim-kos/node-retry#retryoperationoptions
- Because of
Removed
- CUMULUS-1559
logToSharedDestination
has been migrated to the Terraform deployment aslog_api_gateway_to_cloudwatch
and will ONLY apply to egress lambdas.
Due to the differences in the Terraform deployment model, we cannot support a global log subscription toggle for a configurable subset of lambdas.
However, setting up your own log forwarding for a Lambda with Terraform is fairly simple, as you will only need to add SubscriptionFilters to your Terraform configuration, one per log group.
See the Terraform documentation for details on how to do this.
An empty FilterPattern ("") will capture all logs in a group.
v1.15.0
This release includes breaking deployment and workflow changes.
Migration Instructions
Deployment configuration updates
- If you have pinned your message adapter version to < v.1.1.1, please update it to use 1.1.1 or above
- Update your ECS Autoscaling configurations. Remove the following params from your app/config.yml:
- ecs.services.<NAME>.minTasks
- ecs.services.<NAME>.maxTasks
- ecs.services.<NAME>.scaleInActivityScheduleTime
- ecs.services.<NAME>.scaleInAdjustmentPercent
- ecs.services.<NAME>.scaleOutActivityScheduleTime
- ecs.services.<NAME>.scaleOutAdjustmentPercent
- ecs.services.<NAME>.activityName
- Define your autoscaling settings in a kes override file. See the notes for CUMULUS-1470 below and this example
Workflow updates
- Remove StartStatus and StopStatus tasks from each workflow. These have been replaced by CloudWatch events.
- Remove
CumulusConfig
for task configuration and addcma.task_config
for task configuration.
NOTE: These instructions require the use of Cumulus Message Adapter v1.1.x+.
Please ensure you are using a compatible version before attempting to migrate
workflow configurations. When defining workflow steps, remove any
CumulusConfig
section, as shown below:
ParsePdr:
CumulusConfig:
provider: '{$.meta.provider}'
bucket: '{$.meta.buckets.internal.name}'
stack: '{$.meta.stack}'
Instead, use AWS Parameters to pass task_config
for the task directly into
the Cumulus Message Adapter:
ParsePdr:
Parameters:
cma:
event.$: '$'
task_config:
provider: '{$.meta.provider}'
bucket: '{$.meta.buckets.internal.name}'
stack: '{$.meta.stack}'
In this example, the cma
key is used to pass parameters to the message
adapter. Using task_config
in combination with event.$: '$'
allows the
message adapter to process task_config
as the config
passed to the Cumulus
task. See example/workflows/sips.yml
in the core repository for further
examples of how to set the Parameters.
Additionally, workflow configurations for the QueueGranules
and QueuePdrs
tasks need to be updated:
queue-pdrs
config changes:parsePdrMessageTemplateUri
replaced withparsePdrWorkflow
, which is
the workflow name (i.e. top-level name inconfig.yml
, e.g. 'ParsePdr').internalBucket
andstackName
configs now required to look up
configuration from the deployment. Brings the task config in line with
that ofqueue-granules
.
queue-granules
config change:ingestGranuleMessageTemplateUri
replaced
withingestGranuleWorkflow
, which is the workflow name (e.g.
'IngestGranule').
Breaking Changes
-
CUMULUS-1449 - Cumulus now uses a universal workflow template when
starting a workflow that contains general information specific to the
deployment, but not specific to the workflow. Workflow task configs must be
defined using AWS step function parameters. As part of this change,
CumulusConfig
has been retired and task configs must now be defined under
thecma.task_config
key in the Parameters section of a step function
definition. -
CUMULUS-1396 - Workflow steps at the beginning and end of a workflow
using theSfSnsReport
Lambda have now been deprecated (e.g.StartStatus
,
StopStatus
) and should be removed from your workflow definitions. These
steps were used for publishing ingest notifications and have been replaced by
an implementation using Cloudwatch events for Step Functions to trigger a
Lambda that publishes ingest notifications. For further detail on how ingest
notifications are published, see the notes below on CUMULUS-1394. For
examples of how to update your workflow definitions, see our
example workflow definitions. -
CUMULUS-1470
- Remove Cumulus-defined ECS service autoscaling, allowing integrators to
better customize autoscaling to meet their needs. In order to use
autoscaling with ECS services, appropriate
AWS::ApplicationAutoScaling::ScalableTarget
,
AWS::ApplicationAutoScaling::ScalingPolicy
, andAWS::CloudWatch::Alarm
resources should be defined in a kes overrides file. See
this example
for an example. - The following config parameters are no longer used:
- ecs.services.<NAME>.minTasks
- ecs.services.<NAME>.maxTasks
- ecs.services.<NAME>.scaleInActivityScheduleTime
- ecs.services.<NAME>.scaleInAdjustmentPercent
- ecs.services.<NAME>.scaleOutActivityScheduleTime
- ecs.services.<NAME>.scaleOutAdjustmentPercent
- ecs.services.<NAME>.activityName
- Remove Cumulus-defined ECS service autoscaling, allowing integrators to
Additional Notable Changes
- The Cloudformation stack for database resources (e.g.
prefix-db
) now includesDeletionPolicy: Retain
for the DynamoDB tables and the Elasticsearch instance, meaning that those resources will not be deleted when you delete the Cloudformation stack. If you want those resources deleted, you will have to delete them manually. - SQS-type rule support
Added
-
CUMULUS-1100
- Added 30-day retention properties to all log groups that were missing those policies.
-
CUMULUS-1396
- Added
@cumulus/common/sfnStep
:LambdaStep
- A class for retrieving and parsing input and output to Lambda steps in AWS Step FunctionsActivityStep
- A class for retrieving and parsing input and output to ECS activity steps in AWS Step Functions
- Added
-
CUMULUS-1574
- Added
GET /token
endpoint for SAML authorization when cumulus is protected by Launchpad.
This lets a user retrieve a token by hand that can be presented to the API.
- Added
-
CUMULUS-1625
- Added
sf_start_rate
variable to theingest
Terraform module, equivalent tosqs_consumer_rate
in the old model, but will not be automatically applied to custom queues as that was.
- Added
-
CUMULUS-1513
- Added
sqs
-type rule support in the Cumulus API@cumulus/api
- Added
sqsMessageConsumer
lambda which processes messages from the SQS queues configured in thesqs
rules.
- Added
Changed
-
CUMULUS-1449
queue-pdrs
&queue-granules
config changes. Details in breaking changes section.- Cumulus now uses a universal workflow template when starting workflow that contains general information specific to the deployment, but not specific to the workflow.
- Changed the way workflow configs are defined, from
CumulusConfig
to atask_config
AWS Parameter.
-
CUMULUS-1452
- Changed the default ECS docker storage drive to
devicemapper
- Changed the default ECS docker storage drive to
-
CUMULUS-1453
- Removed config schema for
@cumulus/sf-sns-report
task - Updated
@cumulus/sf-sns-report
to always assume that it is running as an intermediate step in a workflow, not as the first or last step
- Removed config schema for
Removed
- CUMULUS-1449
- Retired
CumulusConfig
as part of step function definitions, as this is an artifact of the way Kes parses workflow definitions that was not possible to migrate to Terraform. Use AWS Parameters and thetask_config
key instead. See change note above. - Removed individual workflow templates.
- Retired
Fixed
-
CUMULUS-1620 - Fixed bug where
message_adapter_version
does not correctly inject the CMA -
CUMULUS-1396 - Updated
@cumulus/common/StepFunctions.getExecutionHistory()
to recursively fetch execution history whennextToken
is returned in response -
CUMULUS-1571 - Updated
@cumulus/common/DynamoDb.get()
to throw any errors encountered when trying to get a record and the record does exist -
CUMULUS-1452
- Updated the EC2 initialization scripts to use full volume size for docker storage
- Changed the default ECS docker storage drive to
devicemapper
v1.13.6
This release provides a bug fix to 1.13.5. Users of the v1.13.x baseline are advised to upgrade to this release instead of 1.13.5 if you are utilizing or plan to utilize the message_adapter_version configuration key.
Fixed
- CUMULUS-1620 - Fixes bug where message_adapter_version does not correctly inject the CMA.
v1.14.4
Fixed
- CUMULUS-1632 - Pinned
aws-elasticsearch-connector
package in@cumulus/api
to version8.1.3
, since8.2.0
includes breaking changes
v1.14.3
This release provides a bug fix to 1.14.2. Users are advised to upgrade to this release instead of 1.14.2 if you are utilizing or plan to utilize the message_adapter_version
configuration key.
Fixed
- CUMULUS-1620 - Fixes bug where
message_adapter_version
does not correctly inject the CMA.
v1.14.2
Migration Instructions
Your Cumulus Message Adapter version should be pinned to v1.0.13
or lower in your app/config.yml
using message_adapter_version: v1.0.13
OR
you should use the workflow migration steps below to work with CMA v1.1.1+.
The newest release of the Cumulus Message Adapter (v1.1.1) requires that parameterized configuration be used for remote message functionality. Once released, Kes will automatically bring in CMA v1.1.1 without additional configuration.
Migration instructions for CMA 1.1.1+
Oversized messages are no longer written to S3 automatically. In order to utilize remote messaging functionality, configure a ReplaceConfig
AWS Step Function parameter on your CMA task:
ParsePdr:
Parameters:
cma:
event.$: '$'
ReplaceConfig:
FullMessage: true
Accepted fields in ReplaceConfig
include MaxSize
, FullMessage
, Path
and TargetPath
.
See https://github.com/nasa/cumulus-message-adapter/blob/master/CONTRACT.md#remote-message-configuration for full details.
As this change is backward compatible in Cumulus Core, users wishing to utilize the previous version of the CMA may opt to transition to using a CMA lambda layer, or set message_adapter_version
in their configuration to a version prior to v1.1.0.
Workflow reporting - CUMULUS-1394
The implementation of the SfSnsReport
Lambda requires additional environment variables for integration with the new ingest notification SNS topics. Therefore, you must update the definition of SfSnsReport
in your lambdas.yml
like so:
SfSnsReport:
handler: index.handler
timeout: 300
source: node_modules/@cumulus/sf-sns-report/dist
tables:
- ExecutionsTable
envs:
execution_sns_topic_arn:
function: Ref
value: reportExecutionsSns
granule_sns_topic_arn:
function: Ref
value: reportGranulesSns
pdr_sns_topic_arn:
function: Ref
value: reportPdrsSns
Notable Changes
-
NASA Launchpad is an option for authentication for CMR, the API, and dashboard. This must be set up and configured. NASA users can find Launchpad configuration documentation here for CMR and here for the API. Please note you will have to get the latest version of the Cumulus dashboard as well.
-
Ingest notifications are now provided via 3 separate SNS topics for executions, granules, and PDRs, instead of a single
sftracker
SNS topic. Whereas thesftracker
SNS topic received a full Cumulus execution message, the new topics all receive generated records for the given object. The new topics are only published to if the given object exists for the current execution. For a given execution/granule/PDR, two messages will be received by each topic: one message indicating that ingest is running and another message indicating that ingest has completed or failed. The new SNS topics are:reportExecutions
- Receives 1 message per executionreportGranules
- Receives 1 message per granule in an executionreportPdrs
- Receives 1 message per PDR
Added
-
CUMULUS-1394
- Added
Granule.generateGranuleRecord()
method to granules model to generate a granule database record from a Cumulus execution message - Added
Pdr.generatePdrRecord()
method to PDRs model to generate a granule database record from a Cumulus execution message - Added helpers to
@cumulus/common/message
:getMessageExecutionName()
- Get the execution name from a Cumulus execution messagegetMessageStateMachineArn()
- Get the state machine ARN from a Cumulus execution messagegetMessageExecutionArn()
- Get the execution ARN for a Cumulus execution messagegetMessageGranules()
- Get the granules from a Cumulus execution message, if any.
- Added
@cumulus/common/cloudwatch-event/isFailedSfStatus()
to determine if a Step Function status from a Cloudwatch event is a failed status
- Added
-
CUMULUS-639
- Adds SAML JWT and launchpad token authentication to Cumulus API (configurable)
- NOTE to authenticate with Launchpad ensure your launchpad user_id is in the
<prefix>-UsersTable
- when Cumulus configured to protect API via Launchpad:
- New endpoints
GET /saml/login
- starting point for SAML SSO creates the login request url and redirects to the SAML Identity Provider Service (IDP)POST /saml/auth
- SAML Assertion Consumer Service. POST receiver from SAML IDP. Validates response, logs the user in, and returnes a SAML-based JWT.
- Disabled endpoints
GET /token
POST /refresh
- Changes authorization worklow:ensureAuthorized
now presumes the bearer token is a JWT and tries to validate. If the token is malformed, it attempts to validate the token against Launchpad. This allows users to bring their own token as described here https://wiki.earthdata.nasa.gov/display/CUMULUS/Cumulus+API+with+Launchpad+Authentication. But it also allows dashboard users to manually authenticate via Launchpad SAML to receive a Launchpad-based JWT.
- New endpoints
- NOTE to authenticate with Launchpad ensure your launchpad user_id is in the
- Adds SAML JWT and launchpad token authentication to Cumulus API (configurable)
Changed
-
CUMULUS-1485 Update
@cumulus/cmr-client
to return error message from CMR for validation failures. -
CUMULUS-1394
- Renamed
Execution.generateDocFromPayload()
toExecution.generateRecord()
on executions model. The method generates an execution database record from a Cumulus execution message.
- Renamed
-
CUMULUS-1432
logs
endpoint takes the level parameter as a string and not a number- Elasticsearch term query generation no longer converts numbers to boolean
-
CUMULUS-1447
- Consolidated all remote message handling code into @common/aws
- Update remote message code to handle updated CMA remote message flags
- Update example SIPS workflows to utilize Parameterized CMA configuration
-
CUMULUS-1448 Refactor workflows that are mutating cumulus_meta to utilize meta field
-
CUMULUS-1375
- Migrate Cumulus from deprecated Elasticsearch JS client to new, supported one in
@cumulus/api
- Migrate Cumulus from deprecated Elasticsearch JS client to new, supported one in
-
CUMULUS-1451
- Elasticsearch cluster setting
auto_create_index
will be set to false. This had been causing issues in the bootstrap lambda on deploy.
- Elasticsearch cluster setting
-
CUMULUS-1456
@cumulus/api
endpoints default error handler usesboom
package to format errors, which is consistent with other API endpoint errors.
Fixed
- CUMULUS-1432
logs
endpoint filter correctly filters logs by level - CUMULUS-1484
useMessageAdapter
now does not set CUMULUS_MESSAGE_ADAPTER_DIR whentrue
Removed
- CUMULUS-1394
- Removed
sfTracker
SNS topic. Replaced by three new SNS topics for granule, execution, and PDR ingest notifications. - Removed unused functions from
@cumulus/common/aws
:getGranuleS3Params()
setGranuleStatus()
- Removed