v1.18.0

This is a fairly minor breaking release. See migration steps below for required changes.

Migration Steps

• If not already set, set ecs_cluster_instance_image_id in your terraform.tfvars for your cumulus module deployment

• Change variable saml_launchpad_metadata_path to saml_launchpad_metadata_url in the cumulus Terraform module.

• Remove process from PostToCmr task_config in workflow definitions.

• Please note function removals and deprecations listed below in the Removed and Deprecated sections.

Breaking Changes

• CUMULUS-1686

  • ecs_cluster_instance_image_id is now a required variable of the cumulus module, instead of optional.

• CUMULUS-1698

  • Change variable saml_launchpad_metadata_path to saml_launchpad_metadata_url in the tf-modules/cumulus Terraform module.

• CUMULUS-1703

  • Remove the unused forceDownload option from the sync-granule tasks's config
  • Remove the @cumulus/ingest/granule.Discover class
  • Remove the @cumulus/ingest/granule.Granule class
  • Remove the @cumulus/ingest/pdr.Discover class
  • Remove the @cumulus/ingest/pdr.Granule class
  • Remove the @cumulus/ingest/parse-pdr.parsePdr function

Other Notable Changes

• Upgrade Thin Egress App (TEA) to version 48
• Fixed granule discovery recursion algorithm used in S/FTP protocols.

All Changes

Added

• CUMULUS-1040

  • Added @cumulus/aws-client package to provide utilities for working with AWS services and the Node.js AWS SDK
  • Added @cumulus/errors package which exports error classes for use in Cumulus workflow code
  • Added @cumulus/integration-tests/sfnStep to provide utilities for parsing step function execution histories

• CUMULUS-1102

  • Adds functionality to the @cumulus/api package for better local testing.
    • Adds data seeding for @cumulus/api's localAPI.
      • seed functions allow adding collections, executions, granules, pdrs, providers, and rules to a Localstack Elasticsearch and DynamoDB via addCollections, addExecutions, addGranules, addPdrs, addProviders, and addRules.
    • Adds eraseDataStack function to local API server code allowing resetting of local datastack for testing (ES and DynamoDB).
    • Adds optional parameters to the @cumulus/api bin serve to allow for launching the api without destroying the current data.

• CUMULUS-1697

  • Added the @cumulus/tf-inventory package that provides command line utilities for managing Terraform resources in your AWS account

• CUMULUS-1703

  • Add @cumulus/aws-client/S3.createBucket function
  • Add @cumulus/aws-client/S3.putFile function
  • Add @cumulus/common/string.isNonEmptyString function
  • Add @cumulus/ingest/FtpProviderClient class
  • Add @cumulus/ingest/HttpProviderClient class
  • Add @cumulus/ingest/S3ProviderClient class
  • Add @cumulus/ingest/SftpProviderClient class
  • Add @cumulus/ingest/providerClientUtils.buildProviderClient function
  • Add @cumulus/ingest/providerClientUtils.fetchTextFile function

• CUMULUS-1731

  • Add new optional input variables to the Cumulus Terraform module to support TEA upgrade:
    • thin_egress_cookie_domain - Valid domain for Thin Egress App cookie
    • thin_egress_domain_cert_arn - Certificate Manager SSL Cert ARN for Thin
      Egress App if deployed outside NGAP/CloudFront
    • thin_egress_download_role_in_region_arn - ARN for reading of Thin Egress
      App data buckets for in-region requests
    • thin_egress_jwt_algo - Algorithm with which to encode the Thin Egress
      App JWT cookie
    • thin_egress_jwt_secret_name - Name of AWS secret where keys for the Thin
      Egress App JWT encode/decode are stored
    • thin_egress_lambda_code_dependency_archive_key - Thin Egress App - S3
      Key of packaged python modules for lambda dependency layer

• CUMULUS-1733

  • Add discovery-filtering operator doc to document previously undocumented functionality.

Changed

• CUMULUS-1102

  • Updates @cumulus/api/auth/testAuth to use JWT instead of random tokens.
  • Updates the default AMI for the ecs_cluster_instance_image_id.

• CUMULUS-1622

  • Mutex class has been deprecated in @cumulus/common/concurrency and will be removed in a future release.

• CUMULUS-1686

  • Changed ecs_cluster_instance_image_id to be a required variable of the cumulus module and removed the default value.
    The default was not available across accounts and regions, nor outside of NGAP and therefore not particularly useful.

• CUMULUS-1688

  • Updated @cumulus/aws.receiveSQSMessages not to replace message.Body with a parsed object. This behavior was undocumented and confusing as received messages appeared to contradict AWS docs that state message.Body is always a string.
  • Replaced sf_watcher CloudWatch rule from cloudwatch-events.tf with an EventSourceMapping on sqs2sf mapped to the start_sf SQS queue (in event-sources.tf).
  • Updated sqs2sf with an EventSourceMapping handler and unit test.

• CUMULUS-1698

  • Change variable saml_launchpad_metadata_path to saml_launchpad_metadata_url in the tf-modules/cumulus Terraform module.
  • Updated @cumulus/api/launchpadSaml to download launchpad IDP metadata from configured location when the metadata in s3 is not valid, and to work with updated IDP metadata and SAML response.

• CUMULUS-1731

  • Upgrade the version of the Thin Egress App deployed by Cumulus to v48
    • Note: New variables available, see the 'Added' section of this changelog.

Fixed

• CUMULUS-1664

  • Updated dbIndexer Lambda to remove hardcoded references to DynamoDB table names.

• CUMULUS-1733

  • Fixed granule discovery recursion algorithm used in S/FTP protocols.

Removed

• CUMULUS-1481

  • removed process config and output from PostToCmr as it was not required by the task nor downstream steps, and should still be in the output message's meta regardless.

• CUMULUS-1703

  • Remove the unused forceDownload option from the sync-granule tasks's config
  • Remove the @cumulus/ingest/granule.Discover class
  • Remove the @cumulus/ingest/granule.Granule class
  • Remove the @cumulus/ingest/pdr.Discover class
  • Remove the @cumulus/ingest/pdr.Granule class
  • Remove the @cumulus/ingest/parse-pdr.parsePdr function

Deprecated

• CUMULUS-1040
  • Deprecated the following code. For cases where the code was moved into another package, the new code location is noted:
    • @cumulus/common/CloudFormationGateway -> @cumulus/aws-client/CloudFormationGateway
    • @cumulus/common/DynamoDb -> @cumulus/aws-client/DynamoDb
    • @cumulus/common/errors -> @cumulus/errors
    • @cumulus/common/StepFunctions -> @cumulus/aws-client/StepFunctions
    • All of the exported functions in @cumulus/commmon/aws (moved into @cumulus/aws-client), except:
      • @cumulus/common/aws/isThrottlingException -> @cumulus/errors/isThrottlingException
      • @cumulus/common/aws/improveStackTrace (not deprecated)
      • @cumulus/common/aws/retryOnThrottlingException (not deprecated)
    • @cumulus/common/sfnStep/SfnStep.parseStepMessage -> @cumulus/integration-tests/sfnStep/SfnStep.parseStepMessage
    • @cumulus/common/sfnStep/ActivityStep -> @cumulus/integration-tests/sfnStep/ActivityStep
    • @cumulus/common/sfnStep/LambdaStep -> @cumulus/integration-tests/sfnStep/LambdaStep
    • @cumulus/common/string/unicodeEscape -> @cumulus/aws-client/StepFunctions.unicodeEscape
    • @cumulus/common/util/setErrorStack -> @cumulus/aws-client/util/setErrorStack
    • @cumulus/ingest/aws/invoke -> @cumulus/aws-client/Lambda/invoke
    • @cumulus/ingest/aws/CloudWatch.bucketSize
    • @cumulus/ingest/aws/CloudWatch.cw
    • @cumulus/ingest/aws/ECS.ecs
    • @cumulus/ingest/aws/ECS
    • @cumulus/ingest/aws/Events.putEvent -> @cumulus/aws-client/CloudwatchEvents.putEvent
    • @cumulus/ingest/aws/Events.deleteEvent -> @cumulus/aws-client/CloudwatchEvents.deleteEvent
    • @cumulus/ingest/aws/Events.deleteTarget -> @cumulus/aws-client/CloudwatchEvents.deleteTarget
    • @cumulus/ingest/aws/Events.putTarget -> @cumulus/aws-client/CloudwatchEvents.putTarget
    • @cumulus/ingest/aws/SQS.attributes -> @cumulus/aws-client/SQS.getQueueAttributes
    • @cumulus/ingest/aws/SQS.deleteMessage -> @cumulus/aws-client/SQS.deleteSQSMessage
    • @cumulus/ingest/aws/SQS.deleteQueue -> @cumulus/aws-client/SQS.deleteQueue
    • @cumulus/ingest/aws/SQS.getUrl -> @cumulus/aws-client/SQS.getQueueUrlByName
    • @cumulus/ingest/aws/SQS.receiveMessage -> @cumulus/aws-client/SQS.receiveSQSMessages
    • @cumulus/ingest/aws/SQS.sendMessage -> @cumulus/aws-client/SQS.sendSQSMessage
    • @cumulus/ingest/aws/StepFunction.getExecutionStatus -> @cumulus/aws-client/StepFunction.getExecutionStatus
    • @cumulus/ingest/aws/StepFunction.getExecutionUrl -> @cumulus/aws-client/StepFunction.getExecutionUrl

v1.17.0

This release updates all Cumulus internal lambdas/provided workflow lambdas to utilize a node 10 runtime in anticipation of node 8's end-of-life (https://docs.aws.amazon.com/lambda/latest/dg/runtime-support-policy.html).
All users on releases > 1.15.0 should update to this release prior to node 8's deprecation dates on AWS.

This release also includes an update (CUMULUS-1498) that updates Cumulus's CMR functions such that they utilize AWS Secrets Manager, this is considered a breaking change due to the changed dependency.

Migration Instructions

• If your permissions boundaries or IAMs do not support AWS Secrets Manager, they will need to be updated. For NASA NGAP users, no change should be required.
• This update requires use of the Cumulus Message Adapter >= 1.1.2 to allow Cumulus's internal lambdas to update to node 10, and should also be used for any custom workflow lambdas that need to use node 10 or later:
  • Follow the CMA deployment instructions to update your CMA layer, and ensure your terraform configuration has been updated to use the updated CMA layer
  • Ensure that custom node workflow lambdas are updated to utilize CMA >= 1.1.2 and cumulus-message-adapter-js >= 1.1.0

Changes

Added

• CUMULUS-630

  • Added support for replaying Kinesis records on a stream into the Cumulus Kinesis workflow triggering mechanism: either all the records, or some time slice delimited by start and end timestamps.
  • Added /replays endpoint to the operator API for triggering replays.
  • Added Replay Kinesis Messages documentation to Operator Docs.
  • Added manualConsumer lambda function to consume a Kinesis stream. Used by the replay AsyncOperation.

• CUMULUS-1687

  • Added new API endpoint for listing async operations at /asyncOperations
  • All asyncOperations now include the fields description and operationType. operationType can be one of the following. [Bulk Delete, Bulk Granules, ES Index, Kinesis Replay]

Changed

• CUMULUS-1626

  • Updates Cumulus to use node10/CMA 1.1.2 for all of its internal lambdas in prep for AWS node 8 EOL

• CUMULUS-1498

  • The @cumulus/cmrjs.publish2CMR function expects that the value of its
    creds.password parameter is a plaintext password.
  • Rather than using an encrypted password from the cmr_password environment
    variable, the @cumulus/cmrjs.updateCMRMetadata function now looks for an
    environment variable called cmr_password_secret_name and fetches the CMR
    password from that secret in AWS Secrets Manager.
  • The @cumulus/post-to-cmr task now expects a
    config.cmr.passwordSecretName value, rather than config.cmr.password.
    The CMR password will be fetched from that secret in AWS Secrets Manager.

v1.14.5

This release updates all Cumulus internal lambdas/provided workflow lambdas to utilize a node 10 runtime in anticipation of node 8's end-of-life (https://docs.aws.amazon.com/lambda/latest/dg/runtime-support-policy.html).

All users on releases < 1.14.5 who are not > 1.16 should update to this prior to node 8's deprecation dates on AWS.

Migration Instructions

• This update requires use of the Cumulus Message Adapter >= 1.1.2 to allow Cumulus's internal lambdas to update to node 10, and should also be used for any custom workflow lambdas that need to use node 10 or later:
  • Follow the CMA deployment instructions to update your CMA configuration to utilize version >= 1.1.2
  • Ensure that custom node workflow lambdas are updated to utilize CMA >= 1.1.2 and cumulus-message-adapter-js >= 1.1.0

Updated

• CUMULUS-1626
  • Updates Cumulus to use node10/CMA 1.1.2 for all of its internal lambdas in prep for AWS node 8 EOL

v1.16.1

Please Note

• The region argument to the cumulus Terraform module has been removed. You may see a warning or error if you have that variable populated.

• Your workflow tasks should use the following versions of the CMA libraries to utilize new granule, parentArn, asyncOperationId, and stackName fields on the logs:

  • cumulus-message-adapter-js version 1.0.10+
  • cumulus-message-adapter-python version 1.1.1+
  • cumulus-message-adapter-java version 1.2.11+

• The data-persistence module no longer manages the creation of an Elasticsearch service-linked role for deploying Elasticsearch to a VPC. Follow the deployment instructions on preparing your VPC for guidance on how to create the Elasticsearch service-linked role manually.

• There is now a distribution_api_gateway_stage variable for the tf-modules/cumulus Terraform module that will be used as the API gateway stage name used for the distribution API (Thin Egress App)

• Default value for the urs_url variable is now https://uat.urs.earthdata.nasa.gov/ in the tf-modules/cumulus and tf-modules/archive Terraform modules. So deploying the cumulus module without a urs_url variable set will integrate your Cumulus deployment with the UAT URS environment.

Added

• CUMULUS-1563

  • Added custom_domain_name variable to tf-modules/data-persistence module

• CUMULUS-1654

  • Added new helpers to @cumulus/common/execution-history:
    • getStepExitedEvent() returns the TaskStateExited event in a workflow execution history after the given step completion/failure event
    • getTaskExitedEventOutput() returns the output message for a TaskStateExited event in a workflow execution history

Changed

• CUMULUS-1578

  • Updates SAML launchpad configuration to authorize via configured userGroup.
    See the NASA specific documentation (protected)

• CUMULUS-1579

  • Elasticsearch list queries use match instead of term. term had been analyzing the terms and not supporting - in the field values.

• CUMULUS-1619

  • Adds 4 new keys to @cumulus/logger to display granules, parentArn, asyncOperationId, and stackName.
  • Depends on cumulus-message-adapter-js version 1.0.10+. Cumulus tasks updated to use this version.

• CUMULUS-1654

  • Changed @cumulus/common/SfnStep.parseStepMessage() to a static class method

• CUMULUS-1641

  • Added meta.retries and meta.visibilityTimeout properties to sqs-type rule. To create sqs-type rule, you're required to configure a dead-letter queue on your queue.
  • Added sqsMessageRemover lambda which removes the message from SQS queue upon successful workflow execution.
  • Updated sqsMessageConsumer lambda to not delete message from SQS queue, and to retry the SQS message for configured number of times.

Removed

• Removed create_service_linked_role variable from tf-modules/data-persistence module.

• CUMULUS-1321

  • The region argument to the cumulus Terraform module has been removed

Fixed

• CUMULUS-1668 - Fixed a race condition where executions may not have been
  added to the database correctly

• CUMULUS-1654 - Fixed issue with publishReports Lambda not including workflow execution error information for failed workflows with a single step

• Fixed tf-modules/cumulus module so that the urs_url variable is passed on to its invocation of the tf-modules/archive module

v1.16.0

Migration Instructions

As of this release, kes deployments are considered deprecated. Deployments will now be done using Terraform and all new feature deployments will be available in Terraform only.

It is recommended you replace your stack with a Terraform deployment. If you need to migrate an existing kes stack, please reach out to the Cumulus team.

The documentation, especially the deployment documentation has been revamped, so please check out how to deploy and the updated template deploy repo.

Note the artifacts attached to this release. They will be needed for your Terraform deployment.

Changelog Notes

Added

• CUMULUS-1580
  • Added /granules/bulk endpoint to @cumulus/api to perform bulk actions on granules given either a list of granule ids or an Elasticsearch query and the workflow to perform.

Changed

• CUMULUS-1561

  • Fix the way that we are handling Terraform provider version requirements
  • Pass provider configs into child modules using the method that the
    Terraform documentation
    suggests
  • Remove the region input variable from the s3_access_test Terraform module
  • Remove the aws_profile and aws_region input variables from the
    s3-replicator Terraform module

• CUMULUS-1639

  • Because of
    S3's Data Consistency Model,
    there may be situations where a GET operation for an object can temporarily
    return a NoSuchKey response even if that object has been created. The
    @cumulus/common/aws.getS3Object() function has been updated to support
    retries if a NoSuchKey response is returned by S3. This behavior can be
    enabled by passing a retryOptions object to that function. Supported
    values for that object can be found here:
    https://github.com/tim-kos/node-retry#retryoperationoptions

Removed

• CUMULUS-1559
  • logToSharedDestination has been migrated to the Terraform deployment as log_api_gateway_to_cloudwatch and will ONLY apply to egress lambdas.
    Due to the differences in the Terraform deployment model, we cannot support a global log subscription toggle for a configurable subset of lambdas.
    However, setting up your own log forwarding for a Lambda with Terraform is fairly simple, as you will only need to add SubscriptionFilters to your Terraform configuration, one per log group.
    See the Terraform documentation for details on how to do this.
    An empty FilterPattern ("") will capture all logs in a group.

v1.15.0

This release includes breaking deployment and workflow changes.

Migration Instructions

Deployment configuration updates

1. If you have pinned your message adapter version to < v.1.1.1, please update it to use 1.1.1 or above
2. Update your ECS Autoscaling configurations. Remove the following params from your app/config.yml:
   • ecs.services.<NAME>.minTasks
   • ecs.services.<NAME>.maxTasks
   • ecs.services.<NAME>.scaleInActivityScheduleTime
   • ecs.services.<NAME>.scaleInAdjustmentPercent
   • ecs.services.<NAME>.scaleOutActivityScheduleTime
   • ecs.services.<NAME>.scaleOutAdjustmentPercent
   • ecs.services.<NAME>.activityName
3. Define your autoscaling settings in a kes override file. See the notes for CUMULUS-1470 below and this example

Workflow updates

1. Remove StartStatus and StopStatus tasks from each workflow. These have been replaced by CloudWatch events.
2. Remove CumulusConfig for task configuration and add cma.task_config for task configuration.

NOTE: These instructions require the use of Cumulus Message Adapter v1.1.x+.
Please ensure you are using a compatible version before attempting to migrate
workflow configurations. When defining workflow steps, remove any
CumulusConfig section, as shown below:

ParsePdr:
  CumulusConfig:
    provider: '{$.meta.provider}'
    bucket: '{$.meta.buckets.internal.name}'
    stack: '{$.meta.stack}'

Instead, use AWS Parameters to pass task_config for the task directly into
the Cumulus Message Adapter:

ParsePdr:
  Parameters:
    cma:
      event.$: '$'
      task_config:
        provider: '{$.meta.provider}'
        bucket: '{$.meta.buckets.internal.name}'
        stack: '{$.meta.stack}'

In this example, the cma key is used to pass parameters to the message
adapter. Using task_config in combination with event.$: '$' allows the
message adapter to process task_config as the config passed to the Cumulus
task. See example/workflows/sips.yml in the core repository for further
examples of how to set the Parameters.

Additionally, workflow configurations for the QueueGranules and QueuePdrs
tasks need to be updated:

• queue-pdrs config changes:
  • parsePdrMessageTemplateUri replaced with parsePdrWorkflow, which is
    the workflow name (i.e. top-level name in config.yml, e.g. 'ParsePdr').
  • internalBucket and stackName configs now required to look up
    configuration from the deployment. Brings the task config in line with
    that of queue-granules.
• queue-granules config change: ingestGranuleMessageTemplateUri replaced
  with ingestGranuleWorkflow, which is the workflow name (e.g.
  'IngestGranule').

Breaking Changes

• CUMULUS-1449 - Cumulus now uses a universal workflow template when
  starting a workflow that contains general information specific to the
  deployment, but not specific to the workflow. Workflow task configs must be
  defined using AWS step function parameters. As part of this change,
  CumulusConfig has been retired and task configs must now be defined under
  the cma.task_config key in the Parameters section of a step function
  definition.

• CUMULUS-1396 - Workflow steps at the beginning and end of a workflow
  using the SfSnsReport Lambda have now been deprecated (e.g. StartStatus,
  StopStatus) and should be removed from your workflow definitions. These
  steps were used for publishing ingest notifications and have been replaced by
  an implementation using Cloudwatch events for Step Functions to trigger a
  Lambda that publishes ingest notifications. For further detail on how ingest
  notifications are published, see the notes below on CUMULUS-1394. For
  examples of how to update your workflow definitions, see our
  example workflow definitions.

• CUMULUS-1470

  • Remove Cumulus-defined ECS service autoscaling, allowing integrators to
    better customize autoscaling to meet their needs. In order to use
    autoscaling with ECS services, appropriate
    AWS::ApplicationAutoScaling::ScalableTarget,
    AWS::ApplicationAutoScaling::ScalingPolicy, and AWS::CloudWatch::Alarm
    resources should be defined in a kes overrides file. See
    this example
    for an example.
  • The following config parameters are no longer used:
    • ecs.services.<NAME>.minTasks
    • ecs.services.<NAME>.maxTasks
    • ecs.services.<NAME>.scaleInActivityScheduleTime
    • ecs.services.<NAME>.scaleInAdjustmentPercent
    • ecs.services.<NAME>.scaleOutActivityScheduleTime
    • ecs.services.<NAME>.scaleOutAdjustmentPercent
    • ecs.services.<NAME>.activityName

Additional Notable Changes

• The Cloudformation stack for database resources (e.g. prefix-db) now includes DeletionPolicy: Retain for the DynamoDB tables and the Elasticsearch instance, meaning that those resources will not be deleted when you delete the Cloudformation stack. If you want those resources deleted, you will have to delete them manually.
• SQS-type rule support

Added

• CUMULUS-1100

  • Added 30-day retention properties to all log groups that were missing those policies.

• CUMULUS-1396

  • Added @cumulus/common/sfnStep:
    • LambdaStep - A class for retrieving and parsing input and output to Lambda steps in AWS Step Functions
    • ActivityStep - A class for retrieving and parsing input and output to ECS activity steps in AWS Step Functions

• CUMULUS-1574

  • Added GET /token endpoint for SAML authorization when cumulus is protected by Launchpad.
    This lets a user retrieve a token by hand that can be presented to the API.

• CUMULUS-1625

  • Added sf_start_rate variable to the ingest Terraform module, equivalent to sqs_consumer_rate in the old model, but will not be automatically applied to custom queues as that was.

• CUMULUS-1513

  • Added sqs-type rule support in the Cumulus API @cumulus/api
  • Added sqsMessageConsumer lambda which processes messages from the SQS queues configured in the sqs rules.

Changed

• CUMULUS-1449

  • queue-pdrs & queue-granules config changes. Details in breaking changes section.
  • Cumulus now uses a universal workflow template when starting workflow that contains general information specific to the deployment, but not specific to the workflow.
  • Changed the way workflow configs are defined, from CumulusConfig to a task_config AWS Parameter.

• CUMULUS-1452

  • Changed the default ECS docker storage drive to devicemapper

• CUMULUS-1453

  • Removed config schema for @cumulus/sf-sns-report task
  • Updated @cumulus/sf-sns-report to always assume that it is running as an intermediate step in a workflow, not as the first or last step

Removed

• CUMULUS-1449
  • Retired CumulusConfig as part of step function definitions, as this is an artifact of the way Kes parses workflow definitions that was not possible to migrate to Terraform. Use AWS Parameters and the task_config key instead. See change note above.
  • Removed individual workflow templates.

Fixed

• CUMULUS-1620 - Fixed bug where message_adapter_version does not correctly inject the CMA

• CUMULUS-1396 - Updated @cumulus/common/StepFunctions.getExecutionHistory() to recursively fetch execution history when nextToken is returned in response

• CUMULUS-1571 - Updated @cumulus/common/DynamoDb.get() to throw any errors encountered when trying to get a record and the record does exist

• CUMULUS-1452

  • Updated the EC2 initialization scripts to use full volume size for docker storage
  • Changed the default ECS docker storage drive to devicemapper

v1.13.6

This release provides a bug fix to 1.13.5. Users of the v1.13.x baseline are advised to upgrade to this release instead of 1.13.5 if you are utilizing or plan to utilize the message_adapter_version configuration key.

Fixed

• CUMULUS-1620 - Fixes bug where message_adapter_version does not correctly inject the CMA.

v1.14.4

Fixed

• CUMULUS-1632 - Pinned aws-elasticsearch-connector package in @cumulus/api to version 8.1.3, since 8.2.0 includes breaking changes

v1.14.3

This release provides a bug fix to 1.14.2. Users are advised to upgrade to this release instead of 1.14.2 if you are utilizing or plan to utilize the message_adapter_version configuration key.

Fixed

• CUMULUS-1620 - Fixes bug where message_adapter_version does not correctly inject the CMA.

v1.14.2

Migration Instructions

Your Cumulus Message Adapter version should be pinned to v1.0.13 or lower in your app/config.yml using message_adapter_version: v1.0.13
OR
you should use the workflow migration steps below to work with CMA v1.1.1+.

The newest release of the Cumulus Message Adapter (v1.1.1) requires that parameterized configuration be used for remote message functionality. Once released, Kes will automatically bring in CMA v1.1.1 without additional configuration.

Migration instructions for CMA 1.1.1+
Oversized messages are no longer written to S3 automatically. In order to utilize remote messaging functionality, configure a ReplaceConfig AWS Step Function parameter on your CMA task:

ParsePdr:
  Parameters:
    cma:
      event.$: '$'
      ReplaceConfig:
        FullMessage: true

Accepted fields in ReplaceConfig include MaxSize, FullMessage, Path and TargetPath.
See https://github.com/nasa/cumulus-message-adapter/blob/master/CONTRACT.md#remote-message-configuration for full details.

As this change is backward compatible in Cumulus Core, users wishing to utilize the previous version of the CMA may opt to transition to using a CMA lambda layer, or set message_adapter_version in their configuration to a version prior to v1.1.0.

Workflow reporting - CUMULUS-1394

The implementation of the SfSnsReport Lambda requires additional environment variables for integration with the new ingest notification SNS topics. Therefore, you must update the definition of SfSnsReport in your lambdas.yml like so:

SfSnsReport:
  handler: index.handler
  timeout: 300
  source: node_modules/@cumulus/sf-sns-report/dist
  tables:
    - ExecutionsTable
  envs:
    execution_sns_topic_arn:
      function: Ref
      value: reportExecutionsSns
    granule_sns_topic_arn:
      function: Ref
      value: reportGranulesSns
    pdr_sns_topic_arn:
      function: Ref
      value: reportPdrsSns

Notable Changes

• NASA Launchpad is an option for authentication for CMR, the API, and dashboard. This must be set up and configured. NASA users can find Launchpad configuration documentation here for CMR and here for the API. Please note you will have to get the latest version of the Cumulus dashboard as well.

• Ingest notifications are now provided via 3 separate SNS topics for executions, granules, and PDRs, instead of a single sftracker SNS topic. Whereas the sftracker SNS topic received a full Cumulus execution message, the new topics all receive generated records for the given object. The new topics are only published to if the given object exists for the current execution. For a given execution/granule/PDR, two messages will be received by each topic: one message indicating that ingest is running and another message indicating that ingest has completed or failed. The new SNS topics are:

  • reportExecutions - Receives 1 message per execution
  • reportGranules - Receives 1 message per granule in an execution
  • reportPdrs - Receives 1 message per PDR

Added

• CUMULUS-1394

  • Added Granule.generateGranuleRecord() method to granules model to generate a granule database record from a Cumulus execution message
  • Added Pdr.generatePdrRecord() method to PDRs model to generate a granule database record from a Cumulus execution message
  • Added helpers to @cumulus/common/message:
    • getMessageExecutionName() - Get the execution name from a Cumulus execution message
    • getMessageStateMachineArn() - Get the state machine ARN from a Cumulus execution message
    • getMessageExecutionArn() - Get the execution ARN for a Cumulus execution message
    • getMessageGranules() - Get the granules from a Cumulus execution message, if any.
  • Added @cumulus/common/cloudwatch-event/isFailedSfStatus() to determine if a Step Function status from a Cloudwatch event is a failed status

• CUMULUS-639

  • Adds SAML JWT and launchpad token authentication to Cumulus API (configurable)
    • NOTE to authenticate with Launchpad ensure your launchpad user_id is in the <prefix>-UsersTable
    • when Cumulus configured to protect API via Launchpad:
      • New endpoints
        • GET /saml/login - starting point for SAML SSO creates the login request url and redirects to the SAML Identity Provider Service (IDP)
        • POST /saml/auth - SAML Assertion Consumer Service. POST receiver from SAML IDP. Validates response, logs the user in, and returnes a SAML-based JWT.
      • Disabled endpoints
        • GET /token
        • POST /refresh
          - Changes authorization worklow:
        • ensureAuthorized now presumes the bearer token is a JWT and tries to validate. If the token is malformed, it attempts to validate the token against Launchpad. This allows users to bring their own token as described here https://wiki.earthdata.nasa.gov/display/CUMULUS/Cumulus+API+with+Launchpad+Authentication. But it also allows dashboard users to manually authenticate via Launchpad SAML to receive a Launchpad-based JWT.

Changed

• CUMULUS-1485 Update @cumulus/cmr-client to return error message from CMR for validation failures.

• CUMULUS-1394

  • Renamed Execution.generateDocFromPayload() to Execution.generateRecord() on executions model. The method generates an execution database record from a Cumulus execution message.

• CUMULUS-1432

  • logs endpoint takes the level parameter as a string and not a number
  • Elasticsearch term query generation no longer converts numbers to boolean

• CUMULUS-1447

  • Consolidated all remote message handling code into @common/aws
  • Update remote message code to handle updated CMA remote message flags
  • Update example SIPS workflows to utilize Parameterized CMA configuration

• CUMULUS-1448 Refactor workflows that are mutating cumulus_meta to utilize meta field

• CUMULUS-1375

  • Migrate Cumulus from deprecated Elasticsearch JS client to new, supported one in @cumulus/api

• CUMULUS-1451

  • Elasticsearch cluster setting auto_create_index will be set to false. This had been causing issues in the bootstrap lambda on deploy.

• CUMULUS-1456

  • @cumulus/api endpoints default error handler uses boom package to format errors, which is consistent with other API endpoint errors.

Fixed

• CUMULUS-1432 logs endpoint filter correctly filters logs by level
• CUMULUS-1484 useMessageAdapter now does not set CUMULUS_MESSAGE_ADAPTER_DIR when true

Removed

• CUMULUS-1394
  • Removed sfTracker SNS topic. Replaced by three new SNS topics for granule, execution, and PDR ingest notifications.
  • Removed unused functions from @cumulus/common/aws:
    • getGranuleS3Params()
    • setGranuleStatus()