ETL-Template

Template Repository for creating new ETLs

Development

DFPC provided Lambda ETLs are currently all written in NodeJS through the use of a AWS Lambda optimized Docker container. Documentation for the Dockerfile can be found in the AWS Help Center

npm install

Add a .env file in the root directory that gives the ETL script the necessary variables to communicate with a local ETL server. When the ETL is deployed the ETL_API and ETL_LAYER variables will be provided by the Lambda Environment

{
    "ETL_API": "http://localhost:5001",
    "ETL_LAYER": "19"
}

To run the task, ensure the local CloudTAK server is running and then run with typescript runtime or build to JS and run natively with node

ts-node task.ts

npm run build
cp .env dist/
node dist/task.js

Deployment

Deployment into the CloudTAK environment for configuration is done via automatic releases to the DFPC AWS environment.

Github actions will build and push docker releases on every version tag which can then be automatically configured via the CloudTAK API.

Non-DFPC users will need to setup their own docker => ECS build system via something like Github Actions or AWS Codebuild.