awstee

awstee is a tee command-like tool with AWS as the output destination.

The awstee command reads from standard input and writes to standard output and AWS S3 and CloudWatch Logs. awstee is the util tool for one time script for mission critical (especially for preventing rerunning it).

Usage

Basically, it can be used as follows

$ your_command |  awstee -s3-url-prefix s3://awstee-example-com/logs/  -log-group-name /awstee/logs hoge.log
2022/06/03 17:28:48 [info] s3 destination:  s3://awstee-example-com/logs//hoge.log
2022/06/03 17:28:49 [info] cloudwatch logs destination:  LogGroup=/awstee/test, LogStream=hoge
...

with default config ~/.config/awstee/default.yaml or ~/.config/awstee/default.yml.

aws_region: "ap-northeast-1"

s3:
  url_prefix: "s3://awstee-example-com/logs/" # Required if used. If blank, output setting is turned off
  allow_overwrite: true # Whether to allow overwriting if the object already exists

cloudwatch:
  log_group: "/awstee/logs" # Required if used. If blank, output setting is turned off
  flush_interval: "5s" # Duration of buffer flush output to cloudwatch logs
  buffer_lines: 50 # If more than this number of lines are output within the flush period, it is output once to Cloudwatch logs.
  create_log_group: true # Whether to create a LogGroup if it does not exist

$ your_command |  awstee hoge.log
2022/06/03 17:28:48 [info] s3 destination:  s3://awstee-example-com/logs//hoge.log
2022/06/03 17:28:49 [info] cloudwatch logs destination:  LogGroup=/awstee/test, LogStream=hoge
...

Install

Homebrew (macOS and Linux)

$ brew install mashiike/tap/awstee

Binary packages

Releases

Options

$ awstee -h    
awstee is a tee command-like tool with AWS as the output destination
version: v0.3.0 
  -aws-region string
        aws region
  -buffer-lines int
        cloudwatch logs output buffered lines (default 50)
  -config string
        config file path
  -create-log-group
        cloudwatch logs log group if not exists, create target log group
  -flush-interval string
        cloudwatch logs output flush interval duration (default "5s")
  -i    ignore interrupt signal
  -log-group-name string
        destination cloudwatch logs log group name
  -log-level string
        awstee log level (default "info")
  -s3-allow-overwrite
        allow overwriting if the s3 object already exists?
  -s3-firstly-put-empty-object
        put object from first for authority checks, etc.
  -s3-url-prefix string
        destination s3 url prefix
  -x    exit if an error occurs during initialization

IAM Role Policy

Permissions that awstee may have access to are as follows

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3Access",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:ListBucket"
            ],
            "Resource": "*"
        },
        {
            "Sid": "CloudwatchLogsAccess",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:DescribeLogStreams",
                "logs:CreateLogGroup",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

Note: logs:CreateLogGroup privilege is used only when the -create-log-group option is enabled.

LICENSE

MIT License

Copyright (c) 2022 IKEDA Masashi