Structured log output for all functions

[timoguin]

All functions should output fully structured logs.

Considerations

Every individual event that triggers a Lambda should result in at least one line of structured output. This will improve observability in a number of ways. By default they're going to go to CloudWatch Logs, which allows us to create alarms based on metrics calculated from the logs. Then they could also be easily streamed from there into Kinesis Firehose, and then delivered in parquet right back into Matano.

A good structured logging library will allow us to create a log context that we can flow through the application logic, attaching relevant log data along the way, and then flushing one line at the end. High cardinality log output is key.

Tasks

• Research structured log libraries
  • Rust
  • Should we provide something for Python code? For example, to assist debugging if any of the detections are failing.
  • Java / Kotlin?
  • NodeJS?
• Design work
• Implementation for each language

References

Will add later.