Terraform Module: AWS Centralized Logging

I’ve completed a new module on the public Terraform Registry – a full Terraform conversion of the Centralized Logging Solution from AWS! This solution deploys the following:

  • VPC-hosted Opensearch/Elasticsearch Cluster
  • Kinesis Data with Cloudwatch Destination for cross-account log ingestion
  • Lambda function to transform log data into ES/OS-readable format
  • Kinesis Firehose which deposits data into ES/OS cluster
  • S3 bucket to hold flow logs and life-cycled indices
    • Option to enable ultrawarm storage nodes
  • Cognito user/identity pool for controlling access to dashboard
  • Bastion host for accessing dashboard from outside the VPC
    • Option to create TLS key and store in Secrets Mgr

The infrastructure matches what we see from the Central Logging Solution page, but done in Terraform rather than Cloudformation:

NOTE: I have not included the demo resources option from the Cloudformation deployment. That may come in a later revision.

The template as-designed needs to be run in an environment that can utilize local-exec provisioners. This can be done when using a local execution environment, of course – but it could also be done using something like Github Actions, where the command could be executed on a runner node. The template can NOT be fully executed on the Terraform Cloud platform as local provisioners will not run in that environment – although TF Cloud could be used for remote state storage while the execution environment is run somewhere else.

Building this template was a major challenge – while I could simply transcribe the basic structure from Cloudformation, there are subtleties in the differences between Terraform and Cloudformation which required a lot of thinking to work around. This, of course, made it all the more satisfying to finish.

In the future, I’d like to implement the following improvements:

  1. Include demo assets option.
  2. Include optional best practice Cloudwatch alarms for Opensearch.
  3. Build out more options for Opensearch clusters – fine-grained access control, custom endpoint, etc.

Questions? Comments? Biting criticism? Don’t hesitate to let me know in the comments!

Categories: Cloud

Tags: , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: