I’ve completed a new module on the public Terraform Registry – a full Terraform conversion of the Centralized Logging Solution from AWS! This solution deploys the following:

• VPC-hosted Opensearch/Elasticsearch Cluster
• Kinesis Data with Cloudwatch Destination for cross-account log ingestion
• Lambda function to transform log data into ES/OS-readable format
• Kinesis Firehose which deposits data into ES/OS cluster
• S3 bucket to hold flow logs and life-cycled indices
  • Option to enable ultrawarm storage nodes
• Cognito user/identity pool for controlling access to dashboard
• Bastion host for accessing dashboard from outside the VPC
  • Option to create TLS key and store in Secrets Mgr

The infrastructure matches what we see from the Central Logging Solution page, but done in Terraform rather than Cloudformation:

NOTE: I have not included the demo resources option from the Cloudformation deployment. That may come in a later revision.

The template as-designed needs to be run in an environment that can utilize local-exec provisioners. This can be done when using a local execution environment, of course – but it could also be done using something like Github Actions, where the command could be executed on a runner node. The template can NOT be fully executed on the Terraform Cloud platform as local provisioners will not run in that environment – although TF Cloud could be used for remote state storage while the execution environment is run somewhere else.

Building this template was a major challenge – while I could simply transcribe the basic structure from Cloudformation, there are subtleties in the differences between Terraform and Cloudformation which required a lot of thinking to work around. This, of course, made it all the more satisfying to finish.

In the future, I’d like to implement the following improvements:

1. Include demo assets option.
2. Include optional best practice Cloudwatch alarms for Opensearch.
3. Build out more options for Opensearch clusters – fine-grained access control, custom endpoint, etc.

Questions? Comments? Biting criticism? Don’t hesitate to let me know in the comments!

Categories: Cloud

Tags: AWS, Cloudformation, Logging, Opensearch, Terraform