I’ve completed a new module on the public Terraform Registry – a full Terraform conversion of the Centralized Logging Solution from AWS! This solution deploys the following:
- VPC-hosted Opensearch/Elasticsearch Cluster
- Kinesis Data with Cloudwatch Destination for cross-account log ingestion
- Lambda function to transform log data into ES/OS-readable format
- Kinesis Firehose which deposits data into ES/OS cluster
- S3 bucket to hold flow logs and life-cycled indices
- Option to enable ultrawarm storage nodes
- Cognito user/identity pool for controlling access to dashboard
- Bastion host for accessing dashboard from outside the VPC
- Option to create TLS key and store in Secrets Mgr
The infrastructure matches what we see from the Central Logging Solution page, but done in Terraform rather than Cloudformation:
NOTE: I have not included the demo resources option from the Cloudformation deployment. That may come in a later revision.
The template as-designed needs to be run in an environment that can utilize local-exec provisioners. This can be done when using a local execution environment, of course – but it could also be done using something like Github Actions, where the command could be executed on a runner node. The template can NOT be fully executed on the Terraform Cloud platform as local provisioners will not run in that environment – although TF Cloud could be used for remote state storage while the execution environment is run somewhere else.
Building this template was a major challenge – while I could simply transcribe the basic structure from Cloudformation, there are subtleties in the differences between Terraform and Cloudformation which required a lot of thinking to work around. This, of course, made it all the more satisfying to finish.
In the future, I’d like to implement the following improvements:
- Include demo assets option.
- Include optional best practice Cloudwatch alarms for Opensearch.
- Build out more options for Opensearch clusters – fine-grained access control, custom endpoint, etc.
Questions? Comments? Biting criticism? Don’t hesitate to let me know in the comments!