This template is to create a standard monitoring pipeline. First, Imagine you’re part of a software development team working on a web application …

Some key points in this repo:

• GitHub Actions for automating CI/CD pipeline
• Terraform and Ansible for creating and building infrastructure (keypair, security group, ec2, nginx, Prometheus and Grafana, …)
• Monitor nginx status & statistics with Prometheus and Grafana

What is IaC ?

Imagine you’re part of a software development team working on a web application.

One day, a key infrastructure engineer in your company who was responsible for manually managing the deployment and configuration of various services decides to leave the company. This engineer had extensive knowledge about the intricate details of the infrastructure setup, making it difficult for others to take over their responsibilities. Without proper documentation and automation, maintaining the infrastructure becomes a challenge.

Other day, as the application gains popularity, the user load increases, and your team realizes that the current infrastructure setup is struggling to handle the traffic. Manual provisioning of servers and configurations has become a bottleneck, leading to inconsistent environments and frequent deployment issues. Scaling the infrastructure manually is time-consuming and error-prone.

In these practical examples, Infrastructure as Code (IaC) can be useful: A methodology that involves managing and provisioning infrastructure using code and automation instead of manual processes. IaC treats infrastructure configuration and management as software development, which brings consistency, repeatability, and version control to the process. It allows you to define and manage your infrastructure using code, making it easier to deploy, scale, and maintain.

Terraform & Ansible

Terraform is an open-source IaC tool developed by HashiCorp. It enables you to define and manage your infrastructure as declarative configuration files. Terraform uses a Domain-Specific Language (DSL) that allows you to specify the desired state of your infrastructure components (like virtual machines, networks, databases, etc.). Terraform then plans and executes the necessary changes to match the desired state, creating, updating, or destroying resources as needed.

Ansible is another popular open-source automation tool that falls under the IaC umbrella. It uses a different approach, focusing on agentless automation and configuration management. With Ansible, you define the tasks you want to perform on your infrastructure in YAML files known as playbooks. Ansible playbooks contain a series of tasks that define the desired configuration and state of systems. Ansible connects to target machines over SSH and executes these tasks to bring the systems to the desired state.

In short, combining Terraform and Ansible allows you to take advantage of their respective strengths: Terraform for infrastructure provisioning and Ansible for configuration management. This combination enhances the overall IaC process by providing a well-organized, scalable, and maintainable approach to managing your entire infrastructure stack.

Practice with AWS services

In this repo, Fluentd will continuously collect NGINX access logs and send them to Prometheus. Prometheus will scrape NGINX metrics endpoints to collect metrics data. Grafana will visualize the collected metrics and provide real-time dashboards for monitoring. You can access these dashboards or retrieve metrics data through the APIs for further analysis or integration with other systems.

How it works?

To run this, you first need to create an AWS account and then get the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. In order to activate GitHub Actions (if you want), you also need to add these above keys to the repo:

By the way, make sure you have installed Terraform, and Ansible, then just follow the below simple format:

cd infras/
terraform init 
terraform apply # also need to enter `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`

Now you can see the public IP address of EC2 from the terraform output. Use it to access Nginx(port 8080), Prometheus(port 9090), Grafana(port 3000). Below is just some simple metrics from nginx, you can customize some more useful metrics by importing specific JSON file to Grafana dashboard.

In this repo, I did not focus on building APIs, so after all, you should access the EC2 and create ones then use the nginx as the reverse proxy in your own system. Monitoring nginx in this situation is more interesting. Okay, see you in the next template of mine 😄