Observe Consul service mesh traffic

12min
|
Consul
Interactive

Workflow

In the previous tutorial, you learned how to configure and deploy your services using Consul service mesh solution. By using Consul service mesh, you enabled zero trust security in your network by having all service-to-service traffic encrypted and managed by Envoy sidecar proxies.

In this tutorial, you will configure and use Consul to observe traffic within your service mesh. This enables you to quickly understand how services interact with each other and effectively debug your services' traffic.

Using the Grafana suite, you can have your monitoring configuration embedded in your service VM and have the metrics collection configuration automatically scale when you deploy new services without the need to manually add new scraping endpoints for each new service you want to deploy.

In this tutorial, you will:

Create configuration files for Grafana agent tool
Start Grafana agent on all Consul nodes
Visualize metrics in Grafana
Visualize available metrics in Consul UI

Note

This tutorial is part of the Get Started collection, for this reason all the steps used to configure Consul agents and services are shown and require to be executed manually. If you are setting up a production environment you should codify and automate the installation and deployment process. Refer to the VM production patterns tutorial collection for Consul production deployment best practices.

Tutorial scenario

This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.

At the beginning of the tutorial, you have a fully deployed Consul service mesh with Envoy sidecar proxies running alongside each service.

Architectural diagram 03

By the end of this tutorial, you will have enabled metrics collection on your Consul nodes and will be able to visualize metrics both in Consul UI and in a Grafana dashboard.

Architectural diagram 05

Prerequisites

If you completed the previous tutorial, the infrastructure is already in place with all prerequisites needed.

For this tutorial, you will need:

An AWS account configured for use with Terraform
aws-cli >= 2.0
terraform >= 1.0
consul >= 1.15.0

Clone GitHub repository

Clone the GitHub repository containing the configuration files and resources.

$ git clone https://github.com/hashicorp-education/learn-consul-get-started-vms

Change into the directory that contains the complete configuration files for this tutorial.

$ cd learn-consul-get-started-vms/self-managed/infrastructure/aws

Create infrastructure

With these Terraform configuration files, you are ready to deploy your infrastructure.

Issue the terraform init command from your working directory to download the necessary providers and initialize the backend.

$ terraform initInitializing the backend...Initializing provider plugins......Terraform has been successfully initialized!...

Then, deploy the resources. Confirm the run by entering yes.

$ terraform apply -var-file=./conf/03_service_mesh.tfvars## ...Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes## ...Apply complete! Resources: 49 added, 0 changed, 0 destroyed.

Tip

The Terraform deployment could take up to 15 minutes to complete. Feel free to explore the next sections of this tutorial while waiting for the environment to complete initialization or learn more about the Raft protocol in a fun, interactive way.

Once the deployment is complete, Terraform will return a list of outputs you can use to interact with the newly created environment.

Outputs:connection_string = "ssh -i certs/id_rsa.pem admin@`terraform output -raw ip_bastion`"ip_bastion = "<redacted-output>"remote_ops = "export BASTION_HOST=<redacted-output>"retry_join = "provider=aws tag_key=ConsulJoinTag tag_value=auto-join-hcoc"ui_consul = "https://<redacted-output>:8443"ui_grafana = "http://<redacted-output>:3000/d/hashicups/hashicups"ui_hashicups = "http://<redacted-output>"

The Terraform outputs provide useful information, including the bastion host IP address. The following is a brief description of the Terraform outputs:

The ip_bastion provides IP address of the bastion host you will use to run the rest of the commands in this tutorial.
The remote_ops lists the bastion host IP, which you can use access the bastion host.
The retry_join output lists Consul's retry_join configuration parameter. The next tutorial will use this to generate Consul server and client configuration.
The ui_consul output lists the Consul UI address. The Consul UI is not currently running. You will use this in a later tutorial to verify Consul started correctly.
The ui_grafana output lists the Grafana UI address. You will use this in a future tutorial.
The ui_hashicups output lists the HashiCups UI address. You can use it to verify the HashiCups demo application is running properly.

Terraform output provides a series of useful information, including bastion host IP address.

$ ssh -i certs/id_rsa.pem admin@`terraform output -raw ip_bastion`

Verify Grafana agent binary

Check on each of the Consul nodes (Consul server, NGINX, Frontend, API, and Database) to verify Grafana agent is installed.

$ grafana-agent --versionagent, version v0.33.2 (branch: HEAD, revision: 93a15c09)  build user:       root@808a765754ec  build date:       2023-05-11T20:58:39Z  go version:       go1.20.3  platform:         linux/amd64  tags:             netgo,builtinassets,promtail_journal_enabled

Configure Grafana Agent

You can configure Grafana Agent to collect several kinds of data from your VM. In this tutorial, you will use configurations for:

metrics block, to define a collection of Prometheus-compatible scrape configs to be written in Mimir.
logs block, to configure how the Agent collects logs and sends them to a Loki push API endpoint.

Generate configuration for Grafana Agent

This tutorial and interactive lab environment uses scripts in the tutorial's GitHub repository to generate the Consul configuration files for your client agents.

The Bastion Host includes the script in the ops/scenarios/99_supporting_scripts folder.

$ tree ./ops/scenarios/99_supporting_scripts/./ops/scenarios/99_supporting_scripts/|-- generate_consul_client_config.sh|-- generate_consul_monitoring_config.sh|-- generate_consul_server_config.sh|-- generate_consul_server_tokens.sh|-- generate_consul_service_config.sh`-- generate_consul_service_intentions.sh 1 directory, 6 files

The script requires a few parameters to work correctly:

an OUTPUT_FOLDER to place the files generated
a PROMETHEUS_URI to push metrics to. In this scenario we configured Grafana Mimir for this task listening on the bastion host.
a LOKI_URI to push logs to. In this scenario we configured Grafana Loki for this task listening on the bastion host.

$ export OUTPUT_FOLDER="./assets/scenario/conf/"; \  export PROMETHEUS_URI=`getent hosts mimir | awk '{print $1}'`; \  export LOKI_URI=`getent hosts loki | awk '{print $1}'`

With these values configured, generate the configuration.

$ bash ops/scenarios/99_supporting_scripts/generate_consul_monitoring_config.sh -- Parameter Check -- Generating Grafana Agent configuration

The script creates the Grafana Agent configuration for all agents.

$ tree ${OUTPUT_FOLDER}monitoring./assets/scenario/conf/monitoring├── grafana-agent-consul-server-0.yaml├── grafana-agent-gateway-api.yaml├── grafana-agent-hashicups-api.yaml├── grafana-agent-hashicups-db.yaml├── grafana-agent-hashicups-frontend.yaml└── grafana-agent-hashicups-nginx.yaml 0 directories, 6 files

Copy configuration on client VMs

After the script generates the configuration files, you will copy these files in each client node.

$ scp -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-consul-server-0.yaml consul-server-0:grafana-agent.yaml; \  scp -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-db.yaml hashicups-db:grafana-agent.yaml; \  scp -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-api.yaml hashicups-api:grafana-agent.yaml; \  scp -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-frontend.yaml hashicups-frontend:grafana-agent.yaml; \  scp -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-nginx.yaml hashicups-nginx:grafana-agent.yaml

Start Grafana Agent on VMs

Once you copied the configuration files on the different VMs, login on each Consul client VMs and start the Grafana Agent.

Note

For production environments, consider using systemd to start the Grafana Agent as a daemon service.