Automating Monitoring in Kubernetes using GitOps, Prometheus, and Grafana
Part -11
In traditional DevOps workflows, tools like Jenkins are commonly used for both Continuous Integration (CI) and Continuous Deployment (CD). Jenkins plays a key role in automating tasks such as building applications, running tests, and creating deployment artifacts. However, in many modern DevOps environments, Jenkins is primarily used for CI, while deployment responsibilities are handled by GitOps-based tools like Argo CD.
What is GitOps?
GitOps is an operational framework that uses Git as the single source of truth for managing infrastructure and application deployments. In this approach, all configuration files such as Kubernetes manifests, Helm charts, and infrastructure definitions are stored in a Git repository.
Whenever changes are required, developers update the configuration files in Git through commits or pull requests. Once the changes are approved and merged, the GitOps tool automatically applies those changes to the target environment. This process provides better version control, auditability, and the ability to easily roll back to previous states if something goes wrong.
What is Argo CD?
Argo CD is a popular GitOps continuous delivery tool designed specifically for Kubernetes environments. It continuously monitors the Git repository and compares the desired state defined in Git with the actual state of the Kubernetes cluster.
If there is any difference between the two states, Argo CD synchronizes the cluster to match the configuration stored in Git. One of the key advantages of Argo CD is its user-friendly graphical interface, which allows teams to visualize applications, track deployment status, and manage synchronization easily.
Sync Policies in Argo CD
Argo CD provides two types of synchronization policies:
Manual Sync – Changes from the Git repository are applied only when a user manually triggers the synchronization.
Automatic Sync – Argo CD automatically applies changes to the cluster whenever it detects updates in the Git repository.
Best Practice in GitOps
In a proper GitOps workflow, changes should always be made through the Git repository rather than directly from the Argo CD interface. Making direct changes in Argo CD without updating Git can lead to configuration drift, where the actual cluster state differs from the desired state defined in Git.
By following the GitOps approach with tools like Argo CD, teams can achieve more reliable, traceable, and automated deployments in Kubernetes environments.
Some Core Features of Argo CD
Argo CD provides several powerful features that help DevOps teams manage Kubernetes deployments efficiently using the GitOps approach.
Application
In Argo CD, an application represents a deployed service or workload running in the Kubernetes cluster. It defines the source Git repository, the target cluster, and the Kubernetes resources that should be deployed.
Repository
Argo CD connects to Git repositories that contain Kubernetes manifests, Helm charts, or Kustomize configurations. The repository acts as the source of truth, where all infrastructure and deployment configurations are stored.
Sync
Synchronization is the process where Argo CD compares the desired state from the Git repository with the actual state of the Kubernetes cluster. If differences are detected, Argo CD applies the necessary changes to make the cluster match the configuration defined in Git.
Auto Sync
Auto Sync allows Argo CD to automatically apply changes whenever a new commit is detected in the Git repository. This enables fully automated continuous deployment without manual intervention.
Self-Healing
Self-healing is a feature that automatically restores resources if someone manually changes them in the Kubernetes cluster. Argo CD detects the drift and re-applies the configuration from Git to bring the system back to the desired state.
Pruning
Pruning removes Kubernetes resources that are no longer defined in the Git repository. This ensures that unused or outdated resources do not remain in the cluster.
Rollback
Argo CD allows teams to roll back to a previous application version using Git history. If a deployment introduces issues, the system can quickly revert to a stable configuration.
Application Health Status
Argo CD continuously monitors the health of applications deployed in Kubernetes. It provides visual indicators in the dashboard showing whether an application is healthy, degraded, or out of sync, helping teams quickly identify issues.
let's setup ArgoCd
Create a Namespace for ArgoCd
kubectl create namespace argocd
Install ArgoCD
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
Key components created:
| Component | Purpose |
|---|---|
| argocd-server | Web UI & API |
| argocd-repo-server | Pulls Git repositories |
| argocd-application-controller | Syncs cluster with Git |
| argocd-dex-server | Authentication |
Expose ArgoCd server
kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'
Get intial admin password
argocd admin initial-password -n argocd
Default username:
admin
Check Secrets
kubectl get secret -n argocd
You will see a secret like:
argocd-initial-admin-secret
You can also retrieve password manually:
kubectl get secret argocd-initial-admin-secret \
-n argocd \
-o jsonpath="{.data.password}" | base64 -d
Install ArgoCd CLI
curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd
Along with the graphical interface, Argo CD also provides a powerful Command Line Interface (CLI) that allows DevOps engineers to interact with applications, manage deployments, and automate operations directly from the terminal.
Login to Argo CD
Before running any commands, we need to authenticate with the Argo CD server.
argocd login <ARGOCD_SERVER>
Example:
argocd login 34.120.45.67
You will be prompted to enter:
Username: admin
Password: Initial admin password
List Applications
To view all applications managed by Argo CD:
argocd app list
This command shows important details such as:
Application name
Sync status
Health status
Target namespace
Create an Application
Applications can also be created directly from the CLI.
argocd app create my-app \
--repo https://github.com/example/repo \
--path manifests \
--dest-server https://kubernetes.default.svc \
--dest-namespace default
This command tells Argo CD:
Which Git repository to use
Which folder contains Kubernetes manifests
Which cluster and namespace to deploy to
Synchronize an Application
If manual sync is enabled, we need to trigger synchronization manually.
argocd app sync my-app
View Application Details
To see detailed information about an application:
argocd app get my-app
This shows:
Deployment status
Resource details
Sync status
Health status
Rollback an Application
If a deployment introduces issues, we can roll back to a previous version.
argocd app rollback my-app <revision-id>
This restores the application to a previously deployed state.
Delete an Application
To remove an application from Argo CD:
argocd app delete my-app
Polling vs Webhook in Argo CD
One of the key responsibilities of Argo CD is detecting changes in the Git repository and applying them to the Kubernetes cluster. Argo CD can detect these changes using two methods: Polling and Webhooks.
Polling (Default Method)
Polling is the default mechanism used by Argo CD to detect changes in the Git repository.
In this approach, Argo CD periodically checks the Git repository for updates.
How Polling Works
Argo CD regularly checks the Git repository.
It compares the latest commit with the currently deployed version.
If a new commit is detected, Argo CD identifies a difference.
Depending on the sync policy, it either:
Automatically applies the changes, or
Waits for manual synchronization.
Polling Interval
By default, Argo CD checks the repository approximately every 3 minutes.
Advantages
Simple to configure
Works without additional setup
Reliable in most environments
Disadvantages
Changes are not applied instantly
There may be a delay before Argo CD detects new commits
Webhooks (Event-Driven Method)
Webhooks provide a faster and more efficient way to notify Argo CD about changes in the Git repository.
Instead of continuously checking the repository, the Git provider sends a notification to Argo CD whenever a new commit is pushed.
How Webhooks Work
A developer pushes changes to the Git repository.
The Git platform (such as GitHub or GitLab) sends a webhook notification to Argo CD.
Argo CD immediately checks the repository.
The changes are synchronized with the Kubernetes cluster.
Advantages
Faster deployment
Immediate change detection
Reduces unnecessary Git polling
Disadvantages
Requires additional configuration
Needs network access from the Git provider to the Argo CD server
Monitoring in Kubernetes using Prometheus and Grafana
Monitoring is a critical part of managing applications in Kubernetes. It helps DevOps teams understand the health, performance, and resource usage of applications and infrastructure running inside the cluster.
Two of the most widely used open-source tools for monitoring Kubernetes environments are Prometheus and Grafana.
These tools work together to provide metrics collection, storage, and visualization, allowing teams to monitor cluster performance and troubleshoot issues efficiently.
Prometheus
Prometheus is an open-source monitoring and alerting system designed for cloud-native environments like Kubernetes.
In a Kubernetes cluster, Prometheus collects metrics from different components such as:
Nodes
Pods
Containers
Applications
Cluster services
Prometheus works using a pull-based model, where it periodically scrapes metrics from endpoints exposed by applications and Kubernetes components.
Grafana
Grafana is an open-source visualization and analytics platform used to create dashboards for monitoring metrics.
While Prometheus is responsible for collecting and storing metrics, Grafana is used to visualize those metrics in dashboards.
Using Grafana dashboards, DevOps teams can easily monitor important metrics such as:
CPU usage
Memory usage
Pod health
Network traffic
Application performance
Grafana provides interactive charts and customizable dashboards that make it easier to understand complex monitoring data.
Prometheus and Grafana Integration
Prometheus and Grafana work together to provide a complete monitoring solution.
The monitoring workflow typically works like this:
Applications and Kubernetes components expose metrics.
Prometheus scrapes and stores these metrics.
Grafana connects to Prometheus as a data source.
Grafana dashboards visualize the collected metrics.
This integration allows teams to monitor cluster performance in real time.
Steps to Setup Prometheus and Grafana in Kops Cluster
The following steps demonstrate how to install Prometheus and Grafana in a Kubernetes cluster created using Kops.
Install Helm
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
Install Kubernetes Metrics Server
The Metrics Server collects resource metrics such as CPU and memory usage for nodes and pods.
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
Verify the deployment:
kubectl get pods -n kube-system
kubectl get deployment metrics-server -n kube-system
Add Prometheus Helm Repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
Update Helm repositories:
helm repo update
helm repo list
Create Prometheus Namespace
kubectl create namespace prometheus
kubectl get ns
Install Prometheus
Install Prometheus using Helm:
helm install prometheus prometheus-community/prometheus \
--namespace prometheus \
--set alertmanager.persistentVolume.storageClass="gp2" \
--set server.persistentVolume.storageClass="gp2"
Verify the installation:
kubectl get pods -n prometheus
kubectl get all -n prometheus
Create Grafana Namespace
kubectl create namespace grafana
Install Grafana
Add the Grafana Helm repository:
helm repo add grafana https://grafana.github.io/helm-charts
Install Grafana using Helm:
helm install grafana grafana/grafana \
--namespace grafana \
--set persistence.storageClassName="gp2" \
--set persistence.enabled=true \
--set adminPassword='EKS!sAWSome' \
--set service.type=LoadBalancer
Verify the deployment:
kubectl get pods -n grafana
kubectl get service -n grafana
Copy the EXTERNAL-IP of the Grafana service and open it in a browser.
Configure Prometheus as Grafana Data Source
After accessing the Grafana dashboard:
Go to Settings → Data Sources
Click Add Data Source
Select Prometheus
Enter the following URL:
http://prometheus-server.prometheus.svc.cluster.local
Click Save & Test to verify the connection.
Import Grafana Dashboards
Grafana allows importing pre-built dashboards from Grafana Labs.
To import a dashboard:
Go to Grafana Dashboard
Click New → Import
Enter Dashboard ID
Example dashboards:
Dashboard ID 315
Used for monitoring cluster-level metrics:
Network I/O pressure
Cluster CPU usage
Cluster memory usage
Cluster filesystem usage
Pod CPU usage
Dashboard ID 1860
Used for monitoring individual Kubernetes nodes.
Conclusion
Adopting a GitOps-driven approach for deploying and managing monitoring in Kubernetes — using Argo CD alongside Prometheus and Grafana — turns observability into a repeatable, auditable, and version-controlled process. By treating monitoring configuration (Prometheus rules, exporters, ServiceMonitors, Grafana dashboards and alerting provisioning) as code in Git, teams gain faster, safer rollouts, clear change history, and automatic reconciliation of drift. Combined with CI for building artifacts and sound practices for secrets, RBAC, and testing, this stack reduces operational toil, improves mean time to detection and recovery, and scales with your cluster landscape. Start small (monitor a single service or namespace), codify your alerts and dashboards, automate syncs with Argo CD, and iterate—your monitoring will become more reliable, transparent, and easier to maintain as your platform grows.
If you found this guide useful, please like ❤️, comment, and share your thoughts or questions. Stay tuned for the next part, where we'll explore advanced Docker concepts and practices.
Thank you,
Yours, Rama Grandhi