Monitor Go Applications with Prometheus, Grafana, and Docker: A Complete Guide

Application monitoring is crucial for ensuring smooth operation and early issue detection. Instead of manually testing each service, monitoring provides a proactive approach to identify and resolve problems before they escalate. This guide will walk you through setting up a monitoring system for a Go application using Prometheus, Grafana, and Docker.

In this article, you will learn:

• How to expose metrics from a Go application using the go-prometheus library.
• How to visualize those metrics using Grafana.
• How to use Docker and Docker Compose to run and connect your application with the monitoring stack.

Prerequisites

Before you begin, ensure you have a foundational understanding of:

• Golang programming.
• Docker and Docker Compose.
• Prometheus and Grafana.

Create a Go Application for Monitoring

Let's start by building a simple Go server. Create a new directory for your project and initialize it using go mod init <project-name>. Then, create a main.go file with the following code:

package main

import (
	"strconv"

	"github.com/gin-gonic/gin"
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

// Define metrics
var (
	HttpRequestTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "api_http_request_total",
		Help: "Total number of requests processed by the API",
	}, []string{"path", "status"})
	HttpRequestErrorTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "api_http_request_error_total",
		Help: "Total number of errors returned by the API",
	}, []string{"path", "status"})
)

// Custom registry (without default Go metrics)
var customRegistry = prometheus.NewRegistry()

// Register metrics with custom registry
func init() {
	customRegistry.MustRegister(HttpRequestTotal, HttpRequestErrorTotal)
}

func main() {
	router := gin.Default()

	// Register /metrics before middleware
	router.GET("/metrics", PrometheusHandler())
	router.Use(RequestMetricsMiddleware())

	router.GET("/health", func(c *gin.Context) {
		c.JSON(200, gin.H{
			"message": "Up and running!",
		})
	})

	router.GET("/v1/users", func(c *gin.Context) {
		c.JSON(200, gin.H{
			"message": "Hello from /v1/users",
		})
	})

	router.Run(":8000")
}

// Custom metrics handler with custom registry
func PrometheusHandler() gin.HandlerFunc {
	h := promhttp.HandlerFor(customRegistry, promhttp.HandlerOpts{})
	return func(c *gin.Context) {
		h.ServeHTTP(c.Writer, c.Request)
	}
}

// Middleware to record incoming requests metrics
func RequestMetricsMiddleware() gin.HandlerFunc {
	return func(c *gin.Context) {
		path := c.Request.URL.Path
		c.Next()
		status := c.Writer.Status()

		if status < 400 {
			HttpRequestTotal.WithLabelValues(path, strconv.Itoa(status)).Inc()
		} else {
			HttpRequestErrorTotal.WithLabelValues(path, strconv.Itoa(status)).Inc()
		}
	}
}

After adding the code, execute go mod tidy in your terminal to install the necessary dependencies.

Code Breakdown: Defining and Registering Metrics

This code defines and registers the metrics we want to monitor.

• The code imports necessary packages, including gin-gonic for creating the server and prometheus/client_golang for exposing metrics.
• HttpRequestTotal: Counts the total number of requests processed by the API.
• HttpRequestErrorTotal: Counts the total number of errors returned by the API.
• Both metrics use CounterVec, a metric type that tracks event occurrences, along with labels for path and status.

A custom registry (customRegistry) is created to avoid registering default Golang metrics, ensuring you only monitor what's relevant to your application's performance.

Setting Up the API Routes and Metrics Endpoint

This section focuses on creating the API endpoints and exposing the metrics.

• A new Gin router is created, and the /metrics endpoint is registered before the middleware to ensure metrics are collected before the middleware is executed.
• Two simple endpoints, /health and /v1/users, are created to simulate API functionality.
• The server starts on port 8000.

Custom Metrics Handler and Middleware

This part implements the mechanisms for serving metrics and recording request data.

• PrometheusHandler is a custom handler that serves the metrics using the custom registry.
• RequestMetricsMiddleware is a middleware that records metrics for incoming requests, capturing the path and status code, and incrementing the corresponding metrics.

Run the Go Application locally

To verify that the metrics are registered correctly, run the application using go run main.go.

• Test the Endpoints: Open your browser and navigate to http://localhost:8000/health and http://localhost:8000/v1/users. You should see JSON responses confirming the server is running.
• Check the Metrics: Visit http://localhost:8000/metrics. You should see the metrics defined in the code, including api_http_request_total and api_http_request_error_total, along with their labels.

A successful display of metrics confirms that your application is working correctly and the metrics are properly registered.

Dockerize the Go Application

To containerize the application, create a file named Dockerfile in the root directory. Add the following instructions:

FROM golang:1.24-alpine AS builder

# Set environment variables
ENV CGO_ENABLED=0 \
    GOOS=linux \
    GOARCH=amd64

# Set working directory inside the container
WORKDIR /build

# Copy go.mod and go.sum files for dependency installation
COPY go.mod go.sum ./

# Download dependencies
RUN go mod download

# Copy the entire application source
COPY . .

# Build the Go binary
RUN go build -o /app .

# Final lightweight stage
FROM alpine:3.17 AS final

# Copy the compiled binary from the builder stage
COPY --from=builder /app /bin/app

# Expose the application's port
EXPOSE 8000

# Run the application
CMD ["/bin/app"]

Understanding the Dockerfile stages

The Dockerfile uses a multi-stage build to create a streamlined Docker image. This approach separates the build environment from the runtime environment, resulting in a smaller final image size.

• Build Stage: Uses the official Golang Alpine image to compile the Go application.
  • Sets environment variables to build a static binary for Linux.
  • Downloads dependencies and builds the Go binary.
• Final Stage: Uses the official Alpine image as the base.
  • Copies the compiled binary from the build stage.
  • Exposes port 8000 and defines the command to run the application.

Build and Run the Docker Image

Build the Docker image using the following command:

docker build -t go-prom-monitor .

Run the Docker container:

docker run -d -p 8000:8000 --name go-prom-monitor go-prom-monitor

Just like before, you can confirm the application is running correctly by checking the /health, /v1/users, and /metrics endpoints in your browser.

Connect with Prometheus and Grafana

Docker Compose allows for easy management and connection of multiple Docker containers. Create a file named compose.yml in the root directory of the project and add the following:

services:
  api:
    container_name: go-api
    build:
      context: .
      dockerfile: Dockerfile
    image: go-api:latest
    ports:
      - 8000:8000
    networks:
      - go-network
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 5
    develop:
      watch:
        - path: .
          action: rebuild
  prometheus:
    container_name: prometheus
    image: prom/prometheus:v2.55.0
    volumes:
      - ./Docker/prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - 9090:9090
    networks:
      - go-network
  grafana:
    container_name: grafana
    image: grafana/grafana:11.3.0
    volumes:
      - ./Docker/grafana.yml:/etc/grafana/provisioning/datasources/datasource.yaml
      - grafana-data:/var/lib/grafana
    ports:
      - 3000:3000
    networks:
      - go-network
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=password
volumes:
  grafana-data:
networks:
  go-network:
    driver: bridge

Also, create a new directory Docker in the root directory of your project. Inside the Docker directory, create two new files named prometheus.yml and grafana.yml.

Add the following code to the prometheus.yml file:

global:
  scrape_interval: 10s
  evaluation_interval: 10s

scrape_configs:
  - job_name: 'go-api'
    metrics_path: /metrics
    static_configs:
      - targets: ['api:8000']

Add the following code to the grafana.yml file:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    access: proxy
    isDefault: true

Now, start the entire stack using Docker Compose:

docker compose up -d

Access Grafana in your browser at http://localhost:3000. Log in with the username admin and password password. Prometheus is pre-configured as a data source, and you can now create dashboards to visualize the metrics exposed by your Go application.

Conclusion

By following this guide, you've successfully set up a monitoring system for your Go applications using Prometheus, Grafana, and Docker. This setup empowers you to proactively monitor your application's health, identify potential issues early, and ensure a smooth user experience.