Monitor Your Go Applications with Prometheus, Grafana, and Docker: The Ultimate Guide

Monitoring your applications is no longer optional; it's a critical necessity for identifying and resolving issues before they impact your users. In today's complex, distributed environments, manually testing each service is simply not feasible. This guide will walk you through setting up robust Go application monitoring using Prometheus, Grafana, and Docker, ensuring your applications run smoothly.

Why Monitor Your Go Apps?

• Ensure consistent performance: Spot bottlenecks and optimize code.
• Proactive issue detection: Identify and fix problems before users notice.
• Simplified debugging: Quickly pinpoint the root cause of errors.
• Data-driven decisions: Make informed choices based on real-time insights.

Prerequisites

Before diving in, ensure you have a working knowledge of:

• Golang
• Docker and Docker Compose
• Prometheus and Grafana

Step-by-Step Guide to Monitoring Go Applications

Let's break down the process into manageable steps.

1. Create a Golang Application

First, let's build a simple Go server.

• Create a new directory for your project.
• Initialize a new Go project: go mod init <project-name>.
• Create a main.go file and add the code below.

package main

import (
	"strconv"

	"github.com/gin-gonic/gin"
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

// Define metrics
var (
	HttpRequestTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "api_http_request_total",
		Help: "Total number of requests processed by the API",
	}, []string{"path", "status"})
	HttpRequestErrorTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "api_http_request_error_total",
		Help: "Total number of errors returned by the API",
	}, []string{"path", "status"})
)

// Custom registry (without default Go metrics)
var customRegistry = prometheus.NewRegistry()

// Register metrics with custom registry
func init() {
	customRegistry.MustRegister(HttpRequestTotal, HttpRequestErrorTotal)
}

func main() {
	router := gin.Default()

	// Register /metrics before middleware
	router.GET("/metrics", PrometheusHandler())
	router.Use(RequestMetricsMiddleware())

	router.GET("/health", func(c *gin.Context) {
		c.JSON(200, gin.H{
			"message": "Up and running!",
		})
	})
	router.GET("/v1/users", func(c *gin.Context) {
		c.JSON(200, gin.H{
			"message": "Hello from /v1/users",
		})
	})
	router.Run(":8000")
}

// Custom metrics handler with custom registry
func PrometheusHandler() gin.HandlerFunc {
	h := promhttp.HandlerFor(customRegistry, promhttp.HandlerOpts{})
	return func(c *gin.Context) {
		h.ServeHTTP(c.Writer, c.Request)
	}
}

// Middleware to record incoming requests metrics
func RequestMetricsMiddleware() gin.HandlerFunc {
	return func(c *gin.Context) {
		path := c.Request.URL.Path
		c.Next()
		status := c.Writer.Status()
		if status < 400 {
			HttpRequestTotal.WithLabelValues(path, strconv.Itoa(status)).Inc()
		} else {
			HttpRequestErrorTotal.WithLabelValues(path, strconv.Itoa(status)).Inc()
		}
	}
}

2. Install Dependencies

Run go mod tidy in your terminal. This command fetches and installs all necessary dependencies specified in your go.mod file. This includes the gin-gonic web framework and the prometheus/client_golang library.

3. Understand the Code

• Import Packages: We import gin-gonic/gin for the server and prometheus/client_golang/prometheus for exposing metrics.
• Define Metrics: HttpRequestTotal and HttpRequestErrorTotal are defined as CounterVec types, tracking total requests and errors. Labels path and status provide dimensionality.
• Custom Registry: A customRegistry avoids default Go metrics, giving you finer control.
• Register Endpoints: /health and /v1/users are basic endpoints; /metrics exposes Prometheus metrics.
• Middleware: RequestMetricsMiddleware intercepts requests, recording path and status to increment the relevant counters.

4. Run the Application Locally

Execute go run main.go from your project's root directory.

• Visit http://localhost:8000/health to confirm the server is running ("Up and running!").
• Visit http://localhost:8000/v1/users to see the "Hello from /v1/users" message.
• Visit http://localhost:8000/metrics to view the exposed Prometheus metrics. You should see api_http_request_total and api_http_request_error_total.

5. Dockerize the Application

Containerizing your Go application makes deployment and scaling easier.

• Create a Dockerfile in the project root:

FROM golang:1.24-alpine AS builder

# Set environment variables
ENV CGO_ENABLED=0 \
    GOOS=linux \
    GOARCH=amd64

# Set working directory inside the container
WORKDIR /build

# Copy go.mod and go.sum files for dependency installation
COPY go.mod go.sum ./

# Download dependencies
RUN go mod download

# Copy the entire application source
COPY . .

# Build the Go binary
RUN go build -o /app .

# Final lightweight stage
FROM alpine:3.17 AS final

# Copy the compiled binary from the builder stage
COPY --from=builder /app /bin/app

# Expose the application's port
EXPOSE 8000

# Run the application
CMD ["/bin/app"]

Understanding Multi-Stage Builds

• Build Stage: Uses a Golang Alpine image to compile the Go application. CGO_ENABLED=0 is important for creating static binaries.
• Final Stage: Uses a minimal Alpine image, copying only the compiled binary from the build stage, resulting in a smaller image size.

6. Build and Run the Docker Image

• Build the image: docker build -t go-prom-monitor .
• Run the container: docker run -d -p 8000:8000 --name go-prom-monitor go-prom-monitor

Test the application endpoints (/health, /v1/users, /metrics) again to confirm it behaves as expected within the container.

7. Connect to Prometheus and Grafana with Docker Compose

Docker Compose simplifies the management of multi-container applications.

• Create a compose.yml in the project root:

services:
  api:
    container_name: go-api
    build:
      context: .
      dockerfile: Dockerfile
    image: go-api:latest
    ports:
      - 8000:8000
    networks:
      - go-network
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 5
    develop:
      watch:
        - path: .
          action: rebuild
  prometheus:
    container_name: prometheus
    image: prom/prometheus:v2.55.0
    volumes:
      - ./Docker/prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - 9090:9090
    networks:
      - go-network
  grafana:
    container_name: grafana
    image: grafana/grafana:11.3.0
    volumes:
      - ./Docker/grafana.yml:/etc/grafana/provisioning/datasources/datasource.yaml
      - grafana-data:/var/lib/grafana
    ports:
      - 3000:3000
    networks:
      - go-network
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=password
volumes:
  grafana-data:
networks:
  go-network:
    driver: bridge

• Create a Docker directory in the project root. Inside it, create prometheus.yml and grafana.yml.

• prometheus.yml

global:
  scrape_interval: 10s
  evaluation_interval: 10s

scrape_configs:
  - job_name: 'go-api'
    static_configs:
      - targets: ['api:8000']

• grafana.yml

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    access: proxy
    isDefault: true

8. Run the Docker Compose File

Run docker compose up -d in the directory containing compose.yml. This will start all three services: your Go application, Prometheus, and Grafana.

• Prometheus: Access Prometheus at http://localhost:9090.
• Grafana: Access Grafana at http://localhost:3000 (login with admin/password).

9. Configure Grafana to Visualize Metrics

1. Log in to Grafana. Since we have configured the grafana.yml to provision the Prometheus datasource, a data source is already configured.
2. Create a new dashboard in Grafana.
3. Add a new panel and select "Prometheus" as the data source.
4. Use the Prometheus query language (PromQL) to query your metrics. For example, to visualize the total number of HTTP requests over time, you might use the query sum(rate(api_http_request_total[5m])).
5. Customize the panel to display the data in a meaningful way (e.g., using a line graph, gauge, or table).

Benefits of This Setup

• Automated Monitoring: Prometheus automatically scrapes metrics from your Go application, eliminating manual data collection.
• Real-time Visualization: Grafana provides powerful dashboards to visualize your metrics, making it easier to identify trends and anomalies.
• Scalability: The Docker Compose setup allows you to easily scale your monitoring stack as your application grows.

Next Steps and Long-Tail Keywords To Explore

• Alerting: Configure Prometheus alerting rules to automatically notify you when critical issues arise.
• Advanced PromQL: Learn more about PromQL to create complex queries and derive deeper insights from your metrics.
• Custom Exporters: Build custom Prometheus exporters to monitor application-specific metrics.
• Distributed Tracing: Integrate tracing tools like Jaeger or Zipkin for end-to-end request tracing in distributed systems.

This setup is suitable for anyone interested in Kubernetes Golang monitoring, and Go application performance monitoring using Grafana. It leverages Prometheus metric collection to create dashboards that offer complete visibility.