We have a server with Docker Swarm initialized. A few services are running on it. We need to scrap metrics to prometheus and we want it to use direct connection to the services without going through a load balancer.
I’ve spent a bunch of time figuring out how to do it, so here is a tutorial just for my future self.
There are several ways to scrap metrics with service discovery:
The first one doesn’t work well since Docker Swarm doesn’t expose SRV DNS records of services. I have no idea why, it sounds logic to do so. I would expose SRV DNS records.
The second one —
docker_sd_config is less useful than
dockerswarm_sd_config since it scans all the
containers, even the containers that are not a part of swarm service.
dockerswarm_sd_config has a field called
filters which just doesn’t work at all, I’ve tried
several options such as
label:x=y and all these combinations. Just doesn’t work.
So, the solution is to use
relabel_configs and use
action field to filter out services that
don’t want to be scrapped.
There are other roles such as
nodes, but we need
tasks since a container is a
task in the Docker Swarm dictionary.
- job_name: 'swarm' scrape_interval: 1s dockerswarm_sd_configs: - host: unix:///var/run/docker.sock role: tasks relabel_configs: - source_labels: [__meta_dockerswarm_task_desired_state] regex: running action: keep - source_labels: [__meta_dockerswarm_service_label_metrics_job] regex: .+ action: keep - source_labels: [__meta_dockerswarm_service_label_metrics_job] target_label: job - action: replace regex: (.+) source_labels: - __meta_dockerswarm_service_label_metrics_path target_label: __metrics_path__ - action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 source_labels: - __address__ - __meta_dockerswarm_service_label_metrics_port target_label: __address__
- The first relabel config keeps only services that have desired state equal to ‘running’.
- The second one looks for new name of the job, we don’t want all the metrics to have the same name “swarm”.
- The third one uses
metrics-pathlabel as path to metrics. By default it is
- The 4th one joins internal docker IP and
Note about ansible
Do not use
container_labels to put labels on containers. Just use
labels instead. I don’t know
container_labels aren’t scrapped at all for some reasons.
Follow me on Twitter: @reconquestio