Docker Docker monitoring platform

Monitoring dimension

  • Host dimension
    • Host cpu
    • Host memory
    • Host local image
    • Container operation on the host
  • Mirror dimension
    • Basic information of image
    • Basic information of image and container
    • Historical information of image construction (level dependent information)
  • Container dimension
    • Basic information of container
    • Operation status information of container
    • Container usage information

docker monitoring command

docker ps
docker images
docker stats
# Note: docker stats can only be used when libcontainer is selected as the execution driver
# docker stats often has some limitations. Using the stats api will show more information
echo -e "GET /containers/tools/stats HTTP/1.0\r\n" | nc -U /var/run/docker.sock
docker inspac
docker top
docker port

cAdvisor

google cAdvisor Some features of:

  • Rich api support for remote management api docs
  • With web ui management interface
  • It's another google project that supports k8s
  • Monitoring information is output to the influxdb database for storage and reading, supporting many other Plug-in unit
  • Support to output the container's statistics in the form of prometheus standard index and store it in the / metrics HTTP service endpoint
sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  google/cadvisor:latest

DataDog

Promtheus of SoundCloud

Promtheus is an open source service monitoring system combined with time series database exporter Using, exporter is based on prometheus open Http interface redevelopment to grab the indicator data of application program

Global setting

network configure

docker network create --driver bridge --subnet 10.0.0.0/24 --gateway 10.0.0.1 monitor

Grafana configure

Grafana Template

# pull grafana image
docker pull grafana/grafana

grafana directory

  • Configuration file / etc/grafana/
  • sqlite3 database file /var/lib/grafana

Environment variable of grafana in docker

  • GF? Server? Root? Url = http://grafana.server.name specifies the access path of grafana
  • GF? Security? Admin? Password = secret specifies the login password of grafana
  • GF? Install? Plugins = grafana clock panel, grafana simple JSON datasource specifies the variable to install the plug-in

Build persistent storage

docker run \
 -d \
 -v /var/lib/grafana \
 --name grafana-storage \
 busybox:latest

Starting container

# Start the Grafana container
docker run \
 -d \
 -p 3000:3000 \
 --name grafana \
 --volumes-from grafana-storage \
 -e "GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-simple-json-datasource" \
 -e "GF_SERVER_ROOT_URL=http://10.0.0.10:3000" \
 -e "GF_SECURITY_ADMIN_PASSWORD=marion" \
 --network monitor \
 --ip 10.0.0.10 \
 --restart always \
 grafana/grafana

# View the mount location of the configuration file and data directory
docker inspect grafana

cAdvisor

sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --detach=true \
  --name=cadvisor \
  --network monitor \
  --ip 10.0.0.11 \
  google/cadvisor:latest

Deploy by docker-compose yaml file

docker-compose.yml

prometheus:
    image: prom/prometheus:latest
    container_name: monitoring_prometheus
    restart: unless-stopped
    volumes:
      - ./data/prometheus/config:/etc/prometheus/
      - ./data/prometheus/data:/prometheus
    command:
      - '-config.file=/etc/prometheus/prometheus.yml'
      - '-storage.local.path=/prometheus'
      - '-alertmanager.url=http://alertmanager:9093'
    expose:
      - 9090
    ports:
      - 9090:9090
    links:
      - cadvisor:cadvisor
      - node-exporter:node-exporter

  node-exporter:
    image: prom/node-exporter:latest
    container_name: monitoring_node_exporter
    restart: unless-stopped
    expose:
      - 9100

  cadvisor:,
    image: google/cadvisor:latest
    container_name: monitoring_cadvisor
    restart: unless-stopped
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    expose:
      - 8080

  grafana:
    image: grafana/grafana:latest
    container_name: monitoring_grafana
    restart: unless-stopped
    links:
      - prometheus:prometheus
    volumes:
      - ./data/grafana:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=MYPASSWORT
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_SERVER_DOMAIN=myrul.com
      - GF_SMTP_ENABLED=true
      - GF_SMTP_HOST=smtp.gmail.com:587
      - GF_SMTP_USER=myadrress@gmail.com
      - GF_SMTP_PASSWORD=mypassword
      - GF_SMTP_FROM_ADDRESS=myaddress@gmail.com



prometheus.yml

# my global config
global:
  scrape_interval:     120s # By default, scrape targets every 15 seconds.
  evaluation_interval: 120s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'my-project'

# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
  # - "alert.rules"
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 120s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
         - targets: ['localhost:9090','cadvisor:8080','node-exporter:9100', 'nginx-exporter:9113']

command

docker-compose up -d

sysdig

sysdig offical

docker pull sysdig/sysdig
docker run -i -t --name sysdig --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/sysdig

docker container exec -it sysdig bash
csysdig

Weave Scope

weave scope

Common container monitoring tools

Docker Runtime Metrics Guide

Tags: Linux Docker Google network Database

Posted on Tue, 03 Dec 2019 20:24:21 -0800 by jumphopper