Network probe: Blackbox Exporter

What is a blackbox exporter?

Blackbox Exporter is the official black box monitoring solution provided by Prometheus community, which allows users to detect the network through: http\HTTPS\DNS\TCP\ICMP

Running blackbox exporter as docker

Pull mirror image

docker pull prom/blackbox-exporter

Check the dockerfile (this is a good habit)

$ cat check_docker_file.sh
#!/bin/bash
export PATH=$PATH
if [ $# -eq 1 ];then
        docker history --format {{.CreatedBy}} --no-trunc=true $1 |sed "s/\/bin\/sh\ -c\ \#(nop)\ //g"|sed "s/\/bin\/sh\ -c/RUN/g" | tac
    else
        echo "sh Obtain_dockerfile.sh $DOCKER_IMAGE"
fi

$ sh check_docker_file.sh prom/blackbox-exporter:latest
ADD file:b265aa0ea2ef7ff1f4a3e087217e75aca2c90f5c345406299664cc7969b2b28e in /
 CMD ["sh"]
 MAINTAINER The Prometheus Authors <prometheus-developers@googlegroups.com>
COPY dir:b1c1c3c551755544b818d03ad9136b137ca12c48393ba5cdd58d7f845647e042 in /
 LABEL maintainer=The Prometheus Authors <prometheus-developers@googlegroups.com>
 ARG ARCH=amd64
 ARG OS=linux
COPY file:2bfe91827ebb767bc51f40cd84675a3c315d9da8a70f6d8071c806e0b2b1ee73 in /bin/blackbox_exporter
COPY file:6e820c2d591d3433d139b66241b74e9b7ffc90c9e120bac49cf97014e16f070a in /etc/blackbox_exporter/config.yml
 EXPOSE 9115
 ENTRYPOINT ["/bin/blackbox_exporter"]
 CMD ["--config.file=/etc/blackbox_exporter/config.yml"]

Run blackbox exporter

docker run -id --name blackbox-exporter -p 9115:9115  prom/blackbox-exporter

Interpretation of blackbox exporter configuration file

Official explanation: https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md

modules:
  http_2xx:
    prober: http
  http_post_2xx:
    prober: http
    http:
      method: POST
  tcp_connect:
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp

When running the blackbox exporter, the user needs to provide the configuration information of the probe. The configuration information may be customized HTTP header information, some TSL (secret key certificate) configuration required by the probe, or the verification behavior of the probe itself. Each probe configuration in the blackbox exporter is called a module, and is provided to the blackbox in the form of YAML configuration file Exporter. Each module mainly includes the following configuration contents, probe type, authentication access timeout, and specific configuration items of the current probe:

# Probe type: http https tcp dns icmp
prober: <prober_string>   #Mandatory

# Timeout:
[timeout: <duration>] #Default unit second

# Detailed configuration of probe, at most one of which can be configured
[ http: <http_probe> ]
[ tcp: <tcp_probe> ]
[ dns: <dns_probe> ]
[ icmp: <icmp_probe> ]

< HTTP probe > configurable parameters

# Status code accepted by this probe. The default is 2xx.
  [ valid_status_codes: <int>, ... | default = 2xx ]

# The HTTP version accepted by this probe
  [ valid_http_versions: <string>, ... ]

#The HTTP method that the probe will use.
  [ method: <string> | default = "GET" ]

# The HTTP header set for the probe.
  headers:
    [ <string>: <string> ... ]

# Will the probe follow any redirection
  [ no_follow_redirects: <boolean> | default = false ]

# If SSL is present, the probe fails.
  [ fail_if_ssl: <boolean> | default = false ]

# If SSL does not exist, the probe fails.
  [ fail_if_not_ssl: <boolean> | default = false ]

# If the response body matches the regular expression, the probe fails.
  fail_if_body_matches_regexp:
    [ - <regex>, ... ]

# If the response body does not match the regular expression, the probe fails.
  fail_if_body_not_matches_regexp:
    [ - <regex>, ... ]

# If the response header matches the regular expression, the probe fails. For headers with multiple values, if * matches at least one * then it fails.
  fail_if_header_matches:
    [ - <http_header_match_spec>, ... ]

# If the response header does not match the regular expression, the probe fails. For headers with multiple values, if * none * does not match, it fails.
  fail_if_header_not_matches:
    [ - <http_header_match_spec>, ... ]

# Configuration of TLS protocol of HTTP probe.
  tls_config:
    [ <tls_config> ]

# HTTP basic authentication credentials for the target.
  basic_auth:
    [ username: <string> ]
    [ password: <secret> ]

# The host token for the target.
  [ bearer_token: <secret> ]

# Hosted token file for target
  [ bearer_token_file: <filename> ]

# The HTTP proxy server used to connect to the destination.
  [ proxy_url: <string> ]

# IP protocol of HTTP probe (ip4, ip6)
  [ preferred_ip_protocol: <string> | default = "ip6" ]
  [ ip_protocol_fallback: <boolean> | default = true ]

# The body of the HTTP request used in the probe.
  body: [ <string> ]

###################################################################
<http_header_match_spec>
    header: <string>,
    regexp: <regex>,
    [ allow_missing: <boolean> | default = false ]

Several application scenarios are introduced respectively

ping detection

ping(icmp) can be used to detect the survival of the server, and ping module is configured in the prometheus configuration file:

  icmp:
    prober: icmp

Integration with prometheus

  - job_name: 'blackbox-ping'
    metrics_path: /probe
    params:
      modelus: [icmp]
    static_configs:
    - targets:
      - 223.5.5.5
      lables:
        instance: aliyun
    - targets:
      - 47.92.229.67
      lables:
        instance: zsf
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.111.65:9115

HTTP

blackbox config file

modules:
  http_2xx:
    prober: http
    http:
      method: GET
  http_post_2xx:
    prober: http
    http:
      method: POST

Run as command after configuration

#DNS resolution time, unit s
probe_dns_lookup_time_seconds 0.039431355
#Time from the beginning to the end of detection, unit s, response time of the request page
probe_duration_seconds 0.651619323

probe_failed_due_to_regex 0

#Length of HTTP content response
probe_http_content_length -1
#Count the time of each stage according to the stage
probe_http_duration_seconds{phase="connect"} 0.050388884   #Connection time
probe_http_duration_seconds{phase="processing"} 0.45868667 #Time to process request
probe_http_duration_seconds{phase="resolve"} 0.040037612  #response time
probe_http_duration_seconds{phase="tls"} 0.145433254    #Time to verify certificate
probe_http_duration_seconds{phase="transfer"} 0.000566269 
#Number of redirects
probe_http_redirects 1
#SSL indicates whether SSL is used for final redirection
probe_http_ssl 1
#Status code returned
probe_http_status_code 200
#Uncompressed response body length
probe_http_uncompressed_body_length 40339
#Version of http protocol
probe_http_version 1.1
#Version number of the ip protocol used
probe_ip_protocol 4

probe_ssl_earliest_cert_expiry 1.59732e+09
#Whether the detection is successful
probe_success 1
#Version number of TLS
probe_tls_version_info{version="TLS 1.2"} 1

Integrated with prometheus, adopting its Relabelinng capability (service discovery)

  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      modelue: [http_2xx]
    static_configs:
    - targets:
      - http://www.zhangshoufu.com
      - http://www.xuliangwei.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.111.65:9115  #The machine and port of blackbox exporter

Here, we define a collection task for each probe service (such as HTTP? 2XX), and directly define the collection target of the task as the site we need to probe. Before collecting the sample data, set the collection task dynamically through relabel ﹣ configs.

1. According to the address of the Target instance, write it to the "param" Target label. The label in the form of "param" & lt;name> indicates that the < name > parameter will be added to the request Target address when collecting tasks, which is equivalent to the setting of params
2. Get the value of "param" target and write it to the instance tag;
3. Override the address tag value of the Target instance to the access address of the BlockBox Exporter instance.

Custom HTTP request
HTTP services are usually presented in different forms. Some may be simple web pages, while others may be REST based API services. For different types of HTTP probes, administrators need to be able to make more customized settings for the behavior of HTTP probes, including: http request method, HTTP header information, request parameters, etc. For some services with security authentication enabled, you need to be able to set corresponding Auth support for HTTP probe. For HTTPS type services, you need to be able to customize the certificate settings.
As shown below, the method is used to define the request method to be used in the detection. For some services that need request parameters, you can also define the relevant request header information through headers, and use body to define the request content:

http_post_2xx:
    prober: http
    timeout: 5s
    http:
      method: POST
      headers:
        Content-Type: application/json
      body: '{}'

If the HTTP service enables security authentication, the Blockbox Exporter has built-in support for basic auth. You can directly set the relevant authentication information:

http_basic_auth_example:
    prober: http
    timeout: 5s
    http:
      method: POST
      headers:
        Host: "login.example.com"
      basic_auth:
        username: "username"
        password: "mysecret"

For services that use Bear Token, you can also specify the token string directly through the Bear Token configuration item, or you can specify the token file through the Bear Token file.
For some services that have HTTPS enabled but need custom certificates, you can specify the relevant certificate information through TLS config:

http_custom_ca_example:
    prober: http
    http:
      method: GET
      tls_config:
        ca_file: "/certs/my_cert.crt"

Custom probe behavior
By default, the HTTP probe will only check the HTTP return status code. If the status code is 2XX (200 < = statuscode < 300), the probe is successful, and the probe return indicator probe "success" value is 1.
If the user needs to specify the HTTP return status code, or has special requirements for the HTTP version, as shown below, you can define it by using valid HTTP versions and valid status codes:

http_2xx_example:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2"]
      valid_status_codes: [200,301,302]

By default, the sample data returned by the Blockbox also contains the indicator probe? HTTP? SSL, which indicates whether the current probe uses SSL:

# HELP probe_http_ssl Indicates if SSL was used for the final redirect
# TYPE probe_http_ssl gauge
probe_http_ssl 0

If the user has a mandatory standard for whether or not to enable SSL for HTTP service. You can use fail if not SSL and fail if not SSL for configuration. If fail_if_ssl is true, the probe fails if the site has SSL enabled, otherwise it succeeds. Fail if not SSL is the opposite.

  http_2xx_example:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: []
      method: GET
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: false

In addition to HTTP status code, HTTP protocol version and whether SSL is enabled as the standard to control the success of probe detection behavior, the response content of HTTP service can also be matched. Using fail if matches regexp and fail if not matches regexp users can define a set of regular expressions to verify whether the content returned by HTTP conforms to or does not conform to the content of the regular expression.

  http_2xx_example:
    prober: http
    timeout: 5s
    http:
      method: GET
      fail_if_matches_regexp:
        - "Could not connect to database"
      fail_if_not_matches_regexp:
        - "Download the latest version here"

See the above configuration details

Show it in grafana

Import 9965 dashboards in the grafana web interface, and remember to install the pie chart
Manual installation

wget  https://grafana.com/api/plugins/grafana-piechart-panel/versions/latest/download -O ./grafana-piechart-panel.zip
 Download it to the plug-in directory, extract it to the directory named grafana piechart panel, and restart grafana

Reference resources:

blackbox-configs --> https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md
prometheus-configs --> https://prometheus.io/docs/prometheus/latest/configuration/configuration/
https://yunlzheng.gitbook.io/prometheus-book/part-ii-prometheus-jin-jie/exporter/commonly-eporter-usage/install_blackbox_exporter
https://www.li-rui.top/2018/11/23/monitor/blackbox_exporter%E4%BD%BF%E7%94%A8/

Tags: Linux SSL DNS Docker github

Posted on Thu, 06 Feb 2020 01:32:03 -0800 by gapern