Nginx monitoring installation: Filebeat+ES+Grafana (full)

Preface

This chapter describes Nginx monitoring installation

1. Install Nginx / OpenResty

1.1 Install OpenResty

VeryNginx is based on OpenResty, so you need to install it first:

Prepare for OpenResty installation

  • Centos:yum install -y readline-devel pcre-devel openssl-devel gcc GeoIP-devel
  • Debian|Ubuntu: apt-get install libreadline-dev libncurses5-dev libpcre3-dev libssl-dev perl make build-essential libgeoip-dev
  • Base libraries needed by geoip-devel for geoip modules
●wget  https://openresty.org/download/openresty-1.11.2.1.tar.gz
●tar -zxvf openresty-1.11.2.1.tar.gz
●cd openresty-1.11.2.1
●./configure --with-luajit --with-pcre --with-http_gzip_static_module --with-http_realip_module --with-http_geoip_module --with-http_ssl_module --with-http_stub_status_module --prefix=/app/openresty
●make && make install

VeryNginx actually uses these modules in OpenResty

  • lua-nginx-module
  • http_stub_status_module
  • lua-cjson library

If you don't want to install OpenResty or you already have a working Nginx, you can also manually install these modules for Nginx compilation yourself

1.2 Install and deploy VeryNginx

Clone the VeryNginx repository locally and copy the nginx.conf and VeryNginx folders to Nginx's working directory.

cd ~
git clone https://github.com/alexazhou/VeryNginx.git
rm -f /app/openresty/nginx/conf/nginx.conf
cp ~/VeryNginx/nginx.conf /app/openresty/nginx/conf/nginx.conf
# Create/app/openresty/nginx/waf/verynginx directory
mkdir -p /app/openresty/nginx/waf/verynginx
# Copy/VeryNginx/verynginx to/app/openresty/nginx/waf/verynginx
cp -r ~/VeryNginx/verynginx /app/openresty/nginx/waf/verynginx

1.3 Edit Nginx Profile

The VeryNginx project provides a configuration template/app/openresty/nginx/conf.You need to add the NGX configuration of your site to this template.But remember not to modify that part of the code that configures VeryNginx (unless you know what you're doing).

Next, it is important to note that there are three points to a configuration file:

include /app/openresty/nginx/waf/verynginx/verynginx/nginx_conf/in_external.conf;
include /app/openresty/nginx/waf/verynginx/verynginx/nginx_conf/in_http_block.conf;

include /app/openresty/nginx/waf/verynginx/verynginx/nginx_conf/in_server_block.conf;

in_http_block.conf z qualified profile needs to be modified

upstream vn_upstream{
    server 127.0.0.1;
    balancer_by_lua_file /app/openresty/nginx/waf/verynginx/verynginx/lua_script/on_banlance.lua;
        keepalive 1024; #Connection pool
}

lua_package_path '/app/openresty/nginx/waf/verynginx/verynginx/lua_script/?.lua;;/app/openresty/nginx/waf/verynginx/verynginx/lua_script/module/?.lua;;';
lua_package_cpath '/app/openresty/nginx/waf/verynginx/verynginx/lua_script/?.so;;';
lua_code_cache on;

lua_shared_dict status 1m;
lua_shared_dict frequency_limit 10m;
lua_shared_dict summary_long 10m;
lua_shared_dict summary_short 10m;

init_by_lua_file /app/openresty/nginx/waf/verynginx/verynginx/lua_script/on_init.lua;
rewrite_by_lua_file /app/openresty/nginx/waf/verynginx/verynginx/lua_script/on_rewrite.lua;
access_by_lua_file /app/openresty/nginx/waf/verynginx/verynginx/lua_script/on_access.lua;
log_by_lua_file /app/openresty/nginx/waf/verynginx/verynginx/lua_script/on_log.lua;

Do not change the other 2 files

The code to configure VeryNginx is in the following sections:

#-----------------VeryNginx config code------------------
 #user  nginx;
worker_processes  auto;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}

include /app/openresty/nginx/waf/verynginx/verynginx/nginx_conf/in_external.conf;
#include /app/openresty/nginx/waf/verynginx/nginx_conf/in_http_block.conf;
http {
    charset utf-8;
    sendfile     on;
    tcp_nopush   on;
    tcp_nodelay  on;
    server_tokens off;
    types_hash_max_size 2048;
    server_names_hash_bucket_size 128;
    large_client_header_buffers 4 32k;
    fastcgi_intercept_errors on;
    proxy_intercept_errors on;
    underscores_in_headers on;

    client_header_timeout      15s;
    client_body_timeout        15s;
    client_max_body_size      100m;
    client_body_buffer_size   2m;
    client_header_buffer_size 256k;

    #keepalive_timeout  0;
    keepalive_timeout  90s;
    keepalive_requests 2000;

    # MIME
    include       mime.types;
    default_type  application/octet-stream;
    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;
    #tcp_nopush     on;

    #keepalive_timeout  0;

    # gzip
    gzip               on;
    gzip_min_length    1k;
    gzip_buffers       4 16k;
    gzip_http_version  1.1;
    gzip_comp_level    6;
    gzip_types         text/plain text/css application/javascript application/x-javascript application/xml application/json;
    gzip_vary          on;

    proxy_connect_timeout    5s;
    proxy_read_timeout       1800s;
    proxy_send_timeout       1800s;
    proxy_buffer_size        128k;
    proxy_buffers            100 128k;
    #proxy_busy_buffers_size 1m;
    #proxy_temp_file_write_size 512k;

        #this line shoud be include in every http block
    include /app/openresty/nginx/waf/verynginx/verynginx/nginx_conf/in_http_block.conf;

    log_format   main   '$remote_addr -'
                        ' $remote_user'
                        ' [$time_local]'
                        ' "$request"'
                        ' $status'
                        ' $body_bytes_sent'
                        ' "$http_referer"'
                        ' "$http_user_agent"'
                        ' "$http_x_forwarded_for"'
                        ' $upstream_response_time'
                        ' $upstream_addr';

    access_log  logs/access.log  main;


    upstream gateway_pool {
# Address of gateway
        server 106.12.129.14:9200;

    }



    server {
        listen       80;

        #this line shoud be include in every server block
        include /app/openresty/nginx/waf/verynginx/verynginx/nginx_conf/in_server_block.conf;
        #include /app/openresty/nginx/waf/verynginx/nginx_conf/in_http_block.conf;
        location / {
           proxy_http_version 1.1;
           proxy_set_header Connection "";
           proxy_redirect      off;
           proxy_set_header    X-Real-IP $remote_addr;
           proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header    X-Forwarded-Host $host;
           proxy_set_header    X-Forwarded-Proto $http_x_forwarded_proto;
           proxy_set_header    Host $host;
           proxy_pass http://gateway_pool;
        }
    }

}

#---------------VeryNginx config code end-----------------

If you do not use the configuration template provided by VeryNginx, you can also manually add this section to your own Nginx configuration file. (If the installation path is not/app/openresty, you need to modify the values of lua_package_cpath and lua_package_path)

1.4 Start/Stop Services

Start service: /app/openresty/nginx/sbin/nginx

Stop service: /app/openresty/nginx/sbin/nginx-s stop

1.5 Configure VeryNginx

Open your browser to access http://127.0.0.1/VeryNginx/dashboard/index.html.

Default user name and password: verynginx / verynginx.

After logging in, you can view the status and modify the configuration.After modifying the configuration, remember to "Configure > System > Configure All" to save.

Tips

  • The new configuration is saved through the VeryNginx control panel and takes effect immediately without requiring restart/reload Nginx.

  • VeryNginx saves the configuration in / app/openresty/nginx/waf/verynginx/configs/config.json.

  • If you are unable to log in because of any mismatched options, you can manually delete config.json to empty the configuration.

openresty installed successfully
Overlay/app/openresty/nginx/waf directory with a network disk

Link: https://pan.baidu.com/s/1nt2SfrDtk83Q4gvwIlpBfQ Extraction Code: g5hs

2. Install elasticsearch

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.4.tar.gz
tar -zxvf  elasticsearch-6.5.4.tar.gz -C /usr/local/


useradd es
chown -R es:es /usr/local/elasticsearch-6.5.4/
cd /usr/local/elasticsearch-6.5.4
 Modify config/jvm.options to half the size of memory
vi config/jvm.options  
-Xms512m 
-Xmx512m
 Modify max file and max virtual memory parameters
 Use root or sudo users
vi /etc/sysctl.conf
 Add the following configuration:
vm.max_map_count=655360
 And execute the command:
sysctl -p

Configure ports across domains
vi /usr/local/elasticsearch-6.5.4/config/elasticsearch.yml
cluster.name: elasticsearch
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
node.max_local_storage_nodes: 2
http.cors.enabled: true
http.cors.allow-origin: "*"

Switch es user
su - es
 Run the following commands under the Elasticsearch home directory to install these plug-ins:
bin/elasticsearch-plugin install ingest-geoip
bin/elasticsearch-plugin install ingest-user-agent

es User Start
/usr/local/elasticsearch-6.5.4/bin/elasticsearch -d
 root User Start
su - es -c '/usr/local/elasticsearch-6.5.4/bin/elasticsearch -d'

2.1 windows installation elasticseach-head

  • Visit https://github.com/mobz/elasticsearch-head to download the headplug-in (select the zip package download method).

  • Modify ~\elasticsearch-6.6.2\elasticsearch-head-master\Gruntfile.js by adding the hostname:'*'configuration item to the corresponding location.

  • Execute npm install under ~\elasticsearch-6.6.2\elasticsearch-head-master to start the installation, and then either grunt server or npm run start to run the head plug-in when finished.

  • Installation was successful, accessing http://localhost:9100/.

2.2 Notes

  • Failed to connect ES in head er.

For Access-Control-Allow-Origin issues, you can add the following code at the end of the ~\config\elasticsearch.yml file in ElasticSearch 6.x:

http.cors.enabled: true 
http.cors.allow-origin: "\*" 
node.master: true node.data: true

After the configuration update, restart ES to connect successfully.

3. Install Filebeat

Recommended blog: https://www.cnblogs.com/cjsblog/p/9495024.html

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.5.4-linux-x86_64.tar.gz
tar -zxvf  filebeat-6.5.4-linux-x86_64.tar.gz  -C /usr/local/
cd /usr/local/
mv filebeat-6.5.4-linux-x86_64 filebeat-6.5.4
cd /usr/local/filebeat-6.5.4
vi filebeat.yml
filebeat.inputs:


- type: log

  enabled: false

  paths:
    - /app/openresty/nginx/logs/access.log


#============================= Filebeat modules ===============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false


setup.template.settings:
  index.number_of_shards: 3

setup.template.name: "nginx-log-"
setup.template.pattern: "nginx-log-*"
setup.template.overwrite: true





output.elasticsearch:
  enabled: true
  hosts: ["106.13.3.200:9200"]
  index: "nginx-log-%{+yyyy-MM-dd}"


#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  #- add_host_metadata: ~
  #- add_cloud_metadata: ~
  - drop_fields:
        fields: ["beat.name", "beat.version", "host.architecture","host.architecture","host.name","beat.hostname","log.file.path"]

3.1 Enable module nginx

cd /usr/local/filebeat-6.5.4/modules.d
vi  nginx.yml 

Perform the following configuration

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/app/openresty/nginx/logs/access.log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/app/openresty/nginx/logs/error.log"]

3.2 Configuration default.json

cd /usr/local/filebeat-6.5.4/module/nginx/access/ingest
vi default.json

{
  "description": "Pipeline for parsing Nginx access logs. Requires the geoip and user_agent plugins.",
  "processors": [{
    "grok": {
      "field": "message",
      "patterns":[
        "\"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{GREEDYDATA:nginx.access.info}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\" \"%{GREEDYDATA:nginx.access.xforwardedfor}\" %{GREEDYDATA:nginx.access.upstream_response_time} %{GREEDYDATA:nginx.access.upstream_addr}"
        ],
      "pattern_definitions": {
        "IP_LIST": "%{IP}(\"?,?\\s*%{IP})*"
      },
      "ignore_missing": true
    }
  }, {
    "grok": {
      "field": "nginx.access.info",
      "patterns": [
          "%{WORD:nginx.access.method} %{DATA:nginx.access.url} HTTP/%{NUMBER:nginx.access.http_version}",
          ""
      ],
      "ignore_missing": true
    }
  }, {
    "remove": {
      "field": "nginx.access.info"
    }
  }, {
    "split": {
      "field": "nginx.access.remote_ip_list",
      "separator": "\"?,?\\s+"
    }
  }, {
    "script": {
      "lang": "painless",
      "inline": "boolean isPrivate(def ip) { try { StringTokenizer tok = new StringTokenizer(ip, '.'); int firstByte = Integer.parseInt(tok.nextToken());       int secondByte = Integer.parseInt(tok.nextToken());       if (firstByte == 10) {         return true;       }       if (firstByte == 192 && secondByte == 168) {         return true;       }       if (firstByte == 172 && secondByte >= 16 && secondByte <= 31) {         return true;       }       if (firstByte == 127) {         return true;       }       return false;     } catch (Exception e) {       return false;     }   }   def found = false;   for (def item : ctx.nginx.access.remote_ip_list) {     if (!isPrivate(item)) {       ctx.nginx.access.remote_ip = item;       found = true;       break;     }   }   if (!found) {     ctx.nginx.access.remote_ip = ctx.nginx.access.remote_ip_list[0];   }"
      }
  }, {
    "remove":{
      "field": "message"
    }
  }, {
    "rename": {
      "field": "@timestamp",
      "target_field": "read_timestamp"
    }
  }, {
    "date": {
      "field": "nginx.access.time",
      "target_field": "@timestamp",
      "formats": ["dd/MMM/YYYY:H:m:s Z"]
    }
  },{
    "remove": {
      "field": "nginx.access.time"
    }
  }, {
    "user_agent": {
      "field": "nginx.access.agent",
      "target_field": "nginx.access.user_agent"
    }
  }, {
    "rename": {
      "field": "nginx.access.agent",
      "target_field": "nginx.access.user_agent.original"
    }
  }, {
    "geoip": {
      "field": "nginx.access.remote_ip",
      "target_field": "nginx.access.geoip"
    }
  }, {
    "script": {
      "lang": "painless",
      "inline": "String tmp=ctx.nginx.access.upstream_response_time; if (tmp=='-'){ctx.nginx.access.upstream_response_time=-1.0}else{ctx.nginx.access.upstream_response_time=Float.parseFloat(tmp)}"
      }
  }],
  "on_failure" : [{
    "set" : {
      "field" : "error.message",
      "value" : "{{ _ingest.on_failure_message }}"
    }
  }]
}

cd /usr/local/filebeat-6.5.4
nohup ./filebeat -e -c filebeat.yml >&/dev/null &

4. Install Grafana

4.1 Installation steps

wget https://dl.grafana.com/oss/release/grafana-6.1.6.linux-amd64.tar.gz
tar -zxvf grafana-6.1.6.linux-amd64.tar.gz
cd /app/grafana-6.1.6/bin
grafana-cli plugins install grafana-piechart-panel
grafana-cli plugins install grafana-piechart-panel
grafana-cli plugins install grafana-worldmap-panel
grafana-cli plugins install grafana-piechart-panel
grafana-cli plugins install grafana-piechart-panel

./grafana-server 

4.1 Grafana Configuration

Install port 3000 by default at 106.13.3.200:3000
Default username/password admin/admin

4.1.1 Configuration Data Source

4.1.2 json template

Because the Json template is too long, it is saved in Baidu netdisk.Welcome to Personal Public Coder programming, Background Reply: Grafana imports Json templates.This is available.
If invalid, please trust the public number privately.I'll get back to you as soon as I see it.

4.1.3 Importing nginx dashboard


Focus

The above is from the open source project OCP: https://gitee.com/owenwangwen/open-capacity-platform

Project presentation address
http://59.110.164.254:8066/login.html username/password: admin/admin

Project Monitoring
http://106.13.3.200:3000 username/password: admin/1q2w3e4r

Project Code Address
https://gitee.com/owenwangwen/open-capacity-platform

Group number: 483725710 (Note: Coder programming) Welcome to join ~

end of document

Welcome to Personal WeChat Public Number: Coder Programming
Get the latest original technical articles and free learning materials, more boutique mind maps, interview materials, PMP reference materials and so on, to help you learn technical knowledge anytime, anywhere!

Welcome to your attention and star~

Tags: Nginx ElasticSearch JSON Linux

Posted on Tue, 03 Sep 2019 18:58:05 -0700 by webdes03