Collection and analysis of syslog by ELK

Introduction to ELK

First, ELK is a centralized logging system. It is not a software, but consists of three components: Elastic search, Logstash and Kibana.

Elastic search is a search server based on Lucence, but it is not only a search engine, but also a database, but also an index database to improve performance. Although developed in Java, Elastic search encapsulates a set of http access interfaces and uses restful design style. It is the most popular search quotation at present. One of the engines

Logstash is a tool for receiving, processing and forwarding logs. It can access almost any data, integrate with a variety of external applications, and support flexible expansion. In web development, more data is imported from existing databases. But in collecting syslog, it's the core component

Kibana is an open source, free tool that can better display the data provided by Elastic search and Logstash on the web interface to help us aggregate, analyze and search.

Before describing the installation and configuration, I should first state that the grammar of ELK 2.X, 5.X and 6.X is different. The ELK that I use is not compatible with the version 6.3 that runs in the installation and configuration of CenOS 7.5. There may be some slight differences with other versions.

Installation and configuration of Elastic search``

Both Elastic search and Logstash depend on JDK, so they need to be installed.

[python@localhost ~]$ sudo tar -zxvf jdk-8u171-linux-x64.tar.gz -C /usr/local/   # Unzip the downloaded jdk installation package
[python@localhost ~]$ cd /usr/local/
[python@localhost local]$ sudo vi /etc/profile  # Setting environment variables
# Insert the following paragraph
export JAVA_HOME=/usr/local/jdk1.8.0_171
export JRE_HOME=$JAVA_HOME/jre
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

[python@localhost local]$ source /etc/profile    # Enabling Environmental Variables
[python@localhost local]$ java -version		# Verify that the installation was successful
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

Es boot cannot use root users, so please create a new one if it already exists, ignore this section

root@localhost ~]# adduser python		# Create a new user, python
[root@localhost ~]# passwd python		# Setting Password
//Change user python's password.
//New password:
//Re-enter the new password:
passwd: All authentication tokens have been successfully updated.
[root@localhost ~]# whereis sudoers		
sudoers: /etc/sudoers /etc/sudoers.d /usr/share/man/man5/sudoers.5.gz
[root@localhost ~]# vi /etc/sudoers
root    ALL=(ALL)       ALL
python  ALL=(ALL)       ALL  			# Add your new users
[root@localhost ~]# chmod -v u-w /etc/sudoers		#  Setting New User Rights
"/etc/sudoers" The permission mode is reserved at 0440 (r--r-----)

The following is the installation configuration of Es

[root@localhost ~]# cd /usr/local/
[root@localhost ~]# mkdir elk			# Create an elk folder
[root@localhost ~]# cd /home/python/
[root@localhost python]# tar -zxvf elasticsearch-6.3.1.tar.gz -C /usr/local/elk/		# Unzip installation package
[python@localhost elk]$ sudo chown -R python:python /usr/local/elk/elasticsearch-6.3.1/		# Change to python bootable permissions
[python@localhost elk]$ cd elasticsearch-6.3.1/
[python@localhost elasticsearch-6.3.1]$ ./bin/elasticsearch		# Start Es
[python@localhost ~]$ curl		# Accessing es Library
  "name" : "Dv_SjHh",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "JHASOy-eQRCTKTY_pt-Bqg",
  "version" : {
    "number" : "6.3.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "eb782d0",
    "build_date" : "2018-06-29T21:59:26.107521Z",
    "build_snapshot" : false,
    "lucene_version" : "7.3.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  "tagline" : "You Know, for Search"

Es can now run locally. If you want to visit Es from outside, you need the following steps

[python@localhost elasticsearch-6.3.1]$ sudo vi ./config/elasticsearch.yml 

[python@localhost elasticsearch-6.3.1]$ sudo vi /etc/security/limits.conf 
# Add at the bottom of the file
* hard nofile 65536   
* soft nofile 131072		
* hard nproc 4096
* soft nproc 2048
# nofile - Maximum number of open files
# noproc - Maximum number of processes
# soft refers to the settings in effect for the current system.
# hard indicates the maximum value that can be set in the system

[python@localhost elasticsearch-6.3.1]$ sudo vi /etc/sysctl.conf 
vm.max_map_count=65530 		# So by default, the maximum number of threads that a single jvm can open is half
file-max		# Set the total number of files open for all processes in the system 
[python@localhost elasticsearch-6.3.1]$ sudo sysctl -p
[python@localhost elasticsearch-6.3.1]$ ./bin/elasticsearch -d		# Background Start Es

At this time, the single machine configuration is completed.


[root@localhost python]# tar -zxvf logstash-6.3.1.tar.gz -C /usr/local/elk/
[python@localhost ~]$ cd /usr/local/elk/logstash-6.3.1/
[python@localhost logstash-6.3.1]$ sudo vi syslog.conf
input {
  syslog {
    port => "8899"		# Ports below 1024 cannot be set in later versions of 6.x
output {
  elasticsearch {
    hosts => [""]		# Your own Es IP address
    index => "logstash_syslog-%{+YYYY.MM.dd}"		# This is the name that identifies the library.
[python@localhost logstash-6.3.1]$ service rsyslog stop		# Shut down the rsyslog of the local machine to receive syslog
[python@localhost logstash-6.3.1]$ nohup ./bin/logstash -f syslog.conf		# Background start logstash


[root@localhost python]# tar -zxvf kibana-6.3.1-linux-x86_64.tar.gz -C /usr/local/elk/
[python@localhost ~]$ cd /usr/local/elk/kibana-6.3.1-linux-x86_64/
[python@localhost kibana-6.3.1-linux-x86_64]$ sudo vi config/kibana.yml 
# Add the following
erver.port: 5601 ""
elasticsearch.url: ""
kibana.index: ".kibana"

Then go to the browser and open it

Enter here and see the data you save.

Tags: Python ElasticSearch sudo Java

Posted on Thu, 29 Aug 2019 00:16:39 -0700 by maxpouliot