Elasticsearch log alert

Elasticsearch log alarm

elastalert use posture correctly

Preface

In May, Github found an open-source software, elastalert, to find an ELK log alert solution online Portal

Look at the latest update for two years ago, so have been in a semi questioning state to look at the software, until further... In depth

Latest news: as of the time of release, see the latest changes of the project. Add Kibana plug-in support!

Wrong posture

Because I don't know much about elastalert, I didn't use es query in the early use process (maybe ES6 was just released at that time, and there was no targeted modification to ES6 version), so I did some unnecessary secondary development

Try again and again at any time later, and find that elastalert is still quite powerful, more and more optimistic, and gradually understand the real use... Understand the correct posture

Correct posture

Targeting demand

50X alarm of monitoring station, when there are 100 50X alarms in one minute

name: nginx access 50X rule
type: frequency
index: access*
num_events: 100
timeframe:
    minutes: 1
filter:
- range:
    status:
      from: 500
      to: 599
alert: "modules.eagle_post.EagleAlerter"
eagle_post_url: ""
eagle_post_all_values: False
eagle_time_start: "02:00"
eagle_time_end: "06:00"
eagle_post_payload:
  host: "host"
  status: "status"
  request: "request_uri"
  remoteaddr: "remote_addr"
  requesttime: "request_time"
  useragent: "http_user_agent"
  method: "request_method"
  time: "time_local"

Design sketch
Elastaticsearch(elastalert)

The monitoring system performs hazardous operations

index: system_history*
timeframe:
    minutes: 0
filter:
- query:
    - bool:
        should:
            - match: {"command":"mysqldump"}
            - match: {"command":"rm -rf"}
            - match: {"command":"shutdown"}
            - match: {"command":"passwd"}
            ...

Monitoring database error log

type: blacklist_v2
index: mysql_log*
timeframe:
    minutes: 1
reverse: False
compare_key: message
filter:
- query:
    - bool:
        must:
            - term: {"tag":"error_log"}
        must_not:
            - match: {"message":"[Warning]"}
            - match: {"message":"[Note]"}
blacklist_v2:
    - "Too many connections"
    - "ERROR"
    - "error"
    - "table full"

Posterior language

Technicians can't jump to conclusions after they understand it a little bit!

In the future, we will continue to pay attention to elastalert and welcome everyone to exchange experience

Tags: Operation & Maintenance ElasticSearch github Nginx mysqldump

Posted on Sun, 08 Dec 2019 20:50:57 -0800 by alex.hughson