return and job management of SaltStack

Article directory

1. return of saltstack component

The return component can be understood as the SaltStack system stores or returns data returned by Minion to other programs. It supports multiple storage methods, such as MySQL, MongoDB, Redis, Memcache, etc. through return, we can record each operation of SaltStack and provide data sources for future log audit. At present, the government has supported 30 kinds of return data storage and interface. We can easily configure and use it. Of course, it also supports the return defined by itself. The custom return needs to be written by python. After selecting and configuring the return to use, just specify return after the salt command.

//View all return lists
[root@master ~]# salt '*' sys.list_returners
192.168.69.202:
    - carbon
    - couchdb
    - elasticsearch
    - etcd
    - highstate
    - hipchat
    - local
    - local_cache
    - mattermost
    - multi_returner
    - pushover
    - rawfile_json
    - slack
    - smtp
    - splunk
    - sqlite3
    - syslog
    - telegram

1.1 return process

Return is to trigger the task on the Master side, and then minion will directly establish a connection with the return storage server after receiving the processing task, and then save the data return to the storage server. We must pay attention to this point, because this process is all Minion side operation storage server, so we should ensure that the configuration of Minion side and the dependent package are correct, which means that we will have to install the specified return mode dependent package on each Minion. If Mysql is used as the return storage mode, then we will install the python-mysql module on each Minion.

1.2 use mysql as the return storage method

Install MySQL Python module on all minion s

[root@master ~]# salt '*' pkg.install MySQL-python
192.168.69.202:
    ----------
    MySQL-python:
        ----------
        new:
            1.2.5-1.el7
        old:
[root@master ~]# salt '*' cmd.run 'rpm -qa|grep MySQL-python'
192.168.69.202:
    MySQL-python-1.2.5-1.el7.x86_64

Deploy a mysql server as the storage server. Here, deploy it directly on the host 192.168.69.202

//Deploying mysql
[root@minion ~]# yum -y install mariadb-server
[root@minion ~]# systemctl start mariadb
[root@minion ~]# systemctl enable mariadb
Created symlink from /etc/systemd/system/multi-user.target.wants/mariadb.service to /usr/lib/systemd/system/mariadb.service.
[root@minion ~]# ss -antl
State      Recv-Q Send-Q          Local Address:Port                         Peer Address:Port
LISTEN     0      128                         *:22                                      *:*
LISTEN     0      100                 127.0.0.1:25                                      *:*
LISTEN     0      50                          *:3306                                    *:*
LISTEN     0      128                        :::22                                     :::*
LISTEN     0      100                       ::1:25                                     :::*

//Create database and table structure
[root@minion ~]# mysql
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> CREATE DATABASE  `salt`
    ->   DEFAULT CHARACTER SET utf8
    ->   DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.01 sec)

MariaDB [(none)]>
MariaDB [(none)]> USE `salt`;
Database changed
MariaDB [salt]>
MariaDB [salt]> --
MariaDB [salt]> -- Table structure for table `jids`
MariaDB [salt]> --
MariaDB [salt]>
MariaDB [salt]> DROP TABLE IF EXISTS `jids`;
Query OK, 0 rows affected, 1 warning (0.00 sec)

MariaDB [salt]> CREATE TABLE `jids` (
    ->   `jid` varchar(255) NOT NULL,
    ->   `load` mediumtext NOT NULL,
    ->   UNIQUE KEY `jid` (`jid`)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.02 sec)

MariaDB [salt]> CREATE INDEX jid ON jids(jid) USING BTREE;
ERROR 1061 (42000): Duplicate key name 'jid'
MariaDB [salt]>
MariaDB [salt]> drop database salt;
Query OK, 1 row affected (0.02 sec)

MariaDB [(none)]>
MariaDB [(none)]> CREATE DATABASE  `salt`
    ->   DEFAULT CHARACTER SET utf8
    ->   DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]>
MariaDB [(none)]> USE `salt`;
 NOT NULL,
`alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
`master_id` varchar(255) NOT NULL,
PRIMARY KEY (`id`),
KEY `tag` (`tag`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;Database changed
MariaDB [salt]>
MariaDB [salt]> --
MariaDB [salt]> -- Table structure for table `jids`
MariaDB [salt]> --
MariaDB [salt]>
MariaDB [salt]> DROP TABLE IF EXISTS `jids`;
Query OK, 0 rows affected, 1 warning (0.00 sec)

MariaDB [salt]> CREATE TABLE `jids` (
    ->   `jid` varchar(255) NOT NULL,
    ->   `load` mediumtext NOT NULL,
    ->   UNIQUE KEY `jid` (`jid`)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.01 sec)

MariaDB [salt]> CREATE INDEX jid ON jids(jid) USING BTREE;
ERROR 1061 (42000): Duplicate key name 'jid'
MariaDB [salt]>
MariaDB [salt]> --
MariaDB [salt]> -- Table structure for table `salt_returns`
MariaDB [salt]> --
MariaDB [salt]>
MariaDB [salt]> DROP TABLE IF EXISTS `salt_returns`;
Query OK, 0 rows affected, 1 warning (0.00 sec)

MariaDB [salt]> CREATE TABLE `salt_returns` (
    ->   `fun` varchar(50) NOT NULL,
    ->   `jid` varchar(255) NOT NULL,
    ->   `return` mediumtext NOT NULL,
    ->   `id` varchar(255) NOT NULL,
    ->   `success` varchar(10) NOT NULL,
    ->   `full_ret` mediumtext NOT NULL,
    ->   `alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    ->   KEY `id` (`id`),
    ->   KEY `jid` (`jid`),
    ->   KEY `fun` (`fun`)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.02 sec)

MariaDB [salt]>
MariaDB [salt]> --
MariaDB [salt]> -- Table structure for table `salt_events`
MariaDB [salt]> --
MariaDB [salt]>
MariaDB [salt]> DROP TABLE IF EXISTS `salt_events`;
Query OK, 0 rows affected, 1 warning (0.00 sec)

MariaDB [salt]> CREATE TABLE `salt_events` (
    -> `id` BIGINT NOT NULL AUTO_INCREMENT,
    -> `tag` varchar(255) NOT NULL,
    -> `data` mediumtext NOT NULL,
    -> `alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    -> `master_id` varchar(255) NOT NULL,
    -> PRIMARY KEY (`id`),
    -> KEY `tag` (`tag`)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.06 sec)

MariaDB [salt]> show tables;
+----------------+
| Tables_in_salt |
+----------------+
| jids           |
| salt_events    |
| salt_returns   |
+----------------+
3 rows in set (0.00 sec)

//Authorized access
MariaDB [salt]> grant all on salt.* to salt@'%' identified by 'salt';
Query OK, 0 rows affected (0.00 sec)

MariaDB [salt]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

Configure minion

[root@minion ~]# vim /etc/salt/minion
.....Omit here N That's ok
mysql.host: '192.168.69.202'
mysql.user: 'salt'
mysql.pass: 'salt'
mysql.db: 'salt'
mysql.port: 3306

[root@minion ~]# systemctl restart salt-minion
Test stored in mysql on Master

[root@master ~]# salt '*' test.ping --return mysql
192.168.69.202:
    True

Query in database

MariaDB [salt]> select * from salt_returns\G
*************************** 1. row ***************************
      fun: test.ping
      jid: 20190307183744856327
   return: true
       id: 192.168.69.202
  success: 1
 full_ret: {"fun_args": [], "jid": "20190307183744856327", "return": true, "retcode": 0, "success": true, "cmd": "_return", "_stamp": "2019-03-07T10:37:45.457140", "fun": "test.ping", "id": "192.168.69.202"}
alter_time: 2019-03-07 18:37:45

2. job cache

2.1 job cache process

In return, Minion directly interacts with the storage server. Therefore, it is necessary to install modules with the specified storage mode on each Minion, such as Python mysql. Can we directly store the returned results on the Master to the storage server?

The answer is yes. This is called job cache. This means that when Minion returns the result to Master, Master will cache the result locally, and then store the cached result to the specified storage server, such as mysql.
Open the master job cache on the master side

[root@master ~]# vim /etc/salt/master
....Omit here N That's ok
master_job_cache: mysql
mysql.host: '192.168.69.202'
mysql.user: 'salt'
mysql.pass: 'salt'
mysql.db: 'salt'
mysql.port: 3306

[root@master ~]# systemctl restart salt-master

Empty table contents in database server

[root@minion ~]# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 20
Server version: 5.5.60-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> delete from salt.salt_returns;
Query OK, 3 rows affected (0.02 sec)

MariaDB [(none)]> select * from salt.salt_returns;
Empty set (0.00 sec)

Test whether it can be stored in the database again on the master

[root@master ~]# salt '*' cmd.run 'df -h'
192.168.69.202:
    Filesystem               Size  Used Avail Use% Mounted on
    /dev/mapper/centos-root   17G  3.9G   14G  23% /
    devtmpfs                 3.9G     0  3.9G   0% /dev
    tmpfs                    3.9G   16K  3.9G   1% /dev/shm
    tmpfs                    3.9G  8.8M  3.9G   1% /run
    tmpfs                    3.9G     0  3.9G   0% /sys/fs/cgroup
    /dev/sda1               1014M  143M  872M  15% /boot
    tmpfs                    781M     0  781M   0% /run/user/0

Query in database

MariaDB [(none)]> select * from salt.salt_returns\G
*************************** 1. row ***************************
       fun: cmd.run
       jid: 20190307184928082837
    return: "Filesystem               Size  Used Avail Use% Mounted on\n/dev/mapper/centos-root   17G  3.9G   14G  23% /\ndevtmpfs                 3.9G     0  3.9G   0% /dev\ntmpfs                    3.9G   16K  3.9G   1% /dev/shm\ntmpfs                    3.9G  8.8M  3.9G   1% /run\ntmpfs                    3.9G     0  3.9G   0% /sys/fs/cgroup\n/dev/sda1               1014M  143M  872M  15% /boot\ntmpfs                    781M     0  781M   0% /run/user/0"
        id: 192.168.69.202
   success: 1
  full_ret: {"fun_args": ["df -h"], "jid": "20190307184928082837", "return": "Filesystem               Size  Used Avail Use% Mounted on\n/dev/mapper/centos-root   17G  3.9G   14G  23% /\ndevtmpfs                 3.9G     0  3.9G   0% /dev\ntmpfs                    3.9G   16K  3.9G   1% /dev/shm\ntmpfs                    3.9G  8.8M  3.9G   1% /run\ntmpfs                    3.9G     0  3.9G   0% /sys/fs/cgroup\n/dev/sda1               1014M  143M  872M  15% /boot\ntmpfs                    781M     0  781M   0% /run/user/0", "retcode": 0, "success": true, "cmd": "_return", "_stamp": "2019-03-07T10:49:28.333981", "fun": "cmd.run", "id": "192.168.69.202"}
alter_time: 2019-03-07 18:49:28
1 row in set (0.00 sec)

2.2 job management

Get jid of task

[root@master ~]# salt '*' cmd.run 'uptime' -v
Executing job with jid 20190307191019053009     //Here is the jid of this command
-------------------------------------------

192.168.69.202:
     19:10:19 up 40 min,  1 user,  load average: 0.00, 0.03, 0.07

Get the return result of this task through jid

[root@master ~]# salt-run jobs.lookup_jid 20190307191019053009
192.168.69.202:
     19:10:19 up 40 min,  1 user,  load average: 0.00, 0.03, 0.07
Published 52 original articles, won praise 11, visited 1182
Private letter follow

Tags: MariaDB MySQL Database Python

Posted on Sat, 01 Feb 2020 21:36:18 -0800 by davidx714