Primary ideas for scripting full-network backup

Article Directory

Preparations: Deploy rsync service

Deploy rsync

Server:

First mileage: confirm software installation
yum install -y rsync
systemctl enable rsyncd Start Up
Second mileage: writing configuration files

Back up before writing any profile!

cp /etc/rsyncd.conf{,.bak}

vi /etc/rsyncd.conf
#rsync_config
#created by HQ at 2017
##rsyncd.conf start##
    
uid = rsync
gid = rsync
port = 873
fake super = yes
use chroot = no
max connections = 200
timeout = 300
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log
ignore errors
read only = false
list = false
hosts allow = 172.16.1.0/24
hosts deny = 0.0.0.0/32
auth users = rsync_backup
secrets file = /etc/rsync.password
[backup]
comment = "backup dir by oldboy"
path = /backup

Restart service after modification:

systemctl restart rsyncd
Third mileage: Create backup directory management user

Create a virtual user

useradd  rsync  -M -s /sbin/nologin  (Can be specified if necessary uid    -u  xxx)
#id rsync
uid=1042(rsync) gid=1046(rsync) groups=1046(rsync)
Fourth mileage: Create a backup data directory and modify the ownership group
[root@backup ~]# mkdir /backup
chown rsync.rsync  /backup 
Fifth Mileage: Create an Access Certification File
vim /etc/rsync.password  
rsync_backup:oldboy123                 

Change file permissions:

[root@backup ~]# chmod 600 /etc/rsync.password 
[root@backup ~]# ll /etc/rsync.password 
-rw------- 1 root root 25 1 August 10:34 /etc/rsync.password
Sixth mileage: Start the backup service program
systemctl start rsyncd
systemctl enable rsyncd

Client:

Remote backup data works (similar to scp command functions)

Pull data: Client downloads data <--Backup server (restore)

rsync parameter remote host address or name: pulled data information directory information to store data locally

rsync 172.16.1.41:/oldgirl/oldgirl01.txt  /tmp
Push Data: Clients upload data --> Backup Server (Backup)

rsync parameter local backup push data remote host address or name: /backup data path information

rsync -avz /oldboy/oldboy.txt  backup:/tmp  
(stay/etc/hosts File Add Domain Name and IP To achieve local resolution)
PS:rsync When pushing catalog data, there is a fundamental difference between having/not having/slash behind the catalog

Push data content under directory with / slash
Push the catalog itself and the data below when there is no/slash

daemon Backup Data Method (Server/Client) daemon (Real-time Backup)

Pull: rsync [OPTION...] [USER@]HOST::SRC...[DEST]

rsync Parameter Remote Authentication User Information@Server Host Name or Address:: Module Information Locally Save Data Path
(Module information is bracketed information in rsync profile/etc/rsyncd.conf and can be modified by itself)

rsync -avz rsync_backup@172.16.1.41::backup /tmp

(Some devices with unknown name cause will fail, module information will be restored followed by a specified path)

rsync -avz rsync_backup@172.16.1.41::backup/tmp/etc/host /tmp

The path directory needs to be slashed: otherwise the directory is gone

rsync -avz rsync_backup@172.16.1.41::backup/tmp/etc/host/ /tmp
Push Data (Upload Backup):Push: rsync [OPTION...] SRC...[USER@]HOST::DEST

rsync parameter data authentication user name @server host name or address that needs to be pushed locally:: module information

rsync -avz /oldboy  rsync_backup@172.16.1.41::backup

Secret-free upload backup:

Password file needs to be created on the client with 600 modification rights

[root@backup ~ 17:22:26]$ vim /etc/rsync.passwd

[root@nfs01 ~]# cat /etc/rsync.passwd
oldboy123

[root@backup ~ 17:23:07]$  chmod 600 /etc/rsync.passwd

[root@nfs01 ~]# ll /etc/rsync.passwd
-rw------- 1 root root 10 Jan  9 17:11 /etc/rsync.passwd

[root@nfs01 ~]# rsync -avz /tmp rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.passwd
sending incremental file list
tmp/
tmp/2233.txt
tmp/.ICE-unix/
tmp/.Test-unix/
tmp/.X11-unix/
tmp/.XIM-unix/
tmp/.font-unix/
tmp/vmware-root_5573-4148014734/
tmp/vmware-root_5836-1003073759/
tmp/vmware-root_5933-1958095513/

Network-wide backup requirements:

1) The backup directory of all servers must be / backup

mkdir -p  /backup 

2) The system configuration files to be backed up include, but are not limited to:

mkdir -p /server/scripts
touch /etc/sysconfig/iptables
crontab -e
* * * * * echo 123 &>/etc/null
Method 1: Compress the absolute path of the data
tar zcvhf /backup/system_bak_`date -d "-1day" +%F_%A`.tar.gz  /var/spool/cron/root  /etc/rc.local  /server/scripts  /etc/sysconfig/iptables
Method 2: Compress the relative path of the data
cd /
tar zcvhf /backup/system_bak_`date -d "-1day" +%F_%A`.tar.gz  ./var/spool/cron/root  ./etc/rc.local  ./server/scripts  ./etc/sysconfig/iptables

3) Web server site directory is assumed to be (/var/html/www)

mkdir -p /var/html/www
cd /
tar zcvhf /backup/web_wwwbak_`date -d "-1day" +%F_%A`.tar.gz ./var/html/www

4) Web server A access log path is assumed to be (/app/logs)

mkdir -p /app/logs
cd /
tar zcvhf /backup/web_logbak_`date -d "-1day" +%F_%A`.tar.gz ./app/logs

5) Web servers can keep backup data for 7 days after packaging (local storage cannot be more than 7 days because too many hard drives are full)

find /backup/ -type f -name "*.tar.gz" -mtime +7|xargs rm -f

6) On the backup server, keep all copies of the data every Monday and within 6 months.

backup server side:
Two ways to keep Monday data information

Method 1: Keep file name information
File name information followed by _date +%F_%A or%w

find /backup/ -type f -name "*.tar.gz" ! -name  "*_Monday.tar.gz" -mtime +180 | xargs rm -rf

Method 2: Monday data can be saved separately

find /backup/ -type f  -name  "*_Monday.tar.gz"|xargs mv -t /backup_Monday
find /backup/ -type f -name "*.tar.gz" -mtime +180 -dalete

7) On the backup server, the backup should be saved as directory according to the internal IP of the backup data server, and the backup files should be saved as time name

Method 1: On the client host to be backed up, create a subdirectory with IP address information in the backup directory

[root@web01 backup 19:03:18]$ mkdir /backup/172.16.1.31
cd /
tar zcvhf /backup/172.16.1.31/system_bak_`date +%F_%A`.tar.gz  ./var/spool/cron/root  ./etc/rc.local  ./server/scripts  ./etc/sysconfig/iptables

rsync -avz /backup/ rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.passwd

Method 2: Create subdirectories using rsync command

rsync -avz /buckup/ rsync_backup@172.16.1.41::backup/172.16.1.31/ --password-file=/etc/rsync.passwd

8) You need to make sure that the backup data is as complete and correct as possible, check the backup data on the backup server, and send the success and failure results information of the backup to the system administrator mailbox

How to send mail: Install on backup server

qq mailbox authorization number:
xxxxxx
Configure Sender Method: Test Mailbox: Sender-->Recipient Sends Message
[root@backup ~]# yum install mailx -y
[root@backup ~]# vim/etc/mail.rc #Last line added
set bsdcompat
set from=673945260@qq.com
set smtp=smtp.qq.com
set smtp-auth-user=673945260@qq.com
set smtp-auth-password=mailbox authorization number

Restart mailbox after configuration:

systemctl restart postfix.service

Send mail command:

mail -s "chenbeiping" "673945260@qq.com" < /etc/hosts
a Verify data transfer integrity

First step: Client generates file fingerprint information

[root@web01 172.16.1.31 19:21:05]$ md5sum system_bak_2020-01-12_Sunday.tar.gz 
5c6c511f3ca98ea8dafa42e769ce97bc  system_bak_2020-01-12_Sunday.tar.gz

Generate fingerprint information

find /backup/ -type f -name "*.tar.gz"|xargs md5sum >/backup/172.16.1.31/web01.md5

Step 2: Compare the file fingerprint information on the server side

[root@backup 172.16.1.31 19:39:40]$ md5sum -c web01.md5 
/backup/172.16.1.31/system_bak_2020-01-12_Sunday.tar.gz: OK
b Mail the verification results to the maintenance personnel
find /backup/ -type f -name "*.md5" |xargs md5sum -c > /tmp/md5.txt
mail -s "backupfile notification" "673945260@qq.com" </tmp/md5.txt

Scripting full network backup:

Configure timer task generation profile before running

crontab -e
* * * * * echo 123 &>/etc/null

And configure secret-free upload of backup password files

Backup service client script:

web01 Server Script

crontab -e
* * * * * echo 123 &>/etc/null
//And configure secret-free upload of backup password files


vim /server/scripts/backup.sh
#!/bin/bash
#01. Create directory information
dir="/backup/"
IP=$(hostname -I|awk  '{print $NF}')
mkdir -p $dir/$IP      /var/html/www   /app/logs
touch /etc/sysconfig/iptables

#02. Package backup data to name it at the current time
cd / && \
tar zchf $dir/$IP/web01system_bak_`date -d "-1day" +%F_%A`.tar.gz  ./var/spool/cron/root  ./etc/rc.local  ./server/scripts  ./etc/sysconfig/iptables
tar zchf $dir/$IP/web01html_bak_`date -d "-1day" +%F_%A`.tar.gz  ./var/html/www/
tar zchf $dir/$IP/web01logs_bak_`date -d "-1day" +%F_%A`.tar.gz  ./app/logs/

#03. Generate the md5 check code for the day
find $dir/ -type f -name "*.tar.gz" -mtime -1 | xargs md5sum >$dir$IP/web01.md5

#04.push rsync
rsync -az $dir rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.passwd

#05.Delete compressed files 7 days ago
#find $dir/ -type f -name "*.tar.gz" -mtime +7|xargs rm -f >/dev/null
find $dir/ -type f -name "*.tar.gz" -mtime +7 -delete

nfs01: Initialize backup script

vim /server/scripts/backup.sh
#!/bin/bash
#01. Create directory information
dir="/backup/"
IP=$(hostname -I|awk  '{print $NF}')
mkdir -p $dir$IP      
touch /etc/sysconfig/iptables

#02. Package backup data to name it at the current time
cd / && \
tar zchf $dir$IP/nfs01system_bak_`date -d "-1day" +%F_%A`.tar.gz  ./var/spool/cron/root  ./etc/rc.local  ./server/scripts  ./etc/sysconfig/iptables

#03.make md5 checkcode
find $dir -type f -name "*.tar.gz"|xargs md5sum >$dir/$IP/nfs01.md5

#04.push rsync
rsync -az $dir rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.passwd

#05.Delete compressed files 7 days ago
#find $dir/ -type f -name "*.tar.gz" -mtime +7|xargs rm -f >/dev/null
find $dir -type f -name "*.tar.gz" -mtime +7 -delete

Backup Service Side Script

mkdir -p /server/scripts/
chown rsync.rsync /backup/

vim /server/scripts/ck.sh
#!/bin/bash

#01.check md5sum
#find /backup/ -type f -name "*.md5" |xargs md5sum -c > /tmp/md5.txt
#cat /tmp/md5.txt|sed -r 's#/backup|/|:# #G'|awk'BEGIN{print'Backup server address information','Backup file information','Backup result information'}{print $0}'|column-t >/tmp/$(date +%F_%A) mail.txt

#01 Verify data integrity and view md5 files generated that day
file_count=$(find /backup -type f -name "*.md5" -mtime -1|wc -l)

if [ $file_count -ne 2 ] 
then
   echo "md5.txt not sent" | mail -s "finger.txt sent error" 673945260@qq.com
   find /backup -type f -name "*.md5"|xargs md5sum -c > /tmp/md5.txt
else
   find /backup -type f -name "*.md5"|xargs md5sum -c > /tmp/md5.txt
fi


#02.send mail
mail -s "backupfile notification" "673945260@qq.com" < /tmp/$(date +%F_%A)mail.txt


#03.delete 180 ago file
find /backup/ -type f -name "*.tar.gz" ! -name  "*_Monday.tar.gz" -mtime +180 | xargs rm -rf

Write timed tasks:

Client timer task time 00:00
0 0 * * * /bin/sh /server/scripts/backup.sh &>/dev/null


Server Timed Task Time 06:00 Later Avoid File Not Ending
0 6 * * *  /bin/sh /server/scripts/ck.sh &>/dev/null
23 original articles published, 0 praised, 474 visited
Private letter follow

Tags: rsync iptables vim Unix

Posted on Wed, 15 Jan 2020 20:01:25 -0800 by 2005